Visual object alignment method based on global motion

Abstract

The present invention proposes a visual target alignment method based on global motion, so as to make the training of curling athletes more scientific and technological, and to better observe the complete motion path of curling. This method can obtain the tracking data of curling in the video by analyzing the curling video, and can obtain the movement of the camera lens by calculation, and use the movement to restore the venue in the video, and then obtain the motion data of the curling venue. The venue motion data and the curling tracking data are combined, and finally the complete curling motion trajectory is calculated and the visualization result is output. The feasibility of this method has been verified by testing, and it has important application value.

Description

Technical Field

[0001] The invention relates to the field of image processing, and more particularly to global motion-based visual target alignment. The method sequentially inputs images based on a curling game video shot by a dedicated camera set up in a stadium, and obtains the curling tracking trajectory through target tracking and lens motion restoration. Background Art

[0002] Video is composed of one or more shots, and each shot is composed of continuous frames. Therefore, the problem of obtaining the trajectory of curling can be converted into the problem of obtaining the position coordinates of the curling stone in each video frame and the problem of obtaining the motion of the lens. The position coordinates of the curling stone should be obtained through the target detection algorithm. At present, great progress has been made in the research on target detection. The task of target detection is to find all the targets (objects) of interest in the image and determine their categories and positions. It is one of the core problems in the field of computer vision. Since various objects have different appearances, shapes, and postures, coupled with the interference of factors such as illumination and occlusion during imaging, target detection has always been the most challenging problem in the field of computer vision. The methods of target detection can be divided into two categories. One is to use traditional methods, such as frame difference method, background subtraction method, optical flow method, etc. Most of these traditional target detection methods are calculation methods based on manual features. The other is the target detection algorithm based on deep learning. Compared with traditional target detection algorithms, deep learning can extract higher-level and better features that express targets, and can also integrate feature extraction, selection and classification in one model.

[0003] The lens motion information can be obtained through optical flow, but considering the particularity of the curling venue, during the shooting process, the screen will be occupied by the white venue for most of the time, and the movement of the lens cannot be well reflected. Therefore, the optical flow extracted from such a video cannot express the global motion. Therefore, subsequent calculations cannot be performed. Summary of the invention

[0004] To solve the above problems, the present invention provides a visual target alignment method based on global motion. The target detection part is implemented using a deep learning algorithm, and the estimation of the lens motion is achieved by matching the corner points between frames, thus achieving a good visual target alignment effect.

[0005] The specific steps of the present invention are as follows:

[0006] Step 1: For the input video, read it in the form of a video frame sequence, perform target detection on the curling stones appearing in the picture, and obtain the position coordinates of each curling stone in each frame.

[0007] Step 2: Perform global motion estimation on the input video and detect the corners of each frame of the video. Generally speaking, corners are extreme points, that is, points with particularly prominent properties in some aspects. The basic idea of ​​corner detection is to use a fixed window (a neighborhood window of a pixel) to slide in any direction on the image, and compare the grayscale change of the pixels in the window before and after sliding. If there is a large grayscale change in sliding in any direction, then we can think that there is a corner in the window. Then pair the corners detected between every two frames, and calculate the affine transformation matrix between every two frames based on the successfully paired corners. Extract the global motion variables of the next frame relative to the previous frame from the affine transformation matrix, that is, the horizontal displacement, vertical displacement, tilt angle, and scaling. After the global motion variables between every two frames of the video graphic frame sequence are output in sequence, the global motion path of the camera can be estimated through subsequent calculations.

[0008] Step 3: Mark the position coordinates of the center of the curling scoring area in the last frame of the video, and use the global motion variables between the two frames obtained in step 2 to calculate the position coordinates of the center of the scoring area in the previous frame and output them. And so on, starting from the last frame of the video, the position coordinates of the center point of the scoring area in each frame are finally obtained.

[0009] Step 4: Based on step 3, a coordinate system is established with the center point of the curling scoring area as the origin. The x-axis value of the coordinate system increases from left to right, and the y-axis value increases from top to bottom. Subtract the position coordinates of the curling stone in each frame obtained in step 1 from the position coordinates of the center point of the scoring area in the frame, and the result is the coordinates of the curling stone relative to the center point of the scoring area. After processing each frame in this way, all the results are plotted on a panoramic view of the curling competition venue to obtain visual data of the complete motion trajectory of the curling stone.

[0010] The present invention has the following advantages:

[0011] The affine transformation matrix between two frames is calculated by pairing corner points between frames, and the global motion variable of the latter frame relative to the previous frame is extracted from the affine transformation matrix. Using this method, the problem that the optical flow method cannot express the global motion from the curling video is solved with high accuracy of the final result, and the interference of the shooting angle on viewing the curling motion trajectory can be eliminated. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Schematic diagram of corner point matching between two frames. DETAILED DESCRIPTION

[0013] The present invention proposes a visual target alignment method based on global motion. The specific implementation steps of the invention are as follows:

[0014] Step 1: Read the video. The video is divided into frames to obtain video frame images. The siamRPN++ deep learning algorithm is used to detect the target for each image to obtain the position coordinates of the curling stone. The siamRPN algorithm is a twin network structure based on RPN, which consists of two parts: Siamese Network and RPN. The former is used to extract features from video frames, and the latter is used to generate candidate regions. Among them, the RPN network consists of two branches, one is the classification branch used to distinguish between targets and backgrounds, and the other is the regression branch for fine-tuning candidate regions. On this basis, the siamRPN++ algorithm has been further improved and innovated to make the algorithm perform better. For the curling stone detected in the video frame, the algorithm will use the coordinates of the four corners of the circumscribed rectangular box of the curling stone as the output result, continue to process the result, obtain the coordinates of the center position of the rectangular box, and regard it as the coordinates of the center position of the curling stone.

[0015] Step 2: Perform Shi-Tomasi corner point detection on the video frame image, where the maximum number of corner points is set to 500. This number is set taking into account the particularity of curling videos. Most of the area in the video screen is white curling field. If the number of detected corner points is too small, it will cause large errors in the subsequent global motion estimation. Corner points are automatically ignored when the corner point quality is below 0.01. If the corner point quality threshold is set too high, there will be too few corner points that meet the conditions, affecting subsequent calculations. The minimum Euclidean distance between any two corner points is 12, and a neighborhood with a distance of 12 is taken from the corner point as the center. The above four parameter settings are a set of parameters with better effects determined after comparing the experimental results of multiple sets of parameters.

[0016] The corner point quality evaluation method is as follows:

[0017]

[0018] Where w(x, y) is the window function, I(x, y) is the intensity of the point, I(x+u, y+v) is the displacement intensity of the point, and E(u, v) is the grayscale change of the image in the direction [u, v]. Corner point detection requires maximizing the value of E(u, v).

[0019] From I(x+u,y+v)=I(x,y)+I x u+I y v+O(u 2 +v 2 ), among which, O(u 2 +v 2) represents the expansion of I(x+u, y+v) and the result obtained by dividing I(x, y)+I x u+I y The size of the remainder other than v can be ignored. Therefore, we can get:

[0020]

[0021] Will Denoted as Δ, then,

[0022]

[0023]

[0024] Here I x and I y are the derivatives in the x and y directions. Let λ 1 ,λ 2 is the eigenvalue of M, then the scoring function is:

[0025] R=min(λ 1 ,λ 2 )

[0026] If the score R exceeds the set corner point quality evaluation value, it is considered to be a corner point.

[0027] Then, the corner points detected between every two frames are paired, and the affine transformation matrix between every two frames is calculated based on the successfully paired corner points. In the present invention, a four-parameter affine transformation model is used to estimate global motion, and the model of the affine transformation matrix A is shown in equation (1).

[0028]

[0029] Where x and y represent the horizontal and vertical displacement of the next frame image relative to the previous frame image, s represents the zoom scale, and θ represents the tilt angle.

[0030] After replacing and simplifying the parameters in the affine transformation matrix, the global motion variable of the next frame relative to the previous frame is obtained. The model of the global motion variable matrix B is shown in equation (2).

[0031]

[0032] Among them, a is equal to s*cosθ in the affine transformation matrix, b is equal to -s*sinθ in the affine transformation matrix, c is equal to x in the affine transformation matrix, d is equal to s*sinθ in the affine transformation matrix, e is equal to s*cosθ in the affine transformation matrix, and f is equal to y in the affine transformation matrix.

[0033] The global motion variable matrix between every two frames of the video is output sequentially for subsequent estimation of the global motion.

[0034] Step 3: Manually obtain the position coordinates of the center point of the curling scoring area in the last frame of the video in the video frame image, and use the global motion variables between the last frame and the second to last frame in the result of step 2 to calculate the coordinates. The calculation process is shown in equations (3) and (4).

[0035] x 2 =ax 1 +by 1 +c (3)

[0036] y 2 =dx 1 +ey 1 +f (4)

[0037] The two unknowns on the right side of the equation (x 1 ,y 1 ) is the position coordinate of the center point of the curling scoring area in the next frame of the input, a, b, c, d, e, f are the same as the parameters a, b, c, d, e, f in the global motion variable matrix B between the next frame and the previous frame, respectively. The two unknowns on the left side of the equation (x 2 ,y 2 ) is the position coordinate of the center point of the curling scoring area in the previous frame. Therefore, the position coordinate of the center point of the curling scoring area in the last frame is used as input to calculate the result on the left side of the equation (x 2 ,y 2 ) is the position coordinate of the center point of the curling scoring area in the penultimate frame. Then, the global motion variables between the penultimate and third-to-last frames of the video are used to process the position coordinate of the center point of the curling scoring area in the penultimate frame in the same way, and the position coordinate of the center point of the curling scoring area in the third-to-last frame is obtained. By analogy, the position coordinate of the center point of the curling scoring area in the first frame of the video can be calculated, and the position coordinate of the center point of the curling scoring area in all frames of the video has been obtained.

[0038] Step 4: Establish a coordinate system with the center point of the curling scoring area as the origin. The x-axis value of the coordinate system increases from left to right, and the y-axis value increases from top to bottom. Subtract the center position coordinates (x, y) of the curling stone in each frame obtained in step 1 from the position coordinates (x′, y′) of the center point of the curling scoring area in the current frame obtained in step 3 in the x direction and y direction respectively. The obtained (xx′, yy′) is the position coordinates of the curling stone relative to the center point of the scoring area. In this way, the center position coordinates of the curling stone in the current frame and the position coordinates of the center point of the curling scoring area are used to calculate in each frame, and the position coordinates of the curling stone relative to the center point of the curling scoring area in each frame can be obtained. All the results are drawn on the panoramic view of the curling venue, so as to realize the restoration and visualization of the complete motion trajectory of the curling stone.

[0039] In order to verify the effectiveness of the present invention, the method of the present invention was experimented on a curling competition video, and the visualization result was compared with the original video. It can be found that the result obtained by the present method is basically consistent with the original result.

Claims

1. A visual target alignment method based on global motion, characterized in that: The following steps are involved: (1) Tracking the curling stone in the curling video to obtain the pixel coordinates of the curling stone in each frame; (2) Pair the corner points between video frames to obtain the affine transformation matrix, and then estimate the global motion data of the camera in the video. (3) Manually obtain the coordinates of the center point of the curling scoring area in the last frame of the video, and process them with global motion data to obtain the position coordinates of the center point of the scoring area in all frames; (4) Combining the curling stone position data and the curling stone scoring area center point position data, the complete motion trajectory of the curling stone on the field is calculated, avoiding the interference caused by the camera movement and achieving visual target alignment; In step (2), the method for estimating global motion is as follows: detecting corner points for each frame of the video using the Shi-Tomasi algorithm; The corner point quality evaluation method is as follows: Where w(x,y) is the window function, I(x,y) is the intensity of the point, I(x+u,y+v) is the displacement intensity of the point, and E(u,v) is the grayscale change of the image in the direction [u,v]. Corner point detection requires maximizing the value of E(u,v). From I(x+u,y+v)=I(x,y)+I x u+I y v+Ο(u 2 +v 2 ), where, Ο(u 2 +v 2 ) represents the expansion of I(x+u,y+v) and the result obtained by dividing I(x,y)+I x u+I y The remainder other than v is negligible; therefore, we get: Will Denoted as Δ, then, Here I x and I y are the derivatives in the x and y directions; let λ1,λ2 be the eigenvalues ​​of M, then the scoring function is: R=min(λ1,λ2) If the score R exceeds the set corner point quality evaluation value, it is considered to be a corner point; the set corner point quality evaluation value is set to 0.01; Then, based on the successfully paired corner points between the two frames, the affine transformation matrix between the two frames is estimated, from which the global motion variables of the latter frame relative to the previous frame are extracted, including horizontal displacement, vertical displacement, tilt angle, and scaling.

2. The method according to claim 1, characterized in that In step (1), the specific method for target tracking of the curling stone is as follows: target detection of the curling stone is performed using the siamRPN++ deep learning algorithm, which directly tracks the curling stone in each frame of the video to obtain the position of the curling stone in the current image, and returns the coordinate values ​​of the four corners of the curling stone detection rectangle in each frame; the result is further processed to obtain the coordinates of the center position of the rectangular frame, and is regarded as the coordinates of the center position of the curling stone.

3. The method according to claim 1, characterized in that In step (3), the manually acquired position coordinates of the center point of the curling scoring area are processed using the global motion variables obtained in step (2) to obtain the position coordinates of the center point of the scoring area in the previous frame; and so on, until the position coordinates of the center point of the scoring area in all frames of the video are obtained.

4. The method according to claim 1, characterized in that In step (4), the curling position data obtained in step (1) is processed to obtain the center position coordinates of the curling detection frame, and the data is combined with the position coordinates of the center point of the curling scoring area in each frame obtained in step (3) to calculate the relative position coordinates of the curling relative to the center of the scoring area in each frame of the video; the result is plotted on a panoramic view of the curling competition venue to obtain visual data of the complete motion trajectory of the curling.

Images