Target tracking method based on camera motion compensation

Abstract

This application discloses a target tracking method based on camera motion compensation. By stitching together the field of view of the moving camera, the images of the moving target are stitched together into a panoramic image. Then, motion compensation is performed on the camera motion interference or moving object interference, thereby improving the accuracy of moving target tracking.

Description

Technical Field

[0001] This application relates to the field of target tracking technology, specifically to a target tracking method based on camera motion compensation. Background Technology

[0002] Recording the movement of moving targets, such as athletes, using video cameras for motion analysis has become widely used. When tracking a moving target, the camera itself is also moving; therefore, the target's motion information includes its motion relative to the surrounding environment and its motion relative to the camera.

[0003] For larger venues, multiple action cameras are typically used to record the motion information of moving targets. Then, relevant algorithms are used to stitch together the fields of view from different cameras, and finally, the motion trajectory of the moving target is plotted onto the stitched field of view. However, moving cameras or other moving objects within a camera's field of view can interfere with the motion trajectory of the currently tracked target. Therefore, when tracking moving targets, it is necessary to remove the motion information of the camera or other moving objects within the camera's field of view. How to remove these interfering factors to improve the tracking performance of moving targets has become a pressing technical problem in this field. Summary of the Invention

[0004] This application aims to improve the accuracy of moving target tracking by removing interference from camera motion or other moving objects when tracking a moving target, and provides a target tracking method based on camera motion compensation.

[0005] To achieve this objective, the following technical solution is adopted in this application:

[0006] A target tracking method based on camera motion compensation is provided, including the following steps:

[0007] S1, stitch together the images captured by the action camera from different rotation angles to obtain a panoramic image ponoImg of the moving target, and then calculate the corner points of the panoramic image ponoImg to obtain the corner point set panoPoints corresponding to ponoImg;

[0008] S2, based on the corner relationship between the current frame and historical frames, estimate the camera motion, and then perform motion interference compensation on the moving target to obtain the true motion information of the moving target;

[0009] S3. Perform corner point matching between the current frame and the panoramic image ponoImg to obtain the positional relationship between the current frame and the panoramic image ponoImg. Map the center coordinates of the moving target in the current frame to the panoramic image space and record the trajectory of the moving target in the panoramic image space.

[0010] Preferably, step S2 specifically includes the following steps:

[0011] S21, after filtering out moving objects in the foreground image of the current frame, the mask image maskFrame of the current frame is obtained; the relationship between the current frame and the panoramic image ponoImg is: the motion camera forms several current frames at different times during the rotation process, and the several current frames are stitched together to form the panoramic image ponoImg;

[0012] S22, perform corner point calculation on the mask image maskFrame to obtain the corner point set ORBPoints corresponding to maskFrame;

[0013] S23, determine whether the mask image maskFrame is the first frame for tracking the moving target.

[0014] If so, update the corner point set ORBPoints to the historical corner point set histORBPoints; update the current frame to the historical frame histFrame, and then jump to step S3;

[0015] If not, proceed to step S24;

[0016] S24, Calculate the affine matrix H of the current frame relative to the previous frame based on ORBPoints and histORBPoints. motion H motion Used to describe the relative positional relationship between the camera field of view of the current frame and the camera field of view of the previous frame;

[0017] S25, for H motion The prediction operator in the middle is corrected, and the camera motion interference or moving object interference of the moving target is compensated based on the correction result. The corner point set ORBPoints associated with the current frame is updated to the historical corner point set histORBPoints.

[0018] Preferably, moving objects in the foreground image of the current frame are filtered out by masking.

[0019] Preferably, the corner points of the current frame or the mask image maskFrame are calculated using the ORB algorithm.

[0020] Preferably, the affine matrix H motion Expressed as:

[0021]

[0022] in, Represents a 2x2 linear transformation matrix; , These represent the offsets of the moving target from its true position in the horizontal and vertical directions, respectively.

[0023] Preferably, in step S25, the method for compensating for camera motion interference or moving body interference of the moving target is as follows:

[0024] Construct the correction matrix expressed by equation (1) as follows:

[0025] (1)

[0026] Construct the offset matrix expressed by equation (2) below. :

[0027] (2)

[0028] The Kalman observations are updated using the following formula (3). and :

[0029]

[0030]

[0031] express The transpose of a matrix, where T operation denotes the matrix transpose operation;

[0032] This represents the 8x8 variance matrix in Kalman predictions;

[0033] according to and Update the Kalman state parameters of the moving target to compensate for motion disturbances caused by camera movement.

[0034] Preferably, step S3 specifically includes the following steps:

[0035] S31, calculate the corner points of the mask image maskFrame of the current frame to obtain the corner point set curPoints of the current frame;

[0036] S32, calculate the affine matrix H between the panoramic image pnoImg and the current frame based on the corner point set panoPoints and the corner point set curPoints. pano ;

[0037] S33, the set of center points of each moving target detected in the current frame, ObjCenters: {imgPT0,imgPT1,...,imgPT...} N Each point in} imgPT n ( , ), n={0,1,...,N}, points affine to the panoramic image space coordinate system globalPT n ( , ); imgPT n ( , ), globalPT n ( , ) represent the coordinates of the moving target in the coordinate system of the current frame and the coordinates of the target in the image coordinate system of the panoramic image ponoImg, respectively, and N represents the total number of center points;

[0038] If it is the first frame of a moving target, then update the coordinates of the center point of each tracked target. To the historical movement point And cache the trajectory; otherwise, proceed to step S34;

[0039] S34, Calculate the coordinates of the center point of each tracked moving target. Coordinates of the historical motion points of the moving target distance and judge Is it less than the preset distance threshold?

[0040] If so, then update the coordinates of the historical movement point as follows:

[0041]

[0042]

[0043] If not, the history will not be updated;

[0044] Then, put the corresponding Track caching is implemented so that the athlete's track length can be calculated at the end.

[0045] Preferably, the affine method in step S33 is expressed as follows:

[0046]

[0047] in, , These represent the horizontal and vertical coordinates of the moving target in the ponoImg panoramic image, respectively.

[0048] , These represent the center coordinates of the detected target within the current frame;

[0049] It is a 2x2 real number rotation matrix; , These represent the horizontal and vertical offsets of the moving target, respectively.

[0050] Preferably, .

[0051] This application stitches together the field of view of the action camera to create a panoramic image of the moving target. Then, by performing motion compensation for camera motion interference or moving object interference, the accuracy of moving target tracking is improved. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly described below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a diagram illustrating the implementation steps of the target tracking method based on camera motion compensation provided in this embodiment. Detailed Implementation

[0054] The technical solution of this application will be further described below with reference to the accompanying drawings and specific embodiments.

[0055] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual images. They should not be construed as limiting the scope of this application. To better illustrate the embodiments of this application, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0056] In the accompanying drawings of the embodiments of this application, the same or similar reference numerals correspond to the same or similar components. In the description of this application, it should be understood that if terms such as "upper," "lower," "left," "right," "inner," and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting this application. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0057] In the description of this application, unless otherwise expressly specified and limited, the term "connection" or similar designation indicating a connection between components should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral part; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication between two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0058] The target tracking method based on camera motion compensation provided in the embodiments of this application is as follows: Figure 1 As shown, the steps include:

[0059] S1. Stitch together the images captured by the action camera from different rotation angles to obtain a panoramic image of the moving target, ponoImg. Then, use the ORB (Oriented FAST and Rotated BRIEF) algorithm to calculate the corner points of the panoramic image ponoImg to obtain the corner point set panoPoints corresponding to ponoImg.

[0060] In this embodiment, the method for stitching images captured by the action camera from different perspectives adopts the existing image stitching algorithm based on ORB features, which will not be described in detail.

[0061] The method for calculating corner points in the panoramic image ponoImg using the ORB (Oriented FAST and Rotated BRIE) corner extraction algorithm is as follows: ORB features are extracted using operators provided by OpenCV. ORB is a relatively fast and stable feature point detection and extraction algorithm. ORB uses the FAST (features from accelerated segment test) algorithm to detect feature points. The core idea of ​​the FAST algorithm is to compare the current point with surrounding points. If most of the surrounding points are different from the current point, it can be considered a feature point. The calculation process of ORB feature corner points is briefly described below:

[0062] First, a multi-resolution image pyramid is used, setting the scale factor (default 1.2) and the number of pyramid layers n (usually 8). The source image is progressively scaled down according to the scale factor, generating a total of N images. In the constructed scale space, FAST feature point detection is performed on each image. The sum of feature points extracted from the N images at different scales is used as the FAST feature points of the input image.

[0063] Secondly, for the current detection point P, determine whether it is a FAST feature point. Using P as the center and R as the radius, uniformly select M surrounding points in the image and compare them with P. If the absolute value of the grayscale difference between point P and its surrounding points is greater than a threshold t, the two points are considered different. If K consecutive points among the M surrounding points are different from point P, then point P is considered a feature point. Preferably, M is 16, K is 9, and t is 45.

[0064] Finally, ORB uses the moments of the image to achieve rotation invariance and completes the description of feature point attributes, thus completing the ORB feature corner point extraction of the image.

[0065] After obtaining the ORB feature corner point set panoPoints, as follows Figure 1 As shown, the target tracking method based on camera motion compensation provided in this embodiment proceeds to the following steps:

[0066] S2, based on the corner relationship between the current frame and historical frames, estimate the camera motion, and then perform motion interference compensation on the moving target to obtain the true motion information of the moving target;

[0067] The motion interference compensation method specifically includes the following steps:

[0068] S21, after filtering out moving objects in the foreground image of the current frame through masking, the mask image maskFrame of the current frame is obtained; the moving objects in the foreground image of the current frame are all tracked moving targets, such as the currently tracked athlete 'a', as well as spectators walking or standing on the sidelines. The relationship between the current frame and the panoramic image ponoImg is as follows: the moving camera forms several current frames at different moments during the rotation process, and these several current frames are stitched together to form the panoramic image ponoImg;

[0069] The method of filtering out moving objects in the foreground image of the current frame using masking is briefly described below:

[0070] The pedestrian target detection algorithm detects a set of bounding boxes (Rects) of human body regions: {rt0, rt1, ..., rt N}, where rt n (x, y, width, height) represents the top-left (x, y) and width (width, height) of the rectangle containing the foreground pedestrian in the frame image. For each rt in the set of rectangles (Rects) of the current frame... n The pixel values ​​in the region n = {0, 1, ..., N} are all set to white or black to obtain a maskFrame. N represents the total number of moving objects in the foreground image.

[0071] S22, calculate the corner points of the mask image maskFrame to obtain the corner point set ORBPoints corresponding to maskFrame;

[0072] In this embodiment, the corner points of the mask image maskFrame are also calculated using the ORB algorithm. The calculation process is the same as that for the corner points of the panoramic image ponoImg, and will not be described again.

[0073] S23, determine whether the mask image maskFreme is the first frame for tracking the moving target;

[0074] If so, update the corner point set ORBPoints to the historical corner point set histORBPoints; update the current frame to the historical frame histFrame, and then jump to step S3;

[0075] If not, proceed to step S24;

[0076] S24, Calculate the affine matrix H between the current frame and the historical frame histFrame based on ORBPoints and histORBPoints. motion H motionUsed to describe the relative positional relationship between the camera field of view of the current frame and the camera field of view of the previous frame;

[0077] Affine transformation maps points in one plane to a two-dimensional projection onto another plane. Affine transformations preserve the "flatness" of the two-dimensional shape; that is, where there were lines before, they remain lines. An affine matrix is ​​a 3x3 matrix, mathematically represented as follows:

[0078]

[0079] An affine transformation is a linear transformation from two-dimensional coordinates (x, y) to two-dimensional coordinates (u, v). The corresponding homogeneous coordinate matrix representation is as follows:

[0080]

[0081] Among them, h20 and h21 are set to 0, and h22 is set to 1.

[0082] Corresponding coordinate transformation relationship:

[0083]

[0084]

[0085] The affine matrix only needs to be solved , , , , , These 6 parameters are sufficient. To solve for the affine transformation relationship between two images, as long as there are more than 3 matching pairs of ORB corner points corresponding to the two images, the affine transformation matrix of the two images can be obtained. The solution method can be the least squares method, and the specific process will not be elaborated here.

[0086] For ease of subsequent description, let the 3x3 affine matrix H be... motion Expressed as:

[0087]

[0088] in, Denotes the 2x2 linear transformation matrix in the affine matrix. , =h 02 , =h 12 These represent the offsets of the moving target in the horizontal x-axis direction and the vertical y-axis direction, respectively.

[0089] This embodiment only analyzes two-dimensional affine transformations. The corresponding affine matrix H... motionThe calculation employs the existing RANSAC (RANdom Sampling Consensus) algorithm, an iterative algorithm that correctly estimates the parameters of a mathematical model from a set of data containing "outliers." "Outliers" refer to noise in the data, such as mismatches in matching and outliers in the estimated curve. Therefore, RANSAC is also an outlier detection algorithm. Furthermore, RANSAC is a nondeterministic algorithm; in a sense, it produces a reasonable result with a certain probability, allowing for more iterations to increase its probability.

[0090] H motion The RANSAC calculation process is existing technology and will not be elaborated here.

[0091] S25, according to H motion The values ​​in the table are used to correct the Kalman linear prediction operator in the multi-target tracking algorithm. The correction process is briefly described below:

[0092] In this embodiment, the Deepsort multi-target tracking algorithm is used for athlete tracking. The algorithm employs a Kalman linear motion prediction operator for each tracked target. The prediction principle is to predict the possible location of the moving target in the next frame based on its historical and current positions. This is achieved through H... motion The camera's motion is estimated, and the Kalman predictor is modified to restore the athlete's true position.

[0093] In tracking algorithms, Kalman state estimation uses an 8-dimensional space to characterize the state of the trajectory at a certain moment. These represent the center position, aspect ratio, height, and corresponding variable changes in the image coordinates of the bounding box used to select the moving target; Kalman prediction employs a uniform velocity model and a linear observation model: the corresponding observation variable is the center position. Aspect Ratio ,high This ultimately yields the predicted location. Correspondingly... These observations are directly used as Kalman predictions. The true motion position is then calculated using camera motion estimation. .

[0094] When making corrections, first use the previously calculated 2x2 affine rotation matrix R. motion_2X2 ,as well as The Kalman state parameters are corrected by offsets in the horizontal and vertical directions to complete parameter compensation for camera motion during tracking. The method is as follows:

[0095] First, the Kalman prediction algorithm, for the aforementioned eight dimensions, consists of two parts: the mean and the variance. The mean is a 1*8 matrix, denoted as... A matrix with variance of 8x8 is denoted as . .

[0096] Construct the correction matrix as follows:

[0097]

[0098] Construct the corrected offset matrix as follows:

[0099]

[0100] Then, the updated Kalman observations are obtained. and The corresponding calculation formula is:

[0101]

[0102]

[0103] express The transpose of a matrix, where T operation denotes the matrix transpose operation;

[0104] This represents the 8x8 variance matrix in Kalman predictions;

[0105] Finally, after the calculation is complete, update the Kalman state parameters of the trajectory in the multi-target deepsort tracking algorithm; then use the new parameters for trajectory prediction. Specifically, according to and The method for updating the Kalman state parameters of a moving target falls under the category of multi-target tracking and will not be elaborated here.

[0106] After compensating for camera motion interference or moving object interference on the moving target, the target tracking method based on camera motion compensation provided in this embodiment is as follows: Figure 1 As shown, proceed to the following steps:

[0107] S3, based on the interference compensation results for the moving target in step S2, estimates the tracking trajectory of the moving target, specifically including the following steps:

[0108] S31, perform corner point matching between the current frame and the panoramic image ponoImg to obtain the positional relationship between the current frame and the panoramic image ponoImg, map the center coordinates of the moving target in the current frame to the panoramic image space, and record the trajectory of the moving target in the panoramic image space.

[0109] Specifically, corner points are calculated on the mask image maskFrame of the current frame to obtain the corner point set curPoints of the current frame;

[0110] The method for calculating corner points for the current frame is the same as the method for calculating corner points for the panoramic image ponoImg, and will not be repeated here.

[0111] S32, calculate the affine matrix H between the panoramic image pnoImg and the current frame based on the corner point set panoPoints and the corner point set curPoints. pano ;

[0112] Affine matrix H pano It is a 3x3 matrix, and the solution process is the same as that for H above. moiton The solution principle is the same, so I won't go into details again. For simplicity, H... pano The expression form is:

[0113]

[0114] This represents a 2x2 rotation and scaling matrix. These represent the offsets of the moving target in the horizontal x-direction and the vertical y-direction, respectively.

[0115] H pano The calculation is also solved using the RANSAC algorithm, which will not be described in detail here.

[0116] S33, the target center ObjCenters detected in the current frame: {imgPT0,imgPT1,...,imgPT N Each center point imgPT in} n ( , ), n={0,1,...,N}, points affine to the panoramic image space coordinate system globalPT n ( , The affine method is expressed as:

[0117]

[0118] in, , These represent the horizontal and vertical coordinates of the moving target in the ponoImg panoramic image, respectively. , These represent the horizontal and vertical coordinates of the moving target in the current frame, respectively.

[0119] imgPT n ( , ), globalPT n ( , ) represent the coordinates of the moving target in the coordinate system of the current frame and the coordinates of the target in the image coordinate system of the panoramic image ponoImg, respectively.

[0120] If it is the first frame of a moving target, then update the coordinates of the center point of each tracked target. To the historical movement point And cache the trajectory; otherwise, proceed to step S34;

[0121] S34, Calculate the coordinates of the center point of each tracked moving target. The coordinates of the moving point in the history of the moving target (the previous frame of the current frame). distance and judge Is it less than the preset distance threshold?

[0122] If so, then update the coordinates of that historical movement point as follows:

[0123]

[0124]

[0125] If not, the historical movement point information will not be updated.

[0126] Put the corresponding Track caching is implemented so that the athlete's track length can be calculated at the end;

[0127] The historical motion point in step S34 is ,and The distance calculation formula is:

[0128]

[0129] In summary, this application stitches together the field of view of the action camera to create a panoramic image of the moving target, and then improves the accuracy of moving target tracking by performing motion compensation for camera motion interference or moving object interference.

[0130] It should be stated that the above-described specific embodiments are merely preferred embodiments and technical principles applied in this application. Those skilled in the art should understand that various modifications, equivalent substitutions, and variations can be made to this application. However, such variations, as long as they do not depart from the spirit of this application, should be within the scope of protection of this application. Furthermore, some terminology used in this application's specification and claims is not limiting but merely for ease of description.

Claims

1. A target tracking method based on camera motion compensation, characterized in that, Including the following steps: S1, stitch together the images captured by the action camera from different rotation angles to obtain a panoramic image ponoImg of the moving target, and then calculate the corner points of the panoramic image ponoImg to obtain the corner point set panoPoints corresponding to ponoImg; S2, based on the corner relationship between the current frame and historical frames, estimate the camera motion, and then perform motion interference compensation on the moving target to obtain the true motion information of the moving target; S3, perform corner point matching between the current frame and the panoramic image ponoImg to obtain the positional relationship between the current frame and the panoramic image ponoImg, map the center coordinates of the moving target in the current frame to the panoramic image space, and record the trajectory of the moving target in the panoramic image space. Step S2 specifically includes the following steps: S21, after filtering out moving objects in the foreground image of the current frame, the mask image maskFrame of the current frame is obtained; the relationship between the current frame and the panoramic image ponoImg is: the motion camera forms several current frames at different times during the rotation process, and the several current frames are stitched together to form the panoramic image ponoImg; S22, perform corner point calculation on the mask image maskFrame to obtain the corner point set ORBPoints corresponding to maskFrame; S23, determine whether the mask image maskFrame is the first frame for tracking the moving target. If so, update the corner point set ORBPoints to the historical corner point set histORBPoints; update the current frame to the historical frame histFrame, and then jump to step S3; If not, proceed to step S24; S24, Calculate the affine matrix H of the current frame relative to the previous frame based on ORBPoints and histORBPoints. motion H motion Used to describe the relative positional relationship between the camera field of view of the current frame and the camera field of view of the previous frame; S25, for H motion The prediction operator in the middle is corrected, and the camera motion interference or moving object interference of the moving target is compensated based on the correction result. The corner point set ORBPoints associated with the current frame is updated to the historical corner point set histORBPoints.

2. The target tracking method based on camera motion compensation according to claim 1, characterized in that, Moving objects in the foreground image of the current frame are filtered out by masking.

3. The target tracking method based on camera motion compensation according to claim 1, characterized in that, Corner point calculation is performed on the current frame or the mask image maskFrame using the ORB algorithm.

4. The target tracking method based on camera motion compensation according to claim 1, characterized in that, The affine matrix H motion Expressed as: ； in, Represents a 2x2 linear transformation matrix; , These represent the offsets of the moving target from its true position in the horizontal and vertical directions, respectively.

5. The target tracking method based on camera motion compensation according to claim 4, characterized in that, In step S25, the method for compensating for camera motion interference or moving body interference of the moving target is as follows: Construct the correction matrix expressed by equation (1) as follows: （1） Construct the offset matrix expressed by equation (2) below. : （2） The Kalman observations are updated using the following formula: and : ； ； express The transpose of a matrix, where T operation denotes the matrix transpose operation; This represents the 8x8 variance matrix in Kalman predictions; according to and Update the Kalman state parameters of the moving target to compensate for motion disturbances caused by camera movement.

6. The target tracking method based on camera motion compensation according to claim 1, characterized in that, Step S3 specifically includes the following steps: S31, calculate the corner points of the mask image maskFrame of the current frame to obtain the corner point set curPoints of the current frame; S32, based on the corner point set panoPoints and the corner point set curPoints, calculate the affine matrix H between the panoramic image pnoImg and the current frame. pano ; S33, the set of center points of each moving target detected in the current frame, ObjCenters: {imgPT0,imgPT1,...,imgPT...} N Each point in} imgPT n ( , ), n={0,1,...,N}, points affine to the panoramic image space coordinate system globalPT n ( , ); imgPT n ( , ), globalPT n ( , ) represent the coordinates of the moving target in the coordinate system of the current frame and the coordinates of the target in the image coordinate system of the panoramic image ponoImg, respectively, and N represents the total number of center points; If it is the first frame of a moving target, then update the coordinates of the center point of each tracked target. To the historical movement point ; And perform trajectory caching; Otherwise, proceed to step S34; S34, Calculate the coordinates of the center point of each tracked moving target. Coordinates of the historical motion points of the moving target distance and judge Is it less than the preset distance threshold? If so, then update the coordinates of the historical movement point as follows: ； ； If not, the history will not be updated; Then, put the corresponding Track caching is implemented so that the athlete's track length can be calculated at the end.

7. The target tracking method based on camera motion compensation according to claim 6, characterized in that, The affine method in step S33 is expressed as follows: ； in, , These represent the horizontal and vertical coordinates of the moving target in the ponoImg panoramic image, respectively. , These represent the center coordinates of the detected target within the current frame; It is a 2x2 real number rotation matrix; , These represent the horizontal and vertical offsets of the moving target, respectively.

8. The target tracking method based on camera motion compensation according to claim 6, characterized in that, 。

Images