Target tracking-oriented low-delay camera motion estimation and compensation method

Abstract

The invention provides a target tracking-oriented low-delay camera motion estimation and compensation method, and solves the technical problem of time delay in existing camera motion compensation. The method comprises the steps that an image sequence is acquired, two threads including a motion estimation thread and a target tracking thread are initialized, and the two threads are carried out synchronously; the motion estimation thread comprises the following steps: extracting feature points of front and rear frames of images, and performing feature matching to obtain a feature matching result; performing bidirectional optical flow calculation by using the feature points to obtain a bidirectional optical flow matching result, and performing correspondence with a feature matching result to obtain corresponding matching point pairs; calculating a homography matrix according to the corresponding matching point pairs between the front frame and the rear frame, updating the Kalman filter and predicting the homography matrix of the next frame; and the target tracking thread compensates and adjusts the search area of the next frame by using the homography matrix of the next frame. The method can be widely applied to the technical field of target tracking.

Description

Technical Field

[0001] The present application belongs to the technical field of target tracking, and more specifically, to a low-latency camera motion estimation and compensation method for target tracking. Background Art

[0002] With the rapid development of computer vision technology, target tracking has been widely used in video surveillance, unmanned driving, robot vision and other fields. In actual application scenarios, the camera often needs to perform translation, rotation and other movements as the target moves in order to keep tracking the target. However, the movement of the camera will cause additional visual motion of the target on the image plane. This motion is superimposed on the actual motion of the target itself, which seriously affects the accuracy and robustness of target tracking. Therefore, how to accurately estimate and compensate for the target visual motion caused by camera motion has become a key technical problem that needs to be solved urgently.

[0003] The existing technical solutions mainly use the following methods to perform camera motion compensation:

[0004] For example, the Chinese invention patent with publication number CN118052851A proposes a method for multi-target tracking of infrared images based on motion estimation. This method uses an image feature extractor to extract feature points and descriptors of two compensated frames, and uses feature points to perform bidirectional optical flow calculation. The accurate image feature point matching relationship obtained by bidirectional optical flow and feature matching is used to calculate the homography matrix using the matching feature points between the two frames, and the background motion between the two frames is further accurately estimated, and motion compensation is completed based on the homography matrix. Although this method can estimate camera motion and perform motion compensation more accurately, it starts calculations only after the image is acquired for each frame. Its optical flow calculation and feature point matching are time-consuming, which directly increases the serial time consumption and is difficult to meet the needs of low-latency camera motion estimation and compensation.

[0005] Therefore, a new camera motion estimation and compensation method is urgently needed to solve the delay problem in existing camera motion compensation. Summary of the invention

[0006] The purpose of the embodiments of the present application is to provide a low-latency camera motion estimation and compensation method for target tracking, so as to solve the technical problem of delay occurring during camera motion compensation in the prior art.

[0007] To achieve the above-mentioned purpose, an embodiment of the present application provides a low-latency camera motion estimation and compensation method for target tracking, comprising the following steps: acquiring an image sequence and initializing two threads, including a motion estimation thread and a target tracking thread, the two threads being performed synchronously;

[0008] The motion estimation thread includes: extracting feature points of the previous and next frames, performing feature matching to obtain feature matching results; using the feature points to perform bidirectional optical flow calculation to obtain bidirectional optical flow matching results, and matching the feature matching results to obtain corresponding matching point pairs; calculating the homography matrix based on the corresponding matching point pairs between the previous and next frames, updating the Kalman filter and predicting the homography matrix of the next frame;

[0009] The target tracking thread uses the homography matrix of the next frame to compensate and adjust the search area of ​​the next frame.

[0010] Preferably, the process of compensating and adjusting includes: calculating the predicted position of the target in the next frame after the camera moves according to the homography matrix of the next frame, and compensating and adjusting the search area of ​​the target tracker in the next frame according to the predicted position.

[0011] Preferably, the process of obtaining the predicted position includes: initializing the target tracker, calculating the center point of the search area, calculating the position of the transformed center point according to the homography matrix of the next frame, and obtaining the predicted position of the target in the next frame after the camera moves.

[0012] Preferably, the position formula of the center point after transformation is:

[0013]

[0014] In the formula, H k+1|k is the homography matrix of the next frame, c and d are the horizontal and vertical coordinates of the center point of the target search area of ​​the current frame, a' and b' are the horizontal and vertical coordinates of the center point of the target search area of ​​the next frame, and z is the coefficient.

[0015] Preferably, feature matching refers to searching for descriptors of feature points using a K-nearest neighbor classification algorithm, filtering out matching point pairs with poor quality using a ratio test, and retaining matching point pairs with good quality as feature matching results.

[0016] Preferably, the bidirectional optical flow calculation includes calculating the forward optical flow and the reverse optical flow, removing inconsistent feature points of the forward optical flow and the reverse optical flow, and obtaining a bidirectional optical flow matching result.

[0017] Preferably, the process of calculating the homography matrix includes: constructing a set of hyperstatic equations according to corresponding matching point pairs between two frames, and solving the homography matrix using the least squares method.

[0018] Preferably, the statically indeterminate system of equations:

[0019]

[0020] Where x' and y' are the horizontal and vertical coordinates of the feature point in the corresponding matching point pair of the previous frame, and x and y are the horizontal and vertical coordinates of the feature point in the corresponding matching point pair of the current frame.

[0021] Preferably, the updating process includes: defining a state vector according to a homography matrix, initializing a Kalman filter, setting an observation matrix and calculating a Kalman gain, and updating a state estimate and a posterior error covariance.

[0022] Preferably, an image feature extractor is used to extract feature points of the two preceding and following image frames.

[0023] The beneficial effect of the present application is that the present application provides a low-latency camera motion estimation and compensation method for target tracking, including a motion estimation thread and a target tracking thread, abandoning the traditional practice of performing motion estimation first and then performing target tracking, but adopting a strategy of performing motion estimation thread and target tracking thread simultaneously, using the homography matrix of the next frame predicted by the motion estimation thread to connect the two threads, ensuring that the homography matrix of the next frame corresponding to it can be directly used when the next frame image arrives. In this way, the accumulation of serial delays is avoided, the delay of camera motion estimation is effectively reduced, and the real-time performance and accuracy of the system are improved. Not only that, the motion estimation thread uses bidirectional optical flow, feature matching and correspondence to obtain corresponding matching point pairs. In this process, three screenings are performed, and the matching point pairs obtained are more accurate and the matching rate is higher. At the same time, since the motion estimation thread is responsible for motion estimation alone, feature matching can be performed using feature points with better robustness without being limited by the running time consumption, which significantly improves the robustness and accuracy of camera motion estimation. In summary, the present application has shorter delay, higher accuracy and better robustness for camera motion estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0025] Figure 1 A schematic diagram of the overall process of a low-latency camera motion estimation and compensation method for target tracking provided in one embodiment of the present application;

[0026] Figure 2 A schematic diagram of low-latency motion estimation using dual threads provided in an embodiment of the present application;

[0027] Figure 3 A result diagram of background alignment using the present application provided in an embodiment of the present application;

[0028] Figure 4 A result diagram of target tracking compensation using the present application provided in one embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0030] This application provides a low-latency camera motion estimation and compensation method for target tracking. It abandons the traditional practice of performing motion estimation first and then target tracking, and instead adopts a strategy of running the motion estimation thread and the target tracking thread simultaneously, which effectively reduces the latency. The operation and interaction of the motion estimation thread and the target tracking thread are as follows: Figure 1 As shown, for the first image frame in the image sequence, that is, image frame 1, only motion estimation and updating and prediction using the Kalman filter are performed, and motion compensation is not performed. For the motion estimation thread, each time a new image frame is captured, the homography matrix with the previous frame is calculated, and the Kalman filter is updated. The homography matrix of the next frame is predicted by the Kalman filter, and the homography matrix of the next frame and the current frame is predicted by the Kalman filter to ensure that the homography matrix parameters can be directly used when the next image frame arrives to connect the two threads. For the target tracking thread, each time a new image frame is obtained, the homography matrix of the current frame predicted by the previous frame can be directly obtained in the motion estimation thread, and motion compensation can be performed on the target tracker.

[0031] In this way, the accumulation of serial delay is avoided, thereby effectively reducing the delay of the motion estimation thread and improving the real-time performance and accuracy of the system. At the same time, since the motion estimation thread is responsible for motion estimation alone, it can use more robust feature points and descriptors for feature matching, so that the runtime will not be limited by the runtime, improving the robustness and accuracy of system integration.

[0032] See also Figure 2 , is a schematic diagram of the overall process of a low-latency camera motion estimation and compensation method for target tracking provided by an embodiment of the present application, including:

[0033] S1: Acquire the image sequence and initialize two threads, including a motion estimation thread and a target tracking thread, and the two threads are performed synchronously.

[0034] Use the camera to acquire multiple images to obtain an image sequence. Initialize the motion estimation thread and the target tracking thread, where the motion estimation thread and the target tracking thread are performed synchronously. It is worth noting that there is no restriction on the initialization method here, and it can be set according to the actual situation.

[0035] S2: extracting feature points of the two frames of images before and after in the motion estimation thread, including descriptors corresponding to the feature points, and performing feature matching to obtain feature matching results.

[0036] Image feature extractors such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Robust Feature), Shi-Tomasi, etc. are used to extract feature points of the two frames of images, including descriptors corresponding to the feature points, for feature point matching of the two frames of images. This application does not limit the image feature extractor, and you can choose it according to the actual situation.

[0037] In an optional embodiment, Shi-Tomasi corner point detection is preferably used to obtain feature points, and a K-nearest neighbor classification algorithm is used to perform feature matching on the feature points to obtain feature matching results.

[0038] Specifically, the K nearest neighbor classification algorithm searches for the descriptor corresponding to each feature point and finds the K nearest neighbors to determine the potential feature matching results. Then, the ratio test is used to filter out the matching point pairs with poor quality and retain the matching point pairs with good quality as the feature matching results.

[0039] Among them, in an optional embodiment, the ratio test calculates the distance ratio of the nearest neighbor based on the K nearest neighbors searched, that is, calculates the distance between the nearest neighbor and the next nearest neighbor, and then filters according to the set threshold. The threshold set in this application is 0.8. The setting of the threshold is not limited here and can be selected according to the actual situation. If the distance ratio is less than the threshold, it is considered to be a good quality matching point pair and it is retained; otherwise, it is considered to be a poor quality matching point pair and it is filtered out.

[0040] S3: The motion estimation thread uses the feature points to perform bidirectional optical flow calculations to obtain bidirectional optical flow matching results, matches the bidirectional optical flow matching results with the feature matching results, filters out erroneous matching results and motion points, and obtains corresponding matching point pairs.

[0041] Bidirectional optical flow is calculated for the feature points, where the bidirectional optical flow includes forward optical flow and reverse optical flow. It is determined whether the forward optical flow and reverse optical flow of the feature points after the bidirectional optical flow calculation are consistent. If they are inconsistent (i.e., the position calculated by the forward optical flow is inconsistent with the position calculated by the reverse optical flow), it is removed as a wrong match; otherwise, it is used as a bidirectional optical flow matching result. It is worth noting that the calculation method of the bidirectional optical flow is not limited here, and the Lucas-Kanade method is preferred in this application.

[0042] The bidirectional optical flow matching results are matched with the feature point matching results, the consistency and error of the optical flow are checked, and dynamic points and low-confidence points, i.e., false matching results and moving points, are removed.

[0043] It is worth noting that only the corresponding matching point pairs corresponding to the bidirectional optical flow and feature matching results are retained here and used as the calculation point pairs of the final homography matrix.

[0044] S4: The motion estimation thread uses the corresponding matching point pairs between the previous and next frames to calculate the homography matrix and accurately estimate the background motion between the previous and next frames.

[0045] Specifically, the coordinates (x', y') of the feature points in the matching point pair corresponding to the previous frame and the coordinates (x, y) of the feature points in the matching point pair corresponding to the current frame are used to construct a hyperstatic equation group, and the homography matrix H can be solved using the least squares method.

[0046] Specifically, the formula of the hyperstatic equations is as follows:

[0047]

[0048] Where x' and y' are the horizontal and vertical coordinates of the feature point in the corresponding matching point pair of the previous frame, and x and y are the horizontal and vertical coordinates of the feature point in the corresponding matching point pair of the current frame.

[0049] S5: The motion estimation thread inputs the homography matrix H into the Kalman filter, updates the Kalman filter parameters, and predicts the homography matrix of the next frame relative to the current frame, that is, the homography matrix of the next frame.

[0050] In an optional embodiment, a constant velocity motion Kalman filter is selected for updating and prediction.

[0051] Convert the homography matrix H to the state vector x:

[0052] First, extract the eight independent parameters of the homography matrix H [h 11 ,h 12 ,h 13 ,h 21 ,h 22 ,h23 ,h 31 ,h 32 Then, the state vector x = [h 11 ,h 12 ,h 13 ,h 21 ,h 22 ,h 23 ,h 31 ,h 32 ,Δh 11 ,Δh 12 ,Δh 13 ,Δh 21 ,Δh 22 ,Δh 23 ,Δh 31 ,Δh 32 ]. Where Δh ij Indicates the rate of change of the parameter, i.e. Δh ij =dh ij / dt.

[0053] Update the Kalman filter parameters according to the homography matrix H:

[0054] First, the Kalman gain K is calculated.

[0055] K=P k|k-1 *M T *(M*P k|k-1 *M T +R) -1 ;

[0056] P k|k-1 =F*P k-1|k-1 *F T +Q;

[0057] Where K is the Kalman gain of the current frame, P k|k-1 is the prior estimation error covariance of the current frame, Q is the noise covariance matrix of the state transfer process, M is the observation matrix, which is set to a matrix of 8 rows and 16 columns in the present invention, the first 8*8 are set to a unit matrix, and the last 8*8 are set to all 0s, R is the observation noise covariance matrix, which is set to 0.1*I in the present invention, where I is a 16*16 unit matrix.

[0058] Second, update the state estimate.

[0059]

[0060] In the formula, is the state estimation vector of the current frame, calculated by the state transfer equation, z k is the current measured state vector, that is, the calculated homography matrix parameters, is the predicted state vector, is the observed correction state at the last moment, F is the state transfer equation, and in the present invention, F is set to a 16*16 matrix.

[0061] Finally, the posterior error covariance is updated.

[0062] P k|k =(IK*M)*P k|k-1 ;

[0063] Where P k|k is the posterior error covariance matrix of the current frame, and I is the identity matrix.

[0064] The specific process of predicting the homography matrix of the next frame and the current frame is:

[0065] Use the state transition equation to predict the next frame state:

[0066]

[0067] In the formula, is the state estimation vector of the next frame, and F is the state transfer matrix, which is initialized as:

[0068]

[0069] Considering the constant speed motion, the Kalman filter uses the constant speed motion model, that is, the parameter change rate Δh ij remains unchanged, so the formula for the forecast error covariance is:

[0070] P k+1|k =F*P k|k *F T +Q;

[0071] Where P k+1|k is the prediction error covariance matrix of the current frame, Q is the noise covariance matrix of the state transfer process, which is set to 1*I in this embodiment, where I is a 16*16 unit matrix. Extract the prediction state vector The 8 homography matrix parameters in construct the 3×3 predicted homography matrix H of the next frame k+1|k , ensuring that the matrix satisfies the scale normalization condition h 33 =1.

[0072] S6: The target tracking thread uses the predicted homography matrix H of the next frame k+1|k Calculate the predicted position of the target in the next frame after the camera motion, and compensate and adjust the search area ROI of the target tracker in the next frame according to the predicted position.

[0073] First, initialize the target tracker, and calculate the center point of the search area ROI (a, b, w, h) according to the initialized target tracker. The search area ROI here is a rectangular box, where a is the horizontal coordinate of the upper left corner of the rectangular box, b is the vertical coordinate of the upper left corner of the rectangular box, w is the width of the rectangular box, and h is the height of the rectangular box.

[0074] Specifically, the calculation formula of the center point (c, d) is as follows:

[0075] c = a + 0.5 * w;

[0076] d = b + 0.5 * h;

[0077] Where c and d are the horizontal and vertical coordinates of the center point respectively.

[0078] Secondly, use the predicted homography matrix H of the next frame k+1|k Calculate the position of the center point after transformation. The calculation formula is as follows:

[0079]

[0080] In the formula, H k+1|k is the predicted homography matrix of the next frame, c and d are the horizontal and vertical coordinates of the center point of the target search area of ​​the current frame, c' and d' are the horizontal and vertical coordinates of the center point of the target search area of ​​the next frame, and z is the coefficient.

[0081] Finally, the new ROI area (c', d', w, h) is calculated according to the position of the transformed center point. (c', d') is calculated according to the above formula, (w, h) remains unchanged, and the new ROI area (c', d', w, h) is assigned to the target tracker to compensate and adjust the search area of ​​the next frame.

[0082] Specific embodiment 1: actual measurement experiment

[0083] In order to better compare the beneficial effects of motion compensation on background motion and target tracking using the present application, all pixels of the previous frame image are mapped to the current frame, and the background alignment between the mapped image and the current frame image is compared, such as Figure 3 As shown. In the figure (a), the background comparison of the original adjacent frames is shown, and in the figure (b), the background comparison of the adjacent frames after motion compensation of the present application is shown. It can be seen that there is a large error in the background comparison of the original adjacent frames. For example, the adjacent frames of the road in the figure (a) cannot be aligned, while the adjacent frames after compensation of the present application are aligned. It can be seen that the compensation effect of the present application is better.

[0084] Figure 4Comparison diagrams of the target tracking thread for compensation in this application, where (a) and (b) are ROI images of the area to be searched, (c) is an image obtained using an uncompensated target tracker, and (d) is an image obtained using a compensated target tracker in this application. The mapping of the search area ROI can be clearly seen in the figure. From the mapping relationship between (a) and (c), it can be seen that the background of the camera has changed greatly, but the target tracker without compensation cannot align the backgrounds of the two. For example, the person in (a) and (c) walks forward, and the camera moves forward with it, causing the person in (c) to move backward relative to (a). For another example, compared with the bench in the background of (a) and the bench in the park in (c), the bench in (c) moves forward, and it is obvious that the backgrounds of (a) and (c) are not aligned.

[0085] Compared with (d) obtained after compensation by the present application, it can be clearly seen that the backgrounds of (b) and (d) are aligned. After the ROI in the search area is mapped, the displacement caused by the background movement is well compensated, and the remaining displacement is caused by the movement of the target itself. Therefore, after compensation by the present application, the compensation effect of target tracking is better, which can well avoid the error caused by the movement of the camera itself and well avoid the target visual movement caused by the camera movement.

[0086] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0087] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A low-latency camera motion estimation and compensation method for target tracking, characterized in that: The following steps are involved: Acquire an image sequence, initialize two threads, including a motion estimation thread and a target tracking thread, and the two threads are performed synchronously; The motion estimation thread includes: extracting feature points of two frames of images, performing feature matching to obtain feature matching results; The feature points are used to perform bidirectional optical flow calculation to obtain a bidirectional optical flow matching result, and the result of the feature matching is matched to obtain a corresponding matching point pair; a homography matrix is ​​calculated based on the corresponding matching point pairs between the previous and next frames, a Kalman filter is updated, and a homography matrix of the next frame is predicted; The target tracking thread uses the homography matrix of the next frame to compensate and adjust the search area of ​​the next frame.

2. A low-latency camera motion estimation and compensation method for target tracking according to claim 1, characterized in that: The compensation and adjustment process includes: calculating the predicted position of the target in the next frame after the camera moves according to the homography matrix of the next frame, and compensating and adjusting the search area of ​​the next frame of the target tracker according to the predicted position.

3. The low-latency camera motion estimation and compensation method for target tracking according to claim 2, characterized in that: The process of obtaining the predicted position includes: initializing the target tracker, calculating the center point of the search area, calculating the position of the transformed center point according to the homography matrix of the next frame, and obtaining the predicted position of the target in the next frame after the camera moves.

4. The low-latency camera motion estimation and compensation method for target tracking according to claim 3, characterized in that: The position formula of the center point after the transformation is: In the formula, H k+1|k is the homography matrix of the next frame, c and d are the horizontal and vertical coordinates of the center point of the target search area of ​​the current frame, a' and b' are the horizontal and vertical coordinates of the center point of the target search area of ​​the next frame, and z is the coefficient.

5. The low-latency camera motion estimation and compensation method for target tracking according to claim 4, characterized in that: The feature matching refers to searching the descriptors of the feature points using a K-nearest neighbor classification algorithm, filtering out matching point pairs with poor quality using a ratio test, and retaining matching point pairs with good quality as the feature matching results.

6. The low-latency camera motion estimation and compensation method for target tracking according to claim 5, characterized in that: The bidirectional optical flow calculation includes calculating the forward optical flow and the reverse optical flow, removing the feature points where the forward optical flow and the reverse optical flow are inconsistent, and obtaining the bidirectional optical flow matching result.

7. The low-latency camera motion estimation and compensation method for target tracking according to claim 6, characterized in that: The process of calculating the homography matrix includes: constructing a hyperstatic equation group according to the corresponding matching point pairs between the previous and next frames, and solving the homography matrix using the least squares method.

8. The low-latency camera motion estimation and compensation method for target tracking according to claim 7, characterized in that: The formula of the statically indeterminate system of equations: Where x' and y' are the horizontal and vertical coordinates of the feature point in the corresponding matching point pair of the previous frame, and x and y are the horizontal and vertical coordinates of the feature point in the corresponding matching point pair of the current frame.

9. The low-latency camera motion estimation and compensation method for target tracking according to claim 8, characterized in that: The process of updating the Kalman filter includes: defining a state vector according to the homography matrix, initializing the Kalman filter, setting an observation matrix and calculating a Kalman gain, and updating a state estimate and a posterior error covariance.

10. The low-latency camera motion estimation and compensation method for target tracking according to claim 9, characterized in that: An image feature extractor is used to extract feature points of the two frames of images.

Images