A UAV target tracking method based on Staple

Abstract

The present invention provides a Staple-based unmanned aerial vehicle target tracking method. At the beginning of the tracking task, the unmanned aerial vehicle obtains the first frame image from the camera, selects the target to be tracked, thereby determining the center position and scale of the target, extracts features from the image for calculating the response, respectively uses the directional gradient histogram features and the scale features to obtain the target position and scale response, calculates the target position and scale, makes a confidence judgment on the tracking result, and updates the filter model. The present invention implements the judgment of the tracking effect through a dual mechanism, thereby achieving a high-confidence update of the model, avoiding errors caused by the model update, and improving the accuracy of target tracking. By introducing two tracking confidence judgment mechanisms, namely the position response distance and the average peak correlation energy, the offset caused by the model update in the tracking algorithm is reduced when the target is occluded during the tracking process, making the tracking process more stable and reliable.

Description

Technical Field

[0001] The invention relates to the field of machine vision and to a target tracking algorithm based on correlation filtering. Background Art

[0002] Object tracking is one of the most important research directions in the field of machine vision. It plays an important role in the fields of unmanned driving, video surveillance and human-computer interaction. The task of object tracking is to select the target to be tracked by a rectangular box in the first frame of the video sequence, and then use the tracking algorithm to determine the position and size of the tracked target in the subsequent video frames.

[0003] In 2010, Bolme et al. proposed the Minimum Squared Error Output Correlation Filter (MOSSE) (see Bolme DS, Beveridge JR, Draper BA, et al. Visual object tracking using adaptive correlation filters [C] / / Computer Vision and Pattern Recognition. IEEE, 2010: 2544-2550.), which was the first to introduce correlation filtering in the field of signal processing into the field of target tracking. With its excellent performance and outstanding computing speed, it has attracted widespread attention in the field of target tracking. Later studies introduced a circulant matrix structure based on the MOSSE algorithm to obtain more training samples (CSK) (Henriques JF, Rui C, Martins P, et al. Exploiting the Circulant Structure of Tracking-by-Detection with Kernels [C] / / European Conference on Computer Vision, Springer, 2012: 702-715.), and used the Histogram of Oriented Gradients (HOG) feature to replace the grayscale feature to better characterize the target model (KCF) (Henriques JF, Caseiro R, Martins P, et al. High-Speed ​​Tracking with Kernelized Correlation Filters [J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2015, 37 (3): 583-596.). In order to solve the problem of scale change of the tracked target, DSST (Danelljan M, G, Khan FS, et al. Discriminative Scale Space Tracking. [J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2016, 39 (8): 1561-1575.) and SAMF (Li Y, Zhu JA scale adaptive kernel correlation filter tracker with feature integration [C] / / European conference on computer vision. Springer, Cham, 2014: 254-265.) were proposed respectively. However, this type of algorithm has inevitable edge effects, and the tracking effect of deformed targets is not ideal. SRCDF (Martin D, Gustav H, Fahad SK, et al. Learning Spatially Regularized Correlation Filters for Visual Tracking [C] / / International Conference on Computer Vision, IEEE, 2015: 4310-4318.), CFLB (Kiani Galoogahi H, Sim T, Lucey S. Correlation filters with limited boundaries [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015: 4630-4638.) and BACF (Kiani Galoogahi H, Fagg A, Lucey S. Learning background-aware correlation filters for visual tracking [C] / / Proceedings of the IEEE international conference on computer vision. 2017: 1135-1143.) have proposed their own solutions to the edge effects.The Staple (Bertinetto L, Valmadre J, Golodetz S, et al. Staple: Complementary learners for real-time tracking [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 1401-1409.) algorithm simultaneously introduces HOG features and color statistical features to learn the target, which can handle the problems of target deformation and motion blur well. The color statistical features do not belong to the correlation filtering method and have no edge effect. The complementary combination of HOG features and color features can improve the tracking effect.

[0004] The above algorithms have studied the scale change problem and the defects of the algorithms themselves in the target tracking process, but there are many other challenges in the target tracking process, such as fast motion, object occlusion, and illumination changes, which will have a great impact on the tracking results. For tracking in complex backgrounds, these algorithms will introduce background noise in the model update stage, causing the model to shift and eventually leading to tracking failure. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a UAV target tracking method based on Staple, which realizes the judgment of the tracking effect through a dual mechanism, thereby achieving high-confidence update of the model, avoiding errors caused by model update, and improving the accuracy of target tracking.

[0006] The technical solution adopted by the present invention to solve the technical problem comprises the following steps:

[0007] Step 1: At the beginning of the tracking mission, the drone obtains the first frame of image from the camera;

[0008] Step 2: In the first frame, select the target to be tracked in the image with a rectangular frame according to the task requirements, so as to determine the center position and scale of the target;

[0009] Step 3: Initialize based on the first frame image and target information respectively;

[0010] Step 4: The drone uses the camera to obtain the next frame of image;

[0011] Step 5: Extract features from the image obtained in step 4 to calculate the response;

[0012] (5.1) For the current frame, the target center position of the previous frame image is expanded by 1.5 times to determine the target area, and the color histogram features are extracted in the target area.

[0013] (5.2) Extracting directional gradient histogram features in the target area;

[0014] (5.3) Extracting scale features in the target area;

[0015] Step 6: Use the directional gradient histogram features and scale features extracted in step 5 to obtain the target position and scale response;

[0016] Step 7: Calculate the target position and scale;

[0017] The maximum response position in the final position response is taken as the target position, and the scale corresponding to the maximum response position in the scale response is taken as the target scale. The target position and scale information are output to the UAV to provide a basis for subsequent processing and UAV control.

[0018] Step 8: Make a confidence judgment on the tracking results;

[0019] A dual tracking effect judgment mechanism is proposed, which is the position response distance and the average peak correlation energy (APCE). If the position response distance of the current frame is less than the feature response distance threshold D threhold , and the APCE value is greater than the APCE value threshold APCE threhold , it is determined that the tracking result has a high confidence level, and the model needs to be updated, and the process goes to step 9. If the above judgment conditions are not met, the process returns to step 4;

[0020] Step 9: Update the filter model. After the update is completed, go to step 4;

[0021] (9.1) Determine the target area according to the current position and scale, obtain the color histogram features of the target in the area, and perform weighted fusion update on the original color features;

[0022] (9.2) Obtain the directional gradient histogram feature in the current target area, train the translation filter, and perform weighted fusion update on the original translation filter model;

[0023] (9.3) Using the scale feature, the scale filter is trained and the original scale filter model is updated by weighted fusion.

[0024] The steps for initialization in step 3 are:

[0025] (3.1) Obtaining the color histogram features of the background and foreground;

[0026] (3.2) Obtain the directional gradient histogram features of the target and use the correlation filtering method to train and calculate the translation filter;

[0027] (3.3) Obtain the scale feature of the directional gradient histogram based on the target, and use the correlation filtering method to train and calculate the scale filter;

[0028] The steps of obtaining the target position and scale response in step 6 are:

[0029] (6.1) Using the color histogram feature, obtain the target color feature position response;

[0030] (6.2) Using a translation filter, a translation filter position response of the target is obtained by filtering method;

[0031] (6.3) The final position response is obtained by weighted fusion of the color feature position response and the translation filter position response. The feature response and fusion process are as follows: Figure 2 As shown;

[0032] (6.4) Using the scale filter, the scale response is obtained through filtering method.

[0033] In step 8, the calculation formula of the position response distance is as follows:

[0034] like Figure 3 As shown, the position response distance calculation formula is as follows:

[0035] Distance=(x 1 -x 2 ) 2 +(y 1 -y 2 ) 2

[0036] where x 1 ,y 1 is the maximum position coordinate of the translation filter position response, x 2 ,y 2 The maximum position coordinate of the color feature position response;

[0037] Threshold D threhold The settings are as follows:

[0038]

[0039] The set {D i |i=1,2,…,t-1} represents the set of position response distances of all frames before the current frame, u 1 is the mean of the response distances of all previous frame positions, σ 1 is the standard deviation of the response distances of all previous frames, λ dis the threshold adjustment parameter, t is the current time node;

[0040] When the tracking effect is good, the translation filter response position and the color feature response position should maintain a certain stability, that is, the position response distance should be less than the set threshold;

[0041] In step 8, the calculation formula of average peak correlation energy (APCE) is as follows:

[0042] APCE (Average Peak-To-Correlation Energy) is the average peak correlation energy of the response graph. The average peak correlation energy of the response is calculated as follows:

[0043]

[0044] where F max 、F min 、F w,h They are the maximum response value, the minimum response value, and the response at a width of w and a height of h;

[0045] Threshold APCE threhold The settings are as follows:

[0046]

[0047] The set {APCE i |i=1,2,…,n} represents the average peak correlation energy set of n frames before the current frame, u 2 is the mean of the average peak correlation energy of the previous n frames, σ 2 is the standard deviation of the average peak correlation energy of the previous n frames, λ a is the threshold adjustment parameter;

[0048] When the target tracking effect is good, the response APCE value is greater than or equal to the entire historical average of the APCE values. When the target is blocked or interfered with, the response APCE value is less than the entire historical average of the APCE values.

[0049] The beneficial effect of the present invention is that in the process of UAV target tracking, by introducing two tracking confidence judgment mechanisms, namely position response distance and average peak correlation energy, the offset caused by model update in the tracking algorithm is reduced when the target is occluded during the tracking process, making the tracking process more stable and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is the algorithm flow chart of the present invention.

[0051] Figure 2 It is the response fusion process of the present invention.

[0052] Figure 3 It is the characteristic response position distance of the present invention.

[0053] Figure 4 It is the Girl2 video sequence response distance and threshold curve of the present invention.

[0054] Figure 5 It is the response diagram of Girl2 of the present invention.

[0055] Figure 6 It is the APCE and threshold curve of the Girl2 video sequence of the present invention.

[0056] Figure 7 It is a qualitative comparison of the algorithm effects of the present invention.

[0057] Figure 8 It is the algorithmic precision diagram of OTB100 of the present invention.

[0058] Fig. 9 It is the algorithm success diagram of OTB100 of the present invention. DETAILED DESCRIPTION

[0059] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0060] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0061] Example:

[0062] The technical solution provided by the present invention is a target tracking method based on the Staple algorithm. The overall algorithm process is as follows: Figure 1 As shown, the following steps are included:

[0063] To verify the effectiveness of the algorithm, the Girl2 video sequence in the OTB100 tracking dataset is taken as an example.

[0064] Step 1: Get the image of the first frame in the video sequence and determine the target area based on the position and scale information of the target.

[0065] Step 2: In the target area, obtain the color histogram features of the foreground and background.

[0066] Step 3: Extract the directional gradient histogram features in the target area.

[0067] Step 4: Train to obtain the translation filter. The translation filter h is obtained by optimizing the following cost function ε:

[0068]

[0069] where h l and x l are the lth layer of filtering and the lth layer of feature extraction, respectively, d 1 is the total number of feature layers, which is set to 28 here, y is the response label matrix of the two-dimensional Gaussian distribution, whose peak position corresponds to the center position of the target, ★ is the cyclic correlation operation, and λ is the regularization parameter, which is set to 0.001 here.

[0070] Using the properties of the circulant matrix, the above equation is converted to the frequency domain and solved to obtain

[0071]

[0072] in are the frequency domain forms of h, x, and y respectively, for The conjugate of is the Hadamard product, A and B are the numerator and denominator of the translation filter.

[0073] Step 5: Extract scale features. Extract several images of different scales with the target location as the center, and extract features from images of different scales and splice them into a one-dimensional vector.

[0074] Step 6: Train to obtain the scale filter. The scale filter h is obtained by optimizing the following formula: s :

[0075]

[0076] where x s l is the extracted scale feature of the lth layer, d 2 is the total number of feature layers, which is set to 31 layers here, y s is the response label of a one-dimensional Gaussian distribution.

[0077] Solve for the post-scaling filter h s The form is similar to the translation filter above:

[0078]

[0079] in h s 、x s ,ys The frequency domain form of for The conjugate of A s and B s are the numerator and denominator of the scaling filter.

[0080] Step 7: Get the next frame image.

[0081] Step 8: Extract color histogram features at the original target position in the new image.

[0082] Step 9: Obtain color feature position response based on the color feature.

[0083] Step 10: Extract the directional gradient histogram features at the original target position in the new image.

[0084] Step 11: Use the translation filter to obtain the translation filter position response.

[0085] Step 12: Extract scale features at the original location.

[0086] Step 13: Use the scale filter to obtain the scale response.

[0087] Step 14: Perform weighted fusion of the color feature position response and the translation filter position response to obtain the total position response.

[0088] Step 15: Calculate the position and scale of the target. The maximum position of the total position response is taken as the target position estimation result, and the scale corresponding to the maximum scale response is taken as the final scale estimation result.

[0089] Step 16: Determine whether it is the last frame image, if yes, end the task, otherwise go to step 17.

[0090] Step 17: Determine the reliability of the tracking result based on the dual judgment mechanism. When the characteristic response distance is less than the set threshold and the APCE value is greater than the set threshold, the tracking result is determined to have a high confidence level, the model is updated, and the process goes to step 17. Otherwise, the tracking result is determined to be poor, the update is suspended, and the process goes to step 7.

[0091] (1) Feature response position distance

[0092] Threshold parameter setting: threshold adjustment parameter λ d Set to 3.

[0093] Response distance and threshold curves are as follows: Figure 4 As shown, it can be seen that the response distance fluctuates greatly near frame 106 of the video sequence, and the target is blocked at this time.

[0094] (2) Average peak correlation energy

[0095] Threshold parameter settings: n is 50, λ a is 3.

[0096] like Figure 5 and Figure 6 As shown in the figure, at 100 frames, the target tracking effect is good, only a single peak appears in the response graph, and the APCE value is large, but at 106 frames, the target is occluded, and the response graph has multiple peaks, and the APCE value is small. If the filter is still updated, the model will be offset and the tracking effect will deteriorate.

[0097] Step 18: If the tracking is reliable, extract the color histogram features in the tracking result area.

[0098] Step 19: Extract the directional gradient histogram in the tracking result area.

[0099] Step 20: Train the translation filter, which is the same as step 3.

[0100]

[0101] in A t ' and B t ' is the newly acquired translation filter of the tth frame and its numerator and denominator.

[0102] Step 21: Extract target scale features based on the tracking result area.

[0103] Step 22: Train to obtain the scale filter. Same as step 4.

[0104]

[0105] in A st ' and B st ' is the newly acquired scale filter of the tth frame and its numerator and denominator.

[0106] Step 23: Model update. After the update is completed, proceed to step 7.

[0107]

[0108]

[0109] in and is the updated translation filter h t and scale filter h st frequency domain form.

[0110] Algorithm effect verification:

[0111] Qualitative:

[0112] like Figure 7 As shown in the qualitative effect of the algorithm, if the model is still updated when the target is occluded, error information will be introduced, resulting in subsequent tracking failure. The introduction of a confidence judgment mechanism can stop updating the model when it is determined that the target is occluded, and can reposition the target after it appears, thereby increasing robustness and achieving more stable tracking.

[0113] Quantification:

[0114] At the same time, in order to verify the effectiveness of the improved algorithm and quantify the algorithm performance, the OTB100 dataset is used to experiment the algorithm effect. The experimental results are as follows: Figure 8 , Fig. 9 shown.

Claims

1. A Staple-based UAV target tracking method, characterized in that The steps include: Step 1: At the beginning of the tracking mission, the drone obtains the first frame of image from the camera; Step 2: In the first frame, select the target to be tracked in the image with a rectangular frame according to the task requirements, so as to determine the center position and scale of the target; Step 3: Initialize based on the first frame image and target information respectively; Step 4: The drone uses the camera to obtain the next frame of image; Step 5: Extract features from the image obtained in step 4 to calculate the response; (5.1) For the current frame, the target center position of the previous frame image is expanded by 1.5 times to determine the target area, and the color histogram features are extracted in the target area; (5.2) Extracting directional gradient histogram features in the target area; (5.3) Extracting scale features in the target area; Step 6: Use the directional gradient histogram features and scale features extracted in step 5 to obtain the target position and scale response; Step 7: Calculate the target position and scale; The maximum response position in the final position response is taken as the target position, and the scale corresponding to the maximum response position in the scale response is taken as the target scale. The target position and scale information are output to the UAV to provide a basis for subsequent processing and UAV control. Step 8: Make a confidence judgment on the tracking results; A dual tracking effect judgment mechanism is proposed, which is the position response distance and the average peak correlation energy (APCE). If the position response distance of the current frame is less than the feature response distance threshold D threhold , and the APCE value is greater than the APCE value threshold APCE threhold , it is determined that the tracking result has a high confidence level, and the model needs to be updated, and the process goes to step 9. If the above judgment conditions are not met, the process returns to step 4; Step 9: Update the filter model. After the update is completed, proceed to step 4; (9.1) Determine the target area according to the current position and scale, obtain the color histogram features of the target in the area, and perform weighted fusion update on the original color features; (9.2) Obtain the directional gradient histogram feature in the current target area, train the translation filter, and perform weighted fusion update on the original translation filter model; (9.3) Using the scale feature, the scale filter is trained and the original scale filter model is updated by weighted fusion.

2. The Staple-based UAV target tracking method according to claim 1, characterized in that: The steps for initialization in step 3 are: (3.1) Obtaining the color histogram features of the background and foreground; (3.2) Obtain the directional gradient histogram features of the target and use the correlation filtering method to train and calculate the translation filter; (3.3) Obtain the scale features of the target-based directional gradient histogram and use the correlation filtering method to train and calculate the scale filter.

3. The Staple-based UAV target tracking method according to claim 1, characterized in that: The steps of obtaining the target position and scale response in step 6 are: (6.1) Using the color histogram feature, obtain the target color feature position response; (6.2) Using a translation filter, a translation filter position response of the target is obtained by filtering method; (6.3) Using the weighted fusion of the color feature position response and the translation filter position response, the final position response is obtained; (6.4) Using the scale filter, the scale response is obtained through filtering method.

4. The Staple-based UAV target tracking method according to claim 1, characterized in that: In step 8, the calculation formula of the position response distance is as follows: Distance=(x1-x2) 2 +(y1-y2) 2 Among them, x1 and y1 are the maximum position coordinates of the translation filter position response, and x2 and y2 are the maximum position coordinates of the color feature position response; Threshold D threhold The settings are as follows: The set {D i |i=1,2,…,t-1} represents the set of position response distances of all frames before the current frame, u1 is the mean of the position response distances of all previous frames, σ1 is the standard deviation of the position response distances of all previous frames, and λ d is the threshold adjustment parameter, t is the current time node; When the tracking effect is good, the translation filter response position and the color feature response position should maintain a certain stability, that is, the position response distance should be less than the set threshold.

5. The Staple-based UAV target tracking method according to claim 1, characterized in that: In step 8, the calculation formula of the average peak correlation energy APCE is as follows: APCE is the average peak correlation energy of the response graph. The average peak correlation energy of the response is calculated as follows: where F max 、F min 、F w,h They are the maximum response value, the minimum response value, the width is w, and the height is h Response at; threshold APCE threhold The settings are as follows: The set {APCE i |i=1,2,…,n} represents the average peak correlation energy set of n frames before the current frame, u2 is the mean of the average peak correlation energy of the previous n frames, σ2 is the standard deviation of the average peak correlation energy of the previous n frames, and λ a is the threshold adjustment parameter; When the target tracking effect is good, the response APCE value is greater than or equal to the entire historical average of the APCE values. When the target is blocked or interfered with, the response APCE value is less than the entire historical average of the APCE values.

Images