A Correlation Filter Tracking Method Based on Relative Entropy

Abstract

The present invention is a correlation filtering tracking method based on relative entropy, comprising step 1: reading the position information of the first frame image of the video and the target to be tracked, and adding them to the training set; step 2: extracting the features in the current training set, training the correlation filter, initializing, and fitting the response graph using the relative entropy measurement method; step 3: extracting the image block for target detection with the target position of the previous frame as the center, and extracting features for the candidate samples of the current frame image; step 4: model optimization, generating an optimized and updated correlation filter based on relative entropy; step 5: making the optimized and updated correlation filter and the candidate image block perform convolution operation in the frequency domain, and the obtained result is converted to the response graph of the time domain by inverse Fourier transform; step 6: repeating steps 3-5 until all frames of the video are read. The method has high recognition accuracy and can stably track targets in complex environments such as illumination changes, target occlusion, deformation and rapid motion.

Description

Technical Field

[0001] The invention relates to the field of single target tracking, and in particular to a correlation filtering tracking method based on relative entropy. Background Art

[0002] As a branch of computer vision, target tracking continuously infers the target state in the video sequence, locates the target in each frame of the video, and then generates the target's motion trajectory to provide a complete target area at every moment. With the continuous advancement of social technology, tracking technology is also constantly innovating and improving, especially in the fields of intelligent monitoring, smart cities, medical assistance, etc., and has received more and more attention and application.

[0003] With the development of correlation filtering technology in the field of target tracking in recent years, Bolme et al. first introduced the correlation filtering method into the field of target tracking in 2010 and proposed the MOSSE (Minimum Output Sum of Squared Error) target tracking algorithm. Since then, many excellent algorithms based on correlation filtering have emerged.

[0004] In 2014, Henriques et al. used multi-channel HOG features to replace grayscale features and obtained a multi-channel kernel correlation filter target tracking algorithm (Kernel Correlation Filters, KCF, JFHenriques, C.Rui, P.Martins, and J.Batista, "Exploiting the circulant structure of tracking-by-detection with kernels," in Computer Vision ECCV 2012, vol.7575, 2012, pp.702–715.). KCF improves the algorithm's adaptability to complex tracking environments by replacing low-dimensional features with high-dimensional features.

[0005] In 2014, YANG Li et al. combined the color names (CN) feature with the histogram of oriented gradient (HOG) feature and proposed a scale adaptive multiple feature target tracking algorithm (SAMF, Y.Li and J.Zhu, "A scale adaptive kernel correlation filter tracker with feature integration," in Computer Vision ECCV 2014Workshops, vol.8926, 2014, pp.254–265.). SAMF algorithm combines HOG features and CN features to track the target and proposes a scale pyramid-based target tracking scale solution to solve the scale change problem of the target during the tracking process.

[0006] In 2018, Cheng Tian et al. proposed a spatial-temporal regularized correlation filter target tracking algorithm based on the KCF algorithm to reduce the impact of boundary effects (Spatial-Temporal Regularized Correlation Filters, STRCF, F. Li, C. Tian, ​​W. Zuo, L. Zhang, and M HYang, "Learning spatial temporal regularized correlation filters for visual tracking," in IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 4904–4913.). STRCF uses spatial regularization and spatial-temporal regularization to solve boundary effects, thereby alleviating the problem of reduced filter tracking reliability and degradation caused by the introduction of too many background noise samples.

[0007] In 2020, Changhong Fu et al. proposed a new bidirectional inconsistency-aware correlation filter tracking algorithm (Bidirectional Incongruity-Aware Correlation Filter, BiCF, F.Lin, C.Fu, Y.He, F.Guo and Q.Tang, "BiCF: Learning BidirectionalIncongruity-Aware Correlation Filter for Efficient UAV Object Tracking," in 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 2365-2371.) based on the high computational efficiency of the correlation filter model. BiCF integrates response-based bidirectional inconsistency errors into CF, allowing the filter to not only predict the position of the object forward, but also track backward to locate its position in the previous frame, thereby effectively learning appearance changes and suppressing inconsistency errors.

[0008] In 2021, in order to solve the problem of poor tracking performance of the tracker under complex background at night, Changhong Fu et al. integrated an efficient low-light image enhancer into a tracker based on correlation filtering and proposed a new tracker with anti-dark function (Tracker With Anti-Dark Function, ADTrack, B.Li, C.Fu, F.Ding, J.Ye and F.Lin, "ADTrack: Target-Aware Dual Filter Learning for Real-Time Anti-Dark UAV Tracking," 2021IEEE International Conference on Robotics and Automation (ICRA), 2021, pp.496-502.). The target-aware mask is generated simultaneously using the image illumination changes. The target-aware mask can be used to jointly train a filter focused on the target, which assists the background filter in robust tracking.

[0009] In recent years, target tracking technology based on correlation filtering has developed rapidly. Some have made improvements in feature extraction and integrated deep features based on traditional manual features; some have added training samples to data samples to improve fitting effect and robustness; others have modified the correlation filter model and added regularization terms to prevent the degradation of tracking performance caused by model degradation during target tracking.

[0010] However, in terms of model modification, most of the fixed response map approximation problems are measured by some linear measurement methods, such as the common Euclidean distance, norm, weighted distance, Hausdorff distance, etc. Relatively simple linear measurement methods have certain deficiencies and defects in performance, and are easily affected by external backgrounds during target tracking, causing distortion and drift. Summary of the invention

[0011] In order to solve the above problems, the present invention provides a correlation filtering tracking method based on relative entropy. A regularization term of relative entropy is introduced on the basis of the correlation filtering model. The responses of the previous and next frames are fitted by a nonlinear measurement method to find the highest peak position of the response graph. By adding this regularization term, the possibility of the tracker being controlled by the unreliable part of the tracked target is reduced, thereby solving the problem of model degradation in the tracking process.

[0012] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0013] The present invention is a correlation filtering tracking method based on relative entropy, comprising the following process:

[0014] Step 1: Read the first frame of the video and the location information of the target to be tracked, and add them to the training set;

[0015] Step 2: Extract the color features and gradient features of the target in the current training set, train the correlation filter based on relative entropy, initialize the filter, and use the relative entropy measurement method to replace the traditional linear measurement method to fit the response graph in model building;

[0016] Step 3: Extract image blocks centered on the target position of the previous frame for target detection, and extract features from candidate samples of the current frame image;

[0017] Step 4: Use the Alternating Direction Method of Multipliers (ADMM) optimization algorithm to optimize the correlation filter model based on relative entropy to generate an optimized and updated correlation filter based on relative entropy;

[0018] Step 5: Perform convolution operation on the optimized correlation filter and the candidate image block in the frequency domain. The result is converted into a response map in the time domain through inverse Fourier transform. The highest point of the response map is the position of the target.

[0019] Step 6: Repeat steps 3-5 until all video frames are read to obtain the position information of the target to be tracked in each frame of the video image.

[0020] The specific training process of step 4 is as follows:

[0021] Step 41: construct a correlation filtering model based on relative entropy;

[0022] Step 42: Update and optimize the relevant filter model through the ADMM algorithm.

[0023] Furthermore, the specific process of step 42 is as follows: after introducing the relative entropy regularization term, the correlation filter is obtained by minimizing the filter training formula E, where E d (w d ,h d ) Specifically:

[0024]

[0025] In this work, the dth channel is selected for the following model derivation. is the bidirectional inconsistency error, y represents the expected value, x represents the characteristics of the current frame image, and x p represents the features of the previous frame image; w represents the current frame filter, w p represents the previous frame filter, is the relative entropy measure we introduced, γ is the regularization term, and S represents the spatial regularizer.

[0026] Furthermore, the step 42 minimizes the filter training formula E d (w d ,h d ) process, the optimal correlation filter is obtained by adopting the ADMM optimization method.

[0027] Furthermore, the specific process of the convolution operation in step 5 is as follows:

[0028]

[0029] Among them, x is the image feature of the search area, d represents the number of feature channels, and α d Represents the weight of each feature channel.

[0030] The beneficial effects of the present invention are as follows: This patent takes the BiCF target tracking algorithm as the basic model, innovates on the BiCF model, adds a relative entropy measurement method, performs relative entropy fitting on the effects in the forward error and backward error respectively, then weights them, and finally introduces them into the BiCF model.

[0031] In the long-term target tracking process, the tracking method of the present invention has a better fitting effect than the Euclidean distance compared with the relatively complex relative entropy measurement method. In the updating process of the relevant filter model, it has better adaptability to complex scenes, and effectively prevents the model from degrading and being unable to adapt to more complex tracking scenes during the long-term tracking process, thereby making it difficult to track the correct target, causing response map distortion and tracking frame drift, and more effectively enhancing the robustness of the model and the accuracy of tracking.

[0032] The correlation filtering tracking method of the present invention is suitable for high robustness problems in target tracking such as intelligent monitoring and human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of the tracking algorithm based on relative entropy of the present invention.

[0034] Figure 2 It is the tracking accuracy and success rate graph of each tracker on the UAV123 dataset.

[0035] Figure 3 It is the tracking accuracy and success rate graph of each tracker on the DTB70 dataset.

[0036] Figure 4 It is the tracking accuracy and success rate graph of each tracker on the UAVDT-M dataset.

[0037] Figure 5 This is a comparison table of tracking accuracy under 6 attribute conditions based on the UAV123 dataset.

[0038] Figure 6 This is a comparison table of tracking success rates under six attribute conditions based on the UAV123 dataset.

[0039] Figure 7 This is a comparison table of tracking accuracy under 6 attribute conditions based on the DTB70 dataset.

[0040] Figure 8 This is a comparison table of tracking success rates under six attribute conditions based on the DTB70 dataset.

[0041] Fig. 9 This is a comparison table of tracking accuracy under six attribute conditions based on the UAVDT-M dataset.

[0042] Fig.10 This is a comparison table of tracking success rates under six attribute conditions based on the UAVDT-M dataset. DETAILED DESCRIPTION

[0043] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.

[0044] The present invention takes the BiCF target tracking algorithm as the basic model, makes innovations on the BiCF model, adds a relative entropy measurement method, performs relative entropy fitting on the effects in the forward error and backward error respectively and then weights them, and finally introduces them into the BiCF model to obtain a correlation filtering tracking method based on asymmetric relative entropy of this patent.

[0045] The basic model of BiCF is as follows:

[0046]

[0047] On this basis, the present invention adds a relative entropy measurement method. Since the traditional measurement method makes the model too simple, it is easy to cause the filter model to degenerate during the tracking process, thereby causing the drift of the tracking frame and the distortion of the response graph, affecting the tracking performance. Therefore, a more complex relative entropy measurement method is introduced to make the model more complex in construction, alleviate the problem of degradation during the model update process, and enhance robustness. The model of the present invention is as follows:

[0048]

[0049] in, is the bidirectional inconsistency error, y represents the expected value, x represents the characteristics of the current frame image, and x p represents the features of the previous frame image; w represents the current frame filter, w p represents the previous frame filter, is the relative entropy measure we introduced, γ is the regularization term, and S represents the spatial regularizer.

[0050] Combine the following Figure 1 The invention is further described by ADMM derivation.

[0051] In terms of models, most fixed response map approximation problems are measured by some linear measurement methods. Relatively simple linear measurement methods have certain deficiencies and defects in performance. In the process of tracking the target, they are easily affected by the external background and cause distortion and drift. Relatively complex nonlinear performance measurement methods have good robustness and effectively prevent model degradation. We introduce a relative entropy regularization term, and the total loss function can be expressed as:

[0052]

[0053] in, is our relative entropy regularization term. is the bidirectional inconsistency error, y represents the expected value, x represents the characteristics of the current frame image, and x p represents the features of the previous frame image; w represents the current frame filter, w p represents the previous frame filter, S represents the spatial regularizer, and γ is the regularization term.

[0054] Based on this, Figure 1 As shown, the present invention provides a correlation filtering tracking method based on asymmetric relative entropy, including the following process:

[0055] Step 1: Read the first frame of the video and the location information of the target to be tracked, and add them to the training set;

[0056] Step 2: Extract the color features and gradient features of the target in the current training set, train the correlation filter based on relative entropy, initialize the filter, and use the relative entropy measurement method to replace the traditional linear measurement method to fit the response graph in model building;

[0057] Step 3: Extract image blocks centered on the target position of the previous frame for target detection, and extract features from candidate samples of the current frame image;

[0058] Step 4: Use the Alternating Direction Method of Multipliers (ADMM) optimization algorithm to optimize the correlation filter model based on relative entropy to generate an optimized and updated correlation filter based on relative entropy;

[0059] Step 5: Perform convolution operation on the relative entropy-based correlation filter optimized and updated in step 4 with the candidate image block in the frequency domain. The result is converted to a response map in the time domain through inverse Fourier transform. The highest point of the response map is the position of the target. The specific convolution operation process is as follows:

[0060]

[0061] Among them, x is the image feature of the search area, d represents the number of feature channels, and α d Represents the weight of each feature channel.

[0062] Step 6: Repeat steps 3-5 until all video frames are read to obtain the position information of the target to be tracked in each frame of the video image.

[0063] The specific method of step 4 is:

[0064] Step 41: construct a correlation filtering model based on relative entropy;

[0065] At frame t, the objective function of the correlation filter model based on relative entropy is expressed as:

[0066]

[0067] In this work, the dth channel is selected for the following model derivation, where, is the bidirectional inconsistency error, x represents the characteristics of the current frame image, w represents the current frame filter, and w p represents the previous frame filter, is the relative entropy measurement method, γ is the regularization term, s is the spatial regularizer, λ represents the regularization parameter, * represents the cyclic correlation operator, is the ridge regression error between the label function y and the response reflected by the feature x extracted from the current training sample.

[0068] Step 42: updating and optimizing the relevant filter model through the ADMM algorithm;

[0069] Step 4-1: To obtain the optimal filter w d Need to minimize E d , introduce an auxiliary variable h d , so that it is convenient to write the objective function of the relative entropy correlation filter at t frame equivalently into an equality constraint form through several sub-problems solved by ADMM iteration:

[0070]

[0071] swt d =h d ,d=1,...,D

[0072] Where: st is the constraint term of ADMM, ADMM optimization process;

[0073] In order to improve computational efficiency, the above formula is converted from the time domain to the frequency domain using Parseval's theorem:

[0074]

[0075]

[0076] in, represents the Hadamard product, ∧ represents the discrete Fourier transform of the signal, * represents the conjugate vector of the complex number, F∈C N×N is a v∈R in the time domain N×1 The DFT matrix of the signal after frequency domain transformation. According to the solution process of ADMM in the optimization method, it is expressed as an augmented Lagrangian form:

[0077]

[0078] in, represents the Lagrange multiplier of the dth channel, and μ represents the penalty factor.

[0079] The specific solution of ADMM is: by solving the sub-problems, we can get the specific sub-problem update formula:

[0080]

[0081] The subproblem w is as follows:

[0082]

[0083] By taking the derivative of both sides of the equation, The update formula is:

[0084]

[0085] Among them, because the formula is too long, the specific expression of the molecule A, B, and C is as follows:

[0086]

[0087]

[0088] μ is the penalty factor; γ is the regularization term.

[0089] The subproblem h is as follows:

[0090] By taking the derivative on both sides of the equation, the update formula of h is:

[0091]

[0092] F -1 represents the inverse Fourier transform.

[0093] The subproblem ζ is as follows:

[0094] By taking the derivative on both sides of the equation, the update formula of ζ is:

[0095]

[0096] in, As the number of ADMM iterations increases, the parameter μ is updated as follows:

[0097] μ i+1 =min(μ max ,βμ i )

[0098] Initial penalty factor μ, step size β and maximum value μ of ADMM optimization parametersmax They are set to 100, 10 and 105 respectively, and the number of ADMM iterations is 4.

[0099] As the number of ADMM iterations ends, the filter of the current frame is updated, and the update formula is as follows:

[0100]

[0101] in, represents the current filter model, represents the filter model of the previous frame, and η is the update rate.

[0102] When the tracking video image reaches the next frame, the above steps are repeated.

[0103] By adopting the above target tracking algorithm, the problem of model degradation can be effectively prevented, and the tracking performance in terms of accuracy and success rate can be improved.

[0104] The results are shown in Figures 2-10. The tracker of the present invention is compared with 10 trackers on the UAV123, DTB70 and UAVDT-M data sets, demonstrating the superiority of the tracker of the present invention in tracking distance accuracy and success rate. The high anti-interference ability of the tracker of the present invention under the influence of external conditions such as scale change, target occlusion, rapid motion, motion blur, deformation and illumination change is demonstrated in the form of a table.

[0105] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A correlation filtering tracking method based on relative entropy, Features: The tracking method comprises the following steps: Step 1: Read the first frame of the video and the location information of the target to be tracked, and add them to the training set; Step 2: Extract the color features and gradient features of the target in the current training set, train the correlation filter based on relative entropy, initialize the filter, and fit the response map using the relative entropy measurement method; Step 3: Extract image blocks centered on the target position of the previous frame for target detection, and extract features from candidate samples of the current frame image; Step 4: Optimize the correlation filter model based on relative entropy to generate an optimized and updated correlation filter based on relative entropy; Step 5: Perform convolution operation on the frequency domain of the relative entropy-based correlation filter optimized and updated in step 4 and the candidate image block. The result is converted to a response map in the time domain through inverse Fourier transform. The highest point of the response map is the position of the target. Step 6: Repeat steps 3-5 until all video frames are read, and obtain the position information of the target to be tracked in each frame of the video image, where: In step 4, the alternating direction multiplier method ADMM optimization algorithm is used to optimize the correlation filter model based on relative entropy, which specifically includes the following steps: Step 4-1: Write the objective function of the relative entropy correlation filter at frame t equivalently into an equality constraint form: Where: st is the constraint term of ADMM, ADMM optimization process; Step 4-2: Convert the equality constraint form in step 4-1 from the time domain to the frequency domain using Parseval's theorem: in, represents the Hadamard product, ∧ represents the discrete Fourier transform of the signal, * represents the conjugate vector of the complex number, F∈C N×N is a v∈R in the time domain N×1 The DFT matrix of the signal after frequency domain transformation, N represents the constant of matrix dimension; Step 4-3: According to the solution process of ADMM in the optimization method, it is expressed as an augmented Lagrangian form: in, represents the Lagrange multiplier of the dth channel, and μ represents the penalty factor.

2. According to the correlation filtering tracking method based on relative entropy as claimed in claim 1, Features: In step 2, the objective function of the correlation filter of relative entropy at frame t is: in, is the bidirectional inconsistency error, x represents the characteristics of the current frame image, w represents the current frame filter, and w p represents the previous frame filter, is the relative entropy measurement method, γ is the regularization term, s is the spatial regularizer, λ represents the regularization parameter, * represents the cyclic correlation operator, is the ridge regression error between the label function y and the response reflected by the feature x extracted from the current training sample.

3. According to the correlation filtering tracking method based on relative entropy in claim 2, Features: In step 4, the alternating direction multiplier method ADMM optimization algorithm is used to optimize the correlation filtering model based on relative entropy.

4. According to claim 1, a correlation filtering tracking method based on relative entropy, Features: The ADMM solution in step 4-3 is specifically as follows: by solving the sub-problem to obtain a specific sub-problem update formula: The subproblem w is as follows: By taking the derivative of both sides of the equation, The update formula is: Among them, because the formula is too long, the specific expression of the molecule A, B, and C is as follows: μ is the penalty factor; γ is the regularization term; The subproblem h is as follows: By taking the derivative on both sides of the equation, the update formula of h is: F -1 represents inverse Fourier transform; The subproblem ζ is as follows: By taking the derivative on both sides of the equation, the update formula of ζ is: in, As the number of ADMM iterations increases, the parameter μ is updated as follows: m i+1 =min(μ max ,bm i ) Initial penalty factor μ, step size β and maximum value μ of ADMM optimization parameters max They are set to 100, 10, and 105, respectively, and the number of ADMM iterations is 4; As the number of ADMM iterations ends, the filter of the current frame is updated, and the update formula is as follows: in, represents the current filter model, represents the filter model of the previous frame, η is the update rate; When the tracking video image reaches the next frame, the above steps are repeated.

5. According to the correlation filtering tracking method based on relative entropy as claimed in claim 1, Features: The specific process of the convolution operation in step 5 is as follows: Among them, x is the image feature of the search area, d represents the number of feature channels, and α d Represents the weight of each feature channel.

Images