A target tracking method based on multi-kernel learning and aggregated correlation filtering

Abstract

The present invention relates to an aggregated correlation filtering target tracking method based on multi-core learning, comprising: reading an initial frame of a video image, intercepting a target image area, and extracting different modal features of the target image at the same time; initializing a target template, a detection model and other related parameters of the aggregated correlation filtering based on multi-core learning; reading the video image data of the current frame, intercepting a potential tracking area of ​​the target in the current frame according to the tracking result of the previous frame, then using a cyclic shift strategy to complete a dense division of the potential tracking area of ​​the current frame and obtain multiple candidate samples, and then completing the extraction of different modal features of each candidate sample; using multiple filter functions with different parameters, learning different features separately, obtaining an optimal classification function suitable for the current tracking environment, and obtaining a related filter response map of the potential tracking area in the current frame image under different features; evaluating the contribution of different features to the final prediction result, and then dynamically allocating weights of the above different features.

Description

Technical Field

[0001] The present invention relates to the fields of pattern recognition and computer vision, and in particular to a multi-kernel learning-based aggregate correlation filtering target tracking method. Background Art

[0002] According to incomplete statistics, approximately 70% of all information people encounter daily comes from visual information. With the rapid development of artificial intelligence and video processing technologies, researchers are attempting to use computers and other visual devices to simulate the human visual formation mechanism to quickly perceive the world. Object tracking, a fundamental task in computer vision, not only effectively detects and tracks objects of interest but also enables reasonable interpretation of observed image or video data. In real-world video scenes, the appearance and morphology of the target being tracked often change significantly over time due to the complexity of the environment and the uncontrollable nature of its changes, posing a significant challenge to the task of object tracking.

[0003] A literature search of existing technologies revealed that Chinese patent document CN108154118B discloses a target detection system and method based on adaptive combined filtering and multi-stage detection. In the target tracking module proposed by this method, HoG features, CN features, and corner features are extracted from the target image in the initial frame from three angles. A simple weighted fusion is then used to construct an appearance feature model of the target. Although different types of features can be used in different tracking scenarios, treating them as equivalent clearly ignores the differences between features, thereby weakening their discriminative power in complex tracking environments.

[0004] Further research revealed that Chinese patent document CN106250855B discloses a multimodal emotion recognition method based on multi-kernel learning. This method first extracts the target from the video and obtains its features. It then transforms the multimodal features into new features through certain linear or nonlinear transformations. Different kernel functions are then created to form corresponding kernel matrices. Finally, multi-kernel learning is used in conjunction with a support vector machine to fuse the multimodal features. Although this solution provides a multi-kernel learning strategy, the complex computational process makes it difficult to meet the real-time requirements of the system's target tracking.

[0005] The two aforementioned solutions address target tracking issues in various scenarios to a certain extent. However, their tracking performance degrades significantly when the target's shape changes significantly or when there is significant interference from the external environment. This paper proposes a multi-kernel learning-based aggregated correlation filtering target tracking method to meet the system's requirements for target tracking accuracy, real-time performance, and robustness. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this paper proposes a multi-kernel learning-based aggregated correlation filtering target tracking method based on the classic correlation filtering technology. Specifically, the target image area is captured from the video image and multiple different features are extracted. The feature templates corresponding to each feature type are independently learned. The contribution of different features to the tracking results is measured by a credibility evaluation mechanism, and the adaptive fusion of multiple features is completed. The system is trained and tracked using correlation filtering technology to achieve more accurate and robust tracking results. The technical solution is as follows:

[0007] A multi-kernel learning-based aggregate correlation filtering target tracking method includes the following steps:

[0008] Step 1: Read the initial frame of the video image, capture the target image area, and extract different modal features of the target image; initialize the target template, detection model and other related parameters of the aggregated correlation filter based on multi-kernel learning;

[0009] Step 2: Read the current frame video image data, intercept the target's potential tracking area in the current frame based on the tracking result of the previous frame, then use the cyclic shift strategy to complete the dense division of the potential tracking area of ​​the current frame and obtain multiple candidate samples, and then complete the extraction of different modal features of each candidate sample;

[0010] Step 3: Use multiple filter functions with different parameters to learn the different features obtained in step 2 respectively. At the same time, use the multi-core learning algorithm and correlation filtering technology to train the tracker to obtain the optimal classification function suitable for the current tracking environment. The correlation filter response map of the potential tracking area in the current frame image under different features is obtained as follows:

[0011] Step 4: Combine the fluctuation and peak values ​​of the correlation filter response graph obtained in step 3 to evaluate the contribution of different features to the final prediction result, and then dynamically assign weights to the above different features as follows:

[0012] (1) Use the average peak correlation energy method to evaluate the fluctuation degree of the correlation filter response graph:

[0013]

[0014] Among them, C max i,t , C min i,t and C p,k i,t They represent the maximum response value, minimum response value and response value at position (p, k) of the correlation filter response map of the i-th type feature extracted from the current frame, i.e., the t-th frame image;

[0015] (2) There are three types of features of different modalities. The output response values ​​of the three features are dynamically weighted and fused according to the credibility of the tracking results. Specifically:

[0016]

[0017] Among them, C max t and η i,t They represent the maximum response value after fusion and the redistributed weight coefficient corresponding to the i-th feature in the response graph of the t-th frame image respectively;

[0018] Use the following formula to predict the target at the tth frame, that is, the center position of the current frame (x c t ,y c t ), get the tracking result of the current frame:

[0019]

[0020] Step 5: Use the overlap of the tracking results to decide whether to update. If an update is required, use the prediction results of the current frame to update the target template and detection model, and jump to step 2 until the end of the video. The method is as follows:

[0021] (1) Determine the overlap degree based on the tracking result of the current frame. When the overlap degree is greater than the set threshold, the target template and detection model remain unchanged. Conversely, when the overlap degree is less than the set threshold, the target template and detection model need to be updated. The overlap degree threshold is set to 0.5.

[0022] (2) Update the target template and filter function through linear interpolation.

[0023] Furthermore, in step 1, the method of reading the initial frame of the video image, intercepting the target image area, and extracting different modal features of the target image is as follows:

[0024] (1) Read the initial frame of the video image and capture the target image area [x c ,y c ,w,h] rectangular box, where x c and y c They represent the horizontal and vertical coordinates of the target center, w is the width of the rectangular box, and h is the height of the rectangular box;

[0025] (2) Calculate the three features of the above rectangular box respectively: Histogram of Oriented Gradients (HoG) feature, Color Name (CN) feature and Convolution (CNNs) feature;

[0026] (3) Initialize feature fusion weights

[0027] (4) Initialize the regularization coefficient and learning rate.

[0028] The method of step 2 is as follows:

[0029] (1) Let the current frame be the t-th frame, and according to the tracking result of the t-1-th frame [x c t-1 ,y c t-1 ,w t-1 ,h t-1 ] Intercept the target potential tracking area in the t-th frame image, whose size is M×N;

[0030] (2) Use the cyclic shift strategy to complete the dense division of the potential tracking area in the t-th frame image to obtain multiple candidate samples, and extract different modal features of each candidate sample respectively where (m,n)∈{0,1,...,M-1}×{0,1,...,N-1};

[0031] (3) Describe the features of the potential tracking area in the t-th frame image i is the serial number of different features;

[0032] Furthermore, the method of step three is as follows:

[0033] (1) Using independent filter functions with different parameters to learn different features;

[0034] (2) Using Gaussian kernel function All features are mapped to high dimensions to achieve linear separability, and the filter function w is obtained i,t Isomorphism mapping α in the dual space i,t ;

[0035] (3) Further use of classification functions Calculate the similarity between each candidate sample and the target template respectively, and then obtain the relevant filter response map C of the potential tracking area under different features in the current frame, i.e., the t-th frame image i,t .

[0036] There are three types of features obtained in step 2. In step 3 (1), ridge regression is used to learn the filter function w of the i-th feature. i,t The method is:

[0037]

[0038] Among them, y t represents the Gaussian label corresponding to the target location in the t-th frame, and λ is the regularization coefficient used to control the overfitting of the model.

[0039] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0040] (1) The present invention uses multiple modal features to describe the target and uses different filters to learn the classification functions under various features, ensuring that the dynamic fusion modeling of the target appearance model is carried out under optimal conditions. In addition, it improves the traditional correlation filter algorithm's assumption of using only the maximum value in the response map as the target prediction position, which can significantly improve the accuracy of the system tracking effect;

[0041] (2) The present invention comprehensively considers the problem of different feature representation capabilities in different tracking environments, combines the fluctuation degree and peak value of the relevant filter response graph under different features, uses the credibility evaluation mechanism to measure the importance of the tracking results, and dynamically allocates the fusion weights under different features, thereby improving the tracking robustness of the system under different environments;

[0042] (3) To address the impact of model update frequency on system computing power and model discrimination, the present invention seeks a criterion for determining whether the model has been updated by calculating the degree of overlap in tracking results. When the overlap is greater than a set threshold, the target template and detection model are maintained to reduce the introduction of interference information; conversely, when the overlap is less than the set threshold, the tracking model needs to be updated to ensure that the system can perceive target changes in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Flow chart of the method of the present invention.

[0044] Figure 2 (a) is the accuracy curve of the method of the present invention and other classic correlation filtering algorithms under the OPE results on the OTB2013 dataset.

[0045] Figure 2 (b) is a success curve diagram of the OPE results of the method of the present invention and other classic correlation filtering algorithms on the OTB2013 dataset.

[0046] Figure 3 (a) is the accuracy curve of the method of the present invention and other classic correlation filtering algorithms under the OPE results on the OTB2015 dataset.

[0047] Figure 3 (b) is a success curve diagram of the OPE results of the method of the present invention and other classic correlation filtering algorithms on the OTB2015 dataset. DETAILED DESCRIPTION

[0048] The technical solutions of the present invention will be described clearly and completely below in conjunction with the accompanying drawings and specific embodiments of the present invention. It should be understood that the described embodiments are intended only to explain the present invention and are not to be construed as limiting the present invention. In addition, all other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0049] The multi-core learning-based aggregate correlation filtering target tracking method of the present invention specifically includes the following steps:

[0050] Step 1: Read the initial frame of the video image, complete the interception of the target image area based on the known information, extract the different modal features of the target image, initialize the target template, detection model and other related parameters of the aggregated correlation filter based on multi-kernel learning.

[0051] (1) Read the initial frame of the video image, and cut out the target image area rectangle [x c ,y c ,w,h], where x c and y c They represent the horizontal and vertical coordinates of the target center, w is the width of the rectangular box, and h is the height of the rectangular box;

[0052] (2) Calculate the Histogram of Oriented Gradients (HoG) features, Color Name (CN) features and Convolution (CNNs) features of the above rectangular box respectively;

[0053] (3) Initialize feature fusion weights

[0054] (4) Initialize the regularization coefficient λ = 10 -4 and learning rate τ = 0.015.

[0055] Step 2: Read the next frame of video image data, intercept the potential tracking area of ​​the target in the current frame based on the tracking results of the previous frame, and then use the cyclic shift strategy to complete the dense division of the potential tracking area of ​​the current frame and obtain multiple candidate samples, and then complete the extraction of different modal features of each candidate sample.

[0056] (1) According to the tracking result of frame t-1 [x c t-1 ,y c t-1 ,w t-1 ,h t-1 ] Intercept the target potential tracking area in the t-th frame image, whose size is M×N;

[0057] (2) Use the cyclic shift strategy to complete the dense division of the potential tracking area in the t-th frame image to obtain multiple candidate samples, and extract different modal features of each candidate sample respectively where (m,n)∈{0,1,...,M-1}×{0,1,...,N-1};

[0058] (3) Describe the features of the potential tracking area in the t-th frame image i is the serial number of different features;

[0059] In step three, multiple filter functions with different parameters are used to learn the three features obtained in step two separately. At the same time, the tracker is trained using a multi-core learning algorithm and related filtering technology to obtain the optimal classification function suitable for the current tracking environment.

[0060] (1) When constructing the target appearance model, using only a single filter function can easily weaken the discriminative power between features. To this end, we use three independent filter functions with different parameters to learn different features. Specifically, we use ridge regression to learn the filter function w of the i-th feature. i,t The method is:

[0061]

[0062] Among them, y t represents the Gaussian label corresponding to the target location at the t-th frame, and λ is the regularization coefficient used to control the overfitting of the model;

[0063] (2) Using Gaussian kernel function All features are mapped to high dimensions to achieve linear separability, and the filter function w is obtained i,t Isomorphism mapping α in the dual space i,t ;

[0064] (3) Further use of classification functions Calculate the similarity between each candidate sample and the target template respectively, and then obtain the relevant filter response map C of the potential tracking area under different features in the t-th frame image i,t .

[0065] In step 4, the contribution of different features to the final prediction result is evaluated by combining the fluctuation degree and peak value of the correlation filter response graph obtained in step 3, and then the weights of the above different features are dynamically allocated.

[0066] (1) A large number of experimental results show that the accuracy of the system in predicting the target state is directly related to the fluctuation degree and the number of peaks in the correlation filter response graph. To this end, the present invention proposes a method for evaluating the fluctuation degree of the correlation filter response graph using the average peak correlation energy method:

[0067]

[0068] Among them, C max i,t , C min i,t and C p,k i,t They represent the maximum response value, minimum response value and response value at position (p, k) of the correlation filter response map of the i-th type feature extracted from the t-th frame image;

[0069] (2) In the process of synthesizing the target appearance model, the fixed parameter fusion mechanism is difficult to alleviate the interference of the external environment and the target itself on the system. Therefore, the output response values ​​of the three features should be weighted and fused according to the credibility of the tracking results. The present invention designs a redistribution mechanism to dynamically allocate the fusion modeling weight coefficients corresponding to different modal features, specifically:

[0070]

[0071] Among them, C max t and η i,t They respectively represent the maximum response value after fusion and the redistributed weight coefficient corresponding to the i-th feature in the response map of the t-th frame image.

[0072] Use the following formula to predict the target at the tth frame, that is, the center position of the current frame (x c t ,y c t ), obtain the tracking result of the current frame;

[0073]

[0074] In step 5, a strategy is proposed that uses the degree of overlap in tracking results to determine whether to update the system tracking model. If an update is required, the target template and detection model are updated using the prediction results of the current frame. Jump to step 2 until the end of the video.

[0075] (1) An appropriate update strategy can resolve the contradiction between the system computing power and the model's discriminative power. To this end, the present invention seeks a criterion for determining whether the system tracking model is updated by calculating the degree of overlap of tracking results. When the overlap is greater than the set threshold, the target template and the detection model are kept unchanged to reduce the introduction of interference information; conversely, when the overlap is less than the set threshold, the tracking model needs to be updated to ensure that the system perceives the target changes in a timely manner. The present invention sets the overlap threshold to 0.5. Specifically:

[0076]

[0077] Among them, O t Indicates the overlap of tracking results in the current frame, B t-1 and B t They represent the target bounding boxes predicted by the system in the t-1th frame and the tth frame video image respectively.

[0078] (2) Next, the target template is interpolated by linear interpolation. and filter function Update, specifically:

[0079]

[0080] This paper uses HoG features, CN features, and CNNs features to construct an object appearance model for three reasons. First, HoG features can mitigate the effects of environmental changes, such as lighting variations, on the object's appearance. Second, CN features can handle changes in the object itself, such as rotation and scale. Third, CNNs features deeply mine the correlations between image data to obtain more semantic information, effectively solving object classification problems such as deformation and motion blur.

[0081] The simulation experiment of the method of the present invention is as follows:

[0082] In order to evaluate the tracking effect of the proposed method (Ours), this experiment selected the public OTB dataset, used the one-time evaluation method (OPE) to conduct experiments on all video images in the dataset, and compared them with the mainstream kernel correlation filter algorithm (KCF), discriminant scale space algorithm (DSST) and fully convolutional twin network algorithm (SiamFC). Specifically, the evaluation method divides the factors affecting the tracking results into 11 different attributes, using accuracy and success rate as evaluation criteria. Under 11 different attributes, the accuracy and success rate of the tracking results of all algorithms are shown in Tables 1 and 2, and the best results of each group are underlined.

[0083] Table 1 Accuracy of four algorithms under 11 attributes

[0084] Ours KCF DSST SiamFC IV <![CDATA[ 75.1 ]]> 65.8 69.2 73.6 SV <![CDATA[ 76.9 ]]> 52.9 64.5 73.5 OCC <![CDATA[ 72.5 ]]> 53.7 58.3 72.2 DEF <![CDATA[ 71.8 ]]> 55.7 55.0 69.7 MB <![CDATA[ 73.4 ]]> 39.8 55.7 72.4 FM <![CDATA[ 73.2 ]]> 39.8 56.4 73.0 IPR <![CDATA[ 75.8 ]]> 64.3 68.7 74.2 OPR 75.7 60.6 65.1 <![CDATA[ 76.0 ]]> OV <![CDATA[ 68.1 ]]> 32.0 48.0 66.9 BC <![CDATA[ 74.3 ]]> 63.7 71.0 69.8 LR 67.1 50.1 56.9 <![CDATA[ 79.6 ]]>

[0085] Table 2 Success rates of four algorithms under 11 attributes

[0086] Ours KCF DSST SiamFC IV <![CDATA[ 74.1 ]]> 49.7 52.9 71.0 SV <![CDATA[ 69.2 ]]> 31.4 41.8 68.9 OCC <![CDATA[ 70.1 ]]> 42.6 45 68.3 DEF <![CDATA[ 65.7 ]]> 44.2 44.5 64.1 MB <![CDATA[ 71.2 ]]> 31.3 48.5 70.3 FM <![CDATA[ 70.0 ]]> 28.5 45.3 69.3 IPR <![CDATA[ 71.8 ]]> 48.8 53.7 69.7 OPR 70.2 46.4 49.9 <![CDATA[ 70.9 ]]> OV <![CDATA[ 63.5 ]]> 25.7 41.5 62.8 BC <![CDATA[ 69.2 ]]> 52.2 58.0 66.3 LR 52.4 28.2 28.7 <![CDATA[ 75.1 ]]>

[0087] It can be seen from Table 1 and Table 2 that, compared with the mainstream correlation filter tracking algorithm, the tracking accuracy and success rate of the method of the present invention are better than those of other algorithms in most properties.

[0088] In order to further verify the effectiveness of the method of the present invention, the precision curve (Precision Plot) based on the center position error and the success curve (Success Plot) based on the overlap rate are used to demonstrate the comprehensive performance of the algorithm and complete the ranking, as shown in the following figure: Figure 2 and Figure 3 As shown above, thanks to the addition of convolutional features, the overall tracking performance of the proposed method has significantly improved compared to the classic filtering algorithms KCF and DSST, which only use handcrafted features. Furthermore, compared to the SiamFC algorithm, which only uses convolutional features, the proposed method also outperforms similar algorithms in tracking performance due to its enhanced credibility assessment, multi-feature autonomous fusion, and model update determination.

Claims

1. A multi-kernel learning-based aggregate correlation filtering target tracking method, comprising the following steps: Step 1: read the initial frame of the video image, intercept the target image area, and extract different modal features of the target image; Initialize the target template, detection model and other related parameters of the aggregate correlation filter based on multi-kernel learning; Step 2: Read the current frame video image data, intercept the target's potential tracking area in the current frame based on the previous frame tracking result, and then use the cyclic shift strategy to complete the dense division of the potential tracking area of ​​the current frame and obtain multiple candidate samples. Then, complete the extraction of different modal features of each candidate sample. The method is as follows: (1) Let the current frame be the t-th frame, and according to the tracking result of the t-1-th frame [x c t-1 ,y c t-1 ,w t-1 ,h t-1 ] Intercept the target potential tracking area in the t-th frame image, whose size is M×N; (2) Use the cyclic shift strategy to complete the dense division of the potential tracking area in the t-th frame image to obtain multiple candidate samples, and extract different modal features of each candidate sample respectively where (m,n)∈{0,1,...,M-1}×{0,1,...,N-1}; (3) Describe the features of the potential tracking area in the t-th frame image i is the serial number of different features; Step 3: Use multiple filter functions with different parameters to learn the different features obtained in step 2 respectively. At the same time, use the multi-kernel learning algorithm and correlation filtering technology to train the tracker to obtain the optimal classification function suitable for the current tracking environment, and obtain the correlation filter response map of the potential tracking area in the current frame image under different features; Step 4: Combine the fluctuation and peak values ​​of the correlation filter response graph obtained in step 3 to evaluate the contribution of different features to the final prediction result, and then dynamically assign weights to the above different features as follows: (1) Use the average peak correlation energy method to evaluate the fluctuation degree of the correlation filter response graph: Among them, C max i,t , C min i,t and C p,k i,t They represent the maximum response value, minimum response value and response value at position (p, k) of the correlation filter response map of the i-th type feature extracted from the current frame, i.e., the t-th frame image; (2) There are three types of features of different modalities. The output response values ​​of the three features are dynamically weighted and fused according to the credibility of the tracking results. Specifically: Among them, C max t and η i,t They represent the maximum response value after fusion and the redistributed weight coefficient corresponding to the i-th feature in the response graph of the t-th frame image respectively; Use the following formula to predict the target at the tth frame, that is, the center position of the current frame (x c t ,y c t ), get the tracking result of the current frame: Step 5: Use the overlap of the tracking results to decide whether to update. If an update is required, use the prediction results of the current frame to update the target template and detection model, and jump to step 2 until the end of the video. The method is as follows: (1) Determine the overlap degree based on the tracking result of the current frame. When the overlap degree is greater than the set threshold, the target template and detection model remain unchanged. Conversely, when the overlap degree is less than the set threshold, the target template and detection model need to be updated. The overlap degree threshold is set to 0.

5. (2) Update the target template and filter function through linear interpolation.

2. The method for tracking a target using aggregated correlation filtering according to claim 1, wherein: In step 1, the method of reading the initial frame of the video image, intercepting the target image area, and extracting different modal features of the target image is as follows: (1) Read the initial frame of the video image and capture the target image area [x c ,y c ,w,h] rectangular box, where x c and y c They represent the horizontal and vertical coordinates of the target center, w is the width of the rectangular box, and h is the height of the rectangular box; (2) Calculate the three features of the above rectangular box respectively: Histogram of Oriented Gradients (HoG) feature, Color Name (CN) feature and Convolution (CNNs) feature; (3) Initialize feature fusion weights (4) Initialize the regularization coefficient and learning rate.

3. The target tracking method using aggregated correlation filtering according to claim 1, wherein: The method for step three is as follows: (1) Using independent filter functions with different parameters to learn different features; (2) Using Gaussian kernel function All features are mapped to high dimensions to achieve linear separability, and the filter function w is obtained i,t Isomorphism mapping α in the dual space i,t ; (3) Further use of classification functions Calculate the similarity between each candidate sample and the target template respectively, and then obtain the relevant filter response map C of the potential tracking area under different features in the current frame, i.e., the t-th frame image i ,t .

4. The method for tracking a target using aggregated correlation filtering according to claim 3, wherein: There are three types of features obtained in step 2. In step 3 (1), ridge regression is used to learn the filter function w of the i-th feature. i,t The method is: Among them, y t represents the Gaussian label corresponding to the target location in the t-th frame, and λ is the regularization coefficient used to control the overfitting of the model.

Images