Single target tracking template updating method based on attention mechanism

Abstract

The invention discloses a single target tracking template updating method based on an attention mechanism, and the method comprises the steps: 1, introducing an SENet channel attention mechanism into the two-layer convolution of UpdatNet to order channel information, completing the fusion of an initial template, an accumulation template and a prediction template, and obtaining a template for tracking a next frame of picture; 2, in the fusion process of the initial template, the accumulation template and the prediction template, determining whether the prediction template of the frame can participate in accumulation or not by using a confidence score output by an offline model tracking head; 3, generating data of a training model, and judging whether tracking fails or not by using different thresholds so as to generate training data under the different thresholds; and 4, training a model by adopting the training data generated in the step 3, wherein various learning rates are adopted. When the model is trained, a plurality of training data and a training learning rate which are generated under a threshold value for judging whether tracking fails are provided, so that the model can be converged more quickly.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and target tracking, and specifically relates to a single target tracking template updating method based on an attention mechanism. Background Art

[0002] Target tracking is a model that determines the location of the target in subsequent frames based on the target annotation of the first frame. Target tracking technology has a wide range of applications in autonomous driving, intelligent monitoring and other fields. Common single target tracking methods include traditional algorithms that manually select features and intelligent algorithms that automatically extract features based on deep learning. The performance of intelligent algorithms is far superior to traditional algorithms. Intelligent algorithms usually use CNN structures or transformer structures to extract features. In single target tracking, the commonly used CNN structure is the twin network, which is to build a symmetrical dual-branch feature extraction network for the tracking template and the search area, and then filter the outputs of the two networks. Finally, the position with the highest response in the search area is the target location predicted by the model. Twin networks are widely used and studied because of their simple structure and superior performance.

[0003] The template used by the twin network in the entire process of target tracking is generally the first frame, but the targets in the video stream often deform or partially disappear over time. If the target shape in the first frame is always used as a template, it is obviously not advisable; but if the prediction result of the previous frame is used as a template for each prediction, the tracking result may be worse because the prediction result is not completely reliable.

[0004] At present, there are many works on updating templates for single target tracking: (1) Using linear accumulation to accumulate previous prediction results to obtain the template of the next frame. This method can fuse all the forms of the target. However, the predicted tracking results are not completely reliable. The fusion process will bring in a lot of harmful information, and the accumulation parameters also need to be carefully set, which ultimately leads to the fact that this method has very limited effect on improving the performance of target tracking. (2) Establish a template pool to store a certain amount of templates, and use relevant indicators to judge whether the prediction results can be added to the template pool. The template used for tracking is obtained by fusing all the templates in the template pool using a certain fusion method, but the selected relevant indicators are often unable to completely and accurately select reliable or effective templates. (3) Use multiple indicator information to train a linear network to judge the reliability of the predicted template, but most of the data used to train the network are based on the prediction results, so the reliability is not high.

[0005] Existing template updating methods have very limited effects on improving single target tracking performance, and the effects are unstable and the generalization is poor. Therefore, a template updating method that can significantly and stably improve single target tracking performance is needed. Summary of the invention

[0006] The present invention aims at the current situation that the existing template update algorithm in the field of single target tracking does not significantly improve the tracking performance and has poor generalization performance, and provides a single target tracking template update method based on the attention mechanism. The method uses UpdateNet as the basis for template update, introduces the SENet channel attention mechanism into the two-layer convolution of UpdateNet, highlights the channels carrying important information, and uses the confidence score of the predicted template output by the tracking head of the offline model to determine whether the predicted template of the current frame can participate in the template accumulation process; in the experimental process, the model is provided with more abundant data and parameters, and the model can converge faster.

[0007] The technical solution adopted by the present invention is as follows: a single target tracking template updating method based on an attention mechanism comprises the following four steps: Step 1: Introduce the SENet channel attention mechanism in the two-layer convolution of UpdateNet to sort the channel information, complete the fusion of the initial template, the accumulated template and the predicted template, and obtain the template for tracking the next frame of the picture; Step 2: In the fusion process of the initial template, the accumulated template and the predicted template, the confidence score output by the offline model tracking head is used to determine whether the predicted template of this frame can participate in the accumulation; Step 3: Generate data for training the model, wherein different thresholds are used to determine whether tracking fails, so as to generate training data under different thresholds; Step 4: Use the training data generated in step 3 to train the model, using multiple learning rates.

[0008] Beneficial effects: By providing multiple training data generated under thresholds for determining whether tracking fails and training learning rates, the model can converge faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a specific schematic diagram of the single target tracking template updating method based on the attention mechanism of the present invention; Figure 2 (a) shows the target tracking results obtained by executing the three tracking methods shown in Figure 2 (d) on different frames of a basketball game video, where the boxes represent the target tracking results; Figure 2 (b) shows the target tracking results obtained by executing the three tracking methods shown in Figure 2 (d) on different frames of the movie video, where the boxes represent the target tracking results; Figure 2 (c) shows the target tracking results obtained by executing the three tracking methods shown in Figure 2 (d) on different frames of the concert video, where the boxes represent the target tracking results; FIG2( d ) shows three tracking methods, namely: UPDT, ECO, and the method of the present invention. DETAILED DESCRIPTION

[0010] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms, and the present disclosure should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0011] Figure 1 FIG. 2 is a schematic diagram showing a single target tracking template updating method based on the attention mechanism of the present invention. Figure 1 As shown, the method comprises the following steps: Step 1: Introduce the SENet channel attention mechanism in the two-layer convolution of UpdateNet to sort the channel information, complete the fusion of the initial template, cumulative template and predicted template, and obtain the template for tracking the next frame of the picture.

[0012] Step 2: In the fusion process of the initial template, the accumulated template and the predicted template, the confidence score output by the offline model tracking head is used to determine whether the predicted template of this frame can participate in the accumulation. This step plays a screening role, determining whether the predicted template in step 1 can participate in the fusion of the next frame template.

[0013] Step 3: Generate data for training the model, wherein different thresholds are used to determine whether tracking fails, so as to generate training data under different thresholds. This step can provide more abundant training data for model training. The model refers to a model used to execute the single target tracking template update method of the present invention.

[0014] Step 4: Use the training data generated in step 3 to train the model, using multiple learning rates.

[0015] The following describes step 1: Introducing the SENet channel attention mechanism in the two-layer convolution of UpdateNet to sort the channel information and complete the fusion process of the initial template, the accumulated template and the predicted template.

[0016] Template fusion consists of four parts: the data needs to be processed in sequence by the first convolution layer, the activation function ReLU, the channel attention mechanism SENet, and the second convolution layer. The size of all templates used is The initial template, cumulative template and predicted template of this frame are spliced ​​together by channel to obtain the input data Then the concatenated data is input into the first convolutional layer and activation function to obtain data The data is fed into SENet to sort the information. SENet first performs a transformation operation on the input data, which can be regarded as a simple convolution. In this process, the size of the data does not change. SENet then performs a transformation on the output data (denoted as ) performs a squeeze operation. Squeeze is to perform a global average pooling operation on the input data to obtain a value that can represent each channel. This value can be calculated by After this operation, the data becomes The calculation process is shown in formula (1): (1) The meanings of the symbols in formula (1) are: : Output data; : Input data; : data height; : data width; : Squeeze operation function expression.

[0017] In order to achieve automatic weight design, SENet performs an excitation operation on the data. The specific operation is to let the input data first pass through a linear layer to reduce the channel dimension, then use the ReLU function to activate it, and then pass through a linear layer to increase the channel dimension, so that the dimension of the output data channel is the same as the input data. Finally, the weight data is generated. The calculation process is shown in formula (2): (2) The meanings of the symbols in formula (2) are: : ReLU function; z: input data; : Function expression of dimension raising and lowering operation; W: specific numerical expression of channel dimension increase or decrease; : Excitation operation function expression; s: weight data; : Sigmoid function; , C is the number of data channels, r is the dimension-raising and lowering parameter; .

[0018] Finally, the obtained channel weights are used to calibrate the data that originally entered SENet. The calibration process is shown in formula (3): (3) in, is the calibrated data, is the calibration function expression, is the weight data, It is the input data to be calibrated, that is, the output data after the above SENet transformation operation.

[0019] At this point, we get a feature map where the channel information has been recalibrated and the channels carrying important information have been enhanced. Then, this feature map data is input into the second convolutional layer to get the template data Finally, in order to reduce the risk of unreliable prediction templates, the initial template is added to the template data again. In this way, the template of the next tracking frame is obtained. The template of the tracking frame refers to the template obtained by fusing the initial template, the prediction template, and the accumulated template. One of the purposes of the present invention is to effectively fuse these three templates so that the final template can be as similar as possible to the target shape in the next tracking frame.

[0020] The screening mechanism described in step 2 is as follows: In one embodiment, an offline tracking model DaSiamRPN is used. The confidence score of the prediction result output by the tracking head of the offline model is used as the basis for judging whether the prediction template of the current frame is reliable, and then determining whether the prediction template of a frame can be added to the template accumulation. The selection mechanism is defined as follows: (4) Where s is the confidence score, is the screening result, is the threshold set for the confidence score. If the confidence of the model prediction result is lower than the preset threshold, the prediction template will not be added to the accumulation. In the template filtering process, the threshold needs to be set very finely. The setting is done automatically by the program. The program will try to set the threshold every 0.01 from 0 to 1, and then the program will select the best threshold corresponding to the tracking effect as the final selected threshold. If it is set too high, the accumulated templates will lose too much information, resulting in a decrease in tracking performance; if it is set too low, harmful information will not be filtered out, and the model performance cannot be effectively improved. In theory, this threshold cannot be set too high, because the confidence scores of the prediction results output by the offline model are not completely reliable. Some high-confidence prediction results may not be of high quality, while some prediction results may not have a high enough confidence, but the tracking results are very satisfactory. It can only filter out some prediction templates with too low confidence and serious pollution. If the threshold is set high, some valid information can easily be excluded from the accumulation process. At this time, the screening mechanism will have side effects.

[0021] Step 3: Generate data for training the model, where different thresholds are used to determine whether tracking fails, so as to generate training data under different thresholds, including: When testing the offline model DaSiamRPN on a selected portion of LaSOT data, different thresholds (for example, a value of 0.1 to 0.65 at intervals of 0.05) are used to determine whether tracking fails, so that the initial template, accumulated template, predicted template data, and label template under different thresholds can be obtained. The higher the threshold is set, the higher the accuracy of the saved template with predicted information. In this way, template data of different qualities can be selected. When training the model, let the model select the most suitable training data at any training stage.

[0022] The training method described in step 4 is as follows: The training of UpdateNet is very cumbersome and takes a long time to reach convergence. The training process is divided into three stages, each of which contains two parts. The two parts of the same stage use the same training data generated by the same offline model DaSiamRPN. In each part of the training, let the model try multiple training learning rates (for example, you can take a value every 0.5 from 4.5 to 11 to obtain data in the form of [4.5, 5], [5,5], [5,5.5], ...), so that the model has multiple training data and learning rates generated by the same offline model at different thresholds to choose from, so that the training data and learning rate can be freely matched, so that the model can obtain the best training effect. Experiments show that using training data generated at different thresholds in two parts of the same training stage will achieve better tracking results and faster convergence speed. Using this method, the model can reach convergence in the second part of the second stage or even the first part, and the model also shows good performance earlier.

[0023] An experimental example is described below: 1) Experimental conditions: The computer used in the experiment is an Intel(R) Core(TM) i5-8500 CPU with 32.0GB memory, a GeForce RTX 3090 GPU with 24GB memory, and a programming platform of PyCharm 2023.2.5. The programming tools mainly use python 3.8 and PyTorch 2.1.1, and the test datasets used in the experiment are VOT2016 and VOT2018.

[0024] 2) Experimental content: The method proposed in this invention is compared with other classical single target tracking algorithms, among which the commonly used single target tracking algorithms mainly include: (1) ECO: Danelljan M, Bhat G, Shahbaz Khan F, et al. Eco: Efficientconvolution operators for tracking[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 6638-6646.

[0025] (2) DaSiamRPN: Zhu Z, Wang Q, Li B, et al. Distractor-aware siamese networks for visual object tracking[C] / / Proceedings of the European conference on computervision (ECCV). 2018: 101-117.

[0026] (3) UPDT: Bhat G, Johnander J, Danelljan M, et al. Unveiling the power of deeptracking[C] / / Proceedings of the European conference on computer vision(ECCV). 2018: 483-498.

[0027] (4) ATOM: Danelljan M, Bhat G, Khan F S, et al. Atom: Accurate tracking byoverlap maximization[C] / / Proceedings of the IEEE / CVF conference on computervision and pattern recognition. 2019: 4660-4669。

[0028] (5) DIMP: Bhat G, Danelljan M, Gool L V, et al. Learning discriminative modelprediction for tracking[C] / / Proceedings of the IEEE / CVF internationalconference on computer vision. 2019: 6182-6191。

[0029] (6) SiamRPN: Li B, Yan J, Wu W, et al. High performance visual tracking withsiamese region proposal network[C] / / Proceedings of the IEEE conference oncomputer vision and pattern recognition. 2018: 8971-8980。

[0030] (7) ASRCF: Dai K, Wang D, Lu H, et al. Visual tracking via adaptive spatially-regularized correlation filters[C] / / Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition. 2019: 4670-4679。

[0031] (8) UpdateNet-DaSiamRPN: Zhang L, Gonzalez-Garcia A, Weijer JVD, et al. Learning the model update for siamese trackers[C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2019: 4010-4019.

[0032] Figure 2 (a), Figure 2 (b), and Figure 2 (c) show the target tracking results of the three methods shown in Figure 2 (d) for the selected experimental sequence. The experimental results show that compared with the other two methods, the algorithm of the present invention has no tracking failure; after accurately locating the tracking target, the image framed by the algorithm of the present invention is more consistent with the target as a whole.

[0033] This experiment uses accuracy, robustness and expected average overlap (EAO) as evaluation indicators, which can reflect the tracking accuracy and robustness of the algorithm. Table 1 shows the accuracy, robustness and EAO of other classic single target tracking methods and the present invention on the VOT2016 dataset. The experimental results show that the present invention has the largest accuracy and EAO and the smallest robustness on the test dataset, indicating that the target tracking accuracy and reliability of the present invention are better than other classic single target tracking methods.

[0034] Table 1

[0035] The present invention proposes a single target tracking template update method based on an attention mechanism. The present invention analyzes the limitations of existing single target tracking template update methods. The present invention introduces the SENet channel attention mechanism into the two-layer convolution of UpdateNet, sorts the information carried by each channel, and highlights the channels carrying important information; uses the confidence score of the prediction template output by the offline model to determine whether the prediction template can participate in the template accumulation process; when generating model training data, multiple thresholds are used to determine whether the tracking fails, so that more abundant training data can be obtained; when training the model, the model is provided with training data and model learning rates generated under different thresholds, so that the model can converge faster. The present invention proposes an improved method based on UpdateNet, which effectively releases the performance of UpdateNet and improves the tracking performance of the offline model.

[0036] The parts not described in detail in the present invention belong to the known technologies in the technical field. The above description is only a specific example of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A single target tracking template updating method based on attention mechanism, characterized in that: The following steps are involved: Step 1: Introduce the SENet channel attention mechanism in the two-layer convolution of UpdateNet to sort the channel information, complete the fusion of the initial template, the accumulated template and the predicted template, and obtain the template for tracking the next frame of the picture; Step 2: In the fusion process of the initial template, the accumulated template and the predicted template, the confidence score output by the offline model tracking head is used to determine whether the predicted template of this frame can participate in the accumulation; Step 3: Generate data for training the model, wherein different thresholds are used to determine whether tracking fails, so as to generate training data under different thresholds; Step 4: Use the training data generated in step 3 to train the model, using multiple learning rates.

2. The single target tracking template updating method based on the attention mechanism according to claim 1 is characterized in that: Step 1 includes: The fusion of the initial template, the accumulated template and the predicted template consists of four parts: the data needs to be processed by the first convolution layer, the activation function ReLU, the channel attention mechanism SENet and the second convolution layer in sequence. The size of all templates used is , the initial template, cumulative template and predicted template of this frame are spliced ​​together by channel to get the input data , and then input the concatenated data into the first convolutional layer and activation function to obtain data , the data is fed into SENet to sort the information. SENet first performs a transformation operation on the input data, and the output data is recorded as , then SENet performs a squeeze operation on the output data after the transformation operation to obtain data .

3. The single target tracking template updating method based on the attention mechanism according to claim 2 is characterized in that: SENet performs excitation operations on the data, including: letting the input data pass through a linear layer to reduce the channel dimension, then using the ReLU function for activation, and then passing through a linear layer to increase the channel dimension so that the dimension of the output data channel is the same as the input data, and finally, generating weight data , the calculation process is shown in formula (2): (2) The meanings of the symbols in formula (2) are: : ReLU function; z: input data; : Function expression of dimension raising and lowering operation; W: specific numerical expression of channel dimension increase or decrease; : Excitation operation function expression; s: weight data; : Sigmoid function; , C is the number of data channels, r is the dimension-raising and lowering parameter; ； Finally, the obtained channel weights are used to calibrate the data that originally entered SENet. The calibration process is shown in formula (3): (3) in, is the calibrated data, is the calibration function expression, is the weight data, is the input data to be calibrated; At this point, we have obtained a feature map in which the channel information has been recalibrated and the channel carrying important information has been enhanced. This feature map data is input into the second convolutional layer to obtain the template data , add the initial template to the template data again , obtain the template for the next tracking frame.

4. The single target tracking template updating method based on the attention mechanism according to claim 1 is characterized in that: Step 2 includes: The confidence score of the prediction result output by the tracking head of the offline tracking model DaSiamRPN is used as the basis for judging whether the prediction template of this frame participates in the accumulation. The selection mechanism is defined as follows: (4) Where s is the confidence score, is the screening result, is the threshold set for the confidence score; Among them, if the confidence of the model prediction result is lower than the preset threshold, the prediction template will not participate in the accumulation.

5. The single target tracking template updating method based on the attention mechanism according to claim 4 is characterized in that: Threshold The setting is automatically completed by the program, and the process includes: trying to set the threshold every 0.01 from 0 to 1, and selecting the best threshold corresponding to the tracking effect as the final selected threshold.

6. The single target tracking template updating method based on the attention mechanism according to claim 1 is characterized in that: Step three includes: The offline model DaSiamRPN uses thresholds of different sizes to determine whether tracking fails when tested on a selected portion of LaSOT data, thereby obtaining initial templates, accumulated templates, predicted template data, and label templates under different thresholds.

7. The single target tracking template updating method based on the attention mechanism according to claim 1 is characterized in that: The training process in step 4 is divided into three stages, each of which contains two parts. The two parts of the same stage use the same training data generated by the same model. In each part of the training, the model is allowed to try multiple training learning rates, so that the model has multiple training data and learning rates generated by the same offline model under different thresholds to choose from, allowing the training data and learning rate to match freely, so that the updated model can achieve the best training effect.

Images