Single-target tracking method and tracking system based on channel attention and spatiotemporal awareness
By using a channel attention and spatiotemporal awareness-based approach, the template and search region feature maps are fused together. Target prediction is performed by combining classification and regression branches, and the template is updated in real time. This solves the robustness problem of single-target tracking in complex scenarios and improves tracking success rate and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF AERONAUTICS & ASTRONAUTICS
- Filing Date
- 2023-06-14
- Publication Date
- 2026-05-26
AI Technical Summary
Existing single-target tracking methods lack robustness when faced with complex situations such as target motion, background changes, lighting changes, and occlusion, making it difficult to maintain tracking accuracy and success rate.
We employ a channel attention and spatiotemporal awareness-based approach, extracting feature maps using two weight-sharing RepVGG backbone networks, combining them with an autonomous network for feature extraction, fusing template and search region feature maps, introducing channel attention and cross attention mechanisms, and combining classification branches, bounding box regression branches, and centrality regression branches for target prediction. We also achieve real-time template updates through threshold judgment.
It improves the robustness of target tracking, better copes with challenges such as occlusion and deformation, and enhances tracking success rate and accuracy.
Smart Images
Figure CN116797799B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning target tracking technology, and in particular to a single target tracking method and system based on channel attention and spatiotemporal awareness. Background Technology
[0002] Single-object video tracking is an important research area in computer vision, playing a crucial role in applications such as video surveillance, intelligent transportation, and airport surface monitoring. Object tracking uses the initial position of the tracked object as a reference, aiming to accurately predict the target's state parameters in subsequent frames. In real-world scenarios, the performance of object tracking is limited by factors such as target motion, background and lighting changes, and occlusion.
[0003] Existing methods based on attention mechanisms effectively improve the semantic relevance between the template and the feature map of the search region through self-attention and cross-attention mechanisms. However, different channel features often assign the same weight to the feature channels, without fully considering the different contributions of different channels to the tracking task.
[0004] Most existing work does not employ an online template update strategy. However, in actual tracking tasks, the appearance of the target often changes significantly, and there are complex situations such as occlusion by interference objects, changes in lighting, and even the target disappearing and then reappearing in subsequent video frames during movement. Using only a single initial frame as a template cannot effectively handle these complex changes, which may lead to target tracking drift or even loss. Summary of the Invention
[0005] The technical problem to be solved by the present invention is to provide a single target tracking method and tracking system based on channel attention and spatiotemporal awareness, which allows the template features to adaptively enhance their completeness as they change with time and space, thereby improving the robustness of tracking.
[0006] To address the aforementioned technical problems, this invention provides a single-target tracking method based on channel attention and spatiotemporal awareness, comprising the following steps:
[0007] Step 1: Input the template image and the search region image, and extract the hierarchical features of the two images respectively through two RepVGG backbone networks with shared weights;
[0008] Step 2: Perform feature fusion of the two feature maps using a fusion model based on channel attention mechanism, combining self-attention and cross-attention.
[0009] Step 3: Complete the target tracking and prediction through classification branch, bounding box regression branch, and centrality regression branch;
[0010] Step 4: Real-time template updates are completed by using threshold judgment, and the temporal information of subsequent frames is added to the fusion network to achieve complete target tracking;
[0011] Step 5: Demonstrate tracking performance.
[0012] Preferably, in step 1, the hierarchical features of the template image and the search region image are extracted using two RepVGG backbone networks with shared weights. First, the template image and the search region image are fed into the RepVGG backbone network to extract features. The template features are output at the third layer and the search region features are output at the fifth layer to ensure that their scales are the same. Then, the two sets of outputs are processed... The convolution kernel adjusts the number of channels to 256.
[0013] Preferably, in step 2, the feature fusion of the two feature maps using a fusion model based on channel attention mechanism includes the following steps:
[0014] Step 21: Input sequence template features and enhance the features through a self-attention module to fully associate the pixels of each part of the template features;
[0015] Step 22: Receive the search features as one part of the input, and receive the output from the template features as another part of the input. Calculate the cross-attention between the two parts through the cross-attention module. The calculation formula of the cross-attention module is the same as that of the self-attention module, only the input is different.
[0016] Step 23: Input the results of the cross-attention module into the global max pooling layer and average pooling layer in the length and width dimensions. Then, feed the two pooling outputs into a parameter-shared MLP layer to learn the relationship between channels. After adding the results, feed them into the sigmoid function for activation to obtain the channel attention weights.
[0017] Preferably, in step 21, the pixels of each part of the template feature are fully correlated:
[0018]
[0019]
[0020]
[0021] in Representing the Multiple heads of attention output, Representing the first The query, key, and value change matrix of the head. Representing the first The query, key, and value matrix of each head is used to concatenate multiple multi-head attention results, followed by a linear transformation. The final output result is obtained.
[0022] Preferably, in step 23, the feature map size is restored to maintain size matching, and the calculation expression of the channel attention module is as follows:
[0023]
[0024] in, This represents the sigmoid activation function, and AvgPool represents the average pooling operation. The output of average pooling is... MaxPool represents the max pooling operation, and the output is... express, These represent two convolutional layers in an MLP.
[0025] Preferably, step 3, which involves tracking and predicting the target through classification branch, bounding box regression branch, and centrality regression branch, specifically includes the following steps:
[0026] Step 31: The prediction head maps the feature map after information fusion through different convolutional layers to three different spaces in the prediction head network: classification branch Cls, centrality branch Ctr, and regression branch Reg, thereby achieving accurate prediction.
[0027] Step 32: Calculate the total target loss.
[0028] Preferably, in step 32, For classifying losses, For centrality loss, To regress the loss, For the sample size, The number of positive samples is denoted by x, and y represents the coordinates of the output feature map. and For positive and negative sample labels and predicted values, and For the centrality label and predicted value, and Let these represent the true bounding box position and the predicted bounding box position, respectively. The classification loss uses binary Focal-Loss, the centrality branch uses cross-entropy loss, and the regression branch uses GIoU-Loss. The total loss is calculated as follows:
[0029] .
[0030] Preferably, in step 4, the template is updated in real time by judging the threshold, and the time information of subsequent frames is added to the fusion network to achieve complete target tracking. Specifically, this includes the following steps:
[0031] Step 41: Introduce confidence level and confidence threshold When the current frame sequence number and update interval The judgment condition is met:
[0032]
[0033] That is, when confidence level Greater than the confidence threshold And the current image sequence number Divisible by the update interval If the condition is met, the template is updated; otherwise, it is not updated. The update expression is:
[0034]
[0035] The input features are all sequence features that satisfy the Transformer conditions. Assuming that the conditional update is satisfied, the template to be updated is the current frame feature. Features accumulated previously The result of direct addition, when there is no cumulative characteristic, is... The template features of the first frame ;
[0036] Step 42: Serialize the updated template and add it to the fusion network to achieve complete target tracking.
[0037] Preferably, in step 41, the confidence threshold is... Take 0.7, and update interval The tracker performs optimally when 120 frames are captured.
[0038] Accordingly, a single-target tracking system based on channel attention and spatiotemporal awareness includes: a feature extraction module, a feature fusion module, a tracking prediction module, and a performance display module. The feature extraction module extracts hierarchical features from the input template image and the search region image through two weight-sharing RepVGG backbone networks. The feature fusion module performs self-attention and cross-attention feature fusion on the two feature maps through a fusion model based on channel attention mechanism. The tracking prediction module completes the tracking prediction of the target through classification branch, bounding box regression branch, and centrality regression branch. The template is updated in real time through threshold judgment, and the temporal information of subsequent frames is added to the fusion network to achieve complete target tracking. The performance display module displays the tracking performance.
[0039] The beneficial effects of the present invention are as follows: (1) The fusion network based on channel attention proposed in the present invention adds a channel attention module after self-attention and cross-attention learning of template features and search region features, further enhancing the contribution of different channel features to the tracking task. In the feature fusion part, the channel of interest is given a higher weight, reducing the influence of redundant channels; (2) The template update module based on spatiotemporal awareness proposed in the present invention introduces an online template update module, controls the template update frequency through a threshold, and continuously enriches the template sequence through accumulation and merging in order to ensure the diversity of templates, thereby enhancing the robustness of tracking; (3) The tracking network model proposed in the present invention has a significant improvement in tracking success rate and accuracy, and can well cope with challenges such as occlusion and deformation, improving the robustness of the tracker. Attached Figure Description
[0040] Figure 1 This is a schematic diagram of the tracking network structure of the present invention.
[0041] Figure 2 This is a schematic diagram of the fusion network of the present invention.
[0042] Figure 3 This is a schematic diagram of the channel attention module of the present invention.
[0043] Figure 4 This is a schematic diagram illustrating the principle of the prediction head network of the present invention.
[0044] Figure 5 This is a schematic diagram illustrating the comparative experimental results of this invention in dealing with "scale changes" in the LaSOT dataset.
[0045] Figure 6 This is a schematic diagram illustrating the comparative experimental results of the present invention in dealing with "complete occlusion" in the LaSOT dataset.
[0046] Figure 7 This is a schematic diagram illustrating the comparative experimental results of the present invention in dealing with "target deformation" in the LaSOT dataset. Detailed Implementation
[0047] like Figure 1As shown, this invention provides a single-target tracking method based on channel attention and spatiotemporal awareness. Its design principle is as follows: First, the template image and the search image are input together into the RepVGG backbone network for feature extraction. The outputs of the third and fifth layers of the backbone network are then fed into the Transformers of the upper and lower branches for feature fusion. The information from the two Transformers is merged and then fed into a classification and localization network to determine the target location. Finally, a threshold is used to update the template in real time. After updating the template, the temporal information of subsequent frames can be added to the network. Combined with the Transformer, the spatiotemporal information can be used for target tracking.
[0048] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.
[0049] The detailed steps of the single-target tracking method based on channel attention and spatiotemporal awareness in this embodiment are as follows:
[0050] S1: Input template image and search region image, and extract the hierarchical features of the two through two RepVGG backbone networks with shared weights;
[0051] S2: Feature fusion of two feature maps is performed using a fusion model based on channel attention mechanism, which incorporates self-attention and cross-attention.
[0052] S3: Tracking and predicting the target is accomplished through classification branch, bounding box regression branch, and centrality regression branch;
[0053] S4: Real-time template updates are achieved by using threshold judgment, and the time information of subsequent frames is added to the fusion network to achieve complete target tracking.
[0054] S5: Demonstrates the tracking performance of this embodiment when faced with challenging problems such as object deformation and target occlusion.
[0055] The following is a detailed description of each step.
[0056] In step S1, the template image and the search region image are input, and their hierarchical features are extracted using two RepVGG backbone networks with shared weights. The specific steps are as follows:
[0057] S101: Input the template image and the search image into the backbone network to extract features, setting a template image of 127×127×3 and a search image of 303×303×3 as input.
[0058] S102: Training and inference on the input using a RepVGG network. This example designs a five-layer RepVGG neural network as the feature extraction backbone, with each layer containing 1, 4, 6, 16, and 1 RepVGG blocks respectively. The training process includes three branches: a 3×3 convolution with a BN layer, a 1×1 convolution with a BN layer, and a branch where the input is processed separately through a BN layer. These are then fused together and followed by a ReLU activation function. A 1×1 convolution can be viewed as a 3×3 convolution with one parameter at the center of the kernel, while a branch without convolution can be viewed as a 3×3 convolution with a kernel centered on parameter 1. Therefore, both 1×1 convolutions and branches without convolution can be transformed into 3×3 convolutions. The calculation formula for the RepVGG training process is as follows:
[0059]
[0060] in For the final output, For the initial input, It is a 3×3 convolution. It is a 1×1 convolution.
[0061] The calculation formula for the RepVGG fusion process is shown in Equation 2. Taking the 1×1 convolution branch as an example, , , , These are the parameters of the BN layer. Represents the mean. Represents variance. , These are two learnable parameters.
[0062]
[0063] After merging, the convolution becomes a new 3×3 convolution. The convolution parameters became and The same transformation can be performed on branches that do not contain convolutions, and finally, the three different branches of the reasoning process are all transformed into reasoning process structures containing only 3×3 convolutions.
[0064] S103: Combine the outputs of layers 3 and 5 of the backbone network to enhance the model's feature representation capability. Define the output feature maps of layers 3 and 5 of the convolutional neural network as follows: A new feature map is obtained by superimposing feature maps along the channel dimension. The calculation formula is:
[0065]
[0066] Concat is a stacking operation along the channel dimension, making the number of channels 512. It contains both shallow and deep feature information.
[0067] S104: The two sets of outputs are processed by a 1×1 convolution kernel to adjust the number of channels to 256, so that the upper and lower branches can obtain a set of template features and search region features of the same size respectively.
[0068] In step S2, a fusion model based on channel attention mechanism is used to perform feature fusion of the two feature maps using self-attention and cross-attention. For example... Figure 2 As shown, the specific steps are as follows:
[0069] S201: Input sequence template features, and perform feature enhancement through a self-attention module to fully associate the pixels of each part of the template features:
[0070]
[0071]
[0072]
[0073] in Representing the Multiple heads of attention output, Representing the first The query, key, and value change matrix of the head. Representing the first The query, key, and value matrix of each head is used to concatenate multiple multi-head attention results, followed by a linear transformation. The final output result is obtained.
[0074] S202: Receives the search features as one part of the input, and the output from the template features as another part of the input. The cross-attention module calculates the cross-attention between the two parts. The calculation formula for the cross-attention module is the same as that for the self-attention module, only the input is different.
[0075] S203: The results of the cross-attention module are input to a global max-pooling layer and an average-pooling layer in both length and width dimensions. The outputs of these two pooling layers are then fed into a parameter-shared MLP layer to learn the relationships between channels. The sum of the results is then fed into a sigmoid function for activation, yielding the channel attention weights. The feature map dimensions are then restored to maintain size matching. For example... Figure 3 As shown, the computational expression of the channel attention module is as follows:
[0076]
[0077] in, This represents the sigmoid activation function, and AvgPool represents the average pooling operation. The output of average pooling is... MaxPool represents the max pooling operation, and the output is... express. These represent two convolutional layers in an MLP.
[0078] In addition, the comparison results of this embodiment on the GOT-10k dataset are shown in Table 1. The experiment shows that the tracking method with added channel attention has improved performance in many aspects.
[0079] Table 1. Comparison of tracking performance before and after adding channel attention.
[0080]
[0081] In step S3, target tracking and prediction are completed through classification branch, bounding box regression branch, and centrality regression branch. For example... Figure 4 As shown, the specific steps are as follows:
[0082] S301: The prediction head maps the feature map after information fusion through different convolutional layers to three different spaces in the prediction head network: classification branch Cls, centrality branch Ctr, and regression branch Reg, thereby achieving accurate prediction.
[0083] S302: Calculate the total target loss, where For classification loss, For centrality loss, To regress the loss, For the sample size, This represents the number of positive samples. x and y represent the coordinates of the output feature map. and For positive and negative sample labels and predicted values, and For the centrality label and predicted value, and Let represent the true bounding box position and the predicted bounding box position, respectively. The classification loss uses binary Focal-Loss, the centerness branch uses cross-entropy loss, and the regression branch uses GIoU-Loss. The total loss is calculated as follows:
[0084]
[0085] In step S4, the template is updated in real time by judging the threshold, and the temporal information of subsequent frames is added to the fusion network to achieve complete target tracking. The specific steps are as follows:
[0086] S401: Introducing Confidence Level and confidence threshold When the current frame sequence number and update interval The following conditions must be met:
[0087]
[0088] That is, when confidence level Greater than the confidence threshold And the current image sequence number Divisible by the update interval If the confidence threshold is met, the template is updated; otherwise, it is not updated. Take 0.7, and update interval Take 120 frames. The final update expression is:
[0089]
[0090] The input features are all sequence features that satisfy the Transformer conditions. Assuming the conditional update is satisfied, the template to be updated is the current frame feature. Features accumulated previously The result is obtained by direct addition. When there is no cumulative feature... The template features of the first frame .
[0091] S402: The updated template is serialized and added to the fusion network to achieve complete target tracking. To verify the effectiveness of the template update method, ablation experiments were conducted on GOT-10k, and the results are shown in Table 2. Method a represents the tracking method without the template update module, and method b represents the tracking method with the template update module. The experiments demonstrate that adding the template update module improves the tracking performance.
[0092] Table 2 Comparison of tracking performance before and after template update
[0093]
[0094] In step S5, the comparative experimental results of the tracking algorithm of this embodiment on the LaSOT dataset will be presented to address challenging problems, including target scale changes, complete occlusion, and target deformation, as detailed below. Figure 5 , Figure 6 and Figure 7 As shown in the figure. The results demonstrate that the present invention can effectively address these property challenges.
[0095] In summary, this invention provides a single-target tracking method based on channel attention and spatiotemporal awareness. Its design principle is as follows: First, the template image and the search image are input together into the RepVGG backbone network for feature extraction. The outputs of the third and fifth layers of the backbone network are then fed into the Transformers of the upper and lower branches for feature fusion. The information from the two Transformers is merged and then fed into a classification and localization network to determine the target location. Finally, a threshold is used to update the template in real time. After updating the template, the temporal information of subsequent frames can be added to the network. Combined with the Transformer, spatiotemporal information can be used for target tracking. Due to the lightweight and concise inference process of RepVGG, this invention can be widely applied in scenarios such as video surveillance, autonomous driving, and human-computer interaction. It can also be deployed on large computing servers to provide users with high-precision target tracking services.
Claims
1. A single-target tracking method based on channel attention and spatiotemporal awareness, characterized in that, Includes the following steps: Step 1: Input the template image and the search region image, and extract the hierarchical features of the two images respectively through two RepVGG backbone networks with shared weights; Step 2: Perform feature fusion of the two feature maps using a fusion model based on channel attention mechanism, combining self-attention and cross-attention. Specifically, this includes the following steps: Step 21: Input sequence template features and perform feature enhancement through a self-attention module to fully associate the pixels of each part of the template features: in Representing the Multiple heads of attention output, Representing the first The query, key, and value change matrix of the head. Representing the first The query, key, and value matrix of each head is used to concatenate multiple multi-head attention results, followed by a linear transformation. Obtain the final output result; Step 22: Receive the search features as one part of the input, and receive the output from the template features as another part of the input. Calculate the cross-attention between the two parts through the cross-attention module. The calculation formula of the cross-attention module is the same as that of the self-attention module, only the input is different. Step 23: Input the results of the cross-attention module into the global max pooling layer and average pooling layer in both length and width dimensions. Then, feed the outputs of the two pooling layers into a parameter-shared MLP layer to learn the relationship between channels. The results are summed and then activated by the sigmoid function to obtain the channel attention weights. Finally, restore the feature map size to maintain size matching. The calculation expression of the channel attention module is as follows: in, This represents the sigmoid activation function, and AvgPool represents the average pooling operation. The output of average pooling is... MaxPool represents the max pooling operation, and the output is... express, These represent two convolutional layers in an MLP; Step 3: Complete the target tracking and prediction through classification branch, bounding box regression branch, and centrality regression branch; Step 4: Real-time template updates are completed by using threshold judgment, and the temporal information of subsequent frames is added to the fusion network to achieve complete target tracking; Step 5: Demonstrate tracking performance.
2. The single-target tracking method based on channel attention and spatiotemporal awareness as described in claim 1, characterized in that, In step 1, the hierarchical features of the template image and the search region image are extracted using two RepVGG backbone networks with shared weights. First, the template image and the search region image are fed into the RepVGG backbone network to extract features. Template features are output at the third layer, and search region features are output at the fifth layer to ensure they are at the same scale. Then, the two sets of outputs are processed... The convolution kernel adjusts the number of channels to 256.
3. The single-target tracking method based on channel attention and spatiotemporal awareness as described in claim 1, characterized in that, Step 3, which involves tracking and predicting the target through classification branch, bounding box regression branch, and centrality regression branch, specifically includes the following steps: Step 31: The prediction head maps the feature map after information fusion through different convolutional layers to three different spaces in the prediction head network: classification branch Cls, centrality branch Ctr, and regression branch Reg, thereby achieving accurate prediction. Step 32: Calculate the total target loss.
4. The single-target tracking method based on channel attention and spatiotemporal awareness as described in claim 3, characterized in that, In step 32, For classifying losses, For centrality loss, To regress the loss, For the sample size, The number of positive samples is denoted by x, and y represents the coordinates of the output feature map. and For positive and negative sample labels and predicted values, and For the centrality label and predicted value, and Let these represent the true bounding box position and the predicted bounding box position, respectively. The classification loss uses binary Focal-Loss, the centrality branch uses cross-entropy loss, and the regression branch uses GIoU-Loss. The total loss is calculated as follows: 。 5. The single-target tracking method based on channel attention and spatiotemporal awareness as described in claim 1, characterized in that, In step 4, the template is updated in real time by using a threshold, and the temporal information of subsequent frames is added to the fusion network to achieve complete target tracking. Specifically, this includes the following steps: Step 41: Introduce confidence level and confidence threshold When the current frame sequence number and update interval The judgment condition is met: That is, when confidence level Greater than the confidence threshold And the current image sequence number Divisible by the update interval If the condition is met, the template is updated; otherwise, it is not updated. The update expression is: The input features are all sequence features that satisfy the Transformer conditions. Assuming that the conditional update is satisfied, the template to be updated is the current frame feature. Features accumulated previously The result of direct addition, when there is no cumulative characteristic, is... The template features of the first frame ; Step 42: Serialize the updated template and add it to the fusion network to achieve complete target tracking.
6. The single-target tracking method based on channel attention and spatiotemporal awareness as described in claim 5, characterized in that, In step 41, for the confidence threshold Take 0.7, and update interval The tracker performs optimally when 120 frames are captured.
7. The tracking system of the single-target tracking method based on channel attention and spatiotemporal awareness as described in claim 1, characterized in that, include: The module includes a feature extraction module, a feature fusion module, a tracking and prediction module, and a performance display module. The feature extraction module extracts hierarchical features from the input template image and the search region image through two weight-shared RepVGG backbone networks. The feature fusion module performs self-attention and cross-attention feature fusion on the two feature maps through a fusion model based on channel attention mechanism. The tracking and prediction module completes the tracking and prediction of the target through classification branch, bounding box regression branch and centrality regression branch. The template is updated in real time through threshold judgment. The temporal information of subsequent frames is added to the fusion network to achieve complete target tracking. The performance display module displays the tracking performance.