Spatial-semantic-aware attention tracking method based on adaptive weight template updating

Abstract

The present invention discloses a spatial-semantic-aware attention tracking method based on adaptive weighted template updating, which belongs to the field of target tracking technology and is used to solve the problem that traditional classification-regression-based tracking algorithms are difficult to handle continuous changes in the target's appearance. The present invention first establishes a spatial-semantic-aware attention model, and uses a single convolutional attention network with a loss function to online identify the importance of different regions and channels on the target feature map to the target representation, and then strengthens the specific regions and semantic information on the feature map by adding corresponding weights in the spatial and channel dimensions; secondly, in order to better adapt to changes in the target's appearance, we propose a new template update strategy to adaptively adjust the contribution weight of the tracking result to the new template, further enhancing the reliability of the template. The present invention highlights the target features through the spatial-semantic-aware attention model while suppressing background information, thereby obtaining a more discriminative target appearance model and improving the robustness of the tracking results.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and in particular to a spatial-semantic perception attention tracking method based on adaptive weight template updating. Background Art

[0002] Visual tracking has gained widespread application in computer vision applications such as smart cities, autonomous driving, and video surveillance. In recent years, the performance of visual tracking has been further improved, from trackers based on artificial features to trackers based on deep learning. However, designing robust tracking algorithms remains a challenge in practical applications due to factors such as shape variations, complex backgrounds, and interference from similar targets.

[0003] In recent years, trackers based on Siamese networks have garnered widespread attention for their balance of accuracy and efficiency. However, their inability to adapt to continuous changes in the target's appearance leads to reduced tracking accuracy. To address this issue, some have improved tracking performance by adding a region proposal network (RPN) to the Siamese network. This method classifies and regresses preset bounding boxes, enabling better prediction of multi-scale targets. Others have introduced anchor-free mechanisms into the tracking field, addressing the complex parameter adjustment issues inherent in anchor-based methods and improving tracking efficiency. These algorithms have improved tracking results to a certain extent, and much work has focused on optimizing feature extraction networks or regression functions to enhance efficiency and accuracy. However, in actual tracking scenarios, optimizing only the feature extraction network or regression function still cannot address the accuracy deviation of the classification tracking head.

[0004] To address these issues, existing classification-regression-based trackers consider adding a branch to the classification branch to estimate positioning accuracy. Some have added a center-degree branch parallel to the classification branch to suppress excessive displacement and better determine the location of the target center. Others have proposed a center-prediction network, which uses an offset branch to appropriately correct the target position. Others have introduced a quality estimation score independent of classification to address the performance degradation associated with directly using classification confidence for bounding box selection. However, most of these methods still cannot address the low accuracy of the classification tracking head caused by the continuous changes in target appearance. Summary of the Invention

[0005] To address the above shortcomings, the present invention proposes a spatial-semantic perception attention tracking method based on adaptive weight template updating. This method optimizes the target appearance model through the perception attention strategy, improves the tracker's ability to distinguish target and background information, and alleviates the model degradation by updating the template. It can effectively solve the drift problem generated during the tracking process and improve tracking robustness.

[0006] The technical solutions of the present invention are as follows:

[0007] A spatial-semantic aware attention tracking method based on adaptive weight template updating comprises the following steps:

[0008] Step 1: Crop the template image based on the target position information in the initial frame image, and use the pre-trained ResNet-50 network to obtain the three-layer features of Covn3, Covn4 and Covn5 of the template image;

[0009] Step 2: Use the spatial-perceptual attention module to enhance the area with the largest target information in the template Covn3 layer feature map, while reducing the importance of irrelevant areas to obtain spatial-perceptual attention features; use the semantic-perceptual attention model to enhance the channel weights containing rich target semantic information in the Covn4 and Covn5 layer features to obtain semantic-perceptual attention features;

[0010] Step 3: Read the next frame and determine the search area based on the target position of the previous frame. Use the pre-trained ResNet-50 network to obtain the three-layer features of Covn3, Covn4, and Covn5 of the search area;

[0011] Step 4: Cross-correlate the search area features with the corresponding layers of the perceptual attention features obtained in step 2, and finally weight them to obtain a classification output for foreground and background classification; cross-correlate the search area features with the corresponding layers of the template image features obtained in step 1, and then weight them to obtain a regression output for prediction box prediction;

[0012] Step 5: The classification output is combined with the regression output to determine the target location;

[0013] Step 6. Crop the tracking result image according to the target position; and use the pre-trained ResNet-50 extraction network to extract the three-layer features of Covn3, Covn4 and Covn5 of the tracking result image.

[0014] Step 7: Calculate the Euclidean distance between the template image and the corresponding layer features of the tracking result, and compare them with the historical mean. When the update condition is met, execute step 8, otherwise return to step 3.

[0015] Step 8. The template image features and the tracking result features are cross-correlated and then compared with the autocorrelation of the template image features to generate an updated weight. The historical template and the tracking result are fused according to the updated weight to generate a new target template for tracking the next frame. At the same time, return to step 3 until the end of the video.

[0016] Furthermore, the method for acquiring the spatial perception attention features and the semantic perception attention features in step 2 is as follows:

[0017] Global maximum pooling and average pooling are applied to the template Covn3 layer features. A convolution filter is used to downsample the number of dual-pool feature channels to 1 to obtain a single-channel feature map. After broadcasting the single-channel convolution feature map through the sigmoid operation, the spatial attention feature map is obtained by multiplying it with the previous Covn3 layer features:

[0018]

[0019]

[0020] in Represents the spatial attention feature map,φ s (·) H×W×1 is the spatial attention weight, represents local matrix multiplication, Represents the template Covn3 layer features, GP max (·) represents the global maximum pooling operation, GP avg (·) represents the global average pooling operation, δ(·) represents the sigmoid function, It is a 3×3 convolution operation.

[0021] Global average pooling is applied to the Covn4 and Covn5 layer features of the template, and the pooled features are fully connected at two levels (excitation process). The obtained feature weights are multiplied by the previous Covn4 and Covn5 layer features to obtain the channel attention feature map:

[0022]

[0023]

[0024] in represents the channel attention feature map, φ c (·) 1×1×C represents the channel attention weight, represents local matrix multiplication, Represents the template Covn4, Covn5 layer features, GP avg (·) represents the average pooling operation, fc1(·) and fc2(·) represent the fully connected operation, ReLU(·) represents the activation function, and σ(·) represents the sigmoid function.

[0025] The obtained spatial attention feature map and channel attention feature map are convolved into one-dimensional features and regressed to Gaussian label mapping:

[0026]

[0027] where (i, j) is the offset to the target and σ is the kernel width.

[0028] The perceptual attention weight is obtained by minimizing the following objective function:

[0029]

[0030] in Representing spatial attention features and channel attention features represents the convolution operation, W is the regression weight, and λ is the regularization parameter to control overfitting.

[0031] The perceptual attention weights are multiplied by the previous Covn3, Covn4, and Covn5 layer features to obtain the spatial perceptual attention feature map and the semantic perceptual attention feature map;

[0032]

[0033]

[0034] in is the spatial perception attention feature map, φ s (·) H×W×1′ Attention weight for spatial perception;

[0035] in is the semantic-perceptual attention feature map, φ c (·) 1×1×C′ is the semantic-aware attention weight.

[0036] Furthermore, the classification output and regression output in step 4 are expressed as follows:

[0037]

[0038]

[0039] Among them, P cls Represents the classification diagram, P reg represents the regression plot, Represents the search area characteristics, Represents perceptual attention features, represents the template features obtained in step 1, and ★ represents the cross-correlation operation.

[0040] Furthermore, in step seven, the update condition can be described as:

[0041]

[0042] Indicates the current frame template features, represents the tracking result feature, ||·|| represents the calculation of Euclidean distance, represents the historical mean of the Euclidean distance, and λ is a hyperparameter.

[0043] Furthermore, in step eight, the method for generating the new template is:

[0044]

[0045]

[0046] in Represents the new template feature, Represents the history template feature, represents the tracking result feature, γ is the correlation coefficient, α is the weight factor, and ★ represents the cross-correlation operation.

[0047] The present invention can produce the following beneficial effects:

[0048] (1) This paper uses a perceptual-attention model to focus on the target appearance and suppress irrelevant information. The proposed spatial-perceptual attention is responsible for highlighting the location of the target; while the semantic-perceptual attention strengthens important feature channels online, improving the tracker's ability to distinguish target and background information.

[0049] (2) This paper designs a new adaptive weighted template update strategy. This strategy sets update conditions based on an evaluation mechanism based on Euclidean distance, selectively updates the template to prevent it from being contaminated, and adaptively assigns weights to the tracking results and the target template based on their relevance to generate a more reliable template. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more concisely and clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments.

[0051] Figure 1 This is a flow chart of the overall structure of the spatial-semantic perception attention tracking method based on adaptive weight template update of the present invention;

[0052] Figure 2 The impact of the adaptive weight template update strategy on template features and tracking performance is visualized;

[0053] Figure 3 The following is a comparison chart of the tracking accuracy and tracking success rate of the present invention and other algorithms on the OTB-100 dataset; DETAILED DESCRIPTION

[0054] The following will be combined with the accompanying drawings in the embodiments of the present invention to more intuitively describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] like Figure 1 As shown, a spatial-semantic awareness attention tracking method based on adaptive weight template updating includes the following steps:

[0056] Step 1: Crop the template image based on the target position information in the initial frame image, and use the pre-trained ResNet-50 network to obtain the three-layer features of Covn3, Covn4 and Covn5 of the template image;

[0057] Step 2: Use the spatial-perceptual attention module to enhance the area with the largest target information in the template Covn3 layer feature map, while reducing the importance of irrelevant areas to obtain new spatial-perceptual attention features; use the semantic-perceptual attention model to enhance the channel weights containing rich target semantic information in the Covn4 and Covn5 layer features to obtain semantic-perceptual attention features;

[0058] The method for obtaining the spatial perception attention feature and the semantic perception attention feature is:

[0059] Global maximum pooling and average pooling are applied to the template Covn3 layer features. A convolution filter is used to downsample the number of dual-pool feature channels to 1 to obtain a single-channel feature map. After broadcasting the single-channel convolution feature map through the sigmoid operation, the spatial attention feature map is obtained by multiplying it with the previous Covn3 layer features:

[0060]

[0061]

[0062] in Represents the spatial attention feature map,φ s (·) H×W×1 is the spatial attention weight, represents local matrix multiplication, Represents the template Covn3 layer features, GP max (·) represents the global maximum pooling operation, GP avg (·) represents the global average pooling operation, δ(·) represents the sigmoid function, It is a 3×3 convolution operation.

[0063] Global average pooling is applied to the Covn4 and Covn5 layer features of the template, and the pooled features are fully connected at two levels (excitation process). The obtained feature weights are multiplied by the previous Covn4 and Covn5 layer features to obtain the channel attention feature map:

[0064]

[0065]

[0066] in represents the channel attention feature map, φ c (·) 1×1×C represents the channel attention weight, represents local matrix multiplication, Represents the template Covn4, Covn5 layer features, GP avg (·) represents the average pooling operation, fc1(·) and fc2(·) represent the fully connected operation, ReLU(·) represents the activation function, and σ(·) represents the sigmoid function.

[0067] The obtained spatial attention feature map and channel attention feature map are convolved into one-dimensional features and regressed to Gaussian label mapping:

[0068]

[0069] where (i, j) is the offset to the target and σ is the kernel width.

[0070] The perceptual attention weight is obtained by minimizing the following objective function:

[0071]

[0072] in Representing spatial attention features and channel attention features represents the convolution operation, W is the regression weight, and λ is the regularization parameter to control overfitting.

[0073] The perceptual attention weights are multiplied by the previous Covn3, Covn4, and Covn5 layer features to obtain the spatial perceptual attention feature map and the semantic perceptual attention feature map;

[0074]

[0075]

[0076] in is the spatial perception attention feature map, φ s (·) H×W×1′Attention weight for spatial perception;

[0077] in is the semantic-perceptual attention feature map, φ c (·) 1×1×C′ is the semantic-aware attention weight.

[0078] Step 3: Read the next frame and determine the search area based on the target position of the previous frame. Use the pre-trained ResNet-50 network to obtain the three-layer features of Covn3, Covn4, and Covn5 of the search area;

[0079] Step 4: Cross-correlate the search area features obtained with the corresponding layers of the perceptual attention features obtained in step 2, and finally weight them to obtain a classification output for foreground and background classification; cross-correlate the search area features with the corresponding layers of the template image features obtained in step 1, and then weight them to obtain a return output for prediction box prediction;

[0080] The method for obtaining the classification output and regression output is:

[0081]

[0082]

[0083] Among them, P cls represents the classification output, P reg represents the regression output, Represents the search area characteristics, Represents perceptual attention features, represents the template features obtained in step 1, and ★ represents the cross-correlation operation.

[0084] Step 5: The classification output is combined with the regression output to determine the target location;

[0085] Step 6: Crop the tracking result image according to the target position; and use the pre-trained ResNet-50 extraction network to extract the three-layer features of Covn3, Covn4 and Covn5 of the tracking result image;

[0086] Step 7: Calculate the Euclidean distance between the template image and the corresponding layer features of the tracking result, and compare them with the historical mean. If the template update condition is met, execute step 8, otherwise return to step 3.

[0087] The update condition can be described as:

[0088]

[0089] Indicates the current frame template features, represents the tracking result feature, ||·|| represents the calculation of Euclidean distance, represents the historical mean of the Euclidean distance, and λ is a hyperparameter.

[0090] Step 8: Cross-correlate the template image features with the tracking result features and compare them with the autocorrelation of the template image features to generate updated weights. Based on the updated weights, fuse the historical template and the tracking result to generate a new target template for tracking the next frame. Simultaneously, return to step 3 until the video ends.

[0091] The method for generating the new template is:

[0092]

[0093]

[0094] in Represents the new template feature, Represents the history template feature, represents the tracking result feature, γ is the correlation coefficient, α is the weight factor, and ★ represents the cross-correlation operation.

[0095] The method proposed in this paper is implemented based on Pytorch. The experiments are carried out on a PC with 16.0GB memory, Intel(R) Core(TM) i7-10700 (2.90GHz) CPU and NVIDIA GeForce GTX 1660SUPER GPU.

[0096] like Figure 2 As shown in Figure 2, the impact of the adaptive weight template update strategy on the template features and tracking effect is visualized.

[0097] like Figure 3 As shown in the figure, the spatial-semantic-aware attention tracking method (Ours) based on adaptive weight template updating provided by the example of the present invention is compared with tracking algorithms such as (DaSiamRPN, DeepSRDCF, SiamRPN, SiamDW-FC, Staple, SiamFC) on the OTB-100 dataset. The results show that the algorithm provided by the present invention has better performance in success rate and accuracy.

[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A spatial-semantic awareness attention tracking method based on adaptive weight template updating, characterized by: The following steps are involved: Step 1: Crop the template image based on the target position information in the initial frame image, and use the pre-trained ResNet-50 network to obtain the three-layer features of Covn3, Covn4 and Covn5 of the template image; Step 2: Use the spatial perception attention module to enhance the area with the largest target information in the template Covn3 layer feature map, while reducing the importance of irrelevant areas to obtain a new spatial perception attention feature; The semantic-aware attention module is used to enhance the channel weights of the Covn4 and Covn5 layer features containing rich target semantic information to obtain semantic-aware attention features; Step 3: Read the next frame and determine the search area based on the target position of the previous frame. Use the pre-trained ResNet-50 network to obtain the three-layer features of Covn3, Covn4, and Covn5 of the search area; Step 4: Cross-correlate the search area features with the corresponding layers of the perceptual attention features obtained in step 2, and finally weight them to obtain a classification output for foreground and background classification; cross-correlate the search area features with the corresponding layers of the template image features obtained in step 1, and then weight them to obtain a regression output for prediction box prediction; Step 5: Combine the classification output with the regression output to determine the target location; Step 6: Crop the tracking result image according to the target position, and use the pre-trained ResNet-50 extraction network to extract the three-layer features of Covn3, Covn4 and Covn5 of the tracking result image; Step 7: Calculate the Euclidean distance between the template image and the corresponding layer features of the tracking result, and compare them with the historical mean. If the update condition is met, execute step 8, otherwise return to step 3. Step 8. The template image features and the tracking result features are cross-correlated and then compared with the autocorrelation of the template image features to generate an updated weight. The historical template and the tracking result are fused according to the updated weight to generate a new target template for tracking the next frame. At the same time, return to step 3 until the end of the video.

2. The spatial-semantic awareness attention tracking method based on adaptive weight template updating according to claim 1 is characterized in that: In the step 2, the process of acquiring the spatial perception attention features and the semantic perception attention features is as follows: Global maximum pooling and average pooling are applied to the template Covn3 layer features. A convolution filter is used to downsample the number of dual-pool feature channels to 1 to obtain a single-channel feature map. After broadcasting the single-channel convolution feature map through the sigmoid operation, the spatial attention feature map is obtained by multiplying it with the previous Covn3 layer features: in Represents the spatial attention feature map,φ s (·) H×W×1 is the spatial attention weight, represents local matrix multiplication, Represents the template Covn3 layer features, GP max (·) represents the global maximum pooling operation, GP avg (·) represents the global average pooling operation, δ(·) represents the sigmoid function, It is a 3×3 convolution operation; Global average pooling is applied to the Covn4 and Covn5 layer features of the template, and the pooled features are fully connected at two levels (excitation process). The obtained feature weights are multiplied by the previous Covn4 and Covn5 layer features to obtain the channel attention feature map: in represents the channel attention feature map, φ c (·) 1×1×C represents the channel attention weight, represents local matrix multiplication, Represents the template Covn4, Covn5 layer features, GP avg (·) represents the average pooling operation, fc1(·) and fc2(·) represent the fully connected operation, ReLU(·) represents the activation function, and σ(·) represents the sigmoid function; The obtained spatial attention feature map and channel attention feature map are convolved into one-dimensional features and regressed to Gaussian label mapping: Where (i, j) is the offset to the target and σ is the kernel width; The perceptual attention weight is obtained by minimizing the following objective function: in Representing spatial attention features and channel attention features represents the convolution operation, W is the regression weight, and λ is the regularization parameter to control overfitting; The perceptual attention weights are multiplied by the previous Covn3, Covn4, and Covn5 layer features to obtain the spatial perceptual attention feature map and the semantic perceptual attention feature map; in is the spatial perception attention feature map, φ s (·) H×W×1′ Attention weight for spatial perception; in is the semantic-perceptual attention feature map, φ c (·) 1×1×C′ is the semantic-aware attention weight.

3. The spatial-semantic awareness attention tracking method based on adaptive weight template updating according to claim 1 is characterized in that: In step 4, the classification output and regression output are expressed as follows: Among them, P cls represents the classification output, P reg represents the regression output, Represents the search area characteristics, represents the perceptual attention features obtained in step 2, represents the template features obtained in step 1, and ★ represents the cross-correlation operation.

4. The spatial-semantic awareness attention tracking method based on adaptive weight template updating according to claim 1 is characterized in that In step seven, the update condition can be described as: Indicates the current frame template features, represents the tracking result feature, ||·|| represents the calculation of Euclidean distance, represents the historical mean of the Euclidean distance, and λ is a hyperparameter.

5. The spatial-semantic awareness attention tracking method based on adaptive weight template updating according to claim 1 is characterized in that: In step eight, the new template is generated by: in Represents the new template feature, Represents the history template feature, represents the tracking result feature, γ is the correlation coefficient, α is the weight factor, and ★ represents the cross-correlation operation.

Images