A target tracking method combining efficient self-attention and pixel cross-correlation in a complex scene

Abstract

The application discloses a target tracking method combining efficient self-attention and pixel cross-correlation in a complex scene, and is based on SiamCAR as a basic algorithm.The features of a basic template image and a search image are introduced into a template-guided double-dimensional attention feature enhancement module in the algorithm through a feature extraction stage of a backbone network ResNet-50, a spatial and channel attention mechanism is combined, and cross-branch channel interaction is combined, so that the semantic consistency of a target is improved, and similar background interference is inhibited; then, a dynamic template modeling strategy is used to generate dynamic template features by adaptively fusing the correlation feedback between multiple frames of templates and search regions, so as to improve the stability of the template; then, a pixel-level deformable cross-correlation mechanism is introduced to combine cross-level feature fusion and a step-by-step correlation refining strategy, so as to improve the target positioning and matching precision.

Description

Technical Field

[0001] This invention relates to the field of target tracking in complex scenes, and in particular to a target tracking method that combines efficient self-attention and pixel cross-correlation in complex scenes. Background Technology

[0002] With the development of autonomous driving, surveillance systems, drones, and intelligent security, target tracking technology has been widely applied. Traditional target tracking methods typically rely on the color, texture, or edge features of an image to locate the target, but these methods exhibit low robustness in complex backgrounds and dynamic environments. For example, when the target is occluded, moves rapidly, or the lighting changes, existing technologies may fail to accurately track the target, leading to tracking failures or misjudgments.

[0003] Early target tracking methods, such as correlation filtering algorithms, treated tracking as a discrimination problem, quickly learning a filter in the frequency domain to distinguish the target from the background. Their core advantage lay in their extremely high computational efficiency, enabling high-speed real-time tracking on a central processing unit. However, traditional methods are limited by their heavy reliance on hand-designed features, such as histograms of oriented gradients (HBPs). Some deep learning target tracking methods, such as convolutional neural networks (CNNs) and long short-term memory (LSTM) methods, typically rely on large amounts of training data and feature learning. These methods lack the ability to represent changes in target appearance in complex scenes, resulting in poor robustness to sudden changes in lighting, severe occlusion, and target deformation. Especially when the boundary between the target and background is blurred, or when there is no significant visual difference between the target and background, deep learning methods may fail to effectively distinguish between them. Furthermore, deep learning methods in target tracking, especially in real-time tasks, often require massive computational resources. Particularly when processing high-resolution images or video streams, traditional deep learning models may cause tracking delays, failing to meet real-time requirements. Therefore, a target tracking method is needed that can effectively distinguish between the target and the background in complex scenarios, especially when the boundary between the target and the background is blurred. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a target tracking method that combines efficient self-attention and pixel cross-correlation in complex scenarios.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] A target tracking method combining efficient self-attention and pixel cross-correlation in complex scenarios is characterized by the following steps:

[0007] S1: Extract the template image and search image, input them into the ResNet-50 feature extraction network, and obtain dual-branch features;

[0008] S2: The dual-branch features are introduced into the dual-dimensional attention feature enhancement module, and through the cross-branch information interaction mechanism, the dual-branch features are enhanced in the channel and spatial dimensions to output a new dual-branch enhanced feature representation.

[0009] S3: Perform temporal modeling and weighted fusion of the template content features in the dual-branch enhanced features output in step S2 to generate dynamic template features;

[0010] S4: Input the search content features in the dual-branch enhanced features output in step S2 and the dynamic template features generated in step S3 into the hierarchical pixel-aware correlation refining module to refine and fuse cross-level features and obtain unified features.

[0011] S5: Input the fused unified features into the prediction head of the fully convolutional network architecture to generate the target's precise location, category, and corresponding confidence information;

[0012] S6: Extract 12 feature attributes from the output of step S5 for comparison and evaluation.

[0013] Furthermore, in step S1, the template image is cropped from the truth box given in the first frame of the video sequence, and the search image is a region of 2-4 times the size of the target in the current frame, with the predicted target center as the base point.

[0014] Furthermore, the dual-branch feature includes template content features in the template image and search content features in the search image. The template content features and search content features are extracted from the ResNet-50 backbone network and a feature decoupling mechanism is introduced. The deep semantic features are divided into content features and style features. In the subsequent target tracking process, only the content features are retained, that is, the dual-branch feature of template content features and search content features.

[0015] Furthermore, the dual-dimensional attention feature enhancement module in step S2 includes a spatial self-attention mechanism and a channel self-attention mechanism to perform dual-dimensional modeling of the dual-branch features.

[0016] Furthermore, the principle of spatial self-attention mechanism is as follows: input features Query features are generated by linear projection using 1×1 convolution. Bond features ,in

[0017] ,and ,

[0018] in The number of channels in the feature map. This represents the channel dimension after channel compression. and The height and width of the feature map; and Compression is performed along the channel dimension to obtain... ,in Through matrix multiplication and column direction Operations to generate spatial self-attention maps Thus, the weights of spatial self-attention are obtained, as shown in the following formula:

[0019] ,

[0020] Here, Q and K are obtained from the input feature map through a 1×1 convolutional linear transformation, and are used to measure the correlation between pixels by normalizing column-wise. Ensure that the sum of the attention weights for each pixel is 1; the final output feature formula is as follows:

[0021] ,

[0022] in It is a scalar parameter. The feature content is derived from the input features. The result is obtained after 1×1 convolution and flattening. This represents a weighted summation of all features using attention weights; The final output feature is obtained by connecting the weighted feature with the residual of the original feature.

[0023] Furthermore, the principle of channel self-attention mechanism is similar to that of spatial self-attention mechanism: for input features Apply two independent Convolution operation to generate query features Bond features ,in ,and ;Will and In spatial dimension Compress the above to obtain And through matrix multiplication and row direction The operation generates a channel self-attention map. As shown below:

[0024] ,

[0025] ,

[0026] in It is a scalar parameter, and the output will be restored to its original size. .

[0027] Furthermore, in step S3, the temporal modeling and weighted fusion to generate dynamic template features involves first taking the enhanced template content features from the dual-branch enhanced features obtained in step S2, and then storing the enhanced template content features in a temporal memory queue. For N consecutive frames of template features... Calculate the corresponding weight coefficient based on the matching response strength between it and the current search feature. Dynamic template features are generated through weighted fusion, with weights... The peak response of historical templates and current search features is determined by the following formula:

[0028] .

[0029] Furthermore, step S4 refines and fuses cross-level features in the hierarchical pixel-aware correlation refinement module, namely cross-correlation matching and feature fusion, as follows: First, pixel-level cross-correlation is performed to expand the dynamic template features into H×W small kernels in the spatial dimension. Each kernel is cross-correlated with the search content features to obtain the initial correlation response feature map. It is the first-stage output of the hierarchical pixel-aware correlation refining module, and its calculation formula is:

[0030] ,

[0031] in, For search features , Template features H and W are the template space dimensions, and m is the template pixel index. During pixel-level cross-correlation, the spatial offset Δp is predicted based on the dynamic template features, and the search features are bilinearly resampled based on this offset.

[0032] ,

[0033] This enables geometrically perceptive deformable pixel-level cross-correlation matching; the obtained response map is then concatenated with the original search features to form a new fused feature tensor. The aggregated new features are then deeply cross-correlated with the template content features. Cross-correlation is performed on the template content feature *x* and the search content feature *y* within the same channel, and then summed along the channel dimension to obtain the final matching response map. The cross-correlation is calculated and summed channel by channel, as shown in the following formula:

[0034] ,

[0035] in and They represent the first Feature mapping of each channel Represents cross-correlation operations. This represents the total number of channels.

[0036] Furthermore, the prediction head of the fully convolutional network architecture in step S5 includes a regression branch, a classification branch, and a centrality branch.

[0037] The present invention has the following beneficial effects:

[0038] This invention is based on the SiamCAR algorithm and introduces a template-guided two-dimensional attention feature enhancement module. By jointly constructing spatial self-attention and channel self-attention mechanisms, compared with existing deep learning tracking methods that rely only on local convolution operations and are difficult to capture long-distance dependencies, this invention can significantly improve the discriminativeness and stability of target features in scenarios such as occlusion, lighting changes, and complex backgrounds, thereby improving the overall tracking robustness. In the feature extraction stage, a feature decoupling mechanism is introduced to separate deep features into content feature components and style feature components, and only content features are used in the subsequent tracking process, thereby effectively reducing the impact of different scene conditions (such as lighting, weather, and imaging style differences) on feature distribution. Compared to traditional domain adaptation methods that require the introduction of adversarial discriminators or target domain labeled data, this invention improves the model's generalization ability across scenes and datasets without additional labeling by implementing implicit domain-invariant modeling at the feature level. The hierarchical pixel-aware correlation refinement module combines pixel-level cross-correlation with deep cross-correlation to construct a cross-level correlation modeling and progressive refinement strategy. This achieves precise modeling of the fine-grained spatial correspondence between the target and the search region during the feature matching stage. Compared to existing methods that rely solely on high-level semantic feature similarity or coarse-grained convolutional matching, this module fully utilizes the target's local structural information and high-level semantic information, maintaining stable and accurate target localization capabilities even under rapid motion, scale changes, and complex backgrounds. It also improves the geometric alignment accuracy between the template and the search region. Furthermore, temporal modeling and weighted fusion of template-enhanced features to generate dynamic template features avoid the degradation of static templates over time. This mechanism tightly couples the template update process with the current search matching feedback, significantly improving stability and reliability during long-term tracking while ensuring computational efficiency. Attached Figure Description

[0039] Figure 1 This is the overall network structure diagram of the present invention;

[0040] Figure 2This is a framework diagram of the template-guided two-dimensional attention feature enhancement module of the present invention;

[0041] Figure 3 This is a framework diagram of the hierarchical pixel perception correlation refining module of the present invention;

[0042] Figure 4 This is a success rate graph of different algorithms of this invention on 12 attributes of the UAV20L dataset;

[0043] The 12 attributes correspond to the following: Figure 4 (a) Aspect Ratio Change Figure 4 (b) Background Clutter Figure 4 (c) Camera Motion Figure 4 (d) FastMotion Figure 4 (e) Full Occlusion Figure 4 (f) Out of View Figure 4 (g) Low resolution Figure 4 (h) Partial Occlusion Figure 4 (i) Illumination Variation Figure 4 (j) Scale Variation Figure 4 (k) Viewpoint Change Figure 4 (l) Similar Object; The values ​​in [] in the figure represent the AUC value under the attribute, which is the area under the curve; Ours, or SiamDC, is the name used in the algorithm of this invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0045] This invention provides a target tracking method that combines efficient self-attention and pixel cross-correlation in complex scenes, comprising the following steps:

[0046] S1: Extract the template image and search image, input them into the ResNet-50 feature extraction network, and obtain dual-branch features;

[0047] S2: The dual-branch features are introduced into the dual-dimensional attention feature enhancement module, and through the cross-branch information interaction mechanism, the dual-branch features are enhanced in the channel and spatial dimensions to output a new dual-branch enhanced feature representation.

[0048] S3: Perform temporal modeling and weighted fusion of the template content features in the dual-branch enhanced features output in step S2 to generate dynamic template features;

[0049] S4: Input the search content features in the dual-branch enhanced features output in step S2 and the dynamic template features generated in step S3 into the hierarchical pixel-aware correlation refining module to refine and fuse cross-level features and obtain unified features.

[0050] S5: Input the fused unified features into the prediction head of the fully convolutional network architecture to generate the target's precise location, category, and corresponding confidence information;

[0051] S6: Extract 12 feature attributes from the output of step S5 for comparison and evaluation.

[0052] The detailed implementation methods for each step are as follows:

[0053] First, the hardware and software environment was set up. The SiamCAR framework was used as the basic framework on Ubuntu 18.04. The system was accelerated by GPU using PyTorch 1.8.1 and CUDA 11.1. The UAV20L dataset was used, which contains multiple long video sequences. From the first frame of these video sequences, template images were cropped according to the given ground truth to initialize the appearance representation of the target. In the subsequent tracking process, the features of the most recent frames were stored using a temporal memory queue, and dynamic template images were generated through correlation feedback to cope with appearance degradation. Simultaneously, using the currently predicted target center as the base point, a region approximately 2 to 4 times the target size is cropped from the current frame as the search region. This region includes the possible displacement range of the target and its adjacent background. The obtained template image and search region are input into the ResNet-50 feature extraction network to obtain dual-branch features, namely template features and search region features. Then, through the feature decoupling mechanism embedded in its multi-layer residual structure (Conv1_x to Conv5_x), the obtained template features and search region features are decoupled, and the template style features and search style features are removed, while the template content features and search content features are retained.

[0054] like Figure 2The template content features and search content features obtained from the branch features are input into the template-guided two-dimensional attention feature enhancement module. The two-dimensional attention feature enhancement module includes spatial self-attention feature enhancement and channel self-attention feature enhancement; the spatial self-attention feature enhancement operation is to input the features... Query features are generated by linear projection (channel dimensionality reduction or channel decomposition) using 1×1 convolution. Bond features ,in

[0055] ,and ,

[0056] in The number of channels in the representative feature map is 256. This represents the channel dimension after channel compression. and With both the height and width of the feature map set to 31, the number of channels is typically compressed to the original value. This reduces computational cost while maintaining sufficient feature diversity to capture long-range spatial dependencies; and Compression is performed along the channel dimension to obtain... ,in Through matrix multiplication and column direction Operations to generate spatial self-attention maps Thus, the weights of spatial self-attention are obtained, as shown in the following formula:

[0057] ,

[0058] Here, Q and K are obtained from the input feature map through a 1×1 convolutional linear transformation, and are used to measure the correlation between pixels by normalizing column-wise. Ensure that the sum of the attention weights for each pixel is 1; the final output feature formula is as follows:

[0059] ,

[0060] in It is a scalar parameter; setting it to 0 ensures that the model's behavior is consistent with the original SiamCAR during the initial training phase. The feature content is derived from the input features. The result is obtained after 1×1 convolution and flattening. This represents a weighted summation of all features using attention weights; The final output feature is obtained by concatenating the weighted feature with the residual of the original feature; the channel self-attention feature enhancement logic is similar to the spatial self-attention feature enhancement logic, and it applies the same logic to the input features. Apply two independent Convolution operation to generate query features Bond features ,in ,and ;Will and In spatial dimension Compress the above to obtain And through matrix multiplication and row direction The operation generates a channel self-attention map. As shown below:

[0061] ,

[0062] ,

[0063] in It is a scalar parameter, and the output will be restored to its original size. To further enhance the information complementarity between the target and the search region, a method similar to computational channel self-attention was used. Taking search features as an example, template features... Remodeling and deformation are performed to obtain ,in The formula for calculating the cross-attention feature map from the target branch is as follows:

[0064] ,

[0065] Encode the cross-attention feature map obtained from the target branch into the search feature. This yields a new feature map, calculated using the following formula:

[0066] ,

[0067] in The learnable coefficient.

[0068] Temporal modeling and weighted fusion of the template content features enhanced by two-dimensional attention are performed to generate dynamic template features. The operation is as follows: The template content features enhanced by two-dimensional attention are stored in a temporal memory queue. For N consecutive frames of template features... Calculate the corresponding weight coefficient based on the matching response strength between it and the current search feature. .

[0069] like Figure 3 The dynamic template features shown Features of search content The input is fed into the hierarchical pixel-aware correlation refining module for cross-correlation matching and feature fusion. First, a pixel-level cross-correlation operation is performed: the template content features are expanded into H×W small kernels in the spatial dimension. Each kernel is cross-correlated with the search content features to obtain the initial correlation response feature map. It is the first-stage output of the Hierarchical Pixel-Aware Relevance Refinement (HPCR) module, and its calculation formula is as follows:

[0070] ,

[0071] in, For search content features , Template content features H and W are the template space dimensions, and m is the template pixel index, with a value ranging from 1 to 49. During pixel-level cross-correlation, the spatial offset Δp is predicted based on the dynamic template features, and bilinear resampling is performed on the search content features based on this offset.

[0072] ,

[0073] This achieves geometrically perceptive deformable pixel-level cross-correlation matching. Subsequently, the aggregated new features are deeply cross-correlated with the template content features. Cross-correlation is performed on the template feature *x* and the search feature *y* within the same channel, and then summed along the channel dimension to obtain the final matching response map. The cross-correlation is calculated channel-by-channel and summed, as shown in the following formula:

[0074] ,

[0075] in, and They represent the first Feature mapping of each channel Represents cross-correlation operations. This represents the total number of channels.

[0076] The fused unified features are further input into a prediction head based on a fully convolutional network (FCN) architecture, including a regression branch, a classification branch, and a centrality branch. The classification branch extracts the spatial response through convolution operations to accurately determine the probability of each pixel belonging to the target in order to eliminate background interference. The regression branch simultaneously predicts four offset distances from the pixel to the target boundary, using the nonlinear fitting capability of the convolutional neural network to flexibly capture the target's scale changes and non-rigid deformations. The centrality branch provides a quality assessment by measuring the degree of offset between the predicted position and the target center, effectively suppressing low-quality prediction boxes that are far from the center.

[0077] like Figure 4The image shows a comparison of 12 attribute features of the same dataset under various algorithms, output using the above method. It can be observed that the algorithm of this invention (labeled Ours, i.e., SiamDC) differs from 12 traditional mainstream algorithms—SiamBAN, SiamAPN++, SiamMask, SiamAPN, SiamRPN, SiamRPN++, SiamCAR, SiamFC++, TCTrack, DaSiamRPN, UpdateNet, SiamDW, and Ocean—in 9 attributes. Figure 4 (a) Aspect Ratio Change Figure 4 (b) Background Clutter Figure 4 (c) Camera Motion Figure 4 (d) Fast Motion Figure 4 (f) Out of View Figure 4 (g) Low resolution Figure 4 (h) Partial Occlusion Figure 4 (j) Scale Variation Figure 4 (k) The success rate curves for viewpoint change are all above the comparison algorithm, which quantitatively proves the synergistic gains of the innovative modules described in the embodiment: the leading advantage in cluttered backgrounds and camera motion, fast motion, low resolution, and beyond the field of view intuitively reflects the template-guided two-dimensional attention module's ability to enhance the semantic consistency of the target and suppress background noise; while the performance improvement in scale change and viewpoint change confirms the fine-grained accurate matching effect achieved by the hierarchical pixel perception correlation refinement module through the geometric perception deformable cross-correlation mechanism; at the same time, the stability in partial occlusion reflects the effective solution of the dynamic template modeling strategy to the template degradation problem, thus comprehensively verifying that the present invention has higher tracking accuracy and robustness in complex environments.

[0078] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features.

Claims

1. A target tracking method combining efficient self-attention and pixel cross-correlation in complex scenes, characterized in that, Includes the following steps: S1: Extract the template image and search image, input them into the ResNet-50 feature extraction network, and obtain dual-branch features; S2: The dual-branch features are introduced into the dual-dimensional attention feature enhancement module, and through the cross-branch information interaction mechanism, the dual-branch features are enhanced in the channel and spatial dimensions to output a new dual-branch enhanced feature representation. S3: Perform temporal modeling and weighted fusion of the template content features in the dual-branch enhanced features output in step S2 to generate dynamic template features; S4: Input the search content features in the dual-branch enhanced features output in step S2 and the dynamic template features generated in step S3 into the hierarchical pixel-aware correlation refining module to refine and fuse cross-level features and obtain unified features. S5: Input the fused unified features into the prediction head of the fully convolutional network architecture to generate the target's precise location, category, and corresponding confidence information; S6: Extract 12 feature attributes from the output of step S5 for comparison and evaluation.

2. The target tracking method according to claim 1, characterized in that: The template image in step S1 is cropped from the truth box given in the first frame of the video sequence. The search image is a region 2-4 times the size of the target, cropped from the current frame with the center of the predicted target as the base point.

3. The target tracking method according to claim 2, characterized in that: The dual-branch feature includes template content features in the template image and search content features in the search image. The template content features and search content features are extracted from the ResNet-50 backbone network and a feature decoupling mechanism is introduced. The deep semantic features are divided into content features and style features. In the subsequent target tracking process, only the content features are retained, that is, the dual-branch features of template content features and search content features.

4. The target tracking method according to claim 1, characterized in that: The dual-dimensional attention feature enhancement module in step S2 includes a spatial self-attention mechanism and a channel self-attention mechanism, which perform dual-dimensional modeling of the dual-branch features.

5. The target tracking method according to claim 4, characterized in that: The spatial self-attention mechanism works as follows: Input features... Query features are generated by linear projection using 1×1 convolution. Bond features ,in ,and , in The number of channels in the feature map. This represents the channel dimension after channel compression. and The height and width of the feature map; and Compression is performed along the channel dimension to obtain... ,in Through matrix multiplication and column direction Operations to generate spatial self-attention maps Thus, the weights of spatial self-attention are obtained, as shown in the following formula: ， Here, Q and K are obtained from the input feature map through a 1×1 convolutional linear transformation, and are used to measure the correlation between pixels by normalizing column-wise. Ensure that the sum of the attention weights for each pixel is 1; the final output feature formula is as follows: ， in It is a scalar parameter. The feature content is derived from the input features. The result is obtained after 1×1 convolution and flattening. This represents a weighted summation of all features using attention weights; The final output feature is obtained by connecting the weighted feature with the residual of the original feature.

6. The target tracking method according to claim 4, characterized in that: The principle of the channel self-attention mechanism is similar to that of the spatial self-attention mechanism: for input features... Apply two independent Convolution operation to generate query features Bond features ,in ,and ;Will and In spatial dimension Compress the above to obtain And through matrix multiplication and row direction The operation generates a channel self-attention map. As shown below: ， ， in It is a scalar parameter, and the output will be restored to its original size. .

7. The target tracking method according to claim 1, characterized in that: In step S3, temporal modeling and weighted fusion to generate dynamic template features involves first obtaining the enhanced template content features from the dual-branch enhanced features acquired in step S2, and then storing the enhanced template content features in a temporal memory queue. For N consecutive frames of template features... Calculate the corresponding weight coefficient based on the matching response strength between it and the current search feature. Dynamic template features are generated through weighted fusion, with weights... The peak response of historical templates and current search features is determined by the following formula: 。 8. The target tracking method according to claim 1, characterized in that: Step S4 involves refining and fusing cross-level features, namely cross-correlation matching and feature fusion, in the hierarchical pixel-aware correlation refining module. The operation is as follows: First, a pixel-level cross-correlation operation is performed to expand the dynamic template features into H×W small kernels in the spatial dimension. Each kernel is cross-correlated with the search content features to obtain the initial correlation response feature map. It is the first-stage output of the Hierarchical Pixel-Aware Relevance Refinement (HPCR) module, and its calculation formula is as follows: ， in, For search features , Template features H and W are the template space dimensions, and m is the template pixel index. During pixel-level cross-correlation, the spatial offset Δp is predicted based on the dynamic template features, and the search features are bilinearly resampled based on this offset. ， This enables geometrically perceptive deformable pixel-level cross-correlation matching; the obtained response map is then concatenated with the original search features to form a new fused feature tensor. The aggregated new features are then deeply cross-correlated with the template content features. Cross-correlation is performed on the template content feature *x* and the search content feature *y* within the same channel, and then summed along the channel dimension to obtain the final matching response map. The cross-correlation is calculated and summed channel by channel, as shown in the following formula: ， in and They represent the first Feature mapping of each channel Represents cross-correlation operations. This represents the total number of channels.

9. The target tracking method according to claim 1, characterized in that: The prediction head of the fully convolutional network architecture in step S5 includes a regression branch, a classification branch, and a centrality branch.

Images