A transformer target tracking method and device based on multi-scale feature compression representation

Abstract

The application belongs to the technical field of visual target tracking, and discloses a Transformer target tracking method and device based on multi-scale feature compression representation, which comprises the following steps: obtaining multi-scale features of a template image and a search area image, template tokens and search area tokens respectively; performing similarity calculation on each spatial unit feature by using a learnable query vector, performing weighted summation on the spatial unit features according to the similarity calculation results, and obtaining prototype tokens; inputting the prototype tokens, the template tokens and the search area tokens after splicing into a Transformer encoder for feature interaction, and predicting a target position according to the feature interaction results; the application solves the problems of high calculation complexity and memory consumption and the decline of discrimination ability in a complex scene in the prior art, reduces sequence redundancy caused by the introduction of multi-scale features, and improves the target discrimination ability in a complex scene.

Description

Technical Field

[0001] This invention belongs to the field of visual target tracking technology, specifically relating to a Transformer target tracking method and apparatus based on multi-scale feature compression representation. Background Technology

[0002] Video object tracking is a crucial research area in computer vision. Its goal is to continuously and stably predict the spatial location of a target in subsequent frames, given the target's position in the first frame of a video sequence. This technology has wide-ranging practical applications in intelligent surveillance, unmanned systems, human-computer interaction, autonomous driving, and video analytics.

[0003] With the development of deep learning technology, Siamese network structures based on convolutional neural networks have made significant progress in target tracking tasks. However, these methods usually rely on local convolution operators, making it difficult to model the long-range dependency between the template and the search region on a global scale. When the target undergoes significant deformation, scale changes, occlusion, or the presence of similar interfering targets, localization drift or tracking failure is likely to occur.

[0004] In recent years, the Transformer architecture has been gradually introduced into the field of video object tracking due to its global modeling capabilities based on the self-attention mechanism. Existing Transformer-based object tracking methods typically enhance the feature interaction capabilities between the template image and the search region image by dividing the template image and the search region image into a large number of visual tokens and inputting them into the Transformer network for joint modeling. However, when introducing multi-scale features, this method often requires a significant increase in the number of tokens, leading to a sharp increase in computational complexity and memory overhead. At the same time, background and interference information mixed in with multi-scale features are also introduced into deep interactions, reducing its discrimination ability in complex scenes such as target occlusion and similar backgrounds. Summary of the Invention

[0005] The purpose of this invention is to provide a Transformer target tracking method and apparatus based on multi-scale feature compression representation, which divides multi-scale features into spatial unit features and aggregates a large number of spatial unit features into a small number of prototype tokens, thereby reducing computational and memory overhead while improving discriminative power.

[0006] This invention adopts the following technical solution: a Transformer target tracking method based on multi-scale feature compression representation, comprising: The multi-scale features of the template image and the search region image are obtained respectively, as well as the template token corresponding to the template image and the search region token corresponding to the search region image; each scale feature includes multiple spatial units, and each spatial unit has a corresponding spatial unit feature; Similarity is calculated using a preset number of learnable query vectors and spatial unit features. The spatial unit features are then weighted and summed based on the similarity calculation results to obtain a preset number of prototype tokens. The prototype token, template token, and search area token are concatenated to form the input sequence; The input sequence is fed into the Transformer encoder for feature interaction, and the location of the target in the search region image is predicted based on the result of the feature interaction.

[0007] Another technical solution of the present invention: a Transformer target tracking device based on multi-scale feature compression representation, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the above-mentioned Transformer target tracking method based on multi-scale feature compression representation.

[0008] The beneficial effects of this invention are as follows: This invention acquires multi-scale features with different spatial resolutions, and calculates the similarity between a preset number of learnable query vectors and spatial unit features of each scale feature. Based on the similarity calculation results, it performs a weighted summation of each spatial unit feature, adaptively filters discriminative features, and suppresses background interference to obtain a preset number of prototype tokens. This compresses a large number of spatial unit features into a very small number of discriminative prototype tokens. In the prior art, the computational complexity of the Transformer self-attention mechanism is proportional to the square of the sequence length. However, this invention changes the way multi-scale features are introduced from directly splicing long sequences to injecting compressed and discriminative prototype tokens. This reduces computational and memory overhead while improving discriminative power, effectively enhancing robustness in complex scenarios such as target occlusion, deformation, scale changes, and background interference. Attached Figure Description

[0009] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a flowchart of the feature reweighting method in this invention; Figure 3 This is a flowchart of the feature compression method in this invention; Figure 4 This is a flowchart of the method for concatenating prototype tokens in this invention; Figure 5A radar chart showing the success rate under various complex scenarios on the LaSOT dataset; Figure 6 This is a schematic diagram of the success rate curve for scenarios with varying target aspect ratios on the LaSOT dataset. Figure 7 This is a schematic diagram of the success rate curve in a cluttered background scenario on the LaSOT dataset. Figure 8 This is a schematic diagram of the success rate curve for camera motion scenes on the LaSOT dataset. Figure 9 This is a schematic diagram of the success rate curve for target deformation scenarios on the LaSOT dataset. Figure 10 This is a schematic diagram of the success rate curve in a fast-moving target scenario on the LaSOT dataset. Figure 11 This is a schematic diagram of the success rate curve in a scenario where the target is completely occluded on the LaSOT dataset. Figure 12 This is a schematic diagram of the success rate curve under varying lighting conditions on the LaSOT dataset. Figure 13 This is a schematic diagram of the success rate curve for low-resolution scenes on the LaSOT dataset. Figure 14 This is a schematic diagram of the success rate curve for a target motion blur scene on the LaSOT dataset. Figure 15 This is a schematic diagram of the success rate curve for a target rotation scenario on the LaSOT dataset. Figure 16 This is a schematic diagram of the success rate curve in a target out-of-view scenario on the LaSOT dataset. Figure 17 This is a schematic diagram of the success rate curve in a scenario where the target is partially occluded on the LaSOT dataset. Figure 18 This is a schematic diagram of the success rate curve for a target scale variation scenario on the LaSOT dataset. Figure 19 This is a schematic diagram of the success rate curve for scenarios with changing viewpoints on the LaSOT dataset. Detailed Implementation

[0010] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0011] In one embodiment, a Transformer target tracking method based on multi-scale feature compression representation is provided, comprising: acquiring multi-scale features of a template image and multi-scale features of a search region image, as well as a template token corresponding to the template image and a search region token corresponding to the search region image; wherein each scale feature includes multiple spatial units, and each spatial unit has corresponding spatial unit features; performing feature reweighting and feature compression on the spatial unit features in sequence to obtain a preset number of prototype tokens; concatenating the prototype tokens, template tokens, and search region tokens to form an input sequence; inputting the input sequence into a Transformer encoder for feature interaction, and predicting the position of the target in the search region image based on the result of the feature interaction.

[0012] The multi-scale features include first-scale features and second-scale features, and the prototype token includes first-scale prototype token and second-scale prototype token.

[0013] Specifically, the overall process of this invention is as follows: Figure 1 As shown: First, the template image (128×128 pixels) and the search region image (256×256 pixels) are input into a Hierarchical Vision Transformer (HiViT) backbone network with shared parameters for feature extraction. In the HiViT backbone network, the first stage of its shallow network outputs first-scale features, which include the first-scale features of both the template image and the search region image. The second stage of its shallow network outputs second-scale features, which also include the second-scale features of both the template image and the search region image. Simultaneously, the template token corresponding to the template image and the search region token corresponding to the search region image are obtained from the ends of the HiViT backbone network. Each scale feature comprises multiple spatial units, and each spatial unit has corresponding spatial unit features.

[0014] Then, the first-scale features and the second-scale features are reweighted. The reweighting operation is as follows: each spatial unit in each scale feature is used as the processing unit, and spatial reweighting and channel reweighting are performed on each spatial unit feature to enhance the target-related response and suppress background noise.

[0015] In complex scenarios, a single spatial unit may simultaneously contain local target features, background texture, and interference responses. If average pooling is directly applied to the spatial unit features, background noise and irrelevant responses may be aggregated into the prototype token, thus reducing its discriminative power. To address this issue, this invention performs feature reweighting before feature compression. By adaptively reweighting both spatially and channel-wise for each spatial unit feature, background interference is suppressed, and target-related responses are highlighted, thereby providing more discriminative feature input for feature compression.

[0016] Subsequently, feature compression is performed on the first-scale and second-scale features after feature reweighting, compressing a large number of spatial unit features into a small number (e.g., 2 per scale) of prototype tokens. These prototype tokens include first-scale prototype tokens and second-scale prototype tokens. The first-scale prototype tokens include the first-scale prototype token of the template image and the first-scale prototype token of the search region image, while the second-scale prototype tokens include the second-scale prototype token of the template image and the second-scale prototype token of the search region image.

[0017] Finally, the prototype token, template token, and search region token are concatenated to form the input sequence, which is then fed into the Transformer encoder for global attention interaction. In the Transformer encoder's output, the portion corresponding to the search region token is fed into the prediction head, which outputs the target bounding box coordinates.

[0018] This invention utilizes the inherent hierarchical structure of the HiViT backbone network to directly extract multi-scale features from the first and second stages of its shallow network for subsequent feature reweighting and feature compression. These multi-scale features adopt a representation method with spatial units as the basic granularity, meaning that each scale feature contains multiple spatial units, and each spatial unit retains a finer-grained two-dimensional grid structure. Specifically, the first-scale feature output by the first stage has a fine-grained spatial structure (4×4 sub-grids), and the second-scale feature output by the second stage has a coarse-grained spatial structure (2×2 sub-grids). Thus, multi-scale representations are generated during the forward propagation of the HiViT backbone network without the need to introduce an additional feature pyramid to construct multi-scale features.

[0019] The hierarchical unit features provided by the HiViT backbone network can avoid the computation and memory overhead caused by additional multi-scale branches, and preserve the two-dimensional structure of image patches within spatial units, so that subsequent operations can perform discriminative aggregation in the already formed local structure.

[0020] In another embodiment, such as Figure 2As shown, feature reweighting includes: using each spatial unit in the features at each scale as the processing unit, generating channel weights and spatial weights of the spatial units in parallel; multiplying the spatial weights element-wise with the spatial unit features to perform spatial reweighting of the spatial unit features; and multiplying the channel weights element-wise with the spatially reweighted spatial unit features to perform channel reweighting of the spatial unit features.

[0021] Specifically, spatial weight generation branches and channel weight generation branches are set within each spatial unit to generate spatial and channel weights of the input features in parallel. Subsequently, the channel and spatial weights jointly modulate the input features along the main feature path to enhance the target-related response and suppress background texture and interference responses. The width and height of the input features are equal, taking values ​​of 2 or 4, corresponding to the window size of the first or second stage of the shallow network in the HiViT backbone network, i.e., the height and width of the sub-grid within the spatial unit.

[0022] The method for generating spatial weights includes: performing average pooling and max pooling along the spatial dimensions within the spatial unit; inputting the results of average pooling and max pooling into a multilayer perceptron; and then adding the two outputs of the multilayer perceptron element-wise and activating them with a sigmoid function to obtain the spatial weights, specifically: , in, The spatial unit characteristics after spatial reweighting. As a spatial unit characteristic, For element-wise multiplication, The spatial modulation intensity coefficients are learnable and initialized to 0. For spatial weights.

[0023] The average pooling branch and the max pooling branch share the same set of multilayer perceptron parameters; the sigmoid function is used to map the output of the multilayer perceptron to the range of 0-1 as spatial weights.

[0024] The method for generating channel weights includes: performing average pooling and max pooling along the channel dimensions within the spatial unit; concatenating the results of average pooling and max pooling, and then sequentially performing convolution operations and sigmoid function activation to obtain the channel weights, specifically: , in, The spatial unit characteristics after channel reweighting. The learnable channel modulation intensity coefficients are initialized to 0. For channel weights, the convolutional layer uses a 3×3 kernel size, a stride of 1, and a padding size of 1 to keep the spatial size of the output features unchanged.

[0025] At the start of training, and Initializing them to 0 degenerates the spatial and channel reweighting processes into identity mappings, thus preventing the spatial and channel weight generation branches from excessively perturbing the input features. As training progresses, the modulation intensity coefficients and the parameters of the corresponding weight generation branches gradually learn to enhance the target-related response and suppress background and interference responses.

[0026] In another embodiment, feature compression includes: performing average pooling on the reweighted spatial unit features; performing layer normalization and linear projection on the average pooled spatial unit features sequentially to unify the spatial unit features to the same feature dimension; calculating the similarity between a preset number of learnable query vectors and the linearly projected spatial unit features, and performing a weighted summation on the linearly projected spatial unit features based on the similarity calculation results to obtain a preset number of prototype tokens, including a first-scale prototype token and a second-scale prototype token; adding corresponding learnable scale embeddings to the first-scale prototype token and the second-scale prototype token respectively; concatenating the first-scale prototype token with the added learnable scale embeddings and the second-scale prototype token, and then inputting them into a residual multilayer perceptron for fusion.

[0027] The process involves calculating the similarity between a predetermined number of learnable query vectors and the linearly projected spatial unit features. Based on the similarity calculation results, a weighted sum is performed on the linearly projected spatial unit features to obtain a predetermined number of prototype tokens. This includes: calculating the normalized cosine similarity between each learnable query vector and each linearly projected spatial unit feature; scaling the normalized cosine similarity using a temperature parameter; performing softmax normalization on the scaled normalized cosine similarity to obtain the contribution weight of each spatial unit feature to each learnable query vector; and using the contribution weight to perform a weighted sum on each spatial unit feature to obtain the prototype token corresponding to each learnable query vector.

[0028] Specifically, Figure 3 This is a flowchart of the feature compression method of the present invention, as shown below. Figure 3 As shown, the feature compression process includes three steps.

[0029] First, average pooling is performed on the reweighted features of each spatial unit, compressing the multi-dimensional features within each spatial unit into a single vector. Then, layer normalization and linear projection are sequentially applied to the average-pooled spatial unit features, unifying the features of spatial units at different scales to the same feature dimension (preferably 512 dimensions). After the above processing, each spatial unit is converted into a feature vector of a unified dimension, which is used for subsequent similarity calculation with the learnable query vector.

[0030] Secondly, learnable query vectors are used to perform prototype encoding and discriminative compression on the processed spatial unit features. Specifically, a predetermined number (preferably two) of learnable query vectors are introduced to filter and aggregate the most discriminative information from a large number of spatial unit features. The number of learnable query vectors is the number of prototype tokens generated subsequently. For each learnable query vector and each spatial unit feature, the normalized cosine similarity between them is calculated, and a temperature parameter is introduced to scale the normalized cosine similarity to control the amplification of the difference in normalized cosine similarity. The smaller the temperature parameter, the greater the amplification of the difference in normalized cosine similarity, making the weight distribution after softmax normalization sharper and paying more attention to spatial unit features with high similarity; the larger the temperature parameter, the smoother the weight distribution, as shown below: , in, To normalize the cosine similarity, The characteristics of the normalized spatial units, For learnable query vectors, This is the normalized learnable query vector. The dot product of the normalized spatial unit features and the normalized learnable query vector. This refers to the temperature parameter.

[0031] Then, softmax normalization is applied to each normalized cosine similarity corresponding to each learnable query vector, converting the normalized cosine similarity into the contribution weights of each spatial unit feature to the learnable query vector, expressed as: , in, To contribute weight, This represents the summation of the normalized cosine similarity index values ​​of all spatial unit features, such that the sum of all contribution weights is 1.

[0032] Next, the features of all spatial units after linear projection are weighted and summed using contribution weights to obtain the prototype token corresponding to the learnable query vector, i.e.: , in, For prototype token, This involves a weighted summation of all spatial unit features based on their contribution weights.

[0033] In this way, each learnable query vector independently generates a prototype token, and a preset number of learnable query vectors collectively generate a preset number of prototype tokens. This mechanism can adaptively select the most discriminative spatial unit features for aggregation, while suppressing redundant or interfering information, compressing the original large number of spatial unit features into a small number of prototype tokens rich in discriminative information.

[0034] Compared with the method of directly concatenating all multi-scale spatial unit features, the feature compression of the present invention has two advantages: First, it significantly reduces the token sequence length, compressing the original large number of spatial unit features into a small number of prototype tokens; Second, it can adaptively select the most discriminative spatial unit features for aggregation based on the current input, rather than processing all spatial unit features indiscriminately, thereby forming a discriminative representation that is closer to the target feature distribution.

[0035] Finally, as Figure 4 As shown, cross-scale fusion is performed on prototype tokens of different scales. Specifically, firstly, learnable scale embeddings are superimposed on the first-scale prototype token and the second-scale prototype token respectively to distinguish the feature sources of different scales; then, layer normalization is performed on the superimposed first-scale prototype token and the second-scale prototype token respectively, and then the layer-normalized first-scale prototype token and the second-scale prototype token are concatenated in the sequence dimension; next, the concatenated result is input into a residual multilayer perceptron for further fusion to enhance the coupling between information of different scales. This residual multilayer perceptron refers to a multilayer perceptron containing residual connections, whose output is the superposition of its input and the output of the multilayer perceptron. The last linear layer of the residual multilayer perceptron is preferably initialized to 0, so that the approximate identity mapping is obtained in the early stage of training, thereby obtaining a smoother optimization path; finally, learnable position encoding is added to the output of the residual multilayer perceptron to preserve the position information of each prototype token in the sequence.

[0036] Existing tracking methods have verified the effectiveness of introducing multi-scale features in improving tracking performance. However, directly concatenating multi-scale features using Transformer encoders often leads to a significant increase in the number of tokens, thereby introducing high attention overhead in the Transformer encoder. To compress multi-scale features into a small number of tokens, allowing them to be injected into the Transformer encoder in a compact cue format, and to achieve discriminative feature enhancement while maintaining computational control, this invention proposes a feature compression method.

[0037] In another embodiment, the input sequence is fed into the Transformer encoder for feature interaction. Specifically, a prefix concatenation method is used, where the first-scale prototype token of the template image, the second-scale prototype token of the template image, the first-scale prototype token of the search region image, and the second-scale prototype token of the search region image are concatenated as prefixes before the template token and the search region token, and then fed into the Transformer encoder for global attention interaction.

[0038] Specifically, the input sequence is formed by concatenating the following tokens in the order of "first-scale prototype token of the template image, second-scale prototype token of the template image, first-scale prototype token of the search region image, second-scale prototype token of the search region image, template token, and search region token". Each of these tokens carries multi-scale discriminative information for its respective branch.

[0039] In the above input sequence, since the number of prototype tokens is much smaller than the number of template tokens and search region tokens (preferably, each scale is compressed into 2 prototype tokens), the increase in sequence length is minimal, and the overall computational overhead remains within a controllable range. Simultaneously, placing the search region token at the very end of the sequence facilitates the subsequent prediction head's rapid location and extraction of the portion corresponding to the search region from the Transformer encoder output for bounding box prediction.

[0040] Meanwhile, the first-scale prototype token of the template image, the second-scale prototype token of the template image, the first-scale prototype token of the search region image, and the second-scale prototype token of the search region image serve as compact representations that aggregate multi-scale discriminative information. During the self-attention interaction process, these tokens can provide multi-scale discriminative priors for the template token and the search region token, guiding the present invention to focus on the key features of the target.

[0041] Compared with the approach of directly increasing the number of template tokens or search region tokens to introduce multi-scale features, this invention compresses a large number of spatial unit features into a small number of prototype tokens and injects them in the form of sparse prefixes. With only a very small number of tokens added, multi-scale discriminative information can be effectively guided to feature interaction, keeping the overall computational cost within a controllable range, while significantly improving tracking performance in complex scenes.

[0042] Finally, the constructed input sequence is fed into the Transformer encoder for global attention interaction. The Transformer encoder uses a multi-head self-attention mechanism to enable global information exchange and feature fusion between all pairs of tokens in the sequence. In the output of the Transformer encoder, since the search region token is located at the end of the sequence, the part corresponding to the search region can be directly located and extracted, and then fed into the prediction head for target classification and bounding box regression, outputting the precise location of the target.

[0043] In another embodiment, the present invention also provides a Transformer target tracking device based on multi-scale feature compression representation, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the above-described Transformer target tracking method based on multi-scale feature compression representation.

[0044] The core of this invention lies in constructing an efficient, compact, and highly discriminative multi-scale information fusion framework. First, by leveraging the inherent hierarchical features of the HiViT backbone network, complementary multi-scale feature sources containing fine-grained texture and coarse-grained semantics are acquired with zero additional overhead. Second, a feature reweighting method and a feature compression method are designed. The feature reweighting method performs lightweight channel reweighting and spatial reweighting on each spatial unit before feature compression, effectively enhancing the target-related response and suppressing background noise. The feature compression method, through a prototype encoding mechanism based on learnable query vectors, adaptively aggregates massive spatial unit features into a very small number of highly representative prototype tokens, achieving a fundamental transformation of multi-scale information from spatial redundancy to semantically refined cues. Finally, these compressed prototype tokens are injected into the Transformer's attention interaction with a very low sequence length increment, jointly modeling with template tokens and search region tokens.

[0045] The effects of the present invention will be further illustrated below with simulation experiments.

[0046] 1. Experimental setup details: (1) Experimental environment: This experiment was conducted on a server equipped with an Intel Xeon Platinum 8473C processor, 504GB of RAM, and four NVIDIA RTX 4090 graphics cards (each with 24GB of VRAM). The model was implemented using Python 3.9 and PyTorch 1.13.

[0047] (2) Model settings: This experiment builds a model based on the Transformer tracking framework, using the HiViT backbone network for the visual core and initializing it with a Masked Autoencoder (MAE). The proposed feature reweighting and feature compression methods are introduced into this framework to enhance the discriminative representation of multi-scale features. In the experiment, the input resolutions of the search region image and the template image are set to 256×256 and 128×128, respectively.

[0048] (3) Training strategy: The model was jointly trained on the training sets of COCO, TrackingNet, LaSOT, and GOT-10k. The video dataset directly employed sequential sampling to construct training samples. COCO generated pseudo-sequences using random geometric and illumination perturbations to simulate cross-frame changes. Conventional data augmentation (translation, scale perturbation, brightness jitter, etc.) was used during training. The optimizer was AdamW, with a backbone learning rate of 4×10⁻⁶. -5 The learning rate for the remaining parameters is 4×10. -4 The weight decays to 1×10 -4 The training process consisted of 150 epochs, with the learning rate decreasing to one-tenth of its original value after the 120th epoch. Training employed a sequential sampling and causal sampling mechanism, setting the sampling interval of the template frame to 4 and the sampling span of the search frame to 8.

[0049] (4) Predictor head and loss function: This invention uses a center confidence graph for classification supervision and L1 and GIoU joint loss for bounding box regression.

[0050] 2. Evaluation of digital simulation experiments: (1) Tracking performance evaluation: To verify the effectiveness of this invention, this experiment was conducted using LaSOT and LaSOT. ext The results were compared and evaluated against three public benchmark datasets: GOT-10k. The comparison methods included SiamFC, MDNet, ECO, SiamPRN++, Ocean, TrDiMP, TransT, AutoMatch, STARK, GTELT, AiATrack, MixFormer, SimTrack, OSTrack, VideoTrack, SeqTrack, ARTrack, and EVPTrack, covering classic tracking frameworks and representative Transformer trackers in the current field. To ensure fairness and reliability, methods with input resolutions similar to those of this invention were selected for comparison. For evaluation metrics, LaSOT was compared with LaSOT... extThe success rate was evaluated using the area under the curve, normalized precision, and accuracy. For GOT-10k, the average overlap rate, SR0.5, and SR0.75 were used, where SR0.5 represents the medium threshold success rate and SR0.75 represents the high threshold success rate. The comparison results are shown in Table 1.

[0051] Table 1. Performance comparison of the present invention and the comparison method on three major benchmark datasets. As can be seen from Table 1, this invention utilizes LaSOT and LaSOT... ext Stable and consistent performance gains were achieved on the three mainstream benchmark datasets GOT-10k, verifying the effectiveness of the invention under different evaluation dimensions and different data distributions.

[0052] On the LaSOT dataset, this invention achieved a success rate area under the curve (AUC) of 70.9%, a normalized accuracy of 82.0%, and an accuracy of 78.2%. Compared with the comparison methods, all three metrics were improved, with the gains in normalized accuracy and accuracy being particularly significant. Considering that LaSOT is known for its long sequences and multi-attribute challenges, tracking errors are prone to accumulate during long-term tracking and be amplified by subsequent frames. This result demonstrates that the feature compression method proposed in this invention can provide more stable target cues for feature interaction, enabling the invention to maintain high localization consistency and accuracy even when target deformation, target scale changes, target occlusion, and cluttered backgrounds occur repeatedly, thus leading to simultaneous improvements in the AUC and accuracy metrics.

[0053] In LaSOT ext On the dataset, this invention achieved a success rate of 50.1% with an area under the curve, 60.7% normalized accuracy, and 56.8% precision, demonstrating a more significant overall gain across all metrics compared to other methods. Due to LaSOT... ext Building upon LaSOT, this invention further enhances the evaluation difficulty under cross-class and complex interference conditions, and its performance improvement better reflects its resistance to interference from similar targets, cluttered target backgrounds, and target deformation. The results show that the proposed feature reweighting method reweights target-related responses within spatial cells, effectively suppressing the propagation of background noise. Meanwhile, the feature compression method injects fine-grained texture and high-level semantics into feature interactions in the form of compressed prototype tokens, making it easier to maintain the target-background discrimination boundary in strongly interfered scenarios, thus improving performance in LaSOT. ext It demonstrates a more prominent robustness advantage.

[0054] On the GOT-10k dataset, this invention achieved an average overlap rate of 75.2%, an SR0.5 of 85.2%, and an SR0.75 of 73.5%. The improvements in average overlap rate and SR0.5 demonstrate the robustness of this invention under average overlap quality and moderate threshold success rate. The gain in SR0.75 is particularly crucial, indicating that this invention has a greater advantage in high-precision localization. Since GOT-10k emphasizes cross-class generalization ability and employs strict training and test set partitioning, SR0.75 often relies more on the fine-grained discriminative ability of feature representations and stable modeling of factors such as target scale changes and target deformation. Experimental results show that the feature compression method of this invention not only improves overall tracking performance but also maintains more stable regression localization under stricter overlap thresholds, thus leading to a significant improvement in SR0.75.

[0055] In summary, this invention demonstrates a consistent performance gain trend across the three benchmark datasets: it improves temporal consistency and localization accuracy on long-sequence, multi-interference LaSOT datasets, and further enhances performance on LaSOT datasets with stronger interference and greater demands on generalization ability. ext This further amplifies the robustness advantage and significantly improves high-precision localization capability on GOT-10k, which emphasizes cross-class generalization. The above results demonstrate that this invention, through feature reweighting and feature compression methods, compresses and injects multi-scale feature cues from the template image and the search region image, effectively improving the discriminability of feature representation and localization robustness without significantly increasing sequence redundancy.

[0056] (2) Success rate (area under the success rate curve) analysis under different complex scenarios: Figure 5 The image shows a radar chart of the success rate under various complex scenes on the LaSOT dataset. In the chart, the red line represents the success rate of this invention, the green line represents the success rate of EVPTrack, the purple line represents the success rate of OSTrack, the orange line represents the success rate of SeqTrack, and the blue line represents the success rate of MixFormer. As can be seen from the chart, the radar chart of this invention on the 14 complex attributes of the LaSOT dataset (target rotation, target aspect ratio change, target out of view, target partial occlusion, target scale change, target deformation, low resolution, target complete occlusion, camera motion, target motion blur, target background clutter, illumination change, viewpoint change, and rapid target movement) is generally extended beyond the representative Transformer tracking frameworks of EVPTrack, OSTrack, SeqTrack, and MixFormer. This indicates that this invention has more stable localization robustness under multi-factor interference. Furthermore, the overall success rate of this invention reaches 0.709.

[0057] Figures 6-19This is a schematic diagram of the success rate curves for various complex scenarios on the LaSOT dataset. In the diagram, the horizontal axis represents the overlap threshold, and the vertical axis represents the success rate.

[0058] Figure 6 For scenarios where the target aspect ratio changes, it can be seen from the figure that the success rate of this invention is 0.693, the success rate of EVPTrack is 0.691, the success rate of SeqTrack is 0.683, the success rate of MixFormer is 0.677, and the success rate of OSTrack is 0.674. It can be seen that the success rate of this invention is higher than that of other methods in scenarios where the target aspect ratio changes.

[0059] Figure 7 For cluttered target background scenarios, as shown in the figure, the success rate of this invention is 0.630, the success rate of SeqTrack is 0.629, the success rate of EVPTrack is 0.627, the success rate of OSTrack is 0.615, and the success rate of MixFormer is 0.599. Therefore, it can be seen that the success rate of this invention is higher than that of other methods in cluttered target background scenarios.

[0060] Figure 8 As can be seen from the figure, in the corresponding camera motion scene, the success rate of the present invention is 0.742, the success rate of SeqTrack is 0.733, the success rate of EVPTrack is 0.731, the success rate of OSTrack is 0.720, and the success rate of MixFormer is 0.716. It can be seen that the success rate of the present invention in the camera motion scene is higher than that of other methods.

[0061] Figure 9 As can be seen from the figure, in the corresponding target deformation scenario, the success rate of the present invention is 0.722, the success rate of EVPTrack is 0.721, the success rate of OSTrack is 0.712, the success rate of SeqTrack is 0.708, and the success rate of MixFormer is 0.697. It can be seen that the success rate of the present invention in the target deformation scenario is higher than that of other methods.

[0062] Figure 10 In the context of a fast-moving target scenario, the success rate of EVPTrack is 0.581, while that of the present invention is 0.579, SeqTrack is 0.578, MixFormer is 0.566, and OSTrack is 0.557. Therefore, the success rate of the present invention in a fast-moving target scenario is only lower than that of EVPTrack, which has the highest success rate.

[0063] Figure 11 For scenarios where the target is completely occluded, as shown in the figure, the success rate of this invention is 0.631, while that of EVPTrack is 0.623, SeqTrack is 0.616, MixFormer is 0.614, and OSTrack is 0.595. This indicates that the success rate of this invention is higher than that of other methods in scenarios where the target is completely occluded.

[0064] Figure 12 As can be seen from the figure, in the corresponding lighting change scenario, the success rate of the present invention is 0.710, the success rate of EVPTrack is 0.698, the success rate of MixFormer is 0.696, the success rate of OSTrack is 0.687, and the success rate of SeqTrack is 0.686. It can be seen that the success rate of the present invention in the lighting change scenario is higher than that of other methods.

[0065] Figure 13 In low-resolution scenarios, as shown in the figure, the success rate of EVPTrack is 0.642, the success rate of the present invention is 0.638, the success rate of MixFormer is 0.629, the success rate of SeqTrack is 0.622, and the success rate of OSTrack is 0.616. Therefore, it can be seen that the success rate of the present invention in low-resolution scenarios is only lower than that of EVPTrack, which has the highest success rate.

[0066] Figure 14 For the target motion-blurred scene, as can be seen from the figure, the success rate of SeqTrack is 0.692, the success rate of the present invention is 0.684, the success rate of EVPTrack is 0.681, the success rate of MixFormer is 0.665, and the success rate of OSTrack is 0.664. Therefore, it can be seen that the success rate of the present invention in the target motion-blurred scene is only lower than that of SeqTrack, which has the highest success rate.

[0067] Figure 15 As can be seen from the figure, in the corresponding target rotation scenario, the success rate of the present invention is 0.702, the success rate of EVPTrack is 0.701, the success rate of SeqTrack is 0.698, the success rate of OSTrack is 0.686, and the success rate of MixFormer is 0.686. It can be seen that the success rate of the present invention in the target rotation scenario is higher than that of other methods.

[0068] Figure 16For scenarios where the target is out of sight, the success rate of EVPTrack is 0.661, the success rate of the present invention is 0.655, the success rate of SeqTrack is 0.646, the success rate of MixFormer is 0.644, and the success rate of OSTrack is 0.632. Therefore, the success rate of the present invention in scenarios where the target is out of sight is only lower than that of EVPTrack, which has the highest success rate.

[0069] Figure 17 For the target partial occlusion scenario, as can be seen from the figure, the success rate of the present invention is 0.685, the success rate of EVPTrack is 0.678, the success rate of SeqTrack is 0.677, the success rate of OSTrack is 0.666, and the success rate of MixFormer is 0.665. It can be seen that the success rate of the present invention is higher than that of other methods in the target partial occlusion scenario.

[0070] Figure 18 For scenarios involving changes in target scale, the success rate of this invention is 0.708, while that of EVPTrack is 0.703, SeqTrack is 0.695, OSTrack is 0.690, and MixFormer is 0.689. This demonstrates that the success rate of this invention is higher than that of other methods in scenarios involving changes in target scale.

[0071] Figure 19 As can be seen from the figure, in the corresponding perspective change scenario, the success rate of the present invention is 0.719, the success rate of EVPTrack is 0.718, the success rate of SeqTrack is 0.713, the success rate of MixFormer is 0.706, and the success rate of OSTrack is 0.691. It can be seen that the success rate of the present invention in perspective change scenario is higher than that of other methods.

[0072] Depend on Figures 6-19 It can be seen that the present invention shows particularly significant improvements in attributes strongly correlated with changes in the target's appearance and structure, such as target deformation, viewpoint changes, target rotation, and aspect ratio changes. It also performs exceptionally well in occlusion and background interference scenarios, such as complete target occlusion, partial target occlusion, and cluttered backgrounds. Furthermore, it maintains its advantages under global perturbations caused by imaging conditions and motion, such as changes in illumination and camera movement, and demonstrates stronger scale adaptability in response to changes in target scale.

[0073] In summary, this invention successfully overcomes the core challenges faced by existing Transformer trackers when introducing multi-scale features. First, by employing a feature reweighting method, the target signal is adaptively refined during feature preprocessing, effectively suppressing background and noise interference and enhancing robustness in complex scenarios such as occlusion and interference from similar objects. Second, through feature compression, rich multi-scale discriminative cues are carried with extremely low sequence length increments (only a few to dozens of tokens are added), significantly improving performance while keeping additional computational overhead extremely low, achieving a balance between efficiency and effectiveness. Finally, the entire framework adopts an end-to-end design, enabling seamless collaboration from feature extraction to cue injection. The compressed prototype tokens can naturally integrate into the Transformer's attention interaction, guiding this invention to more accurately focus on the multi-scale characteristics of the target, thereby achieving stable and significant performance improvements.

Claims

1. A Transformer target tracking method based on multi-scale feature compression representation, characterized in that, include: The multi-scale features of the template image and the multi-scale features of the search region image are obtained respectively, as well as the template token corresponding to the template image and the search region token corresponding to the search region image; wherein, each scale feature includes multiple spatial units, and the spatial unit has corresponding spatial unit features; A preset number of learnable query vectors are used to calculate the similarity between the spatial unit features and the spatial unit features. The spatial unit features are then weighted and summed based on the similarity calculation results to obtain the preset number of prototype tokens. The prototype token, template token, and search area token are concatenated to form the input sequence; The input sequence is fed into a Transformer encoder for feature interaction, and the location of the target in the search region image is predicted based on the result of the feature interaction.

2. The Transformer target tracking method based on multi-scale feature compression representation according to claim 1, characterized in that, Before calculating the similarity between the spatial unit features and a preset number of learnable query vectors, the method further includes: Using each spatial unit in the features at each scale as a processing unit, the spatial weights and channel weights of the spatial units are generated in parallel. The spatial weights are multiplied element by element by the spatial unit features to perform spatial reweighting on the spatial unit features; The channel weights are multiplied element-wise by the spatially reweighted spatial unit features to perform channel reweighting on the spatial unit features.

3. The Transformer target tracking method based on multi-scale feature compression representation according to claim 2, characterized in that, The method for generating the spatial weights includes: Average pooling and max pooling are performed along the spatial dimensions within the spatial unit, respectively. The results of average pooling and max pooling are input into the multilayer perceptron. The spatial weights are obtained by adding the two outputs of the multilayer perceptron element by element and then activating them with the Sigmoid function.

4. The Transformer target tracking method based on multi-scale feature compression representation according to claim 2, characterized in that, The method for generating the channel weights includes: Average pooling and max pooling are performed along the channel dimensions within the spatial unit, respectively; The results of average pooling and max pooling are concatenated, and then sequentially processed through convolution and Sigmoid activation to obtain the channel weights.

5. The Transformer target tracking method based on multi-scale feature compression representation according to claim 2, characterized in that, The spatial weights are multiplied element-wise by the spatial unit features, and the spatial unit features are spatially reweighted, specifically as follows: ， in, The spatial unit characteristics after spatial reweighting. As a spatial unit characteristic, For element-wise multiplication, The spatial modulation intensity coefficients are learnable and initialized to 0. For spatial weights.

6. The Transformer target tracking method based on multi-scale feature compression representation according to claim 5, characterized in that, The channel weights are multiplied element-wise by the spatially reweighted spatial unit features, and the spatial unit features are then subjected to channel reweighting, specifically as follows: ， in, The spatial unit characteristics after channel reweighting. The learnable channel modulation intensity coefficients are initialized to 0. This represents the channel weight.

7. The Transformer target tracking method based on multi-scale feature compression representation according to claim 1, characterized in that, Before calculating the similarity between the spatial unit features and a preset number of learnable query vectors, the method further includes: The spatial unit features are subjected to average pooling; The spatial unit features after average pooling are sequentially subjected to layer normalization and linear projection to unify the spatial unit features to the same feature dimension.

8. The Transformer target tracking method based on multi-scale feature compression representation according to claim 1, characterized in that, A preset number of learnable query vectors are used to calculate similarity with the spatial unit features. Based on the similarity calculation results, the spatial unit features are weighted and summed to obtain the preset number of prototype tokens, including: Calculate the normalized cosine similarity between each learnable query vector and the features of each spatial unit; The normalized cosine similarity is scaled using a temperature parameter; The scaled normalized cosine similarity is subjected to softmax normalization to obtain the contribution weight of each spatial unit feature to each learnable query vector. The contribution weights are used to perform a weighted summation of the features of each spatial unit to obtain the prototype token corresponding to each learnable query vector.

9. The Transformer target tracking method based on multi-scale feature compression representation according to claim 1, characterized in that, The multi-scale feature includes a first-scale feature and a second-scale feature, and the prototype token includes a first-scale prototype token and a second-scale prototype token. After obtaining the preset number of prototype tokens, the process also includes: Add the corresponding learnable scale embeddings to the first scale prototype token and the second scale prototype token respectively. The first-scale prototype token with added learnable scale embedding is concatenated with the second-scale prototype token, and then fed into a residual multilayer perceptron for fusion.

10. A Transformer target tracking device based on multi-scale feature compression representation, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-9.

Images