A target-aware Transformer drone tracking method focusing on key information

Abstract

This application proposes a target-aware Transformer drone tracking method that focuses on key information. First, a single-stream tracking framework that integrates feature learning and target search is constructed to improve the information interaction between tokens; then an adaptive relationship modeling mechanism is constructed to model the relationship between the target template token and the search area token and divide the type tags, and terminate the subsequent reasoning and learning of the tokens marked as background in advance, so as to focus on the key information of the key area; finally, a multi-layer feature aggregation module is constructed to retain the detailed features of the target to further enhance the discriminative power of the feature representation, and finally the target two-dimensional feature map is output through the prediction head. Experiments on the UAV123, DTB70, and UavDrak135 drone tracking benchmarks show that this method has achieved superior performance.

Description

Technical Field

[0001] The present invention relates to the field of intelligent recognition, and in particular to a Transformer-based unmanned aerial vehicle tracking method. Background Art

[0002] Unmanned aerial vehicle (UAV) target tracking aims to determine the position of the target for each frame of the UAV video based on the target's initial state. It is used in forest fire prevention, disaster detection, emergency search and rescue, etc. Due to the high maneuverability of UAVs, UAV target tracking frequently faces challenges such as extreme viewing angles, similar background interference, and severe occlusion. In addition, UAVs have limited battery and computing resources, which puts more stringent requirements on their lightweight tracking. Therefore, a superior UAV tracker must meet the two requirements of high efficiency and high precision.

[0003] At present, the UAV target tracking methods are mainly divided into trackers based on correlation filters (CF) and trackers based on deep convolutional neural networks (CNN). Correlation filters convert spatial domain similarity into element calculations in the Fourier domain to obtain higher tracking efficiency. In contrast, deep convolutional neural network trackers obtain higher tracking accuracy by learning the semantic information and appearance features of the target, but they also require a lot of computing resources. Therefore, discriminative correlation filters dominate the field of UAV target tracking due to their high tracking rate. Many methods improve tracking accuracy by suppressing background abnormal response information, enhancing nighttime illumination, constructing spatiotemporal regularization, and enhancing spatial feature learning on the basis of ensuring tracking efficiency. In addition, some lightweight deep convolutional neural network trackers based on Siamese networks have been proposed for UAV tracking, and the accuracy and efficiency of Siamese network UAV tracking methods have been further improved by exploring methods such as enhancing target feature learning, mining spatiotemporal information, or pruning rank information.

[0004] Since the tracker based on Siamese network ignores global information and easily leads to local optimization, the ViT network framework is applied to target tracking and shows excellent performance. Thanks to its excellent attention mechanism and contextual relationship capture ability, a variety of ViT-based tracking methods have been proposed and show better performance than CNN-based trackers. However, since the current Transformer network divides the input image into regular and fixed image blocks, and gives the same attention to all input image blocks and interacts with information, the target representation learned by the tracker does not have sufficient discriminative power, and it is easy to confuse the target and background in the drone target tracking scene. This shows that the input of fixed image blocks is suboptimal in drone tracking. Just as in human visual analysis tracking, the line of sight will focus on objects near the target and similar to the target, and automatically ignore the rest of the area.

[0005] In this regard, this paper proposes a target-aware UAV Transformer Tracker Focusing on Key Information (TUTTFKI), referred to as TUTTFKI. First, feature learning and template search are integrated into a single-stream visual Transformer network for efficient learning and reasoning. This single-stream visual Transformer network not only simplifies the tracking process, but also enhances the interaction between feature learning and template search in the tracking process, so that the learned features contain more target and background relationship information and better capture the correlation. Secondly, an adaptive relationship modeling mechanism is constructed to model the relationship between the tokens of the search area and the target template tokens and divide them into background tokens and target tokens, blocking the reasoning and learning of the background tokens, so that the tracker only focuses on the area that is more similar to the target, thereby strengthening the focus on the key area and reducing the interference of background information on feature learning, further improving the discriminative ability of model learning. Finally, a feature aggregation module is constructed. Due to the fixed image block input method, some target features are divided into background tokens and terminated and discarded in advance. This application further enhances the discriminative power of feature representation learned by the tracker by retaining image details of each layer of the network and gradually aggregating the marked token features from the previous layer in the final inference stage, and finally outputs the target two-dimensional feature map through the prediction head. Through comparative experiments on three mainstream drone datasets, UAV123, DTB70 and UAVDark135, TUTTFKI achieved tracking accuracies of 0.863, 0.859 and 0.746 at a tracking speed of 69.73 frames per second. Summary of the invention

[0006] The present invention proposes a target-aware Transformer UAV tracking method focusing on key information, which solves the problems of tracking and recognition accuracy and speed in the prior art.

[0007] The technical solution of the present invention is as follows: a target-aware UAV Transformer Tracker Focusing on Key Information (TUTTFKI), referred to as TUTTFKI;

[0008] Firstly, a single-stream tracking framework integrating feature learning and target search is constructed to improve the information interaction between tokens. Secondly, an adaptive relationship modeling mechanism is constructed to model the relationship between target template tokens and search area tokens and classify the type tags, so as to terminate the subsequent reasoning and learning of tokens marked as background in advance, and focus on the key information of key areas. Finally, a multi-layer feature aggregation module is constructed to retain the detailed features of the target and further enhance the discriminative power of the feature representation, and finally output the target two-dimensional feature map through the prediction head.

[0009] The steps of constructing a single-stream tracking framework integrating feature learning and target search are as follows: 1. Setting the target template as , whose formula is , set the search image to , the formula is: , target template image Search the image with the target object as the center Represents a larger region in subsequent frames that contains the target; the target template and search image are then segmented and flattened into a sequence of image patches through a patch embedding layer, followed by a trainable linear projection layer produce tokens, whose formula is in, is the embedding dimension of each token that is input to the encoder. Let For the The token passes through the Transformer block of the first The layer Transformer decoder output is: ,The output search tokens from the last encoder layer are decoupled and reshaped into a 2D feature map according to their original spatial locations.

[0010] The image block sequence is Image block sequences, the number of which is: and .

[0011] Each encoder layer updates the input token through a multi-head attention module and a feed-forward network.

[0012] The adaptive relational modeling mechanism allocates a neuron in the multi-layer perceptron layer to complete the relational modeling, and then the relational information embedded in the token is merged into the single-stream backbone tracking network; the input tokens are divided into two categories, namely target tokens and background tokens right layer The adaptive relationship modeling mechanism calculates its classification probability score , defined by the following formula: .

[0013] Said It is a relational modeling operation through the multi-layer perceptron layer of Transformer. is a logistic sigmoid function, Indicated in dimension of , and are token shift and scale parameters, which are shared across layers for all token parameters. Through the relational modeling operation, each token is classified as: in is a very small constant and allows , when the token termination score exceeds When the token is considered to be a background token , otherwise it is the target token ; To avoid the category The background tokens of interfere with the learning of the model's target features, and the adaptive relationship modeling mechanism will terminate the category , the token value will be cleared to zero and prevented from paying attention to other tokens, and the token will not be updated thereafter. In the token relationship modeling process, each token is assigned to the target and background classes, and the feature aggregation module records the relationship between the tokens marked as targets; in the tag upsampling process, according to the token tag, the token features saved by the network are copied to the corresponding upsampled tags, and the feature aggregation module continues to add the token features extracted by the previous layer of the network to the upsampled tokens; these target tokens are learned and processed by the transformer block; this operation is performed step by step until all the tokens marked as target categories are aggregated; through the feature aggregation module, the local features of each image block can be easily reshaped into a global feature map of the tracked target for subsequent target tracking.

[0014] The two-dimensional feature map is used as the input of the convolutional head for target bounding box prediction, through a prediction head composed of multiple convolutional layers, batch normalization layers, and activation function layers that is completely based on the convolutional network. , directly estimate the bounding box of the target and generate the target classification score , Local Offset and the normalized bounding box size ; The target position is determined by the highest classification score: ; and the final target bounding box is estimated by the following formula ; For tracking tasks, weighted focal loss is introduced for target classification, and Loss and Perform bounding box regression; the total loss function is: .

[0015] The algorithm of the tracking method is as follows:

[0016] 1) Input the first frame target initialization template and search images ;

[0017] 2) Flatten the input image and target into K image blocks, and use the image embedding operation formula: Generate token embeddings for each image patch;

[0018] 3) Initialize the classification of each token ;

[0019] 4) Through the formula Calculate each token in Classification probability of the layer;

[0020] 5) For tokens classified as background, , with a score exceeding The token of is assigned the value zero;

[0021] 6) Aggregate the token features marked as targets in each layer through the feature aggregation module;

[0022] 7) Decouple the outputs of the last encoder layer and reshape them into a 2D feature map according to their original spatial positions;

[0023] 8) Through the prediction head formula: Get the target location and predicted bounding box;

[0024] 9) If it is the last frame, stop the algorithm; otherwise, repeat steps 3) to 8);

[0025] 10) Return the tracking results.

[0026] The working principle and beneficial effects of the present invention are as follows: 1) a single-stream UAV tracking method integrating feature learning and target search is proposed to achieve accurate tracking at a faster tracking rate;

[0027] 2) Based on human visual analysis, an adaptive relationship modeling mechanism is constructed to model the relationship between search area tokens, dividing them into current and background tokens, and performing termination reasoning on background tokens, thereby focusing on the key information of the key area and further improving the feature discrimination of the model extraction;

[0028] 3) Construct a feature aggregation module to preserve the image details of each layer of the network and further strengthen the feature representation of the tracker by gradually aggregating the token features marked in the previous layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0030] Figure 1 It is a schematic diagram of the tracking flow chart of the present invention;

[0031] Figure 2 It is a schematic diagram of a feature aggregation module of the present invention;

[0032] Figure 3 It is a comprehensive evaluation result diagram of the present invention;

[0033] Figure 4 This is a visualization result diagram of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] like Figure 1 As shown, this embodiment proposes a target-aware Transformer UAV tracking method focusing on key information, Target-aware UAV Transformer Tracker Focusing on Key Information, TUTTFKI, referred to as TUTTFKI;

[0036] The specific operations are as follows:

[0037] Experimental configuration: All experiments were conducted using Ubuntu 18.04, with an Intel Core I9-9900K CPU and 32GB memory. The TUTTFKI prediction head is constructed using a lightweight fully convolutional network, with each output consisting of four stacked Conv-BN-ReLU layers; the sizes of the template and search area are set to 128×128 and 256×256, respectively, with the parameters Set to 2, Set to 5, Set to 0.0001.

[0038] 1. Dataset

[0039] In order to verify the effectiveness of the TUTTFKI algorithm and its tracking performance in various complex scenarios, the UAV123, DTB70 and UAVDark135 drone video datasets were used to compare the algorithm with 9 tracking algorithms, including AutoTrack, ASRCF, ARCF, ECO, MCPF, HiFT, SiamAPN++, SiamRPN++ and TCTrack. The UAV123 dataset contains a total of 123 video sequences shot by drones and more than 110,000 frames of drone images, which can cover most drone tracking scenarios, such as intersections, highways, etc., as well as different types of tracking objects, such as people, ships, cars, etc. The two data benchmarks, DTB70 and UAVDark135, are small target tracking and night tracking images, representing the extremely challenging environment for drone tracking. In addition, comparisons are performed on 12 challenging attributes of the UAV123 dataset, including aspect ratio change (ARC), background clutter (BC), camera motion (CM), fast motion (FM), full occlusion (FOC), illumination change (IV), low resolution (LR), visual out-of-bounds (OV), partial occlusion (POC), scale change (SV), similar objects (SOB) and viewpoint change (VC).

[0040] 2. Evaluation criteria

[0041] In order to fairly and effectively evaluate the tracking accuracy of the algorithm, center location error (CLE) and overlap rate (OR) are used as evaluation indicators. Center error refers to the Euclidean distance between the true position of the target and the position tracked by the method. If the distance is less than a set threshold, it means that the tracking is successful, otherwise it is a failure; the overlap rate calculates the proportion of the overlapping area between the true bounding box of the target and the bounding box tracked by the method to the entire target area, and is also compared with the threshold to determine whether the tracking is successful. All data sets use a center error threshold of 20 and an overlap rate threshold of 0.5.

[0042] 3. Example

[0043] After the TUTTFKI model of the present invention is built, it can be directly verified without training. The verification data of this embodiment uses the UAV123 data set, which contains 123 videos.

[0044] The first step is to initialize the target template

[0045] Select the first frame image, determine the target template to be searched, and set the target template image as , the formula is: Search image set to , the formula is: ; Input video sequence I and the first frame target initialization template; Output the target position and size of each frame in video sequence I.

[0046] The second step is image segmentation input

[0047] The input image and target are segmented and flattened into K image blocks, and a token embedding is generated for each image block through the image embedding operation; TUTTFKI first segments the image pair through the patch embedding layer (PaE), and then segments and flattens it into Image block sequence, the number of image blocks is and . Template Image Centered on the target object, the search image Represents a larger region in subsequent frames that contains the object. The sequence of image patches is passed through a trainable linear projection layer produce The token embedding formula is as follows: in is the dimensionality of each token embedding, calculated as: , indicates a template token embed, Denoted as search region token embedding. Afterwards, these tokens are fed into the encoder, and each encoder layer updates the input tokens through a multi-head attention (MHA) module and a feed-forward network (FFN). Let For the The Transformer block of the layer, the token embedding is The layer Transformer decoder output is: ,in Indicates a connection operation, using , and represents the query, key, and value passed to the multi-head attention block, Since the template token and search token are jointly processed by a multi-head attention block, cross-relation and self-relation modeling are integrated in each encoder layer.

[0048] Step 3: Adaptive Relationship Modeling

[0049] The segmented image blocks need to be initialized first to initialize the classification of each token , since each token in the ViT network is embedded in the In order to avoid TUTTFKI consuming too much resources for relationship modeling, the adaptive relationship modeling mechanism built allocates a neuron in the multilayer perceptron (MLP) layer to complete the relationship modeling. Then the relationship information embedded in the token is merged into the single-stream backbone tracking network. Therefore, the adaptive relationship modeling mechanism does not need to introduce any additional learning parameters;

[0050] The adaptive relational modeling mechanism divides the tokens input to the model into two categories: target tokens and background tokens .right layer The adaptive relationship modeling mechanism calculates its classification probability score Defined by: ,in is the relational modeling operation through the MLP layer of the visual Transformer block, is a logistic sigmoid function, Indicated in dimension of , and are token shift and scaling parameters, which are shared across layers for all token parameters. Through the relational modeling operation, each token is classified as: ,in is a very small constant and allows . When the token expiration score exceeds , we consider the token to be a target token , otherwise it is a background token To avoid the category The background tokens of interfere with the learning of the model's target features, and the adaptive relationship modeling mechanism will terminate the category The token value will be cleared to zero and prevented from following other tokens, and the token will not be updated afterwards;

[0051] set up To pass the Transformer block to tokens The total number of updates is: The termination probability of all tokens available is: ,because and , if there is no limit on the number of updates for each token, it will tend to wait as long as possible to avoid errors. In order to reduce the computational effort of the TUTTFKI tracker, the following waiting loss is used to encourage tokens to terminate early: in Representation Token The waiting function, however, this loss is The weight of treats each token equally. When learning prior knowledge about tokens, it will reduce the ability to learn the correlation between tokens and targets or backgrounds during the model training phase. In order to further improve the token perception ability of the model, the waiting loss is further optimized to : in and is the indicator function, defined as follows: in is a predefined constant used to scale the waiting loss of background tokens. hour, Reduce to The identified background tokens are discarded, the target tokens are retained and proceed to the next step.

[0052] Step 4: Feature aggregation

[0053] Since the input of the VIT-based network module is to cut and flatten the image into a sequence of image blocks, the target edge image block may be marked as background after the relationship modeling mechanism because of the larger background features, thus terminating the reasoning early. Therefore, to ensure that the tracker can obtain shallow features such as texture and shape of the target from the shallow network, we proposed a Transformer-based multi-layer feature aggregation module that can retain image details of all network layers.

[0054] like Figure 2 As shown in the figure, during the token relationship modeling process, each token is assigned to the target and background classes. The feature aggregation module records the relationship between tokens labeled as targets. During the label upsampling process, we copy the token features saved by the network to the corresponding upsampled tokens according to the token labels. After upsampling the labeled tokens, the feature aggregation module continues to add the token features extracted by the previous layer of the network to the upsampled tokens. These target tokens are then learned by the transformer block. This operation is performed step by step until all tokens labeled as target categories are aggregated. Through the feature aggregation module, the local features of each image block can be easily reshaped into a global feature map of the tracked target for subsequent target tracking. In the fifth step, the output search tokens from the last encoder layer are decoupled and reshaped into a two-dimensional feature map according to their original spatial positions.

[0055] The two-dimensional feature map is used as the input of the convolutional head for target bounding box prediction, through a prediction head composed of multiple convolutional layers-batch normalization layers-activation function layers (Conv-BN-ReLU) that is completely based on the convolutional network. , directly estimate the bounding box of the target and generate the target classification score , Local Offset and the normalized bounding box size The target position is determined by the highest classification score: , and the final target bounding box is estimated by the following formula For tracking tasks, a weighted focal loss is introduced for target classification, and Loss and Perform bounding box regression. The total loss function is: , the target position and predicted bounding box can be obtained through the above algorithm; each frame of the video sequence I in the dataset is input into the model for verification. If it is the last frame, stop, otherwise repeat the second to fifth steps.

[0056] Experimental results and analysis

[0057] 1. Overall assessment

[0058] like Figure 3 As shown in FIG. 1 , the comprehensive evaluation results of the present invention and the nine compared methods are shown. The results show that the TUTTFKI of the present invention achieves the first performance in comprehensive accuracy and success rate on the UAV123 dataset and the DTB70 dataset.

[0059] In the UAV123 dataset, the TUTTFKI algorithm achieved excellent performance, with an accuracy and accuracy improvement of 0.023 and 0.018 respectively compared to the second place SiamRPN++. This shows that the TUTTFKI algorithm enhances the information interaction between the target template and the search area by building an integrated target feature learning and target search in a single-stream backbone tracking network, and improves TUTTFKI's ability to model target features. At the same time, the adaptive relationship modeling mechanism is used to model the relationship between the target token and the search token, and the learning and reasoning of the background token is terminated in advance, which further reduces the interference of similar background information in the target feature modeling process, thereby making the learned target features more robust. In the DTB70 dataset, TUTTFKI also achieved excellent performance, with an accuracy and accuracy improvement of 0.027 and 0.021 respectively compared to the second place (SiamRPN++). This shows that when the tracking target is small and effective information is insufficient, the proposed method can focus more on target information mining and extraction through global attention broadcasting and adaptive relationship modeling mechanisms, while avoiding the influence of background information.

[0060] In the UAVDark135 dataset, although TUTTFKI did not win the first place in the comparative experiment, it also won the second place, and the accuracy and correctness differed from the first place (SiamRPN++) by 0.011 and 0.005 respectively. This shows that even in the case of insufficient light, TUTTFKI can still terminate some background tokens in advance through the adaptive relationship modeling mechanism to ensure that the learned features are not interfered by the background. However, due to insufficient light, only a single-stream framework is used, and it is impossible to extract sufficiently discriminative feature representations, so there is still room for further improvement on this dataset.

[0061] 2. Attribute analysis

[0062]

[0063]

[0064] In order to more clearly evaluate the tracking performance of the tracking algorithm under specific challenges, Tables 1 and 2 show the tracking results of the proposed method and nine other advanced trackers in 12 drone tracking challenge attributes in the UAV123 dataset. The results show that the TUTTFKI algorithm achieves the best tracking performance in all other environments except the two challenge attributes of IV and VC. Under the IV and VC challenges, the center position error results of 0.816 and 0.895 and the overlap rate results of 0.611 and 0.706 were obtained respectively. At the same time, TUTTFKI achieved a tracking speed of 69.73 frames per second on the UAV123 and DTB70 datasets. The excellent performance is due to the fact that the current visual Transformer has been trained with large model data, and its ability to extract model features and semantic information of the target has been greatly improved. On the other hand, it is due to the fact that TUTTFKI integrates feature learning and target search in a single-stream network for tracking reasoning, and uses the prior information of the background and the target to model the token relationship, and adaptively stops the learning of background information, so as to achieve more efficient and robust drone tracking.

[0065] 3. Qualitative analysis

[0066] In order to more intuitively demonstrate the effectiveness of the TUTTFKI algorithm, Figure 4As shown in the figure, a visual comparison of TUTTFKI and four tracking algorithms, including SiamRPN++, AutoTrack, SiamAPN++, and ASRCF, on three videos. These include challenges such as background clutter, fast motion, full occlusion, low resolution, and similar objects. The proposed TUTTFKI algorithm can better maintain continuous tracking of the target. This shows that TUTTFKI makes full use of the self-attention mechanism of Transformer to extract the contextual information of the video image and the semantic information of the target, and uses the integrated single-stream network to enhance the information-related interaction between the target template and the search area, thereby improving the model's learning ability. At the same time, the adaptive relationship modeling mechanism and the global attention broadcast mechanism constructed can enhance the model's perception of the target, which further improves the tracker's anti-interference ability to similar targets in the background.

[0067] 4. Ablation experiment

[0068] To verify the effectiveness of the proposed adaptive relationship modeling mechanism and global attention broadcast mechanism, an ablation experiment was conducted on the UAV123 dataset. The results are shown in Table 3.

[0069]

[0070] As can be seen from Table 3, the construction of the adaptive relationship modeling mechanism contributes 4.95% and 2.01% to the improvement of tracking accuracy and success rate respectively. By modeling the relationship between the input tokens, each token is divided into the target class and the background class, thereby reducing the interference of the background tokens in the search area on the target feature learning and ensuring the purity of the feature representation of the target template. Through the global attention broadcast mechanism, the effective attention of the tracker to key information is further strengthened, thus further improving the tracking accuracy and success rate by 7.07% and 4.08%.

[0071] Experimental conclusion

[0072] The present invention proposes a Transformer tracking method with target perception capability that focuses on key information, integrating feature learning and target search into a single-stream tracking backbone network. By constructing an adaptive relationship modeling mechanism, the relationship between the target template and the search area can be modeled, additional spatial interactions can be added, and the rate and accuracy of drone tracking can be improved synchronously through feature fusion. However, when faced with visual changes in the target, the tracker lacks the ability to perceive and learn the background around the target, which reduces the tracker's robustness in extracting target features. In the future, further research will be conducted on extracting spatial and temporal clues from the target template to better refine the relationship modeling between the target template and the search area.

[0073] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A target-aware Transformer drone tracking method focusing on key information, characterized by ； Firstly, a single-stream tracking framework integrating feature learning and target search is constructed to improve the information interaction between tokens; secondly, an adaptive relationship modeling mechanism is constructed to model the relationship between the target template token and the search area token and divide the type tags, so as to terminate the subsequent reasoning and learning of the tokens marked as background in advance, and then focus on the key information of the key area; finally, a multi-layer feature aggregation module is constructed to retain the detailed features of the target to further enhance the discriminative power of the feature representation, and finally output the target two-dimensional feature map through the prediction head; the single-stream tracking framework is a single-stream tracking framework based on ViT, and its steps are as follows:

1. Set the target template to Z, and its formula is Let the search image be X, and the formula is: The target template image Z is centered on the target object, and the search image X represents the image of the area in the subsequent frame that contains the target; the target template and the search image are then segmented and flattened into a sequence of image patches through a patch embedding layer, and then passed through a trainable linear projection layer E(·) to generate K tokens, which is formulated as Where D is the embedding dimension of each token that is input to the encoder. Let is the l-th layer Transformer block, then the token is output through the l-th layer Transformer decoder as: The output search tokens from the last encoder layer are decoupled and reshaped into a two-dimensional feature map according to their original spatial positions; the adaptive relational modeling mechanism allocates a neuron in the multi-layer perceptron layer to complete the relational modeling, and then the relational information embedded in the token is merged into the single-stream backbone tracking network; the input tokens are divided into two categories, namely target tokens C t and background token C b For the token at k in layer l, the adaptive relationship modeling mechanism calculates its classification probability score Defined by the following formula: The B(·) is a relational modeling operation through the multi-layer perceptron layer of the Transformer; the two-dimensional feature map is used as the input of the convolutional head for target bounding box prediction, and a prediction head composed of multiple convolutional layers, batch normalization layers, and activation function layers is completely based on the convolutional network. Directly estimate the bounding box of the target and generate the target classification score Local Offset and the normalized bounding box size The target position is determined by the highest classification score: (x c ,y c ) = argmax (x,y) p(x,y); and the final target bounding box [(x t ,y t ); (w,h)] = [(x c ,y c )+o(x c ,y c );s(x c ,y c )]; For tracking tasks, weighted focal loss is introduced for target classification, and Loss and Perform bounding box regression; the total loss function is: The algorithm of the tracking method is as follows: 1) Input the first frame target initialization template and search images 2) The input image and target are split and flattened into K image blocks, and the image embedding operation formula is used: Generate token embeddings for each image patch; 3) Initialize the classification of each token 4) Through the formula Calculate the classification probability of each token at layer l; 5) For tokens classified as background, Tokens with scores exceeding 1-∈ are assigned a value of zero; 6) Aggregate the token features marked as targets in each layer through the feature aggregation module; 7) Decouple the outputs of the last encoder layer and reshape them into a 2D feature map according to their original spatial positions; 8) Through the prediction head formula: (x c ,y c ) = argmax (x,y) p(x,y), get the target position and predicted bounding box; 9) If it is the last frame, stop the algorithm; otherwise, repeat steps 3) to 8); 10) Return the tracking results.

2. The tracking method according to claim 1, characterized in that: The image block sequence is a P×P image block sequence, and the number of the image blocks is: K Z =H Z W Z / P 2 and K X =H X W X / P 2 .

3. The tracking method according to claim 1, characterized in that: Each encoder layer updates the input token through a multi-head attention module and a feed-forward network.

4. The tracking method according to claim 1, characterized in that: The B(·) is the relational modeling operation through the multi-layer perceptron layer of Transformer, is a logistic sigmoid function, Indicates that in dimension e β and γ are the token shift and scaling parameters, which are shared across layers for all token parameters. By relation modeling operations, each token is classified as: Where ∈ is a constant, and n=0 is allowed. When the token termination score exceeds 1-∈, we consider the token to belong to the background token C. b , otherwise it is the target token C t ; To avoid category C b The background tokens of interfere with the learning of the model's target features, and the adaptive relationship modeling mechanism will terminate the category C b The token will be b The token value is cleared to zero and prevents it from paying attention to other tokens, and it will not be used for C afterwards. b The token is updated.

5. The tracking method according to claim 1, characterized in that: In the token relationship modeling process, each token is assigned to the target and background classes, and the feature aggregation module records the relationship between the tokens marked as targets; in the tag upsampling process, according to the token tag, the token features saved by the network are copied to the corresponding upsampled tags, and the feature aggregation module continues to add the token features extracted by the previous layer of the network to the upsampled tokens; these target tokens are learned and processed by the transformer block; by gradually performing the above operations until all the tokens marked as target categories are aggregated; Through the feature aggregation module, the local features of each image block can be reshaped into a global feature map of the tracked target for subsequent target tracking.

Images