Autoregressive visual target tracking algorithm based on token fusion

Abstract

The invention discloses an autoregressive visual target tracking algorithm based on token fusion, and the algorithm specifically comprises the following steps: 1, obtaining a public visual target tracking data set, and converting the data set into a central coordinate format; 2, preprocessing an initial template of a given video sequence, an online template at the current moment and a search image to obtain visual embedding of the initial template, a dynamic template and the search image; the method comprises the following steps: preprocessing an angular coordinate sequence of a target frame in previous historical N frames of a current search image of an input video to obtain historical track word embedding of the current search image; 3, constructing an autoregressive visual target tracking model based on token fusion, and designing a loss function; step 4, obtaining a final autoregressive visual target tracking model based on token fusion; the method solves the problem of insufficient model feature extraction capability caused by insufficient utilization of spatio-temporal information and redundant background information of a search area in an existing method.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an autoregressive visual target tracking algorithm based on token fusion. Background Art

[0002] Visual object tracking has important applications in the field of computer vision. Visual object tracking aims to continuously estimate the position of an object in a video sequence based on its initial state. It plays an important role in traffic flow monitoring, human-computer interaction, autonomous driving, and military. In practical application scenarios, visual object tracking faces multiple complex challenges, including motion blur, morphological changes, size fluctuations, occlusion, background noise interference, and confusion between similar objects. These factors together make visual object detection a very challenging task.

[0003] Thanks to the rapid development of deep learning, visual target tracking methods based on deep learning are constantly being proposed. Some of the related works are: SiamFC (Bertinetto L, Valmadre J, Henriques JF, et al.Fully-convolutional siamese networks for object tracking[C] / / Computer Vision–ECCV2016Workshops:Amsterdam,The Netherlands,October 8-10and15-16,2016,Proceedings,Part II 14.Springer International Publishing,2016:850-865.),SiamRPN(Li B,Yan J,Wu W,et al.High performance visual tracking with siameseregion proposal network[C] / / Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition.2018:8971-8980.),SiamFC++(Xu Y,Wang Z,Li Z,etal.SiamFC++:Towards robust and accurate visual tracking with target estimation guidelines[C] / / Proceedings of the AAAI Conference on ArtificialIntelligence.2020,34(07):12549-12556.). SiamFC is the first to apply the twin network to visual tracking tasks. It achieves real-time tracking by building a twin network framework with shared weights to evaluate the similarity between the template and the search region. SiamRPN is the first to combine the twin network with the Region Proposal Network (RPN), using RPN to generate candidate regions and perform classification and regression to more accurately locate the target. SiamFC++ adds position regression and quality score based on SiamFC, and uses multiple loss functions for joint training to improve tracking performance.

[0004] After the Transformer was proposed, it was widely used in computer vision tasks due to its powerful sequence modeling capabilities, and gradually expanded to the field of visual target tracking. Among them, the work that uses Transformer for visual tracking includes STARK (Yan B, Peng H, Fu J, et al. Learning spatio-temporal transformer for visual tracking [C] / / Proceedings of the IEEE / CVF International Conference on ComputerVision.2021:10448-10457.), SwinTrack (Lin L, Fan H, Zhang Z, et al. Swintrack: Asimple and strong baseline for transformer tracking[J].Advances in NeuralInformation Processing Systems,2022,35:16743-16754.), ARTrack(Wei STARK uses a specially designed Transformer encoder and decoder to simultaneously capture global dependencies in the spatial and temporal dimensions, thereby generating more discriminative spatiotemporal features, treating target tracking as a problem of directly predicting bounding boxes, simplifying the tracking process and improving tracking accuracy. SwinTrack uses Transformer in feature learning and fusion to enhance feature interactions, and introduces motion tokens to embed historical trajectory information to provide temporal context and improve the model's understanding of spatiotemporal information. ARTrack eliminates complex head networks and loss functions by modeling tracking as a coordinate sequence generation problem, and introduces spatiotemporal cue information. Finally, it combines the generation of visual information autoregression to predict the final target position in the current search frame, thereby improving tracking accuracy.

[0005] However, although the above algorithms have achieved remarkable results in visual tracking tasks, there is still room for improvement. This is mainly because existing methods only use template frames and search frames for feature extraction. This approach ignores the temporal and spatial information association between frames and has difficulty dealing with significant changes in target appearance and position. In addition, the target and background areas in the search frame are not effectively distinguished. This results in a large amount of redundant background information in the search frame, which interferes with the model's feature extraction and matching capabilities. Summary of the invention

[0006] The purpose of the present invention is to provide an autoregressive visual target tracking algorithm based on token fusion, aiming to solve the problems of insufficient model feature extraction capability caused by insufficient utilization of spatiotemporal information and redundant background information in the search area in existing methods.

[0007] The technical solution adopted by the present invention is an autoregressive visual target tracking algorithm based on token fusion, which specifically includes the following steps:

[0008] Step 1: Obtain a public visual target tracking dataset and convert the dataset into a central coordinate format; Step 2: Preprocess the initial template of a given video sequence, the online template at the current moment, and the search image to obtain the visual embedding of the initial template, dynamic template, and search image; Preprocess the angular coordinate sequence of the target box in the previous N historical frames of the current search image of the input video to obtain the historical trajectory word embedding of the current search image; Step 3: Construct an autoregressive visual target tracking model based on token fusion and design a loss function; Step 4: Use the visual target tracking dataset in step 1 to train the autoregressive visual target tracking model based on token fusion to obtain the final autoregressive visual target tracking model based on token fusion.

[0009] The present invention is also characterized in that:

[0010] In step 1, the public visual object tracking dataset is TrackingNet, LaSOT, COCO, OTB100 or UAV123 dataset.

[0011] Step 2 is as follows:

[0012] Step 2.1: Preprocess the initial template of the given video sequence, the dynamic template at the current moment, and the search image to obtain the visual embedding of the initial template, dynamic template, and search image, so as to facilitate the input encoder for feature extraction. The specific operation is as follows: Dynamic template at the current frame t and search images Among them, H and W represent the height and width of the image respectively. First, use image segmentation to and are divided into M blocks of the same size; then, the divided Z, Z t and X t After being connected in sequence, the image block is mapped into a visual embedding block through a linear mapping layer; then, Z, Z t and X t The visual embedding is added to the position embedding to retain the position information between image blocks; finally, the corresponding preprocessed initial template embedding, dynamic template embedding and search image embedding results are expressed as: and Where (P×P) is the number of blocks, M = HW / P 2 ;

[0013] Step 2.2: Preprocess the corner coordinate sequence of the target frame of the historical N frames of the current search image of the input video, and process the coordinate sequence into word embedding; the specific operation is as follows: the search image at the current time t is X t , the corner coordinates of its target box are expressed as: in, and Respectively represent the coordinates of the upper left corner and lower right corner of the target box; therefore, in X t The target frame coordinate sequence of the previous N historical search images is expressed as follows: Currently searching for image X in the input video t When preprocessing the angular coordinate sequence of the target frame of the historical N-frame search image, first, the angular coordinate sequences of the target frames of the N historical frames are cascaded, and the cascaded sequence is expressed as: After that, the command token Start is added to the corner coordinate sequence of the target frame of the cascaded N historical frames to indicate the target coordinates of the current search image. The sequence after adding the command token can be expressed as At this time, the sequence is simplified to Y t-N:t-1 , where (tN:t-1) represents traversing from the first N frames of the current search image to the previous frame; finally, Y t-N:t-1 Get the historical track word embedding of the current search image through a word embedding layer and Each token in is an integer in the shared vocabulary V; where V is in the range [1,n bins ], so |V|=n bins , each word in V corresponds to a learnable embedding.

[0014] Step 3 is as follows:

[0015] Step 3.1: Construct an improved ViT encoder. The improved ViT encoder network structure is as follows: Figure 5As shown in FIG, it consists of 12 encoder layers, and the multi-head attention module in each encoder layer of the original ViT is replaced by a template mask multi-head attention module; compared with the original multi-head attention, the template mask multi-head attention integrates the dynamic template mask module during attention calculation; in addition, a token fusion module is added between the template mask multi-head attention module and the multi-layer perceptron in the improved ViT encoder layer; in the improved ViT encoder layer that integrates the multi-head attention module and the token fusion module with the dynamic template mask module, the initial template embedding Z p , Dynamic Template Embedding and search image embeddings The specific operation is as follows: Z p , and After splicing, it is used as the initial input information of the first layer of the encoder of the improved ViT network, denoted as In the encoder layer of the improved ViT, a layer normalization is first performed to ensure the stability of the output, and then the template mask multi-head attention module is input to capture The correlation information between all embeddings in Z is obtained by using dynamic template mask p and Only with and itself; next, the initial input information Add the output of the template mask multi-head attention module to get a preliminary feature representation Where ε∈[1,12] represents the number of layers of the improved ViT encoder; then, Divided into two parts, namely the preliminary template features and preliminary search for image features Then The input token fusion module performs similar token fusion to reduce redundant background information, and outputs token features after token reduction. and with Splicing After that, the layer is normalized again to ensure the stability of the output and compared with Add together to get the initial features after token fusion Finally, The features are further aggregated through a multi-layer perceptron to obtain an overall feature vector, which is the visual feature of the final output of the ε-th encoder layer. When ε<12, the visual features As the input of the next improved ViT encoder layer, repeat the above steps to obtain the visual feature output, where when ε = 12, the output of the encoder layer of the last improved ViT network is

[0016] Among them, the improved ViT encoder layer uses a token fusion module to extract the X t Preliminary features For all X t The tokens of X are merged into a specific number of tokens; the token fusion is divided into six steps. The first step is to merge all X t Token Split into two equal-sized sets A and B, forming a bipartite graph, where |A| = M A ,|B|=M B ; The second step is to establish an edge between each token in set A and the token in set B with the highest cosine similarity; specifically, the tokens in set A 1≤i≤M A and the middle token of set B 1≤j≤M B The similarity calculation method is:

[0017]

[0018] In the formula, · represents the dot product operation, × represents the cross product operation, and M = M A +M B , and Then they represent the tokens in set A respectively and the tokens in set B Key value, Represents the tokens in set A and the tokens in set B similarity; therefore, the dot product similarity matrix of the above formula (1) is used to calculate the similarity between the key values ​​of the tokens, and then it is used as the similarity between the tokens; the third step, according to formula (1), the r edges with the highest similarity between set A and set B are retained; the fourth step, the 2r tokens in set A and set B connected by these r edges will be merged, and the merging method is to average the token feature values ​​of the 2r tokens; the fifth step, all the remaining tokens in set A and set B are spliced ​​to form the search token feature after token fusion As the output of the token fusion module. Finally, search for regional tokens After feature extraction by the improved ViT encoder, the final number of tokens is reduced to (M-12r);

[0019] Among them, the above token fusion module process is summarized as the following expression:

[0020]

[0021] In the formula, and They represent the output X after the εth layer and (ε-1)th layer of the improved ViT encoder respectively. t Token, TOME represents the token fusion operation performed, and r represents the number of tokens to be merged in each layer of the improved ViT encoder;

[0022] Step 3.2: Construct a feature fusion encoder. The feature fusion encoder consists of three encoder layers, each of which contains a multi-head self-attention layer, a cross-attention layer, and a feedforward network. Among them, the FFN contains a two-layer multilayer perceptron, and the GELU activation layer is inserted after the first linear layer. Layer normalization is performed before each module. The feature fusion encoder is used to fuse Z and Z obtained in step 3.1. t and X t Visual features Perform feature fusion. The specific operations are as follows:

[0023] First, the first encoder layer of the feature fusion encoder transforms the visual features According to Z, Z t and X t The spatial dimension of the tokens is separated to generate three separate tables Z, Z t and X t The visual features of Z , and Subsequently, in each encoder layer of the feature encoder, the visual feature f Z , and Through the multi-head self-attention module, the multi-head self-attention of the three is calculated respectively, and the residual calculation is performed to obtain the self-attention representation of the initial template, dynamic template and search image and The above calculation expression is:

[0024]

[0025] Where LN represents the layer normalization operation, MHSA represents the multi-head self-attention operation, and η∈[1,3] represents the number of encoder layers. When η=1,

[0026] Then, using the obtained self-attention representation and Calculate Z, Z t and X t The cross attention is used, and the residuals are calculated to obtain Z and Z respectively. t and X t Cross-attention representation and The above calculation expression is:

[0027]

[0028] In the formula, CSA represents the cross attention operation; next, we calculate X t For Z and Z t The cross attention is summed and averaged, and the residual is calculated to get X t Token pair Z and Z t Representation of Cross Attention The above calculation expression is:

[0029]

[0030] Finally, the feedforward network is used to refine the feature results generated by the above process and perform residual calculation to obtain Z, Z t and X t The final feature fusion result of the visual features in the first encoder layer and The above calculation expression is:

[0031]

[0032] After that, the output results of the encoder layer are and Input the next encoder layer and repeat the above formula (3)-formula (6) operation. After passing the last encoder layer of the feature fusion encoder, Z and Z are obtained. t and X t Visual features The final output is and

[0033] Step 3.3, build the Transformer decoder. The Transformer decoder consists of 6 decoder layers, each of which contains a masked multi-head attention layer, a cross attention layer and a feedforward network; the feedforward network contains a two-layer multilayer perceptron, and the GELU activation layer is inserted after the first linear layer; layer normalization is performed before each module; the input of the Transformer decoder is the historical trajectory word embedding sequence obtained by preprocessing and Z, Z obtained by the feature fusion encoder t and X t Visual features and Perform the current X t The autoregressive generation of the corner coordinates of the target box in the target box; the specific operation is as follows:

[0034] First, in each decoder layer of the Transformer decoder, a sequence of command tokens Start responsible for providing trajectory suggestions and historical trajectory words as spatiotemporal cues are embedded. The spatial and temporal information in the historical trajectory is captured and transmitted by inputting the masked multi-head attention layer for calculation, and the causal mask in the masked multi-head attention layer is used to ensure that the output of each coordinate value in the sequence only depends on its previous sequence coordinates and historical trajectory information; then, the residual calculation is performed on the obtained attention representation to obtain the historical trajectory word embedding sequence Masked multi-head attention representation of The above calculation expression is:

[0035]

[0036] Where MMHA represents the masked multi-head attention operation, δ∈[1,6] represents the number of layers of the Transformer decoder, and when δ=1,

[0037] Then, using the obtained masked multi-head attention representation After fusion of features Z, Z t and X t Visual features and Cross-attention calculations are performed, and residual calculations are performed to obtain the embedded features of the corner coordinates of the target box generated by the fusion of contextual spatiotemporal information and visual features. The historical trajectory word embedding sequence after layer normalization The and image features are used as the Query value and Key value of the multi-head attention respectively, while the original image features are input into the multi-head attention layer as the Value value of the multi-head attention. The above calculation expression is:

[0038]

[0039] Where f is and The result after splicing, MHA is a multi-head attention operation;

[0040] Next, use the feedforward network to embed the feature results of the corner coordinates of the target box generated by the above process Perform refinement and residual calculation, and output the embedded features of the corner coordinates of the final target box The calculation expression of the above process is:

[0041]

[0042] Finally, the embedded feature output of the corner coordinates of the target box is Continue to repeat the above operation as the input of the next decoder layer until the last decoder layer of the decoder outputs the embedding token feature result of the corner coordinates of the target box Then, the embedding token feature results of the corner coordinates of the target box The token is matched with the words in the shared vocabulary V to find the most likely position value of the token. This process is usually called vocabulary projection, and finally generates X t Corner coordinate sequence of the target box in The above calculation expression is as follows:

[0043]

[0044] Where Softmax represents normalization calculation, h1, h2, h3 and h4 are the embedding vectors of the last four positions in the sequence, b is a learnable bias vector, and W emb is the weight matrix of the word embedding of the vocabulary V; during the projection process, the embedding vector h is passed through the word embedding weight matrix W emb and the bias vector b are converted into an unnormalized score vector, and then the score vector is converted into a probability distribution through the Softmax function. Finally, the words w1, w2, w3 and w4 with the largest probability are selected as and The values ​​of are used to form the search image X t The corner coordinate sequence Y of the target box t , and add it to the historical trajectory set as output, expressed as

[0045] Step 3.4: Construct an online template update mechanism stage integrating spatiotemporal information. The online template update mechanism stage integrating spatiotemporal information consists of a target coordinate confidence calculation module, a target appearance reliability judgment module, and a template clipping module. The online template update mechanism integrating spatiotemporal information first calculates the X obtained in step 3.3 through the target coordinate confidence calculation module. t The sum of the probability of each value in the angular coordinate sequence of the target box is taken as X t The score of the target box Xt , calculated as follows:

[0046]

[0047] If the score is higher, it means that X t The target appearance in the image is relatively clear and is suitable for use as a new template; otherwise, it means that the target may be disturbed by occlusion, loss, or scale change during tracking, resulting in significant changes in its appearance or position;

[0048] Then, the target box score is input into the target appearance reliability judgment module, which sets the score threshold θ and the update interval threshold T u To determine whether the tracking result is suitable as a new Z t ; Next will And (t+1) / / T u =0 of X t Its target frame sequence is input into the template cropping module, and X is converted into t Crop and set to new Z t ;

[0049] Step 3.5: Design the loss function of the token fusion-based autoregressive visual target tracking model. The specific expression is as follows:

[0050]

[0051] In the formula, and They are the cross entropy loss function and SIoU loss function respectively; and λ is the hyperparameter that balances the two loss functions;

[0052] in, The settings are as follows:

[0053]

[0054] Where P(·) is the Softmax probability, Y t-N:t-1 It is used to help predict the target coordinate sequence of the historical N frames of the target coordinate sequence of the current search image, and T represents the length of the entire video sequence;

[0055] SIoU loss further considers X t The ground-truth target box and the output of the token fusion-based autoregressive visual target tracking model X t The vector angle between the predicted boxes is redefined to The settings are as follows:

[0056]

[0057] In the formula, It specifically includes four parts: distance loss Shape loss and intersection and comparison loss.

[0058] Among them, the distance loss The definition of settings is as follows:

[0059]

[0060] In the formula, They are X t The horizontal and vertical coordinates of the real box and the horizontal and vertical coordinates of the predicted box; C w and C h For X t The width and height of the minimum enclosing rectangle of the real box and the predicted box;

[0061] Shape loss The definition of settings is as follows:

[0062]

[0063] In the formula, They are search images X t The width and height of the predicted box and the true box, β controls the degree of attention to the shape loss. In order to avoid paying too much attention to the shape loss and reducing the movement of the predicted box, it is set to 4 here.

[0064] Step 4 is as follows:

[0065] The token fusion-based autoregressive visual target tracking model constructed in step 3 is used as the visual target tracking model, and the invented token fusion-based autoregressive visual target tracking model is trained using the visual target tracking datasets TrackingNet, LaSOT and COCO in step 1 to obtain the final token fusion-based autoregressive visual target tracking model.

[0066] The beneficial effects of the present invention are:

[0067] The method of the present invention is based on the Transformer-based autoregressive visual tracking model, and proposes an autoregressive visual target tracking model called token fusion-based. First, by adding a token fusion module to the ViT backbone network to reduce the interference of redundant background information on visual extraction, the model's ability to extract visual features and spatiotemporal information is improved; secondly, based on the online template update, the historical trajectory is introduced as a spatiotemporal cue to enhance the model's ability to perceive changes in target appearance and position; finally, a dual-template dynamic mask mechanism is proposed and used to prevent the dual templates from interfering with each other during feature extraction without increasing additional computational costs, thereby further enhancing the tracking performance of the model. A large number of experiments have shown that the token fusion-based autoregressive visual target tracking model has achieved competitive results on four tracking datasets, and has shown excellent performance and broad application prospects in visual target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is a flow chart of the method of the present invention;

[0069] Figure 2The process of preprocessing the input image in the method of the present invention;

[0070] Figure 3 It is the process of preprocessing the angular coordinate sequence of the historical target frame of the current search image in the method of the present invention;

[0071] Figure 4 The structure diagram of the autoregressive visual target tracking model based on token fusion constructed by the present invention;

[0072] Figure 5 The improved ViT encoder network structure diagram in the method of the present invention;

[0073] Figure 6 Schematic diagram of a dual-template dynamic masking strategy in the template mask multi-head attention mechanism module in the method of the present invention;

[0074] Figure 7 The operation process of the token fusion module in the improved ViT encoder network in the method of the present invention;

[0075] Figure 8 It is a network structure diagram of feature fusion encoder in the method of the present invention;

[0076] Fig. 9 It is a Transformer decoder network structure diagram in the method of the present invention;

[0077] Fig.10 Flow chart of the online template updating mechanism integrating spatiotemporal information in the method of this invention;

[0078] Fig.11 This is a tracking result diagram of the method of the present invention when the target moves quickly, is partially blocked, and has multiple similar individuals;

[0079] Fig.12 This is a tracking result diagram of the method of the present invention when the target motion is blurred;

[0080] Fig.13 This is a diagram of the tracking result of the method of the present invention when the lighting conditions are poor; DETAILED DESCRIPTION

[0081] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0082] The present invention provides an autoregressive visual target tracking model based on token fusion, such as Figure 1-9 As shown, the specific implementation steps are as follows:

[0083] Step 1: Obtain a public visual target tracking dataset and convert it into a center coordinate format;

[0084] In step 1, the public visual object tracking dataset is TrackingNet, LaSOT, COCO, OTB100 or UAV123 dataset.

[0085] Step 2: Preprocess the initial template of the given video sequence, the online template at the current moment, and the search image to obtain the visual embedding of the initial template, the dynamic template, and the search image; preprocess the corner coordinate sequence of the target box in the previous N historical frames of the current search image of the input video to obtain the historical trajectory word embedding of the current search image;

[0086] Step 2 is as follows:

[0087] Step 2.1: Preprocess the initial template of the given video sequence, the dynamic template at the current moment, and the search image. Figure 2 , get the visual embedding of the initial template, dynamic template and search image, which is convenient for input encoder to extract features. The specific operation is as follows: for the initial template of a given video sequence Dynamic template at the current frame t and search images Among them, H and W represent the height and width of the image respectively. First, use image segmentation (Patch Partition) to and are divided into M blocks of the same size; then, the divided Z, Z t and X t After being connected in sequence, the image block is mapped into a visual embedding block through a linear projection layer; then, Z, Z t and X t The visual embedding is added to the position embedding (PositionEmbedding) to retain the position information between image blocks; finally, the corresponding preprocessed initial template embedding, dynamic template embedding and search image embedding results are expressed as: and Where (P×P) is the number of blocks, M = HW / P 2 ;

[0088] Step 2.2: Preprocess the corner coordinate sequence of the target frame of the current search image of the input video in the history of N frames of search images, such as Figure 3 As shown, it processes the coordinate sequence into word embedding; the specific operation is as follows: the search image at the current time t is X t , the corner coordinates of its target box are expressed as: in, and Respectively represent the coordinates of the upper left corner and lower right corner of the target box; therefore, in X t The target frame coordinate sequence of the previous N historical search images is expressed as follows: Currently searching for image X in the input video t When preprocessing the angular coordinate sequence of the target frame of the historical N-frame search image, first, the angular coordinate sequences of the target frames of the N historical frames are cascaded, and the cascaded sequence is expressed as: After that, the command token Start is added to the corner coordinate sequence of the target frame of the cascaded N historical frames to indicate the target coordinates of the current search image. The sequence after adding the command token can be expressed as At this time, the sequence is simplified to Y t -N:t-1 , where (tN:t-1) represents traversing from the first N frames of the current search image to the previous frame; finally, Y t-N:t-1 Get the historical track word embedding of the current search image through a word embedding layer (Word to Embedding Layer) and Each token in is an integer in the shared vocabulary V; where V is in the range [1,n bins ], so |V|=n bins , each word in V corresponds to a learnable embedding that can be optimized during model training.

[0089] Step 3: Build an autoregressive visual target tracking model based on token fusion and design the loss function.

[0090] In step 3, the token fusion-based autoregressive visual target tracking model is composed of an improved ViT encoder, a feature fusion encoder, a Transformer decoder, and an online template update mechanism that integrates spatiotemporal information. The overall network structure is as follows: Figure 4 As shown in the improved ViT encoder stage, the initial template embedding Z preprocessed in step 2 is input p , Dynamic Template Embedding and search image embeddings Output Z obtained after the improved ViT encoder feature extraction p , and The visual features of Z , and After cascading, it is expressed as: In the feature fusion encoder stage, the visual features f of the dual template and the search image obtained in the improved ViT encoder are Z , and As input; output the visual features of the two templates and the search image obtained after feature fusion by the feature fusion encoder, which are expressed as: and In the Transformer decoder stage, the historical trajectory word embedding obtained in step 2.2 is And the visual features output by the feature fusion encoder and As input, the final X is generated under multi-level guidance t The corner coordinate sequence of the target box It can be expressed as Y t , and add it as output to the historical trajectory set; finally, in the online template update mechanism stage that integrates spatiotemporal information, input X t The Y coordinate of the corner of the target box t ; The output is the new Z t .

[0091] Step 3.1: Construct an improved ViT encoder. The improved ViT encoder network structure is as follows: Figure 5 As shown in FIG, it consists of 12 encoder layers, and the multi-head attention module in each encoder layer of the original ViT is replaced by a template masked multi-head attention module (Template Masked Multi-Head Attention); compared with the original multi-head attention, the template masked multi-head attention integrates the dynamic template mask module in the attention calculation; in addition, a token fusion module (Token Merging Module) is added between the template masked multi-head attention module and the multi-layer perceptron (MLP) of the improved ViT encoder layer; in the improved ViT encoder layer that integrates the multi-head attention module and the token fusion module with the dynamic template mask module, the initial template embedding Z p , Dynamic Template Embedding and search image embeddings The specific operation is as follows: Z p , and After splicing, it is used as the initial input information of the first layer of the encoder of the improved ViT network, denoted as In the encoder layer of the improved ViT, a layer normalization is first performed to ensure the stability of the output, and then the template mask multi-head attention module is input to capture The correlation information between all embeddings in Z is obtained by using dynamic template mask p and Only with and itself, see Figure 6 ; Next, the initial input information Add the output of the template mask multi-head attention module to get a preliminary feature representation Where ε∈[1,12] represents the number of layers of the improved ViT encoder; then, Divided into two parts, namely the preliminary template features and preliminary search for image features Then The input token fusion module performs similar token fusion to reduce redundant background information, and outputs token features after token reduction. and with Splicing After that, the layer is normalized again to ensure the stability of the output and compared with Add together to get the initial features after token fusion Finally, The features are further aggregated through a multi-layer perceptron to obtain an overall feature vector, which is the visual feature of the final output of the ε-th encoder layer. When ε<12, the visual features As the input of the next improved ViT encoder layer, repeat the above steps to obtain the visual feature output, where when ε = 12, the output of the encoder layer of the last improved ViT network is

[0092] Among them, the improved ViT encoder layer uses a token fusion module to extract the X t Preliminary features For all X t The tokens of the tokens are merged into a specific number of tokens; token fusion is divided into six steps, see Figure 7 The first step is to set all X t Token Split into two equal-sized sets A and B, forming a bipartite graph, where |A| = M A ,|B|=M B ; The second step is to establish an edge between each token in set A and the token in set B with the highest cosine similarity; specifically, the tokens in set A 1≤i≤M A and the middle token of set B 1≤j≤M B The similarity calculation method is:

[0093]

[0094] In the formula, · represents the dot product operation, × represents the cross product operation, and M = M A +MB , and Then they represent the tokens in set A respectively and the tokens in set B Key value, Represents the tokens in set A and the tokens in set B similarity; therefore, the dot product similarity matrix of the above formula (1) is used to calculate the similarity between the key values ​​of the tokens, and then it is used as the similarity between the tokens; the third step, according to formula (1), the r edges with the highest similarity between set A and set B are retained; the fourth step, the 2r tokens in set A and set B connected by these r edges will be merged, and the merging method is to average the token feature values ​​of the 2r tokens; the fifth step, all the remaining tokens in set A and set B are spliced ​​to form the search token feature after token fusion As the output of the token fusion module. Finally, search for regional tokens After feature extraction by the improved ViT encoder, the final number of tokens is reduced to (M-12r);

[0095] Among them, the above token fusion module process is summarized as the following expression:

[0096]

[0097] In the formula, and They represent the output X after the εth layer and (ε-1)th layer of the improved ViT encoder respectively. t Token, TOME represents the token fusion operation performed, and r represents the number of tokens to be merged in each layer of the improved ViT encoder;

[0098] Step 3.2: Construct a feature fusion encoder, such as Figure 8 As shown in Figure 2, the feature fusion encoder consists of three encoder layers, each of which contains a multi-head self-attention layer (Multi-Head Self-Attention, MHSA), a cross-attention layer (Cross Self-Attention, CSA) and a feed forward network (Feed Forward Network, FFN), see Figure 7 FFN consists of a two-layer multilayer perceptron (MLP), with a GELU activation layer inserted after the first linear layer; layer normalization is performed before each module; the feature fusion encoder is used to fuse Z and Z obtained in step 3.1. t and X t Visual features Perform feature fusion. The specific operations are as follows:

[0099] First, the first encoder layer of the feature fusion encoder transforms the visual features According to Z, Z t and X t The spatial dimension of the tokens is separated to generate three separate tables Z, Z t and X t The visual features of Z , and Subsequently, in each encoder layer of the feature encoder, the visual feature f Z , and Through the multi-head self-attention module, the multi-head self-attention of the three is calculated respectively, and the residual calculation is performed to obtain the self-attention representation of the initial template, dynamic template and search image and The above calculation expression is:

[0100]

[0101] Where LN represents the layer normalization operation, MHSA represents the multi-head self-attention operation, and η∈[1,3] represents the number of encoder layers. When η=1,

[0102] Then, using the obtained self-attention representation and Calculate Z, Z t and X t The cross attention is used, and the residuals are calculated to obtain Z and Z respectively. t and X t Cross-attention representation and The above calculation expression is:

[0103]

[0104] In the formula, CSA represents the cross attention operation; next, we calculate X t For Z and Z t The cross attention is summed and averaged, and the residual is calculated to get X t Token pair Z and Z t The representation of cross attention The above calculation expression is:

[0105]

[0106] Finally, the feedforward network is used to refine the feature results generated by the above process and perform residual calculation to obtain Z, Z t and X tThe final feature fusion result of the visual features in the first encoder layer and The above calculation expression is:

[0107]

[0108] After that, the output results of the encoder layer are and Input the next encoder layer and repeat the above formula (3)-formula (6) operation. After passing the last encoder layer of the feature fusion encoder, Z and Z are obtained. t and X t Visual features The final output is and

[0109] Step 3.3, build the Transformer decoder, such as Fig. 9 As shown in Figure 1, the Transformer decoder consists of 6 decoder layers, each of which contains a masked multi-head attention layer, a cross attention layer, and a feedforward network. The feedforward network contains a two-layer multi-layer perceptron, and the GELU activation layer is inserted after the first linear layer. Layer normalization is performed before each module. The Transformer decoder structure is shown in Figure 1. Fig. 9 As shown; its input is the historical trajectory word embedding sequence obtained by preprocessing and Z, Z obtained by the feature fusion encoder t and X t Visual features and Perform the current X t The autoregressive generation of the corner coordinates of the target box in the target box; the specific operation is as follows:

[0110] First, in each decoder layer of the Transformer decoder, a sequence of command tokens Start responsible for providing trajectory suggestions and historical trajectory words as spatiotemporal cues are embedded. The spatial and temporal information in the historical trajectory is captured and transmitted by inputting the masked multi-head attention layer for calculation, and the causal mask (Casual Mask) in the masked multi-head attention layer is used to ensure that the output of each coordinate value in the sequence only depends on its previous sequence coordinates and historical trajectory information; then, the residual calculation is performed on the obtained attention representation to obtain the historical trajectory word embedding sequence Masked multi-head attention representation of The above calculation expression is:

[0111]

[0112] Where MMHA represents the masked multi-head attention operation, δ∈[1,6] represents the number of layers of the Transformer decoder, and when δ=1,

[0113] Then, using the obtained masked multi-head attention representation After fusion of features Z, Z t and X t Visual features and Cross-attention calculations are performed, and residual calculations are performed to obtain the embedded features of the corner coordinates of the target box generated by the fusion of contextual spatiotemporal information and visual features. The historical trajectory word embedding sequence after layer normalization The and image features are used as the Query value and Key value of the multi-head attention respectively, while the original image features are input into the multi-head attention layer as the Value value of the multi-head attention. The above calculation expression is:

[0114]

[0115] Where f is and The result after splicing, MHA is a multi-head attention operation;

[0116] Next, use the feedforward network to embed the feature results of the corner coordinates of the target box generated by the above process Perform refinement and residual calculation, and output the embedded features of the corner coordinates of the final target box The calculation expression of the above process is:

[0117]

[0118] Finally, the embedded feature output of the corner coordinates of the target box is Continue to repeat the above operation as the input of the next decoder layer until the last decoder layer of the decoder outputs the embedding token feature result of the corner coordinates of the target box Then, the embedding token feature results of the corner coordinates of the target box The token is matched with the words in the shared vocabulary V to find the most likely position value of the token. This process is usually called vocabulary projection, and finally generates X t Corner coordinate sequence of the target box in The above calculation expression is as follows:

[0119]

[0120] Where Softmax represents normalization calculation, h1, h2, h3 and h4 are the embedding vectors of the last four positions in the sequence, b is a learnable bias vector, and W emb is the weight matrix of the word embedding of the vocabulary V; during the projection process, the embedding vector h is passed through the word embedding weight matrix W emb and the bias vector b are converted into an unnormalized score vector, and then the score vector is converted into a probability distribution through the Softmax function. Finally, the words w1, w2, w3 and w4 with the largest probability are selected as and The values ​​of are used to form the search image X t The corner coordinate sequence Y of the target box in t , and add it to the historical trajectory set as output, expressed as

[0121] Step 3.4: Construct the online template update mechanism stage that integrates spatiotemporal information. The online template update mechanism stage that integrates spatiotemporal information consists of a target coordinate confidence calculation module, a target appearance reliability judgment module, and a template clipping module. The process can be seen in Fig.10 ; The online template update mechanism integrating spatiotemporal information first calculates the X obtained in step 3.3 through the target coordinate confidence calculation module t The sum of the probability of each value in the angular coordinate sequence of the target box is taken as X t The score of the target box The calculation is as follows:

[0122]

[0123] If the score is higher, it means that X t The target appearance in the image is relatively clear and is suitable for use as a new template; otherwise, it means that the target may be disturbed by occlusion, loss, or scale change during tracking, resulting in significant changes in its appearance or position;

[0124] Then, the target box score is input into the target appearance reliability judgment module, which sets the score threshold θ and the update interval threshold T u To determine whether the tracking result is suitable as a new Z t ; Next will And (t+1) / / T u =0 of X t Its target frame sequence is input into the template cropping module, and X is converted into t Crop and set to new Z t ;

[0125] Step 3.5: Design the loss function of the token fusion-based autoregressive visual target tracking model. The specific expression is as follows:

[0126]

[0127] In the formula, and They are the cross entropy loss function and SIoU loss function respectively; and λ is the hyperparameter that balances the two loss functions and is set to 2;

[0128] in, The settings are as follows:

[0129]

[0130] Where P(·) is the Softmax probability, Y t-N:t-1 It is used to help predict the target coordinate sequence of the historical N frames of the target coordinate sequence of the current search image, and T represents the length of the entire video sequence; Use the cross entropy loss function to maximize X based on the previous subsequence, historical trajectory sequence and input t Log-likelihood of target coordinates.

[0131] SIoU loss further considers X t The ground-truth target box and the output of the token fusion-based autoregressive visual target tracking model X t The vector angle between the predicted boxes is redefined to The settings are as follows:

[0132]

[0133] In the formula, It specifically includes four parts: distance loss Shape loss and intersection and comparison loss.

[0134] Among them, the distance loss The definition of settings is as follows:

[0135]

[0136] In the formula, They are X t The horizontal and vertical coordinates of the real box and the horizontal and vertical coordinates of the predicted box; C w and C h For X t The width and height of the minimum enclosing rectangle of the real box and the predicted box;

[0137] Shape loss The definition of settings is as follows:

[0138]

[0139] In the formula, They are search images X t The width and height of the predicted box and the true box, β controls the degree of attention to the shape loss. In order to avoid paying too much attention to the shape loss and reducing the movement of the predicted box, it is set to 4 here.

[0140] Step 4: Use the token fusion-based autoregressive visual target tracking model constructed in step 3 as the visual target tracking model, and use the visual target tracking datasets TrackingNet, LaSOT and COCO in step 1 to train the invented token fusion-based autoregressive visual target tracking model to obtain the final token fusion-based autoregressive visual target tracking model.

[0141] Autoregressive visual target tracking model based on token fusion. In order to verify the effectiveness of the autoregressive visual target tracking model based on token fusion, the autoregressive visual target tracking model based on token fusion obtained in step 4 and the current mainstream target tracking model are tested on the test sets of TrackingNet and LaSOT, as well as the OTB100 and UAV123 datasets. The success rate, accuracy, and normalized accuracy of the above models on different target tracking datasets are evaluated and recorded, thereby verifying the effectiveness and accuracy of the model in target tracking.

[0142] Example 1

[0143] Case analysis and method validation

[0144] In order to verify the effectiveness of the present invention, the present invention is programmed in the PyTorch environment and the model is trained using two NVIDIA Tesla A100 80GB GPUs. The detailed parameter settings of the training model of the present invention are as follows: TemplateSize: [128×128]; Search Size: [256×256]. Among them, the initial encoder is ViT with token fusion added, and is initialized using the pre-trained parameters in MAE. The training data includes the training sets of COCO, LaSOT and TrackingNet. The model is optimized using AdamW, and the learning rate of Backbone is 1×10 -5 , the learning rate of the decoder and other modules is 1×10 -4 , and set the weight decay to 1×10 -4The network was trained for 300 epochs, with 60,000 image pairs randomly selected in each round of training. After 240 rounds of training, the learning rate was reduced by 10 times and the batch size was set to 8.

[0145] The comparison results are shown in Table 1. The performance of the proposed method is evaluated on several benchmark datasets, including TrackingNet, LaSOT, OTB100, and UAV123. On the TrackingNet test set, the proposed method achieves the best results when compared with other trackers with similar input resolution and network depth, achieving 84.7% in success rate (AUC), 84.0% in accuracy (P), and 84.0% in normalized accuracy rate (P). Norm ) and achieved a score of 89.6%.

[0146] Table 1 Comparison of the results of the proposed method with other methods on the TrackingNet dataset

[0147]

[0148] In the LaSOT dataset, its test set consists of 280 video sequences with an average length of 2448 frames. Therefore, the robustness of the model is the key to obtaining a high score on this dataset. As shown in Table 2, compared with other tracking algorithms, this method achieves the best AUC of 70.5% on the LaSOT dataset, and other indicators also reach advanced results.

[0149] Table 2 Comparison of the results of the proposed method with other methods on the LaSOT dataset

[0150]

[0151]

[0152] In the OTB100 dataset, it contains 100 video sequences. As shown in Table 3, the proposed method achieves the best results on this dataset compared with other methods, achieving 72.6% AUC and 95.0% Precision respectively.

[0153] Table 3 Comparison of the results of the proposed method and other methods on the OTB100 dataset

[0154]

[0155] On the UAV123 dataset, which is a dataset containing 123 video sequences shot by drones, it can be seen from Table 4 that the proposed method also achieves competitive results on UAV123, which is better than the recently proposed OSTrack and ARTrack models with the same input size.

[0156] Table 4 Comparison of the results of the proposed method with other methods on the UAV123 dataset

[0157]

[0158] The application results on the tracking task are as follows Figure 11-13 shown. Figure 11-13 The following figure shows the tracking results of the token fusion-based autoregressive visual target tracking model under the conditions of fast moving target, partially blocked target, multiple similar individuals, blurred target motion, and low light in the city at night. Fig.11 As can be seen, thanks to the continuous learning and capturing of the appearance and position of the target that changes over time by the online updating of the template that incorporates spatiotemporal information, the method of the present invention can continuously and accurately track the target that moves quickly, is partially occluded, and has multiple similar individuals in the background. Fig.12 The tracking results of the method for the blurred target posture in motion are shown. Thanks to the addition of the token fusion module and the dynamic template mask mechanism, the model's ability to suppress background interference and feature extraction learning is significantly enhanced. Therefore, even in the case of blurred images, the model of the present invention can still show excellent tracking performance. Fig.13 The paper selects a road scene in the city at night, showing that even in poor lighting conditions, the method can still stably track the target. This proves the effectiveness and accuracy of the method in complex lighting and urban environments.

[0159] The method of the present invention is based on an autoregressive visual target tracking model with token fusion. First, by adopting an autoregressive network architecture based on an encoder-decoder, the complex head network and complicated training optimization problems are avoided, and the tracking process is simplified. Secondly, token fusion and a dual-template mask mechanism are added to the backbone network to effectively improve the ability to extract the visual features of the target; finally, an online template update mechanism that integrates spatiotemporal information is added, so that the model can fully perceive the changes in the appearance and spatial position of the target over time, further improving the tracking ability of the target. Experimental results show that compared with other advanced methods on multiple data sets, the present invention has improved various evaluation indicators. In TrackingNet and OTB100, it has achieved the most advanced results compared with previous methods. The problem of insufficient model feature extraction capability caused by insufficient utilization of spatiotemporal information and redundant background information in the search area in existing methods is solved. In the actual tracking process, the success rate and performance of the visual target tracking task are significantly improved, proving that the present method is feasible.

[0160] Example 2

[0161] The autoregressive visual target tracking algorithm based on token fusion includes the following steps:

[0162] Step 1: Obtain a public visual target tracking dataset and convert the dataset into a central coordinate format; Step 2: Preprocess the initial template of a given video sequence, the online template at the current moment, and the search image to obtain the visual embedding of the initial template, dynamic template, and search image; Preprocess the angular coordinate sequence of the target box in the previous N historical frames of the current search image of the input video to obtain the historical trajectory word embedding of the current search image; Step 3: Construct an autoregressive visual target tracking model based on token fusion and design a loss function; Step 4: Use the visual target tracking dataset in step 1 to train the autoregressive visual target tracking model based on token fusion to obtain the final autoregressive visual target tracking model based on token fusion.

[0163] Example 3

[0164] The autoregressive visual target tracking algorithm based on token fusion includes the following steps:

[0165] Step 1: Obtain a public visual target tracking dataset and convert it into a central coordinate format; in step 1, the public visual target tracking dataset is TrackingNet, LaSOT, COCO, OTB100 or UAV123 dataset. Step 2: Preprocess the initial template of a given video sequence, the online template at the current moment, and the search image to obtain the visual embedding of the initial template, dynamic template, and search image; preprocess the angular coordinate sequence of the target box in the previous N historical frames of the current search image of the input video to obtain the historical trajectory word embedding of the current search image; Step 3: Construct an autoregressive visual target tracking model based on token fusion and design a loss function; Step 4: Use the visual target tracking dataset in step 1 to train the autoregressive visual target tracking model based on token fusion to obtain the final autoregressive visual target tracking model based on token fusion.

Claims

1. An autoregressive visual target tracking algorithm based on token fusion, characterized in that: The specific steps include: Step 1: Obtain a public visual target tracking dataset and convert it into a center coordinate format; Step 2: preprocessing the initial template of the given video sequence, the online template at the current moment, and the search image to obtain the visual embedding of the initial template, the dynamic template, and the search image; Preprocess the corner coordinate sequence of the target box in the previous N historical frames of the current search image of the input video to obtain the historical trajectory word embedding of the current search image; Step 3: Build an autoregressive visual target tracking model based on token fusion and design a loss function; Step 4: Use the visual target tracking dataset in step 1 to train the token fusion-based autoregressive visual target tracking model to obtain the final token fusion-based autoregressive visual target tracking model.

2. The token fusion-based autoregressive visual target tracking algorithm according to claim 1 is characterized in that: In step 1, the public visual object tracking dataset is TrackingNet, LaSOT, COCO, OTB100 or UAV123 dataset.

3. The token fusion-based autoregressive visual target tracking algorithm according to claim 1 is characterized in that: Step 2 is as follows: Step 2.1: Preprocess the initial template of the given video sequence, the dynamic template at the current moment, and the search image to obtain the visual embedding of the initial template, dynamic template, and search image, so as to facilitate the input encoder for feature extraction. The specific operation is as follows: Dynamic template at the current frame t and search images Among them, H and W represent the height and width of the image respectively. First, use image segmentation to and are divided into M blocks of the same size; then, the divided Z, Z t and X t After being connected in sequence, the image block is mapped into a visual embedding block through a linear mapping layer; then, Z, Z t and X t The visual embedding is added to the position embedding to retain the position information between image blocks; finally, the corresponding preprocessed initial template embedding, dynamic template embedding and search image embedding results are expressed as: and Where (P×P) is the number of blocks, M = HW / P 2 ; Step 2.2: Preprocess the corner coordinate sequence of the target frame of the historical N frames of the current search image of the input video, and process the coordinate sequence into word embedding; the specific operation is as follows: the search image at the current time t is X t , the corner coordinates of its target box are expressed as: in, and Respectively represent the coordinates of the upper left corner and lower right corner of the target box; therefore, in X t The target frame coordinate sequence of the previous N historical search images is expressed as follows: In the current search image X of the input video t When preprocessing the angular coordinate sequence of the target frame of the historical N-frame search image, first, the angular coordinate sequences of the target frames of the N historical frames are cascaded, and the cascaded sequence is expressed as: After that, the command token Start is added to the corner coordinate sequence of the target frame of the cascaded N historical frames to indicate the target coordinates of the current search image. The sequence after adding the command token can be expressed as At this time, the sequence is simplified to Y t-N:t-1 , where (tN:t-1) represents traversing from the first N frames of the current search image to the previous frame; finally, Y t-N:t-1 Get the historical track word embedding of the current search image through a word embedding layer and Each token in is an integer in the shared vocabulary V; where V is in the range [1,n bins ], so V|=n bins , each word in V corresponds to a learnable embedding.

4. The token fusion-based autoregressive visual target tracking algorithm according to claim 1 is characterized in that: Step 3 is as follows: Step 3.1, build an improved ViT encoder. The improved ViT encoder network structure consists of 12 encoder layers. The multi-head attention module in each encoder layer of the original ViT is replaced by a template mask multi-head attention module; Compared with the original multi-head attention, the template mask multi-head attention integrates the dynamic template mask module during attention calculation; In addition, a token fusion module is added between the template mask multi-head attention module and the multi-layer perceptron in the improved ViT encoder layer; in the improved ViT encoder layer that integrates the multi-head attention module and the token fusion module with the dynamic template mask module, the initial template embedding Z p , Dynamic Template Embedding and search image embeddings The specific operation is as follows: Z p , and After splicing, it is used as the initial input information of the first layer of the encoder of the improved ViT network, denoted as In the encoder layer of the improved ViT, a layer normalization is first performed to ensure the stability of the output, and then the template mask multi-head attention module is input to capture The correlation information between all embeddings in Z is obtained by using dynamic template mask p and Only with and itself; next, the initial input information Add the output of the template mask multi-head attention module to get a preliminary feature representation Where ε∈[1,12] represents the number of layers of the improved ViT encoder; then, Divided into two parts, namely the preliminary template features and preliminary search for image features Then The input token fusion module performs similar token fusion to reduce redundant background information, and outputs token features after token reduction. and with Splicing After that, the layer is normalized again to ensure the stability of the output and compared with Add together to get the initial features after token fusion Finally, The features are further aggregated through a multi-layer perceptron to obtain an overall feature vector, which is the visual feature of the final output of the ε-th encoder layer. When ε<12, the visual features As the input of the next improved ViT encoder layer, repeat the above steps to obtain the visual feature output, where when ε = 12, the output of the encoder layer of the last improved ViT network is Among them, the improved ViT encoder layer uses a token fusion module to extract the X t Preliminary features For all X t The tokens of X are merged into a specific number of tokens; the token fusion is divided into six steps. The first step is to merge all X t Token Split into two equal-sized sets A and B, forming a bipartite graph, where |A| = M A ,|B|=M B ; The second step is to establish an edge between each token in set A and the token in set B with the highest cosine similarity; specifically, the tokens in set A 1≤i≤M A and the middle token of set B 1≤j≤M B The similarity calculation method is: In the formula, · represents the dot product operation, × represents the cross product operation, and M = M A +M B , and Then they represent the tokens in set A respectively and the tokens in set B Key value, Represents the tokens in set A and the tokens in set B similarity; therefore, the dot product similarity matrix of the above formula (1) is used to calculate the similarity between the key values ​​of the tokens, and then it is used as the similarity between the tokens; the third step, according to formula (1), the r edges with the highest similarity between set A and set B are retained; the fourth step, the 2r tokens in set A and set B connected by these r edges will be merged, and the merging method is to average the token feature values ​​of the 2r tokens; the fifth step, all the remaining tokens in set A and set B are spliced ​​to form the search token feature after token fusion As the output of the token fusion module; finally, search for regional tokens After feature extraction by the improved ViT encoder, the final number of tokens is reduced to (M-12r); Among them, the above token fusion module process is summarized as the following expression: In the formula, and They represent the output X after the εth layer and (ε-1)th layer of the improved ViT encoder respectively. t Token, TOME represents the token fusion operation performed, and r represents the number of tokens to be merged in each layer of the improved ViT encoder; Step 3.2: Construct a feature fusion encoder. The feature fusion encoder consists of three encoder layers, each of which contains a multi-head self-attention layer, a cross-attention layer, and a feedforward network. Among them, the FFN contains a two-layer multilayer perceptron, and the GELU activation layer is inserted after the first linear layer. Layer normalization is performed before each module. The feature fusion encoder is used to fuse Z and Z obtained in step 3.

1. t and X t Visual features Perform feature fusion. The specific operations are as follows: First, the first encoder layer of the feature fusion encoder transforms the visual features According to Z, Z t and X t The spatial dimension of the tokens is separated to generate three separate tables Z, Z t and X t The visual features of Z , and Subsequently, in each encoder layer of the feature encoder, the visual feature f Z , and Through the multi-head self-attention module, the multi-head self-attention of the three is calculated respectively, and the residual calculation is performed to obtain the self-attention representation of the initial template, dynamic template and search image and The above calculation expression is: Where LN represents the layer normalization operation, MHSA represents the multi-head self-attention operation, and η∈[1,3] represents the number of encoder layers. When η=1, Then, using the obtained self-attention representation and Calculate Z, Z t and X t The cross attention is used, and the residuals are calculated to obtain Z and Z respectively. t and X t Cross-attention representation and The above calculation expression is: In the formula, CSA represents the cross attention operation; next, we calculate X t For Z and Z t The cross attention is summed and averaged, and the residual is calculated to get X t Token pair Z and Z t Representation of Cross Attention The above calculation expression is: Finally, the feedforward network is used to refine the feature results generated by the above process and perform residual calculation to obtain Z, Z t and X t The final feature fusion result of the visual features in the first encoder layer and The above calculation expression is: After that, the output results of the encoder layer are and Input the next encoder layer and repeat the above formula (3)-formula (6) operation. After passing the last encoder layer of the feature fusion encoder, Z and Z are obtained. t and X t Visual features The final output is and Step 3.3, build the Transformer decoder. The Transformer decoder consists of 6 decoder layers, each of which contains a masked multi-head attention layer, a cross attention layer and a feedforward network; the feedforward network contains a two-layer multilayer perceptron, and the GELU activation layer is inserted after the first linear layer; layer normalization is performed before each module; the input of the Transformer decoder is the historical trajectory word embedding sequence obtained by preprocessing and Z, Z obtained by the feature fusion encoder t and X t Visual features and Perform the current X t The autoregressive generation of the corner coordinates of the target box in the target box; the specific operation is as follows: First, in each decoder layer of the Transformer decoder, a sequence of command tokens Start responsible for providing trajectory suggestions and historical trajectory words as spatiotemporal cues are embedded. The spatial and temporal information in the historical trajectory is captured and transmitted by inputting the masked multi-head attention layer for calculation, and the causal mask in the masked multi-head attention layer is used to ensure that the output of each coordinate value in the sequence only depends on its previous sequence coordinates and historical trajectory information; then, the residual calculation is performed on the obtained attention representation to obtain the historical trajectory word embedding sequence Masked multi-head attention representation of The above calculation expression is: Where MMHA represents the masked multi-head attention operation, δ∈[1,6] represents the number of layers of the Transformer decoder, and when δ=1, Then, using the obtained masked multi-head attention representation After fusion of features Z, Z t and X t Visual features and Cross-attention calculations are performed, and residual calculations are performed to obtain the embedded features of the corner coordinates of the target box generated by the fusion of contextual spatiotemporal information and visual features. The historical trajectory word embedding sequence after layer normalization The and image features are used as the Query value and Key value of the multi-head attention respectively, while the original image features are input into the multi-head attention layer as the Value value of the multi-head attention. The above calculation expression is: Where f is and The result after splicing, MHA is a multi-head attention operation; Next, use the feedforward network to embed the feature results of the corner coordinates of the target box generated by the above process Perform refinement and residual calculation, and output the embedded features of the corner coordinates of the final target box The calculation expression of the above process is: Finally, the embedded feature output of the corner coordinates of the target box is Continue to repeat the above operation as the input of the next decoder layer until the last decoder layer of the decoder outputs the embedding token feature result of the corner coordinates of the target box Then, the embedding token feature results of the corner coordinates of the target box The token is matched with the words in the shared vocabulary V to find the most likely position value of the token. This process is usually called vocabulary projection, and finally generates X t Corner coordinate sequence of the target box in The above calculation expression is as follows: Where Softmax represents normalization calculation, h1, h2, h3 and h4 are the embedding vectors of the last four positions in the sequence, b is a learnable bias vector, and W emb is the weight matrix of the word embedding of the vocabulary V; during the projection process, the embedding vector h is passed through the word embedding weight matrix W emb and the bias vector b are converted into an unnormalized score vector, and then the score vector is converted into a probability distribution through the Softmax function; finally, the words w1, w2, w3 and w4 with the largest probability are selected as and The values ​​of are used to form the search image X t The corner coordinate sequence Y of the target box in t , and add it to the historical trajectory set as output, expressed as Step 3.4: Construct an online template update mechanism stage integrating spatiotemporal information. The online template update mechanism stage integrating spatiotemporal information consists of a target coordinate confidence calculation module, a target appearance reliability judgment module, and a template clipping module. The online template update mechanism integrating spatiotemporal information first calculates the X obtained in step 3.3 through the target coordinate confidence calculation module. t The sum of the probability of each value in the angular coordinate sequence of the target box is taken as X t The score of the target box The calculation is as follows: If the score is higher, it means that X t The target appearance in the image is relatively clear and is suitable for use as a new template; otherwise, it means that the target may be disturbed by occlusion, loss, or scale change during tracking, resulting in significant changes in its appearance or position; Then, the target box score is input into the target appearance reliability judgment module, which sets the score threshold θ and the update interval threshold T u To determine whether the tracking result is suitable as a new Z t ; Next will And (t+1) / / T u =0 of X t Its target frame sequence is input into the template cropping module, and X is converted into t Crop and set to new Z t ; Step 3.5: Design the loss function of the token fusion-based autoregressive visual target tracking model. The specific expression is as follows: In the formula, and They are the cross entropy loss function and SIoU loss function respectively; and λ is the hyperparameter that balances the two loss functions; in, The settings are as follows: Where P(·) is the Softmax probability, Y t-N:t-1 It is used to help predict the target coordinate sequence of the historical N frames of the target coordinate sequence of the current search image, and T represents the length of the entire video sequence; SIoU loss further considers X t The ground-truth target box and the output of the token fusion-based autoregressive visual target tracking model X t The vector angle between the predicted boxes is redefined to The settings are as follows: In the formula, It specifically includes four parts: distance loss Shape loss and intersection and comparison loss; among them, distance loss The definition of settings is as follows: In the formula, They are X t The horizontal and vertical coordinates of the real box and the horizontal and vertical coordinates of the predicted box; C w and C h For X t The width and height of the minimum enclosing rectangle of the real box and the predicted box; Shape loss The definition of settings is as follows: In the formula, They are search images X t The width and height of the predicted box and the real box, β controls the degree of attention to the shape loss. In order to avoid paying too much attention to the shape loss and reducing the movement of the predicted box, it is set to 4 here.

5. The token fusion-based autoregressive visual target tracking algorithm according to claim 1, characterized in that: Step 4 is as follows: The token fusion-based autoregressive visual target tracking model constructed in step 3 is used as the visual target tracking model, and the invented token fusion-based autoregressive visual target tracking model is trained using the visual target tracking datasets TrackingNet, LaSOT and COCO in step 1 to obtain the final token fusion-based autoregressive visual target tracking model.

Images