Transform visual target tracking method based on adaptive mark division

Abstract

The invention provides a Transform visual target tracking method based on adaptive mark division, and relates to the technical field of visual target tracking, the method converts a target tracking problem into a sequence generation problem, firstly, a Transform-based encoder-decoder architecture is used, an additional header network is eliminated, and a tracking architecture is simplified; secondly, a self-adaptive mark division module is added into an encoder, so that search marks and template marks are subjected to optimal cross relation modeling, and the target and background distinguishing capability of the model is improved; the method comprises the following steps: constructing a network model; constructing a self-adaptive mark division module, and integrating the self-adaptive mark division module into an encoder in the network model; training the network model according to the loss function; and tracking a target in the video by using the trained network model. The method provided by the invention has higher accuracy and robustness when facing complex scenes with similar object interference, shielding, scale change and the like.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning, and in particular to a Transformer visual target tracking method based on adaptive labeling. Background Technology

[0002] Visual object tracking is an important research area in computer vision, aiming to locate and track the position and trajectory of specific targets in real time within continuous video sequences. With the rapid development of artificial intelligence technology, visual object tracking has shown broad application prospects in many fields such as autonomous driving, intelligent monitoring, and human-computer interaction.

[0003] Currently, mainstream visual object tracking methods are mainly divided into two-stream and single-stream methods. These methods typically treat the tracking problem as a template matching problem, and mainly consist of three stages: (i) extracting features from the search image and the template image; (ii) performing feature matching and fusion using convolution or attention mechanisms; and (iii) using a custom head network to perform operations such as center point localization and scale estimation. Compared to two-stream methods, single-stream methods have a simpler network structure and higher computational efficiency. Single-stream methods utilize attention mechanisms to jointly extract visual features and model the relationship between the template region and the search region. However, using a custom head network increases the complexity of the network framework and requires additional training.

[0004] Cui and Ye et al. proposed that early feature interactions between the template and the search region can help generate more target-discriminative features. However, not all parts of the search region are suitable for interaction with the template. In reality, the search region often contains a lot of interfering information, such as objects that resemble the tracked target in appearance. This can cause these interfering objects in the search region to be misclassified as interaction objects, leading to incorrect cross-relationship modeling and reducing tracking accuracy.

[0005] In recent years, researchers have leveraged the high flexibility of attention mechanisms to propose various label organization strategies to optimize Transformer performance. Fu et al. used a sparse attention mechanism to focus on and integrate only the most similar label features, thereby improving the distinction between foreground and background. Ye et al. used an early candidate elimination module to remove irrelevant labels in the search region to improve computational efficiency. Gao et al. used a lightweight prediction module to classify input labels, achieving flexible relation modeling. Although these methods have achieved excellent performance, they are structurally complex, requiring the design of dedicated head networks for subsequent target localization and other operations, and necessitating the introduction of multiple loss functions.

[0006] Existing single-target tracking algorithms typically suffer from the following three problems: 1) A customized header network needs to be designed, increasing the complexity of the framework; 2) Introducing multiple loss functions requires adjusting more hyperparameters, increasing the difficulty and uncertainty of model tuning; 3) Using template features to interact with the entire search area is not optimal. Similar interfering objects within the search area may be misidentified as interactive objects, reducing the accuracy of tracking.

[0007] To address this, the present invention proposes a Transformer visual target tracking method based on adaptive label partitioning. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a Transformer-based visual target tracking method based on adaptive label partitioning. This method transforms the target tracking problem into a sequence generation problem. First, it employs a Transformer-based encoder-decoder architecture, eliminating the need for an additional head network and simplifying the tracking architecture. Second, an adaptive label partitioning module is added to the encoder, enabling optimal cross-relationship modeling between search labels and template labels, thus improving the model's ability to distinguish between the target and the background. Experiments show that the proposed algorithm can complete tracking tasks in complex scenes on standard tracking datasets with challenges such as severe occlusion, interference from similar objects, and scale variations.

[0009] A Transformer-based visual target tracking method based on adaptive label partitioning includes the following steps: Step 1: Construct the network model; The network model includes a linear projection layer, an encoder, and a decoder. The linear projection layer converts image patches into visual embeddings, which are then input into the encoder. The encoder extracts visual features, and the decoder generates bounding box values ​​for the target. The encoder includes an adaptive label partitioning module, a parallel attention calculation module, and a feedforward neural network layer; the adaptive label partitioning module is used to classify search labels; the parallel attention calculation module is used to perform parallel attention calculations on labels of different categories; the feedforward neural network layer is used to generate embeddings for the next encoder block; The decoder includes a word-to-embedding layer, a masked multi-head attention layer, a multi-head cross-attention layer, a feedforward neural network layer, and an embedding-to-word layer. The word-to-embedding layer converts discrete coordinates into continuous word embeddings. The masked multi-head attention layer restricts the output of each sequence element to depend only on the previous element. The multi-head cross-attention layer fuses the output of the masked multi-head attention layer with the encoder's output. The feedforward neural network layer generates embeddings for the next decoder block. The embedding-to-word layer converts the decoder's output back into discrete coordinates to generate the final prediction result. Step 2: Construct an adaptive label partitioning module and integrate it into the encoder in the network model; Step 2.1: Aggregate all template tags through global max pooling to generate an overall representation for target perception; Step 2.2: Concatenate the target-aware representation with the features of each search tag; Step 2.3: Process the concatenated features using a lightweight multilayer perceptron to predict whether each search tag belongs to G. S and G A The probability of a category is calculated using the following formula: Among them, G S G represents the target marker within the search area. A This indicates the background marker within the search area. G z and Represent the template tag embedding and the i-th search tag embedding, respectively, π i,0 and π i,1 These represent the i-th search tag belonging to G. S Category and G A The probability of the category. Based on the predicted value π. i,0 and π i,1 Choose the category with the higher probability; Step 3: Based on the loss function, maximize the log-likelihood between the generated target sequence and the actual target sequence, and train the network model. The loss function is as follows: Where P(·) represents the probability of softmax, x and z represent the search image and template image respectively, and s j Let s represent the target sequence for the current prediction, j represent the position of the currently generated token in the sequence, and s represent the position of the token in the sequence. ＜j Let j represent the target sequence preceding j, and L represent the length of the target sequence. Step 3.1: Divide the search image and template image into image blocks, then convert the image blocks into visual embeddings through linear projection and input them into the encoder; Step 3.2: Use the adaptive tag partitioning module to divide the search tags into G. S and G A Two categories, G T Mark all templates; Step 3.3: Model the cross-relationship of the three label categories through three parallel attention calculations and output visual features; Step 3.3 specifically involves: Step 3.3.1: Convert the probability π into a one-hot tensor Step 3.3.2: Define a one-hot tensor Category assignment for template tags, Q z Set all values ​​in the first column to 1, and set the other two columns to 0. Step 3.3.3: Obtain the value by adding a column of zeros before Q. Category assignment for search tags; Step 3.3.4: Utilize the merged... Constructing attention masks The value of each element is M. i,j Calculate using the following formula: Step 3.3.5: Multiply the attention mask matrix M element-wise with the original attention weight matrix, and set the attention weights of the interaction positions that are prohibited by the mask to zero; Step 3.3.6: Multiply the masked attention weight matrix with the value matrix to obtain the final output; Step 3.4: Convert the target bounding box into a series of discrete tokens and input them into the word to embedding layer; Step 3.5: Add positional embedding to the word to embedding layer, and input the fused embedding to the mask multi-head attention layer of the decoder; Step 3.6: The word embeddings output from the masked multi-head attention layer and the visual features output from the encoder are input together into the multi-head cross-attention layer of the decoder; Step 3.7: Using the embedding to word layer, the decoder output is converted back to discrete coordinates to generate the final target boundary coordinates; Step 4: Use the trained network model to track the target in the video and obtain the tracking results.

[0010] The beneficial effects of adopting the above technical solution are as follows: This invention proposes a Transformer-based visual target tracking method based on adaptive label partitioning. Addressing the high complexity of existing tracking frameworks, it employs a Transformer-based encoder-decoder architecture, eliminating the need for a customized head network and simplifying the tracking framework. To address the issue of redundant loss functions in existing tracking methods requiring more hyperparameter adjustments and increasing the difficulty and uncertainty of model tuning, this invention uses only cross-entropy loss to maximize the log-likelihood between the generated target sequence and the actual target sequence, without requiring other loss functions. To address the problem of similar interfering objects within the search region being misclassified as interactive objects, thus weakening the model's ability to distinguish between the target and background, an adaptive label partitioning module is introduced. This module models the optimal cross-relationship between search labels and template labels, improving the model's ability to distinguish between the target and background. Attached Figure Description

[0011] Figure 1 This is a structural diagram of the Transformer visual target tracking method based on adaptive label partitioning proposed in this invention. Detailed Implementation

[0012] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0013] The experiments conducted in this embodiment of the invention were implemented using the PyTorch framework on a CPU with a clock speed of 3GHz, memory of 32GB, and GPU of GeForce RTX4090.

[0014] A Transformer-based visual target tracking method based on adaptive label partitioning, such as Figure 1 As shown, it includes the following steps: Step 1: Construct the network model; The network model includes a linear projection layer, an encoder, and a decoder. The linear projection layer converts image patches into visual embeddings, which are then input into the encoder. The encoder extracts visual features, and the decoder generates bounding box values ​​for the target. The encoder includes an adaptive label partitioning module, a parallel attention calculation module, and a feedforward neural network layer; the adaptive label partitioning module is used to classify search labels; the parallel attention calculation module is used to perform parallel attention calculations on labels of different categories; the feedforward neural network layer is used to generate embeddings for the next encoder block; The decoder includes a word-to-embedding layer, a masked multi-head attention layer, a multi-head cross-attention layer, a feedforward neural network layer, and an embedding-to-word layer. The word-to-embedding layer converts discrete coordinates into continuous word embeddings. The masked multi-head attention layer restricts the output of each sequence element to depend only on the previous element. The multi-head cross-attention layer fuses the output of the masked multi-head attention layer with the encoder's output. The feedforward neural network layer generates embeddings for the next decoder block. The embedding-to-word layer converts the decoder's output back into discrete coordinates to generate the final prediction result. Step 2: Construct an adaptive label partitioning module and integrate it into the encoder in the network model; Step 2.1: Aggregate all template tags through global max pooling to generate an overall representation for target perception; Step 2.2: Concatenate the target-aware representation with the features of each search tag; Step 2.3: Process the concatenated features using a lightweight multilayer perceptron to predict whether each search tag belongs to G. S and G A The probability of a category is calculated using the following formula: Among them, G z and Represent the template tag embedding and the i-th search tag embedding, respectively, π i,0 and π i,1 These represent the i-th search tag belonging to G. S Category and G A The probability of the category. Based on the predicted value π. i,0 and π i,1 Choose the category with the higher probability; Step 3: Based on the loss function, maximize the log-likelihood between the generated target sequence and the actual target sequence, and train the network model. The loss function is as follows: Where P(·) represents the probability of softmax, x and z represent the search image and template image respectively, and s j Let s represent the target sequence for the current prediction, j represent the position of the currently generated token in the sequence, and s represent the position of the token in the sequence. ＜j Let j represent the target sequence preceding j, and L represent the length of the target sequence. Step 3.1: Divide the search image and template image into image blocks, then convert the image blocks into visual embeddings through linear projection and input them into the encoder; Step 3.2: Use the adaptive tag partitioning module to divide the search tags into G. S and G A Two categories, G T Mark all templates; Step 3.3: Model the cross-relationship of the three label categories through three parallel attention calculations and output visual features; Step 3.3 specifically involves: Step 3.3.1: Convert the probability π into a one-hot tensor Step 3.3.2: Define a one-hot tensor Category assignment for template tags, Q z Set all values ​​in the first column to 1, and set the other two columns to 0. Step 3.3.3: Obtain the value by adding a column of zeros before Q. Category assignment for search tags; Step 3.3.4: Utilize the merged... Constructing attention masks The value of each element is M. i,j Calculate using the following formula: Step 3.3.5: Multiply the attention mask matrix M element-wise with the original attention weight matrix, and set the attention weights of the interaction positions that are prohibited by the mask to zero; Step 3.3.6: Multiply the masked attention weight matrix with the value matrix to obtain the final output; Step 3.4: Convert the target bounding box into a series of discrete tokens and input them into the word to embedding layer; Step 3.5: Add positional embedding to the word to embedding layer, and input the fused embedding to the mask multi-head attention layer of the decoder; Step 3.6: The word embeddings output from the masked multi-head attention layer and the visual features output from the encoder are input together into the multi-head cross-attention layer of the decoder; Step 3.7: Using the embedding to word layer, the decoder output is converted back to discrete coordinates to generate the final target boundary coordinates; Step 4: Use the trained network model to track the target in the video and obtain the tracking results.

[0015] In this embodiment, the resolutions of the search image and the template image are set to 256×256 and 128×128, respectively. The search image and the template image are divided into image blocks, and then the image blocks are converted into visual embeddings through linear projection and input into the encoder. The adaptive labeling module divides the input labels into G. T G A and G S Three categories, including G T It can aggregate G T and G A The tagging information in G S It can aggregate G S and G A The tagging information in G A It can aggregate information from all the labels, and finally, calculate the output visual features through three parallel attention operations.

[0016] To provide target-related contextual information to the search tags, all template tags are first aggregated using a global max-pooling operation to generate a holistic target-aware representation. Then, the target-aware representation is concatenated with the features of each search tag. Finally, a lightweight multilayer perceptron processes the concatenated features to predict whether each search tag belongs to G. S and G A The probability of the category.

[0017] To achieve efficient parallel attention computation, an attention masking strategy is designed. First, the probability π is converted into a one-hot tensor. Then define two one-hot tensors: Category assignment for template tags Category assignment for search tags. In Q z In the diagram, the first column is set to all values ​​of 1, and the other two columns are set to 0, indicating that all template tags are divided into G. T Q is obtained by adding a column of zero values ​​before Q. x This ensures that search tags are not classified as G. T Finally, using the merged... Constructing attention masks By multiplying the attention mask matrix M element-wise with the original attention weight matrix, the attention weights at the masked-off interaction positions are set to zero. This effectively merges the three independent attention calculations into a single calculation, enabling parallel attention computation.

[0018] Before the input decoder, the target bounding box is discretized. Each consecutive coordinate is uniformly discretized into integers between 1 and n, and mapped to a unified vocabulary V. These integers are considered as words in vocabulary V. The final input sequence is [start, x...].min ,y min ,x max ,y max The target sequence is [x] min ,y min ,x max ,y max [,end], where start and end are used to control the start and end of the sequence generation task, respectively. min ,y min [x] indicates the bottom left corner of the target. max ,y max The symbol [] indicates the top-right corner of the target. Masked multi-head attention limits the model to focus only on preceding information when generating each marker, and prevents it from considering subsequent information.

[0019] The decoder output is converted into the target bounding box coordinates using a multilayer perceptron with softmax. Based on the vocabulary V, the coordinates are converted back into discrete words to obtain the target bounding box coordinates.

[0020] To verify the accuracy and effectiveness of the proposed method in complex scenarios, this invention conducts comparative experiments with other mainstream target tracking methods on six datasets: LaSOT, GOT-10k, TrackingNet, UAV123, NFS, and TNL2K.

[0021] LaSOT is a large-scale benchmark dataset designed specifically for single-object tracking tasks, containing 1400 video sequences, each with an average length of approximately 2500 frames, totaling over 1.5 million frames. This dataset covers various object categories and complex scenes, involving challenges such as lighting variations, object occlusion, and rapid movement, and is widely used to evaluate the robustness and accuracy of tracking algorithms. GOT-10k is a large-scale benchmark dataset for general object tracking, containing over 10,000 video frame sequences, totaling over 1.5 million frames. This dataset covers a rich variety of object types and scenes, including various motion states and complex challenges such as occlusion, making it significant for evaluating the generalization performance of tracking algorithms. TrackingNet is one of the largest short-term object tracking benchmarks currently available. This dataset is divided into training and testing subsets, containing over 30,000 video sequences covering diverse scenes and object categories. UAV123 is a long-term tracking benchmark containing 123 video sequences from the perspective of drones, covering various complex environments and challenges such as rapid movement, changing viewpoints, and occlusion. NFS is a tracking benchmark consisting of 100 fast-moving video sequences, covering a variety of high-speed motion and complex tracking scenarios. TNL2K is a large-scale dataset with natural language descriptions, containing 700 test sequences. Each sequence not only has visual data but also a natural language description, making it a challenging tracking benchmark.

[0022] The comparison results are shown in Table 1, where Succ represents the success rate, and P... norm P and represent normalization precision and accuracy, respectively; AO represents the average value of the overlapping region across all frames; SR 0.5 and SR 0.75 The success rates were calculated with IoU thresholds of 0.5 and 0.75. Comparisons between the proposed method and mainstream methods on the Lasot, GOT-10k, and TrackingNet datasets show that the proposed method achieves superior performance. Specifically, on the Lasot dataset, the proposed method achieved a success rate of 72.6%, a 1.0% improvement over ARTrackV2. Improvements were also observed on the GOT-10K and TrackingNet datasets. Table 1: Test Results of Mainstream Tracking Methods on Lasot, GOT-10k, and TrackingNet Datasets

[0023] To further verify the robustness and accuracy of the proposed method in handling complex scenarios, comparative experiments were conducted on UAV123, NFS, and TNL2K. Table 2 shows the success rate scores of the proposed method and mainstream tracking methods on these three datasets. The proposed method achieved a success rate of 70.7% on the UAV123 dataset, which is 0.8% and 1.5% higher than the mainstream ARTrackV2 and SeqTrack, respectively. Table 2: Test Results of Mainstream Tracking Methods on UAV123, NFS, and TNL2K Datasets

[0024] Experimental results show that the method proposed in this invention can perform excellent tracking tasks in complex scenarios on standard tracking datasets with challenges such as interference from similar objects, changes in illumination and scale, and sudden movement of the target.

[0025] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A Transformer visual target tracking method based on adaptive label partitioning, characterized in that, Includes the following steps: Step 1: Construct the network model; Step 2: Construct an adaptive label partitioning module and integrate it into the encoder in the network model; Step 3: Based on the loss function, maximize the log-likelihood between the generated target sequence and the actual target sequence, and train the network model. Step 4: Use the trained network model to track the target in the video and obtain the tracking results.

2. The Transformer visual target tracking method based on adaptive label partitioning according to claim 1, characterized in that, The network model includes a linear projection layer, an encoder, and a decoder. The linear projection layer converts image patches into visual embeddings, which are then input into the encoder. The encoder extracts visual features. The decoder generates the bounding box values ​​of the target.

3. The Transformer visual target tracking method based on adaptive label partitioning according to claim 2, characterized in that, The encoder includes an adaptive label partitioning module, a parallel attention calculation module, and a feedforward neural network layer; the adaptive label partitioning module is used to classify search labels; the parallel attention calculation module is used to perform parallel attention calculations on labels of different categories; the feedforward neural network layer is used to generate embeddings for the next encoder block; The decoder includes a word-to-embedding layer, a masked multi-head attention layer, a multi-head cross-attention layer, a feedforward neural network layer, and an embedding-to-word layer. The word-to-embedding layer converts discrete coordinates into continuous word embeddings. The masked multi-head attention layer restricts the output of each sequence element to depend only on the previous element. The multi-head cross-attention layer fuses the output of the masked multi-head attention layer with the output of the encoder. The feedforward neural network layer generates embeddings for the next decoder block. The embedding-to-word layer converts the decoder output back into discrete coordinates to generate the final prediction result.

4. The Transformer visual target tracking method based on adaptive label partitioning according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2.1: Aggregate all template tags through global max pooling to generate an overall representation for target perception; Step 2.2: Concatenate the target-aware representation with the features of each search tag; Step 2.3: Process the concatenated features using a lightweight multilayer perceptron to predict whether each search tag belongs to G. S and G A The probability of a category is calculated using the following formula: Among them, G S G represents the target marker within the search area. A G represents the background marker in the search area. z and Represent the template tag embedding and the i-th search tag embedding, respectively, π i,0 and π i,1 These represent the i-th search tag belonging to G. S Category and G A The probability of the category, based on the predicted value π i,0 and π i,1 The category of the search tag is determined by selecting the one with the higher probability.

5. The Transformer visual target tracking method based on adaptive label partitioning according to claim 1, characterized in that, The loss function described in step 3 is as follows: Where P(·) represents the probability of softmax, x and z represent the search image and template image respectively, and s j Let s represent the target sequence for the current prediction, j represent the position of the currently generated token in the sequence, and s represent the position of the token in the sequence. ＜j Let j represent the target sequence preceding j, and L represent the length of the target sequence.

6. The Transformer visual target tracking method based on adaptive label partitioning according to claim 1, characterized in that, Step 3 specifically includes the following steps: Step 3.1: Divide the search image and template image into image blocks, then convert the image blocks into visual embeddings through linear projection and input them into the encoder; Step 3.2: Use the adaptive tag partitioning module to divide the search tags into G. S and G A Two categories, G T Mark all templates; Step 3.3: Model the cross-relationship of the three label categories through three parallel attention calculations and output visual features; Step 3.4: Convert the target bounding box into a series of discrete tokens and input them into the wordtoembedding layer; Step 3.5: Add positional embedding to the wordto embedding layer, and input the fused embedding into the mask multi-head attention layer of the decoder; Step 3.6: The word embeddings output from the masked multi-head attention layer and the visual features output from the encoder are input together into the multi-head cross-attention layer of the decoder; Step 3.7: The decoder output is converted back to discrete coordinates through the embeddingtoword layer to generate the final target boundary coordinates.

7. The Transformer visual target tracking method based on adaptive label partitioning according to claim 6, characterized in that, Step 3.3 specifically involves: Step 3.3.1: Convert the probability π into a one-hot tensor Step 3.3.2: Define a one-hot tensor Category assignment for template tags, Q z Set all values ​​in the first column to 1, and set the other two columns to 0. Step 3.3.3: Obtain the value by adding a column of zeros before Q. Category assignment for search tags; Step 3.3.4: Utilize the merged... Constructing attention masks The value of each element is M. i,j Calculate using the following formula: Step 3.3.5: Multiply the attention mask matrix M element-wise with the original attention weight matrix, and set the attention weights of the interaction positions that are prohibited by the mask to zero; Step 3.3.6: Multiply the attention weight matrix after masking with the value matrix to obtain the final output.

Images