Visual target tracking method based on sparse Transform

Abstract

The invention provides a visual target tracking method based on sparse Transform, and relates to the technical field of visual target tracking, the method converts a target tracking problem into a sequence generation problem, firstly, a simple encoder-decoder architecture based on Transform is used, an additional header network is eliminated, and a tracking architecture is simplified; secondly, a sparse attention mechanism is added into a decoder, so that the model pays more attention to key information, and the distinction degree between a foreground and a background is improved; the method comprises the following steps: constructing a network model; constructing a sparse attention mechanism, and applying the sparse attention mechanism in a decoder in the network model; training the network model according to the loss function; and tracking a target in the video by using the trained network model. The method provided by the invention has higher accuracy and robustness when facing complex scenes with serious shielding, illumination, scale change and the like.

Description

Technical Field

[0001] The present invention relates to the field of visual target tracking technology, and in particular to a visual target tracking method based on sparse Transformer. Background Art

[0002] Visual target tracking is an important research topic in the field of computer vision. Its purpose is to continuously capture the position of the target object in the video sequence and accurately track the target even when the target object is occluded or deformed. With the development of science and technology, visual target tracking technology has been widely used in video surveillance, intelligent transportation, military guidance and other fields.

[0003] At present, the mainstream algorithms in the field of single target tracking can be divided into two categories: tracking based on classification and regression and tracking based on corner prediction. Chen et al. used a tracking algorithm based on classification and regression for target tracking, designed a classification head to locate the target, predicted the foreground and background of the target, and designed a regression head to estimate the scale of the target. Each head network requires one or more loss functions, which increases the complexity of the tracking framework. Yan et al. used a tracking algorithm based on corner prediction for target tracking, designed a head network to predict the corner points of the bounding box of the target object. Similarly, the design of these head networks also requires a large number of loss functions, which increases the number of hyperparameters and thus increases the difficulty of training.

[0004] The Transformer model was originally used for natural language processing tasks, especially machine translation. By introducing the attention mechanism, it replaced the traditional recurrent neural network and convolutional neural network, improving the processing speed and translation quality. With the success of the Transformer model in natural language processing, researchers began to explore its application in the field of computer vision. It has now been successfully introduced into target tracking. The multi-head attention mechanism in the Transformer enables the model to handle complex dependencies and global information more flexibly and efficiently, improving the quality of generated sequences. However, when dealing with scenes containing a lot of background information or other complex scenes, its global attention ability will cause the weight of important information (such as targets in the search area) to be diluted, and unnecessary attention will be allocated to background information, thereby reducing tracking performance.

[0005] Existing single target tracking algorithms usually have the following three problems:

[0006] 1) It is necessary to design a customized head network, which increases the complexity of the framework.

[0007] 2) The introduction of multiple loss functions requires adjusting more hyperparameters, which increases the difficulty and uncertainty of model tuning.

[0008] 3) When dealing with complex scenes, the global attention capability of the multi-head attention mechanism reduces the model's focus on key information. Summary of the invention

[0009] In view of the shortcomings of the prior art, the present invention provides a visual target tracking method based on sparse Transformer. The method transforms the target tracking problem into a sequence generation problem. First, a simple Transformer-based encoder-decoder architecture is used to eliminate the additional head network and simplify the tracking architecture. Second, a sparse attention mechanism is added to the decoder to make the model pay more attention to key information and improve the distinction between foreground and background. Experiments show that the proposed algorithm can complete the tracking task in complex scenes for standard tracking datasets with severe occlusion, illumination and scale changes, and sudden target motion.

[0010] A visual target tracking method based on sparse Transformer includes the following steps:

[0011] Step 1: Construct a network model;

[0012] The network model includes a linear projection layer, a visual embedding layer, an encoder, and a decoder. The linear projection layer maps the image patch to the visual embedding layer; the visual embedding layer inputs the visual embedding after adding position information to the encoder; the encoder extracts visual features; and the decoder generates a bounding box value of the target;

[0013] The encoder includes a multi-head self-attention layer and a word to embedding layer, wherein the multi-head self-attention layer is used to jointly extract visual features in the search image and the template image; the word to embedding layer is used to convert discrete coordinates into continuous word embeddings;

[0014] The decoder includes a masked multi-head attention layer, a sparse multi-head attention layer, a feedforward neural network layer, and an embedding to word layer; the masked multi-head attention layer limits the output of each sequence element to only depend on the previous element; the sparse multi-head attention layer is used to combine the output of the masked multi-head attention with the output of the encoder to make the model pay more attention to important information; the feedforward neural network layer generates an embedding for the next decoder block. The embedding to word layer is used to convert the output of the decoder back to discrete coordinates to generate the final prediction result.

[0015] Step 2: construct a sparse attention mechanism and integrate the sparse attention mechanism into the decoder in the network model;

[0016] Step 2.1: Perform matrix multiplication on the query and the key to obtain the attention matrix;

[0017] Step 2.2: Scale the attention matrix by dividing it by a scaling factor; the scaling factor is the dimension of the key;

[0018] Step 2.3: Normalize the first K largest elements in each row of the scaled attention matrix and replace all other elements with 0;

[0019] Step 2.4: Multiply the normalized matrix with the value matrix to get the final output;

[0020] Step 3: Based on the loss function, the log-likelihood between the generated target sequence and the actual target sequence is maximized to train the network model;

[0021] The loss function is as follows:

[0022]

[0023] Where P(·) represents the softmax probability, s and t represent the search image and template image respectively, j represents the position of the token currently being generated in the sequence, and z j represents the currently predicted target sequence, z ＜j represents the target sequence before j, and L represents the length of the target sequence.

[0024] Step 3.1: Divide the search image and template image into image patches, and then convert the image patches into visual embeddings through a linear projection layer;

[0025] Step 3.2: Add position embedding to the visual embedding layer to preserve the position information of the target;

[0026] Step 3.3: Input the visual embedding with added position information into the encoder for visual feature extraction;

[0027] The step 3.3 is specifically as follows: jointly extracting the visual features in the search image and the template image through the multi-head self-attention layer of the encoder, learning the feature correspondence between them, and only inputting the features of the search image into the decoder.

[0028] The encoder is based on the standard visual Transformer architecture and performs two optimizations:

[0029] (1) The CLASS token has been deleted;

[0030] (2) Add a linear projection to the last layer to align the feature dimensions of the encoder and decoder;

[0031] Step 3.4: Convert the target bounding box into a series of discrete tokens and input them into the word to embedding layer;

[0032] Step 3.5: Add position embedding to the word to embedding layer, and the fused embedding is input to the masked multi-head attention layer of the decoder;

[0033] Step 3.6: The word embedding output by the masked multi-head attention layer and the visual features output by the encoder are input into the sparse multi-head attention layer of the decoder;

[0034] Step 3.7: Convert the decoder output back to discrete coordinates through the embedding to word layer to generate the final object boundary coordinates;

[0035] Step 4: Use the trained network model to track the target in the video and obtain the tracking result.

[0036] The beneficial effects of adopting the above technical solution are:

[0037] The present invention provides a visual target tracking method based on sparse Transformer. In view of the high complexity of the existing tracking framework, a simple Transformer-based encoder-decoder architecture is used, which does not require a customized head network and simplifies the tracking framework. In view of the problem that the redundancy of the loss function in the existing tracking method leads to the need to adjust more hyperparameters, increasing the difficulty and uncertainty of model tuning, only the cross entropy loss is used to maximize the log-likelihood between the generated target sequence and the actual target sequence, without the need for other loss functions. In view of the problem that the global attention ability of the attention mechanism reduces the model's attention to key information when processing complex scenes, a sparse attention mechanism is introduced. When processing scenes containing a large amount of background information or other complex scenes, the model pays more attention to key information and improves the distinction between foreground and background. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a structural diagram of the visual target tracking method based on sparse Transformer proposed in the present invention;

[0039] Figure 2 is a schematic diagram of the working of masked multi-head attention in an embodiment of the present invention;

[0040] Figure 3 is a calculation flow chart of the sparse multi-head attention mechanism in an embodiment of the present invention;

[0041] Figure 4 2. It is a schematic diagram comparing the calculation results of the sparse multi-head attention mechanism and the traditional multi-head attention mechanism in an embodiment of the present invention;

[0042] Figure 5 is a flowchart of online template updating in an embodiment of the present invention;

[0043] Figure 6 It is a flow chart of the window penalty strategy in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0045] The experiments conducted in the embodiments of the present invention were implemented on a 3 GHz CPU, 32 GB memory, and a GeForce RTX4090 GPU, based on PyTorch framework programming.

[0046] A visual object tracking method based on sparse Transformer, such as Figure 1 As shown, the following steps are included:

[0047] Step 1: Construct a network model;

[0048] The network model includes a linear projection layer, a visual embedding layer, an encoder, and a decoder. The linear projection layer maps the image patch to the visual embedding layer; the visual embedding layer inputs the visual embedding after adding position information to the encoder; the encoder extracts visual features; and the decoder generates a bounding box value of the target;

[0049] The encoder includes a multi-head self-attention layer and a word to embedding layer, wherein the multi-head self-attention layer is used to jointly extract visual features in the search image and the template image; the word to embedding layer is used to convert discrete coordinates into continuous word embeddings;

[0050] The decoder includes a masked multi-head attention layer, a sparse multi-head attention layer, a feedforward neural network layer and an embedding to word layer; the masked multi-head attention layer limits the output of each sequence element to only depend on the previous element; the sparse multi-head attention layer is used to combine the output of the masked multi-head attention with the output of the encoder to make the model pay more attention to important information; the feedforward neural network layer generates embedding for the next decoder block; the embedding to word layer is used to convert the output of the decoder back into discrete coordinates to generate the final prediction result.

[0051] Step 2: construct a sparse attention mechanism and integrate the sparse attention mechanism into the decoder in the network model;

[0052] Step 2.1: Perform matrix multiplication on the query and the key to obtain the attention matrix;

[0053] Step 2.2: Scale the attention matrix by dividing it by a scaling factor; the scaling factor is the dimension of the key;

[0054] Step 2.3: Normalize the first K largest elements in each row of the scaled attention matrix and replace all other elements with 0;

[0055] Step 2.4: Multiply the normalized matrix with the value matrix to get the final output;

[0056] Step 3: Based on the loss function, the log-likelihood between the generated target sequence and the actual target sequence is maximized to train the network model;

[0057] The loss function is as follows:

[0058]

[0059] Where P(·) represents the softmax probability, s and t represent the search image and template image respectively, j represents the position of the token currently being generated in the sequence, and z j represents the currently predicted target sequence, z ＜j Denotes the target sequence before j, and L denotes the length of the target sequence. During training, the cross entropy loss is used to maximize the log likelihood to learn how to generate the target sequence so as to predict the next token given the previous subsequence and the input video frame.

[0060] Step 3.1: Divide the search image and template image into image patches, and then convert the image patches into visual embeddings through a linear projection layer;

[0061] Step 3.2: Add position embedding to the visual embedding layer to preserve the position information of the target;

[0062] Step 3.3: Input the visual embedding with added position information into the encoder for visual feature extraction;

[0063] The step 3.3 is specifically as follows: jointly extracting the visual features in the search image and the template image through the multi-head self-attention layer of the encoder, learning the feature correspondence between them, and only inputting the features of the search image into the decoder.

[0064] The encoder makes two adjustments based on the standard Visual Transformer (ViT) architecture:

[0065] (1) The CLASS token was removed because it is designed for image classification tasks;

[0066] (2) Add a linear projection to the last layer to align the feature dimensions of the encoder and decoder.

[0067] Step 3.4: Convert the target bounding box into a series of discrete tokens and input them into the word to embedding layer;

[0068] Step 3.5: Add position embedding to the word to embedding layer, and the fused embedding is input to the masked multi-head attention layer of the decoder;

[0069] Step 3.6: The word embedding output by the masked multi-head attention layer and the visual features output by the encoder are input into the sparse multi-head attention layer of the decoder;

[0070] The sparse multi-head attention layer allows the model to pay more attention to key information, avoid over-focusing on background information, improve the distinction between foreground and background, and make the prediction of target bounding boxes more accurate even when dealing with complex scenes such as deformation and occlusion;

[0071] Step 3.7: Convert the decoder output back to discrete coordinates through the embedding to word layer to generate the final object boundary coordinates;

[0072] Step 4: Use the trained network model to track the target in the video and obtain the tracking result.

[0073] In this embodiment, the search image and the template image have the same resolution. The search image and the template image are divided into image blocks, and then the image blocks are converted into visual embeddings through linear projection, and the position embeddings are added to them, so that the encoder takes the position information into account when learning feature representation, and better distinguishes the features at different positions in the image. Finally, the merged visual embeddings are input into the encoder, and then their visual features are output.

[0074] Among them, before entering the decoder, the target bounding box is discretized, and each continuous coordinate is uniformly discretized into integers between 1 and n and mapped to a unified vocabulary V. These integers are regarded as words in the vocabulary V. The final input sequence is [start, x, y, w, h], and the target sequence is [x, y, w, h, end], where start and end are used to control the start and end of the sequence generation task respectively, [x, y] represents the center point position of the target, and [w, h] represents the scale of the target.

[0075] Masked multi-head attention restricts the model to only pay attention to the previous information when generating each tag, but not the following information. Figure 2 As shown in Figure 1, when generating w, the model can only focus on start, x, y, but cannot focus on the information after w. The calculation method of the sparse multi-head attention mechanism and the calculation comparison with the traditional multi-head attention mechanism are shown in Figure 1. Figure 3-Figure 4As shown in the figure. Sparse multi-head attention is to improve the model's attention to key information and reduce the interference of complex background information on tracking prediction. Multi-head attention considers the relationship between all input features when calculating attention weights, which usually involves calculating the similarity between all pixel pairs to obtain global attention information. Sparse multi-head attention only focuses on the K pixel values ​​most relevant to the current pixel, thereby improving the distinction between foreground and background.

[0076] like Figure 3 As shown in the figure, the traditional multi-head attention mechanism first performs matrix multiplication on the query and the key, and then scales the result, that is, divides it by a scaling factor, which is the dimension of the key, and then uses the softmax function to normalize the attention score matrix row by row, and finally multiplies the normalized matrix with the value matrix to get the final output. However, in sparse multi-head attention, after scaling, only the first K most similar elements in each row of the attention matrix are normalized, and all other elements are replaced by 0. This normalization method reduces the impact of irrelevant similarities and improves the attention to relevant features.

[0077] like Figure 4 As shown, traditional scaled dot-product attention will amplify small attention weights, which may amplify noise and background interference, while sparse multi-head attention alleviates this problem by focusing on the most similar pixel values.

[0078] The output of the decoder is converted into the boundary coordinates of the target through a multi-layer perceptron with softmax, and then converted back into discrete words according to the vocabulary table V to obtain the coordinate values ​​of the target bounding box.

[0079] Furthermore, this method introduces online template updating and window penalty strategies. During the tracking process, the appearance of the target object may change due to perspective changes, deformation, illumination changes, occlusion, etc. By updating the template online, the tracker can adjust the template according to the target appearance information obtained in real time to maintain accurate tracking of the target. During the tracking process, the tracker may fail or misjudge due to sudden large displacement of the tracking window. Through window penalty, large displacement can be punished to improve the accuracy and stability of tracking.

[0080] The specific steps of online template updating are as follows: Figure 5 As shown:

[0081] 1) During the inference process of each frame, generate bounding box labels of the target object;

[0082] 2) Based on the generated tags, perform a softmax operation and take the average to calculate the probability of the generated tags;

[0083] 3) Determine whether the probability of the generated mark is higher than a certain threshold τ and check whether the update interval Tu has been reached;

[0084] 4) If the probability is higher than the threshold and reaches the update interval, the template is updated. If the probability of the generated mark is not higher than the threshold or the update interval is not reached, the dynamic template remains unchanged;

[0085] 5) When updating the template, the tracking result of the current frame is used to update the dynamic template.

[0086] Among them, the specific steps of the window penalty strategy are as follows Figure 6 As shown:

[0087] 1) First, calculate the offset of the generated marker x and y from the center point to determine whether a penalty needs to be imposed;

[0088] 2) Represent the probability of generating a token as a vector whose length is the number of words in the vocabulary V, with one probability for each word;

[0089] 3) Create a Hanning window vector of the same length as the probability vector. The value of the Hanning window vector will be adjusted according to the distance between the marked word and the center point. Words farther from the center point will receive a greater penalty, and words closer to the center point will receive a smaller penalty.

[0090] 4) Multiply the probability vector by the Hanning window vector element-wise to apply the penalty of the Hanning window.

[0091] In order to verify the accuracy and effectiveness of the proposed method in tracking in complex scenarios, the proposed method is compared with other mainstream target tracking methods on six data sets, namely LaSOT, GOT-10k, TrackingNet, UAV123, NFS, and TNL2K.

[0092] Among them, LaSOT is a large-scale single target tracking dataset, containing 1,400 video sequences, with an average video length of about 2,500 frames and a total of more than 1.5 million frames. The LaSOT dataset covers different types of targets and complex scenes, including illumination changes, occlusion, fast motion, camera shake, etc., and is one of the important datasets for evaluating the performance of target tracking algorithms. The GOT-10k dataset is a general target tracking dataset, containing more than 10,000 video sequences with a total of more than 1,500,000 frames. These video sequences cover a variety of target categories and scenes, and have different challenges such as motion and occlusion. The GOT-10k dataset is characterized by a large data scale and contains a variety of target categories and motion patterns, which is of great significance for evaluating the generalization ability of target tracking algorithms. TrackingNet is one of the largest target tracking datasets currently, divided into two parts: training set and test set, containing a total of more than 30,000 video sequences, covering various scenes and target objects. The UAV123 dataset is designed for evaluating target tracking tasks in drone scenarios. It contains 123 video sequences, covering various complex environments and challenges, such as fast motion, perspective changes, illumination changes, and occlusion. NFS is used to evaluate the performance of trackers in high frame rate video scenarios. It contains 100 video sequences, covering various high-speed motion and complex tracking scenarios. TNL2K is a large-scale dataset combined with natural language descriptions. Each video sequence has not only visual data but also natural language descriptions. It is a diverse and challenging benchmark.

[0093] The comparison results are shown in Table 1, where Succ represents the success rate, P norm and P represent the normalized accuracy and precision respectively, AO represents the average value of the overlapping area of ​​all frames, SR 0.5 and SR 0.75 It is the success rate calculated when the IoU threshold is 0.5 and 0.75. The comparison between the method proposed in the present invention and the mainstream method on the Lasot, GOT-10k, and TrackingNet datasets shows that the method of the present invention achieves better performance. Among them, on the Lasot dataset, the present invention achieved a success rate of 71.8%, which is 1.9% higher than SeqTrack. There are also improvements on the GOT-10K and TrackingNet datasets.

[0094] Table 1: Test results of Lasot, GOT-10k, and TrackingNet datasets compared with mainstream tracking methods

[0095]

[0096]

[0097] In order to further verify that the method proposed in this invention has good robustness and accuracy in processing complex scenes, comparative experiments are conducted on UAV123, NFS and TNL2K. Table 2 shows the success rate comparison results of the method proposed in this invention and the mainstream tracking methods on the three datasets. Among them, the method proposed in this invention achieved a success rate score of 69.9% on the UAV123 dataset, which is 0.7% and 1.6% higher than the mainstream SeqTrack and OSTrack, respectively.

[0098] Table 2: Comparison of test success rates of mainstream tracking methods on UAV123, NFS, and TNL2K datasets

[0099]

[0100] Experimental results show that the method proposed in this paper can excellently complete tracking tasks in complex scenes when facing challenging standard tracking datasets with severe occlusion, illumination and scale changes, and sudden target motion.

[0101] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. A visual target tracking method based on sparse Transformer, characterized in that: The following steps are involved: Step 1: Construct a network model; Step 2: construct a sparse attention mechanism and integrate the sparse attention mechanism into the decoder in the network model; Step 3: Based on the loss function, the log-likelihood between the generated target sequence and the actual target sequence is maximized to train the network model; Step 4: Use the trained network model to track the target in the video and obtain the tracking result.

2. According to claim 1, a visual target tracking method based on sparse Transformer is characterized in that: The network model includes a linear projection layer, a visual embedding layer, an encoder, and a decoder. The linear projection layer maps the image patch to the visual embedding layer; the visual embedding layer inputs the visual embedding after adding the position information to the encoder; The encoder extracts visual features; The decoder generates bounding box values ​​for the object.

3. The method for visual target tracking based on sparse Transformer according to claim 2, characterized in that: The encoder includes a multi-head self-attention layer and a word to embedding layer, wherein the multi-head self-attention layer is used to jointly extract visual features in the search image and the template image; the word to embedding layer is used to convert discrete coordinates into continuous word embeddings; The decoder includes a masked multi-head attention layer, a sparse multi-head attention layer, a feed-forward neural network layer, and an embedding to word layer; the masked multi-head attention layer restricts the output of each sequence element to only depend on the previous element; The sparse multi-head attention layer is used to combine the output of the masked multi-head attention with the output of the encoder, so that the model pays more attention to important information; the feedforward neural network layer generates an embedding for the next decoder block; The embedding to word layer is used to convert the output of the decoder back into discrete coordinates to generate the final prediction result.

4. The method for visual object tracking based on sparse Transformer according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 2.1: Perform matrix multiplication on the query and the key to obtain the attention matrix; Step 2.2: Scale the attention matrix by dividing it by a scaling factor; the scaling factor is the dimension of the key; Step 2.3: Normalize the first K largest elements in each row of the scaled attention matrix and replace all other elements with 0; Step 2.4: Multiply the normalized matrix with the value matrix to get the final output.

5. The method for visual object tracking based on sparse Transformer according to claim 1, characterized in that: The loss function described in step 3 is as follows: Where P(·) represents the softmax probability, s and t represent the search image and template image respectively, j represents the position of the token currently being generated in the sequence, and z j represents the currently predicted target sequence, z ＜j represents the target sequence before j, and L represents the length of the target sequence.

6. The method for visual object tracking based on sparse Transformer according to claim 1, characterized in that: The step 3 specifically comprises the following steps: Step 3.1: Divide the search image and template image into image patches, and then convert the image patches into visual embeddings through a linear projection layer; Step 3.2: Add position embedding to the visual embedding layer to preserve the position information of the target; Step 3.3: Input the visual embedding with added position information into the encoder for visual feature extraction; Step 3.4: Convert the target bounding box into a series of discrete tokens and input them into the word to embedding layer; Step 3.5: Add position embedding to the word to embedding layer, and the fused embedding is input to the masked multi-head attention layer of the decoder; Step 3.6: The word embedding output by the masked multi-head attention layer and the visual features output by the encoder are input into the sparse multi-head attention layer of the decoder; Step 3.7: Convert the decoder output back to discrete coordinates through the embedding to word layer to generate the final object boundary coordinates.

7. The method for visual object tracking based on sparse Transformer according to claim 6, characterized in that: The step 3.3 is specifically as follows: jointly extracting visual features in the search image and the template image through the multi-head self-attention layer of the encoder, learning the feature correspondence between them, and only inputting the features of the search image into the decoder.

8. The method for visual object tracking based on sparse Transformer according to claim 6, characterized in that: In step 3.3, the encoder is optimized in two places based on the standard visual Transformer architecture: (1) The CLASS token has been deleted; (2) Add a linear projection to the last layer to align the feature dimensions of the encoder and decoder.

Images