A single target tracking method based on Transformer

Abstract

The present invention proposes a single target tracking method based on Transformer, and the steps are as follows: cropping the target area from the first frame of the video, using a pre-trained ResNet50 network to extract features; using a feature fusion method based on Transformer to fuse the target features, inputting the features into a regression model generator to generate a regression model, and inputting the target features into a classification model generator to generate a classification model; using a feature fusion method based on Transformer to fuse the extracted features of the search area and inputting the features into the regression model to obtain a regression result; using Transformer to enhance the features of the search area in combination with a training set and inputting the features into the classification model to obtain a classification result; generating an accurate bounding box for the target based on the classification result and the regression result to track the target. The present invention integrates Transformer into the tracking framework, and makes full use of the temporal context and spatial context in the tracking process to improve the tracking accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and in particular to a Transformer-based single target tracking method. Background Art

[0002] Object tracking is a fundamental task in computer vision. With the development of deep learning, it has made great progress. The object tracking task usually builds a model based on the first frame of the video, and then uses it to locate the object in subsequent frames. But it is still a challenging task due to sudden motion, occlusion and other problems. At present, according to the network objective function of the tracker, popular trackers can be divided into three categories: classification-based trackers, regression-based trackers, and classification and regression-based trackers. Classification-based trackers regard the tracking task as a detection task for consecutive frames, and use a classifier to classify the candidate regions to locate the target position. Since this classifier focuses on inter-class classification, it is prone to drift problems when similar targets appear. Unlike classification-based trackers, regression-based trackers usually use the minimization of regularized least squares functions to locate objects due to the continuity of the object tracking positioning space. In order to take advantage of the advantages of foreground-background classification and ridge regression at the same time, recent advanced work designs trackers into two branches, namely, classification branch and regression branch, the former is used to roughly locate the target, and the latter is used to generate accurate bounding boxes. The classification and regression based tracker shows better performance compared to the classification based tracker and regression based tracker.

[0003] Although classification and regression based trackers have made great progress by dividing the tracking task into classification and regression tasks, there are still some limitations. For classification tasks, the temporal context containing the state changes of the object across frames is important for accurate classification, but the current popular twin trackers only use spatial context to locate objects. They usually use ordinary convolution, grouped convolution or deep convolution with templates and search regions to extract the spatial dimension relationship between them, ignoring the temporal context. Unlike twin trackers, some trackers use both temporal and spatial context for tracking. They use independent samples collected in tracking to train classifiers, and then use the classifiers to locate objects. Although the latter use temporal context to improve the adaptability of the classifier, they do not explore the relationship between frames. This results in the waste of rich temporal context between frames. For regression tasks, most trackers use the same features as classification tasks. However, regression tasks are different from classification tasks. Classification tasks need to tolerate changes in the appearance of the target and are suitable for using high-level features of the target, while regression tasks require clear target boundaries to generate accurate bounding boxes and are included in low-level features. However, low-level features include many interference factors, so effective high-level and low-level feature fusion is important for regression tasks. In addition, some trackers use methods based on anchor box placement or box sampling and refinement for regression tasks. Due to their complex design methods, it is difficult to online optimize the regression branch to adapt to the appearance changes of the target.

[0004] Recently, Transformer has attracted much attention due to its unique attention mechanism. Compared with the multiple iterations of recurrent neural networks and the multi-layer stacking of convolutional neural networks, Transformer can aggregate the information of the entire input sequence in one operation. This is achieved by generating three features (named Q, K, and V) for each input. The response of each input is calculated as the weighted sum of all V, where the weight is determined by the similarity between its Q and the corresponding K. ViT directly applies the standard Transformer to image classification, completely replacing the previous work of convolution. DETR proposes a Transformer-based object detection model, which achieves significantly better performance on large objects due to the processing of global information performance through self-attention. TransT proposes a novel attention-based feature fusion network, which effectively combines the features of the template and the search area using only attention. TrTr proposes a new tracker network based on Transformer to obtain global and rich contextual correlations. Similarly, Transformer (a model name) has also been prominent in other works. Summary of the invention

[0005] In view of the technical problems that existing target tracking methods are computationally complex and cannot optimize regression analysis online, the present invention proposes a single target tracking method based on Transformer, which integrates Transformer into the tracking framework for feature fusion of regression branches and feature enhancement of classification branches. During the tracking process, the temporal context and spatial context are fully utilized to improve the tracking accuracy.

[0006] In order to achieve the above object, the technical solution of the present invention is implemented as follows: a single target tracking method based on Transformer, the steps of which are as follows:

[0007] Step 1: Crop the target area from the first frame of the video to be tracked, and use the ResNet50 network pre-trained on the ImageNet dataset to extract features of the target area to obtain three layers of different target features;

[0008] Step 2: Use the Transformer-based feature fusion method to fuse the target features obtained in step 1, input the fused features into the regression model generator to generate a regression model, and input the target features of the last layer into the classification model generator to generate a classification model;

[0009] Step 3: The target area in the first frame is rotated and translated to perform feature enhancement to obtain a training data set, and the classification model and regression model obtained in step 2 are pre-trained using the training data set;

[0010] Step 4: Input the image of the search area of ​​the subsequent frame into the ResNet50 network for feature extraction to obtain the features of the three-layer search area;

[0011] Step 5: The features of the search area are fused using the Transformer-based feature fusion method, and the regression model is used as the convolution kernel to convolve the fused features to obtain the regression result;

[0012] Step 6: Use Transformer to enhance the features of the last layer of the search area with the training set, and then use the classification model as the convolution kernel to convolve with the features of the enhanced search area to obtain the classification result;

[0013] Step 7: Generate an accurate bounding box for the target based on the classification result of step 6 and the regression result of step 5. If the tracking result meets the requirements, the frame image is processed and added to the training data set; if the tracking fails or the classification score is too low, the classification model and regression model are updated using the collected training data set;

[0014] Step 8: Get the next frame of video and return to step 4 until the entire tracking task is completed.

[0015] The implementation method of the Transformer-based feature fusion method in step 2 and step 5 is:

[0016] (1) The feature X of the last layer is obtained layer4 Use the attention mechanism in Transformer to calculate and obtain enhanced features

[0017] (2) The features As the residual term and feature X layer4 Add up the features

[0018] (3) The features Upsampling and channel compression are performed to obtain features The second layer feature X layer3 Channel compression to obtain features Then the feature and Features Add element by element to get the feature

[0019] (4) Features Upsampling to get features Then the feature and the first layer feature X layer2 Add element by element to get the fused features

[0020] The method of using Transformer in step 6 to perform feature enhancement in combination with the training set is:

[0021] Step 1: Get the feature X of the last layer of the search area layer4 Use the attention mechanism in Transformer to calculate and obtain enhanced features

[0022] Step 2: Add features As the residual term and feature X layer4 Add together to get

[0023] Step 3: Calculate the obtained training set features S using the attention mechanism in Transformer to obtain the enhanced features

[0024] Step 4: Add features As the residual term, add it to the feature S to get the feature

[0025] Step 5: Using the features from step 2 Calculate the matrix Q and use the enhanced target features of the last layer Calculate the matrices K and V, and then use the attention formula to calculate the features

[0026] Step 6: Add Features As the residual term and feature Add up the features

[0027] The calculation method of the attention mechanism in the Transformer is:

[0028] 1) The features The features are obtained through convolution operation. After the Reshape() operation, we get the tensor Among them, C represents the number of channels of the feature, H represents the height of the feature, W represents the width of the feature, N = H × W, Conv() is the convolution function, and Reshape() represents the operation function that transforms the matrix into a matrix of a specific dimension;

[0029] 2) The features The features are obtained through convolution operation. After the Reshape() operation, we get the tensor

[0030] 3) The features After the Reshape() operation, we get the tensor

[0031] 4) Use the calculated tensors Q, K, and V to input the attention formula to get the output of the attention formula:

[0032]

[0033] Among them, Softmax() represents the normalized exponential function, Attention() is the function of the attention formula, and the coefficient d k =H×W.

[0034] The regression model generator consists of a single-layer convolutional neural network, which processes the target features. and Input a single-layer convolutional neural network for convolution calculation to obtain a regression model M reg ; The classification model generator consists of a PrROI Pooling layer. The input features are subjected to the Pooling operation to obtain a classification model M cls ; The training set feature S is obtained by extracting the collected training set samples using the ResNet50 network.

[0035] The pre-training operation method is to use the samples of the training data set to input the classification model or regression model for forward propagation, then calculate the gradient information of each parameter in the classification model or regression model through back propagation, and then use the gradient information to update the parameters of the current classification model or regression model, so that the classification model or regression model fits the current training samples.

[0036] In step 5, the regression model M is used reg As the feature X after convolution kernel fusion reg Perform convolution operation to obtain the regression result m cls ; The step six classifies the model M cls As the feature X of the search area after convolution kernel and enhancement cls Perform convolution to obtain the classification result m cls .

[0037] The method for obtaining the bounding box in step 7 is: according to the classification result, the approximate position of the target is located, that is, the classification result m cls The position with the largest median value is mapped back to the search area to obtain the center position of the target; the offset values ​​of the upper, lower, left, and right sides of the target corresponding to the regression result relative to the center position are selected according to the position with the largest median value in the classification result; the bounding box is drawn according to the center position and offset value of the target.

[0038] When tracking fails or the classification score is too low in step seven, the classification model and the regression model are updated by using the data training set through the gradient descent method.

[0039] The enhanced features for Among them, Q, K, and V are the characteristics of X layer4 The tensor computed using the attention mechanism in Transformer.

[0040] The beneficial effects of the present invention are as follows: compared with the traditional tracking method, the addition of a Transformer-based feature fusion method can improve the fusion efficiency of useful feature information and alleviate the interference of noise in the features; in addition, a Transformer-based feature enhancement method is proposed, which can use the time context of the target to suppress the interference information in the search area, emphasize the target features in the search area, and improve the accuracy of the classification task. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is the overall framework diagram of the present invention.

[0043] Figure 2 Flowchart of feature calculation of Transformer of the present invention.

[0044] Figure 3 It is a schematic diagram of feature fusion based on Transformer in the present invention.

[0045] Figure 4 Transformer flow chart for feature enhancement of the classification branch in the present invention.

[0046] Figure 5 This is a comparison diagram of the tracking effect of the present invention and other existing trackers on the LaSOT general data set, where (a) is the position error threshold and (b) is the overlap rate threshold.

[0047] Figure 6 This is a comparison chart of the tracking effect of the present invention and other trackers on the OTB dataset. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] like Figure 1As shown, the present invention discloses a single target tracking method based on Transformer, comprising the following steps: at the beginning of the video to be tracked, the image of the object area to be tracked is input into the ResNet50 network for feature extraction to obtain three layers of different target features; these features are fused to generate regression models and classification models respectively; the target image area is data enhanced to obtain a training sample set; the classification model and the regression model are then trained respectively using the training sample set; the search area is input into the ResNet50 network for feature extraction to obtain three layers of features similar to the target features; the features of the search area are fused and then a regression task is performed; the search area features are enhanced using Transformer combined with the training sample set and then a classification task is performed; finally, the classification result and the regression result are combined to accurately locate the target; when appropriate during the tracking process, the classification model and the regression model need to be trained using the training set. The specific steps of the present invention are as follows:

[0050] Step 1: Get the location information of the target of interest from the first frame of the video to be tracked, that is, crop the target area, and use the ResNet50 network pre-trained on the ImageNet dataset to extract features of the target area to obtain three layers of different target features.

[0051] Because in the target tracking task, only the position information of the target to be tracked in the first frame of the video is given, and in the subsequent frames, the model needs to detect the position of the tracked target from it. Using ImageNet to pre-train the ResNet50 network allows the model to learn a general feature expression, and then quickly converge during the offline training process of the tracking task to fit the new task requirements. At present, most trackers in the tracking field use the ResNet50 network for basic feature extraction.

[0052] The three layers of different target features are Among them, Z layer2 , Z layer3 and Z layer4 They are the target features extracted from the 2nd, 3rd, and 4th layers of the ResNet50 network, and their sizes are fixed.

[0053] Step 2: The three layers of target features obtained in step 1 are fused using the Transformer-based feature fusion method, and then the fused features are input into the regression model generator to generate a regression model, and the target features of the last layer are input into the classification model generator to generate a classification model.

[0054] like Figure 3 As shown in the figure, the feature fusion method based on Transformer is:

[0055] (1) The target feature Z is obtained layer4 Use the attention mechanism in Transformer to calculate and obtain enhanced features The specific calculation formula is:

[0056]

[0057]

[0058]

[0059]

[0060] Among them, Conv() is the convolution function, Reshape() represents the operation of transforming the specified matrix into a matrix of a specific dimension. Attention() represents the attention function in Transformer.

[0061] (2) The features As the residual term and Z layer4 Add up the features

[0062]

[0063] Among them, the target feature Z layer4 The size of is 1024×18×18. The addition of residual terms and target features can prevent the model from overfitting and gradient disappearance.

[0064] (3) The features Upsampling and channel compression are performed to obtain features The target feature Z layer3 Channel compression to obtain features Then the feature and Features Add element by element to get the feature

[0065] Upsampling is achieved through linear interpolation, which can expand the width and height of the feature map. For example, a tensor (feature) originally has 512 channels, 18 height, and 18 width. After upsampling, it becomes 512 channels, 36 height, and 36 width. Channel compression is achieved through a convolution layer with a convolution kernel size of 1×1. The number of channels of the tensor (feature) can be compressed. For example, a tensor that originally has 512 channels, 18 height, and 18 width becomes a tensor with 256 channels, 18 height, and 18 width after compression. The expansion and compression ratios of the two operations are not fixed and can be adjusted by parameters. The upsampling of the present invention by 2 times and the reduction of the channel compression to 1 / 2 are set according to the needs of feature fusion.

[0066] (4) Features Upsampling to get features Then the feature and the target feature Z layer2 Add element by element to get the fused features for generating the regression model

[0067] Compared with the general fusion method, the fusion method proposed in the present invention can alleviate the noise accumulation problem caused by the fusion process, suppress interference features, and retain more useful features. Only the fourth layer is calculated by the attention mechanism in Transformer, which is the result of the balance between computational complexity and effectiveness. Because the feature map size in the fourth layer is 18*18, and the other two are 36*36 and 72*72, the computational overhead of the attention mechanism in Transformer increases quadratically with the number of H*W. Only performing attention calculation on the fourth layer also achieves the expected effect.

[0068] like Figure 3 As shown, the regression model generator consists of a single-layer convolutional neural network that transforms the features and Input the single-layer convolutional neural network for convolution calculation to obtain a regression model Because the classification task requires a certain tolerance to target changes, we choose to convert the target feature Z of the last layer of the target area into layer4 Input the classification model generator to generate a classification model, because the features of the last layer have more semantic information and are insensitive to changes in the appearance of the target. The classification model generator consists of a PrROI Pooling layer. The input features are pooled to obtain a classification model.

[0069] All the attention calculation processes in the Transformer designed in this invention, such as Figure 2 As shown, the calculation method of the attention mechanism in Transformer is:

[0070] 1) The features First, the features are obtained through convolution operation. Then the Reshape() operation is performed to obtain the features Where H×W.

[0071]

[0072] 2) The features First, the features are obtained through convolution operation. Then the Reshape() operation is performed to obtain the features Where H×W.

[0073]

[0074] Because the convolutional layers used by Conv() in the calculation of the two features Q and K are not the same, the parameters are different, and they are learned separately during the model training process, so the final Q and K are different.

[0075] 3) The features After Reshape() operation, the features are obtained Where H×W.

[0076]

[0077] 4) Use the calculated features Q, K and V to input the attention formula to get the output of the attention formula.

[0078]

[0079] Among them, Softmax() represents the normalized exponential function, and the formula on the right side of the equal sign is the definition of the Attention() function. k =H×W.

[0080] Step 3: Use rotation and translation operations to enhance the features of the target area in the first frame to obtain a training data set, and use the training data set to pre-train the classification model and regression model obtained in step 2 so that the pre-trained classification model and regression model can adapt to the current target.

[0081] At the beginning of the tracking task, the tracker only obtains the target in the first frame of the video. In order to increase the number of samples used for training, the target area in the first frame is rotated, translated, and other operations to enhance the features to obtain the training data set. The specific operation of pre-training is to use the samples of the training data set to input the model for forward propagation, and then calculate the gradient information of each parameter in the model through back propagation, and then use the gradient information to update the parameters of the current model so that the model fits the current training samples. Pre-training can learn a general target representation method, which can quickly converge in the offline training process of the tracking task.

[0082] Step 4: Input the image of the search area of ​​the subsequent frame into the ResNet50 network for feature extraction to obtain the features of the three-layer search area

[0083] Step 5: The features of the search area are fused using the Transformer-based feature fusion method, and the regression model is used as the convolution kernel to convolve the fused features to obtain the regression result.

[0084] like Figure 3 As shown in the figure, the feature fusion method based on Transformer is:

[0085] S1: The feature X of the search area obtained layer4 Use the attention mechanism in Transformer to calculate and obtain enhanced features

[0086]

[0087]

[0088]

[0089]

[0090] S2: The features As the residual term and feature X layer4 Add up the features

[0091]

[0092] S3: Features Upsampling and channel compression are performed to obtain features The feature X layer3 Channel compression to obtain features Then the feature and Features Add element by element to get the feature

[0093] S4: Features Upsampling to get features Then the feature and feature X layer2 Add element by element to get the fused features

[0094] Using regression model M reg As the feature X after convolution kernel fusion reg Perform convolution operation to get regression results

[0095] Step 6: Use Transformer to enhance the features of the last layer of the search area with the training set, and then use the classification model as the convolution kernel to convolve with the features of the enhanced search area to obtain the classification result.

[0096] like Figure 4 As shown in the figure, the method of using Transformer combined with training set to perform feature enhancement is:

[0097] Step 1: Get the features X of the search area layer4 Use the attention mechanism in Transformer to calculate and obtain enhanced features

[0098]

[0099]

[0100]

[0101]

[0102] Step 2: Add features As the residual term and feature X layer4 Add together to get

[0103]

[0104] Step 3: Get the training set features Use the attention mechanism in Transformer to calculate and obtain enhanced features n is the number of samples in the training set.

[0105]

[0106]

[0107]

[0108]

[0109] The training set feature S is obtained by extracting the collected training set samples using the ResNet50 network. The feature is obtained after extracting n training samples.

[0110] Step 4: Add features As the residual term, add it to the feature S to get the feature

[0111]

[0112] Step 5: Using the features from step 2 Calculate the matrix Q and use To calculate the matrices K and V, and then use the attention formula to calculate the features

[0113]

[0114]

[0115]

[0116]

[0117] Step 6: Add Features As the residual term and feature Add up the features

[0118]

[0119] The classification model M cls As the feature X of the search area after convolution kernel and enhancement cls Perform convolution to obtain classification results

[0120] Step 7: Generate an accurate bounding box for the target based on the classification and regression results. If the tracking result meets the requirements, crop the area where the tracked target is located from the frame image, and then add the cropped image area to the data training set. cls When the maximum value is less than 0.5, the classification model and regression model are updated using the data training set.

[0121] The approximate location of the target is located according to the classification result, that is, the classification result m cls The position with the largest median value. Map this position back to the search area to get the center position of the target. According to the position with the largest median value in the classification result, select the offset values ​​of the corresponding target in the regression result relative to the center position, and draw an accurate bounding box based on the center position and offset value of the target.

[0122] In the case of tracking failure or low classification score, the classification model and regression model are updated by gradient descent using the data training set. This training method is consistent with the training method at the beginning of tracking.

[0123] Step 8: Get the next frame of video and return to step 4 until the entire tracking task is completed.

[0124] A specific example of the present invention is to use the Pytorch framework to conduct experiments on a computer with an Intel(R) Xeon(R) CPU E5-2680 v4@2.40GHz and an NVDIA TITAN RTX. Figure 5The two graphs in the figure are the precision graph and the success graph. The precision graph shows the percentage of video frames where the distance between the center point of the target position estimated by the tracking algorithm and the center point of the manually annotated target is less than a given threshold. The success graph shows the percentage of video frames where the overlap score (OS) between the bounding box drawn by the tracking algorithm (denoted as a) and the manually annotated bounding box (denoted as b) is greater than a given threshold. The overlap rate is defined as Given different thresholds in the two graphs, the percentages obtained are different, so a curve can be obtained. Figure 5 TrCAR is the name of the tracker proposed by the present invention, and GlobalTrack, SiamCAR, SiamBAN, ATOM, SiamRPN++, MDNet, SiamFC, and DSiam are the names of trackers proposed by other scholars in recent years. The numbers behind refer to the average precision and average accuracy of the tracker.

[0125] like Figure 5 As shown, compared with the current advanced trackers on the general data set LaSOT, the tracking method TrCAR of the present invention has obvious improvements in accuracy and success rate. Figure 6 The actual tracking results of the present invention and the current advanced tracker DiMP in the OTB100 dataset are compared. Figure 6 It can be seen from the figure that the tracking method of the present invention generates a more accurate bounding box than the tracker DiMP when similar targets appear in the search area and when tracking is performed for a long time.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A single target tracking method based on Transformer, characterized in that: The steps are as follows: Step 1: Crop the target area from the first frame of the video to be tracked, and use the ResNet50 network pre-trained on the ImageNet dataset to extract features of the target area to obtain three layers of different target features; Step 2: Use the Transformer-based feature fusion method to fuse the target features obtained in step 1, input the fused features into the regression model generator to generate a regression model, and input the target features of the last layer into the classification model generator to generate a classification model; Step 3: The target area in the first frame is rotated and translated to perform feature enhancement to obtain a training data set, and the classification model and regression model obtained in step 2 are pre-trained using the training data set; Step 4: Input the image of the search area of ​​the subsequent frame into the ResNet50 network for feature extraction to obtain the features of the three-layer search area; Step 5: The features of the search area are fused using the Transformer-based feature fusion method, and the regression model is used as the convolution kernel to convolve the fused features to obtain the regression result; Step 6: Use Transformer to enhance the features of the last layer of the search area with the training set, and then use the classification model as the convolution kernel to convolve with the features of the enhanced search area to obtain the classification result; Step 7: Generate an accurate bounding box for the target based on the classification result of step 6 and the regression result of step 5. If the tracking result meets the requirements, the tracking area is cropped from the frame image and added to the training data set; If tracking fails or the classification score is too low, the classification model and regression model are updated using the collected training dataset; Step 8: Get the next frame of video and return to step 4 until the entire tracking task is completed; The implementation method of the Transformer-based feature fusion method in step 2 and step 5 is: (1) The feature X of the last layer is obtained layer4 Use the attention mechanism in Transformer to calculate and obtain enhanced features (2) The features As the residual term and feature X layer4 Add up the features (3) The features Upsampling and channel compression are performed to obtain features The second layer feature X layer3 Channel compression to obtain features Then the feature and Features Add element by element to get the feature (4) Features Upsampling to get features Then the feature and the first layer feature X layer2 Add element by element to get the fused features The method of using Transformer in step 6 to perform feature enhancement in combination with the training set is: Step 1: Get the feature X of the last layer of the search area layer4 Use the attention mechanism in Transformer to calculate and obtain enhanced features Step 2: Add features As the residual term and feature X layer4 Add together to get Step 3: Calculate the obtained training set features S using the attention mechanism in Transformer to obtain the enhanced features Step 4: Add features As the residual term, add it to the feature S to get the feature Step 5: Using the features from step 2 Calculate the matrix Q and use the enhanced target features of the last layer Calculate the matrices K and V, and then use the attention formula to calculate the features Step 6: Add Features As the residual term and feature Add up the features 2. The Transformer-based single target tracking method according to claim 1, characterized in that: The calculation method of the attention mechanism in the Transformer is: 1) The features The features are obtained through convolution operation. After the Reshape() operation, we get the tensor Among them, C represents the number of channels of the feature, H represents the height of the feature, W represents the width of the feature, N = H × W, Conv() is the convolution function, and Reshape() represents the operation function that transforms the matrix into a matrix of dimension N; 2) The features The features are obtained through convolution operation. After the Reshape() operation, we get the tensor 3) The features After the Reshape() operation, we get the tensor 4) Use the calculated tensors Q, K, and V to input the attention formula to get the output of the attention formula: Among them, Softmax() represents the normalized exponential function, Attention() is the function of the attention formula, and the coefficient d k =H×W.

3. The Transformer-based single target tracking method according to claim 1 or 2, characterized in that: The regression model generator consists of a single-layer convolutional neural network, which processes the target features. and Input a single-layer convolutional neural network for convolution calculation to obtain a regression model M reg The classification model generator consists of a PrROIPooling layer, and the input features are pooled to obtain a classification model M cls ; The training set feature S is obtained by extracting features from the collected training set samples using the ResNet50 network.

4. The Transformer-based single target tracking method according to claim 3, characterized in that: The pre-training operation method is to use the samples of the training data set to input the classification model or regression model for forward propagation, then calculate the gradient information of each parameter in the classification model or regression model through back propagation, and then use the gradient information to update the parameters of the current classification model or regression model, so that the classification model or regression model fits the current training samples.

5. The Transformer-based single target tracking method according to claim 4, characterized in that: In step 5, the regression model M is used reg As the feature X after convolution kernel fusion reg Perform convolution operation to obtain the regression result m cls ; The step six classifies the model M cls As the feature X of the search area after convolution kernel and enhancement cls Perform convolution to obtain the classification result m cls .

6. The Transformer-based single target tracking method according to claim 5, characterized in that: The method for obtaining the bounding box in step 7 is: locate the position of the target according to the classification result, that is, the classification result m cls The position with the largest median value is mapped back to the search area to obtain the center position of the target; the offset values ​​of the upper, lower, left, and right sides of the target corresponding to the regression result relative to the center position are selected according to the position with the largest median value in the classification result; the bounding box is drawn according to the center position and offset value of the target.

7. The Transformer-based single target tracking method according to any one of claims 4 to 6, characterized in that: When tracking fails or the classification score is too low in step seven, the classification model and the regression model are updated by using the data training set through the gradient descent method.

8. The Transformer-based single target tracking method according to claim 1, characterized in that: The enhanced features for Among them, Q, K, and V are the characteristics of X layer4 The tensor computed using the attention mechanism in Transformer.

Images