Transformer object tracking method and tracker based on symmetric cross attention and position information enhancement

Abstract

The application discloses a kind of based on symmetric cross attention and position information enhancement's Transform target tracking method and tracker, method includes: template image and search area image are regarded as a pair of image input backbone network, and the feature map of template and search area is respectively extracted;The feature map of the template and search area is sent into feature fusion network, and the feature fusion network includes the encoder network, decoder network and position information enhancement module connected in turn, and the enhanced search area final feature vector is obtained after fusion, enhancement;The enhanced search area final feature vector is input into prediction head, and target tracking result is generated by classification function and the regression function of boundary box.The application solves the problem that the accuracy and tracking speed of existing target tracking need to be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual target tracking, and particularly relates to a Transformer target tracking method based on symmetric cross attention and position information enhancement and a tracker. BACKGROUND

[0002] Estimating the trajectory of an object in a video sequence is called visual object tracking. General visual object tracking is a basic and challenging task in computer vision. This task mainly predicts the position and state of the target in each frame of a video sequence by using the position of the target in the initial frame of the video. Therefore, the correct features of the target need to be extracted from the first frame to locate the target from the subsequent frames. At present, it is widely used in unmanned aerial vehicles, autonomous driving, monitoring and other fields. Due to its wide application, the field of target tracking has attracted more and more attention. The main challenges in the tracking field are light changes, deformations, occlusions, background clutter, and similar target interference. Many problems have not been well solved. In the face of these challenges, many researches have made great efforts in recent years. However, it is still challenging to design a high-accuracy and real-time tracking tracker model.

[0003] In the prior art, the tracking structure based on the twin network has attracted widespread attention due to its performance ability and simplicity, and has become very popular in the field of visual object tracking and has achieved very good performance. The siamese-based tracking framework formulates visual object tracking as a template matching problem. Simply put, tracking is to search for the region most similar to the target template. The cross-correlation operation between the template and the search region is fully utilized to accurately locate the target object in the search region. Most mainstream tracker frameworks first use a backbone network to extract the features of the template and the search region image, and then use a correlation network structure to calculate the similarity of the template and the search region, such as ATOM. The siamese tracker has achieved excellent tracking performance, especially in the balance between precision and speed.

[0004] The Transformer was first proposed in the field of Natural Language Processing (NLP) for machine translation tasks. By enabling each element to pay attention to all other elements, it improves the learning of long-range dependencies in neural network machine translation. The Transformer quickly replaced the LSTM model and has become the mainstream architecture for language modeling, even achieving great success in large-scale pre-trained models, such as the famous GPT. The Transformer, based on an attention mechanism, uses a combination of encoder and decoder to transform one sequence into another. Due to the use of the attention mechanism, it can acquire global information in processing language sequences. In sequential tasks like NLP, the Transformer has largely replaced recurrent neural networks and has shone brightly in computer vision, such as image classification, object detection, semantic segmentation, and multi-object tracking. Currently, there are two main types of transformer structures used in single-object tracking applications. The first type uses a convolutional neural network-based backbone to extract features from the template and search region, and then uses a transformer-based feature fusion structure to deeply fuse the features of the template and search region so that the predictor can better predict the bounding box of the target object, such as Transt. The second type does not use a convolutional backbone for feature extraction, but directly uses a transformer structure for feature extraction and fusion, making full use of the attention mechanism, and the overall structure is more compact, such as Mixtformer.

[0005] However, Siamese-based trackers have certain inherent drawbacks due to their use of cross-correlation operations. For example, cross-correlation operations often get stuck in local optima and lack access to global information, which may affect the prediction of the final target object bounding box, leading to inaccurate predictions. Moreover, the original Transformer has a very high computational cost, affecting the speed of target tracking. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] To address the aforementioned issues, this invention provides a Transformer target tracking method and tracker based on symmetric cross attention and position information enhancement, thereby resolving the problem that the accuracy and tracking speed of existing target tracking methods need to be improved.

[0008] (II) Technical Solution

[0009] To address the aforementioned technical problems, this invention provides a Transformer target tracking method based on symmetric cross-attention and positional information enhancement, comprising:

[0010] S1. Treat the template image and the search region image as a pair of images and input them into the backbone network to extract feature maps of the template and the search region respectively;

[0011] S2. The feature maps of the template and the search region are fed into the feature fusion network, which includes an encoder network, a decoder network and a location information enhancement module connected in sequence. After fusion and enhancement, the final feature vector of the enhanced search region is obtained.

[0012] S3. Input the final feature vector of the enhanced search region into the prediction head, and generate the target tracking result through the classification function and the bounding box regression function.

[0013] Furthermore, step S2 includes:

[0014] S20. Preprocess the feature maps of the template and the search region to convert them into feature vectors of the template and the search region;

[0015] S21. Input the feature vectors of the template and the search region into the encoder network. The encoder network uses a multi-layer symmetric cross attention module to fuse the features of the template and the search region to obtain the fused feature vectors of the template and the search region, respectively.

[0016] S22. The fused template and the feature vector of the search region are further fused by the decoder network to obtain the final feature vector of the search region;

[0017] S23. The final feature vector of the search region is input into the position information enhancement module for enhancement to obtain the enhanced final feature vector of the search region.

[0018] Furthermore, step S20 includes: flattening the feature maps of the template and the search region along the spatial dimension, adding sinusoidal position encoding, then using 1×1 convolution to reduce the channel dimension, and then flattening them along the spatial dimension to obtain the feature vectors of the template and the search region respectively.

[0019] Furthermore, in step S21, the symmetrical cross-attention module includes:

[0020] The feature vectors of the template and the search region, as well as the positional encodings corresponding to the template and the search region, are input into the multi-head cross-attention module to obtain the preliminary fused feature vector Xt:

[0021] X t =MultiHead(X zq +P zq X xk +P xk X xv ),

[0022] MultiHead(Q,K,V)=Concat(H1,...,H h W O ,

[0023]

[0024]

[0025] The feature vector of the template and the corresponding position code of the template are input into the multi-head cross-attention module for further fusion. After fusion, the feature vector and the corresponding position code are added to the feature vector of the template to obtain the fused template feature vector Xz. z After passing through the feedforward network FFN, it is connected to the X z The fused template feature vector is obtained by adding them together.

[0026] X z =X zq +MultiHead(X zq +P zq X tk +P tk X tv ),

[0027]

[0028] Simultaneously, the feature vector of the search region, the position code corresponding to the search region, the initially fused feature vector, and the corresponding position code are input into the multi-head cross-attention module for further fusion. This fusion is then added to the feature vector of the search region to obtain the fused search region feature vector X. x The X x After passing through the feedforward network FFN, it is connected to the X x The features of the fused search region are obtained by adding them together.

[0029] X x =X xq +MultiHead(X xq +P xq X tk +P tk X tv )

[0030]

[0031] in, W O Both are parameter matrices, X zq It is the input of the template branch, P zqIt is the position code corresponding to the template branch, X xk and X xv It is the input for the search region branch, P xk It is the position code corresponding to the branch in the search region, X tk and X tv It is the input of the feature vector branch of the initial fusion, P tk It is the positional encoding corresponding to the feature vectors of the initial fusion.

[0032] Furthermore, in step S22, the decoder network includes: inputting the fused template and the feature vector of the search region into a multi-head cross-attention module for further fusion, then passing it through a feedforward network FFN with an activation function inserted, performing residual connections between the multi-head cross-attention module and the feedforward network FFN, performing norm normalization processing, and outputting the final feature vector of the search region.

[0033] Furthermore, in step S23, the location information enhancement module includes:

[0034] The final feature vector of the search region is input and reshaped to obtain the final feature map X of the search region, which includes a feature map with C channels, a height H * width W.

[0035] Each feature map is subjected to one-dimensional horizontal global pooling and one-dimensional vertical global pooling to obtain intermediate feature maps p in the horizontal and vertical directions. h p w :

[0036]

[0037] The intermediate feature map in the vertical direction is then concatenated with the intermediate feature map in the horizontal direction along the spatial dimension after passing through the permute transformation function to obtain T. After a 1×1 convolution transformation and BatchNorm batch normalization, it is multiplied by the BatchNorm batch normalized feature map after passing through the ReLU linear rectified function to obtain the intermediate feature map f encoding the spatial information in the horizontal and vertical directions.

[0038] T = (Conv(Concat(p) h p w ))),

[0039] f t =φ(T),

[0040] f = T × f t ,

[0041] The intermediate feature map f is split into two independent height tensors along the spatial dimension. hand width tensor f w Then, after passing through 1×1 convolution and sigmoid activation function respectively, the height weight S is obtained. h and width weight S w Multiplying the transformed input of the location information enhancement module, the enhanced search region final feature map Y is output, which is then reshaped to output the enhanced search region final feature vector.

[0042] S h =τ(Conv(f) h )),

[0043] S w =τ(Conv(f) w )),

[0044]

[0045] Wherein, the subscript c represents the feature map corresponding to the c-th channel, i and j are the horizontal and vertical coordinates, τ() represents the sigmoid activation function, and φ() represents the ReLU linear rectified function.

[0046] Furthermore, in step S3, the classification network uses a binary cross-entropy loss function, and the bounding box regression network uses L1 loss and IOU loss functions.

[0047] Furthermore, in step S1, the template image and the search region image need to be preprocessed: the image patch of the template image is obtained by expanding the target bounding box of the first frame of the video sequence outward to twice the side length; the image patch of the search region image is obtained by expanding the target bounding box of the previous frame outward to four times the side length; the backbone network adopts a modified ResNet50 network, which removes the last stage and the fully connected layer of the ResNet50 network, and changes the convolution stride of the downsampling in the fourth stage from 2 to 1, and modifies the 3×3 convolution in the fourth stage to a dilated convolution with a stride of 2.

[0048] Furthermore, the symmetric cross-attention module has 4 layers; after obtaining the fused template feature vector X z , The fused search region feature vector X x , All values ​​were normalized using the norm method.

[0049] This invention also discloses a Transformer target tracker based on symmetric cross attention and positional information enhancement, comprising:

[0050] At least one processor; and at least one memory communicatively connected to said processor, wherein:

[0051] The memory stores program instructions that can be executed by the processor, and the processor can execute the method by calling the program instructions.

[0052] (III) Beneficial Effects

[0053] The above-described technical solution of the present invention has the following advantages:

[0054] (1) Based on the original transformer tracking structure, this invention improves the symmetric cross attention module and the position information enhancement module. First, the feature information of the template and the search area is extracted through the backbone network and then fed into the feature fusion network. The encoder network is superimposed with four symmetric cross attention modules, which can more effectively fuse the search area features and the template features. Then, the features are further fused through the decoder network. The position information enhancement module further enhances the fused feature information. Finally, the prediction head performs classification and bounding box regression based on the enhanced features to generate the final tracking result. Through multiple fusion and enhancement, the feature fusion of the template image and the search area image is better realized, which improves the recognition accuracy of target tracking. At the same time, the cross attention structure adopted can reduce the amount of computation and improve the tracking speed.

[0055] (2) This invention uses an attention mechanism to replace cross-correlation operation. Through the symmetrical cross-attention module, the features of the search region and the template are deeply fused to perform global information interaction, avoiding the situation of local optima. It can better fuse the features of the template image and the search region image and better judge the feature similarity between the template image and the search region image.

[0056] (3) The present invention makes full use of spatial location information through the location information enhancement module, which helps to better identify and locate targets in the prediction head stage, effectively improves the accuracy of target location by the prediction head, and makes the tracker perform better.

[0057] (4) The present invention can effectively cope with various challenges such as changes in illumination, deformation, scale changes, and interference, and provides high-precision, high-robust target tracking with better real-time performance and robustness, and is more suitable for visual single-target tracking. Attached Figure Description

[0058] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings:

[0059] Figure 1 This is an overall structural diagram of the Transformer target tracking method based on symmetric cross attention and position information enhancement according to an embodiment of the present invention;

[0060] Figure 2 This is a structural diagram of the symmetrical cross-attention module according to an embodiment of the present invention;

[0061] Figure 3 This is a structural diagram of the location information enhancement module according to an embodiment of the present invention;

[0062] Figure 4 This is a comparison chart of the tracker of this embodiment with other trackers on the GOT-10k dataset;

[0063] Figure 5 This is a comparison chart of the tracker of this embodiment with other trackers on the LaSOT dataset;

[0064] Figure 6 This is a comparison chart of the tracker of this embodiment with other trackers on the UAV123 dataset;

[0065] Figure 7 This is a comparison chart of the tracker of this embodiment with other trackers on the OTB100 dataset. Detailed Implementation

[0066] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0067] This invention provides a Transformer-based target tracking method based on symmetric cross-attention and positional information enhancement. Figure 1 As shown, it includes the following steps:

[0068] S1. Treat the template image and the search region image as a pair of images and input them into the backbone network to extract feature maps of the template and the search region respectively.

[0069] S10. Preprocess the template image and the image of the search area;

[0070] The template image and the search region image are treated as a pair of images and used as input to the backbone network. To include the target's appearance information and the surrounding scene, the template image patch is obtained by expanding the target's bounding box outward by twice its side length from the first frame of the video sequence. Since the target's movement range in the next frame is usually not particularly large, the search region image patch is obtained by expanding the target's bounding box outward by four times its side length from the previous frame. The template and search region images are then reshaped to be input into the backbone network for processing.

[0071] S11. Treat the preprocessed template image and the search region image as a pair of images and input them into the backbone network to extract feature maps of the template and the search region respectively.

[0072] The backbone network uses a modified ResNet50 network. The last stage and fully connected layer of the existing ResNet50 network are removed, and the stride of the convolution in the fourth stage of downsampling is changed from 2 to 1 to obtain a larger feature resolution. Simultaneously, the 3×3 convolution in the fourth stage is modified to a dilated convolution with a stride of 2 to increase the receptive field. The template image patch... and image patch of the search area After processing by Backbone, we can obtain the template feature map. and feature map of the search region C = 1024.

[0073] S2. The feature maps of the template and the search region are fed into the feature fusion network, which includes an encoder network, a decoder network and a location information enhancement module connected in sequence. After fusion and enhancement, the final feature vector of the enhanced search region is obtained.

[0074] S20. Preprocess the feature maps of the template and the search region to transform them into feature vectors of the template and the search region: flatten the feature maps of the template and the search region along the spatial dimension, add sinusoidal position encoding, then use 1×1 convolution to reduce the channel dimension, and then flatten them along the spatial dimension to obtain the feature vectors of the template and the search region respectively.

[0075] Before inputting the feature maps into the encoder, the feature maps output by the backbone network need to be preprocessed: first, the feature maps are transformed into feature vectors. Because the attention mechanism requires a set of feature vectors as input, we flatten the template feature map and search region feature map output by the backbone network along the spatial dimension to obtain a series of feature vectors, and add sinusoidal positional encoding. Then, we use 1×1 convolutions to reduce the channel dimension, reducing it from 1024 to 256, thus obtaining f. z0 and f x0 Then flatten f from a spatial dimension z0 and f x0 f was obtained z1 and f x1 f z1 and f x1 Can be considered Individual and Each feature vector is 256 in length, and these feature vectors are then used as inputs to the template branch and the search region branch in the encoder.

[0076] S21. Input the feature vectors of the template and the search region into the encoder network. The encoder network uses a 4-layer symmetric cross attention module to fuse the features of the template and the search region, and obtain the fused feature vectors of the template and the search region respectively.

[0077] The Symmetrical Cross-Attention Module (SCA) is as follows: Figure 2 As shown, it includes:

[0078] The feature vectors of the template and the search region, as well as the positional encodings corresponding to the template and the search region, are input into the multi-head cross-attention module. After normalization, a preliminary fused feature vector X is obtained. t :

[0079] The template's feature vector and its corresponding positional encoding are input into a multi-head cross-attention module for further fusion. This fusion is then added to the template's feature vector, and after normalization, the fused template feature vector X is obtained. z The X z After passing through the feedforward network FFN, it is connected to the X z The summation, followed by norm normalization, yields the fused template feature vector.

[0080] Simultaneously, the feature vector of the search region, the position code corresponding to the search region, the initially fused feature vector, and the corresponding position code are input into the multi-head cross-attention module for further fusion. This is then added to the feature vector of the search region, and after normalization, the fused search region feature vector X is obtained. x The X x After passing through the feedforward network FFN, it is connected to the X x The features of the fused search region are obtained by summing the features and normalizing them using norm.

[0081] The encoder network comprises four sequentially fused symmetric cross-attention modules. In these modules, a preliminary multi-head cross-attention operation is performed using the two inputs from the template branch and the search region branch to enhance feature fusion. The formula for the multi-head attention module is as follows:

[0082]

[0083] MultiHead(Q,K,V)=Concat(H1,...,H h W O

[0084]

[0085] here and They are all parameter matrices.

[0086] Here, the dimension of the parameter matrix is ​​h = 8, d m =256,d k =d v =d m / h=32.

[0087] The formula for a preliminary multi-head cross-attention operation is as follows:

[0088] X t =MultiHead(X zq +P zq X xk +P xk X xv )

[0089] here It is the input for the template branch. It is the corresponding position code. and It is the input for the search region branch. This is the positional encoding corresponding to this branch, with a dimension d = 256.

[0090] Then, the fused feature map is subjected to cross-attention again with the inputs from the previous template branch and search region branch, respectively, to fully integrate the features of the template and search region, thereby enhancing the ability to identify targets.

[0091] X z =X zq +MultiHead(X zq +P zq X tk +P tk X tv )

[0092]

[0093] X x =X xq +MultiHead(X xq +P xq X tk +P tk X tv )

[0094] We call this combination of cross-attention symmetrical cross-attention. and It is the output of our symmetrical cross attention.

[0095] The encoder network performs full feature fusion on the template image and the search region image, and the positional encoding helps the model distinguish between tags from different sources and different locations.

[0096] S22. The fused template and the feature vector of the search region are further fused by the decoder network to obtain the final feature vector of the search region;

[0097] The decoder network includes: inputting the fused template and the feature vector of the search region into a multi-head cross-attention module for further fusion, then passing it through a feedforward network FFN with an activation function inserted, performing residual connection between the multi-head cross-attention module and the feedforward network FFN, performing normalization processing, and outputting the final feature vector of the search region.

[0098] After fusing features from the template and search region using symmetric cross-attention in the encoder network, two feature maps are obtained. In the decoder network, multi-head cross-attention is used, taking these two feature maps as input and fusing them through a final multi-head cross-attention process. This is then passed through a feedforward network to enhance the model's fitting ability. The feedforward network (FFN) contains two multilayer perceptron layers with activation functions inserted. Residual connections are applied in both the multi-head cross-attention module and the FFN module, and normalization is performed after each residual connection. After fusion, a final feature map of the search region is obtained. This fused feature map is then input into a location information enhancement module, which effectively increases the ability to identify the target location without significantly increasing computational cost.

[0099] S23. The final feature vector of the search region is input into the location information enhancement module for enhancement to obtain the enhanced final feature vector of the search region.

[0100] The location information enhancement module PIE, such as Figure 3 As shown, the process includes: inputting the final feature vector of the search region, reshaping it to obtain the final feature map X of the search region, which includes a feature map with C channels and a height H * width W. C It is a two-dimensional matrix representation of the feature map of a certain channel within a feature map; each feature map is then subjected to one-dimensional horizontal global pooling and one-dimensional vertical global pooling to obtain the intermediate feature map p in the horizontal and vertical directions. h p wThe intermediate feature map in the vertical direction is then concatenated with the intermediate feature map in the horizontal direction along the spatial dimension after passing through the permute transformation function to obtain T. This T is then subjected to a 1×1 convolution transformation, batch normalized using BatchNorm, and multiplied with the batch normalized feature map after passing through the ReLU linear rectified function, resulting in an intermediate feature map f encoding the spatial information in the horizontal and vertical directions. This intermediate feature map f is then split along the spatial dimension into two independent height tensors f. h and width tensor f w Then, after passing through 1×1 convolution and sigmoid activation function respectively, the height weight S is obtained. h and width weight S w The enhanced search region is multiplied by the deformed input of the location information enhancement module to output the final feature map Y of the enhanced search region, which is then reshaped to output the final feature vector of the enhanced search region.

[0101] Unlike channel attention, the location information enhancement module also considers encoding spatial information. We perform a one-dimensional horizontal global pooling and a one-dimensional vertical global pooling along the horizontal and vertical directions, respectively. Global pooling is typically used in channel attention to enhance spatial information encoding capabilities. However, global pooling compresses global spatial information into a single channel descriptor, making it difficult to retain location information. This is crucial for capturing the target's spatial location in tracking tasks. To obtain more accurate location information across the spatial region, we use two spatial range pooling operations, one along the horizontal coordinate and the other along the vertical coordinate. Therefore, the output of the c-th channel at height h can be represented as...

[0102]

[0103] Similarly, the output of the c-th channel with width w can be written as

[0104]

[0105] Here, k represents either the kth row or the kth column.

[0106] These two transformations along different spatial directions aggregate features, resulting in a pair of direction-aware feature maps. Unlike previous global pooling, these two transformations allow our attention to capture long-range dependencies along one spatial direction and retain precise location information along the other. Each element in both feature maps reflects whether the target object of interest exists in the corresponding row and column. This enables our model to more accurately locate the target object.

[0107] The aggregated feature maps generated by one-dimensional global pooling are first concatenated, and then a 1×1 convolution transformation is performed on them. The formula is as follows:

[0108] T = (Conv(Concat(p) h p w ))),

[0109] f t =d(T),

[0110] f = T × f t ,

[0111] The splicing operation here is performed along the spatial dimension. This non-linear activation function yields a... The intermediate feature map encodes the horizontal and vertical spatial information. Here, r controls the block size. Then, we split the feature map f into two independent tensors f along the spatial dimensions. h and f w Then apply two 1×1 convolutions to f h and f w The input is converted to a tensor with the same number of channels as the previous input, and then the sigmoid activation function is applied to each tensor. The formula is as follows:

[0112] S h =τ(Conv(f) h )),

[0113] S u =τ(Conv(f) w )),

[0114] We will output S h and S w As weights for attention, the output of our position information enhancement module Y can finally be written as:

[0115]

[0116] Wherein, the subscript c represents the feature map corresponding to the c-th channel, i and j are the horizontal and vertical coordinates, k is the sigmoid activation function, and φ() represents the ReLU linear rectified function.

[0117] S3. Input the final feature vector of the enhanced search region into the prediction head, and generate the target tracking result through the classification function and the bounding box regression function.

[0118] The prediction header receives H x ×W x eigenvectors, output H x ×W xFor the binary classification and regression results, we select the predicted feature vectors corresponding to the ground-truth bounding boxes as positive samples, and the rest as negative samples. The class label for positive samples is foreground, and the class label for negative samples is background. All samples are used for classification loss, while only positive samples are useful for regression loss, so that the feature vectors can predict the target at the corresponding location. We use the standard binary cross-entropy loss for classification, defined as:

[0119]

[0120] For regression, we use L1 loss and the general IOU loss; the regression loss function is defined as:

[0121]

[0122] The above methods were validated on several authoritative datasets, including GOT-10k, LaSOT, TrackingNet, OTB100, UAV123, and VOT2020. Tables 1-4 and... Figures 4-7 The metrics comparisons with other methods demonstrate that the tracker performed better using this method.

[0123] Table 1 provides detailed data comparing the trackers on the GOT-10k dataset with other trackers.

[0124]

[0125] Table 2 provides detailed data comparing the trackers on the TrackingNet dataset.

[0126]

[0127] Table 3 provides detailed data comparing it with other trackers on the LaSOT dataset.

[0128]

[0129] Table 4 provides detailed data comparing the trackers with other trackers on the OTB100 and UAV123 datasets.

[0130]

[0131] Finally, it should be noted that the above methods can be converted into software program instructions. They can be implemented using a Transformer target tracker based on symmetric cross-attention and positional information enhancement, including a processor and memory, or by computer instructions stored in a non-transitory computer-readable storage medium. The integrated unit implemented as a software functional unit can be stored in a computer-readable storage medium. This software functional unit, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] In summary, the Transformer target tracking method and tracker based on symmetric cross attention and positional information enhancement described above have the following beneficial effects:

[0133] (1) Based on the original transformer tracking structure, this invention improves the symmetric cross attention module and the position information enhancement module. First, the feature information of the template and the search area is extracted through the backbone network and then fed into the feature fusion network. The encoder network is superimposed with four symmetric cross attention modules, which can more effectively fuse the search area features and the template features. Then, the features are further fused through the decoder network. The position information enhancement module further enhances the fused feature information. Finally, the prediction head performs classification and bounding box regression based on the enhanced features to generate the final tracking result. Through multiple fusion and enhancement, the feature fusion of the template image and the search area image is better realized, which improves the recognition accuracy of target tracking. At the same time, the cross attention structure adopted can reduce the amount of computation and improve the tracking speed.

[0134] (2) This invention uses an attention mechanism to replace cross-correlation operation. Through the symmetrical cross-attention module, the features of the search region and the template are deeply fused to perform global information interaction, avoiding the situation of local optima. It can better fuse the features of the template image and the search region image and better judge the feature similarity between the template image and the search region image.

[0135] (3) The present invention makes full use of spatial location information through the location information enhancement module, which helps to better identify and locate targets in the prediction head stage, effectively improves the accuracy of target location by the prediction head, and makes the tracker perform better.

[0136] (4) The present invention can effectively cope with various challenges such as changes in illumination, deformation, scale changes, and interference, and provides high-precision, high-robust target tracking with better real-time performance and robustness, and is more suitable for visual single-target tracking.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit it; although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A Transformer target tracking method based on symmetric cross-attention and positional information enhancement, characterized in that, include: S1. Treat the template image and the search region image as a pair of images and input them into the backbone network to extract feature maps of the template and the search region respectively; S2. The feature maps of the template and the search region are fed into the feature fusion network, which includes an encoder network, a decoder network and a location information enhancement module connected in sequence. After fusion and enhancement, the final feature vector of the enhanced search region is obtained. The location information enhancement module includes: The final feature vector of the search region is input and reshaped to obtain the final feature map X of the search region, which includes a feature map with C channels, a height H * width W. Each feature map is subjected to one-dimensional horizontal global pooling and one-dimensional vertical global pooling to obtain intermediate feature maps p in the horizontal and vertical directions. h p w : The intermediate feature map in the vertical direction is then concatenated with the intermediate feature map in the horizontal direction along the spatial dimension after passing through the permute transformation function to obtain T. After a 1×1 convolution transformation and BatchNorm batch normalization, it is multiplied by the BatchNorm batch normalized feature map after passing through the ReLU linear rectified function to obtain the intermediate feature map f encoding the spatial information in the horizontal and vertical directions. T=(Conv(Concat(p h ,p w ))), f t =φ(T), f=T×f t , The intermediate feature map f is split into two independent height tensors along the spatial dimension. h and width tensor f w Then, after passing through 1×1 convolution and sigmoid activation function respectively, the height weight S is obtained. h and width weight S w Multiplying the transformed input of the location information enhancement module, the enhanced search region final feature map Y is output, which is then reshaped to output the enhanced search region final feature vector. S h =τ(Conv(f h )), S w =τ(Conv(f w )), Where, the subscript c represents the feature map corresponding to the c-th channel, i and j are the horizontal and vertical coordinates, k represents the k-th row or the k-th column, τ() represents the sigmoid activation function, and φ() represents the ReLU linear rectifier function; S3. Input the final feature vector of the enhanced search region into the prediction head, and generate the target tracking result through the classification function and the bounding box regression function.

2. The Transformer target tracking method based on symmetric cross attention and position information enhancement according to claim 1, characterized in that, Step S2 includes: S20. Preprocess the feature maps of the template and the search region to convert them into feature vectors of the template and the search region; S21. Input the feature vectors of the template and the search region into the encoder network. The encoder network uses a multi-layer symmetric cross attention module to fuse the features of the template and the search region to obtain the fused feature vectors of the template and the search region, respectively. S22. The fused template and the feature vector of the search region are further fused by the decoder network to obtain the final feature vector of the search region; S23. The final feature vector of the search region is input into the position information enhancement module for enhancement to obtain the enhanced final feature vector of the search region.

3. The Transformer target tracking method based on symmetric cross-attention and position information enhancement according to claim 2, characterized in that, Step S20 includes: flattening the feature maps of the template and the search region along the spatial dimension, adding sinusoidal position encoding, then using 1×1 convolution to reduce the channel dimension, and then flattening them along the spatial dimension to obtain the feature vectors of the template and the search region respectively.

4. The Transformer target tracking method based on symmetric cross attention and position information enhancement according to claim 2, characterized in that, In step S21, the symmetrical cross-attention module includes: The feature vectors of the template and the search region, as well as the positional encodings corresponding to the template and the search region, are input into the multi-head cross-attention module to obtain the preliminary fused feature vector X. t : X t =MultiHead(X zq +P zq ,X xk +P xk ,X xv ), MultiHead(Q,K,V)=Concat(H1,...,H h )W O , The template's feature vector and its corresponding positional encoding are input into a multi-head cross-attention module for further fusion. This fusion is then added to the template's feature vector to obtain the fused template feature vector X. z The X z After passing through the feedforward network FFN, it is connected to the X z The fused template feature vector is obtained by adding them together. X z =X zq +MultiHead(X zq +P zq ,X tk +P tk ,X tv ), Simultaneously, the feature vector of the search region, the position code corresponding to the search region, the initially fused feature vector, and the corresponding position code are input into the multi-head cross-attention module for further fusion. This fusion is then added to the feature vector of the search region to obtain the fused search region feature vector X. x The X x After passing through the feedforward network FFN, it is connected to the X x The features of the fused search region are obtained by adding them together. X x =X xq +MultiHead(X xq +P xq ,X tk +P tk ,X tv ) in, W O Both are parameter matrices, X zq It is the input of the template branch, P zq It is the position code corresponding to the template branch, X xk and X xv It is the input for the search region branch, P xk It is the position code corresponding to the branch in the search region, Z. tk and X tv It is the input of the feature vector branch of the initial fusion, P tk It is the positional encoding corresponding to the feature vectors of the initial fusion.

5. The Transformer target tracking method based on symmetric cross attention and position information enhancement according to claim 2, characterized in that, In step S22, the decoder network includes: inputting the fused template and the feature vector of the search region into a multi-head cross-attention module for further fusion, then passing it through a feedforward network FFN with an activation function inserted, performing residual connection between the multi-head cross-attention module and the feedforward network FFN, performing norm normalization, and outputting the final feature vector of the search region.

6. The Transformer target tracking method based on symmetric cross attention and position information enhancement according to claim 1, characterized in that, In step S3, the classification network uses the binary cross-entropy loss function, and the bounding box regression network uses L1 loss and IOU loss functions.

7. The Transformer target tracking method based on symmetric cross attention and position information enhancement according to claim 1, characterized in that, In step S1, the template image and the search region image need to be preprocessed: the image patch of the template image is obtained by expanding the target bounding box of the first frame of the video sequence outward to twice the side length; the image patch of the search region image is obtained by expanding the target bounding box of the previous frame outward to four times the side length; the backbone network adopts a modified ResNet50 network, which removes the last stage and the fully connected layer of the ResNet50 network, and changes the convolution stride of the downsampling in the fourth stage from 2 to 1, and modifies the 3×3 convolution in the fourth stage to a dilated convolution with a stride of 2.

8. The Transformer target tracking method based on symmetric cross attention and position information enhancement according to claim 4, characterized in that, The symmetric cross-attention module has 4 layers; after obtaining the fused template feature vector X z , The fused search region feature vector X x , All values ​​were normalized using the norm method.

9. A Transformer target tracker based on symmetric cross attention and positional information enhancement, characterized in that, include: At least one processor; and at least one memory communicatively connected to the processor, wherein: The memory stores program instructions that can be executed by the processor, and the processor can invoke the program instructions to perform the method as described in any one of claims 1 to 8.

Images