A Transformer-based spatiotemporal context target tracking method and system

Abstract

The present invention discloses a Transformer-based spatiotemporal context target tracking method and system, which are as follows: S1, image acquisition and preprocessing; S2, inputting the image into a backbone network, and obtaining search area features, initial template features and dynamically updated template features through a Transformer encoder; S3, using the output as the input of an interactive feature enhancement module, and adopting a multi-head cross self-attention mechanism to obtain mixed features; S4, using the mixed features and target queries as inputs of a Transformer decoder, wherein the mask self-attention mechanism part in the Transformer decoder adopts a multi-head self-attention mechanism layer; after obtaining the output of the Transformer decoder, calculating the similarity between the output and the mixed feature embedding and performing feature reshaping, and calculating the expectation of the probability distribution of the corner points to obtain a boundary prediction box; S5, using the output obtained by the Transformer decoder as the input of a scoring head, the scoring head consists of a fully connected layer FFN and a softmax activation function, and setting a threshold to determine whether to perform template update.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and mainly relates to technologies such as target tracking, feature enhancement and feature fusion, and specifically relates to a Transformer-based spatiotemporal context target tracking method or system. Background Art

[0002] Target tracking is an important branch of computer vision and a core technology in video surveillance, intelligent transportation, autonomous driving, and other fields. After continuous research and development in recent years, target tracking technology can be roughly divided into two categories: generative and discriminative. The processing idea of ​​the generative model is to first establish a target model or extract target features, and then search for similar features in subsequent frames to gradually iterate and achieve target positioning; however, the disadvantage of this method is that the background information of the image is not fully utilized, and the establishment of the model will be affected by motion blur, target deformation, and occlusion. The discriminative model introduces background information into the tracking model, extracts the target model by comparing the difference between the target model and the background information, and obtains the position of the current frame. This can address the shortcomings of the generative model to a certain extent and better achieve target tracking.

[0003] The Transformer architecture is a neural network model based on the self-attention mechanism. It has been widely used in the field of computer vision in recent years, especially after the proposal of Vision Transformer. The advantage of the self-attention mechanism in Transformer is that it can capture the global dependencies in the input sequence and is not limited to the local context; and the position encoding can embed the position information into the input sequence, so that the self-attention mechanism can better model the global context relevance. Based on the above status quo, the present invention adopts Transformer as a feature extractor and feature fusion network, which can combine spatiotemporal context information to avoid falling into local optimality. This tracking model based entirely on Transformer can cope with many challenges such as deformation, occlusion, etc. in the case of long-term tracking, thereby improving the robustness of the target tracking system. Summary of the Invention

[0004] In response to the above situation, the present invention discloses a spatiotemporal context target tracking method and system based on Transformer. The main contents of the present invention are as follows: (1) In order to better obtain spatial context information and avoid the lack of global perception and interaction caused by the limited receptive field of the Resnet network, the present invention uses Vision Transformer as the backbone network to extract the features of the image, thereby achieving a mixture of local and global information and enhancing feature extraction. (2) In order to avoid losing semantic information and falling into local optimality when matching the template and the search area, the present invention uses an interactive feature enhancement module for feature fusion, and utilizes the cross-attention mechanism to enhance the representation and generalization capabilities of the model by modeling the characteristics between different input sequences. (3) In order to better locate the bounding box of the target, the bounding box corner probability prediction is used, and a target query is introduced in the input part of the Transformer decoder to calculate the similarity between the mixed features and the decoder output, and finally the predicted box is obtained by calculating the expectation of the corner probability distribution. (4) In order to cope with the challenges of occlusion and deformation encountered during long-term target tracking, the present invention introduces temporal information to dynamically update the template, which better combines the spatiotemporal context information. In order to reduce post-processing hyperparameters, FFN and softmax functions are used to complete the foreground and background classification, and a threshold is set. When the calculated score is greater than the threshold, the template is updated, otherwise it is not updated.

[0005] The present invention adopts the following technical solutions:

[0006] A Transformer-based spatiotemporal context object tracking method includes the following steps:

[0007] S1. Image acquisition and preprocessing:

[0008] Obtain the tracking target image and perform preprocessing.

[0009] S2. Backbone network extracts features:

[0010] The preprocessed image in step S1 is input into the backbone network Vision Transformer, where it is first flattened and linearly mapped, and then added to the corresponding position code to obtain the slice embedding layer. The search area features, initial template features, and dynamically updated template features are then obtained through the Transformer encoder.

[0011] S3, feature enhancement and fusion:

[0012] The output of step S2 is used as the input of the interactive feature enhancement module, which is improved by the Transformer decoder. The original mask self-attention mechanism is deleted, and only the multi-head cross self-attention mechanism is used to make the search area features query the initial template features and the dynamic template features, and finally obtain the output mixed features, which strengthens the interaction of global features.

[0013] S4. Bounding box prediction:

[0014] The hybrid feature obtained in step S3 and a target query are used as input to the Transformer decoder. The masked self-attention mechanism in the Transformer decoder is partially replaced with a multi-head self-attention mechanism layer, adaptively focusing on useful contextual information to enhance feature representation. After obtaining the output of the Transformer decoder, the similarity between this output and the hybrid feature embedding is calculated and the features are reshaped. Finally, the expected probability distribution of the corner points is calculated to obtain the bounding box prediction.

[0015] S5. Score head prediction:

[0016] The output of the Transformer decoder in step S4 is used as the input of the scoring head. The essence of the scoring head is to classify the foreground and background in the image. Here, the scoring head consists of an FFN and a softmax activation function. Finally, a set threshold is used to determine whether to update the template.

[0017] Preferably, in step S1, the present invention uses the GOT-10K dataset to train and verify the model. The input part consists of three parts: the search area, the initial template, and the dynamically updated template. Before inputting into the backbone network, the input image needs to be preprocessed and the image is divided into slices before inputting into the backbone network to reduce the number of parameters and improve the training speed of the model.

[0018] Preferably, in step S1, the image size of the search area R is the area, C is the number of channels, H x is the length of the image, W x is the width of the picture. The preferred search area size of the present invention is 256×256×3. The initial template frame and the dynamically updated template frame are The preferred size of the present invention is 128×128×3. And the search area is expanded from the target's center coordinates in the previous frame to four directions by several times the target size, including the target's possible movement range; first pre-process the search area, decompose a frame of image into Each slice is input into the backbone network as a token. The size of each slice is n×n×C. The initial template area and the dynamically updated template area are decomposed into The size of each slice is n×n×C. In the present invention, n is preferably 16 and the number of slices is 256.

[0019] Preferably, in step S2, each slice is first flattened and linearly mapped into a vector of dimension n×n×C, and then the generated position code is added to the corresponding position of the slice to obtain a slice embedding layer, and then the obtained slice embedding layer is input into the Transformer encoder.

[0020] Preferably, in step S2, the Transformer encoder first passes through a layer normalization, enters the multi-head self-attention layer, then passes through a residual network structure, and finally undergoes normalization again and a multi-layer perceptron to obtain the output of the encoder part, that is, the feature vector of the extracted image.

[0021] Preferably, in step S3, the search area features obtained in the backbone network are used as queries, and the fusion features X2 obtained by concatenating the initial template features and the dynamically updated template features are input together as key-value pairs into the interactive feature enhancement module.

[0022] Preferably, in step S3, the interactive feature enhancement module adopts a multi-head cross attention mechanism, wherein, given a query Q, a key K and a value V, the attention function adopts a scaled dot product, and the formula is as follows:

[0023]

[0024] Where, Q = X1 × W Q ,K=X2×W K ,V=X2×W V ,W Q 、W K 、W V The parameter matrices are obtained from the training data and are used to extract features. X1 is the search area feature, X2 is the fusion feature obtained by concatenating the initial template feature and the dynamic update template feature, and K T is the transposed matrix of key K, d k is the dimension of matrix K; the multi-head attention mechanism is set to several layers because there is more than one kind of feature correlation, which is cascaded by the results of several single heads and then combined with the matrix W O Multiplying, the formula is as follows:

[0025] MultiHead(Q,K,V)=Concat(H 1,...., H n )W O

[0026] Among them, the matrix W O is the learnable parameter matrix, H n For the nth multi-head layer. In the vector matrix H of multiple single-head outputs i After cascading, through W O Extract features from several vector matrices, where H i The output result for each single head is as follows:

[0027] H i =Attention(X1W i Q ,X2W i K ,X2W i V )

[0028] Where W i Q 、W i K 、W i V The i-th layer parameter matrix is ​​obtained from the training data. The query Q and the key K are spatially encoded and added accordingly. Then, they are input into the multi-head cross attention mechanism and connected with the residual through normalization. The formula is as follows:

[0029]

[0030] Among them, P q The spatial position encoding of the corresponding query Q part, P k The spatial position encoding of the corresponding key K part.

[0031] Then perform normalization and residual connection to obtain the final output mixed features. The formula is as follows:

[0032]

[0033] Preferably, in step S4, after obtaining the output of the Transformer decoder, the similarity between it and the hybrid feature vector embedding is calculated, and the obtained similarity score is multiplied element-by-element with the hybrid feature to enhance important areas and weaken less distinguishable areas; the new feature sequence is reshaped into a feature map; and then two corner point probabilities P are output through the fully convolutional network. tl (x,y) and P br (x,y); where x and y are coordinate points, P tl (x,y) is the probability distribution of the upper left corner point, P br(x, y) probability distribution of the lower right corner; the fully convolutional network structure consists of L stacked convolutional layers, batch normalization layers, and ReLU functions; finally, the expected probability distribution of the corner points is calculated to obtain the predicted box coordinates, as shown in the following formula:

[0034]

[0035]

[0036] Model the uncertainty in coordinate estimation to generate more accurate and robust prediction results for target tracking. After obtaining the predicted box, the loss function L1 loss and IOU loss are used in conjunction with the auxiliary network model for bounding box prediction. The formula is as follows:

[0037]

[0038] Among them, b i and are the bounding box label and the predicted bounding box, respectively. λ is the weight hyperparameter between the two loss functions, which adjusts the relative importance between the two.

[0039] Preferably, in step S5, the threshold is set to 0.7, and the output confidence score of the scoring head is compared with the threshold. When the score is greater than 0.7, the template is updated, otherwise it is not updated; the new template is cropped from the search area image and then input into the backbone network for feature extraction; the binary cross entropy loss function is used when optimizing the scoring head, and the formula is as follows:

[0040] L cls =y i log(P i )+(1-y i )log(1-P i )

[0041] Among them, y i Represents 0 or 1 in binary label, P i belongs to y i The probability of the label; when P i The closer the value of is to 1, the closer the value of the loss function is to 0. On the contrary, the predicted value P i The closer the loss function is to 0, the larger it is.

[0042] The present invention also discloses a Transformer-based spatiotemporal context target tracking system, wherein the above method comprises the following modules:

[0043] Image acquisition and preprocessing module: acquires the tracking target image and performs preprocessing;

[0044] Backbone network feature extraction module: The image preprocessed by the image acquisition and preprocessing modules is input into the backbone network Vision Transformer, where it is first flattened and linearly mapped. The slice embedding layer is then added to the position code. The search area features, initial template features, and dynamically updated template features are then obtained through the Transformer encoder.

[0045] Feature enhancement and fusion module: The output of the backbone network feature extraction module is used as the input of the interactive feature enhancement module. The interactive feature enhancement module uses a multi-head cross self-attention mechanism to make the search area features query the fusion features of the initial template features and the dynamic template features to obtain hybrid features;

[0046] Bounding Box Prediction Module: The hybrid features obtained by the feature enhancement and fusion modules and a target query are used as inputs to the Transformer decoder. The masked self-attention mechanism in the Transformer decoder adopts a multi-head self-attention mechanism layer to adaptively focus on useful contextual information. After obtaining the output of the Transformer decoder, the similarity between the output and the hybrid feature embedding is calculated and the features are reshaped. Finally, the expected probability distribution of the corner points is calculated to obtain the bounding box prediction.

[0047] Score Head Prediction Module: The output of the Transformer decoder in the bounding box prediction module is used as the input of the score head. The score head consists of a fully connected layer FFN and a softmax activation function. Finally, a set threshold is used to determine whether to update the template.

[0048] The present invention discloses a Transformer-based spatiotemporal context target tracking method and system, which has the following beneficial technical effects:

[0049] (1) First, the present invention improves the feature extraction part of the backbone network and uses Vision Transformer to replace the CNN model, which improves the model's ability to extract global information, reduces the model's sensitivity to post-processing hyperparameters, and improves the model training speed and effect.

[0050] (2) Secondly, the present invention proposes an interactive feature enhancement module, which queries and matches the search area features with the initial template features and the dynamically updated template features, and finally obtains the fused features, thereby strengthening the correlation between the features of the fused template and the search area.

[0051] (3) Regression and classification are split into two stages: bounding box prediction and score head prediction. Time series information is introduced, which not only strengthens the fusion features spatially but also fully utilizes contextual semantic information. To address challenges such as deformation and occlusion brought about by long-term tracking, this paper sets a threshold in the score head to identify template updates. The updated template is used together with the initial template as the model input, improving the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is the overall framework diagram of the network model involved in the Transformer-based spatiotemporal context target tracking method in the preferred embodiment of the present invention.

[0053] Figure 2 This is the structure diagram of the Vision Transformer model.

[0054] Figure 3 This is the structure diagram of the interactive enhancement module.

[0055] Figure 4 This is the Transformer decoder structure diagram.

[0056] Figure 5 Schematic diagram of the boundary prediction framework.

[0057] Figure 6 Schematic diagram of the scoring head prediction framework.

[0058] Figure 7 A block diagram of a Transformer-based spatiotemporal context target tracking system is provided for a preferred embodiment of the invention. DETAILED DESCRIPTION

[0059] The present invention will be further described below with reference to the accompanying drawings and specific implementation cases, but it should be noted that the present invention is not limited to the following embodiments.

[0060] The environment of the preferred embodiment of the present invention is as follows: cuda10.2, deep learning framework pytorch1.7.0, Intel core i7-12700H CPU, 16G memory and GPU is NVIDIA GeForce RTX 3050, with 4G video memory. Figure 1 Give a detailed introduction to the entire network model.

[0061] This embodiment provides a Transformer-based spatiotemporal context object tracking method, and its specific steps are as follows:

[0062] S1. Image acquisition and preprocessing:

[0063] First, for tracking target data acquisition, this embodiment uses the GOT-10K dataset, which is large-scale and diverse, and provides challenging tracking scenarios. Its multiple evaluation indicators can better evaluate the tracking model. According to the characteristics of the Vision Transformer model, some preprocessing operations are required before inputting the network model, including the image size of the search area. Initial template frame and dynamically updated template frame size The search area is expanded from the target's center coordinates in the previous frame to four times the target size in four directions, usually including the target's possible movement range. First, the search area is preprocessed and a frame of image is decomposed into the number of Each slice is input into the backbone network as a token. The size of each slice is n×n×C. The initial template area and the dynamically updated template area are decomposed into The size of each slice is also n×n×C.

[0064] S2. Backbone network extracts features:

[0065] The present invention proposes to delete the last part of the original Vision Transformer, MLP Head and Class, because this step is used for image classification operations, and the Vision Transformer was originally proposed for image classification tasks. The pre-processed data is input into the backbone network Vision Transformer. First, each slice is flattened and linearly mapped into a vector of dimension n×n×C. Then, the generated position code is added to the corresponding position of the slice to obtain the slice embedding layer. Here, the tracking task is performed, so there is no need to generate CLS tokens. Then, the obtained slice embedding layer is input into the Transformer encoder. Here, it is combined with Figure 2 The encoder part in the Vision Transformer model structure diagram is described in detail. It first passes through a layer of normalization and enters the multi-head self-attention layer. In this embodiment, it is set to 6 layers. Then it passes through a residual network structure and finally undergoes normalization again and a multi-layer perceptron to obtain the output of the encoder part, that is, the feature vector of the extracted image.

[0066] S3. Interactive Enhancement Features:

[0067] The search area feature X1 obtained in the backbone network is used as the query, and the fusion feature X2 obtained by cascading the initial template feature and the dynamically updated template feature is input into the interactive enhancement module as a key-value pair. Figure 3To provide a detailed description, this module is essentially a masked self-attention mechanism with partial deletion of the Transformer decoder. It employs a multi-head cross-attention mechanism to enable the search region feature X1 to query the fused feature X2 of the initial template and the dynamically updated template. This strengthens the feature interaction between the fused template and the search region in the tracking framework, further leveraging global feature information. The following formula describes the module's functionality. The core of the module is the multi-head cross-attention mechanism. Given a query Q, a key K, and a value V, the attention function uses a scaled dot product, as shown in the following formula:

[0068]

[0069] Where, Q = X1 × W Q ,K=X2×W K ,V=X2×W V ,W Q 、W K 、W V Both are parameter matrices obtained from training data, and their function is to extract features. k The dimension of the matrix K is the multi-head attention mechanism because there is more than one kind of feature correlation. In this embodiment, it is set to 6 layers, which is cascaded by the results of 6 single heads and then combined with the matrix W. O Multiply. The formula is as follows:

[0070] MultiHead(Q,K,V)=Concat(H 1,...., H n )W O

[0071] Matrix W O is a learnable parameter matrix, which is the vector matrix H of multiple single-head outputs i After cascading, through W O Extract features from these 6 vector matrices, where H i The output result for each single head is as follows:

[0072] H i =Attention(X1W i Q ,X2W i K ,X2W i V )

[0073] The interactive feature enhancement module of the present invention differs from the Transformer decoder in position encoding. The present invention only performs spatial position encoding on the query Q and key K, adds them together, then inputs the multi-head cross attention mechanism, and connects them with the residual through normalization. The formula is as follows:

[0074]

[0075] In addition, like the forward feedback network FFN in the Transformer decoder, it is used to enhance the model fitting ability, and then normalization and residual connection are performed again to obtain the final output mixed features. The formula is as follows:

[0076]

[0077] S4. Bounding box prediction head:

[0078] The output mixed feature vector obtained by the interactive enhancement module and a target query are used as the input of the Transformer decoder. Figure 4 To give a detailed description, first of all, this module is essentially the Transformer decoder part that uses the multi-head self-attention mechanism to replace the masked multi-head attention mechanism, because the mask matrix introduced by the masked multi-head attention mechanism is mainly used to process variable-length sequences and is a serial output. The input length of the present invention is fixed, and the use of the multi-head self-attention mechanism enhances the model's ability to express multiple points of interest and provides the advantage of parallel computing. In the position encoding part, the position encoding is shared with the interactive enhancement module. Different from the Transformer decoder: the present invention uses position encoding for the multi-head cross-attention mechanism part, and also only performs spatial position encoding and corresponding addition on the query Q and the key K, while in the multi-head self-attention mechanism part, the target query itself is added and self-attention operations are performed. Among them, the multi-head self-attention mechanism part is to learn the features of the anchor box, and the multi-head cross-attention mechanism part is to predict the coordinates and category of the bounding box of the image based on the global feature information of the image and the feature information of the anchor box. In this way, after obtaining the output of the Transformer decoder, it is combined with the mixed feature vector to calculate the similarity between the embeddings of the two. Here, combined with Figure 5 The similarity score is then multiplied element-wise with the mixed feature to enhance the important areas and weaken the less distinguishable areas. The new feature sequence is reshaped into a feature map. The fully convolutional network then outputs two corner point probabilities P tl (x,y) and P br (x, y), representing the upper left corner and lower right corner respectively. The fully convolutional network structure consists of L stacked convolutional layers, batch normalization layers, and ReLU functions. Finally, the expected probability distribution of the corner points is calculated to obtain the predicted box. The formula is as follows:

[0079]

[0080]

[0081] This embodiment models the uncertainty in coordinate estimation to generate more accurate and robust prediction results for target tracking. After obtaining the predicted box, this embodiment uses the loss function L1 loss and IOU loss combined with the auxiliary network model to predict the bounding box. The formula is as follows:

[0082]

[0083] Among them, b i and are the bounding box label and the predicted bounding box, respectively. λ is the weight hyperparameter between the two loss functions, which adjusts the relative importance between the two.

[0084] S5. Score Head Prediction:

[0085] Based on the challenges of occlusion and deformation of the target during long-term tracking, the present invention introduces temporal information into the model structure, because using only the initial template frame information as a reference may cause the tracker to be unable to adapt to the rapidly changing appearance of the target and have poor robustness. A dynamic template design is adopted here. Specifically, the dynamic template will be continuously updated over time and used as the input of the network model together with the search area and the initial template. This can provide more temporal information for the entire framework. However, the target may be occluded and deformed during the tracking process, which will make the dynamic template unreliable. Therefore, a threshold needs to be set to determine whether the dynamic template is updated based on the confidence score. The design of the scoring head takes the output of the Transformer decoder as input, and the scoring head structure consists of a fully connected layer and a softmax activation function. The threshold part is set to 0.7. Finally, the output confidence score of the scoring head is compared with the threshold. When the score is greater than 0.7, the template is updated, otherwise it is not updated. The new template is cropped from the search area image and then input into the backbone network for feature extraction. Bounding box prediction is essentially to locate the target, while the scoring head is to classify the target and background. This embodiment divides the two into two stages. The binary cross entropy loss function is used to optimize the scoring head. The formula is as follows:

[0086] L cls =y i log(P i )+(1-y i )log(1-P i )

[0087] Among them, y i Represents 0 or 1 in binary label, P i belongs to y i The probability of the label. When P i The closer the value of is to 1, the closer the value of the loss function is to 0. On the contrary, the predicted value Pi The closer the loss function is to 0, the larger it is.

[0088] like Figure 7 As shown, the preferred embodiment of the present invention further discloses a Transformer-based spatiotemporal context target tracking system, which is based on the above method embodiment and specifically includes the following modules:

[0089] Image acquisition and preprocessing module: acquires the tracking target image and performs preprocessing;

[0090] Backbone network feature extraction module: The image preprocessed by the image acquisition and preprocessing modules is input into the backbone network Vision Transformer, where it is first flattened and linearly mapped. The slice embedding layer is then added to the position code. The search area features, initial template features, and dynamically updated template features are then obtained through the Transformer encoder.

[0091] Feature enhancement and fusion module: The output of the backbone network feature extraction module is used as the input of the interactive feature enhancement module. The interactive feature enhancement module uses a multi-head cross self-attention mechanism to make the search area features query the fusion features of the initial template features and the dynamic template features to obtain hybrid features;

[0092] Bounding Box Prediction Module: The hybrid features obtained by the feature enhancement and fusion modules and a target query are used as inputs to the Transformer decoder. The masked self-attention mechanism in the Transformer decoder adopts a multi-head self-attention mechanism layer to adaptively focus on useful contextual information. After obtaining the output of the Transformer decoder, the similarity between the output and the hybrid feature embedding is calculated and the features are reshaped. Finally, the expected probability distribution of the corner points is calculated to obtain the bounding box prediction.

[0093] Score Head Prediction Module: The output of the Transformer decoder in the bounding box prediction module is used as the input of the score head. The score head consists of a fully connected layer FFN and a softmax activation function. Finally, a set threshold is used to determine whether to update the template.

[0094] For other contents of this embodiment, please refer to the above method embodiment.

[0095] Those skilled in the art should recognize that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of protection of the present invention should not be limited to the contents disclosed in the embodiments.

Claims

1. A Transformer-based spatiotemporal contextual target tracking method, characterized by The steps include: S1. Image acquisition and preprocessing: Acquire the tracking target image and perform preprocessing; S2, backbone network feature extraction: input the preprocessed image in step S1 into the backbone network Vision In the Transformer, flattening and linear mapping operations are first performed, and then the slice embedding layer is obtained by adding the corresponding position code. Then, the search area feature, initial template feature and dynamic update template feature are obtained through the Transformer encoder respectively. S3, feature enhancement and fusion: The output of step S2 is used as the input of the interactive feature enhancement module, which uses a multi-head cross self-attention mechanism to make the search area feature query the fusion feature of the initial template feature and the dynamic template feature to obtain a hybrid feature; S4, bounding box prediction: The mixed features obtained in step S3 and a target query are used as the input of the Transformer decoder. The masked self-attention mechanism in the Transformer decoder adopts a multi-head self-attention mechanism layer to adaptively focus on useful contextual information. After obtaining the output of the Transformer decoder, the similarity between the embedding of the output and the mixed feature vector is calculated and the features are reshaped. Finally, the expected probability distribution of the corner points is calculated to obtain the bounding box prediction. S5. Score header prediction: The output of the Transformer decoder in step S4 is used as the input of the score header. The score header consists of a fully connected layer FFN and a softmax activation function. Finally, a set threshold is used to determine whether to update the template.

2. The method for tracking a target in a spatiotemporal context based on a Transformer according to claim 1, wherein: In step S1, the GOT-10K data set is used, and the input part consists of a search area, an initial template, and a dynamically updated template; the preprocessing is to divide the image into slices.

3. The method for tracking a target in a spatiotemporal context based on a Transformer according to claim 2, wherein: In step S1, the image size of the search area R is the area, C is the number of channels, H x is the length of the image, W x is the width of the image, the initial template frame and the dynamically updated template frame are The search area is expanded from the target's center coordinates in the previous frame to four directions by several times the target size, including the target's possible movement range; the search area is preprocessed first, and a frame of image is decomposed into the number Slices, each slice is input into the backbone network as a token, and the size of each slice is n×n×C; The initial template area and the dynamically updated template area are decomposed into the following quantities: The size of each slice is n×n×C.

4. A Transformer-based spatiotemporal context target tracking method according to claim 2 or 3, characterized in that: In step S2, each slice is first flattened and linearly mapped into a vector of dimension n×n×C. The generated position code is then added to the corresponding position of the slice to obtain a slice embedding layer, which is then input into the Transformer encoder.

5. The method for tracking a target in a spatiotemporal context based on a Transformer according to claim 4, wherein: In step S2, the Transformer encoder first passes through a layer of normalization, enters the multi-head self-attention layer, then passes through a residual network structure, and finally undergoes normalization again and a multi-layer perceptron to obtain the output of the encoder part, that is, the feature vector of the extracted image.

6. A Transformer-based spatiotemporal context target tracking method according to any one of claims 1 to 3, characterized in that: In step S3, the search area feature X1 obtained in the backbone network is used as a query, and the fusion feature X2 obtained by cascading the initial template feature and the dynamically updated template feature is input into the interactive feature enhancement module as a key-value pair.

7. The method for tracking a target in a spatiotemporal context based on a Transformer according to claim 6, wherein: In step S3, the interactive feature enhancement module adopts a multi-head cross attention mechanism, where, given the query Q, key K and value V, the attention function adopts a scaled dot product, and the formula is as follows: Where, Q = X1 × W Q ,K=X2×W K ,V=X2×W V ,W Q 、W K 、W V The parameter matrices are obtained from the training data and are used to extract features. X1 is the search area feature, X2 is the fusion feature obtained by concatenating the initial template feature and the dynamic update template feature, and K T is the transposed matrix of key K, d k is the dimension of matrix K; the multi-head attention mechanism is set to several layers because there is more than one kind of feature correlation, which is cascaded by the results of several single heads and then combined with the matrix W O Multiplying, the formula is as follows: MultiHead(Q,K,V)=Concat(H 1,...., H n )W O Among them, the matrix W O is the learnable parameter matrix, H n For the nth multi-head layer, the vector matrix H of multiple single-head outputs i After cascading, through W O Extract features from several vector matrices, where H i The output result for each single head is as follows: H i =Attention(X1W i Q ,X2W i K ,X2W i V ) Among them, W i Q 、W i K 、W i V The i-th layer parameter matrix is ​​obtained from the training data. The query Q and the key K are spatially encoded and added accordingly. Then, they are input into the multi-head cross attention mechanism and connected with the residual through normalization. The formula is as follows: Among them, P q The spatial position encoding of the corresponding query Q part, P k The spatial position encoding of the corresponding key K part; Then perform normalization and residual connection to obtain the final output mixed features. The formula is as follows:

8. The method for tracking a target in a spatiotemporal context based on a Transformer according to any one of claims 1 to 3, wherein: In step S4, after obtaining the output of the Transformer decoder, the similarity between the embedding of the Transformer decoder and the hybrid feature vector is calculated, and the obtained similarity score is multiplied element-by-element with the hybrid feature to enhance the important areas and weaken the less distinguishable areas; the new feature sequence is reshaped into a feature map; and then the two corner point probabilities P are output through the fully convolutional network. tl (x,y) and P br (x,y), where x and y are coordinate points, P tl (x,y) is the probability distribution of the upper left corner point, P br (x, y) probability distribution of the lower right corner; the fully convolutional network structure consists of L stacked convolutional layers, batch normalization layers, and ReLU functions; finally, the expected probability distribution of the corner points is calculated to obtain the predicted box coordinates, as shown in the following formula: Model the uncertainty in coordinate estimation to generate more accurate and robust prediction results for target tracking. After obtaining the predicted box, the loss function L1 loss and IOU loss are used in conjunction with the auxiliary network model for bounding box prediction. The formula is as follows: Among them, b i and are the bounding box label and the predicted bounding box, respectively. λ is the weight hyperparameter between the two loss functions, which adjusts the relative importance between the two.

9. The method for tracking a target in a spatiotemporal context based on a Transformer according to any one of claims 1 to 3, wherein: In step S5, the threshold is set to 0.7, and the confidence score of the output of the scoring head is compared with the threshold. If the score is greater than 0.7, the template is updated, otherwise it is not updated. The new template is cropped from the search area image and then input into the backbone network for feature extraction. The binary cross entropy loss function is used to optimize the scoring head, and the formula is as follows: L cls =y i log(P i )+(1-y i )log(1-P i ) Among them, y i Represents 0 or 1 in binary label, P i belongs to y i The probability of the label; when P i The closer the value of is to 1, the closer the value of the loss function is to 0. On the contrary, the predicted value P i The closer the loss function is to 0, the larger it is.

10. A Transformer-based spatiotemporal context target tracking system, based on the method according to any one of claims 1 to 9, characterized in that Includes the following modules: Image acquisition and preprocessing module: acquires the tracking target image and performs preprocessing; Backbone network feature extraction module: The image preprocessed by the image acquisition and preprocessing modules is input into the backbone network Vision Transformer, where it is first flattened and linearly mapped. The slice embedding layer is then added to the position code. The search area features, initial template features, and dynamically updated template features are then obtained through the Transformer encoder. Feature enhancement and fusion module: The output of the backbone network feature extraction module is used as the input of the interactive feature enhancement module. The interactive feature enhancement module uses a multi-head cross self-attention mechanism to make the search area features query the fusion features of the initial template features and the dynamic template features to obtain hybrid features; Bounding Box Prediction Module: The hybrid features obtained by the feature enhancement and fusion modules and a target query are used as input to the Transformer decoder. The masked self-attention mechanism in the Transformer decoder adopts a multi-head self-attention mechanism layer to adaptively focus on useful contextual information. After obtaining the output of the Transformer decoder, the similarity between the embedding of the output and the hybrid feature vector is calculated and the features are reshaped. Finally, the expected probability distribution of the corner points is calculated to obtain the bounding box prediction. Score Head Prediction Module: The output of the Transformer decoder in the bounding box prediction module is used as the input of the score head. The score head consists of a fully connected layer FFN and a softmax activation function. Finally, a set threshold is used to determine whether to update the template.

Images