A spatiotemporal joint UAV image target tracking method

Abstract

This invention discloses a spatiotemporally integrated UAV image target tracking method, belonging to the field of aerial image processing technology. The method includes the following steps: Step 1, constructing a backbone network based on the VIT model; Step 2, constructing a network framework for a single-stream VIT model; Step 3, constructing a loss function based on contrastive learning; Step 4, adding a dynamic template update mechanism; Step 5, constructing two sequence decoder structures; Step 6, inputting the image and past sequences into the network to obtain the final tracking result. This invention provides a spatiotemporally integrated UAV image target tracking method. In the air domain, it designs a single-stream VIT-based backbone network and a contrastive learning-based loss function for small targets; in the temporal domain, it designs a dynamic template update mechanism and a trajectory prediction-based decoder to address background interference. These two structures together form a spatiotemporally integrated tracking algorithm, effectively improving the success rate and accuracy of target tracking from the UAV's perspective.

Description

Technical Field

[0001] This invention belongs to the field of aerial image processing technology, and in particular relates to a method for tracking UAV images based on spatiotemporal coordination. Background Technology

[0002] With the rapid development of science and technology, unmanned aerial vehicle (UAV) technology, as an important technological field, has shown broad application prospects in military, civilian, and scientific research fields. Especially from the perspective of UAVs, target tracking technology, as one of the key components of UAV missions, has increasingly attracted the attention of researchers and engineers. Traditional target tracking methods often exhibit limitations when facing complex and ever-changing environments and target situations. However, in recent years, the rise of deep learning technology has provided new ideas and tools for solving this problem.

[0003] In UAV (Unmanned Aerial Vehicle) ground target tracking missions, target tracking faces numerous challenges. UAVs often operate in complex and ever-changing environments, such as cities, mountains, and forests. These environments are filled with various factors, including occlusion and changing lighting, making traditional target tracking methods often ill-suited to these complex conditions. Furthermore, due to their flight altitude, UAVs face many challenges when performing ground target tracking tasks, including small objects, cluttered backgrounds, weather conditions, bird's-eye view, rapid camera / object movement, sharp camera rotation, and significant viewpoint changes. Targets captured by UAVs at this altitude generally have weak features and small scale. In most cases, due to limited pixels, even humans find it difficult to track small objects in complex backgrounds. Many UAV missions require real-time execution, such as search and rescue and border patrols. In some applications, UAVs need to track targets for extended periods, such as surveillance and tracking. Therefore, algorithms need to possess good target persistence, capable of accurately tracking small targets over long periods. Summary of the Invention

[0004] The purpose of this invention is to provide a spatiotemporal joint UAV image target tracking method to solve the problems of low target resolution, weak features, small scale, and susceptibility to target occlusion, shape changes, and background interference in complex backgrounds in existing technologies, which lead to discontinuous tracking.

[0005] To achieve the above objectives, this invention provides a spatiotemporal joint UAV image target tracking method, comprising the following steps:

[0006] Step 1: Build a backbone network based on the VIT model to extract visual features of small targets. The VIT model divides the input image into fixed-size image blocks through two-dimensional convolution, increases the dimension of the hidden dimension through linear transformation, and inputs it into the transformer for feature extraction and downstream tasks.

[0007] Step 2: For the target tracking task, enhance the utilization of spatial information in the tracking image and build a network framework for a single-stream VIT model.

[0008] Step 3: Construct a loss function based on contrastive learning to distinguish between the foreground and background of the tracked image;

[0009] Step 4: Add a dynamic template update mechanism to update the status of the tracked target using time-domain information during the tracking process;

[0010] Step 5: Construct two sequence decoder structures from both the temporal and spatial domains;

[0011] Step 6: Combine temporal and spatial information, input the image and past sequences into the network, and obtain the final tracking result.

[0012] Preferably, the specific process of building the backbone network based on the VIT model in step 1 is as follows:

[0013] Step 101: First, the input 3D image x in the form of H*W*C is segmented, where H, W, and C represent the height, width, and dimension of the image, respectively. The model uses a 2D convolution transformation. By setting the size of the convolution kernel and the stride, the image x is segmented into multiple non-overlapping 2D image blocks. The specific formula is as follows:

[0014]

[0015] Where x represents the input image, i represents the image patch number, and x p N represents an image patch, and N represents the number of image patches;

[0016] The input H*W*C 3D image x is transformed into a 2D image block encoding with dimension N*D. The specific transformation process is illustrated by the following formula:

[0017] (H*W*C)→N*(P 2 *C)→N*D

[0018] Where N represents the number of image patches in the sequence, (P 2 *C) represents the dimension of each image patch, P is the size of the image patch, C is the dimension of the initial image, D is the dimension of the hidden layer, and "→" represents the transformation process;

[0019] Step 102: Use a learnable positional encoding with dimension N*D. Add the N*D positional encoding to the image block encoding and input it into the Transformerblock to enhance the positional information of different image blocks.

[0020] Step 103: Construct Transformerblock. Transformerblock is the basic component of the VIT model. The input of the VIT model is processed by several stacked Transformerblocks to obtain the final output. Each Transformerblock consists of a self-attention module and a multilayer perceptron (MLP).

[0021] Preferably, the specific process of constructing the Transformer block is as follows: First, a self-attention module is constructed. The attention mechanism assigns different weights to the input information based on the current query. The attention module contains three vectors: query vector, key vector, and value vector. For a given query vector, the k corresponding key vectors are matched by the inner product. The inner product is then normalized by softmax to obtain the k weights. The output of the attention module is the weighted average of the value vectors corresponding to the k key vectors. If the query vector, key vector, and value vector are all obtained by linear transformation of an input sequence, then it becomes self-attention. Next, an MLP module is constructed. The MLP module consists of a linear layer and an activation function.

[0022] Preferably, the specific process of building the network framework of the single-stream VIT model in step 2 is as follows:

[0023] Step 201: Based on the backbone network of the VIT model, adjust the input. First, for a dimension of 3*H... X *W X The search region X and dimension H are 3*H Z *W Z Template Z, where H X W X and H Z W Z Representing the height and width of the search region and the height and width of the template respectively, they are arranged as N X *(3*P 2 ) and N Z *(3*P 2 A sequence of image patches, where P represents the width and height of an image patch, P*P is the resolution of each image patch, and N is the resolution of the image patch. z =H z W z / P 2 N x =H x W x / P 2The number of image patches for the template and the search region are respectively. Then, a learnable one-dimensional linear projection layer E is used to project each image patch of the template and the search region to the D dimension.

[0024] Step 202: Add learnable positional coding, using a learnable one-dimensional positional coding mechanism P. Z and P X , representing the positional codes of the template and the search region, respectively, are added to the corresponding image patch codes to obtain the final template sequence H. Z and search region sequence H X The visual characteristics are defined by the following formula:

[0025]

[0026]

[0027] Step 203: Process the obtained template sequence H Z and search region sequence H X The visual features are spliced ​​together and attention is calculated to obtain the template sequence H. Z and search region sequence H X Connecting them along the 0th dimension, the final vector has a dimension of (N). X +N Z )*D, represented as H ZX =[H Z H X As the input to the encoder in the subsequent transformer, due to the self-attention mechanism in the transformer, after concatenating the template sequence and the search region sequence and inputting them into the transformer, the network can simultaneously complete two tasks: image feature extraction and modeling the relationship between the template and the search region. The formula for the self-attention mechanism suitable for tracking is as follows:

[0028]

[0029] Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively; the subscripts z and x represent matrix items belonging to the template and search region, and D is the hidden layer dimension.

[0030] Step 204: Expand the formula obtained in step 203 to obtain the following formula:

[0031]

[0032]

[0033] Q Z K ZT V Z and Q X K X T V X Q represents self-attention on the template and search region, respectively, and is responsible for image feature extraction. Z K X T V X and Q X K Z T V Z These represent cross-attention to the template and the search region, respectively, and are responsible for aggregating image features, which is equivalent to modeling the relationship between the template and the search region;

[0034] Step 205: Build the overall network framework of the single-stream VIT model based on the target tracking task.

[0035] Preferably, the specific process of constructing the loss function based on contrastive learning in step 3 is as follows:

[0036] Step 301: First, construct the loss function based on SimCLR, with the following specific expression:

[0037]

[0038] Where sim(z) i ,z j ) represents the cosine similarity with respect to examples, sim(z) i ,z k ) represents the cosine similarity of the positive examples and the cosine similarity of the remaining negative examples, Z represents the input image, i, j, and k represent the image numbers respectively, exp represents the exponential operation with base e, T is an adjustable parameter, and M represents the number of positive examples plus the number of negative examples.

[0039] Step 302: Construct a contrastive learning-based loss function for the tracking scenario based on the loss function from step 301. The specific formula is as follows:

[0040]

[0041] Where L represents the total loss, T is an adjustable parameter, M represents the number of positive examples plus the number of negative examples, Z represents the input image, x represents the current frame number, xi and xj represent the i-th and j-th frame numbers before the current frame, respectively, k is the total number of past frames, sim represents cosine similarity calculation, exp represents exponential operation with base e, mask represents masking operation, sim(z x-i ,z x-j) represents the cosine similarity with respect to examples, sim(z) x-i ,mask(z x-j )) represents the cosine similarity of opposing examples.

[0042] Preferably, the specific process of adding a dynamic template update mechanism in step 4 is as follows:

[0043] Step 401: Add a dynamic template to the network input that is continuously updated as tracking progresses. This template is obtained by cropping the current search region from the target bounding box predicted by the network, and the initial value is the same as the static template.

[0044] Step 402: Add a confidence score head to the network. The confidence score head includes a three-layer perceptron and an activation function. If the score of the confidence score head is higher than the set threshold, the tracking is considered accurate and the dynamic template is updated. Otherwise, the template is not updated.

[0045] Preferably, the specific process of constructing the two sequence decoder structures in step 5 from both the time domain and spatial domain perspectives is as follows:

[0046] Step 501: Construct the initialization decoder. The input to the initialization decoder is the four coordinates of the target initialization and the template sequence H output in step 2. Z and search region sequence H X The visual features are used to perform cross-attention operations, where the four initial coordinates are used as q values, and the visual features are used as k and v values. The q, k, and v values ​​are all values ​​used in the attention operation.

[0047] Step 502: Construct a sequence decoder. The input to the sequence decoder is the coordinate features obtained from the previous decoder and the previously tracked coordinate sequence. The coordinate form is (X1, Y1, X2, Y2), where X1 and Y1 represent the coordinates of the upper left corner of the target bounding box, and X2 and Y2 represent the coordinates of the lower right corner of the target bounding box.

[0048] Step 503: Combine the current coordinates with the previously tracked coordinate sequence, perform self-attention calculation, find the relationship between the past tracking trajectory and the current coordinates, and use the past tracking trajectory as a guide to predict the coordinate position of the current target.

[0049] Preferably, the specific process of combining temporal and spatial information in step 6, inputting the image and past sequences into the network to obtain the final tracking result, is as follows:

[0050] Step 601: Construct the network's decoder using the two sequence decoders obtained in Step 5. The overall structure of the network's decoder consists of 6 initial encoders and L+1 sequence encoders, where 2 LTo utilize the length of past sequences, different sequence encoders employ a number of past coordinates ranging from 2... 0 to 2 L Increasing;

[0051] Step 602: Perform coordinate transformation on the input sequence. Using the upper left corner of the cropped area of ​​the previous frame of the current tracked image as the origin of the relative coordinate system, perform coordinate transformation on the input sequence to convert the absolute coordinates in the image into coordinates relative to the previous frame.

[0052] Step 603: Input the search area to be tracked, the template, and the coordinates of the previously tracked trajectory into the network structure. The output coordinates obtained by the network are inversely transformed to the absolute coordinate system with the upper left corner of the image as the origin according to the position of the previous frame, so as to obtain the final tracking result.

[0053] Therefore, the present invention employs the above-mentioned spatiotemporal joint UAV image target tracking method, which has the following beneficial effects:

[0054] (1) Enhancing the network's ability to utilize spatial information is beneficial for tracking targets with low resolution, weak features, and small scale.

[0055] (2) Enhancing the network's ability to utilize time-domain information is beneficial for tracking targets affected by complex background interference.

[0056] (3) Improve the model’s robustness to tracking deviations, making tracking more stable.

[0057] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0058] Figure 1 This is an overall flowchart of a UAV image target tracking method based on spatiotemporal integration according to the present invention;

[0059] Figure 2 This is a schematic diagram of the single-stream Transformer network framework based on target tracking tasks used in this invention;

[0060] Figure 3 This is a schematic diagram of the loss function based on contrastive learning used in this invention;

[0061] Figure 4 This is a schematic diagram of the dynamic template update mechanism used in this invention;

[0062] Figure 5 These are structural diagrams of the two sequence decoders used in this invention;

[0063] Figure 6 This is a schematic diagram of the overall structure of the decoder used in this invention. Detailed Implementation

[0064] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0065] Please see Figure 1-6 A spatiotemporal joint UAV image target tracking method includes the following steps:

[0066] Step 1: Construct a backbone network based on the VIT (Visual Attention) model to extract visual features of small targets. The VIT model divides the input image into fixed-size image patches through two-dimensional convolution, increases the hidden dimension through linear transformation, and inputs it into a transformer for feature extraction and downstream tasks. Since CNN (Convolutional Neural Network) based methods can only process a local neighborhood of the image at a time, and operations such as convolution and downsampling lead to the loss of image detail, and the target resolution in drone-based target tracking tasks is usually low and easily affected by background interference, CNN-based models struggle to extract discriminative features from low-resolution images, making tracking prone to deviations. To overcome the detail loss problem of CNN-based methods, this invention, for drone-based target tracking tasks, adopts a transformer-based target tracking framework, ensuring that all transformer layers have a global receptive field, preserving the detail information of the tracked target. The specific process of constructing the backbone network based on the VIT model is as follows:

[0067] Step 101: First, the 3D image x in the input form of H*W*C is segmented, where H, W, and C represent the height, width, and dimension of the image, respectively. The model uses a 2D convolution transformation. By setting the size of the convolution kernel and the stride, the image x is segmented into multiple non-overlapping 2D image blocks. The specific formula is as follows:

[0068]

[0069] Where x represents the input image, i represents the image patch number, and x p N represents an image patch, and N represents the number of image patches;

[0070] The input H*W*C 3D image x is transformed into a 2D image block encoding with dimension N*D. The specific transformation process is illustrated by the following formula:

[0071] (H*W*C)→N*(P 2 *C)→N*D

[0072] Where N represents the number of image patches in the sequence, (P 2 *C) represents the dimension of each image patch, P is the size of the image patch, C is the dimension of the initial image, D is the dimension of the hidden layer, and "→" represents the transformation process;

[0073] Step 102: Use a learnable positional encoding with dimension N*D. Add the N*D positional encoding to the image block encoding and input it into the Transformerblock to enhance the positional information of different image blocks.

[0074] Step 103: Construct Transformer Blocks. Transformer blocks are the fundamental components of the VIT model. The input of the VIT model is processed through several stacked Transformer blocks to obtain the final output. Each Transformer block consists of a self-attention module and a multilayer perceptron (MLP). The specific process of constructing Transformer blocks is as follows: First, construct the self-attention module. The attention mechanism assigns different weights to the input information based on the current query. The attention includes three vectors: query vector, key vector, and value vector. For a given query vector, the inner product is used to calculate the matching k corresponding key vectors. The inner product is then normalized using softmax to obtain k weights. The output of the attention is the weighted average of the value vectors corresponding to the k key vectors. This attention mechanism is called Scaled Dotproduct Attention. If Q, K, and V are all obtained by linear transformation of an input sequence, then it becomes self-attention. This invention uses multi-head self-attention (MSA), defining h attention heads. The hidden layer dimension D is divided into h parts in the input, and a total of h self-attention calculations are performed. Next, an MLP module is constructed, which consists of linear layers and activation functions. The purpose is to increase the model capacity and make the network more likely to converge.

[0075] Step 2: For target tracking tasks, enhance the utilization of spatial information in the tracking image by constructing a one-stream VIT model network framework. Currently, most target tracking for small targets adopts the common Siamese neural network framework. The Siamese network extracts features from the search region and template through two neural networks sharing weights, mapping them to a new feature space. Then, DW convolution is used, with the template as the convolution kernel on the search region, thus transforming the tracking problem into a similarity comparison problem. However, the separation of feature extraction and similarity relationship modeling has certain limitations. Since the feature extraction network is trained offline, the template and search region do not interact during feature extraction in the tracking stage, while the target is constantly changing, resulting in limited ability of the algorithm to recognize the target background. Background information is extremely important for tracking small targets. This application introduces a one-stream structure, concentrating feature extraction and similarity relationship modeling within the network framework constructed in Step 1. This approach effectively enhances the interaction between the visual features of the tracking image template and the search region when tracking small targets, improving the utilization of spatial information in the tracking image. The specific approach involves concatenating the search region and the template, and then inputting them together into the Transformer structure. The output search region features can be directly used for target classification and regression without further matching with template features. This structure enables dynamic extraction of both template and search region features, thereby giving the model strong discriminative capabilities. The specific process is as follows:

[0076] Step 201: Based on the backbone network of the VIT model, adjust the input. First, for a dimension of 3*H... X *W X The search region X and dimension H are 3*H Z *W Z Template Z, where H X W X and H Z W Z The width and height of the search area and the width and height of the template are respectively arranged as N. X *(3*P 2 ) and N Z *(3*P 2 A sequence of image patches, where P represents the width and height of an image patch, P*P is the resolution of each image patch, and N is the resolution of the image patch. z =H z W z / P 2 N x =H x W x / P 2The number of image patches for the template and the search region are respectively. Then, a learnable one-dimensional linear projection layer E is used to project each image patch of the template and the search region to the D dimension.

[0077] Step 202: Add learnable positional coding, using a learnable one-dimensional positional coding mechanism P. Z and P X , representing the positional codes of the template and the search region, respectively, are added to the corresponding image patch codes to obtain the final template sequence H. Z and search region sequence H X The visual characteristics are defined by the following formula:

[0078]

[0079]

[0080] Step 203: Process the obtained template sequence H Z and search region sequence H X The visual features are spliced ​​together and attention is calculated to obtain the template sequence H. Z and search region sequence H X Connecting them along the 0th dimension, the final vector has a dimension of (N). X +N Z )*D, represented as H ZX =[H Z H X As the input to the encoder in the subsequent transformer, due to the self-attention mechanism in the transformer, after concatenating the template sequence and the search region sequence and inputting them into the transformer, the network can simultaneously complete two tasks: image feature extraction and modeling the relationship between the template and the search region. The formula for the self-attention mechanism suitable for tracking is as follows:

[0081]

[0082] Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively; the subscripts z and x represent matrix items belonging to the template and search region, and D is the hidden layer dimension.

[0083] Step 204: Expand the formula obtained in step 203 to obtain the following formula:

[0084]

[0085]

[0086] Q Z K ZT V Z and Q X K X T V X Q represents self-attention on the template and search region, respectively, and is responsible for image feature extraction. Z K X T V X and Q X K Z T V Z These represent cross-attention to the template and the search region, respectively, and are responsible for aggregating image features, which is equivalent to modeling the relationship between the template and the search region;

[0087] Step 205: Construct the overall network framework for a single-stream VIT model based on the target tracking task, such as... Figure 2 As shown.

[0088] Step 3: Construct a loss function based on contrastive learning to distinguish between the foreground and background of the tracked image. Contrastive learning is a self-supervised algorithm that does not require a dedicated architecture or memory library. This algorithm inputs the image to be detected as a positive example and the data-augmented image as a negative example into the loss function. Compared to supervised learning algorithms, this algorithm benefits from stronger data augmentation and is suitable for large training batches and longer training times. Furthermore, like supervised learning, it can be widely applied in deeper networks. To further distinguish between the foreground and background in the tracking image, reduce the difference between foreground elements, and increase the difference between foreground and background, this application proposes a loss function based on contrastive learning. This loss function utilizes foreground and background information from the current frame and historical frames, using the foreground as a positive example and the background as a negative example. This makes the network's learning of the difference between the foreground and background of the target image more stable and effectively reduces background interference. Moreover, this loss function only operates during network training and does not participate in calculations during network inference, improving the algorithm's real-time performance. The specific process is as follows:

[0089] Step 301: First, construct the loss function based on SimCLR, with the following specific expression:

[0090]

[0091] Where sim(z) i ,z j ) represents the cosine similarity with respect to examples, sim(z) i ,z k) represents the cosine similarity of the positive examples and the cosine similarity of the remaining negative examples, Z represents the input image, i, j, and k represent the image numbers respectively, exp represents the exponential operation with base e, T is an adjustable parameter, and M represents the number of positive examples plus the number of negative examples.

[0092] Step 302: Construct a contrastive learning-based loss function for the tracking scenario based on the loss function from step 301. The specific formula is as follows:

[0093]

[0094] Where L represents the total loss, T is an adjustable parameter, M represents the number of positive examples plus the number of negative examples, Z represents the input image, x represents the current frame number, xi and xj represent the i-th and j-th frame numbers before the current frame, respectively, k is the total number of past frames, sim represents cosine similarity calculation, exp represents exponential operation with base e, mask represents masking operation, sim(z x-i ,z x-j ) represents the cosine similarity with respect to examples, sim(z) x-i ,mask(z x-j The cosine similarity ()) represents the similarity of opposing examples. The following is a detailed explanation of the formula:

[0095] The numerator is the sum of all positive examples, where a positive example consists of any two images from the (k+1) images formed by the current frame and the previous k frames. The denominator is the sum of positive and negative examples, where a negative example consists of any one positive image (from the (k+1) images formed by the current frame and the previous k frames) and one negative image (the image after masking the positive image). The masking operation involves setting the regions containing the target in the search area to 0 according to the coordinates of the target's bounding box. A schematic diagram of the loss function based on contrastive learning is shown below. Figure 3 As shown.

[0096] Step 4: Add a dynamic template update mechanism to update the state of the tracked target using temporal information during the tracking process. To effectively utilize temporal information during tracking and prevent the effects of background interference and target deformation, this application proposes a dynamic template update mechanism. During image tracking, the network input includes not only the static template obtained from the initial frame but also a dynamic template that is continuously updated as tracking progresses. This template is obtained by cropping the current search area from the target bounding box predicted by the network, and its initial value is the same as the static template. This approach can update the state of the tracked target in a timely manner, making it less prone to deviation when the tracking is subjected to abnormal interference. The specific process is as follows:

[0097] Step 401: Add a dynamic template to the network input that is continuously updated as tracking progresses. This template is obtained by cropping the current search region from the target bounding box predicted by the network, and the initial value is the same as the static template.

[0098] Step 402: Add a confidence score header to the network. The confidence score header consists of a three-layer perceptron and an activation function. If the score obtained through the confidence score header is higher than a set threshold, the tracking is considered accurate, and the dynamic template is updated. Otherwise, the template is not updated. A schematic diagram of the overall dynamic template update mechanism is shown below. Figure 4 As shown.

[0099] Step 5: Construct two sequence decoder structures from both the temporal and spatial domains. To effectively utilize the temporal and spatial information during the tracking process, this invention constructs two sequence decoders: one is an initialization decoder, which aims to represent coordinate features using the target's visual features, utilizing spatial features; the other is a sequence decoder, which aims to find the relationship between past trajectories and current coordinates, utilizing temporal features. The specific process is as follows:

[0100] Step 501: Construct the initialization decoder. The input to the initialization decoder is the four coordinates of the target initialization and the template sequence H output in step 2. Z and search region sequence H X The visual features are used to perform cross-attention operations, where the four initial coordinates are used as q values, and the visual features are used as k and v values. The q, k, and v values ​​are all values ​​used in the attention operation.

[0101] Step 502: Construct a sequence decoder. The input to the sequence decoder is the coordinate features obtained from the previous decoder and the previously tracked coordinate sequence. The coordinate form is (X1, Y1, X2, Y2), where X1 and Y1 represent the coordinates of the upper left corner of the target bounding box, and X2 and Y2 represent the coordinates of the lower right corner of the target bounding box.

[0102] Step 503: Combine the current coordinates with the previously tracked coordinate sequence, perform self-attention calculation, find the relationship between the past tracking trajectory and the current coordinates, and use the past tracking trajectory as a guide to predict the coordinate position of the current target.

[0103] The structural diagrams of the two decoders are as follows: Figure 5 As shown in the diagram. Shaded circles represent current coordinate features, while blank circles represent coordinate features from past sequences. Z and X represent the visual features of the template and search region, respectively, and Q, K, and V represent the three projection matrices.

[0104] Step 6: Combine temporal and spatial information by inputting the image and past sequences into the network to obtain the final tracking result. To obtain the final output target position, the input of this application is divided into two parts: one part is the image, consisting of the search area and the target template; the other part is the sequence, consisting of the tracking coordinates of past targets arranged in chronological order. This invention inputs the image and sequence into the network, and through the network's backbone and sequence decoder, combines the temporal and spatial information to finally obtain the target coordinates of the current frame, i.e., the final tracking result. The specific process is as follows:

[0105] Step 601: Construct the network's decoder using the two sequence decoders obtained in Step 5. The overall structure of the network's decoder consists of 6 initial encoders and L+1 sequence encoders, where 2 L To utilize the length of past sequences, different sequence encoders employ a number of past coordinates ranging from 2... 0 to 2 L Incremental scaling enables the decoder to learn sequences of different lengths, while reducing the impact of outlier coordinates and smoothing the trajectory.

[0106] The final decoder output is determined by both the target's visual features and its past trajectory. This allows the network to track the target not only visually but also by its past trajectory, effectively mitigating the effects of occlusion and similar targets. The overall structure of the network's decoder consists of... Figure 6 As shown;

[0107] Step 602: Perform coordinate transformation on the input sequence. In order to place the input sequence under the same reference system and make the sequence prediction more meaningful, this application uses the upper left corner of the cropped area based on the center of the previous frame of the current tracking image as the origin of the relative coordinate system to perform coordinate transformation on the input sequence, converting the absolute coordinates in the image in the sequence into coordinates relative to the previous frame.

[0108] Step 603: Input the search area to be tracked, the template, and the coordinates of the previously tracked trajectory into the network structure. The output coordinates obtained by the network are transformed inversely to the absolute coordinate system with the upper left corner of the image as the origin according to the position of the previous frame, so as to obtain the final coordinates of the target in the image, that is, the tracking result.

[0109] Therefore, this invention proposes a spatiotemporally integrated UAV image target tracking method. Firstly, in the spatial domain, a single-stream VIT backbone network and a contrastive learning-based loss function are designed to address the small target problem, resulting in a spatially enhanced VIT structure. Based on this VIT structure, a dynamic template update mechanism and a trajectory prediction-based decoder are designed in the temporal domain to address background interference, resulting in a temporally correlated correction module. These two structures together form a spatiotemporally integrated tracking algorithm, which can effectively improve the success rate and accuracy of target tracking from the UAV's perspective.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for tracking unmanned aerial vehicle (UAV) images based on spatiotemporal joint tracking, characterized in that, Includes the following steps: Step 1: Build a backbone network based on the VIT model to extract visual features of small targets. The VIT model divides the input image into fixed-size image blocks through two-dimensional convolution, increases the dimension of the hidden dimension through linear transformation, and inputs it into the transformer for feature extraction and downstream tasks. Step 2: For the target tracking task, enhance the utilization of spatial information in the tracking image and build a network framework for a single-stream VIT model. Step 3: Construct a loss function based on contrastive learning to distinguish between the foreground and background of the tracked image; Step 4: Add a dynamic template update mechanism to update the status of the tracked target using time-domain information during the tracking process; Step 5: Construct two sequence decoder structures from both the temporal and spatial domains; Step 6: Combine temporal and spatial information, input the image and past sequences into the network, and obtain the final tracking result; The specific process of building the network framework of the single-stream VIT model in step 2 is as follows: Step 201: Based on the backbone network of the VIT model, adjust the input. First, for a dimension of 3*H... X *W X The search region X and dimension H are 3*H Z *W Z Template Z, where H X W X and H Z W Z Representing the height and width of the search region and the height and width of the template respectively, they are arranged as N X *(3*P 2 ) and N Z *(3*P 2 A sequence of image patches, where P represents the width and height of an image patch, P*P is the resolution of each image patch, and N is the resolution of the image patch. z = H z W z / P 2 N x = H x W x / P 2 The number of image patches for the template and the search region are respectively. Then, a learnable one-dimensional linear projection layer E is used to project each image patch of the template and the search region to the D dimension. Step 202: Add learnable positional coding, using a learnable one-dimensional positional coding mechanism P. Z and P X , representing the positional codes of the template and the search region, respectively, are added to the corresponding image patch codes to obtain the final template sequence H. Z and search region sequence H X The visual characteristics are defined by the following formula: Step 203: Process the obtained template sequence H Z and search region sequence H X The visual features are spliced ​​together and attention is calculated to obtain the template sequence H. Z and search region sequence H X Connecting them along the 0th dimension, the final vector has a dimension of (N). X +N Z )*D, represented as H ZX =[H Z ;H X As the input to the encoder in the subsequent transformer, due to the self-attention mechanism in the transformer, after concatenating the template sequence and the search region sequence and inputting them into the transformer, the network can simultaneously complete two tasks: image feature extraction and modeling the relationship between the template and the search region. The formula for the self-attention mechanism suitable for tracking is as follows: Where Q, K, and V are the query matrix, key matrix, and value matrix, respectively; the subscripts z and x represent matrix items belonging to the template and search region, and D is the hidden layer dimension. Step 204: Expand the formula obtained in step 203 to obtain the following formula: in and These represent self-attention applied to the template and the search region, respectively, and are responsible for image feature extraction. and These represent cross-attention to the template and the search region, respectively, and are responsible for aggregating image features, which is equivalent to modeling the relationship between the template and the search region; Step 205: Build the overall network framework of the single-stream VIT model based on the target tracking task.

2. The UAV image target tracking method based on spatiotemporal joint method according to claim 1, characterized in that, The specific process of building the backbone network based on the VIT model in step 1 is as follows: Step 101: First, process the 3D image in the input format H*W*C. The image is segmented, where H, W, and C represent the image's height, width, and dimension, respectively. The model employs a two-dimensional convolutional transformation, and by setting the kernel size and stride, the image is segmented... The image is divided into multiple non-overlapping two-dimensional blocks, as shown in the following formula: in, x This represents the input image, where i represents the image patch number. x p Represents an image block. Indicates the number of image patches; Input H*W*C 3D image Transform into dimension The formula for the specific transformation process of two-dimensional image block encoding is as follows: Where N represents the number of image patches in the sequence, The dimension of each image patch is represented by P, the size of the image patch is C, the dimension of the initial image is D, the dimension of the hidden layer is "→", and "→" represents the transformation process. Step 102: Employ a learnable positional encoding method with the following dimensions. , with dimensions as The positional encoding and image block encoding are added together and input into the Transformer block to enhance the positional information of different image blocks; Step 103: Construct Transformer blocks. Transformer blocks are the basic components of the VIT model. The input of the VIT model is processed by several stacked Transformer blocks to obtain the final output. Each Transformer block consists of a self-attention module and a multilayer perceptron (MLP).

3. The UAV image target tracking method based on spatiotemporal joint method according to claim 2, characterized in that, The specific process of constructing the Transformer block is as follows: First, a self-attention module is constructed. The mechanism of attention is to assign different weights to the input information according to the current query. Attention contains three vectors: query vector, key vector, and value vector. For a given query vector, the k corresponding key vectors are matched by the inner product. The inner product is normalized by softmax to obtain the k weights. The output of attention is the weighted average of the value vectors corresponding to the k key vectors. If the query vector, key vector, and value vector are all obtained by linear transformation from an input sequence, then it becomes self-attention. Next, an MLP module is constructed, which consists of linear layers and activation functions.

4. The UAV image target tracking method based on spatiotemporal joint method according to claim 3, characterized in that, The specific process of constructing the loss function based on contrastive learning in step 3 is as follows: Step 301: First, construct the loss function based on SimCLR, with the following specific expression: in Represents the cosine similarity to the examples. Z represents the cosine similarity of the positive examples and the cosine similarity of the other negative examples. Z represents the input image. i, j, and k represent the image numbers, respectively. exp represents the exponential operation with base e. T is an adjustable parameter. M represents the number of positive examples plus the number of negative examples. Step 302: Construct a contrastive learning-based loss function for the tracking scenario based on the loss function from step 301. The specific formula is as follows: Where L represents the total loss, T is an adjustable parameter, M represents the number of positive examples plus the number of negative examples, Z represents the input image, x represents the current frame number, xi and xj represent the i-th and j-th frame numbers before the current frame, respectively, k is the total number of past frames, sim represents cosine similarity calculation, exp represents exponential operation with base e, and mask represents masking operation. Represents the cosine similarity to the examples. Cosine similarity to opposing examples.

5. The UAV image target tracking method based on spatiotemporal joint method according to claim 4, characterized in that, The specific process of adding the dynamic template update mechanism in step 4 is as follows: Step 401: Add a dynamic template to the network input that is continuously updated as tracking progresses. This template is obtained by cropping the current search region from the target bounding box predicted by the network, and the initial value is the same as the static template. Step 402: Add a confidence score head to the network. The confidence score head includes a three-layer perceptron and an activation function. If the score of the confidence score head is higher than the set threshold, the tracking is considered accurate and the dynamic template is updated. Otherwise, the template is not updated.

6. The UAV image target tracking method based on spatiotemporal joint method according to claim 5, characterized in that, The specific process of constructing the two sequence decoder structures in step 5, from both the time domain and spatial domain perspectives, is as follows: Step 501: Construct the initialization decoder. The input to the initialization decoder is the four coordinates of the target initialization and the template sequence H output in step 2. Z and search region sequence H X The visual features are used to perform cross-attention operations, where the four initial coordinates are used as q values, and the visual features are used as k and v values. The q, k, and v values ​​are all values ​​used in the attention operation. Step 502: Construct a sequence decoder. The input to the sequence decoder is the coordinate features obtained from the previous decoder and the previously tracked coordinate sequence. The coordinate form is (X1, Y1, X2, Y2), where X1 and Y1 represent the coordinates of the upper left corner of the target bounding box, and X2 and Y2 represent the coordinates of the lower right corner of the target bounding box. Step 503: Combine the current coordinates with the previously tracked coordinate sequence, perform self-attention calculation, find the relationship between the past tracking trajectory and the current coordinates, and use the past tracking trajectory as a guide to predict the coordinate position of the current target.

7. The UAV image target tracking method based on spatiotemporal joint method according to claim 6, characterized in that, The specific process of combining temporal and spatial information in step 6, inputting the image and past sequences into the network to obtain the final tracking result, is as follows: Step 601: Construct the network's decoder using the two sequence decoders obtained in Step 5. The overall structure of the network's decoder consists of 6 initial encoders and L+1 sequence encoders, where 2 L To utilize the length of past sequences, different sequence encoders employ a number of past coordinates ranging from 2... 0 to 2 L Increasing; Step 602: Perform coordinate transformation on the input sequence. Using the upper left corner of the cropped area of ​​the previous frame of the current tracked image as the origin of the relative coordinate system, perform coordinate transformation on the input sequence to convert the absolute coordinates in the image into coordinates relative to the previous frame. Step 603: Input the search area to be tracked, the template, and the coordinates of the previously tracked trajectory into the network structure. The output coordinates obtained by the network are inversely transformed to the absolute coordinate system with the upper left corner of the image as the origin according to the position of the previous frame, so as to obtain the final tracking result.

Images