Target tracking method based on spatio-temporal information joint perception

Abstract

The application discloses a target tracking method based on spatio-temporal information joint perception, and solves the problem that the prior art is prone to tracking failure in the face of interference such as local occlusion of a target and camera view angle change. The implementation scheme is as follows: a training set and a test set are obtained from a public website; a target tracking network composed of a spatio-temporal feature representation module, a mixed attention encoding and decoding module and a target positioning module is constructed, and the spatio-temporal feature representation module and the mixed attention encoding and decoding module are used to strengthen the features of a current tracking target, so that the target tracking network can resist the interference of environment change in tracking; the training set is input into the target tracking network for training until the maximum training round is reached; a test image is input into the trained target tracking network to obtain a classification response graph and a regression response graph, and the target tracking result is obtained by post-processing the classification response graph and the regression response graph. The application effectively improves the success rate and accuracy of target tracking, and can be used for video monitoring or autonomous flight of a unmanned aerial vehicle.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a target tracking method that can be used for video surveillance, autonomous flight of drones, and vehicle recognition. Background Technology

[0002] Target tracking plays an increasingly important role in modern technology, with broad application prospects and great potential. In video surveillance systems, target tracking can monitor and identify specific targets in real time, providing crucial information for security protection. The development of autonomous driving technology also relies heavily on target tracking, helping vehicles accurately identify and respond to their surroundings, improving driving safety. In the field of unmanned aerial vehicles (UAVs), it can significantly enhance the autonomous flight capabilities and mission execution efficiency of UAVs. However, in real-world target tracking scenarios, there are often problems such as frequent target motion deformation, changes in camera perspective, partial target occlusion, and changes in lighting. These problems lead to a decrease in the representation ability of target features, interfering with the target tracking algorithm's acquisition of effective image spatial information, and consequently, tracking deviation.

[0003] Target tracking can be broadly categorized into single-target tracking and multi-target tracking. Multi-target tracking refers to detecting and assigning IDs to multiple targets such as pedestrians, cars, and animals in a video for trajectory tracking when the number of targets is unknown beforehand. Single-target tracking, on the other hand, involves specifying the target to be tracked in the first frame of a video sequence and then accurately locating and selecting the target during subsequent tracking operations.

[0004] Commonly used single-object tracking techniques can be broadly categorized into three types: correlation filtering-based target tracking algorithms, Siamese network-based target tracking algorithms, and attention-based tracking algorithms. Correlation filtering-based algorithms primarily rely on manually designed features for target tracking, resulting in limited image representation capabilities compared to deep networks. Siamese network-based algorithms consist of three steps: 1) extracting features from the search region image and the template image; 2) measuring the similarity of features between the two images; and 3) using the point of highest feature similarity as the predicted target location and resolving the corresponding image coordinates. In these three steps, acquiring effective image features is crucial for accurately determining the target's current position. Attention-based tracking algorithms leverage the Transformer's ability to capture long-range dependencies, enabling them to capture effective target features even under various disturbances.

[0005] Patent document CN114596338A discloses a twin network target tracking method considering temporal relationships. It first uses a ResNet-50 residual network to extract features from the target template image sequence and the search region image; then, it designs a temporal information fusion network based on a multilayer perceptron to incorporate the temporal information of the target template image sequence into the target template features, thereby achieving more robust tracking performance; next, it uses the target template features and search region features containing temporal information to obtain a response map; finally, it achieves precise target localization based on the response map, thus improving the tracker's accuracy. While this method utilizes some temporal information from the video sequence to improve tracking accuracy and robustness to a certain extent, the use of temporal information by fusing features from three independent frames disrupts the continuity of time. Furthermore, its target localization processing does not consider target deformation, leading to a significant decrease in tracking performance when faced with more complex environmental interference or changes in the target itself. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and propose a target tracking method based on spatiotemporal information joint perception to ensure the representation ability of target features when they are disturbed and improve the target tracking success rate.

[0007] The technical idea to achieve the purpose of this invention is to enhance the features of the tracking frame at time t by using the features of frame image fusion from time t0 to time t-1, so that it can still represent the target to be tracked even when it is interfered with, thereby enabling the target localization network to successfully locate the target.

[0008] Based on the above ideas, the implementation steps of the present invention include the following:

[0009] (1) Obtain the training dataset and test set:

[0010] We obtained the GOT-10K dataset, Lasot dataset, DET dataset, and COCO dataset from public websites, and aggregated the video sequences from these datasets to form a training dataset.

[0011] We obtained the UAV123 dataset, UAV123@10fps dataset, and DTB70 dataset from public websites, and then aggregated the video sequences from these datasets to form a test set.

[0012] (2) Construct a target tracking network model based on spatiotemporal information joint perception:

[0013] (2a) Establish a spatiotemporal feature representation module consisting of a motion-controllable filtering unit, an image feature extraction unit, a hybrid attention unit, and a spatiotemporal convolution unit cascaded together;

[0014] (2b) Establish a hybrid attention encoder-decoder module consisting of an encoder and a decoder;

[0015] (2c) Establish a target localization module consisting of classification branches and regression branches;

[0016] (2d) The spatiotemporal feature representation module, the hybrid attention encoder-decoder module, and the target localization module are cascaded in sequence to form a target tracking network model;

[0017] (2e) Set the loss function of the target tracking network to Loss:

[0018] Loss = Weight cls ×loss cls +Weight loc ×loss loc

[0019] Among them, Weight cls These are the weights of the classification loss function, loss cls It is a binary cross-entropy loss, Weight loc These are the weights of the regression loss function, loss loc It is the intersection-union ratio loss function;

[0020] (3) Input the training dataset into the target tracking network model and optimize it using gradient descent to obtain the trained target tracking network model;

[0021] (4) Input the test dataset into the trained target tracking network model to obtain the classification response map and regression response map of the target feature response map;

[0022] (5) Post-process the classification response map and regression response map to achieve target tracking:

[0023] (5a) Apply a cosine window penalty to the classification result to obtain a new classification response map. Map the new classification response map back to the size of the input image during tracking and find the maximum value point as the center point of the target.

[0024] (5b) Based on the center point of the target, find the corresponding distance rectangle and the offset of the four sides of the rectangle on the regression response map, and add or subtract these offsets from the coordinates of the center point of the target to obtain the final rectangle coordinates, which is the final target tracking result.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] Firstly, because the present invention establishes a spatiotemporal feature representation module, it can utilize the continuous temporal information of the video sequence to enhance the tracking input image features extracted by the AlexNet network;

[0027] Secondly, because the present invention establishes a hybrid attention codec module, the enhanced tracking input image features can pay more attention to target information through its internal attention mechanism;

[0028] Third, this invention constructs a target tracking network composed of a spatiotemporal feature representation module, a hybrid attention encoder-decoder module, and a target localization module, which can effectively address the problem of decreased target feature representation capability caused by interference such as partial target occlusion and changes in camera viewpoint, thereby improving the success rate of target tracking. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0030] Figure 2 This is a network structure diagram of the present invention;

[0031] Figure 3 This is a diagram of the spatiotemporal convolution structure of the present invention. Detailed Implementation

[0032] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0033] Reference Figure 1 The implementation steps of this example include the following:

[0034] Step 1: Obtain the training dataset and the test set.

[0035] We obtained the GOT-10K dataset, Lasot dataset, DET dataset, and COCO dataset from public websites, and aggregated the video sequences from these datasets to form a training dataset.

[0036] We obtained the UAV123 dataset, UAV123@10fps dataset, and DTB70 dataset from public websites, and then aggregated the video sequences from these datasets to form a test set.

[0037] Step 2: Construct a target tracking network based on spatiotemporal information joint perception.

[0038] Reference Figure 2 The implementation of this step is as follows:

[0039] (2.1) Establishing a spatiotemporal feature representation module:

[0040] 2.1.1) Establish a motion-controllable filtering unit to transform the input target image x while preserving motion trajectory information:

[0041] First, the input target image x is transformed to polar coordinates, and then the periodic harmonic factor e is used. -ikθPerform the following rotational transformation:

[0042]

[0043] Where, φ θ Scale represents a transformation operation with a transformation angle of θ. jk It is a scale factor. It is a polar coordinate transformation function;

[0044] Then, the rotated target image is superimposed onto the cropped position of the original sample image to obtain the superimposed image, so as to maintain the continuity of the target motion trajectory;

[0045] Finally, the maximum connected component algorithm is used to obtain the mask image of the target image, and the mask image is used to repair the superimposed image. The Gaussian smoothing algorithm is then used to eliminate abrupt changes in the target edge information.

[0046] 2.1.2) An image feature extraction unit composed of an AlexNet network is established to extract image features. It consists of three cascaded convolutional layers C1, C2, and C3 with different structures, where:

[0047] The structure of C1 is: convolution → ReLU activation → max pooling. The convolution operation uses 96 11×11×3 convolution kernels with a stride of 4 to convolve the input image; the ReLU activation function activates each value of the convolution result; the max pooling operation uses one 3×3 pooling kernel with a stride of 2 to pool the activated result.

[0048] The structure of C2 is: convolution → ReLU activation → max pooling. The convolution operation uses 256 5×5×96 convolution kernels with a stride of 1. The ReLU activation function activates each value of the convolution result. The max pooling operation uses one 3×3 pooling kernel with a stride of 2.

[0049] The structure of C3 is: convolution → ReLU activation; the convolution operation uses 384 3×3×256 convolution kernels with a stride of 1. The ReLU activation function activates each value of the convolution result, and its output is the original feature F of the image.

[0050] 2.1.3) Establish a hybrid attention unit that includes channel attention operations and spatial attention operations, wherein:

[0051] This channel attention operation first uses max pooling, average pooling, and a sigmoid activation function to obtain the channel weights W of the original feature F. c Then use W c Multiplying the original feature by the enhanced feature F yields the enhanced feature F. c The calculation formula is as follows:

[0052] W c =Sigmoid(MaxPool(F)+AvgPool(F))

[0053] F c =W c ×F

[0054] This spatial attention operation first obtains the enhanced feature F through max pooling, average pooling, and concatenation. c Position weight D s Then use D s With F c Multiplication yields the output feature F of the hybrid attention unit. r :

[0055] D s =Concat(MaxPool(F c ), AvgPool(F c ))

[0056] F r =D s ×F c

[0057] In the formula, F represents the original features of the image, Sigmoid is the activation function, MaxPool is the max pooling function, AvgPool is the average pooling function, and Concat represents the concatenation function.

[0058] 2.1.4) Construct a spatiotemporal convolution unit that includes spatiotemporal convolution operations and depthwise cross-correlation operations, wherein:

[0059] like Figure 3 As shown, this spatiotemporal convolution operation obtains spatiotemporal features through linear connections and pooling.

[0060]

[0061] This deep cross-correlation operation is used to obtain the feature response M of the tracking input image features and the first frame target template image features. r The calculation formula is as follows:

[0062]

[0063] In the formula, F r It is the fusion feature from time t0 to t-1, F x is the tracking input image feature at time t, w1 and w2 are trainable weight parameters, and b1 and b2 are trainable bias parameters; F is the spatiotemporal feature of the tracking input image at time t.z This represents the features of the first frame of the target template image extracted using the AlexNet network, and ★ indicates the deep cross-correlation operation.

[0064] 2.1.5) The motion-controllable filtering unit, image feature extraction unit, hybrid attention unit, and spatiotemporal convolution unit are sequentially connected to form a spatiotemporal feature representation module.

[0065] (2.2) Establish a hybrid attention encoder-decoder module;

[0066] 2.2.1) Establish an encoder that includes multi-head attention operation, normalization operation, and feedforward propagation operation. Its input is the fused feature F from the initial time t0 to time t-1. r and the characteristic response M at time t r Feature F obtained by straightening and splicing m and the input features F m Enhance the feature F to obtain features that are more relevant to the target information. me Its implementation is as follows:

[0067] First, in multi-head attention operations, F m Multiply by the query transformation matrix W respectively Q , bond transformation matrix W K Value transformation matrix W V The query matrix M is obtained. Q Key matrix MK, value matrix M V It is represented as follows:

[0068]

[0069] Then, the attention score for a single head is calculated using the following formula:

[0070]

[0071] Where Attn is the attention score, Tranpose denotes matrix transpose, and dim denotes matrix M. K The dimension;

[0072] Next, the attention scores of each head are concatenated and subjected to a linear transformation to obtain the multi-head attention score MulAttn(F). m ):

[0073] MulAttn(F m )=Liner(Concat(Attn1, Attn2,…,Attn i , ..., Attn n ))

[0074] Where Liner represents linear transformation, Concat represents concatenation, and Attn i Let i represent the attention score of the i-th head, where i = 1, 2, ..., n, and n is the number of attention heads.

[0075] Finally, the multi-head attention score MulAttn(F) was calculated. m After performing normalization and feedforward propagation operations, the encoder output F is obtained. me :

[0076] F me =LayerNorm(F ml +Feedforword(F ml ))

[0077] Where F ml =LayerNorm(F m +MulAttn(F m LayerNorm represents the normalization operation, and Feedforword represents the feedforward propagation operation.

[0078] 2.2.2) Establish a decoder that includes multi-head attention operation, normalization operation, and feedforward propagation operation, with the encoder output F as its input. me It obtains the output F after undergoing the same operation as the encoder. md ;

[0079] 2.2.3) Connect the encoder and decoder to form a hybrid attention encoder-decoder module.

[0080] (2.3) Establish the target localization module:

[0081] 2.3.1) Establish a classification branch consisting of four convolutional layers with the same structure cascaded together and then connected to a third convolutional layer. Each of these four convolutional layers consists of convolution, GN group normalization, and ReLU activation function. The third convolutional layer consists of convolution and SoftMax activation function, and its output is the classification result.

[0082] 2.3.2) Establish a regression branch consisting of four convolutional layers with the same structure cascaded together and then connected to a third convolutional layer. Each of these four convolutional layers with the same structure consists of convolution, GN group normalization, and ReLU activation function. The output of this convolutional layer is the regression result.

[0083] 2.3.2) Connect the classification branch and the regression branch in parallel to form the target localization module.

[0084] (2.4) The spatiotemporal feature representation module, the hybrid attention encoder-decoder module, and the target localization module are cascaded in sequence to form a target tracking network based on spatiotemporal information joint perception.

[0085] Step 3: Set the loss function of the target tracking network to Loss.

[0086] (3.1) The loss function of the classification branch in the target tracking network is... cls Weight cls Set the value to 1.2; based on the predicted probability value in the classification results and the given label value, adjust the loss... cls The calculations are as follows:

[0087]

[0088] In the formula, N is the number of predicted samples, and label is... i It is the label of the predicted target, pred i It is the probability of predicting the target;

[0089] (3.2) The loss function of the regression branch in the target tracking network is... loc Weight loc Set to 3; based on the distances from the predicted center point to the four sides of the rectangle and the labeled true rectangle, adjust the loss... loc The calculations are as follows:

[0090]

[0091] In the formula, A represents the area of ​​the predicted box, B represents the area of ​​the ground truth box, A∩B represents the intersection of the predicted box and the ground truth box, and A∪B represents the union of the predicted box and the ground truth box.

[0092] (3.3) Multiply the loss of the classification branch and the loss of the regression branch by their respective weights and then add them together to obtain the loss of the target tracking network: Loss = Weight cls ×loss cls +Weight loc ×loss loc .

[0093] Step 4: Optimize the target tracking network model using gradient descent.

[0094] (4.1) Set the learning rate to 0.0001, the number of positive samples to 16, the number of negative samples to 16, and the maximum number of training rounds to 100.

[0095] (4.2) Input the training dataset into the target tracking network model and perform forward propagation to obtain the predicted classification and regression results;

[0096] (4.3) Use the loss function Loss to calculate the loss value between the prediction result and the actual result of the target tracking network, and use the loss value to backpropagate the network to calculate the gradient of each parameter.

[0097] (4.4) Use the SGD optimizer based on gradient Update the network parameters to obtain the currently updated parameter w. k :

[0098]

[0099] In the formula, w k-1 Here are the parameters of the model at the previous time step, and α is the learning rate;

[0100] (4.5) Repeat steps (4.2) to (4.4) until the specified number of training rounds is reached to obtain a trained target tracking network based on spatiotemporal information joint perception.

[0101] Step 5: Use the trained target tracking network to track the target.

[0102] (5.1) Input the image to be tracked into the trained target tracking network to obtain the final classification response map and regression response map;

[0103] (5.2) Apply a cosine window penalty to the classification result to obtain a new classification response map. Map the new classification response map back to the size of the input image during tracking and find the maximum value point as the center point of the target.

[0104] (5.3) Based on the center point of the target, find the corresponding distance rectangle and the offset of the four sides of the rectangle on the regression response map, and add or subtract these offsets from the coordinates of the center point of the target to obtain the final rectangle coordinates, which is the final target tracking result.

[0105] The effects of this invention will be further illustrated below with simulation experiments:

[0106] 1. Simulation experimental conditions:

[0107] The hardware platform for the simulation experiment of this invention is as follows: the processor is an Intel Core i9-12900KF with a main frequency of 3.1GHz, the memory is 32G, and the graphics card is an NVIDIA GeForce RTX3090.

[0108] The hardware platform for the simulation experiment of this invention is: Ubuntu 20.04.6LTS operating system, Python 3.8.0 language, and PyTorch 1.12.1 programming framework based on CUDA 10.2.

[0109] The datasets used in the simulation experiments of this invention are the UAV123 dataset, the UAV123@10fps dataset, and the DTB70 dataset, all obtained from publicly available websites. The UAV123 dataset, proposed in 2016, is a publicly available dataset for testing the performance of target tracking algorithms on UAV platforms. It contains 123 video sequences, exceeding 110K frames, and includes 42 scenes such as urban roads, buildings, and beaches. It addresses common UAV tracking issues such as target scale changes, background clutter, camera movement, scale changes, and viewpoint shifts. The UAV123@10fps dataset is a downsampled dataset obtained from the UAV123 dataset, exhibiting more rapid feature changes and containing 123 challenging video sequences. The DTB70 dataset, proposed in 2017, is a UAV viewpoint dataset providing 70 short video sequences. It contains a large number of similar targets and presents several challenging factors, such as background interference and similar targets.

[0110] The metrics used to evaluate these three datasets are tracking success rate (SR) and tracking accuracy (P).

[0111] The success rate SR is related to the overlap score, which refers to the overlap score of the manually annotated bounding box B. gt The bounding box B estimated by the tracking algorithm tr The ratio of the intersection to the union: When the overlap score of a frame is greater than the set threshold, the frame is considered to be successfully tracked. The percentage of successful frames out of all frames is the success rate.

[0112] The accuracy P is related to the center position error, which refers to the center point C of the manually annotated rectangle. gt The center point C of the bounding box estimated by the tracking algorithm tr Euclidean distance between them: CenterError = ||C gt -C tr ||2, The accuracy is the percentage of video frames whose center position error is less than a given threshold among all frames;

[0113] 2. Simulation content and result analysis:

[0114] Simulation Experiment 1: The present invention and six existing technologies—HiFT, SiamAPN, SiamAPN++, SiamRPN++, DaSiamRPN, and UpdateNet—were used to track targets on the test set UAV123. The success rate and accuracy of each technology were obtained, and the results are shown in Table 1.

[0115] Table 1 Comparison of success rate and accuracy of different tracking methods on the test set UAV123

[0116]

[0117] As can be seen from Table 1, the method proposed in this invention achieves the best success rate and accuracy in tracking targets compared with existing excellent technologies, proving the effectiveness of this method in different environments and scenarios.

[0118] Simulation Experiment 2: The present invention and six existing technologies (SiamAPN++, SiamAPN, HiFT, DASiamRPN, UpdateNet, and Ocean) were used to perform target tracking on the UAV123 test set at 10fps. The success rate and accuracy of each technology were obtained, and the results are shown in Table 2.

[0119] Table 2 Comparison of success rate and accuracy of different tracking methods on the test set UAV123@10fps

[0120]

[0121] As can be seen from Table 2, the method proposed in this invention achieves the best success rate and accuracy in tracking targets compared with existing excellent technologies, proving that this method can effectively cope with situations where targets change rapidly.

[0122] Simulation Experiment 3: The present invention and six existing technologies (SiamAPN++, LightTrack, SiamAPN, SiamGAT, SiamMASK, and DASiamRPN) were used to track targets on the DTB70 test set, and their success rates and accuracy rates were obtained. The results are shown in Table 3.

[0123] Table 3 Comparison of success rate and accuracy of different tracking methods on the DTB70 test set.

[0124]

[0125] As can be seen from Table 3, the method proposed in this invention achieves the best success rate and accuracy in tracking targets compared with existing excellent technologies, proving that the method is still effective in the face of interference from a large number of similar targets.

[0126] Simulation results show that the target tracking method based on spatiotemporal information joint perception proposed in this invention uses continuous temporal information in the image sequence to enhance the features of the currently tracked target, ensuring the representation ability of the target features after encountering interference, thereby improving the success rate and accuracy of target tracking.

[0127] The existing methods used in the simulation experiments are sourced as follows:

[0128] HiFt refers to the target tracking method proposed by Cao et al. in "Hift: Hierarchical feature transformer for aerial tracking", abbreviated as HiFt;

[0129] SiamRPN++ refers to the target tracking method proposed by Li et al. in "Siamrpn++: Evolution of siamese visual tracking with very deep networks", abbreviated as SiamRPN++;

[0130] SiamAPN refers to the target tracking method proposed by Fu et al. in "Siamese anchor proposal network for high-speed aerial tracking", abbreviated as SiamAPN;

[0131] SiamAPN++ refers to the target tracking method proposed by Fu et al. in "SiamAPN++: Siamese attentional aggregation network for real-time UAV tracking", abbreviated as SiamAPN++.

[0132] DaSiamRPN refers to the target tracking method proposed by Zhu et al. in "Distractor-aware siamese networks for visual object tracking", abbreviated as DaSiamRPN;

[0133] UpdateNet refers to the target tracking method proposed by Zhang et al. in "Learning the model update for siamesetrackers", abbreviated as UpdateNet;

[0134] Ocean refers to the target tracking method proposed by Peng et al. in "Ocean: Object-aware anchor-free tracking", abbreviated as Ocean;

[0135] SiamGAT refers to the object tracking method proposed by Guo et al. in "Graph attention tracking", abbreviated as SiamGAT;

[0136] SiamMASK refers to the object tracking method proposed by Hu et al. in "Siammask: A framework for fast online object tracking and segmentation", abbreviated as SiamMASK;

[0137] LightTrack refers to the object tracking method proposed by Yan et al. in "Lighttrack: Finding lightweight neural networks for object tracking via one-shot architecture search", abbreviated as LightTrack.

[0138] The above description is merely a specific example of the present invention and does not constitute any limitation on the present invention. Obviously, those skilled in the art, after understanding the content and principles of the present invention, may make various modifications and changes in form and details without departing from the principles and structure of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0139] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.

Claims

1. A target tracking method based on spatiotemporal information joint sensing, characterized in that, Includes the following steps; (1) Obtain the training dataset and test set: We obtained the GOT-10K dataset, Lasot dataset, DET dataset, and COCO dataset from public websites, and aggregated the video sequences from these datasets to form a training dataset. We obtained the UAV123 dataset, UAV123@10fps dataset, and DTB70 dataset from public websites, and then aggregated the video sequences from these datasets to form a test set. (2) Construct a target tracking network model based on spatiotemporal information joint perception: (2a) Establish a spatiotemporal feature representation module consisting of a motion-controllable filtering unit, an image feature extraction unit, a hybrid attention unit, and a spatiotemporal convolution unit cascaded together; (2b) Establish a hybrid attention encoder-decoder module consisting of an encoder and a decoder; (2c) Establish a target localization module consisting of classification branches and regression branches; (2d) The spatiotemporal feature representation module, the hybrid attention encoder-decoder module, and the target localization module are cascaded in sequence to form a target tracking network model; (2e) Set the loss function of the target tracking network to Loss: Loss=Weight cls ×loss cls +Weight loc ×loss loc Among them, Weight cls These are the weights of the classification loss function, loss cls It is a binary cross-entropy loss, Weight loc These are the weights of the regression loss function, loss loc It is the intersection-union ratio loss function; (3) Input the training dataset into the target tracking network model and optimize it using gradient descent to obtain the trained target tracking network model; (4) Input the test dataset into the trained target tracking network model to obtain the classification response map and regression response map of the target feature response map; (5) Post-process the classification response map and regression response map to achieve target tracking: (5a) Apply a cosine window penalty to the classification result to obtain a new classification response map. Map the new classification response map back to the size of the input image during tracking and find the maximum value point as the center point of the target. (5b) Based on the center point of the target, find the corresponding distance rectangle and the offset of the four sides of the rectangle on the regression response map, and add or subtract these offsets from the coordinates of the center point of the target to obtain the final rectangle coordinates, which is the final target tracking result.

2. The method according to claim 1, characterized in that, The unit structure and parameter settings of the spatiotemporal feature representation module in step (2a) are as follows: The motion-controllable filtering unit is used to transform the input target image x while retaining the motion trajectory information; The image feature extraction unit uses an AlexNet network to extract image features. It consists of three cascaded convolutional layers C1, C2, and C3 with different structures, wherein: The structure of C1 is: convolution → ReLU activation → max pooling. The convolution operation uses 96 11×11×3 convolution kernels with a stride of 4 to convolve the input image; the ReLU activation function activates each value of the convolution result; the max pooling operation uses one 3×3 pooling kernel with a stride of 2 to pool the activated result. The structure of C2 is: convolution → ReLU activation → max pooling. The convolution operation uses 256 5×5×96 convolution kernels with a stride of 1. The ReLU activation function activates each value of the convolution result. The max pooling operation uses one 3×3 pooling kernel with a stride of 2. The structure of C3 is: convolution → ReLU activation; the convolution operation uses 384 3×3×256 convolution kernels with a stride of 1. The ReLU activation function activates each value of the convolution result, and its output is the original feature F of the image. The hybrid attention unit includes channel attention operations and spatial attention operations, wherein: This channel attention operation first uses max pooling, average pooling, and a sigmoid activation function to obtain the channel weights W of the original feature F. c Then use W c Multiplying the original feature by the enhanced feature F yields the enhanced feature F. c The calculation formula is as follows: W c =Sigmoid(MaxPool(F)+AvgPool(F)) F c =W c ×F This spatial attention operation first obtains the enhanced feature F through max pooling, average pooling, and concatenation. c Position weight D s Then use D s With F c Multiplication yields the output feature F of the hybrid attention unit. r : D s =Concat(MaxPool(F c ),AvgPool(F c )) F r =D s ×F c In the formula, F represents the original features of the image, Sigmoid is the activation function, MaxPool is the max pooling function, AvgPool is the average pooling function, and Concat represents the concatenation function. The spatiotemporal convolution unit includes spatiotemporal convolution operations and depthwise cross-correlation operations, wherein: This spatiotemporal convolution operation obtains spatiotemporal features through linear connections and pooling. This deep cross-correlation operation is used to obtain the feature response M of the tracking input image features and the first frame target template image features. r The calculation formula is as follows: In the formula, F r It is the fusion feature from time t0 to t-1, F x is the tracking input image feature at time t, w1 and w2 are trainable weight parameters, and b1 and b2 are trainable bias parameters; F is the spatiotemporal feature of the tracking input image at time t. z This represents the features of the first frame of the target template image extracted using the AlexNet network, and ★ indicates the deep cross-correlation operation.

3. The method according to claim 2, characterized in that, The motion-controllable filtering unit transforms the input target image x while preserving the motion trajectory information, as follows: First, the input target image x is transformed to polar coordinates, and then the periodic harmonic factor e is used. -ikθ Perform the following rotational transformation: Where, φ θ Scale represents a transformation operation with a transformation angle of θ. jk It is a scale factor. It is a polar coordinate transformation function; Then, the rotated target image is superimposed onto the cropped position of the original sample image to obtain the superimposed image, so as to maintain the continuity of the target motion trajectory; Finally, the maximum connected component algorithm is used to obtain the mask image of the target image, and the mask image is used to repair the superimposed image. The Gaussian smoothing algorithm is then used to eliminate abrupt changes in the target edge information.

4. The method according to claim 1, characterized in that, The encoder and decoder that constitute the hybrid attention encoder-decoder module in step (2b) have the following structure: The encoder includes multi-head attention operation, normalization operation, and feedforward propagation operation, and its input is the fused feature F from the initial time t0 to time t-1. r and the characteristic response M at time t r Feature F obtained by straightening and splicing m and the input features F m Enhance the feature F to obtain features that are more relevant to the target information. me ; The decoder includes multi-head attention operation, normalization operation, and feedforward propagation operation, and its input is the encoder output F. me It obtains the output F after undergoing the same operation as the encoder. md .

5. The method according to claim 4, characterized in that, The encoder, for the input feature F m Enhance it to achieve the following: First, in multi-head attention operations, F m Multiply by the query transformation matrix W respectively Q , bond transformation matrix W K Value transformation matrix W V The query matrix M is obtained. Q Key matrix M K Value matrix M V It is represented as follows: Then, the attention score for a single head is calculated using the following formula: Where Attn is the attention score, Tranpose denotes matrix transpose, and dim denotes matrix M. K The dimension; Next, the attention scores of each head are concatenated and subjected to a linear transformation to obtain the multi-head attention score MulAttn(F). m ): MulAttn(F m )=Liner(Concat(Attn1,Attn2,…,Attn i ,…,Attn n )) Where Liner represents linear transformation, Concat represents concatenation, and Attn i Let i represent the attention score of the i-th head, where i = 1, 2, ..., n, and n is the number of attention heads. Finally, the multi-head attention score MulAttn(F) was calculated. m After performing normalization and feedforward propagation operations, the encoder output F is obtained. me : F me =LayerNorm(F ml +Feedforword(F ml )) Where F ml =LayerNorm(F m +MulAttn(F m LayerNorm represents the normalization operation, and Feedforword represents the feedforward propagation operation.

6. The method according to claim 1, characterized in that, The classification and regression branches that constitute the target localization module in step (2c) have the following structure: The classification branch consists of four convolutional layers with the same structure cascaded together and then connected to a convolutional layer that outputs the classification result. Each of these four convolutional layers with the same structure is composed of convolution, GN group normalization, and ReLU activation function. The convolutional layer that outputs the classification result is composed of convolution and SoftMax activation function. The regression branch consists of four convolutional layers with the same structure cascaded together, followed by a convolutional layer that outputs the regression result. Each of these four convolutional layers with the same structure consists of convolution, GN group normalization, and ReLU activation function. The convolutional layer that outputs the classification result consists of convolution and ReLU activation function.

7. The method according to claim 1, characterized in that, The loss function Loss used in step (2e) is expressed as follows: Loss=Weight cls ×loss cls +Weight loc ×loss loc Among them: Weight cls Indicates loss cls The weighting is set to 1.2; Weight loc Indicates loss loc The weighting is set to 3; This represents the binary cross-entropy loss, where N is the number of predicted samples and label is the number of samples predicted. i It is the label of the predicted target, pred i It is the probability of predicting the target; Let A represent the intersection-union loss, where A represents the area of ​​the predicted box, B represents the area of ​​the ground truth box, A∩B represents the intersection of the predicted box and the ground truth box, and A∪B represents the union of the predicted box and the ground truth box.

8. The method according to claim 1, characterized in that, In step (3), the training dataset is input into the target tracking network model, and gradient descent is used to optimize it, as follows: (3a) Set the learning rate to 0.0001, the number of positive samples to 16, the number of negative samples to 16, and the maximum number of training rounds to 100. (3b) The training data input into the target tracking network based on spatiotemporal information joint perception is forward propagated through the network to obtain the predicted classification and regression results; (3c) Use the loss function Loss to calculate the loss value between the prediction result and the actual result of the target tracking network, and use the loss value to backpropagate the network to calculate the gradient of each parameter. (3d) Using the SGD optimizer based on gradient Update the network parameters to obtain the currently updated parameter w. k : In the formula, W k-1 Here are the parameters of the model at the previous time step, and α is the learning rate; (3e) Repeat steps (3b) to (3d) until the specified number of training rounds is reached to obtain a trained target tracking network based on spatiotemporal information joint perception.

Images