A Long-Term Anti-UAV Visual Tracking Method Based on Spatiotemporal Attention Knowledge

Abstract

The present invention provides a long-term anti-UAV visual tracking method based on spatio-temporal attention knowledge, including S1: obtaining an infrared video sequence of a UAV and extracting features from the video frame images; S2: constructing a tracking network model and training it, inputting the extracted feature images into the trained tracking network model, and outputting the predicted position and scale. The tracking network model constructs a two-stage re-detection architecture of cross-correlation RPN similarity learning RCNN based on a Siamese network, introduces a spatial position consistency function and a supervised foreground detector, and at the same time uses a triplet training mechanism to incorporate spatio-temporal attention knowledge into the learning process of the tracker, training the tracking network model, and realizing the position estimation and scale prediction of the UAV target in the infrared video sequence.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and particularly relates to a long-term anti-UAV visual tracking method based on spatio-temporal attention knowledge. Background Art

[0002] Unmanned aerial vehicles (UAVs) have received extensive attention due to their flexibility and portability. They have a large number of applications, including aerial photography, intelligent monitoring, reconnaissance, and rescue. At the same time, it is also a double-edged sword. As the technical barriers and difficulties of modifying UAVs continue to decrease, UAVs are increasingly frequently used to perform illegal tasks, such as physical attacks (carrying explosives) and privacy violations (intruding into critical infrastructure). Therefore, anti-UAV technologies, especially vision-based anti-UAV tracking technologies, have been widely studied to address the potential threats of UAV intrusions.

[0003] Visual object tracking, especially in the Thermal InfraRed (TIR) mode, as an important branch and key technology in anti-UAV systems, relies on the basic theories of machine learning and image processing to assist in the research of subsequent anti-UAV tasks. With the remarkable progress of deep learning technology, Siamese networks have become the mainstream framework for visual object tracking due to their powerful deep feature learning capabilities. The SiamFC tracker first introduced a Siamese network with two shared weights for visual object tracking and used a correlation layer to learn the similarity evaluation function between the two branches. Currently, Siamese trackers have greatly advanced the state-of-the-art tracking performance in multiple recognized benchmarks and competitions by adding region proposal networks (RPNs), attention mechanisms, correlation filters, residual networks, region-based convolutional neural networks (RCNNs), and Transformer architectures. Nevertheless, these trackers are designed for general objects in visible light videos.

[0004] The potential of the TIR mode in some special scenarios cannot be ignored, such as at night, in foggy weather, etc.; obviously, TIR tracking technology is more suitable for low-light scenarios, meeting the all-weather tracking requirements. However, tracking small UAVs in TIR videos has problems such as low contrast, small objects, fast movement, and thermal background noise compared to tracking general objects in natural scenes.

[0005] Summary of the Invention

[0006] To solve the above problems, the present invention provides a long-term anti-UAV visual tracking method based on spatio-temporal attention knowledge, and the specific technical solution is as follows:

[0007] S1: Obtain the UAV infrared video sequence and extract features from the video frame images;

[0008] S2: Construct a tracking network model and train it. Input the extracted feature images into the trained tracking network model to output the predicted position and scale.

[0009] The tracking network model is constructed based on a Siamese network and includes a template branch and at least one search branch. The feature images are respectively input into the template branch and the search branch.

[0010] Among them, the template branch is initialized in the first frame and then remains fixed in subsequent frames.

[0011] The search branch continuously inputs the feature images of each frame and encodes the template branch information into the candidate proposals for re-detection.

[0012] Furthermore, in the feature extraction of video frame images, Swin Transformer is used as the feature extractor.

[0013] Furthermore, the re-detection includes two stages. The first stage is cross-correlation RPN re-detection, which uses a large number of anchor box (Anchor) candidate regions generated by the cross-correlation RPN network.

[0014] The second stage is similarity learning RCNN re-detection, which uses the similarity learning RCNN network to encode the prediction results of historical frames into the search candidate regions. The RCNN network includes a template memory repository and an RCNN detection head.

[0015] Cross-correlation RPN re-detection realizes the information interaction between the search branch and the template branch through a cross-correlation encoder.

[0016] Similarity learning RCNN re-detection calculates the similarity score between the target and each candidate proposal by introducing a similarity learning encoder.

[0017] Furthermore, in the similarity learning RCNN re-detection, based on the template memory repository, the high-confidence prediction results of all historical frames are concatenated into a feature matrix, and the feature matrix is cross-coded with the candidate regions generated by the cross-correlation RPN re-detection and then input into the RCNN detection head.

[0018] This enables each expert to contribute to the target prediction in the current frame, thus maximizing the retention of target information and effectively solving the update problem of the Siamese network.

[0019] Furthermore, the specific operations of the cross-correlation RPN re-detection are as follows:

[0020] The specific operations of the cross-correlation RPN re-detection are as follows:

[0021]

[0022] Among them, and respectively represent the features of the template and the search image, is a convolutional filter of size is the feature output of cross-coding.

[0023] Furthermore, the similarity learning RCNN re-detection is specifically operated as follows:

[0024]

[0025] Among them, is the target alignment feature of the template branch and is the alignment feature of the candidate regions from the search branch , where M is the number of candidate proposals, and ⊙ represents the Hadamard product.

[0026] Furthermore, the tracking network model also includes a supervised foreground detector, and the supervised foreground detector uses an independent RCNN detection head to perform foreground prediction on the UAV.

[0027] The supervised foreground detector does not interact with the template branch and can directly output the probability that the target is a UAV target; the supervised foreground detector independently predicts the target for each frame and is not troubled by the changes in the target appearance, which is beneficial to repairing the tracking failure situation.

[0028] Furthermore, the tracking network model also includes a state-aware converter, which switches between local tracking and global search according to the consistency condition;

[0029] The consistency condition is , if the spatial consistency function outputs zero, then global search is enabled, and the best prediction is found from the global candidate regions according to the evaluation scores of the candidate proposals of the high-confidence tracking trajectory; among them, represents the end frame of a certain sub-trajectory in the current high-confidence tracking trajectory.

[0030] Furthermore, the supervised foreground detector shares weights with the RCNN detection head of the similarity learning RCNN network.

[0031] They can be trained simultaneously in an end-to-end manner.

[0032] Furthermore, for the tracking network model, the search branch includes a first search branch and a spatio-temporal attention branch, and is trained in a triplet manner, specifically as follows:

[0033] Obtain a random frame and two adjacent frames of the UAV infrared video sequence;

[0034] Use a random frame as the input of the template branch, and use two adjacent frames as the inputs of the first search branch and the spatio-temporal attention branch respectively;

[0035] In the re-detection cross-correlation RPN network in the first stage, use binary cross-entropy and smoothness as the classification and regression losses respectively to train the model:

[0036]

[0037] where, represents the predicted classification value, represents the regression offset of the predicted bounding box, and are the true label and the true bounding box respectively, and represent the normalization parameters for classification and regression, is the balance parameter of these two loss functions;

[0038] In the re-detection RCNN network in the second stage, set the following loss function to be minimized:

[0039]

[0040] where, represents the similarity learning loss between the template branch and the search branch, represents the supervised UAV foreground detection loss, represents the spatio-temporal attention loss between adjacent frames.

[0041] The beneficial effects of the present invention are as follows:

[0042] Based on the Siamese network framework, a tracking network model is constructed. By establishing a two-stage re-detection architecture to re-detect the historical prediction results, a new triplet training mechanism is designed, and spatio-temporal attention knowledge is incorporated into the learning process of the tracker. The tracking network model establishes a two-stage re-detection architecture to re-detect the historical prediction results. At the same time, a spatial position consistency function is introduced to suppress background thermal noise interference, and a supervised foreground detector is introduced to recover possible tracking failures. By making full use of spatio-temporal attention knowledge, the position estimation and scale prediction of UAV targets in infrared video sequences are realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a schematic flow chart of the tracking method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] In the following description, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0045] Embodiment 1

[0046] Embodiment 1 of the present invention discloses a long-term anti-UAV visual tracking method based on spatio-temporal attention knowledge. In this embodiment, a high-confidence tracking trajectory is defined , which consists of consecutive non-overlapping sub-trajectories, that is, . For each sub-trajectory , there is a relationship , where and respectively represent the starting frame and the ending frame of the sub-trajectory and ; the following description is based on this.

[0047] As shown in Figure 1 , it includes:

[0048] S1: Obtain the UAV infrared video sequence and extract features from the video frame images;

[0049] In this embodiment, when extracting features from the video frame images, the SwinTransformer based on the shifted window is used as the feature extractor;

[0050] The Swin Transformer is created by replacing the standard MSA module in the Transformer block with a module based on the shifted window, and the other layers remain unchanged.

[0051] First, calculate the self-attention within the local window, and each window is used in a non-overlapping manner The local area is evenly divided; the local MSA reduces the computational burden of the global MSA by , where represents the window size, and represents any feature dimension;

[0052] Then, introduce the shifted window overlapping division strategy to achieve cross-connection between adjacent windows and maintain the efficient calculation of the local MSA.

[0053] Using this division strategy, the continuous block propagation calculation of the Swin Transformer is as follows,

[0054]

[0055] Among them, and respectively represent the output features of the shifted window based multiple-head self-attention (SW-MSA) and the multi-layer perception (MLP) at the th layer; and respectively represent window-based MSA using non-overlapping partitions and shifted overlapping partitions; represents the multi-layer perception, represents the LayerNorm layer.

[0056] S2: Construct a tracking network model and train it, input the extracted feature images into the trained tracking network model, and output the predicted position and scale;

[0057] The tracking network model is constructed based on a Siamese network, including a template branch and two search branches; the feature images are respectively input into the template branch and the search branches;

[0058] Among them, the template branch is initialized in the first frame and then remains fixed in subsequent frames;

[0059] The search branches continuously input the feature images of each frame, and encode the template branch information into the candidate proposals for re-detection.

[0060] In this embodiment, the search branch includes a first search branch and a spatio-temporal attention branch, and the tracking network model is trained in a triplet manner, specifically as follows:

[0061] Obtain a random frame and two adjacent frames of the UAV infrared video sequence;

[0062] Use the random frame as the input of the template branch ( branch), and use the two adjacent frames as the inputs of the first search branch ( branch) and the spatio-temporal attention (STA) branch ( branch) respectively; the purpose of introducing the STA branch is to enable the tracker to accurately locate the target in the face of both the minute differences between adjacent frames of the target and the large morphological changes over a long time.

[0063] The image frames undergo the same backbone feature extraction, and directly encode the target information of the template branch into the cross-correlation RPN re-detection in the first stage;​

[0064] In the second-stage re-detection, the triplet training needs to perform multiple tasks, including: predicting the target position using the similarity learning between branches and ; transferring and learning spatio-temporal attention knowledge from the branch; performing SFD in the independent branch for UAV foreground semantic learning.

[0065] In this embodiment, to simplify the training, the interest target region features of branches and are connected, and a common similarity learning RCNN is used to calculate the output.

[0066] The re-detection cross-correlation RPN network in the first stage uses binary cross-entropy and smooth as the classification and regression losses respectively to train the model:

[0067]

[0068] where represents the predicted classification value, represents the predicted bounding box regression offset, and are the true label and the true bounding box respectively, and represent the normalization parameters for classification and regression, is the balance parameter of these two loss functions;

[0069] The re-detection RCNN network in the second stage sets the following loss function to be minimized:

[0070]

[0071] where represents the similarity learning loss between the template branch and the search branch, represents the supervised UAV foreground detection loss, represents the spatio-temporal attention loss between adjacent frames.

[0072] The re-detection includes two stages. The first stage is the cross-correlation RPN re-detection, which uses a large number of anchor box candidate regions generated by the cross-correlation RPN network;

[0073] The specific operation of the cross-correlation RPN re-detection is as follows:

[0074]

[0075] where and respectively represent the features of the template and the search image, is a sized convolutional filter that transforms into a matrix. The target information carried by is encoded into the feature map by the cross-correlation layer, and then the output is transformed into channels through a convolutional layer; is the cross-encoded feature output, which has the same data structure as and is input into the RPN head for classification and regression; and respectively represent the convolution and cross-correlation operations.

[0076] The second stage is the similarity learning RCNN re-detection, which uses the similarity learning RCNN network to encode the prediction results of the historical frames into the search candidate regions; the RCNN network includes a template memory repository and an RCNN detection head.

[0077] The specific operations of the similarity learning RCNN re-detection are as follows:

[0078]

[0079] First, perform and convolution with padding with one pixel. Since and also have the same size, the Hadamard product is used to encode the correlation relationship between them. Similar to the RPN, the number of channels is transformed back to using the convolution method. Finally, the similarity-encoded is sent into the RCNN head to replace for subsequent classification and regression predictions.

[0080] Among them, is the target alignment feature of the template branch , is the alignment feature of the candidate regions from the search branch , where M is the number of candidate proposals, and ⊙ represents the Hadamard product.

[0081] Specifically, in this embodiment, the evaluation scores of the candidate proposals belonging to the trajectory are calculated in the following way:

[0082]

[0083] In the formula represents the target ground truth given in the first frame, represents the search candidate region of the branch in the test frame ; and are complementary ratios; represents the similarity evaluation, which sends the similarity coding features of and into the RCNN head and returns the confidence score of re-detection; is the position consistency evaluation, where the Intersection over Union (IoU) is used to impose neighborhood restrictions.

[0084] Given that the target does not undergo sudden and large position changes in two consecutive frames. When it is found that of is greater than a given threshold, the neighborhood search is considered successful, and at this time will be added to the sub-trajectory ; Otherwise, it means that no suitable target is found in the neighborhood, and the sub-trajectory will be terminated, and the candidate region with the highest global score will be output as the prediction result. Due to the limitation of spatial position, candidate proposals with similar target appearances will be excluded from the decision-making, which greatly reduces the interference of background thermal noise.

[0085] In this embodiment, in the similarity learning RCNN re-detection, a template memory repository is introduced to make the best decision for the current frame tracking task by fully utilizing the template of the first frame and the prediction results of historical frames. The high-confidence prediction results of all historical frames are concatenated into a feature matrix, and the feature matrix is cross-coded with the candidate regions generated by the cross-correlation RPN re-detection and then input into the RCNN detection head;

[0086] Specifically as follows:

[0087] Record the target alignment features with high confidence in the historical frame prediction, denoted as , and record their corresponding frame numbers , , where represents the capacity of the memory bank, and these alignment features are also called sequence decision experts. To cover as many possible target states as possible, calculate the frame interval between adjacent serial number experts. If the capacity of the repository is fully occupied and new high-confidence prediction results appear, then delete the larger one among the sequence experts with the smallest frame interval;

[0088] Repeat the solution for each sequence expert With respect to the search area of the similarity score, then it is necessary to perform repeated calculations, which is a huge waste of computing resources. In this embodiment, for fast calculation, several of the best-performing candidate region proposals are selected in , and these proposals have high confidence scores output in the first-stage cross-correlation RPN re-detection. Since the similarity learning RCNN is a mirror process, can also be regarded as a search area, and then the search area is encoded in reverse into . Through similarity learning and concatenation, the encoded feature matrix is obtained. Therefore, the evaluation score of each candidate proposal can be calculated, that is:

[0089]

[0090] wherein, represents the th sequence decision expert, and represents the time weighting factor. The above formula shows that in the current tracking task, a large number of historical predicted target states will also participate in the scoring of candidate regions in the current frame, avoiding the template update problem. The prediction score of each sequence decision expert is directly generated by a shared RCNN structure, so no large amount of additional computational cost is introduced.

[0091] Based on the above consideration of spatial consistency, the final prediction score is calculated as follows:

[0092]

[0093] In the above formula, the first term performs re-detection of the template of the first frame, the second term imposes spatial constraints, and the third term performs re-detection of the historical prediction results. The confidence scores returned by all three terms are within . It should be noted that since all high-quality prediction results in the trajectory are considered, a possible error in a sub-trajectory will not cause the model to completely fail.

[0094] When a high-confidence sub-trajectory is damaged, a global repair strategy is required to drive the tracker to return to the correct trajectory. In this embodiment, in the tracking network model, an independent unmanned aerial vehicle target detection module, that is, a Supervised Foreground Detector (SFD), is introduced to mitigate the impact caused by target loss.

[0095] The supervised foreground detector uses an independent RCNN detection head to predict the foreground of the UAV; the supervised foreground detector shares weights with the RCNN detection head of the similarity learning RCNN network and can be trained simultaneously in an end-to-end manner.

[0096] The supervised foreground detector does not interact with the template branch and can directly output the probability that the target is a UAV target; the supervised foreground detector independently predicts the target for each frame and is not troubled by the changes in the target appearance, which is beneficial to repairing tracking failures.

[0097] Specifically, first, a binary class label (foreground or background) is assigned to each search region; the anchor boxes with an overlap greater than 0.7 with the true target box are given a positive label, while other background anchor boxes are given a negative label. These labeled anchor boxes are trained by constructing a two-stage supervised foreground detector to enable the model to have the ability to predict whether each candidate region is a foreground target. In this embodiment, the foregrounds are all UAV targets, so training the SFD can also be regarded as training the UAV detector. The SFD shares the feature extraction structure and the cross-correlation RPN network structure.

[0098] Then, the candidate region features generated by the cross-correlation RPN network are input into the RCNN structure for binary classification prediction, and the tracker can obtain the prior probability that the candidate region is a UAV.

[0099] Since global search is more effective for target occlusion but will introduce more background noise, while local tracking limits background noise to a greater extent but it is difficult to capture the target again after the target is lost.

[0100] In this embodiment, the tracking network model further includes a state-aware converter, which switches between local tracking and global search according to the consistency condition to make the best decision.

[0101] In the next frame of the high-confidence prediction result, local tracking is performed to search for the target, and the consistency condition is , if the spatial consistency function outputs zero, then local tracking fails and global search is started. According to 's function, the best prediction is found from the global candidate regions; among them, represents the end frame of a certain sub-trajectory in the current high-confidence tracking trajectory; let and the best target box predicted by the SFD be and , and their prediction confidences are and . If and , it is considered that the SFD has recaptured a reliable UAV target. At this time, the predicted target with high confidence is used to activate a new sub-trajectory , and then local tracking is reused instead of global search to suppress background noise.

[0102] The present invention is not limited to the foregoing specific embodiments. The present invention extends to any new feature disclosed in this specification or any new combination, as well as any step of any new method or process disclosed or any new combination.

Claims

1. A long-term anti-UAV visual tracking method based on spatio-temporal attention knowledge, characterized in that Including: S1: Obtain the infrared video sequence of the drone, and extract features from the video frame images; S2: Construct a tracking network model and train it, input the extracted feature images into the trained tracking network model, and output the predicted position and scale; The tracking network model is constructed based on a Siamese network, including a template branch and at least one search branch; the feature images are respectively input into the template branch and the search branch; Among them, the template branch is initialized in the first frame and then remains fixed in subsequent frames; The search branch continuously inputs the feature images of each frame, and encodes the template branch information into the candidate proposals for re-detection; The re-detection includes two stages. The first stage is cross-correlation RPN re-detection, which uses a large number of anchor box Anchor candidate regions generated by the cross-correlation RPN network; The second stage is similarity learning RCNN re-detection, which uses the similarity learning RCNN network to encode the prediction results of historical frames into the search candidate regions; the RCNN network includes a template memory repository and an RCNN detection head; in the similarity learning RCNN re-detection, based on the template memory repository, the high-confidence prediction results of all historical frames are concatenated into a feature matrix, and the feature matrix is cross-encoded with the candidate regions generated by the cross-correlation RPN re-detection and then input into the RCNN detection head; The tracking network model also includes a supervised foreground detector, which uses an independent RCNN detection head to perform foreground prediction on the drone; the supervised foreground detector shares weights with the RCNN detection head of the similarity learning RCNN network; The tracking network model also includes a state-aware transformer, which switches between local tracking and global search according to the consistency condition; The consistency condition is that if the spatial consistency function outputs zero, global search is enabled, and the best prediction is found from the global candidate region according to the evaluation scores of the candidate proposals for the high-confidence tracking trajectory; where represents the end frame of a sub-trajectory in the current high-confidence tracking trajectory, represents the candidate region features generated by the cross-correlation RPN network.

2. The long-term anti-UAV vision tracking method according to claim 1, characterized in that When extracting features from the video frame images, Swin Transformer is used as the feature extractor.

3. The long-term anti-UAV visual tracking method according to claim 1, characterized in that The specific operation of the cross-correlation RPN re-detection is as follows: Among them, and represent the features of the template and the search image respectively, is a convolution filter of size is the feature output of cross-coding.

4. The long-time anti-UAV visual tracking method according to claim 1, characterized in that The specific operation of the similarity learning RCNN re-detection is as follows: Among them, is the target alignment feature of the template branch , is the alignment feature of the candidate region from the search branch , where M is the number of candidate proposals, and ⊙ represents the Hadamard product.

5. The long-term anti-UAV vision tracking method according to any one of claims 1-4, characterized in that, For the tracking network model, the search branch includes a first search branch and a spatio-temporal attention branch, and is trained in a triplet manner, specifically as follows: Obtain a random frame and two adjacent frames of the drone infrared video sequence; Use the random frame as the input of the template branch, and use the two adjacent frames as the inputs of the first search branch and the spatio-temporal attention branch respectively; The re-detection cross-correlation RPN network in the first stage uses binary cross-entropy and smoothness to train the model as the classification and regression losses respectively: wherein, represents the predicted classification value, represents the predicted bounding box regression offset, and are the true label and the true bounding box respectively, and represent the normalization parameters for classification and regression, is the balance parameter of these two loss functions; For the re-detection RCNN network in the second stage, set the following loss function to be minimized: Among them, represents the similarity learning loss between the template branch and the search branch, represents the supervised UAV foreground detection loss, represents the spatio-temporal attention loss between adjacent frames.

Images