Unmanned aerial vehicle tracking method and tracking device based on twin network

Abstract

The application provides a kind of unmanned aerial vehicle tracking method and tracking equipment based on twin network, it is related to computer vision technical field.The method comprises: determining search area and target tracking template based on tracked target;First feature of search area and first feature of target tracking template are respectively input into preset residual mixed attention module, to obtain target tracking template fusion feature and search area fusion feature;Target tracking template fusion feature and search area fusion feature are input into bilingual semantic region construction network, to obtain similarity feature map and foreground probability feature map, and generate search candidate box set;Based on bit movement dynamic threshold and the position of tracked target in current frame, the optimal candidate box is screened out in search candidate box set;Based on the quality of optimal candidate box, similarity feature map and background confidence, determine whether the first feature of target tracking template needs to be updated, until the tracking of tracked target is completed.The application can improve tracking effect.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a drone tracking method and tracking device based on twin networks. Background Technology

[0002] With the rapid development of drone technology, its use in both civilian and military scenarios has increased dramatically. Consequently, target tracking technology for drones has become an important research area.

[0003] Currently, target tracking algorithms are mainly divided into correlation filtering and deep learning algorithms. However, correlation filtering-based target tracking algorithms are susceptible to target noise and interference, making it difficult to effectively extract target features. They are also prone to errors when faced with non-rigid deformation of objects, easily leading to tracking failures. Deep learning-based methods, compared to correlation filtering methods, offer higher performance. This is primarily because deep learning models can automatically learn more discriminative and expressive feature representations from large amounts of data, thereby improving the accuracy and robustness of target tracking.

[0004] However, current deep learning models, such as the Transformer model, improve tracking robustness in complex scenarios by modeling long-range dependencies through global attention, but their high computational complexity inhibits the tracking effect. Summary of the Invention

[0005] This invention provides a drone tracking method and device based on twin networks to address the problem of poor tracking performance of current deep learning algorithms.

[0006] In a first aspect, embodiments of the present invention provide a drone tracking method based on twin networks, comprising:

[0007] The target to be tracked is determined in the initial frame, and the search area and target tracking template are determined based on the target to be tracked; wherein, the initial frame is the first frame in the acquired infrared video sequence;

[0008] The first features of the search region and the first features of the target tracking template are extracted respectively, and the first features of the search region and the first features of the target tracking template are input into the preset residual hybrid attention module to obtain the target tracking template fusion feature and the search region fusion feature.

[0009] The target tracking template fusion feature and the search region fusion feature are input into the dual semantic region construction network to obtain similarity feature map and foreground probability feature map, and generate a set of search candidate boxes;

[0010] Based on the dynamic displacement threshold and the position of the tracked target in the current frame, the optimal candidate box is selected from the search candidate box set;

[0011] Based on the quality of the optimal candidate box, similarity feature map, and background confidence, it is determined whether the first feature of the target tracking template needs to be updated until the tracking of the target is completed.

[0012] In one possible implementation, based on a dynamic displacement threshold and the position of the tracked target in the current frame, the optimal candidate box is selected from the search candidate box set, including:

[0013] Based on the historical trajectory of the tracked target, determine the dynamic threshold of displacement;

[0014] The position of the tracked target in the current frame is predicted based on Kalman filtering, and the prediction position error between the predicted position and the actual position of the tracked target is determined.

[0015] Based on the displacement dynamic threshold, the predicted position error, and the average size of each search candidate box in the sliding window, the optimal candidate box is selected from the search candidate box set; where the search candidate box is any one of the search candidate box sets.

[0016] In one possible implementation, a dynamic displacement threshold is determined based on the historical trajectory of the tracked target, including:

[0017] The displacement vector of the tracked target between adjacent frames in the historical trajectory is determined based on the sliding window.

[0018] Based on the displacement vector of the tracked target between adjacent frames, the average velocity and velocity standard deviation are determined.

[0019] The dynamic threshold of displacement is determined based on the average velocity and the velocity standard deviation.

[0020] In one possible implementation, the optimal candidate box is selected from the set of candidate boxes based on a dynamic displacement threshold, predicted position error, and the average size of each candidate box within the sliding window, including:

[0021] When the predicted position error is less than the displacement dynamic threshold, the optimal candidate box is selected from the set of candidate boxes based on the average size of each candidate box in the sliding window.

[0022] In one possible implementation, the optimal candidate box is selected from the set of search candidate boxes based on the average size of each candidate box within the sliding window, including:

[0023] The average width and average height of the bounding box of the tracked target in the historical trajectory are determined based on the sliding window.

[0024] Based on the width and height of each search candidate box, as well as the average width and average height of the bounding box of the tracked target, the optimal candidate box is selected from the search candidate box set.

[0025] In one possible implementation, based on the optimal candidate box, the quality of the similarity feature map, and the background confidence, it is determined whether the first feature of the object tracking template needs to be updated, including:

[0026] Once it is determined that a candidate box exists within the optimal candidate box, the quality of the similarity feature map and the background confidence are calculated.

[0027] When the quality of the similarity feature map is less than a preset quality threshold and the background confidence is greater than a preset background threshold, the first feature of the target tracking template is determined based on the linear interpolation method.

[0028] In one possible implementation, determining the first feature of the target tracking template based on linear interpolation includes:

[0029] Extract features of the tracked target in the current frame;

[0030] The first feature of the updated target tracking template is determined by a weighted sum of the features of the tracked target in the current frame and the first feature of the target tracking template.

[0031] In one possible implementation, the quality of the similarity feature map and the background confidence are determined as follows:

[0032] The quality of the similarity feature map is calculated based on the average peak correlation energy.

[0033] The background confidence is determined based on the confidence scores of multiple candidate interference regions generated by the sliding window within a preset range of the similarity feature map.

[0034] In one possible implementation, the residual hybrid attention module includes a channel attention branch and a spatial attention branch;

[0035] The channel attention branch is used to perform global average pooling on the input features to generate channel weights;

[0036] The spatial attention branch is used to perform bar pooling on the input features along the horizontal and vertical directions respectively to generate spatial weights;

[0037] The residual hybrid attention module performs weighted fusion of input features based on the channel weights output by the channel attention branch and the spatial weights output by the spatial attention branch.

[0038] In a second aspect, embodiments of the present invention provide a tracking device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect or any possible implementation thereof.

[0039] In this embodiment of the invention, to improve tracking quality, firstly, the target to be tracked is determined in the initial frame, and a search region and a target tracking template are determined based on the target. Next, the first features of the search region and the first features of the target tracking template are extracted respectively, and then input into a preset residual hybrid attention module to obtain target tracking template fused features and search region fused features. Then, the target tracking template fused features and search region fused features are input into a dual semantic region construction network to obtain a similarity feature map and a foreground probability feature map, and a set of search candidate boxes is generated. Then, based on a displacement dynamic threshold and the position of the target in the current frame, the optimal candidate box is selected from the search candidate box set. Finally, based on the quality of the optimal candidate box, the similarity feature map, and the background confidence, it is determined whether the first feature of the target tracking template needs to be updated until the tracking of the target is completed. Thus, by using the residual hybrid attention module to further fuse the features, the expressive power of the features can be significantly improved. By employing a dynamic displacement threshold and considering the target's position in the current frame, optimal candidate boxes that conform to motion patterns can be more accurately selected from the search candidate box set. Based on the dual constraints of similarity feature map quality and background confidence, the update frequency can be adaptively controlled, thereby improving the UAV's tracking performance. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the implementation of the UAV tracking method based on twin networks provided in this embodiment of the invention.

[0041] Figure 2 This is a block diagram of the UAV tracking method based on twin networks provided in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the residual hybrid attention module provided in an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the optimal candidate box filtering provided in an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of the optimal candidate box filtering provided in an embodiment of the present invention;

[0045] Figure 6 This is a schematic diagram of the structure of the drone tracking device provided in an embodiment of the present invention. Detailed Implementation

[0046] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0047] As described in the background section, Transformer architectures such as SwinTrack can further improve tracking robustness in complex scenarios by modeling long-range dependencies with global attention. However, their computational complexity is high, making it difficult to directly adapt to infrared small target scenarios.

[0048] Furthermore, due to the lack of color features in thermal infrared images, current common tracking methods still have some problems when tracking infrared UAV targets, such as dynamic background clutter in complex infrared scenes, blurring of small target features, and rapid motion deformation, which seriously restrict the performance of current tracking algorithms.

[0049] See Figure 1 and Figure 2 The document illustrates a flowchart of the implementation of the UAV tracking method based on twin networks provided in this embodiment of the invention, which is described in detail below:

[0050] S110. Determine the target to be tracked in the initial frame, and determine the search area and target tracking template based on the target to be tracked.

[0051] The initial frame is the first frame in the acquired infrared video sequence.

[0052] After acquiring the video sequence of the target to be tracked, the target must first be identified in the first frame, i.e., the initial frame. Then, the search area is dynamically determined based on the size of the target itself, and a target tracking template is determined with the target as the center. The size of the target tracking template is smaller than the size of the search area.

[0053] Specifically, the search area needs to completely cover the tracked target, expanding outward from the tracked target to ensure that the tracked target is located within the search area.

[0054] S120. Extract the first feature of the search area and the first feature of the target tracking template respectively, and input the first feature of the search area and the first feature of the target tracking template into the preset residual hybrid attention module to obtain the target tracking template fusion feature and the search area fusion feature.

[0055] In some embodiments, the Swing Transformer backbone network can be used to extract multi-scale features from the search region and the target tracking template, respectively, to obtain the first feature of the search region and the first feature of the target tracking template.

[0056] In some embodiments, to enable cross-channel and spatial dimension interaction enhancement between the first features of the obtained multi-scale search region and the first features of the target tracking template, and to address the problems of significant background interference and easy loss of target details, this invention also constructs a residual hybrid attention module (RHAM), such as... Figure 3 As shown, this module can output fused features that include spatial and semantic information.

[0057] In this embodiment, the residual hybrid attention module includes a channel attention branch and a spatial attention branch. The channel attention branch performs global average pooling on the input features to generate channel weights. The spatial attention branch performs bar pooling on the input features along the horizontal and vertical directions respectively to generate spatial weights. The residual hybrid attention module performs weighted fusion on the input features based on the channel weights output by the channel attention branch and the spatial weights output by the spatial attention branch.

[0058] Specifically, such as Figure 2 As shown, the channel attention branch includes sequentially connected adaptive average pooling layers, 2D convolutional layers, ReLU activation functions, 2D convolutional layers, sigmoid functions, and channel weight output layers. The spatial attention branch includes parallel 2D max pooling layers and 2D average pooling layers, and the outputs of both the 2D max pooling layers and 2D average pooling layers are connected to tensor concatenation layers. After the tensor concatenation layers, sequentially connected are 2D convolutional layers, sigmoid functions, and spatial weight output layers.

[0059] Specifically, the channel attention branch performs global average pooling (GAP) on the input feature map to generate channel description vectors z∈R. c Channel weights W are generated through two fully connected layers. c The calculation formula is:

[0060] W c =σ(W2·ReLU(W1·GAP(X)));

[0061] Where σ is the Sigmoid function, GAP(·) represents global average pooling, and W1 and W2 are fully connected layers with a reduction ratio of 16.

[0062] The spatial attention branch performs bar pooling on the input feature map along both the horizontal and vertical directions to generate feature F. h ∈R C×H With F v ∈R C×W After splicing, spatial weights W are generated by 3×3 convolution. s The calculation formula is:

[0063] W s =σ(Conv 3×3(Concat(F h ,F v )));

[0064] After obtaining the channel weight W c and spatial weight W s Then, based on these two weights, the first feature of the search area and the first feature of the target tracking template will be weighted and fused separately.

[0065] RHAM(X)=X+X⊙W c +X⊙W s ;

[0066] Where X is the first feature of the input search region or the first feature of the target tracking template, ⊙ represents element-wise multiplication, and the output is the enhanced fused feature.

[0067] Compared with the traditional single-branch attention mechanism, the residual hybrid attention module (RHAM) constructed in this invention can significantly improve the expressive power of the tracked template features under infrared low signal-to-noise ratio conditions, effectively balancing the conflict between detail preservation and background suppression of the tracked target.

[0068] S130. Input the target tracking template fusion feature and the search area fusion feature into the dual semantic region construction network to obtain the similarity feature map and the foreground probability feature map, and generate a set of search candidate boxes.

[0069] Dual Semantic Region Construction Network (DS-RPN) is a network that uses two types of semantic information to generate candidate boxes simultaneously: target matching semantics to determine the similarity between the template and the search region, and foreground probability semantics to determine the probability that the candidate box contains the target.

[0070] The specific processing steps for constructing a dual semantic region network are as follows:

[0071] First, an initial candidate region is generated using preset anchor frames.

[0072] Next, the target matching semantics calculates the similarity between the target tracking template fusion features and the search region fusion features through deep convolution, and outputs a similarity feature map.

[0073]

[0074] in, This indicates a depthwise convolution operation.

[0075] Foreground probabilistic semantics predicts the probability that a candidate bounding box contains the target by using independent fully connected layers. The probability of the background region approaches zero.

[0076] The results of target matching semantics and foreground probabilistic semantics are concatenated along the channel dimension and input into the RPN header for final candidate box filtering and regression, generating a search candidate box set containing multiple candidate boxes. The loss function can be defined as:

[0077]

[0078] Where λ is the classification loss L cls With regression loss L reg The balance coefficient.

[0079] S140. Based on the displacement dynamic threshold and the position of the tracked target in the current frame, the optimal candidate box is selected from the search candidate box set.

[0080] In some embodiments, after determining the set of candidate search boxes, the optimal candidate box can be selected from the set based on relevant constraints. Firstly, a dynamic displacement threshold can be determined based on the historical trajectory of the tracked target.

[0081] Then, the position of the tracked target in the current frame is predicted based on Kalman filtering, and the prediction position error between the predicted position and the actual position of the tracked target is determined.

[0082] Finally, based on the displacement dynamic threshold, the predicted position error, and the average size of each candidate bounding box within the sliding window, the optimal candidate bounding box is selected from the set of candidate bounding boxes. Here, a candidate bounding box is any one of the candidate bounding boxes in the set.

[0083] In this embodiment, the displacement vector of the tracked target between adjacent frames in the historical trajectory can be determined first based on a sliding window.

[0084] Then, based on the displacement vector of the tracked target between adjacent frames, the average velocity and velocity standard deviation are determined.

[0085] Finally, the dynamic threshold of displacement is determined based on the average velocity and the velocity standard deviation.

[0086] Specifically, the center point coordinates of the tracked target can be recorded in T frames of the historical trajectory using a sliding window. and the size of the bounding box of the tracked target (w) t h t ), calculate average speed With standard deviation σ v The historical trajectory was obtained from the acquired infrared video sequence.

[0087] Here, we can set the sliding window size T = 5, the update rate α = 0.9, and the displacement vector as follows:

[0088] d k=(cx k -cx k-1 ,cy k -cy k-1 ).

[0089] The formula is as follows:

[0090]

[0091] After determining the average speed With standard deviation σ v Then, you can set the maximum allowed displacement threshold for the current frame. Maximum allowable displacement threshold Set as the dynamic threshold for displacement.

[0092] In this embodiment, when the predicted position error is less than the displacement dynamic threshold, the optimal candidate box can be selected from the set of search candidate boxes based on the average size of each search candidate box in the sliding window.

[0093] Specifically, one can first determine the average width and average height of the bounding box of the tracked target in the historical trajectory based on the sliding window.

[0094] Then, based on the width and height of each search candidate box, as well as the average width and average height of the bounding box of the tracked target, the optimal candidate box is selected from the search candidate box set.

[0095] Specifically, prediction needs to be performed first using Kalman filtering, with the state vector x... t =[cx,cy,vx,vy] T The state transition equation is:

[0096]

[0097] Where, Δt=1, This represents process noise. The observation model z t =Hx t +v t middle,

[0098] For the observation matrix, To observe noise, predict location. The formula for calculating the predicted location error is as follows:

[0099]

[0100] After determining the predicted position error, when the predicted position error e is determined... p Less than the dynamic displacement threshold τ vFurthermore, it is also necessary to introduce size mutation constraints to further determine how to select the optimal candidate box from the search candidate box set.

[0101] The average width of the bounding box of the tracked target in the historical trajectory is determined based on a sliding window. and average height

[0102]

[0103] When the width w′ and height h′ of any search candidate box satisfy... and When the condition is met, the search candidate box is determined as the optimal candidate box. Next, based on the above method, it is determined whether other search candidate boxes satisfy the above size constraints. If they do, they are also considered optimal candidate boxes.

[0104] It should be noted that there can be multiple candidate boxes for the optimal candidate box.

[0105] Furthermore, when the predicted location error e is determined p Greater than the dynamic displacement threshold τ v If the condition is met, the candidate box in the filter will be removed directly.

[0106] By dynamically determining the displacement threshold through historical trajectories, using Kalman filtering to predict position errors, and combining this with size mutation constraints, the optimal candidate boxes that conform to the motion patterns can be selected more accurately from the search candidate box set. This approach demonstrates higher trajectory integrity in fast-moving and complex trajectory scenarios for UAVs, especially when the target makes sharp turns or hovers, with a significantly lower false screening rate than strategies using a fixed threshold.

[0107] S150. Based on the quality of the optimal candidate box, similarity feature map, and background confidence, determine whether the first feature of the target tracking template needs to be updated until the tracking of the target is completed.

[0108] By using motion constraints and size constraints, we can ensure that the determined optimal candidate box is more in line with the motion laws.

[0109] In some embodiments, to suppress low-quality template contamination and update the interfering feature library to reduce the false detection rate, it is also necessary to dynamically update the first feature of the target tracking template based on the quality of the similarity feature map and the background confidence. For example... Figure 5 As shown, we can first calculate the quality of the similarity feature map and the background confidence when it is determined that there is a candidate box in the optimal candidate box.

[0110] Then, when the quality of the similarity feature map is less than a preset quality threshold and the background confidence is greater than a preset background threshold, the first feature of the target tracking template is determined based on the linear interpolation method.

[0111] When it is determined that there is no candidate box in the optimal candidate box, there is no need to calculate the quality of the similarity feature map and the background confidence, and there is no need to update the first feature of the target tracking template.

[0112] In this embodiment, features of the tracked target in the current frame can be extracted first.

[0113] Then, based on the weighted sum of the features of the tracked target in the current frame and the first feature of the target tracking template, the first feature of the updated target tracking template is determined.

[0114] In this embodiment, the quality of the similarity feature map can be calculated based on the average peak correlation energy. The background confidence can be determined based on the confidence scores of multiple candidate interference regions generated by a sliding window within a preset range of the similarity feature map.

[0115] Specifically, the sharpness of the similarity feature map is quantified using the mean peak correlation energy (APCE), and the quality of the similarity feature map is calculated using the following formula:

[0116]

[0117] Where N is the mode of pixels and R is the similarity feature map.

[0118] Furthermore, multiple candidate interference regions are generated within a preset range of the similarity feature map using a sliding window, and the confidence scores of these multiple candidate interference regions are predicted. Therefore, the background confidence s can be determined based on the confidence scores of multiple candidate interference regions. b The specific calculation process is as follows:

[0119]

[0120] K represents the number of candidate interference regions.

[0121] When the quality APCE of the similarity feature map is less than the preset quality threshold τ1 and the background confidence s b When the value is greater than the preset background threshold τ2, the similarity feature map is blurry and unclear, the tracked target is occluded or interfered with and there is a significant background interference area. Therefore, it is determined that the first feature of the target tracking template needs to be updated.

[0122] When it is determined that the first feature of the target tracking template needs to be updated, a linear interpolation strategy can be used for updating. The specific update method is as follows:

[0123] φ new =α·φ current +(1-α)·φ old ;

[0124] Where, φ current φ represents the features of the tracked target in the current frame. old α represents the first feature of the target tracking template, which is the first feature of the target tracking template that has not been updated, and α represents the update rate.

[0125] In addition, after updating the first feature of the target tracking template, it is also necessary to record the background region features with a confidence level higher than the threshold in the current frame for similarity matching in subsequent frames.

[0126] After updating the first feature of the target tracking template, the optimized best candidate box, along with the ID and confidence score of the tracked target, is output. Tracking continues until the entire video is tracked.

[0127] When updating the first feature of the target tracking template, this invention uses the quality APCE of the similarity feature map and the background confidence s b The dual constraints allow for adaptive control of the update frequency. In dynamic background clutter scenarios, it effectively avoids the impact of low-quality template accumulation on tracking continuity.

[0128] The tracking method provided by this invention, in order to improve tracking quality, firstly, identifies the target to be tracked in the initial frame and determines the search region and target tracking template based on the target. Next, it extracts the first features of the search region and the first features of the target tracking template, respectively, and inputs them into a preset residual hybrid attention module to obtain target tracking template fused features and search region fused features. Then, it inputs the target tracking template fused features and search region fused features into a dual semantic region construction network to obtain a similarity feature map and a foreground probability feature map, and generates a set of search candidate boxes. Then, based on a displacement dynamic threshold and the position of the target in the current frame, the optimal candidate box is selected from the search candidate box set. Finally, based on the quality of the optimal candidate box, the similarity feature map, and the background confidence, it is determined whether the first feature of the target tracking template needs to be updated until the tracking of the target is completed. Thus, by using the residual hybrid attention module to further fuse the features, the expressive power of the features can be significantly improved. By employing a dynamic displacement threshold and considering the target's position in the current frame, optimal candidate boxes that conform to motion patterns can be more accurately selected from the search candidate box set. Based on the dual constraints of similarity feature map quality and background confidence, the update frequency can be adaptively controlled, thereby improving the UAV's tracking performance.

[0129] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0130] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0131] Figure 6 A schematic diagram of a drone tracking device based on a twin network according to an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, and are described in detail below:

[0132] like Figure 6 As shown, the drone tracking device 600 based on twin networks includes:

[0133] The determination module 610 is used to determine the tracked target in the initial frame and determine the search area and target tracking template based on the tracked target; wherein, the initial frame is the first frame in the acquired infrared video sequence;

[0134] The feature fusion module 620 is used to extract the first feature of the search area and the first feature of the target tracking template respectively, and input the first feature of the search area and the first feature of the target tracking template into the preset residual hybrid attention module to obtain the target tracking template fusion feature and the search area fusion feature;

[0135] The generation module 630 is used to input the target tracking template fusion feature and the search area fusion feature into the dual semantic region construction network to obtain the similarity feature map and the foreground probability feature map, and generate a set of search candidate boxes;

[0136] The filtering module 640 is used to filter the optimal candidate box from the search candidate box set based on the displacement dynamic threshold and the position of the tracked target in the current frame.

[0137] The update module 650 is used to determine whether the first feature of the target tracking template needs to be updated based on the quality of the optimal candidate box, the similarity feature map, and the background confidence, until the tracking of the target is completed.

[0138] In one possible implementation, the filtering module 640 is used to determine a dynamic displacement threshold based on the historical trajectory of the tracked target;

[0139] The position of the tracked target in the current frame is predicted based on Kalman filtering, and the prediction position error between the predicted position and the actual position of the tracked target is determined.

[0140] Based on the displacement dynamic threshold, the predicted position error, and the average size of each search candidate box in the sliding window, the optimal candidate box is selected from the search candidate box set; where the search candidate box is any one of the search candidate box sets.

[0141] In one possible implementation, the filtering module 640 is used to determine the displacement vector of the tracked target between adjacent frames in the historical trajectory based on a sliding window;

[0142] Based on the displacement vector of the tracked target between adjacent frames, the average velocity and velocity standard deviation are determined.

[0143] The dynamic threshold of displacement is determined based on the average velocity and the velocity standard deviation.

[0144] In one possible implementation, the filtering module 640 is used to filter the optimal candidate box from the set of search candidate boxes based on the average size of each search candidate box in the sliding window when the predicted position error is less than the displacement dynamic threshold.

[0145] In one possible implementation, the filtering module 640 is used to determine the average width and average height of the bounding box of the tracked target in the historical trajectory based on a sliding window;

[0146] Based on the width and height of each search candidate box, as well as the average width and average height of the bounding box of the tracked target, the optimal candidate box is selected from the search candidate box set.

[0147] In one possible implementation, the update module 650 is used to calculate the quality of the similarity feature map and the background confidence when it is determined that a candidate box exists in the optimal candidate box;

[0148] When the quality of the similarity feature map is less than a preset quality threshold and the background confidence is greater than a preset background threshold, the first feature of the target tracking template is determined based on the linear interpolation method.

[0149] In one possible implementation, update module 650 is used to extract features of the tracked target in the current frame;

[0150] The first feature of the updated target tracking template is determined by a weighted sum of the features of the tracked target in the current frame and the first feature of the target tracking template.

[0151] In one possible implementation, update module 650 is used to calculate the quality of the similarity feature map based on the average peak correlation energy;

[0152] The background confidence is determined based on the confidence scores of multiple candidate interference regions generated by the sliding window within a preset range of the similarity feature map.

[0153] In one possible implementation, the residual hybrid attention module includes a channel attention branch and a spatial attention branch;

[0154] The channel attention branch is used to perform global average pooling on the input features to generate channel weights;

[0155] The spatial attention branch is used to perform bar pooling on the input features along the horizontal and vertical directions respectively to generate spatial weights;

[0156] The residual hybrid attention module performs weighted fusion of input features based on the channel weights output by the channel attention branch and the spatial weights output by the spatial attention branch.

[0157] The tracking device provided by this invention, in order to improve tracking quality, firstly identifies the target to be tracked in an initial frame and determines a search region and a target tracking template based on the target. Next, it extracts the first features of the search region and the first features of the target tracking template, respectively, and inputs them into a preset residual hybrid attention module to obtain fused features of the target tracking template and fused features of the search region. Then, it inputs the fused features of the target tracking template and the fused features of the search region into a dual semantic region construction network to obtain a similarity feature map and a foreground probability feature map, and generates a set of search candidate boxes. Then, based on a displacement dynamic threshold and the position of the target in the current frame, the optimal candidate box is selected from the set of search candidate boxes. Finally, based on the quality of the optimal candidate box, the similarity feature map, and the background confidence, it is determined whether the first feature of the target tracking template needs to be updated until the tracking of the target is completed. Thus, by using the residual hybrid attention module to further fuse the features, the expressive power of the features can be significantly improved. By employing a dynamic displacement threshold and considering the target's position in the current frame, optimal candidate boxes that conform to motion patterns can be more accurately selected from the search candidate box set. Based on the dual constraints of similarity feature map quality and background confidence, the update frequency can be adaptively controlled, thereby improving the UAV's tracking performance.

[0158] This invention also provides a tracking device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the above method embodiments.

[0159] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0160] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A drone tracking method based on twin networks, characterized in that, include: The target to be tracked is determined in the initial frame, and the search area and target tracking template are determined based on the target to be tracked; wherein, the initial frame is the first frame in the acquired infrared video sequence; The first features of the search region and the first features of the target tracking template are extracted respectively, and then input into a preset residual hybrid attention module to obtain the target tracking template fused features and the search region fused features. The residual hybrid attention module includes a channel attention branch and a spatial attention branch. The channel attention branch performs global average pooling on the input features to generate channel weights. The spatial attention branch performs strip pooling on the input features along the horizontal and vertical directions to generate spatial weights. The residual hybrid attention module performs weighted fusion of the input features based on the channel weights output by the channel attention branch and the spatial weights output by the spatial attention branch. The target tracking template fusion feature and the search region fusion feature are input into the dual semantic region construction network to obtain a similarity feature map and a foreground probability feature map, and to generate a set of search candidate boxes; Based on the displacement dynamic threshold and the position of the tracked target in the current frame, the optimal candidate box is selected from the search candidate box set; Based on the optimal candidate box, the quality of the similarity feature map, and the background confidence, it is determined whether the first feature of the target tracking template needs to be updated until the tracking of the target is completed; The step of selecting the optimal candidate box from the search candidate box set based on the displacement dynamic threshold and the position of the tracked target in the current frame includes: determining the displacement dynamic threshold based on the historical trajectory of the tracked target; predicting the position of the tracked target in the current frame based on Kalman filtering, and determining the prediction position error between the predicted position and the actual position of the tracked target; when the prediction position error is less than the displacement dynamic threshold, selecting the optimal candidate box from the search candidate box set based on the average size of each search candidate box in the sliding window; wherein, the search candidate box can be any one of the search candidate box sets; The step of determining whether to update the first feature of the target tracking template based on the optimal candidate box, the quality of the similarity feature map, and the background confidence includes: when it is determined that there is a candidate box in the optimal candidate box, calculating the quality of the similarity feature map and the background confidence; when the quality of the similarity feature map is less than a preset quality threshold and the background confidence is greater than a preset background threshold, determining to update the first feature of the target tracking template based on the linear interpolation method.

2. The UAV tracking method based on twin networks according to claim 1, characterized in that, Determining the displacement dynamic threshold based on the historical trajectory of the tracked target includes: The displacement vector of the tracked target between adjacent frames in the historical trajectory is determined based on a sliding window. Based on the displacement vector of the tracked target between adjacent frames, the average velocity and the velocity standard deviation are determined. The displacement dynamic threshold is determined based on the average velocity and the velocity standard deviation.

3. The UAV tracking method based on twin networks according to claim 1, characterized in that, The step of selecting the optimal candidate box from the set of search candidate boxes based on the average size of each candidate box in the sliding window includes: The average width and average height of the bounding box of the tracked target in the historical trajectory are determined based on the sliding window. Based on the width and height of each search candidate box, and the average width and average height of the bounding box of the tracked target, the optimal candidate box is selected from the set of search candidate boxes.

4. The UAV tracking method based on twin networks according to claim 1, characterized in that, The determination of the first feature for updating the target tracking template based on the linear interpolation method includes: Extract the features of the tracked target in the current frame; The first feature of the updated target tracking template is determined by a weighted sum of the features of the tracked target in the current frame and the first feature of the target tracking template.

5. The UAV tracking method based on twin networks according to any one of claims 1-4, characterized in that, The method for determining the quality and background confidence of the similarity feature map is as follows: The quality of the similarity feature map is calculated based on the average peak correlation energy; The background confidence is determined based on the confidence scores of multiple candidate interference regions generated by the sliding window within a preset range of the similarity feature map.

6. A tracking device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 5.

Images