Air-to-air cluster multi-unmanned aerial vehicle target tracking method based on attitude attention
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INST OF TECH
- Filing Date
- 2025-09-16
- Publication Date
- 2026-04-24
Smart Images
Figure CN121921339A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to an air-to-air swarm multi-UAV target tracking method based on attitude attention, belonging to the field of aircraft control technology. Background Technology
[0002] Air-to-air target tracking technology for swarms of multiple UAVs is a key task for achieving swarm perception and can be used for air traffic planning, countering unauthorized UAV flights, and UAV postal services.
[0003] Visual tracking methods have advantages such as strong anti-interference ability, low application cost and rich information. They are the main technology and development direction for air-to-air swarm multi-UAV target tracking. However, swarms are often composed of multiple UAVs of the same model. From an air-to-air perspective, the characteristics of multiple UAV targets within the swarm are highly homogeneous. In addition, compared with general targets, the characteristics of swarm UAV targets are more similar.
[0004] Existing multi-target tracking methods suffer from the following problems when dealing with the homogeneity of target features and the similarity of features between targets:
[0005] (1) Most existing multi-target tracking association algorithms establish target association in a short period of time (two consecutive frames), while the features of targets in a cluster of UAVs are highly similar in the short period of time, resulting in a low success rate of target association.
[0006] (2) Existing feature extraction components are difficult to extract detailed feature differences between different UAV targets of the same model, and the accuracy of the algorithm is low when using appearance features to distinguish within a class.
[0007] (3) Targets within the cluster have the same short-term movement trend. For targets within the cluster that are close in location, it is difficult for the algorithm to distinguish them based on the short-term position and movement characteristics of the targets, resulting in a low association success rate. Existing target tracking methods.
[0008] Therefore, it is necessary to conduct more in-depth research on existing multi-UAV target tracking methods to solve the above problems. Summary of the Invention
[0009] To overcome the above problems, the inventors conducted in-depth research and proposed an air-to-air swarm multi-UAV target tracking method based on attitude attention, characterized by comprising:
[0010] S1. Detect images in the current frame and historical frames to obtain the confidence level and appearance features of the target in each frame.
[0011] S2. Based on confidence level, the targets are divided into high-confidence targets and low-confidence targets;
[0012] S3. For high-confidence targets and low-confidence targets, perform multi-frame target association to achieve tracking of UAV targets.
[0013] In a preferred embodiment, in S2, a high threshold and a low threshold are set, targets with confidence levels below the low threshold are discarded, and targets with confidence levels above the low threshold are retained. Targets with confidence levels above the high threshold are designated as high-confidence targets, and targets with confidence levels between the high and low thresholds are designated as low-confidence targets.
[0014] In a preferred embodiment, in S3, for high-confidence targets, inter-frame target association is performed using the following method:
[0015] S301. Extract feature maps F of all high-confidence targets in the image. t ;
[0016] S302. Divide the target feature map into multiple sub-feature maps;
[0017] S303. Convert each sub-feature map into a feature vector and expand it along the channel direction to form a feature embedding.
[0018] S304. The feature embedding is enhanced by a self-attention mechanism, and then the enhanced block feature vector is obtained by passing it through a multilayer perceptron.
[0019] S305. The enhanced block feature vector and feature embedding are subjected to a cross-attention mechanism, and then passed through a multi-layer perceptron to obtain multi-frame pose enhancement features.
[0020] S306. Based on the enhanced block feature vector and multi-frame pose enhancement features, obtain the block similarity matrix of each high-confidence target in frame t and all high-confidence targets under the sliding time window T in different sub-feature maps;
[0021] S307. Fuse the block similarity matrix to obtain the attitude feature similarity between each high-confidence target and all high-confidence targets in the previous time window. Based on the attitude feature similarity, obtain the attitude similarity between the target in frame t and the tracking segment in the sliding time window T.
[0022] S308. Based on the similarity level, match different targets in the current frame with different targets in historical frames to achieve target association between multiple frames.
[0023] In a preferred embodiment, in S302, the target feature map is split into three sub-maps along its width: left, middle, and right.
[0024] In a preferred embodiment, the width ratio p of the left, middle, and right sub-images is set to 1:2:1.
[0025] In a preferred embodiment, in S303, each sub-feature map is transformed into a 2-dimensional feature vector.
[0026] In a preferred embodiment, in S304, the enhanced block feature vector F j " is represented as:
[0027]
[0028] in, These are the query matrix, key matrix, and value matrix obtained after weighted computation of feature embeddings, respectively. SA represents self-attention mechanism, MLP represents multilayer perceptron, and ReLU represents linear rectified function.
[0029] In a preferred embodiment, in S305, the enhanced block feature vectors F″ corresponding to all frames within the sliding time window T are... j With the feature embedding F of frame t t j 'A cross-attention mechanism is performed, followed by a multi-layer perceptron to obtain multi-frame pose enhancement features F'. t j ", is represented as:
[0030]
[0031] in, For vector F respectively t j ',F j "The query matrix, key matrix, and value matrix obtained after weighted calculation; CA represents the cross-attention mechanism."
[0032] In a preferred embodiment, in S306, the block similarity matrix G j By enhancing the multi-frame pose feature F j "and the enhanced block feature vector F t j "Obtained by matrix cross product, represented as:
[0033]
[0034] In a preferred embodiment, in S307, the fusion is achieved through convolution.
[0035] The beneficial effects of this invention include:
[0036] (1) For high-confidence targets in the current frame detection results, calculate the similarity of their appearance features with all tracking segments in consecutive frames and associate them based on the long-term features of the targets; for low-confidence targets in the current frame detection results, the long-term appearance features and motion features of the targets are no longer reliable, so only calculate the position cost matrix between them and the tracking segments of the previous frame and associate them based on the short-term features of the targets. By comprehensively utilizing the long-term and short-term features of the tracking segments, the association success rate can be improved.
[0037] (2) By comparing the high-confidence target in each frame with the high-confidence appearance features of all tracking segments in the long-term time window, the differences in appearance and pose features of different targets within the class under the long-term time window are captured, thereby improving the accuracy of class-specific target differentiation. Attached Figure Description
[0038] Figure 1 The diagram illustrates a preferred embodiment of an air-to-air swarm multi-UAV target tracking method based on attitude attention according to the present invention. Detailed Implementation
[0039] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Through these descriptions, the features and advantages of the present invention will become clearer and more apparent.
[0040] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. Although various aspects of embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated otherwise.
[0041] According to the present invention, an air-to-air swarm multi-UAV target tracking method based on attitude attention is provided, such as... Figure 1 As shown, it includes:
[0042] S1. Detect images in the current frame and historical frames to obtain the confidence level and appearance features of the target in each frame.
[0043] S2. Based on confidence level, the targets are divided into high-confidence targets and low-confidence targets;
[0044] S3. For high-confidence targets and low-confidence targets, perform multi-frame target association to achieve tracking of UAV targets.
[0045] In S1, a detector is used to obtain the location and confidence level of the target in the image, and a feature extraction network is used to obtain the appearance features of the target.
[0046] Those skilled in the art can use any known method to detect targets in an image, and no particular limitation is made in this invention. For example, the YOLOX method can be used for detection.
[0047] YOLOX is a single-stage detection network that can identify targets in images and output target bounding boxes and target confidence scores.
[0048] In this invention, the appearance features of the target can be extracted using any feature extraction network, such as ResNet18.
[0049] According to the present invention, the target in the first frame of the image is initialized as a tracking segment.
[0050] In S2, a high threshold and a low threshold are set. Targets with a confidence level below the low threshold are discarded, while targets with a confidence level above the low threshold are retained. Targets with a confidence level above the high threshold are designated as high-confidence targets, while targets with a confidence level between the high and low thresholds are designated as low-confidence targets.
[0051] In this invention, the specific settings of the high threshold and low threshold are not limited, and those skilled in the art can set them freely according to actual needs.
[0052] In S3, for high-confidence targets, the following method is used for inter-frame target association:
[0053] S301. Extract feature maps F of all high-confidence targets in the image. t ;
[0054] S302. Divide the target feature map into multiple sub-feature maps;
[0055] S303. Convert each sub-feature map into a feature vector and expand it along the channel direction to form a feature embedding.
[0056] S304. The feature embedding is enhanced by a self-attention mechanism, and then the enhanced block feature vector is obtained by passing it through a multilayer perceptron.
[0057] S305. The enhanced block feature vector and feature embedding are subjected to a cross-attention mechanism, and then passed through a multi-layer perceptron to obtain multi-frame pose enhancement features.
[0058] S306. Based on the enhanced block feature vector and multi-frame pose enhancement features, obtain the block similarity matrix of each high-confidence target in frame t and all high-confidence targets under the sliding time window T in different sub-feature maps;
[0059] S307. Fuse the block similarity matrix to obtain the attitude feature similarity between each high-confidence target and all high-confidence targets in the previous time window. Based on the attitude feature similarity, obtain the attitude similarity between the target in frame t and the tracking segment in the sliding time window T.
[0060] S308. Based on the similarity level, match different targets in the current frame with different targets in historical frames to achieve target association between multiple frames.
[0061] In clusters, homogeneous targets have nearly identical appearance features at the same time. Existing target tracking methods struggle to extract detailed feature differences between different targets of the same model. In this invention, the differences in motion postures among multiple target instances, namely the differences in the relative positions of the target arms, are utilized to assist in the identification of different UAVs of the same model. Since UAV targets (including rotorcraft and fixed-wing aircraft) achieve different maneuvers through attitude adjustments between their arms or wings, feature extraction of different motion postures can better distinguish homogeneous targets.
[0062] According to a preferred embodiment of the present invention, in S301, feature maps of all high-confidence targets in the image are extracted and represented as follows: Among them, F i t ∈R C×H×W N represents the feature map of the i-th target in the t-th frame. t The number of high-confidence targets in frame t is represented by C, W, and H. Preferably, C = 256, W = 8, and H = 8.
[0063] In S302, the sub-feature map obtained after splitting can be represented as: Where j represents the different subgraphs split.
[0064] Preferably, the target feature map is split into three sub-maps along its width: left, middle, and right, i.e., j = 1, 2, 3.
[0065] This invention targets drone targets. In most datasets, drones are characterized by an aspect ratio greater than 1, and each drone in the image has a typical frame and fuselage structure. Splitting along the width can, in most cases, divide the target into the left frame (wing), fuselage, and right frame (wing), thus capturing the drone's attitude characteristics.
[0066] The width direction refers to the width direction of each UAV target image in a frame, that is, the x-axis direction of the image coordinate system.
[0067] More preferably, the width ratio p of the three sub-images (left, middle, and right) is set to 1:2:1.
[0068] Extensive research has shown that splitting the target structure in a 1:2:1 width ratio yields the best target structural features and optimal performance.
[0069] According to the present invention, a sliding time window T is set, and all high-confidence target feature maps F = Ft+1 in all frames within the sliding time window T, i.e., frames t-T+1 to t-1, are collected. t-T+1 ∪…∪F t-1 The sub-feature map set obtained by splitting the feature map into multiple sub-feature maps can be represented as F. j .
[0070] In S303, each sub-feature map is transformed into a 2-dimensional feature vector.
[0071] According to the present invention, the obtained 2D feature vector is represented as follows: N k W represents the total number of high-confidence targets in frames t-T+1 to t-1. j The width of the j-th subgraph is represented by 'embed', the vector dimension is represented by 'step', and the step size of the transformation process is represented by 'step'.
[0072] Preferably, the patch embedding method is used to convert each sub-feature map into a 2-dimensional feature vector. More preferably, the step size is set to 2 and the output vector dimension is set to embed = 512.
[0073] According to the present invention, the 2D eigenvector F jex The expanded feature embedding is represented as D = embed × f j ′.
[0074] Wherein, the feature embedding of the t-th frame is represented as
[0075] In S304, the enhanced block feature vector F j " is represented as:
[0076]
[0077] in, These are the query matrix, key matrix, and value matrix obtained after weighted computation of feature embeddings, respectively. SA represents self-attention mechanism, MLP represents multilayer perceptron, and ReLU represents linear rectified function.
[0078] In S305, the enhanced block feature vector F″ corresponding to all frames within the sliding time window T is... j With the feature embedding F of frame t t j 'A cross-attention mechanism is performed, followed by a multi-layer perceptron to obtain multi-frame pose enhancement features F'.t j ", is represented as:
[0079]
[0080] in, For vector F respectively t j ',F j "The query matrix, key matrix, and value matrix obtained after weighted calculation; CA represents the cross-attention mechanism."
[0081] Preferably, the hidden layer in the multilayer perceptron (MLP) is set to 1 layer.
[0082] In S306, the block similarity matrix G j By enhancing the multi-frame pose feature F j "and the enhanced block feature vector F t j "Obtained by matrix cross product, represented as:
[0083]
[0084] In S307, the fusion is achieved through convolution, preferably through 1x1 convolution fusion.
[0085] The similarity of the pose features is represented as follows:
[0086] G = ReLU(conv) 1×1 (G1,…,G j ,…,G L ))
[0087] Where L represents the total number of subgraphs split, conv 1×1 This represents a 1x1 convolution.
[0088] Furthermore, in frame t, target i at a certain time t n Next target τ m Similarity P between A (i=τ m |t=t n ) is represented as:
[0089]
[0090] The target i in the current frame is relative to a certain tracking segment τ within a sliding window T. m Similarity P(i=τ) m |T) is:
[0091]
[0092] In S3, for targets with low confidence, the inventors found that long-term target appearance and location features are no longer reliable, and the error rate is high when using traditional methods to match targets using appearance features or simple location features.
[0093] In this invention, the intersection-union ratio (IUU) of the target in the current frame and the target in the previous frame is calculated, and the target is matched based on the IUU ratio.
[0094] Intersection over Union (IoU) is a commonly used performance metric widely employed to evaluate the accuracy of annotation, segmentation, and object detection algorithms. This invention will not elaborate on the specific calculation method for IoU.
[0095] Furthermore, in this invention, high-confidence targets are matched first, and then low-confidence targets are matched among the unmatched targets. According to this invention, for high-confidence targets that are not matched in the current frame, they are initialized as new tracking segments for target matching in subsequent frames.
[0096] Preferably, when a target has high confidence in some frames and low confidence in others, in high-confidence frames, matching is performed according to the matching logic for high-confidence targets. That is, the target's appearance confidence is matched first, and if no match is found, it is initialized as a new target. In low-confidence frames, matching is performed according to the matching logic for low-confidence targets. At this point, after all high-confidence targets in the current frame have been matched, the remaining tracking segments are then matched with these low-confidence targets, and unmatched segments are discarded.
[0097] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.
[0098] Example
[0099] Example 1
[0100] Multi-target tracking experiments were conducted using the AIRMOT and UAVSwarm datasets, which are commonly used datasets for detection and tracking tasks. The experiments included the following steps:
[0101] S1. Detect images in the current frame and historical frames to obtain the confidence level and appearance features of the target in each frame.
[0102] S2. Based on confidence level, the targets are divided into high-confidence targets and low-confidence targets;
[0103] S3. For high-confidence targets and low-confidence targets, perform multi-frame target association to achieve tracking of UAV targets.
[0104] In S1, YOLOX is used for target recognition.
[0105] In S3, for high-confidence targets, the following method is used for inter-frame target association:
[0106] S301. Extract feature maps F of all high-confidence targets in the image. t ;
[0107] S302. Divide the target feature map into multiple sub-feature maps;
[0108] S303. Convert each sub-feature map into a feature vector and expand it along the channel direction to form a feature embedding.
[0109] S304. The feature embedding is enhanced by a self-attention mechanism, and then the enhanced block feature vector is obtained by passing it through a multilayer perceptron.
[0110] S305. The enhanced block feature vector and feature embedding are subjected to a cross-attention mechanism, and then passed through a multi-layer perceptron to obtain multi-frame pose enhancement features.
[0111] S306. Based on the enhanced block feature vector and multi-frame pose enhancement features, obtain the block similarity matrix of each high-confidence target in frame t and all high-confidence targets under the sliding time window T in different sub-feature maps;
[0112] S307. Fuse the block similarity matrix to obtain the attitude feature similarity between each high-confidence target and all high-confidence targets in the previous time window. Based on the attitude feature similarity, obtain the attitude similarity between the target in frame t and the tracking segment in the sliding time window T.
[0113] S308. Based on the similarity level, match different targets in the current frame with different targets in historical frames to achieve target association between multiple frames.
[0114] In S303, each sub-feature map is transformed into a 2D feature vector, and the resulting 2D feature vector is represented as follows: 2D eigenvector F jex The expanded feature embedding is represented as D = embed × f j ′.
[0115] In S307, the target i in the current frame is relative to a certain tracking segment τ within the sliding window T. m The similarity is:
[0116]
[0117] In the experiment, ResNet18, ResNet34, and ResNet50 were used for feature extraction, respectively.
[0118] Comparative Example 1
[0119] The same experiment as in Example 1 was conducted, except that ResNet18, ResNet34, and ResNet50 were used for feature extraction, and the dot-product method was used for target matching based on the extracted features.
[0120] The dot-product method is a typical tracking method in target detection and tracking tasks, and its specific process will not be elaborated here.
[0121] Comparing the results of Example 1 and Comparative Example 1, the commonly used HOTA, MOTA, IDF1 and IDSW were used as indicators for evaluation, and the results are shown in Table 1.
[0122] Table 1
[0123]
[0124] As shown in Table 1, for both the AIRMOT and UAVSwarm datasets, under the same ResNet network, the method in Example 1 demonstrates a significant performance improvement compared to the method in Comparative Example 1, especially in the HOTA and IDF1 metrics. This indicates that the method in Example 1 is better able to extract features between different instances of the same type of UAV, and has a higher association accuracy.
[0125] The present invention has been described above with reference to preferred embodiments; however, these embodiments are merely exemplary and illustrative. Various substitutions and modifications can be made to the present invention based on these embodiments, all of which fall within the scope of protection of the present invention.
Claims
1. A method for tracking multiple UAV targets in an air-to-air swarm based on attitude attention, characterized in that, include: S1. Detect images in the current frame and historical frames to obtain the confidence level and appearance features of the target in each frame. S2. Based on confidence level, the targets are divided into high-confidence targets and low-confidence targets; S3. For high-confidence targets and low-confidence targets, perform multi-frame target association to achieve tracking of UAV targets.
2. The air-to-air swarm multi-UAV target tracking method based on attitude attention according to claim 1, characterized in that, In S2, a high threshold and a low threshold are set. Targets with a confidence level below the low threshold are discarded, while targets with a confidence level above the low threshold are retained. Targets with a confidence level above the high threshold are designated as high-confidence targets, while targets with a confidence level between the high and low thresholds are designated as low-confidence targets.
3. The air-to-air swarm multi-UAV target tracking method based on attitude attention according to claim 1, characterized in that, In S3, for high-confidence targets, the following method is used for inter-frame target association: S301. Extract feature maps F of all high-confidence targets in the image. t ; S302. Divide the target feature map into multiple sub-feature maps; S303. Convert each sub-feature map into a feature vector and expand it along the channel direction to form a feature embedding. S304. The feature embedding is enhanced by a self-attention mechanism, and then the enhanced block feature vector is obtained by passing it through a multilayer perceptron. S305. The enhanced block feature vector and feature embedding are subjected to a cross-attention mechanism, and then passed through a multi-layer perceptron to obtain multi-frame pose enhancement features. S306. Based on the enhanced block feature vector and multi-frame pose enhancement features, obtain the block similarity matrix of each high-confidence target in frame t and all high-confidence targets under the sliding time window T in different sub-feature maps; S307. Fuse the block similarity matrix to obtain the attitude feature similarity between each high-confidence target and all high-confidence targets in the previous time window. Based on the attitude feature similarity, obtain the attitude similarity between the target in frame t and the tracking segment in the sliding time window T. S308. Based on the similarity level, match different targets in the current frame with different targets in historical frames to achieve target association between multiple frames.
4. The air-to-air swarm multi-UAV target tracking method based on attitude attention according to claim 3, characterized in that, In S302, the target feature map is split into three sub-maps along its width: left, middle, and right.
5. The air-to-air swarm multi-UAV target tracking method based on attitude attention according to claim 4, characterized in that, Set the width ratio p of the three sub-images (left, middle, and right) to 1:2:
1.
6. The air-to-air swarm multi-UAV target tracking method based on attitude attention according to claim 3, characterized in that, In S303, each sub-feature map is transformed into a 2-dimensional feature vector.
7. The air-to-air swarm multi-UAV target tracking method based on attitude attention according to claim 3, characterized in that, In S304, the enhanced block feature vector F j " is represented as: in, These are the query matrix, key matrix, and value matrix obtained after weighted computation of feature embeddings, respectively. SA represents self-attention mechanism, MLP represents multilayer perceptron, and ReLU represents linear rectified function.
8. The air-to-air swarm multi-UAV target tracking method based on attitude attention according to claim 3, characterized in that, In S305, the enhanced block feature vector F corresponding to all frames within the sliding time window T is... j "Feature embedding F with frame t" t j 'A cross-attention mechanism is performed, followed by a multi-layer perceptron to obtain multi-frame pose enhancement features F'. t j ", is represented as: in, For vector F respectively t j ',F j "The query matrix, key matrix, and value matrix obtained after weighted calculation; CA represents the cross-attention mechanism." 9. The air-to-air swarm multi-UAV target tracking method based on attitude attention according to claim 3, characterized in that, In S306, the block similarity matrix G j By enhancing the multi-frame pose feature F j "and the enhanced block feature vector F t j "Obtained by matrix cross product, represented as:
10. The air-to-air swarm multi-UAV target tracking method based on attitude attention according to claim 3, characterized in that, In S307, the fusion is achieved through convolution.