A Cross-View Correlation-Based Cooperative Target Detection Method for Dual Unmanned Aerial Vehicles

Abstract

The present invention discloses a cross-view correlation-based dual-UAV collaborative target detection method, belonging to the field of multi-view image information and multi-class target detection. The present invention constructs a dual-UAV collaborative target detection network, selects a training set, inputs the training data into the dual-UAV collaborative target detection network, outputs the predicted values of the classification detection results and the predicted values of the target bounding box detection results, calculates the classification loss and the target bounding box regression loss according to the predicted values and the ground truth in the training set, updates the internal parameters of the dual-UAV collaborative target detection network according to the sum of the losses until the total loss converges, and saves the network parameters; inputs the video or image sets from the two UAVs at the same moment into the trained target detection network to obtain the corresponding classification detection results and target bounding box detection results. The present invention enhances the feature representation of occluded targets and small targets, improves the target detection accuracy; in addition, it avoids generating candidate boxes for each task separately, saving computational resources.

Description

Technical Field

[0001] The present invention relates to the field of multi-view image information and multi-class object detection, and particularly to a cross-view associated dual-UAV collaborative object detection method. Background Art

[0002] Multi-view collaborative multi-object detection (MCMOD) aims to perform multi-class object detection based on multi-view image information with overlapping fields of view. Different from traditional single-view object detection, multi-view images are captured by different devices, providing rich angular features of the object and enabling multi-view modeling.

[0003] In recent years, object detection has witnessed rapid development. From the simple R-CNN series and YOLO series to DETR-based models, the performance of object detection has been significantly improved. Recently, YOLOV8x achieved an accuracy of 53.9 (0.5-0.95 map) on the MSCOCO dataset test-dev2017. The YOLOv8 series, based on modifying the backbone network, decoupled the detection head, separating classification and detection. At the same time, it changed from Anchor-Based to Anchor-Free. In terms of Loss calculation, it adopted the Task Aligned Assigner positive sample assignment strategy and introduced Distribution Focal Loss to improve the detection effect of the model on imbalanced data. Compared with the R-CNN and YOLO series that require post-processing operations such as Non-Maximum Suppression (NMS), the object detection framework of the DETR series does not require NMS. Instead, it uses the Hungarian matching algorithm to supervise the model output, making it correspond one-to-one with the ground truth, and eliminating redundant boxes through network learning. This eliminates the instability caused by manually designed post-processing parameters and enables end-to-end training. DETR pioneered the application of Transformer in the field of end-to-end object detection. Subsequently, numerous object detection works were based on the DETR detection framework, improving aspects such as model convergence and detection accuracy on small objects. For example, Deformable DETR, based on the idea of deformable convolution, realized deformable attention, converting the fully connected attention between feature map pixels into deformable local attention, greatly reducing the computational amount. The model convergence speed was accelerated from the original 50-200 Epochs to within 50 Epochs. Due to the reduction in computational amount, the resolution of the feature map was further improved, enhancing the detection effect of the model on small objects. In Efficient DETR, the impact of different query initialization processes on the model was verified, and a more efficient query initialization method was proposed, enabling the acceleration of model training efficiency by reducing the number of decoder layers (three-layer encoder, one-layer decoder). In subsequent work, DN-DETR defined the query as a reference box with width and height, and DAB-DETR introduced a denoising training strategy, respectively solving problems such as query interpretability and instability caused by Hungarian matching, further accelerating the model convergence speed and improving the detection accuracy.

[0004] Although existing methods have made great progress in object detection, the problems of small objects, object occlusion, and low object resolution still remain the biggest obstacles to improving the detection effect. In recent years, with the development of multi-agent systems, multi-view collaborative perception has proposed a new research idea for solving the problems of small objects and object occlusion in object detection. In the work proposed by Hou et al., the projection transformation of the feature map is used to aggregate multi-camera information, improving the pedestrian detection effect. Iskakov et al. aggregated the features of multiple views in 3D space, obtaining the feature information from all views. Based on this, the positions of key points in 3D space are predicted through 3D convolution, improving the accuracy of human pose modeling. The research found that all existing multi-view collaborative perception methods require accurate camera pose parameters to facilitate 3D modeling, obtain the mapping relationship between 2D images and 3D space, and aggregate multi-view information in 3D space. In the UAV operation environment, the perspective deviation caused by camera jitter brings instability to 3D space modeling, making it impossible to achieve multi-view fusion through spatial modeling. Summary of the Invention

[0005] In view of this, the present invention proposes a cross-view associated dual-UAV collaborative object detection method. The present invention constructs a feature interaction network based on the attention mechanism to perform interaction and fusion between multi-view information at the high-dimensional feature space level, enhancing the feature representation of occluded objects and small objects, and improving the object detection accuracy. In addition, a candidate box sharing module is designed, where different object detection tasks share the same set of candidate boxes, which can avoid generating candidate boxes separately for each task, thus saving computational resources.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A cross-view associated dual-UAV collaborative object detection method, comprising the following steps:

[0008] Step 1, construct a dual-UAV collaborative object detection network, the dual-UAV collaborative object detection network includes a feature interaction network based on the attention mechanism, a candidate box generator, and a decoder; the feature interaction network includes two groups of four-layer feature extraction modules generated based on the residual deep network and the feature pyramid network, a first self-attention module, a first cross-attention module, and an upsampling module; the decoder includes a candidate box sharing module, a second self-attention module, and a second cross-attention module;

[0009] Step 2, select the MDMT dataset as the training set, and use the image A taken by UAV A and the image B taken by UAV B at the same time but from different perspectives as a set of training input values;

[0010] Step 3: Input image A and image B into their respective four-layer feature extraction modules, and obtain the feature maps x output by the four-layer feature extraction modules AL and x BL , where L = 1, 2, 3, 4. L = 1 corresponds to the topmost feature extraction module, and L = 4 corresponds to the bottommost feature extraction module;

[0011] Step 4: Set the number of single heads M and the number of sampling points K. Sequentially input the K feature points on the feature maps x AL and x BL into the first self-attention module to obtain the corresponding multi-head self-attention calculation results q' A and q' B :

[0012]

[0013] where MSDeformAttn represents the calculation of multi-head self-attention, and q A and q B respectively represent the query vectors corresponding to a feature point on the feature maps x AL and x BL ; and respectively represent the coordinates of the feature point query vectors q A and q B ; and respectively represent the sequential calculation of the four-layer feature maps x AL and x BL ; ZW m represents the parameter weight corresponding to the m-th single head in the fully connected layer of the first self-attention module, where m = 1, 2, 3,......, M; q A (m) and q B (m) are respectively the m-th single heads of the feature point query vectors q A and q B , x A (L, k) and x B (L, k) are respectively the k-th feature points on the L-th layer feature maps x AL and x BL ; is the attention score of q A (m) and x A (L, k), is the attention score of q B (m) and x B (L, k), where k = 1, 2, 3,......, K; is the mapping of ZW m on the L-th layer feature extraction module; and respectively represent and 's normalized values, and respectively represent mapping the normalized coordinates and to the L-th layer feature extraction module; is the coordinate offset of q A (m) and x A (L,k), is the coordinate offset of q B (m) and x B (L,k); represents on the L-th layer feature map x AL , at the position of

[0014] the feature point query vector, represents on the L-th layer feature map x BL , at the position of the feature point query vector;

[0015] Step 5, input the multi-head self-attention calculation results q′ A and q′ B into the first cross-attention module in sequence to obtain the corresponding cross-attention calculation results q″ A and q″ B :

[0016] q″ A =q A +Drop(q′ A +CrossAttn(q′ A , x B ))

[0017] q″ B =q B +Drop(q′ B +CrossAttn(q′ B , x A ))

[0018] where Drop is the random inactivation neuron function;

[0019]

[0020]

[0021] CrossAttn represents calculating cross-attention, JW mDenote the parameter weights corresponding to the m-th single head in the fully connected layer of the first cross-attention module, where m = 1, 2, 3,......, M; q′ A (m) and q′ B (m) are respectively the m-th single head of the multi-head self-attention calculation results q′ A and q′ B ; is the attention score of q′ A (m) and x B (L, k); is the attention score of q′ B (m) and x A (L, k); is the mapping of JW m in the feature extraction module of the L-th layer;

[0022] Step 6, transfer the low-resolution features on the feature maps x AL and x BL to the high-resolution feature map by upsampling:

[0023] x AL = x AL + Upsampling (4-L) (x″ A ), L = 1, 2, 3

[0024] x BL = x BL + Upsampling (4-L) (x″ B ), L = 1, 2, 3

[0025] where Upsampling (4-L) represents performing 4 - L times of upsampling; x″ A is the feature map formed according to the cross-attention calculation result q″ A ; x″ B is the feature map formed according to the cross-attention calculation result q″ B ;

[0026] Step 7, input the feature maps x AL and x BL into the candidate box generator to generate multiple candidate boxes, and obtain the corresponding content encoding and position encoding. For the feature maps x AL and x BL , select the N content encodings and position encodings with the largest similarity and input them into the candidate box sharing module:

[0027] f A = cat((c A , d A), dim = 1)

[0028] f Bs = cat((c B , d Bkxx ), dim = 1)

[0029] f' A = cat((f A , f Bs ), dim = 0)

[0030] Among them, f A and f Bs are intermediate variables. cat represents the concatenation function. dim = 1 means concatenation in the feature dimension, and dim = 0 means concatenation in the spatial dimension; c A and d A respectively represent the content encoding and position encoding obtained by inputting the feature map x AL into the candidate box generator; c B and d Bkxx respectively represent the content encoding and a set of learnable position encodings obtained by inputting the feature map x BL into the candidate box generator; f' A represents the output value corresponding to image A obtained by transmitting f A and f Bs to the second self-attention module;

[0031] f B = cat((c B , d B ), dim = 1)

[0032] f As = cat((c A , d Akxx ), dim = 1)

[0033] f' B = cat((f B , f As ), dim = 0)

[0034] f B and f As are intermediate variables. d B represents the position encoding obtained by inputting the feature map x BL into the candidate box generator; d Akxx represents a set of learnable position encodings obtained by inputting the feature map x AL into the candidate box generator; f' B represents the output value corresponding to image B obtained by transmitting f B and f As to the second self-attention module;

[0035] Step 8, f′ A After passing through the second cross-attention module, the predicted values of the classification detection results and the target bounding box detection results corresponding to image A are obtained, f′ B After passing through the second cross-attention module, the predicted values of the classification detection results and the target bounding box detection results corresponding to image B are obtained. The predicted values of the classification detection results and the target bounding box detection results are respectively subjected to Hungarian matching with the corresponding ground truths in the MDMT dataset to obtain the classification loss and the target bounding box regression loss. The classification loss and the target bounding box regression loss are summed to obtain the total loss. According to the total loss, the internal parameters of the entire dual-UAV collaborative target detection network are updated through the backpropagation algorithm until the total loss converges, and the training is ended and the parameters of the dual-UAV collaborative target detection network are saved;

[0036] Step 9, input the video or image set from the dual UAVs at the same moment into the trained dual-UAV collaborative target detection network to obtain the classification detection results and the target bounding box detection results corresponding to the video or image set.

[0037] Due to the adoption of the above technical solutions, the beneficial effects of the present invention compared with the prior art are as follows:

[0038] 1. The present invention constructs a feature interaction network based on the attention mechanism, conducts interaction and fusion between multi-perspective information from the high-dimensional feature space level, effectively improves the detection accuracy of small targets and occluded targets, and provides a reliable implementation method for realizing multi-UAV information interaction enhancement.

[0039] 2. The present invention designs a dual-UAV collaborative target detection model. Starting from the perspective of multi-perspective consistency, the Transformer attention mechanism is adopted to realize complementary feature sharing between multi-perspectives, and the representation ability of different perspective feature maps for small targets and occluded targets is enhanced.

[0040] 3. The present invention proposes a candidate box sharing module, which enhances the recall and accuracy of candidate box queries through the sharing and fusion between multi-perspectives, improves the initialization efficiency of the decoder query vector, and alleviates the feature sampling and target positioning burden of the decoder. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the overall network structure of a cross-perspective associated dual-UAV collaborative target detection method in an embodiment of the present invention.

[0042] Figure 2 It is a schematic diagram of the overall process of a cross-perspective associated dual-UAV collaborative target detection method in an embodiment of the present invention.

[0043] Figure 3 is Figure 1 a schematic diagram of the principle of the feature interaction network in

[0044] Figure 4 is Figure 1 a schematic diagram of the principle of the candidate box generator and the decoder in Specific implementation manners

[0045] The following further describes the content of the present invention in conjunction with the accompanying drawings and specific embodiments.

[0046] A cross-view correlation-based dual-UAV collaborative target detection method, as Figure 2 shown, includes the following steps:

[0047] Step 1, as Figure 1 shown, construct a dual-UAV collaborative target detection network, the dual-UAV collaborative target detection network includes a feature interaction network based on an attention mechanism, a candidate box generator, and a decoder; the feature interaction network includes two groups of four-layer feature extraction modules generated based on a residual depth network and a feature pyramid network, a first self-attention module, a first cross-attention module, and an upsampling module; the decoder includes a candidate box sharing module, a second self-attention module, and a second cross-attention module;

[0048] Step 2, select the MDMT dataset as the training set, and use the image A captured by UAV A and the image B captured by UAV B at the same time and from different perspectives in it as a set of training input values;

[0049] Specifically, in this embodiment, the Multi-Drone Multi-Target Tracking (MDMT) dataset includes 44 pairs of video sequences, which are captured by two drones working together. The flight altitude is between 50 and 80 meters, and there is a large overlapping area between the fields of view of the two drones. This dataset contains a total of 39,678 frames of images, and rectangular boxes and IDs are annotated for three types of targets: people, vehicles, and bicycles in each image. The training set contains 27,916 (27,912 during actual training) images, and the test set contains 11,762 images. The MDMT dataset has a total of 2,204,620 annotated boxes, and 543,444 of the targets in the annotated boxes are occluded, accounting for approximately 24.7% of the total annotated boxes. Among the 908,259 annotated boxes of UAV A, 198,913 of the targets in the boxes are occluded. Among the 1,296,361 annotated boxes of UAV B, 344,531 of the targets in the boxes are occluded. It can be seen that the proportion of occluded targets within the two drones is relatively large, posing a huge challenge to the target detection task. Moreover, the number of targets in the images collected by UAV B is larger and the occlusion situation is more serious, and the detection difficulty is higher than that of the images collected by UAV A. To better analyze the presence of small targets, in this embodiment, according to the definitions of small targets, medium targets, and large targets in the COCO metrics, the number of the three types of targets in the test set and their distribution in the two drones are statistically analyzed. Among them, small targets refer to targets with a pixel area less than 322, medium targets refer to targets with a pixel area greater than 322 but less than 962, and large targets refer to targets with a pixel area greater than 962. In the MDMT dataset, the areas of most annotated objects are less than 322, which poses higher requirements for the detection accuracy of the model for small targets.

[0050] Step 3: Input Image A and Image B into their respective four-layer feature extraction modules to obtain the feature maps \(x^L\) output by the four-layer feature extraction modules, where \(L = 1, 2, 3, 4\). \(L = 1\) corresponds to the uppermost feature extraction module, and \(L = 4\) corresponds to the lowermost feature extraction module. AL and \(x^L\) BL , where \(L = 1, 2, 3, 4\). \(L = 1\) corresponds to the uppermost feature extraction module, and \(L = 4\) corresponds to the lowermost feature extraction module;

[0051] Specifically, in this embodiment, xml annotation files in the COCO detection format are constructed for each perspective. The image numbers are arranged in time series and represented by Image ID. Therefore, multi-perspective images at the same moment have the same Image ID. During model training and inference, the Image ID generated by random sampling is used as an index to obtain the corresponding image and the annotation box information of the corresponding image in the annotation files under different perspectives.

[0052] In this embodiment, the resolution of the input image is 1920×1080, and the target detection categories are three types of objects: people, vehicles, and bicycles. The model uses a two-stage Deformable DETR detection framework, adds an iterative_bbox_refinement strategy, and uses ResNet-50

[80] as the backbone network and FPN feature pyramid network to extract image features. All experiments are trained based on the R50 deformabledetr pre-trained model. In this embodiment, a geometric adaptive Gaussian kernel method is used to generate the ground truth density map. It is trained for 50 Epochs. The optimizer selects the SGD algorithm, the initial learning rate is 2×10 -4 , and the weight decay rate is 10 -4 , the learning rate decay rate is 0.1, and the decay is performed every 40 Epochs. The Dropout rate of the attention network is set to 0.1, the decoder initializes the query vector to 300, and the query vector becomes 600 after being shared between two machines. The training is carried out on a single RTX3090, the batchsize is set to 2, and the normal training time for 50 Epochs is 100 hours.

[0053] Step 4, as shown in Figure 3 , set the number of single heads M and the number of sampling points K, and sequentially input the K feature points on the feature maps x AL and x BL into the first self-attention module to obtain the corresponding multi-head self-attention calculation results q′ A and q′ B :

[0054]

[0055] Among them, MSDeformAttn represents the calculation of multi-head self-attention, q A and q B respectively represent the query vectors corresponding to a feature point on the feature maps x AL and x BL , and respectively represent the coordinates of the feature point query vectors q A and q B ; and respectively represent the sequential calculations on the four-layer feature maps x AL and x BL ; ZW m represents the parameter weight corresponding to the m-th single head in the fully connected layer of the first self-attention module, m = 1, 2, 3,......, M; q A (m) and q B (m) are respectively the feature point query vectors qA with q B the m-th single head of, x A (L, k) and x B (L, k) are respectively the k-th feature point on the L-th layer feature map x AL and x BL : is q A (m) and x A (L, k)'s attention score is q B (m) and x B (L, k)'s attention score, k = 1, 2, 3,......, K; is ZW m 's mapping on the L-th layer feature extraction module; and respectively represent and 's normalization value, and respectively represent mapping the normalized coordinates and to the L-th layer feature extraction module; is q A (m) and x A (L, k)'s coordinate offset, is q B (m) and x B (L, k)'s coordinate offset; represents the feature point query vector at the position of AL on the L-th layer feature map x

[0056] : represents the feature point query vector at the position of BL on the L-th layer feature map x :

[0057] Step 5, as Figure 3 shown, input the multi-head self-attention calculation results q' A and q' B into the first cross-attention module in sequence to obtain the corresponding cross-attention calculation results q″ A and q″ B :

[0058] q″ A = q A + Drop(q' A + CrossAttn(q' A , x B ))

[0059] q″ B =q B +Drop(q' B +CrossAttn(q' B , x A ))

[0060] where Drop is the random inactivation neuron function;

[0061]

[0062] CrossAttn represents the calculation of cross-attention, JW m represents the parameter weight corresponding to the m-th single head in the fully connected layer of the first cross-attention module, m = 1, 2, 3,......, M; q′ A (m) and q′ B (m) are respectively the m-th single heads of the multi-head self-attention calculation results q′ A and q′ B ; is the attention score of q′ A (m) and x B (L, k); is the attention score of q′ B (m) and x A (L, k); is the mapping of JW m in the feature extraction module of the L-th layer;

[0063] Step 6, as Figure 3 shown, transfer the low-resolution features on the feature maps x AL and x BL to the high-resolution feature map by upsampling:

[0064] x AL =x AL +Upsampling (4-L) (x″ A ), L = 1, 2, 3

[0065] x BL =x BL +Upsampling (4-L) (x″ B ), L = 1, 2, 3

[0066] where Upsampling (4-L) represents performing 4 - L times of upsampling; x″ A is the feature map formed according to the cross-attention calculation result q″ A ; x″ BThe feature map formed according to the cross-attention calculation result q″ B ;

[0067] Step 7, as Figure 4 shown, input the feature map x AL and x BL into the candidate box generator to generate multiple candidate boxes, and obtain the corresponding content encoding and position encoding. For the feature map x AL and x BL , select the N content encodings and position encodings with the largest similarity, that is, Top-N, and input them into the candidate box sharing module:

[0068] f A = cat((c A , d A ), dim = 1)

[0069] f Bs = cat((c B , d Bkxx ), dim = 1)

[0070] f′ A = cat((f A , f Bs ), dim = 0)

[0071] where f A and f Bs are intermediate variables, cat represents the concatenation function, dim = 1 means concatenation in the feature dimension, and dim = 0 means concatenation in the spatial dimension; c A and d A respectively represent the content encoding and position encoding obtained by inputting the feature map x AL into the candidate box generator; c B and d Bkxx respectively represent the content encoding and a set of learnable position encodings obtained by inputting the feature map x BL into the candidate box generator; f′ A represents the output value corresponding to image A after transmitting f A and f Bs to the second self-attention module;

[0072] f B = cat((c B , d B ), dim = 1)

[0073] f As = cat((c A , d Akxx ), dim = 1)

[0074] f′B = cat((f B , f As ), dim = 0)

[0075] f B and f As are intermediate variables, d B represents the position encoding obtained by inputting the feature map x BL into the candidate box generator; d Akxx represents a set of learnable position encodings obtained by inputting the feature map x AL into the candidate box generator; f' B represents the output value corresponding to image B obtained by transmitting f B and f As to the second self-attention module;

[0076] Specifically, in this embodiment, the RPN method is used to generate candidate boxes.

[0077] Step 8, after f' A passes through the second cross-attention module, the predicted values of the classification detection results and the target bounding box detection results corresponding to image A are obtained. After f' B passes through the second cross-attention module, the predicted values of the classification detection results and the target bounding box detection results corresponding to image B are obtained. The predicted values of the classification detection results and the target bounding box detection results are respectively subjected to Hungarian matching with the corresponding ground truths in the MDMT dataset to obtain the classification loss and the target bounding box regression loss. The classification loss and the target bounding box regression loss are summed to obtain the total loss. According to the total loss, the internal parameters of the entire dual-UAV collaborative target detection network are updated through the backpropagation algorithm until the total loss converges, and the training is ended and the parameters of the dual-UAV collaborative target detection network are saved;

[0078] Step 9, input the video or image set from the dual UAVs at the same moment into the trained dual-UAV collaborative target detection network to obtain the classification detection results and the target bounding box detection results corresponding to the video or image set.

[0079] In this embodiment, the MS-COCO evaluation metric is used to evaluate the object detection results, which includes the average detection precision Average Precision0.5 (AP0.5), the average detection precision Average Precision0.5-0.95 (MAP-0.95), the average detection precision of small objects (area < 322) AP-0.95-Small (Small-AP), the average detection precision of medium objects (322 < area < 962) AP-0.95-Medium (MediumAP), and the average detection precision of large objects (area > 962) AP-0.95-Large (Large-AP), as well as the average recall rate of small objects (S-AR), the average recall rate of medium objects (M-AR), and the average recall rate of large objects (L-AR). Among them, both AP (Average Precision) and AR (Average Recall) represent the average results in three detection categories (person, vehicle, and bicycle). AP0.5 means that when the intersection over union ratio of the detection box and the ground truth box is greater than 0.5, it is a TP (True Positive), otherwise it belongs to FP (False Positive). The AP value is the AUC area (area under curve) with the recall rate from 0 to 1 as the horizontal axis and the value of TP / (TP + FP) as the vertical axis. AP-0.95 means that the AP is calculated once for every 0.05 increase in the threshold from 0.5 to 0.95, and the average result of ten APs is taken, denoted as AP-0.95. In contrast, AP0.5 has a looser intersection over union threshold, which is suitable for sparse and less overlapping target scenarios. AP-0.95 calculates and averages the AP for different intersection over union thresholds, paying more attention to the position accuracy of the detection box. The larger the value, the more accurate the position of the detection box.

[0080] This embodiment is compared with numerous single - machine object detection algorithms, including Faster - RCNN, Mask - RCN, Tood - FPN, AutoAssign, CascadeRPN, and the original two - stage Deformable - DETR. As shown in Table 1, Deformable - DETR is our baseline method. Co - DET (candidate box sharing) and Co - DET (candidate box fusion) represent dual - machine collaborative object detection methods, which have different candidate box interaction strategies. As the experimental results show, the dual - machine collaborative detection method can achieve the best in any metric. The Co - DET method using the shared candidate box strategy (share proposal) has a MAP0.5:0.95 of 20.0, which is 0.7% higher than the baseline method. The average precision for small object detection reaches 12.7, which is 0.9% higher than the baseline method. The recall rates for small and medium - sized objects reach 19.3 and 36.4 respectively, which are 0.9% and 1.8% higher than the baseline method respectively. The Co - DET method using the fused candidate box strategy (fuse proposal) has a MAP0.5 of 41.7, which is 0.8% higher than the baseline method. The accuracy and recall rates for small objects are 12.0 and 18.7 respectively, with increases of 0.2% and 0.3% respectively. The recall rate for large objects is 73.1, with an increase of 0.6%. The Co - DET method is superior to the baseline method in the detection accuracy and recall rate of small objects, and the multi - perspective collaborative method has shown obvious advantages in small object detection.

[0081] Co - DET (w / candidate box sharing) means only adding the candidate box sharing or fusion module and removing the feature map interaction network and the upsampling process; Co - DET (w / cross - attention) means removing the candidate box sharing module and the upsampling process and retaining the feature map interaction network; Co - DET (w / o upsampling) means retaining both the candidate box sharing module and the feature map interaction network but not adding the upsampling process; Co - DET (w / o candidate box sharing) means retaining the feature map interaction network and the upsampling process but not adding the candidate box sharing module; Co - DET (sharing) means the complete Co - DET network with a candidate box sharing module, a feature map interaction network, and an upsampling process.

[0082] As shown in Table 2, we conducted ablation experiments on the Co-DET (Candidate Box Sharing) model to study the impact of the candidate box sharing strategy, cross-attention feature interaction module, and upsampling strategy when they appear alone. The MAP0.5 of the model with only the candidate box sharing strategy is 41.4, and the MAP0.5-0.95 is 19.9, which are 0.5% and 0.6% higher than the baseline method respectively. The detection accuracy AP and recall rate AR for small targets both increased by 0.5%, indicating that the candidate box sharing strategy effectively improved the detection effect of small targets. As shown in the experimental results of the third row of Table 2, adding only the attention-based feature interaction network module achieved the optimal average detection accuracy of MAP0.5, which was 0.8% higher than the baseline method, and at the same time, MAP-0.95 also increased by 0.4%. In terms of the average detection accuracy and recall rate of large targets, this method achieved improvements of +1.4% and +2.3% respectively, indicating that the attention-based dual-machine feature interaction module can complement information between multiple perspectives, sample and complete the feature information not captured by a single machine from another perspective, enhance feature representation, and improve the detection effect.

[0083] The attention-based feature interaction network only operates between the top two layers of the feature pyramid. To transfer the dual-machine interaction information to the high-resolution feature map, the model performed an upsampling operation. As shown in the fourth row of Table 2, the upsampling process lost the detection accuracy and recall rate of large targets, but the detection accuracy and recall rate for small targets were significantly improved. Small-AP-0.95 and Small-AR-0.95 increased by 0.4% and 0.3% respectively, realizing the original design significance of the upsampling strategy. As shown in the experimental results of the fifth row of Table 2, when the candidate box sharing strategy and the feature interaction network participated together, MAP0.5 and MAP0.95 were 41.4 and 20.0 respectively, increasing by 0.5 and 0.7 percentage points respectively. The detection accuracy and recall rate of small targets increased by 0.6% and 0.9% respectively compared to the baseline method. After continuing to add the upsampling strategy, the small target detection accuracy Small-AP-0.95 reached 12.7, which was 0.3% higher than the method without upsampling and 0.9% higher than the baseline method, further verifying the significant effect of the upsampling method on improving the detection accuracy of small targets.

[0084] To further analyze the impacts of the above-mentioned modules, we list the results of the ablation experiments of the Co-DET (Candidate Box Sharing) model on a single machine. The specific data is shown in Table 3. First, from the perspective of the detection metrics MAP0.5 and MAP-0.95 between the two machines, the results of Machine A are significantly better than those of Machine B. The reason is that Machine A has more low-altitude data, fewer small targets in the field of view, and relatively easier detection; Machine B has more high-altitude data, more small targets and occluded targets in the field of view, and relatively more difficult detection. As shown in Table 3, the candidate box sharing strategy has achieved good improvements in both machines. In Machine A, MAP0.5, MAP0.5:0.95, and Small-AP-0.95 have increased by 0.8%, 1.0%, and 0.9% respectively, and in Machine B, they have increased by 0.2%, 0.2%, and 0.1% respectively. This is consistent with the results of the ablation experiments between the two machines, indicating that the candidate box sharing strategy effectively improves the detection effect of small targets.

[0085] The results of Co-DET (w / cross-attention) on a single machine also verify that the feature interaction network between the two machines can improve the object detection accuracy (mainly reflected in MAP0.5) by enhancing the feature representation of the feature map. In Machine A, MAP0.5 and MAP0.5:0.95 have increased by 0.8% and 0.4% respectively, and in Machine B, they have increased by 1.0% and 0.4% respectively. As shown in the second-to-last row of Table 3, the indicators of the Co-DET (w / o upsampling) method are all better than those of the baseline method. Compared with Co-DET (w / candidate box sharing), adding the feature interaction network between the two machines improves the overall detection accuracy in Machine B, and the three indicators have increased by 0.4%, 0.3%, and 0.8% respectively, verifying that the feature interaction network between the two machines can contribute the object features in Machine A (lower perspective) to the B perspective (higher perspective), enhance the feature representation of small targets in the B perspective, and enhance the detection effect. However, the targets in the B perspective are smaller, and the supplementary effect on the A perspective is weaker. The detection effect of the Co-DET (sharing) method in the B perspective fully verifies the advantages of the two-machine collaborative detection model. The three indicators of MAP0.5, MAP-0.95, and Small-AP-0.95 have increased by 1.3%, 1.5%, and 1.8% respectively, fully demonstrating that the two-machine collaborative detection algorithm can transfer the rich object feature information in the low-altitude perspective to the high-altitude perspective through feature sharing, candidate box sharing, and upsampling strategies between the two machines, significantly improving the detection accuracy (from 13.1 to 14.9) and recall rate (from 0.19 to 0.21) of small targets in the high-altitude perspective.

[0086] Table 1

[0087] Algorithm MAP0.5 MAP-0.95 Small-AP Medium-AP Large-AP S-AR M-AR L-AR Faster-RCNN-Caffe-FPN 26.8 15.3 6.6 22.6 64.5 8.2 28.6 69.5 Faster-RCNN-FPN-Carafe 26.6 16.5 6.8 22.4 65.3 8.3 28.0 70.3 Mask-RCNN 30.3 17.1 8.6 23.1 66.6 10.6 28.4 71.5 Tood-FPN 38.1 19.1 12.3 23.7 63.2 18.0 30.7 69.2 AutoAssign 34.3 16.9 9.5 21.6 52.0 15.5 27.7 58.5 Cascade-RPN 31.8 17.4 11.0 23.4 66.7 16.5 31.3 71.7 Deformable-DETR 40.9 19.3 11.8 24.9 67.6 18.4 34.6 72.5 Co-DET 40.9 20.0 12.7 24.9 66.3 19.3 36.4 71.9

[0088] Table 2

[0089]

[0090] Table 3

[0091]

[0092]

[0093] In summary, the present invention constructs a feature interaction network based on the attention mechanism to perform interaction and fusion among multi-perspective information from the high-dimensional feature space level, enhance the feature representation of occluded targets and small targets, and improve the target detection accuracy. In addition, a candidate box sharing module is designed, and different target detection tasks share the same set of candidate boxes, which can avoid generating candidate boxes separately for each task, thereby saving computing resources.

[0094] Those skilled in the art will realize that the described embodiments are for helping readers understand the principles of the present invention, and it should be understood that the protection scope of the present invention is not limited to the described embodiments. For those skilled in the art, various changes and modifications can be made to the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the scope of the claims of the present invention.

Claims

1. A cross-perspective correlation-based dual-UAV collaborative target detection method, characterized in that, Including the following steps: Step 1: Construct a dual-UAV collaborative target detection network, where the dual-UAV collaborative target detection network includes a feature interaction network based on the attention mechanism, a candidate box generator, and a decoder; the feature interaction network includes two sets of four-layer feature extraction modules generated based on the residual depth network and the feature pyramid network, a first self-attention module, a first cross-attention module, and an upsampling module; the decoder includes a candidate box sharing module, a second self-attention module, and a second cross-attention module; Step 2: Select the MDMT dataset as the training set, and use the image A captured by UAV A and the image B captured by UAV B at the same moment from different perspectives in the MDMT dataset as a set of training input values; Step 3: Input image A and image B into their respective four-layer feature extraction modules to obtain the feature map x output by the four-layer feature extraction module. AL With x BL , L = 1, 2, 3, 4, L = 1 corresponds to the feature extraction module at the top layer, and L = 4 corresponds to the feature extraction module at the bottom layer; Step 4, set the number of single heads M and the number of sampling points K, and sequentially input the K feature points on the feature map x AL and x BL into the first self-attention module to obtain the corresponding multi-head self-attention calculation results q′ A and q′ B : Among them, MSDeformAttn represents the calculation of multi-head self-attention, q A and q B respectively represent the query vectors corresponding to a feature point on the feature maps x AL and x BL ; and respectively represent the coordinates of the feature point query vectors q A and q B ; and respectively represent the sequential calculations on the four-layer feature maps x AL and x BL ; ZW m represents the parameter weight corresponding to the m-th single head in the fully connected layer of the first self-attention module, m = 1, 2, 3,......, M; q A (m) and q B (m) are respectively the m-th single heads of the feature point query vectors q A and q B , x A (L, k) and x B (L, k) are respectively the k-th feature points on the L-th layer feature maps x AL and x BL ; is the attention score of q A (m) and x A (L, k); is the attention score of q B (m) and x B (L, k), k = 1, 2, 3,......, K; is the mapping of ZW m on the L-th layer feature extraction module; and respectively represent the normalization values of p qA and ; and respectively represent the mapping of the normalized coordinates and onto the L-th layer feature extraction module; is the coordinate offset of q A (m) and x A (L, k); is the coordinate offset of q B (m) and x B (L, k); represents the feature point query vector at the position of AL on the L-th layer feature map x ; Denote the feature point query vector at the position on the feature map x of the L-th layer BL above at the position Step 5, input the multi-head self-attention calculation result q′ A and q′ B into the first cross-attention module in sequence to obtain the corresponding cross-attention calculation results q′ A ′ and q′ B ′: q″ A = q A + Drop(q' A + CrossAttn(q' A , x B )) q″ B = q B + Drop(q' B + CrossAttn(q' B , x A )) where Drop is a random inactivation neuron function; CrossAttn represents the calculation of cross-attention, JW m represents the parameter weight corresponding to the m-th single head in the fully connected layer of the first cross-attention module, where m = 1, 2, 3,......, M; q′ A (m) and q′ B (m) are respectively the m-th single heads of the multi-head self-attention calculation results q′ A and q′ B ; is the attention score of q′ A (m) and x B (L,k); is the attention score of q′ B (m) and x A (L,k); is the mapping of JW m in the feature extraction module of the L-th layer; Step 6, transfer the feature map x AL and the low-resolution features on x BL to the high-resolution feature map in an upsampling manner: x AL = x AL + Upsampling (4-L) (x″ A ), L = 1, 2, 3 x BL = x BL + Upsampling (4-L) (x″ B ), L = 1, 2, 3 Among them, Upsampling (4-L) represents performing 4-L times of upsampling; x″ A is the feature map formed according to the cross-attention calculation result q″ A ; x″ B is the feature map formed according to the cross-attention calculation result q″ B ; Step 7, input the feature map x AL and x BL into the candidate box generator to generate multiple candidate boxes, and obtain their corresponding content encodings and position encodings. For the feature map x AL and x BL , select the N content encodings and position encodings with the largest similarity and input them into the candidate box sharing module: f A = cat((c A , d A ), dim = 1) f Bs = cat((c B , d Bkxx ), dim = 1) f′ A = cat((f A , f Bs ), dim = 0) Among them, f A and f Bs are intermediate variables, cat represents the concatenation function, dim = 1 means concatenation in the feature dimension, and dim = 0 means concatenation in the spatial dimension; c A and d A respectively represent the content encoding and position encoding obtained by inputting the feature map x AL into the candidate box generator; c B and d Bkxx respectively represent the content encoding and a set of learnable position encodings obtained by inputting the feature map x BL into the candidate box generator; f' A represents the output value corresponding to image A after transmitting f A and f Bs to the second self-attention module; f B = cat((c B , d B ), dim = 1) f As = cat((c A , d Akxx ), dim = 1) f′ B = cat((f B , f As ), dim = 0) f B and f As is an intermediate variable, d B represents the position encoding obtained by inputting the feature map x BL into the candidate box generator; d Akxx represents the position encoding obtained by inputting the feature map x AL into the candidate box generator; a set of learnable position encodings; f' B represents transmitting f B and f As to the second self-attention module, corresponding to the output value of image B; Step 8, f A ′ After passing through the second cross-attention module, the predicted values of the classification detection results and the predicted values of the target bounding box detection results corresponding to image A are obtained, f′ B After passing through the second cross-attention module, the predicted values of the classification detection results and the predicted values of the target bounding box detection results corresponding to image B are obtained. The predicted values of the classification detection results and the predicted values of the target bounding box detection results are respectively subjected to Hungarian matching with the corresponding ground truths in the MDMT dataset to obtain the classification loss and the target bounding box regression loss. The classification loss and the target bounding box regression loss are summed to obtain the total loss. According to the total loss, the internal parameters of the entire dual-UAV collaborative target detection network are updated through the backpropagation algorithm until the total loss converges, and the training is terminated and the parameters of the dual-UAV collaborative target detection network are saved; Step 9: Input the video or image set from the dual UAVs at the same moment into the trained dual-UAV collaborative target detection network to obtain the classification detection result and the target bounding box detection result corresponding to the video or image set.

Images