A multi-scale target detection method for UAV aerial images based on coordinate and global information aggregation

Abstract

The present invention discloses a multi-scale target detection method for drone aerial images based on coordinate and global information aggregation, comprising: constructing a target detection network; the target detection network is composed of three parts: a backbone network, a multiple feature fusion network, and a prediction head; the backbone network is composed of a coordinate and global information aggregation module; the coordinate and global information aggregation module includes a coordinate information extraction branch and a global information extraction branch; the constructed target detection network is trained to obtain a target detection model, and Focal‑EIOU loss is used as the bounding box loss of the model; and multi-scale target detection is performed on drone aerial images based on the target detection model. The present invention alleviates the interference of background factors in the feature extraction process by combining coordinate information and global information, thereby enhancing the network's ability to extract robust features; the multiple feature fusion network designed by the present invention can better fuse multi-scale features and improve the detection performance of multi-scale targets.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection in drone images, and in particular to a multi-scale target detection method for drone aerial images based on coordinate and global information aggregation. Background Art

[0002] Drone (UAV) technology has made significant progress in recent years. Due to their low cost, maneuverability, and ease of use, UAVs hold great potential for both military and civilian applications. Consequently, object detection in UAV imagery has become a research hotspot in recent years. However, most object detection methods have limited feature extraction capabilities in UAV imagery, hindering detection performance and failing to achieve high detection accuracy. Therefore, a multi-scale object detection method in UAV aerial imagery based on coordinate and global information aggregation is a promising topic.

[0003] Many detection models based on convolutional neural networks have achieved good results. However, due to the special characteristics of drone images, common object detection frameworks are not well suited for drone images. Compared with object detection on natural images, object detection on drone images generally faces the following challenges:

[0004] (1) In drone images, many targets are usually obscured. At the same time, due to the lighting, the boundaries and features of the targets become unclear, making it difficult for the model to extract discriminative features.

[0005] (2) Due to the problem of the drone’s shooting angle, there is a big difference in the scale of targets of the same category or different categories, and most of them are small targets, which puts a great test on the model’s ability to extract features of multi-scale targets and small targets.

[0006] (3) The bounding box regression loss function of the existing model has defects and cannot regress the bounding box well, thereby reducing the performance of the detector. Summary of the Invention

[0007] In response to the above problems, the present invention proposes a multi-scale target detection method for drone aerial images based on coordinate and global information aggregation, which can accurately detect drone images with complex and changeable shooting angles. The present invention alleviates the interference of background factors in the feature extraction process by combining coordinate information and global information, and enhances the network's ability to extract robust features. In order to improve the model's ability to detect multi-scale targets, the present invention improves the feature fusion network and designs a structure called Multi-Feature Fusion Pyramid Network (MF-FPN), which can better fuse multi-scale features and improve the detection performance of multi-scale targets. In order to improve the model's bounding box regression capability, the present invention uses the Focal-EIOU loss function to replace the model's original bounding box regression loss to improve the model's detection performance. The present invention uses the challenging VisDrone dataset for experiments to verify the effectiveness of the proposed model.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A multi-scale target detection method for UAV aerial images based on coordinate and global information aggregation, including:

[0010] Step 1: Construct a target detection network; the target detection network consists of three parts: a backbone network, a multiple feature fusion network, and a prediction head; the backbone network is used to extract feature information of the input drone image and output feature maps of different sizes; the multiple feature fusion network is used to fuse feature maps of different sizes; the prediction head is used to perform detection based on the fused features to obtain the final detection result; the backbone network is composed of a coordinate and global information aggregation module; the coordinate and global information aggregation module includes a coordinate information extraction branch and a global information extraction branch. The coordinate information extraction branch introduces an attention mechanism to allow it to always pay attention to coordinate information while using convolution for feature extraction. The global information extraction branch obtains global information of the feature map through pooling operations. Finally, features extracted by the two branches are fused to obtain features with richer contextual information.

[0011] Step 2: Train the constructed target detection network to obtain the target detection model, and use Focal-EIOUloss as the bounding box loss of the model;

[0012] Step 3: Perform multi-scale target detection on UAV aerial images based on the target detection model.

[0013] Furthermore, the processing flow of the coordinate information extraction branch includes:

[0014] First, use 1×1 convolution to halve the number of channels of the input feature map to obtain the first intermediate feature map M1;

[0015] Then use 3×3 convolution and attention mechanism to extract target features and obtain the second and third intermediate output feature maps M2 and M3:

[0016] M2=CA(Conv 3×3 (CA(Conv 3×3 (M1))))

[0017] M3=CA(Conv 3×3 (CA(Conv 3×3 (M2))))

[0018] Where CA represents the attention mechanism.

[0019] Furthermore, the processing flow of the attention mechanism includes:

[0020] First, input feature map F∈R C×H×W Perform pooling operations in the horizontal and vertical directions to obtain two feature maps f with direction awareness h ∈R C×H×1 and f w ∈R C×1×W :

[0021]

[0022]

[0023] where f c and F c Denotes the c-th channel of the output and input feature maps, respectively. H and W denote the height and width of the input feature map, respectively.

[0024] Then, f h and f w Concat operation is performed on the spatial dimension, and then 1×1 convolution is used to reduce the number of channels, and then features are obtained through batch normalization and activation function. Where r is the scaling factor used to reduce the number of channels;

[0025] M=δ(BN(Conv 1×1 ([f h , f w ])))

[0026] Where [·] represents the channel concat operation, BN represents batch normalization, and δ represents the nonlinear activation function; then, M is split from the spatial dimension to obtain two feature tensors and Perform 1×1 convolution operations on these two feature tensors to increase the number of channels to the same as the number of channels of the input feature map, and then use the Sigmoid function to obtain the attention weight g h and g w :

[0027] g h =σ(Conv 1×1 (y h ))

[0028] g w =σ(Conv 1×1 (y w ))

[0029] Where σ represents the sigmoid function;

[0030] Finally, g h and g w Multiply it with the input feature map F to get the final output feature map Y∈R C×H×W :

[0031]

[0032] Furthermore, the processing flow of the global information extraction branch includes:

[0033] First, use the global pooling operation to increase the global context information of the backbone network. For the input feature X∈R C×H×W First, perform global average pooling and global maximum pooling operations, then add the two results, and then use a fully connected layer and Sigmoid function to assign weights to each channel, so that the model focuses on high-response channel information:

[0034]

[0035] Where GAP and GMP represent global average pooling and global maximum pooling respectively. FC represents the fully connected layer.

[0036] σ represents the sigmoid function;

[0037] Then, the result is multiplied by the input feature X, and the number of channels is halved through a 1×1 convolution to obtain the output feature of the global information extraction branch

[0038]

[0039] Furthermore, the features extracted by the two branches are fused as follows:

[0040] All intermediate output features M1, M2, M3, and M4 of the two branches are first subjected to channel concat operation, and then the channel information is sorted using 1×1 convolution to obtain the final output feature map Z∈R 2C×H×W .

[0041] Furthermore, the feature fusion method in the multi-feature fusion network includes:

[0042] For the case where only two feature maps need to be fused, two learnable weights are assigned to the two feature maps to be fused to determine the importance of each feature map:

[0043]

[0044] Where P is the feature after fusion of F1 and F2, F1 and F2 are the two feature maps to be fused, w1 and w2 are learnable weights, and Δ is a very small number to avoid numerical instability;

[0045] In the case of fusing three feature maps, the three feature maps to be fused are first fused in pairs, where each feature will participate in the fusion twice to achieve the effect of reuse. Then three different intermediate feature maps are obtained. Finally, three learnable weights are assigned to the three intermediate feature maps for fusion to obtain output features with rich contextual information for the final prediction.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1) A coordinate and global information aggregation module is designed to obtain more robust features by fusing local information, coordinate information and global information, thereby improving the feature extraction ability of the model.

[0048] 2) Design a multi-feature fusion pyramid network to better solve the problem of large scale variations of objects in drone aerial images.

[0049] 3) Use Focal-EIOU loss as the bounding box regression loss of the model to improve the quality of the detection box and increase the convergence speed of the model.

[0050] 4) Experiments on the VisDrone dataset demonstrate that our method performs better than other algorithms for object detection in drone imagery. Ablation experiments demonstrate that each of the proposed improved modules effectively improves detection accuracy. Therefore, our method is more suitable for object detection in drone imagery than existing object detection algorithms, and has important practical implications for research on object detection in drone imagery. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a schematic diagram of the overall network structure of a multi-scale target detection method for UAV aerial images based on coordinate and global information aggregation according to an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of the structure of the coordinate and global information aggregation module according to an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of the attention mechanism structure according to an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the feature fusion module structure of MF-FPN according to an embodiment of the present invention;

[0055] Figure 5 These are the detection results of the VisDrone dataset in different scenarios according to an embodiment of the present invention; (a1-a4) are the detection results of YOLOv7; and (b1-b4) are the detection results of the method proposed in the present invention. DETAILED DESCRIPTION

[0056] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0057] A multi-scale target detection method for UAV aerial images based on coordinate and global information aggregation, including:

[0058] Step 1: Construct a target detection network; the target detection network consists of three parts: a backbone network, a multiple feature fusion network, and a prediction head; the backbone network is used to extract feature information of the input drone image and output feature maps of different sizes; the multiple feature fusion network is used to fuse feature maps of different sizes; the prediction head is used to perform detection based on the fused features to obtain the final detection result; the backbone network is composed of a coordinate and global information aggregation module; the coordinate and global information aggregation module includes a coordinate information extraction branch and a global information extraction branch. The coordinate information extraction branch introduces an attention mechanism to allow it to always pay attention to coordinate information while using convolution for feature extraction. The global information extraction branch obtains global information of the feature map through pooling operations. Finally, features extracted by the two branches are fused to obtain features with richer contextual information.

[0059] Step 2: Train the constructed target detection network to obtain the target detection model, and use Focal-EIOUloss as the bounding box loss of the model;

[0060] Step 3: Perform multi-scale target detection on UAV aerial images based on the target detection model.

[0061] Furthermore, the present invention is described as follows:

[0062] (1) Overall network structure

[0063] This paper proposes a multi-scale target detection method for UAV aerial images based on coordinate and global information aggregation. The overall network structure is as follows: Figure 1 As shown in the figure, the network consists of three parts: the backbone network (Backbone), the Multi-Feature Fusion Pyramid Network (MF-FPN), and the prediction head (Head). First, the backbone network is used to extract the feature information of the input drone image. In order to detect targets of different sizes, the backbone network outputs several feature maps of different sizes (160×160, 80×80, 40×40, and 20×20). Among them, we use the four output feature maps of the backbone network as the input of the feature fusion network. Then, the extracted feature information of different sizes is sent to the multi-feature fusion network for feature fusion to obtain three feature maps of different scales with rich contextual information. Finally, these feature maps will be sent to the prediction head for detection to obtain the final detection results.

[0064] (2) Backbone Network

[0065] The backbone network is mainly composed of the Coordinate and Global Information Aggregation Module (CGAM) designed by us. This module can capture local information, coordinate information and global information, and fuse them to alleviate the interference of background factors and obtain more robust features. The CGAM module mainly consists of two branches. The first branch introduces the Coordinate Attention mechanism, which allows it to pay attention to coordinate information while using convolution for feature extraction. The second branch obtains the global information of the feature map through pooling operations. By fusing the features extracted by the two branches, features with richer contextual information are obtained. The Coordinate and Global Information Aggregation Module is as follows: Figure 2 shown.

[0066] 1) Coordinate information extraction branch: The first branch of CGAM first uses 1×1 convolution to transform the input feature map X∈R C ×H×W The number of channels is halved to obtain the first intermediate feature map As shown in formula (1):

[0067] M1=Conv 1×1 (X)#(1)

[0068] Then, 3×3 convolution and Coordinate Attention mechanism are used to extract target features and obtain the second and third intermediate output feature maps M2. As shown in formula (2) and (3):

[0069] M2=CA(Conv 3×3 (CA(Conv 3×3 (M1))))#(2)

[0070] M3=CA(Conv 3×3 (CA(Conv 3×3 (M2))))#(3)

[0071] Among them, CA represents the Coordinate Attention mechanism. The structure of the Coordinate Attention mechanism is as follows Figure 3 shown.

[0072] The Coordinate Attention mechanism first inputs the feature map F∈R C×H×W Perform pooling operations in the horizontal and vertical directions to obtain two feature maps f with direction awareness h ∈R C×H×1 and f w ∈R C×1×W As shown in formula (4) and (5):

[0073]

[0074]

[0075] where f c and F c Denote the cth channel of the output and input feature maps respectively. H and W denote the height and width of the input feature map respectively. Then, f h and f w Concat is performed on the spatial dimension, and then 1×1 convolution is used to reduce the number of channels. Batch Normalization and activation function are then used to obtain features. Where r is a scaling factor used to reduce the number of channels; Batch Normalization is used to avoid gradient explosion or gradient vanishing, making model training more stable; and the activation function is used to introduce nonlinear factors to improve the expressive power of the model. As shown in formula (6):

[0076] M=δ(BN(Conv 1×1 ([f h , fw ])))#(6)

[0077] Where [·] represents the channel concat operation, BN represents Batch Normalization, and δ represents the nonlinear activation function. Then, M is split from the spatial dimension to obtain two feature tensors and Perform 1×1 convolution operations on these two feature tensors to increase the number of channels to the same as the number of channels of the input feature map, and then use the Sigmoid function to obtain the attention weight g h and g w As shown in formula (7) and (8):

[0078] g h =σ(Conv 1×1 (y h ))#(7)

[0079] g w =σ(Conv 1×1 (y w ))#(8)

[0080] Where σ represents the sigmoid function. Finally, g h and g w Multiply it with the input feature map F to get the final output feature map Y∈R c×H×W As shown in formula (9):

[0081]

[0082] 2) Global information extraction branch: The second branch of the CGAM module first uses a global pooling operation to increase the global context information of the backbone network. C×H×W , first perform global average pooling and global maximum pooling, then add the two results, and then assign weights to each channel through a fully connected layer and Sigmoid function, so that the model focuses on high-response channel information. Specifically, as shown in formula (10):

[0083]

[0084] Where GAP and GMP represent global average pooling and global maximum pooling respectively. FC represents the fully connected layer. σ represents the sigmoid function. Then, the result is multiplied by the input feature X, and then a 1×1 convolution is performed to halve the number of channels to obtain the output feature of the second branch. As shown in formula (11):

[0085]

[0086] 3) Feature fusion: The CGAM module first performs channel concat operation on all intermediate output features M1, M2, M3, and M4 of the two branches, and then uses 1×1 convolution to organize the channel information to obtain the final output feature map Z∈R 2C×H×W As shown in formula (12):

[0087] Z=Conv 1×1 ([M1, M2, M3, M4])#(12)

[0088] (3) Multi-Feature Fusion Pyramid Network (MF-FPN)

[0089] The scale of objects in drone images varies greatly, and most of them are small. In order to improve the detection performance of multi-scale objects, the MF-FPN feature fusion structure is proposed. This structure obtains more context information by repeatedly using feature maps for fusion. In order to alleviate the problem of small target detection difficulties, a feature map of size 160×160 is added to the MF-FPN structure for fusion. Figure 4 As shown in the figure, the present invention adds a skip connection fusion path in the Neck module. The feature fusion method of MF-FPN is as follows: Figure 4 shown.

[0090] There are two main ways to fuse features. If you only need to fuse two feature maps, for example, the top-down path in Neck, use Figure 4 The method shown in (a) assigns two learnable weights to the two feature maps to be fused to determine the importance of each feature map. As shown in formula (13):

[0091]

[0092] Where P(F1, F2) is the fused feature of F1 and F2, F1 and F2 are the two feature maps to be fused, w1 and w2 are the learnable weights, and Δ is a small number used to avoid numerical instability.

[0093] For the case of fusing three feature maps, the present invention adopts Figure 4 (b) The three features are fused, as shown in formula (14). The present invention first uses formula (13) to fuse the three feature maps in pairs. Each feature will participate in the fusion twice, achieving the effect of reuse. Then, three different intermediate feature maps are obtained. Finally, three learnable weights are assigned to the three feature maps for fusion, and output features with rich contextual information are obtained for the final prediction.

[0094]

[0095] Among them, N(F1, F2, F3) is the feature after the fusion of F1, F2, and F3.

[0096] Since the fused features contain feature maps of different scales and different numbers of channels, the size and number of channels of the feature maps need to be adjusted to be consistent before fusion.

[0097] In order to retain more feature information, the number of channels in the convolution model is usually large, and a larger number of channels will bring more parameters to the model. The calculation formula for the number of parameters required for convolution is as follows:

[0098] Params=K h ×K w ×C in ×C out #(15)

[0099] Among them, K h and K w are the height and width of the convolution kernel respectively. in and C out Represents the number of input and output channels of the convolution. Therefore, to reduce the number of parameters required by the model, the present invention modifies the number of convolution channels in the model's Neck structure. First, the number of input and output channels of the 3×3 convolutions in the three ELAN-H modules in the top-down path are adjusted to 32. Then, the number of output channels of the first two 1×1 convolutions in the ELAN-H module in the bottom-up path is adjusted to 1 / 4 of the number of input channels.

[0100] (4) Loss Function

[0101] In order to help the model regress the bounding box more accurately and improve the convergence speed of the model, this paper uses Focal-EIOU loss as the bounding box loss of the model. It is defined as follows:

[0102] L Focal-EIOU =IOU γ L EIOU #(16)

[0103] Where IOU is the ratio of the intersection and union between the predicted box and the true box. γ is an adjustable parameter set to 0.5. EIOU represents EIOU loss. IOU and L EIOU The definition of is shown in formulas (17) and (18):

[0104]

[0105]

[0106] Where ρ represents the Euclidean formula. b and b gt Represent the center points of the predicted box and the true value box respectively. gt and h gt are the width and height of the true value box, w and h are the width and height of the predicted box. c represents the diagonal length of the minimum enclosing rectangle of the predicted box and the true value box. h c , w c is the height and width of the minimum bounding rectangle between the prediction box and the true value box. B and B gt Represent the predicted box and the true value box respectively.

[0107] Focal-EIOU loss can obtain more accurate detection results by regressing the center point, width, and height of the predicted box. At the same time, Focal-EIOU loss allows high-quality anchor boxes to contribute more gradients to the model, thereby improving the model's convergence speed.

[0108] In order to verify the effect of the present invention, the following experiments were performed:

[0109] The target detection method proposed in the present invention was verified on the Ubuntu 18.04.6LTS system, and was trained and tested with NVIDIA GeForce RTX 3090 (24G) as the graphics processing unit, the CPU was Intel(R) Xeon(R) Silver 4114CPU@2.20GHz, the Python version was 3.6, the CUDA version was 11.7, and the Pytorch version was 1.10.2.

[0110] During model training, the input image size is 640×640, and the Stochastic Gradient Descent (SGD) optimizer with momentum is used for training. The momentum parameter is set to 0.937, the initial learning rate is set to 0.01, the weight decay coefficient is set to 0.0005, the batch size is set to 8, and the total number of training iterations is 300.

[0111] We conducted experiments on the VisDrone dataset and compared it with other detection methods. As shown in Table 1, the detection method of the present invention outperforms other methods and achieves the best results. Compared with YOLOv7, the proposed model improves mAP0.5 by 1.9%, mAP0.75 by 1.6% and mAP by 1.2%, respectively. The model of the present invention has a significant improvement in the detection performance of small and medium-sized targets. The AP used to evaluate the detection performance of small targets is 0.0447, which is 0.033, and the mAP is 0.036, which is 0.063, which is 0.067 ... S Compared with YOLOv7, it has improved by 1.3% and evaluates the AP of medium target detection performance. M Improved by 1.2% for evaluating the AP of large object detection performance LAlthough it has been reduced, it is only reduced by 0.4%. We believe that the proposed model has more advantages than disadvantages for multi-scale object detection. Compared with QueryDet, although the model of the present invention only improves mAP0.75 by 0.6%, mAP0.5 and mAP are improved by 2.8% and 1% respectively. Compared with RetinaNet, Cascade-RCNN, Faster-RCNN, YOLOv3, YOLOX, YOLOv51 and HawkNet, whether it is mAP0.5, mAP0.75 and mAP or the evaluation index AP of multi-scale object detection performance S 、AP M 、AP L , all the proposed methods achieved the best results.

[0112] Table 1 Comparison of experimental results of the method of the present invention and other methods on the VisDrone dataset.

[0113] Method mAP0.5 mAP0.75 mAP <![CDATA[AP s ]]> <![CDATA[AP M ]]> <![CDATA[AP L ]]> RetinaNet 35.9 18.5 19.4 14.1 29.5 33.7 Cascade R-CNN 39.9 23.4 23.2 16.5 36.8 39.4 Faster R-CNN 40.0 20.6 21.5 15.4 34.6 37.1 YOLOv3 31.4 15.3 16.4 8.3 26.7 36.9 YOLOX 45.0 26.6 26.7 17.4 37.9 45.3 YOLOv51 36.2 20.1 20.5 12.4 29.9 36.4 HawkNet 44.3 25.8 25.6 19.9 36.0 39.1 QueryDet 48.1 28.8 28.3 \ \ \ YOLOv7 49.0 27.8 28.1 18.9 39.4 47.8 Ours 50.9 29.4 29.3 20.2 40.6 47.4

[0114] In order to more intuitively demonstrate the advantages of the detection method proposed in this invention, we provide the detection results of several different scene images. Figure 5 As shown, (a1)-(a4) are the detection results of YOLOv7, and (b1)-(b4) are the detection results of the proposed method. Figure 5 As can be seen from the red dotted boxes in (a1) and (b1), YOLOv7 recognizes the word on the ground as "car", while the method of the present invention can well recognize it as background. Figure 5 As can be seen from (a2) and (b2), the method of the present invention can distinguish two closely adjacent targets with very similar features. Since the features of small targets are not obvious, network learning is more difficult, and it is easy to identify the surrounding similar background as the target. The improved detection method of the present invention improves the accuracy of detecting small targets and can well distinguish the background. Figure 5 (a3) and (b3). In addition, we also tested the detection effect in night scenes, as shown in Figure 5 As shown in the red dotted boxes in (a4) and (b4), YOLOv7 did not detect it, but the method of the present invention accurately marked it.

[0115] We also list the mAP0.5 for each category to further illustrate which categories our method achieves significant improvements for. As shown in Table 2, our method achieves higher mAP0.5 than other methods in every category. Furthermore, except for the tricycle category, where the result is identical to YOLOv7, all other categories show significant improvements, particularly for bicycle and bus, where the improvements are 3.8% and 3.3%, respectively. This demonstrates the effectiveness of our method.

[0116] Table 2 Comparison of results for each category on the VisDrone dataset.

[0117] Method pedestrian people bicycle car van truck tricycle awing-tricycle bus motor mAP0.5 YOLOv3 12.8 7.8 4.0 43.0 23.5 16.5 9.5 5.1 29.0 12.5 31.4 YOLOv5l 44.4 36.8 15.6 73.9 39.2 36.2 22.6 11.9 50.5 42.8 37.4 YOLOv7 57.6 48.7 21.6 85.4 51.9 45.8 37.9 18.3 63.0 60.0 49.0 Ours 59.7 50.7 25.4 86.2 53.4 47.4 37.9 20.2 66.3 61.6 50.9

[0118] To verify the effectiveness of the three improved methods, we conducted ablation experiments on the VisDrone dataset. As shown in Table 3, we used YOLOv7 as the baseline and achieved a mAP of 49% on the VisDrone dataset. Each of the improved methods proposed in this paper can improve detection performance to a certain extent. Moreover, the proposed complete detection method, which integrates the three improvements, improves Precision by 0.5%, Recall by 2.1%, and mAP0.5 by 1.9% compared to YOLOv7, while also reducing the number of parameters by 0.7M.

[0119] Table 3 Ablation experiments on the VisDrone dataset.

[0120]

[0121] In summary, the key improvements of the present invention are:

[0122] (1) The present invention proposes a multi-scale target detection method for UAV aerial images based on coordinate and global information aggregation, which can perform more accurate detection on images taken by UAVs.

[0123] (2) In order to reduce the interference of background factors on the network feature extraction process, a coordinate and global information aggregation module is designed, which can simultaneously extract coordinate information, global information and local information, and fuse them to obtain more robust features, thereby more accurately locating the target.

[0124] (3) In order to better detect multi-scale targets, a multiple feature fusion network is designed. By reusing feature maps and using learnable weights to better fuse features of different scales, it can better solve the problem of large scale variations of objects in drone aerial images.

[0125] (4) In order to obtain more accurate detection results and improve the convergence speed of the model, Focal-EIOU loss is used to replace the original bounding box regression loss of the model.

[0126] (5) We conduct experiments on the challenging VisDrone dataset to verify our improved network structure. We also conduct a series of comparative experiments and ablation experiments to evaluate and analyze the effects of each improvement we propose.

[0127] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-scale target detection method for UAV aerial images based on coordinate and global information aggregation, characterized in that: include: Step 1: Construct a target detection network; the target detection network consists of three parts: a backbone network, a multiple feature fusion network, and a prediction head; the backbone network is used to extract feature information of the input drone image and output feature maps of different sizes; the multiple feature fusion network is used to fuse feature maps of different sizes; the prediction head is used to perform detection based on the fused features to obtain the final detection result; the backbone network is composed of a coordinate and global information aggregation module; the coordinate and global information aggregation module includes a coordinate information extraction branch and a global information extraction branch. The coordinate information extraction branch introduces an attention mechanism to allow it to always pay attention to coordinate information while using convolution for feature extraction. The global information extraction branch obtains global information of the feature map through pooling operations. Finally, features extracted by the two branches are fused to obtain features with richer contextual information. The processing flow of the coordinate information extraction branch includes: First, use 1×1 convolution to halve the number of channels of the input feature map to obtain the first intermediate feature map M1; Then use 3×3 convolution and attention mechanism to extract target features and obtain the second and third intermediate output feature maps M2 and M3: Where CA represents the attention mechanism; The processing flow of the global information extraction branch includes: First, use the global pooling operation to increase the global context information of the backbone network. For the input feature X∈R C×H×W First, perform global average pooling and global maximum pooling operations, then add the two results, and then use a fully connected layer and Sigmoid function to assign weights to each channel, so that the model focuses on high-response channel information: Where GAP and GMP represent global average pooling and global maximum pooling respectively, FC represents the fully connected layer, and σ represents the sigmoid function; Then, the result is multiplied by the input feature X, and the number of channels is halved through a 1×1 convolution to obtain the output feature of the global information extraction branch Step 2: Train the constructed target detection network to obtain the target detection model, and use Focal-EIOU loss as the bounding box loss of the model; Step 3: Perform multi-scale target detection on UAV aerial images based on the target detection model.

2. The multi-scale target detection method for UAV aerial images based on coordinate and global information aggregation according to claim 1 is characterized in that: The processing flow of the attention mechanism includes: First, input feature map F∈R C×H×W Perform pooling operations in the horizontal and vertical directions to obtain two feature maps f with direction awareness h ∈R C×H×1 and f w ∈R C×1×W : where f c and F c Denotes the c-th channel of the output and input feature maps, respectively. H and W denote the height and width of the input feature map, respectively. Then, f h and f w Concat operation is performed on the spatial dimension, and then 1×1 convolution is used to reduce the number of channels, and then features are obtained through batch normalization and activation function. Where r is the scaling factor used to reduce the number of channels; M=δ(BN(Conv 1×1 ([f h ,f w ]))) Where [·] represents the channel concat operation, BN represents batch normalization, and δ represents the nonlinear activation function; Then, M is split from the spatial dimension to obtain two feature tensors and Perform 1×1 convolution operations on these two feature tensors to increase the number of channels to the same as the number of channels of the input feature map, and then use the Sigmoid function to obtain the attention weight g h and g w : g h =σ(Conv 1×1 (y h )) g w =σ(Conv 1×1 (y w )) Where σ represents the sigmoid function; Finally, g h and g w Multiply it with the input feature map F to get the final output feature map Y∈R C×H×W :

3. The multi-scale target detection method for UAV aerial images based on coordinate and global information aggregation according to claim 1 is characterized in that: The features extracted by the two branches are fused as follows: All intermediate output features M1, M2, M3, and M4 of the two branches are first subjected to channel concat operation, and then the channel information is sorted using 1×1 convolution to obtain the final output feature map Z∈R 2C×H×W .

4. The multi-scale target detection method for UAV aerial images based on coordinate and global information aggregation according to claim 1 is characterized in that: The feature fusion methods in the multiple feature fusion network include: For the case where only two feature maps need to be fused, two learnable weights are assigned to the two feature maps to be fused to determine the importance of each feature map: Where P is the feature after fusion of F1 and F2, F1 and F2 are the two feature maps to be fused, w1 and w2 are learnable weights, and Δ is a very small number to avoid numerical instability; In the case of fusing three feature maps, the three feature maps to be fused are first fused in pairs, where each feature will participate in the fusion twice to achieve the effect of reuse. Then three different intermediate feature maps are obtained. Finally, three learnable weights are assigned to the three intermediate feature maps for fusion to obtain output features with rich contextual information for the final prediction.

Images