Unmanned aerial vehicle aerial image target detection method based on multi-scale feature information extraction

Abstract

The invention discloses an unmanned aerial vehicle aerial image target detection method based on multi-scale feature information extraction, and belongs to the field of unmanned aerial vehicle image target detection, and the method comprises the steps: carrying out the preprocessing of an unmanned aerial vehicle aerial image data set; an unmanned aerial vehicle aerial image target detection model based on multi-scale feature information extraction is constructed, the target detection model comprises a multi-scale feature extraction backbone network and a bidirectional feature fusion network, and the backbone network is composed of a multi-scale feature extraction module COFM and a C2f module. The bidirectional feature fusion network comprises a feature fusion structure DiFPN added with a large-scale detection head; and detecting the unmanned aerial vehicle image through the trained model. According to the method, the unmanned aerial vehicle aerial image with a complex scene and a large change scale can be accurately detected.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection in unmanned aerial vehicle (UAV) aerial images, and relates to a method for detecting targets in unmanned aerial vehicle (UAV) aerial images based on multi-scale feature information extraction. Background Art

[0002] With the development of drone technology, drone-based image target detection has been widely used in many fields, such as agricultural monitoring, disaster emergency response, urban management, etc. However, the images from the drone's perspective have the characteristics of large viewing angles and a large number of small targets, making the target detection task particularly complex and challenging. When processing drone images, traditional target detection algorithms often find it difficult to balance detection accuracy and real-time performance, especially for small targets.

[0003] Small object detection is a very challenging task because smaller objects have lower resolutions and the model has limited contextual information to learn. In addition, they often coexist with large objects in the same image, which tend to dominate the feature learning process, making small objects undetectable. To address this problem, in recent years, major research has focused on using feature information at different scales to improve the accuracy of small object detection. By extracting features at different scales, the detailed information of small objects can be better captured, thereby improving the detection effect. Common multi-scale methods include pyramid pooling, multi-scale convolution, and multi-scale feature fusion.

[0004] The existing YOLO series models, especially the latest YOLOv8, have made significant progress in detection speed and accuracy. However, YOLOv8 still has some shortcomings when facing the task of small target detection in drone images. Specifically, YOLOv8's ability in multi-scale feature fusion and processing low-resolution images needs to be further improved in order to achieve higher detection accuracy in complex scenes with a large number of small targets. Summary of the invention

[0005] In view of the above problems, the present invention provides a method for detecting drone aerial images based on multi-scale feature information extraction, which can detect drone aerial images and obtain detection results of various targets in the image. Specifically, the present invention proposes a multi-scale small target feature extraction module COTM, which obtains feature maps of different scales through convolution kernels of different sizes, and uses a residual connection structure to establish feature dependencies, which can effectively extract small target feature information. In order to improve the efficiency of feature fusion, the present invention proposes DiFPN, which can bidirectionally fuse feature maps of different scales. The method comprises the following steps:

[0006] Step S1, preprocessing the drone aerial image data set;

[0007] Step S2, constructing a target detection model for drone aerial images based on multi-scale feature information extraction, wherein the target detection model includes a multi-scale feature extraction backbone network and a bidirectional feature fusion network, wherein the backbone network is composed of a multi-scale feature extraction module COFM and a C2f module, and the bidirectional feature fusion network includes a feature fusion structure DiFPN with a large-scale detection head added;

[0008] Step S3, inputting the training set into the model for training to obtain a UAV aerial image detection model;

[0009] Step S4: input the test set into the model for testing to obtain model detection results and performance evaluation results;

[0010] Furthermore, the step S1 specifically includes:

[0011] S11. Use the VisDrone2019 drone aerial image public dataset to preprocess the dataset, including converting the format of the annotation file and removing incorrect category labels, and dividing the dataset into a training set and a validation set.

[0012] Furthermore, the step S2 of constructing a drone aerial image target detection model based on multi-scale feature information extraction specifically includes:

[0013] S21, constructing a feature extraction backbone network in the target detection model, and introducing a multi-scale feature extraction module COTM into the backbone network, wherein the data processing steps of the multi-scale feature extraction module COTM include:

[0014] F1=FC(DWconv 7x7 (FC(LN(F in )))

[0015] F2=FC(BN(DWconv 3x3 (F in )))

[0016] F out = F1+F2

[0017] Among them, F in is the input feature map, F1 represents the output result of the large convolution branch, DWconv 7x7 represents a depth-wise separable convolution with a kernel size of 7, FC represents a fully connected convolutional layer, LN represents a layer normalization layer, F2 represents the output of a small convolution branch, BN represents a batch normalization layer, and DWconv 3x3 Indicates a depth-wise separable convolution with a kernel size of 3;

[0018] S22. Further, an efficient feature fusion network DIFPN is introduced. The steps of constructing the bidirectional feature fusion network DiFPN specifically include:

[0019] S221, adding a small target detection head to the feature network, wherein the detection size of the detection head is 160x160, and obtaining a new feature fusion network with four detections of different sizes;

[0020] S222, feature fusion stage, FPN takes the different scale feature maps C1, C2, C3, C4 extracted by the backbone network as input, passes through the bidirectional fusion network DiFPN, the output features are marked with P as the number, and the feature maps output to the detection head are P1, P2, P3, P4, and the steps of the feature fusion stage include:

[0021] The fusion stage 1 obtains the output feature map P1 of size 160x160:

[0022] P1 = Cat(C1,Unpsample 8x (C4), Unpsample 4x (C3), Unpsample 2x (C2)

[0023] The fusion stage 2 obtains the output feature map P2 of size 80x80:

[0024] P2 = Cat (Downsample 2x (C1), C2, Unpsample 2x (C3), Unpsample 4x (C4)

[0025] The fusion stage 3 obtains the output feature map P3 of size 40x40:

[0026] P3 = Cat (Downsample 4x (C1), Downsample 2x (C2), C3, Unpsample 2x (C4)

[0027] The fusion stage 4 obtains the output feature map P4 of size 20x20:

[0028] P4 = Cat (Downsample 8x (C1), Downsample 4x (C2), Downsample 2x (C3), C4)

[0029] Among them, Cat is the concatenation operation, Upsample2× means two times upsampling, Downsample2× means two times downsampling, Upsample4× means four times upsampling, Downsample4× means four times downsampling, Upsample2× means two times upsampling, Downsample2× means two times downsampling, Upsample8× means eight times upsampling, and Downsample8× means eight times downsampling.

[0030] Furthermore, the method further comprises:

[0031] The training set is input into the network model for iterative training step S3, which includes:

[0032] S31, input the training set into the network model, and perform 200 rounds of iterative training on the network model through the SGD optimizer until the network model converges to obtain the target network model.

[0033] S32: Input the test set into the target network model to obtain the drone aerial image detection result.

[0034] The proposed detection algorithm is verified on the VisDrone2019 public dataset. The results show that compared with other advanced methods, the method of the present invention can well balance the relationship between detection accuracy and model complexity, and has better detection capabilities for targets in drone aerial images. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of a flow chart of a method for detecting targets in drone aerial images based on multi-scale feature information extraction according to an embodiment of the present invention;

[0036] Figure 2 A COTM module structure diagram constructed for an embodiment of the present invention;

[0037] Figure 3 This is a structural diagram of target detection in drone aerial images based on multi-scale feature information extraction in an embodiment of the present invention.

[0038] Figure 4 A visual comparison of the improvements before and after the embodiments of the present invention on the VisDrone dataset. DETAILED DESCRIPTION

[0039] The technical solution of the present invention is described in detail below with reference to the accompanying drawings:

[0040] As attached Figure 1 The present invention provides a method for detecting targets in unmanned aerial images based on multi-scale feature information extraction, comprising the following steps:

[0041] S1. Preprocess the drone aerial image dataset.

[0042] The public dataset VisDrone2019 dataset is used in the examples of the present invention. The experiment uses the VisDrone2019 dataset, which contains 8629 aerial images from the perspective of drones. Among them, 6470 images are training sets, 548 images are validation sets, and 1610 images are test sets. The dataset covers 10 categories, including bicycles, awning tricycles, tricycles, pedestrians, people, cars, vans, buses, trucks, and motorcycles.

[0043] S2. Construct a target detection model for drone aerial images based on multi-scale feature information extraction, wherein the target detection model includes a multi-scale feature extraction backbone network and a bidirectional feature fusion network. The backbone network is composed of a multi-scale feature extraction module COFM and a C2f module, and the bidirectional feature fusion network includes a feature fusion structure DiFPN with a large-scale detection head added.

[0044] S3, inputting the training set into the model for training to obtain a drone aerial image detection model;

[0045] The training set is input into the network model, and the network model is iteratively trained for 200 rounds through the SGD optimizer until the network model converges to obtain the target network model.

[0046] In the embodiment of the present application, the YOLO v8s model size is used as a training example, and its hyperparameters are set as follows: the initial learning rate is set to 0.01 in the SGD optimizer, the batch size is set to 4, the number of iterations is 200, and the IOU threshold is 0.5. The optimized weight parameter model is obtained through multiple rounds of iterative training as the target network model.

[0047] S4. Input the test set into the model for testing to obtain model detection results and performance evaluation results.

[0048] After the test set is input into the optimal target network model obtained by S3 for testing and the test results are obtained, the performance of the tested target network model is evaluated by the preset evaluation indicators, where the performance evaluation indicators include precision P, recall R, average precision AP, and average AP value mAP.

[0049] The technical solution of the present invention is described in detail below:

[0050] (1) Model network structure

[0051] The overall network structure of the target detection model for drone aerial images based on multi-scale feature extraction constructed in this invention is as follows: Figure 2 As described. First, the present invention proposes a multi-scale feature extraction module COTM, which obtains feature maps of different receptive fields through convolution branches of different sizes, and then superimposes feature maps of different scales through the ADD structure to avoid excessive loss of small target information in drone images during the downsampling process. Secondly, the present invention proposes a DiFPN structure to perform bidirectional feature fusion of shallow feature maps and deep feature maps, while fully retaining target feature information of different scales, improving the efficiency of feature fusion. Finally, the fused feature map is sent to the detection head for detection to obtain the final detection result.

[0052] (2) Backbone network

[0053] The backbone network is mainly composed of the multi-scale feature extraction COTM module designed by the present invention and the C2f module in the original model. The COTM module uses 7x7 convolution as the main branch to expand the receptive field to obtain global features and uses 3x3 convolution as the secondary branch to obtain local features. Finally, the local features and global features are superimposed by the ADD module to obtain the output feature map with different receptive fields superimposed, thereby realizing the multi-scale feature information extraction of drone aerial images. The COTM module structure is as follows: Figure 3 shown.

[0054] Specifically, the calculation process of the main branch is shown in formula (1):

[0055] F1=FC(DWconv 7x7 (FC(LN(F in )))

[0056] Among them, F in is the input feature map, F1 represents the output result of the large convolution branch, DWconv 7x7 represents a depth-wise separable convolution with a kernel size of 7, FC represents a fully connected convolution layer, and LN represents a layer normalization layer. The secondary branch extracts local features of the image, and the calculation process is shown in formula (2):

[0057] F2=FC(BN(DWconv 3x3 (F in )))

[0058] F2 represents the output result of the small convolution branch, BN is the batch normalization layer, DWconv 3x3 It represents a depth-wise separable convolution with a convolution kernel size of 3. Finally, the global features extracted by the main branch and the local features extracted by the secondary branch are added, and the feature maps of different receptive fields are superimposed to obtain the output feature map of multi-scale information. The calculation process is shown in formula (3):

[0059] Fout = F1+F2

[0060] Among them, F out Feature map representing the final output of the COTM module.

[0061] (3) Bidirectional feature fusion network

[0062] After the backbone network extracts feature maps of different scales, in order to improve the fusion efficiency of feature maps, the present invention proposes a bidirectional feature fusion network DiFPN. The network performs bidirectional feature fusion on the shallow feature map rich in target location information and the deep feature map rich in semantic information, and establishes the connection between the shallowest feature map and the deepest feature map. The calculation process of the fusion process of feature maps of different sizes is as follows:

[0063] In the feature fusion stage, FPN takes the feature maps C1, C2, C3, and C4 of different scales extracted by the backbone network as input, passes through the bidirectional fusion network DiFPN, and the output features are marked with P as the number. The feature maps output to the detection head are P1, P2, P3, and P4. The steps of the feature fusion stage include:

[0064] The fusion stage 1 obtains the output feature map P1 of size 160x160:

[0065] P1 = Cat(C1,Unpsample 8x (C4), Unpsample 4x (C3), Unpsample 2x (C2)

[0066] The fusion stage 2 obtains the output feature map P2 of size 80x80:

[0067] P2 = Cat (Downsample 2x (C1), C2, Unpsample 2x (C3), Unpsample 4x (C4)

[0068] The fusion stage 2 obtains the output feature map P3 of size 40x40:

[0069] P3 = Cat (Downsample 4x (C1), Downsample 2x (C2), C3, Unpsample 2x (C4)

[0070] The fusion stage 2 obtains the output feature map P4 of size 20x20:

[0071] P4 = Cat (Downsample 8x (C1), Downsample 4x (C2), Downsample 2x (C3), C4)

[0072] Among them, Cat is the concatenation operation, Upsample2× means two times upsampling, Downsample2× means two times downsampling, Upsample4× means four times upsampling, Downsample4× means four times downsampling, Upsample2× means two times upsampling, Downsample2× means two times downsampling, Upsample8× means eight times upsampling, and Downsample8× means eight times downsampling.

[0073] By fusing multi-scale feature maps rich in different feature information using a bidirectional fusion method, the feature information of small targets in UAV images can be retained to the maximum extent.

[0074] To verify the effect of the present invention, the following experiment was performed:

[0075] We conducted experiments on the VisDrone dataset and ablation experiments to verify the effectiveness of each module, where the number of model training iterations was 200. In order to verify the effectiveness of the proposed algorithm, evaluation indicators such as mAP, precision, recall and COCO evaluation indicators were used to evaluate the performance of multi-scale targets. APs is the detection rating indicator of small targets in the COCO evaluation index. The results of the ablation experiment are shown in Table 1. The data in Table 1 show that the method proposed in the present invention is improved by 3.4%, 2.1%, 1.6% and 2.7% in mAP0.5, mAP0.5:0.95, precision and recall respectively compared with the baseline YOLOv8, verifying the effectiveness of the invented module. And compared with the original model, the APs of the improved model is improved by 3.5%, proving that the accuracy of the algorithm of the present invention for small target detection has been improved. In general, the target detection method proposed in the present invention has better multi-scale target detection capabilities and is more suitable for drone aerial images with large scale changes.

[0076] surface The ablation test results of the method proposed in this invention on the VisDrone dataset are shown in Table 2.

[0077]

[0078] Furthermore, we show the detection visualization results of the accurate model and the improved model. Figure 4As shown in the figure, (a1)-(c1) are the detection results of the baseline model, and (a2)-(c2) are the detection results of the improved model. From the visualization results, it can be seen that the detection accuracy of the improved model in complex environments is better than that of the baseline model, and it reduces the cases of missed detection and false detection.

[0079] The present invention proposes a target detection method for unmanned aerial images based on multi-scale feature information extraction. Firstly, the unmanned aerial image data set is preprocessed and the data set is divided. Then, a multi-scale feature extraction module COTM and an improved feature fusion network DiFPN are introduced into the backbone network of the model. Then, the training set is input into the target network model for iterative training to obtain the optimal target detection model. Finally, the test set is input into the optimal target detection model for testing and outputting the test results. Then, the performance of the model is evaluated by preset evaluation indicators.

Claims

1. A method for detecting targets in drone aerial images based on multi-scale feature information extraction, characterized in that: The method comprises the following steps: Step S1, preprocessing the drone aerial image data set; Step S2, constructing a target detection model for drone aerial images based on multi-scale feature information extraction, wherein the target detection model includes a multi-scale feature extraction backbone network and a bidirectional feature fusion network, wherein the backbone network is composed of a multi-scale feature extraction module COFM and a C2f module, and the bidirectional feature fusion network includes a feature fusion structure DiFPN with a large-scale detection head added; Step S3, inputting the training set into the model for training to obtain a UAV aerial image detection model; Step S4: input the test set into the model for testing to obtain model detection results and performance evaluation results.

2. According to the method for detecting targets in drone aerial images based on multi-scale feature information extraction as described in claim 1, it is characterized in that: The step S1 specifically includes: S11. Use the VisDrone2019 drone aerial image public dataset to preprocess the dataset, including converting the format of the annotation file and removing incorrect category labels, and dividing the dataset into a training set and a validation set.

3. According to the method for detecting targets in drone aerial images based on multi-scale feature information extraction as described in claim 1, it is characterized in that: The data processing steps of the multi-scale feature extraction module COFM include: F1=FC(DWconv 7x7 (FC(LN(F in ))) <h2 style=";text-align:left;direction:ltr">F2=FC(BN(DWconv<h2 style=";text-align:left;direction:ltr"> 3x3 <h2 style=";text-align:left;direction:ltr"> (F<h2 style=";text-align:left;direction:ltr"> in <h2 style=";text-align:left;direction:ltr"> ))) F out = F1+F2 Among them, F in is the input feature map, F1 represents the output result of the large convolution branch, DWconv 7x7 represents a depth-wise separable convolution with a kernel size of 7, FC represents a fully connected convolutional layer, LN represents a layer normalization layer, F2 represents the output of a small convolution branch, BN represents a batch normalization layer, and DWconv 3x3 Represents a depthwise separable convolution with a kernel size of 3.

4. According to the method for detecting targets in drone aerial images based on multi-scale feature information extraction as described in claim 1, it is characterized in that: The bidirectional feature fusion network DiFPN construction step S2 specifically includes: S21, adding a small target detection head to the feature network, wherein the detection size of the detection head is 160x160, and obtaining a new feature fusion network with four detections of different sizes; S22, feature fusion stage, FPN takes the different scale feature maps C1, C2, C3, C4 extracted by the backbone network as input, passes through the bidirectional fusion network DiFPN, and the output features are marked with P as the number, and the feature maps output to the detection head are P1, P2, P3, P4. The steps of the feature fusion stage include: S221, fusion stage 1 obtains the output feature map P1 of size 160x160: P1 = Cat (C1, Unsampled) 8x (C4), Unsample 4x (C3), Unsample 2x (C2) S222, fusion stage 2 obtains the output feature map P2 of size 80x80: P2 = Cat(Downsample 2x (C1),C2,Unpsample 2x (C3),Unpsample 4x (C4)) S223, fusion stage 2 obtains the output feature map P3 of size 40x40: P3 = Cat (Downsample) 4x (C1),Downsample 2x (C2), C3, Unsample 2x (C4) S224, fusion stage 2 obtains the output feature map P4 of size 20x20: P4 = Cat (Downsample) 8x (C1), Downsample 4x (C2), Downsample 2x (C3), C4) Among them, Cat is the concatenation operation, Upsample2× means two times upsampling, Downsample2× means two times downsampling, Upsample4× means four times upsampling, Downsample4× means four times downsampling, Upsample2× means two times upsampling, Downsample2× means two times downsampling, Upsample8× means eight times upsampling, and Downsample8× means eight times downsampling.

5. A method for detecting targets in drone aerial images based on multi-scale feature information extraction according to claim 3 or 4, characterized in that: The step S3 of inputting the training set into the network model for iterative training comprises: S31, inputting the training set into the network model, and performing 200 rounds of iterative training on the network model through the SGD optimizer until the network model converges to obtain a target network model; S32: Input the test set into the target network model to obtain the drone aerial image detection result.

Images