A method for detecting densely small targets in aerial photography of unmanned aerial vehicles

Abstract

The application gives a kind of detection method for unmanned aerial vehicle aerial dense small target. First, a lightweight backbone network CSPDarknet-tiny is constructed based on YOLOv7 model, which reduces the down-sampling rate and retains more semantic information and detailed features. Second, a multi-head attention mechanism MHSA is introduced in the neck network, which effectively alleviates the interference of irrelevant information caused by complex background, helps the network pay more attention to the feature information extraction of small targets, and improves the detection accuracy of small targets. Finally, aiming at the sensitivity of IOU Loss to the position difference of small targets, NWD Loss is introduced, and the combination of IOU Loss and NWD Loss through a certain weight ratio obviously improves the detection accuracy of small targets. The application improves the small target detection problem under the condition of unmanned aerial vehicle aerial, improves the accuracy of small target detection, reduces the missed detection rate of small target detection, guarantees the excellent detection performance of small target, and has wide applicability.

Description

TECHNICAL FIELD

[0001] The present application relates to image processing technology, and in particular, to a detection method for aerial dense small targets of unmanned aerial vehicles, which combines multi-scale feature fusion and a measurement method based on Wasserstein distance, introduces a multi-head attention mechanism MHSA, and constructs a robust small target detection network. TECHNICAL BACKGROUND

[0002] Unmanned aerial vehicles have the advantages of low operating cost, high mobility, portability, multi-view, small size, etc., and can make up for the deficiencies of remote sensing satellite information acquisition. With the gradual opening of the low-altitude field and the continuous development of unmanned aerial vehicle research and development technology, unmanned aerial vehicle target detection technology has become a research hotspot for domestic and foreign experts and scholars.

[0003] Target detection technology refers to distinguishing the target of interest from the background in the obtained pictures and videos, and identifying the target type and the target location. Early target detection methods use manually designed features that cannot capture abstract semantic features well, and can only identify a single specified class, resulting in low recognition efficiency and poor detection performance. Since aerial images have more complex scenes and targets than ordinary images, this is more detrimental to traditional target detection methods and cannot meet the needs of aerial image target detection. At the same time, aerial images often come with a large amount of data, and detection usually needs to be real-time, which makes the requirements for the detection method more stringent. In recent years, with the rapid development of deep learning, image processing using convolutional neural networks has greatly improved the detection performance compared to traditional methods. The algorithm mainly includes two categories: single-stage and two-stage algorithms. Two-stage algorithms are mainly based on the basic idea of region detection, which divides the detection process into two steps. First, methods such as selective search, edge detection, and region extraction network are used to generate candidate regions that may contain targets for feature extraction. Then, a convolutional neural network is used to classify and regress the position of the candidate frame. The current two-stage algorithm generally has low false detection rate and low missed detection rate, and has good detection effect, but it needs multiple detection and classification, and the detection speed is slow. Two-stage algorithms include R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN, and SPP-Net. Single-stage detectors are different from two-stage detectors in that they do not need to generate multiple candidate regions in advance and can directly obtain the detection frame. Therefore, single-stage algorithms usually have fast detection speed, but the detection effect is low, such as SSD and YOLO.

[0004] The current target detection algorithm still has the following problems in target detection for unmanned aerial vehicle aerial image: 1) the target scale changes greatly, and the algorithm has high requirements for feature fusion; 2) the target size is small, the distribution is dense, the background is complex, and there is a contradiction between feature extraction and down-sampling of small targets, which increases the detection difficulty; 3) the algorithm model based on YOLO has large parameters and high calculation complexity. SUMMARY

[0005] The purpose of the present application is to solve the problems of low target detection accuracy and high miss detection rate caused by the large number of targets in unmanned aerial vehicle aerial images and the large proportion of small targets, provide aerial images under different time, different weather conditions and different light conditions, design an algorithm network model, and obtain a model through deep neural network training for target detection, thereby solving the problem of small target detection under unmanned aerial vehicle aerial conditions, improving the accuracy of small target detection, and reducing the miss detection rate of small target detection.

[0006] In order to achieve the above purpose, the present application provides an unmanned aerial vehicle aerial target detection model based on YOLOv7 network, which takes YOLOv7 as the backbone network, reduces the down-sampling rate, and introduces a multi-head attention mechanism (MHSA), so that the model pays more attention to target feature information, and introduces a Normalized Wasserstein Distance (NWD) loss function when calculating the regression loss, which makes up for the deficiency of small target detection, which includes three parts: the first part is to preprocess the data set, the second part is to construct the improved YOLOv7 network, and the third part is network training and testing, and output the best aerial image data set detection result.

[0007] The first part includes three steps:

[0008] Step 1: adopt unmanned aerial vehicle aerial public data set VisDrone, divide training set, validation set and test set;

[0009] Step 2: adjust the obtained data set picture to 640x640 pixels, perform data enhancement on each training picture through Mosaic data enhancement, and perform data enhancement through random flipping, scaling, color gamut transformation and other operations, and four pictures are spliced through picture splicing to obtain the final data set;

[0010] Step 3: for the data set obtained in step 2, K-means++ clustering is performed on the boundary frame to obtain new anchor frame size, and the result is compared with the original set anchor frame to calculate the matching accuracy, and the best anchor frame size is selected;

[0011] The second part includes three steps:

[0012] Step 4: Establish a lightweight backbone feature extraction network CSPDarknet-tiny. Reduce the down-sampling rate on the original YOLOv7 backbone network from 32 times to 16 times, and the output feature maps include a 160x160x256 feature map map1, an 80x80x512 feature map map2, and a 40x40x512 feature map map3;

[0013] Step 5: Process the feature map map3 obtained in step 4 using SPPCSPC to obtain a 40x40x256 feature map P1;

[0014] Step 6: Establish a feature fusion network. In the neck feature extraction network, the path fusion network of YOLOv7 is retained, and different feature layers and detection layers are fused. FPN up-sampling conveys semantic features, and PAN down-sampling conveys positioning features. The specific implementation is as follows:

[0015] (1) P1 obtained in step 5 is transmitted into the deep feature extraction module C3MS. C3MS is a multi-head attention mechanism MHSA introduced on the basis of C3, which can effectively enhance the feature extraction capability of the network, and obtain a feature map P2;

[0016] (2) The map1, map2, and P2 obtained in step 4 are fused through top-down and bottom-up paths, and the final feature maps P3, P4, and P5 are output;

[0017] The third part includes four steps:

[0018] Step 7: The feature maps P3, P4, and P5 output by step 6 are adjusted in channel number by REPConv, and three 1x1 convolution layers are used to predict objectness, class, and bbox. The final detection head used is head0 with a size of 40x40x512, head1 with a size of 80x80x256, and head2 with a size of 160x160x128;

[0019] Step 8: Adjust the network structure hyperparameters and set the network model parameters. The training batch size epoch is set to 200, the momentum Momentum is 0.937, and the initial learning rate Ir is set to 0.01;

[0020] Step 9: Train the aerial target detection model using the training set to obtain the prediction results of the target in each sample, including the target prediction bounding box and the center point position of the prediction bounding box;

[0021] Step 10: Calculate the total loss based on the sample prediction results and label differences obtained in step 9, and update the network model parameters based on the total loss to obtain the final trained model;

[0022] In calculating the regression loss in the total loss, the NWD Loss is introduced, the IOU Loss and the NWD Loss are combined through a certain weight ratio, and the regression loss function is as follows:

[0023] Loss box =λ1×(1.0-IOU)+λ2×(1.0-NWD(N a ,N b ))

[0024] Wherein, λ1 and λ2 are both 0.5, the introduced NWD Loss makes up for the deficiency of IOU in small target detection, retains the detection accuracy of the original model for large and medium targets, and significantly improves the detection ability of the model for small targets;

[0025] Step 11: input the test set in step 2 into the training model in step 10 to obtain the test result of the small target detection of the unmanned aerial vehicle.

[0026] The improvements of the present application in the YOLOv7 network model mainly include:

[0027] (1) a lightweight backbone network CSPDarknet-tiny is constructed, the down-sampling ratio is reduced from the original 32 times to 16 times, more semantic information and detail features are retained, the output feature map is changed from the original 80x80x512, 40x40x1024 and 20x20x1024 to 160x160x256, 80x80x512 and 40x40x1024, not only the parameter amount of the model is significantly reduced, but also the information loss caused by too large multiple in the down-sampling process of the target in the unmanned aerial vehicle aerial image is effectively relieved, and the detection accuracy of the small target is improved;

[0028] (2) since the background of the aerial image is complex and the target to be detected is small, it is not conducive to the extraction of the network target features, when the feature information is extracted in the backbone network, there is a lot of irrelevant information interference, which greatly affects the detection result, therefore, the present application introduces a multi-head attention mechanism MHSA, the MHSA has the ability to capture large-range image target information, and a deep feature extraction module C3MS is designed by combining C3 with the multi-head attention, which effectively relieves the interference of irrelevant information in the aerial image on feature extraction, and enhances the extraction ability of the model for small target feature information;

[0029] (3) The target in the aerial image has the characteristics of large scale difference and many small targets, and the loss function of the initial YOLOv7 model uses CIOU, but it is observed that IOU is very sensitive to the position difference of targets of different scales, so the model is not ideal in the application of the aerial image dataset. The NWD Loss is introduced to make up for the deficiency of the IOU Loss, and the NWD is not sensitive to the scale change of the target, which is beneficial to the similarity comparison between small targets, but only using the NWD Loss is not conducive to the detection of large and medium scale targets, so the NWD Loss and the IOU Loss are combined with a certain weight in the application, which not only compensates for the disadvantage of the IOU in the detection of small targets, but also retains the superiority of the IOU in the detection of large and medium scale targets. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is the overall flowchart of the present application;

[0031] Figure 2 is the network overall framework diagram of the present application;

[0032] Figure 3 is the feature fusion network framework diagram of the present application;

[0033] Figure 4 is the MHSA structure framework diagram of the present application;

[0034] Figure 5 is the test set result output by the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. In the following description, the detailed description of the prior art may weaken the subject matter of the present application, which will be ignored here. It should be understood that the specific embodiments described here are only used to explain the present application, and are not used to limit the present application. The overall flowchart of the embodiments of the present application is shown in Figure 1 The present application will be further described below in combination with the drawings.

[0036] Step 1: Obtain the unmanned aerial vehicle aerial image dataset VisDrone, which contains a total of 10209 images (6471 training set, 548 validation set, 3190 test set), contains 10 types of targets (pedestrian, person, bicycle, car, truck, truck, tricycle, sunshade tricycle, bus, car), and unifies the size of the training picture to 640*640;

[0037] Step 2: Mosaic data augmentation is performed on the data set, four pictures in the data set are randomly selected, and the four pictures are respectively randomly flipped (the original picture is flipped left and right), randomly scaled (the original picture is scaled in size), color gamut changed (the brightness, saturation and hue of the original picture are changed), and then the four pictures are spliced by matrix to obtain pictures with richer background and more information, thereby enhancing the training samples;

[0038] Step 3: For the data set obtained in step 2, the K-means++ clustering algorithm is used to cluster the width and height of all target bounding boxes in the training set, a new anchor box combination is obtained, and the value of the prior box is continuously updated through back propagation, so that it can better fit the data set, and finally the best anchor box combination [3, 4, 4, 9, 8, 7], [8, 14, 16, 9, 14, 18], [31, 17, 25, 33, 58, 42] is obtained;

[0039] Figure 2 It is an improved network model diagram based on the YOLOv7 model, first inputting a picture with a size of 640x640, outputting three different scale feature maps to the neck network through the backbone feature extraction network, outputting the corresponding scale feature maps through the path fusion network in the neck network, outputting the prediction results through Rep and CBS at the detection head, and in the embodiment, the following steps are performed:

[0040] Step 4: A lightweight backbone feature extraction network CSPDarknet-tiny is established, as shown in Figure 3As shown, first, a picture of 640x640x3 is input, a 2x down-sampling feature map of 320x320x64 is output after passing through three CBS convolutional layers, and then a 4x down-sampling feature map is obtained through a CBS convolution with a convolution kernel size of 3 and a step size of 2, wherein the CBS is composed of Conv (convolution) + BN (Batch Normalization) + SiLU (Sigmoid linear unit). The 4x down-sampling feature map obtained by processing through the ELAN module enhances the learning ability of the network without destroying the original gradient path, and outputs a feature map of 160x160x256 map1 through the ELAN module. Then, map1 is continuously down-sampled, and YOLOv7 adopts two branches to achieve this, one branch realizes spatial down-sampling through MP (maxpooling) and connects a 1x1 convolution to compress the channel, and the other branch realizes down-sampling through a 1x1 convolution to compress the channel and a 3x3 convolution with a step size of 2. Finally, the two branches are combined to obtain a feature map of 80x80x256, and 80x80x512 feature map map2 and 40x40x512 feature map map3 are obtained through ELAN and MP operations;

[0041] Step 5: The feature map map3 obtained in step 4 is processed using SPPCSPC. SPPCSPC continues the idea of SPP, increases the receptive field, and obtains different receptive fields through maxpooling. The feature map is divided into two parts, one part is subjected to regular convolution, and the other part is subjected to SPP operation, which not only improves the accuracy but also speeds up the speed, and a feature map of 40x40x256 P1 is obtained;

[0042] Step 6: Construct a feature fusion network. The present application continues the path fusion network PAFPN of YOLO, which is composed of top-down and bottom-up paths, and the specific implementation is as follows:

[0043] (3) P1 obtained in step 5 is input into deep feature extraction module C3MS. C3MS is a multi-head attention mechanism MHSA introduced on the basis of C3, which can effectively enhance the feature extraction capability of the network. Multi-head attention mechanism MHSA (Multi-Head Self-Attention) is composed of multiple self-attention modules, which capture global information in different spaces, and then the obtained information is spliced to form a new feature map, and its structure is as shown in Figure 4 P2 is obtained;

[0044] (4) map1, map2 and P2 obtained in step 4 are fused through top-down and bottom-up paths to obtain final feature maps P3, P4 and P5;

[0045] Step 7: The feature maps P3, P4, P5 output from step 6 are adjusted in channel number by REPConv, and three 1x1 convolution layers are used to predict the objectness, class and bbox, and finally the detection head used is 40x40x512 head0, 80x80x256 head1 and 160x160x128 head2;

[0046] Step 8: Adjust the network structure hyperparameters and set the network model parameters, wherein the training batch size epoch is set to 200, the momentum Momentum is 0.937, and the initial learning rate is set to Ir=0.01;

[0047] Step 9: Use the training set to train the aerial target detection model to obtain the prediction result of the target in each sample, including the target prediction bounding box and the center point position of the prediction bounding box;

[0048] Step 10: Calculate the total loss according to the sample prediction result and label difference obtained in step 9, update the network model parameters based on the total loss, and obtain the final trained model;

[0049] The total loss function used by the original YOLOv7 network when calculating the loss includes the confidence loss (Loss obj ), the regression loss (Loss box ) and the classification loss (Loss cls ), and the loss function Loss is as follows:

[0050] Loss=λ1Loss obj +λ2Loss box +λ3Loss cls

[0051] Wherein λ1, λ2, λ3 represent the weight of different loss functions in the total loss function, and the present application takes 1, the target confidence loss and the classification loss use BCEWithLogitsLoss (log binary cross entropy loss), and the regression loss uses CIOU Loss;

[0052] When calculating the regression loss, the present application introduces a new small target detection evaluation method based on the Wasserstein distance, called Normalized Wasserstein Distance (NWD), which calculates the similarity between targets through Gaussian distribution. For the detected targets, whether they overlap or not, the similarity between the distributions can be used to measure them. This method is not sensitive to target scale and is more suitable for measuring the similarity between small targets. The formula of NWD is as follows:

[0053]

[0054]

[0055] where C is a constant closely related to the data set, is a distance metric, N a and N b are Gaussian distributions modeled by A=(cx a ,cy a ,w a ,h a ) and B=(cx b ,cy b ,w b ,h b );

[0056] But the result is not improved by completely replacing IOU Loss with NWD Loss, although the detection accuracy of small targets and micro targets is improved, the detection performance of large and medium targets is reduced, so the present application retains IOU Loss, combines IOU Loss and NWD Loss through a certain weight ratio, and the complete regression loss function is as follows:

[0057] Loss box =λ1×(1.0-IOU)+λ2×(1.0-NWD(N a ,N b ))

[0058] In the present application, λ1 and λ2 are both 0.5, the introduced NWD Loss fully makes up for the deficiency of IOU in small target detection, retains the detection accuracy of large and medium targets of the original model, and significantly improves the detection ability of small targets of the model;

[0059] Step 11: input the test set in step 2 into the training model in step, and obtain the test result of unmanned aerial vehicle small target detection, as shown in Figure 5 .

[0060] The present application is aimed at the characteristics of large size difference of targets, multiple small targets and large parameter quantity in processing unmanned aerial vehicle aerial image by deep learning, a lightweight detection method of multi-scale feature fusion is proposed, first, the K-means++ clustering algorithm is used to obtain the optimal anchor box combination for the Visdrone unmanned aerial vehicle aerial image dataset, in addition, the down sampling ratio of the original YOLOv7 model backbone network is reduced from the original 32 times to 16 times, the target information lost in the down sampling process of small targets is reduced, more semantic information and target features are retained, not only the parameter quantity is reduced from 71.1M to 17.8M, but also the detection accuracy of small targets is improved through experiments. In order to solve the interference of complex background of aerial image, the multi-head attention mechanism MHSA is introduced, the deep feature extraction module C3MS is designed by combining MHSA on the basis of C3, which effectively relieves the interference of irrelevant information caused by complex background, helps the network pay more attention to the feature information extraction of small targets, and improves the detection accuracy of small targets. In addition, since it is observed that the IOU loss function used in the original model is very sensitive to the position difference of targets of different scales, the NWD Loss is introduced to make up for the deficiency, the NWD Loss and the IOU Loss are combined through a certain weight ratio, which not only retains the superiority of detecting large and medium targets, but also improves the detection ability of small targets, in the prediction aspect, the original 20x20x1024 detection head is abandoned, and the 160x160x256 detection head is added, which is more conducive to the detection of small target scales in unmanned aerial vehicle aerial images.

[0061] Although the above describes the specific embodiments of the present application, it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, all the inventions utilizing the concept of the present application are within the scope of the present application, as long as various changes are obvious within the spirit and scope of the present application defined and determined by the appended claims.

Claims

1. A method for detecting dense small targets in drone aerial photography, characterized in that, Based on YOLOv7, this paper improves upon it by combining target characteristics from UAV aerial images to establish a lightweight backbone feature extraction network. It integrates the Multi-Head Attention (MHSA) mechanism and introduces Non-WD Loss. The process includes three parts: dataset preprocessing, constructing the improved YOLOv7 network, and network training and testing. The first part consists of three steps: Step 1: Using the VisDrone public drone aerial photography dataset, divide it into training, validation, and test sets; Step 2: Adjust the obtained dataset images to 640×640 pixels, perform random flipping, scaling, and color gamut transformation on each training image using Mosaic data augmentation, and then stitch the four images together to obtain the final dataset. Step 3: For the dataset obtained in Step 2, perform K-means++ clustering on its bounding boxes to obtain new anchor box sizes. Compare the results with the original anchor box settings, calculate the matching accuracy, and select the optimal anchor box size setting. Part Two consists of three steps: Step 4: Build a lightweight backbone feature extraction network CSPDarknet-tiny. The downsampling factor is reduced on the original YOLOv7 backbone network from 32x downsampling to 16x downsampling. The output feature maps include a 160×160×256 feature map map1, an 80×80×512 feature map map2, and a 40×40×512 feature map map3. Step 5: Process the feature map map3 obtained in step 4 using SPPCSPC to obtain a 40×40×256 feature map P1; Step 6: Establish a feature fusion network. In the feature extraction network for the neck area, the path fusion network of YOLOv7 is retained, and different feature layers and detection layers are fused. FPN upsampling conveys semantic features, and PAN downsampling conveys localization features. The specific implementation is as follows: The P1 obtained in step 5 is fed into the deep feature extraction module C3MS. C3MS introduces the multi-head attention mechanism MHSA on the basis of C3, which can effectively enhance the feature extraction capability of the network and obtain the feature map P2. The maps P1, P2, and P2 obtained in step 4 are fused using top-down and bottom-up paths to output the final feature maps P3, P4, and P5. Part Three consists of four steps: Step 7: The feature maps P3, P4, and P5 output from Step 6 are adjusted for channel number by REPConv, and three 1×1 convolutional layers are used to predict the objectness, class, and bounding box. The final detection heads used are head0 (40×40×512), head1 (80×80×256), and head2 (160×160×128). Step 8: Adjust the network structure hyperparameters and set the network model parameters, including setting the training batch size (epochs) to 200, momentum (Momentum) to 0.937, and the initial learning rate to Ir to 0.

01. Step 9: Train the aerial target detection model using the training set to obtain the prediction results of the target in each sample, including the target prediction bounding box and the center point position of the prediction bounding box; Step 10: Based on the sample prediction results and label differences obtained in Step 9, calculate the total loss, update the network model parameters based on the total loss, and obtain the final trained model; When calculating the regression loss in the total loss, NWD Loss is introduced, and IOU Loss and NWD Loss are combined with a certain weight ratio. The regression loss function is as follows: Among them, take and Both are 0.

5. The introduced NWD Loss fully makes up for the shortcomings of IOU in small target detection. While retaining the original model's detection accuracy for large and medium targets, it significantly improves the model's ability to detect small targets. Step 11: Input the test set from Step 2 into the training model from Step 10 to obtain the test results of UAV small target detection.

2. The method for detecting dense small targets in UAV aerial photography according to claim 1, characterized in that, In step 4, a lightweight backbone feature extraction network CSPDarknet-tiny was established. The downsampling rate was reduced on the original YOLOv7 backbone network, thereby retaining more feature information, reducing the number of parameters and improving the detection accuracy of small targets.

3. The method for detecting dense small targets in UAV aerial photography according to claim 1, characterized in that, In step 6, a deep feature extraction module C3MS is introduced based on the multi-head attention mechanism (MHSA) to reduce the interference of complex background noise in aerial images, so that the backbone network can focus more on extracting feature information of small targets and ignore irrelevant information.

4. The detection method for dense small targets in UAV aerial photography according to claim 1, characterized in that, Step 10 introduces NWD Loss to compensate for the sensitivity of IOU Loss to the positional differences of targets at different scales. The combination of NWD Loss with certain weight ratios significantly improves the detection accuracy of small targets.

Images