Unmanned aerial vehicle aerial photograph image target detection method based on multi-scale feature fusion

Abstract

The application discloses a kind of unmanned aerial vehicle aerial image target detection methods based on multi-scale feature fusion. Including the following steps. First, using h-swish activation function in backbone network and neck network, increase the performance of model;Second, add attention mechanism (CoordAttention) in the bottleneck layer of backbone network, thereby increase the weight of effective information and inhibit background noise interference;Finally, by removing the redundant nodes of path aggregation network (PANet) and adding additional connection, and using BlurPool instead of downsampling mode, a kind of feature pyramid network (Blur-PANet) is proposed to effectively fuse multi-layer features.This paper trains the proposed network on VisDrone public dataset, and carries out online verification. Experimental results show that the improved YOLOV4 network can effectively improve the detection accuracy of unmanned aerial vehicle aerial image.

Description

TECHNICAL FIELD

[0001] The application relates to a deep neural network, and is an improved YOLOV4 unmanned aerial vehicle aerial image target detection method based on multi-scale feature fusion and attention mechanism and belongs to the field of target detection. BACKGROUND

[0002] With the rapid development of network technology, unmanned aerial vehicle image target detection has wide application, including city management, forest fire prevention, agricultural information, power line detection, traffic monitoring, land change monitoring, military reconnaissance and the like. In addition, the unmanned aerial vehicle image plays a vital role in maintaining social order. The Internet of Things combined with unmanned aerial vehicles can be used for real-time video monitoring and monitoring of blind spots. The unmanned aerial vehicle aerial image data is huge, and the traditional manual processing of image content method will cause omission and commission errors. Therefore, it is unrealistic to rely only on human resources to retrieve, view and process massive image data. The intelligent video monitoring method in the field of artificial intelligence processes massive image data using advanced algorithms, provides useful information meeting the needs of users, saves manpower and material resources, reduces monitoring cost and significantly improves monitoring efficiency. The massive video data collected by the unmanned aerial vehicle in real time can be processed using big data technology and deep learning, which changes the traditional target detection method from the inefficient manual mode to the intelligent real-time efficient mode. Therefore, it has important research value and significance to use deep learning for target detection in unmanned aerial vehicle aerial video.

[0003] Most of the previous methods in this field use traditional detection algorithms. When the unmanned aerial vehicle shoots real-time pictures of the ground from high altitude, it is easily affected by the external environment, resulting in less significant target feature points collected by the traditional detection algorithm. Therefore, when the traditional target detection algorithm is used for detection, the detection speed is slow and the precision is low, and false detection and missed detection are prone to occur. Deep learning is used to identify the target through a neural network. Compared with the traditional target detection algorithm, the detection precision and speed are greatly improved, and the deep learning plays an important role in the development process of the unmanned aerial vehicle target detection of the ground.

[0004] Target detection in drone images is more challenging than in ground images. Drone images contain numerous tiny objects, such as those smaller than 32 pixels. Audebert et al. used a deep fully convolutional network to accurately segment vehicles in aerial images and achieved vehicle detection by extracting connected components, demonstrating that combining semantic segmentation and target detection in aerial images can improve detection performance, especially in extracting target boundary information. Algorithms such as Mask R-CNN and MaskLab combine segmentation and target detection, achieving excellent results in both tasks. Inspired by this, Li et al. constructed a semantic segmentation-guided RPN (sRPN) module to suppress background clutter in aerial images. This module integrates multi-layer pyramid features into a new feature, performs atrous spatial pyramid pooling (ASPP) and convolution operations to obtain a mask and semantic features, which can help guide the RPN and obtain more accurate regression results, respectively. While sRPN improves detection accuracy to some extent, the acquired features are sparse in scale and lack strong contextual connections, easily leading to information loss. Yang et al. introduced an attention mechanism into object detection, proposing SCRDet, which uses a supervised multi-dimensional attention network (MDA-NET) to highlight target features and weaken background features. Li et al. proposed YOLOv4_Drone based on YOLOv4, adding hollow convolutions to resample feature images, thereby improving feature extraction and object detection performance for aerial images. Zhu et al. proposed TPH-YOLOv5 based on YOLOv5, replacing the original detection head with a Transformer prediction head to improve the detection capability for small targets. Summary of the Invention

[0005] To address the current problems in target detection for UAV aerial images, this invention provides an improved YOLOv4 UAV aerial image target detection method based on multi-scale feature fusion and attention mechanisms. This method can effectively improve the detection accuracy of UAV aerial images and has good detection performance. The specific scheme of this invention is as follows:

[0006] 1. Obtain the dataset

[0007] The experimental dataset used in this invention is the VisDrone2019 dataset, which was acquired by a drone platform under different weather and lighting conditions in 14 cities in China.

[0008] 2. Data Preprocessing

[0009] Data preprocessing is used to expand the dataset to improve the detection performance of small targets and to remove noise interference to some extent.

[0010] 3. An improved YOLOv4 UAV aerial image target detection model based on multi-scale feature fusion

[0011] This invention improves upon the YOLOv4 network structure, which mainly consists of a feature extraction stage, a feature fusion stage, and a detection head. First, the Mish and ReLU activation functions are replaced with the h-swish activation function in the backbone and neck layers, increasing the model's performance and reducing the number of parameters. Then, a Coordinate Attention mechanism is added to the bottleneck layer of the backbone network, making the model focus more on positional information, thereby increasing the model's feature expressiveness and suppressing background noise interference. Finally, a Feature Pyramid Network (Blur-PANet) is proposed. By removing redundant nodes from PANet and adding extra connections, and using BlurPool instead of downsampling, feature loss during downsampling is reduced, thus effectively fusing multi-layer features.

[0012] 4. Train the model and save the best model for prediction.

[0013] After building the model, the improved YOLOv4 model was trained on the VisDrone training set, and the optimal model was saved for prediction.

[0014] 5. Model Prediction and Evaluation

[0015] The optimal model is used to predict the validation set to obtain a prediction map and an evaluation metric. Meanwhile, to quantitatively evaluate the model's detection performance, this invention selects mean accuracy (mAP) as the performance evaluation metric. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the complete model structure of the present invention.

[0017] Figure 2 A schematic diagram of the attention mechanism

[0018] Figure 3 Schematic diagram of multi-scale feature enhancement structure

[0019] Figure 4 Schematic diagram of multi-scale feature fusion structure

[0020] Figure 5 A schematic diagram of the detection results on the VisDrone validation set. Detailed Implementation

[0021] Step 1: Data Preparation Stage

[0022] like Figure 5 As shown, the VisDrone2019 dataset was used as the experimental dataset for this invention. The data was acquired by a drone platform under different weather and lighting conditions in 14 cities in China, and includes 6471 training images and 1610 test images. This dataset not only features complex scenes but also generally small targets, increasing the difficulty of detection. It includes 10 types of targets: pedestrians, people, cars, bicycles, vans, buses, trucks, tricycles, sunshade tricycles, and motors.

[0023] Step 2: Construct an improved YOLOv4 drone aerial image target detection model based on multi-scale feature fusion.

[0024] This invention improves upon the classic YOLOv4 model, proposing a target detection method for UAV aerial images based on multi-scale feature fusion. Different modules are used to address different characteristics of the problems in UAV aerial image target detection. The specific work is summarized below:

[0025] (1) To address the problem of insufficient model performance, this invention replaces the Mish activation function and ReLU activation function with the h-swish activation function in the backbone network and neck network, thereby increasing the model's performance and reducing the number of parameters.

[0026] (2) To address the problem of numerous interference factors such as model noise and background, this invention adds an attention mechanism (Coordinate Attention) to the bottleneck layer of the backbone network, making the model pay more attention to positional information, thereby increasing the expressive power of model features and suppressing the interference of background noise.

[0027] (3) To address the problem of insufficient feature fusion and enhancement capabilities of the model, this invention proposes a feature pyramid network (Blur-PANet). By removing redundant nodes of PANet and adding additional connections, and using BlurPool to replace the downsampling method, the loss of features during the downsampling process is reduced, thereby effectively fusing multi-layer features.

[0028] The following sections will describe these three aspects in detail.

[0029] (1) h-swish activation function

[0030] Neural networks are essentially function fitting. Activation functions can introduce non-linearity, making the network output not simply a linear combination of input values. Theoretically, this allows them to approximate any function. Therefore, introducing a suitable activation function can improve model performance to some extent. Hence, we introduce the h-swish activation function:

[0031]

[0032] h-swish possesses unique non-monotonicity, which can improve the performance of detection models on different datasets. Furthermore, due to its lower boundedness and unboundedness, h-swish helps eliminate the saturation problem of output neurons and improves network regularization.

[0033] In this invention, the h-swish function is used as the activation function for the backbone network and the neck network, replacing the Mish activation function in the original YOLOv4, which improves detection accuracy and reduces computational cost.

[0034] (2) Attention mechanism

[0035] like Figure 1 and 2 As shown, the attention mechanism originates from research on human vision. When humans process complex visual information, to improve processing efficiency, the human eye selectively focuses on visual information of interest while automatically ignoring other visual information. As an effective method to improve the feature extraction capabilities of neural networks, the attention mechanism initially played a significant role in natural language processing. Subsequently, the SE module proposed in SENet introduced the attention mechanism into the field of computer vision, achieving significant improvements in model performance. However, the SE attention mechanism only considers encoding inter-channel information, ignoring positional information crucial for capturing target structure in visual tasks. Coordinate Attention is a novel and efficient attention mechanism that embeds positional information into channel attention, enabling the feature extraction network to focus on a larger area without introducing significant computational overhead. The structure diagram of the CA module is shown below. Figure 2 As shown.

[0036] This invention uses the CSPDarknet53 module in YOLOv4 to extract drone image features, reducing computational complexity while maintaining high accuracy. Coordinate Attention is added after the last four feature layers, making the model focus more on positional information and improving target detection accuracy in drone aerial images.

[0037] (3) Multi-scale feature fusion module

[0038] To further improve the performance of detection networks, researchers often add convolutional layers between the feature extraction network and the output layer to better fuse features. To further increase the receptive field, a Spatial Pyramid Pooling Network (SPP-Net) was added after CSPDarknet53 in YOLOv4. This added network effectively separates most of the important contextual features with almost no impact on computational speed.

[0039] like Figure 3 and 4 As shown, using max pooling with {1*1, 5*5, 9*9, 13*13} respectively, SPP-Net performs max pooling on the feature maps obtained in the previous stage, and then aggregates all feature maps by concat.

[0040] Furthermore, YOLO v4 uses PANet as the parameter aggregation method, such as Figure 4 As shown in Figure A. While PANet has proven the effectiveness of bidirectional fusion, it ignores the possibility that features at different levels may contribute differently. To more effectively reuse and fuse multi-layer features and obtain better detection performance, this paper proposes a feature fusion network named Blur-PANet. The structure of Blur-PANet is shown in Figure A. Figure 4 As shown in B.

[0041] Compared to PANet, Blur-PANet improves as follows: (1) To reduce unnecessary computational costs, PANet is simplified by removing redundant nodes at both ends of the network, resulting in P4-P4', as shown below. Figure 4 A; (2) To enhance the fusion of multi-layer features, extra edges from the original feature nodes to the output nodes are added as P6-P5' and P6-P3', and to reduce feature loss during downsampling, BlurPool is used instead of the downsampling method, thereby effectively fusing multi-layer features. Figure 4 As shown in B.

[0042] Step 3: Model Training

[0043] The VisDrone2019 dataset training set was input into the network for training. The network structure was implemented using the PyTorch framework and computed on the CUDA 10.0 architecture platform. The learning rate was 0.009, the momentum was set to 0.9, the batch size of the training dataset was 16, and the number of training iterations was 1000. The CuDNN neural network computing library was used to accelerate the computation, and the stochastic gradient descent method was used for optimization. The optimal model was saved during the training process.

[0044] Step 4: Model Prediction and Evaluation

[0045] The validation set of the VisDrone2019 dataset is input into the network, and the trained model is used for detection and prediction.

[0046] To quantitatively evaluate the detection performance of the model, this paper selects the mean accuracy (mAP) as the performance evaluation index. mAP is defined in the formula as follows.

[0047]

[0048] AP is determined based on precision and recall, and is defined as follows.

[0049]

[0050]

[0051] To more intuitively demonstrate the impact of adding different modules on the detection results, this invention conducted ablation experiments on the VisDrone2019 dataset, adding different modules to the YOLOv4 model sequentially and using the same parameters for the experiments. The specific results are shown in Table 1.

[0052] Table 1 Comparison of detection results with different modules added to YOLOv4

[0053]

[0054] "-" indicates that it is not used, and "√" indicates that it is used.

[0055] As shown in Table 1, we progressively added modules at each level to the baseline to demonstrate that these modules did not conflict. The first row shows the baseline performance. Adding the h-swish activation function improved the mAP by 0.19%; adding the CA attention mechanism improved the mAP by 0.59%; and finally, replacing PANet with Blur-PANet improved the mAP by 1.20%. This demonstrates the effectiveness of each module. Experimental results show that different modules of the proposed network structure improve the detection accuracy of UAV aerial images. Experimental results demonstrate that this invention can effectively improve the detection accuracy of UAV aerial images and has good detection performance.

Claims

1. A UAV aerial image target detection method based on multi-scale feature fusion, characterized in that, The method comprises: Step 1: data acquisition and data preprocessing; Step 2: constructing an improved YOLOV4 model based on attention mechanism and multi-scale feature fusion; Step 3: replacing a more suitable h-swish activation function, then training the improved YOLOV4 model on the VisDrone training set, and saving the optimal model for prediction; Step 4: inputting the VisDrone validation set into the trained optimal model for prediction, saving the prediction results, obtaining the evaluation indexes, and finally comparing the results; The improved YOLOV4 model comprises: adding an attention mechanism at the bottleneck layer of the backbone network, so that the model pays more attention to position information, thereby increasing the expression ability of the model features and suppressing the interference of background noise; The improved YOLOV4 model further comprises: constructing a feature pyramid network, simplifying the PANet by removing redundant nodes at both ends of the network, making the PANet P4-P4', in order to strengthen the fusion of multi-layer features, additional edges from the original feature nodes to the output nodes are added as P6-P5' and P6-P3', and in order to reduce the loss of features in the downsampling process, using BlurPool instead of downsampling, thereby effectively fusing multi-layer features. 2.The method of claim 1, wherein the method further comprises: In step 1, the acquired public data set is preprocessed, which is used to expand the data set to improve the detection performance of small targets, and to remove noise interference to a certain extent. 3.The method of claim 2, wherein the method further comprises: Replace the Mish activation function and the Relu activation function in the backbone network and the neck network with the h-swish activation function, increase the performance of the model, and reduce the parameter amount. 4.The method of claim 1, wherein the method further comprises: After the model is constructed, the improved YOLOV4 model is trained on the VisDrone training set, and the optimal model is saved for prediction.

5. The unmanned aerial vehicle aerial image target detection method based on multi-scale feature fusion according to claim 4, characterized in that: Input the VisDrone validation set into the trained optimal model for prediction, save the prediction results, obtain the evaluation indexes, and finally compare the results.

Images