A method, system and electronic device for detecting small targets in unmanned aerial image

Abstract

The present invention discloses a method, system and electronic device for detecting small targets in drone aerial images, and relates to the technical field of target detection. The method comprises: obtaining drone aerial images to be identified under complex weather conditions; inputting the drone aerial images to be identified under complex weather conditions into a small target recognition network, and obtaining the types of small targets in the drone aerial images to be identified under complex weather conditions; the types include: vehicles and pedestrians; the small target recognition network is obtained by training an improved YOLOv5 network using multiple training drone aerial images under complex weather conditions and the types of corresponding small targets, and the improved YOLOv5 network is obtained by improving the YOLOv5 network through an attention mechanism, a weighted cross-layer feature pyramid network and a variable detection head. The present invention improves the detection accuracy of small targets in drone aerial images under complex weather conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a method, system and electronic equipment for detecting small targets in unmanned aerial vehicle (UAV) aerial images. Background Art

[0002] In recent years, Unmanned Aerial Vehicle (UAV) has been increasingly used in the fields of agricultural rice ear detection, motion detection, urban inspection, etc. due to its light and fast characteristics. However, due to factors such as long-distance shooting and changing lighting conditions, UAV aerial photography may cause problems such as small target size, large size variation, dense distribution, and easy occlusion, thereby increasing the complexity of the background. In actual UAV application tasks, complex weather conditions, mainly foggy days, are often encountered. In this case, the image quality obtained by the UAV is seriously affected, resulting in a decrease in target detection performance. Therefore, improving the ability to detect small targets during UAV aerial photography under complex weather conditions has become a challenging research direction in the field of target detection.

[0003] Traditional drone detection methods usually use target detection algorithms based on manual features. For example, Shao Panyu combined the Histogram of Oriented Gradients (HOG) with the Support Vector Machine (SVM) for small target detection of drones. However, in actual application scenarios, traditional target detection algorithms based on manual features have problems such as low stability and high requirements for the detection environment. When the lighting environment, posture and weather conditions change, the detection accuracy decreases significantly. With the rise of deep learning, the use of deep learning-based target detection algorithms for drone detection has become a research hotspot. Compared with target detection based on manual features, target detection based on deep learning has the advantages of wider application fields, convenient design, and simple data set preparation. Target detection methods based on deep learning can be mainly divided into two categories. One is a two-stage target detection algorithm represented by algorithms such as Fast R-CNN and Faster R-CNN. This type of algorithm regresses the target area based on the generation of candidate boxes. It has the characteristics of high detection accuracy and low missed detection rate, but it has problems such as slow detection speed and large amount of calculation, which makes it difficult to apply to real-time detection. The second category is single-stage target detection algorithms represented by YOLO (You Only Look Once) and Single Shot Multibox Detector (SSD). Such algorithms directly predict the location and category of the target and have the advantages of fast detection speed and low computational complexity, but their ability to detect small targets is poor.

[0004] To address this problem, many scholars have proposed a deep learning-based small target detection algorithm for drones. Kisantal et al. enhanced each image by copying and pasting small targets multiple times, so that the detector can increase the detection weight of small targets, thereby improving the model's detection effect on small targets. However, although data enhancement has improved the detection effect of small targets to a certain extent, it simply increases the proportion of small targets and lacks the fusion and utilization of semantic information. Lin et al. proposed the Feature Pyramid Network (FPN), which improves the network representation ability by fusing shallow feature map information with deep feature map information, but this will increase many parameters and slow down the reasoning speed. Liu et al. proposed the Path Aggregation Network (PAN), which, based on FPN, more efficiently transfers shallow information to the deep layer from bottom to top and fuses it through upsampling, thereby speeding up the reasoning speed and improving the detection effect of small targets. However, due to the large information difference between feature maps at different levels, directly using the addition or channel dimension splicing method when fusing different levels is likely to generate redundant information and noise information, and thus easily ignore small targets at different levels. Wen et al. introduced the Coordinate Attention (CA) mechanism into the module to strengthen the focus on small targets. Yang et al. used a query mechanism to accelerate the inference speed of the target detector based on the feature pyramid. In summary, although the existing small target detection algorithms have improved the detection performance to a certain extent, there are still some shortcomings: 1) Inaccurate positioning of small targets in complex weather. Small targets in complex weather have less feature information and are easily affected by pixel-level errors. 2) Confusion of small target categories in complex weather. The categories of small targets in complex weather are similar to those of other objects in the surrounding environment, which can easily lead to confusion and misclassification. 3) Sample imbalance. Small targets in complex weather usually occupy a smaller proportion in the image, making the algorithm tend to prefer larger targets that are easier to identify. Summary of the invention

[0005] The purpose of the present invention is to provide a method, system and electronic equipment for detecting small targets in drone aerial images, which improves the detection accuracy of small targets in drone aerial images under complex weather conditions.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A method for detecting small targets in unmanned aerial images, comprising:

[0008] Obtain drone aerial images to be identified in complex weather conditions;

[0009] The unmanned aerial images to be identified under complex weather conditions are input into a small target recognition network to obtain the types of small targets in the unmanned aerial images to be identified under complex weather conditions; the types include: cars and pedestrians; the small target recognition network is obtained by training an improved YOLOv5 network using multiple training unmanned aerial images under complex weather conditions and the corresponding types of small targets, and the improved YOLOv5 network is obtained by improving the YOLOv5 network through an attention mechanism, a weighted cross-layer feature pyramid network and a variable detection head.

[0010] Optionally, the improved YOLOv5 network includes: a backbone network, a neck network and a detection network;

[0011] The backbone network is used to extract features from the unmanned aerial vehicle aerial images to be identified under complex weather conditions, and obtain multiple feature maps to be fused;

[0012] The neck network is used to fuse the feature maps to be fused to obtain multiple fused feature maps;

[0013] The detection network is used to identify small targets based on the fused feature maps.

[0014] Optionally, the backbone network includes: 5 CBS modules, 4 C3 modules and 1 SPPF module.

[0015] Optionally, the backbone network of the neck network adopts a cascade of a weighted cross-layer feature pyramid network and PANet.

[0016] Optionally, the neck network includes: 2 first feature extraction structures, 1 second feature extraction structure and 3 third feature extraction structures;

[0017] The first feature extraction structure includes: a CBS module, an upsampling module, a splicing module and a C3 module; the second feature extraction structure includes: a CBS module, an upsampling module, a splicing module and an RFFEB module; the third feature extraction structure includes: a CBS module, a splicing module and an RFFEB module;

[0018] The RFFEB module includes: 5 1×1 convolutional layers, 2 3×3 convolutional layers, 1 5×5 convolutional layer, 1 7×7 convolutional layer, 1 3×3 convolutional layer with a dilation rate of 3, 1 3×3 convolutional layer with a dilation rate of 5, and 1 3×3 convolutional layer with a dilation rate of 7.

[0019] Optionally, the detection network includes: 4 CBS modules and 4 variable detection heads.

[0020] A small target detection system for unmanned aerial imagery, comprising:

[0021] Image acquisition module, used to acquire drone aerial images to be identified in complex weather conditions;

[0022] The small target detection module is used to input the unmanned aerial vehicle aerial images to be identified under complex weather conditions into a small target recognition network to obtain the types of small targets in the unmanned aerial vehicle aerial images to be identified under complex weather conditions; the types include: cars and pedestrians; the small target recognition network is obtained by training an improved YOLOv5 network using multiple training unmanned aerial vehicle aerial images under complex weather conditions and the corresponding types of small targets, and the improved YOLOv5 network is obtained by improving the YOLOv5 network through an attention mechanism, a weighted cross-layer feature pyramid network and a variable detection head.

[0023] An electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the above-mentioned method for detecting small targets in drone aerial images.

[0024] Optionally, the memory is a readable storage medium.

[0025] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0026] The present invention discloses a method, system and electronic device for detecting small targets in drone aerial images. First, drone aerial images to be identified in complex weather are obtained; then, the drone aerial images to be identified in complex weather are input into a small target recognition network to obtain the types of small targets in the drone aerial images to be identified in complex weather; the types include: cars and pedestrians. Among them, the small target recognition network is obtained by training an improved YOLOv5 network using multiple training drone aerial images in complex weather and the corresponding types of small targets, and is obtained by improving the YOLOv5 network through an attention mechanism, a weighted cross-layer feature pyramid network and a variable detection head. The present invention uses an improved YOLOv5 network obtained by improving the YOLOv5 network through an attention mechanism, a weighted cross-layer feature pyramid network and a variable detection head to detect small targets in drone aerial images, thereby improving the detection accuracy of small targets in drone aerial images in complex weather. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1A schematic diagram of a process flow of a method for detecting small targets in drone aerial images provided in Example 1 of the present invention;

[0029] Figure 2 This is a schematic diagram of the YOLOv5 network structure;

[0030] Figure 3 This is a schematic diagram of the C3_N module structure;

[0031] Figure 4 It is a schematic diagram of the CBS module structure;

[0032] Figure 5 It is the schematic diagram of the SPPF module structure;

[0033] Figure 6 To improve the YOLOv5 network structure diagram;

[0034] Figure 7 This is a schematic diagram of the FPN+PANet structure;

[0035] Figure 8 This is a schematic diagram of the BiFPN+PANet structure;

[0036] Fig. 9 It is a schematic diagram of the CBAM module structure;

[0037] Fig.10 It is the schematic diagram of RFFEB module structure;

[0038] Fig.11 It is a schematic diagram of the structure of a variable detection head;

[0039] Fig.12 This is the detection result diagram of YOLOv5 in the case of dense distribution;

[0040] Fig.13 This is the detection result diagram of DIT-YOLOv5 in the case of dense distribution;

[0041] Fig.14 This is the detection result of YOLOv5 under complex background;

[0042] Fig.15 This is the detection result of DIT-YOLOv5 under complex background;

[0043] Fig.16 This is the detection result of YOLOv5 under dark background and low light intensity at night;

[0044] Fig.17 This is the detection result of DIT-YOLOv5 under dark background and low light intensity at night;

[0045] Fig.18This is the detection result diagram of YOLOv5 for extremely small targets;

[0046] Fig.19 This is the detection result diagram of DIT-YOLOv5 for extremely small targets;

[0047] Fig. 20 This is the detection result of YOLOv5 for small targets in haze weather;

[0048] Fig.21 This is the detection result of DIT-YOLOv5 for small targets in haze weather;

[0049] Fig. 22 This is the detection result of YOLOv5 for a small target on the road in dense fog;

[0050] Fig.23 This is the detection result of DIT-YOLOv5 for small targets on the road in dense fog;

[0051] Fig.24 This is the detection result of YOLOv5 for a small target on a country road in the mist;

[0052] Fig.25 This is the detection result of DIT-YOLOv5 for small targets on a country road in the mist;

[0053] Fig.26 This is the detection result of YOLOv5 for small targets in a scene with many vehicles;

[0054] Fig. 27 This is the detection result of DIT-YOLOv5 for small targets in a scene with many vehicles. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] The purpose of the present invention is to provide a method, system and electronic equipment for detecting small targets in drone aerial images, aiming to improve the detection accuracy of small targets in drone aerial images under complex weather conditions.

[0057] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] Example 1

[0059] Figure 1 This is a flow chart of a method for detecting small targets in drone aerial images provided by Embodiment 1 of the present invention. Figure 1 As shown, the method for detecting small targets in drone aerial images in this embodiment includes:

[0060] Step 101: Acquire the drone aerial image to be identified under complex weather conditions.

[0061] Step 102: Input the unmanned aerial vehicle aerial image to be identified under complex weather conditions into a small target recognition network to obtain the type of small targets in the unmanned aerial vehicle aerial image to be identified under complex weather conditions.

[0062] Among them, the types include: cars and pedestrians; the small target recognition network is obtained by training the improved YOLOv5 network using multiple training drone aerial images in complex weather and the corresponding small target types. The improved YOLOv5 network is obtained by improving the YOLOv5 network through the attention mechanism, weighted cross-layer feature pyramid network and variable detection head.

[0063] Specifically, YOLO is a target detection model proposed by Joseph Redmon, Ali Farhadi and others in 2015. It is one of the most popular single target detection algorithms. Thanks to the transformation and innovation of many researchers, the YOLO series of algorithms have been continuously developed and updated. Different versions of the YOLO algorithm have their own advantages and disadvantages. According to different application requirements, each version combines new technologies, enhances various functions, and introduces different auxiliary technologies such as data enhancement and loss functions. At present, the YOLO algorithm has developed to the eighth generation. In view of the needs of drone application scenarios and the maturity of the algorithm, the present invention chooses to improve the YOLOv5 version. YOLOv5 is divided into four types of S, M, L, and X from small to large. The network structures of these four specifications are consistent, and the difference lies in the network width and network depth. Due to the problems of complex background and many small target samples in the target detection task of drone aerial images, a certain network depth is required to extract feature information. At the same time, considering the load-bearing situation of the drone, a certain network lightweight is required, and the comprehensive detection speed and accuracy requirements are required. Therefore, the present invention selects the YOLOv5 algorithm with the fastest detection speed, the least number of parameters and a specification of S as the baseline method.

[0064] The overall framework of YOLOv5 is as follows Figure 2 As shown in Figure 1, it is divided into three parts: backbone network (Backbone), neck network (Neck) and detection network (Head). Backbone is responsible for extracting the features of the input image and generating feature maps. Among them, the CBS module includes three operations: convolution, normalization and activation function. Figure 3As shown in Figure 2, the C3_N module is used for feature extraction and adopts a residual structure without changing the size of the feature map. Figure 4 As shown in , the CBS module is used to encapsulate the combined operation of convolution, batch normalization, and activation function. Figure 5 As shown in the figure, the SPPF module is used to achieve adaptive size output and convert feature maps of any size into feature vectors of fixed size. Neck is responsible for fusing the feature maps generated by the backbone network to enhance the performance of the entire network. Upsample in Neck is an upsampling operation, and Concat is a concatenation operation. In order to enhance the feature extraction capability, Neck adopts the FPN+PANet structure. The FPN structure transfers high-level feature information to the lower layer and fuses it through top-down upsampling to improve the target detection effect; the PANet structure enhances feature fusion through bottom-up downsampling on the basis of FPN. Head performs the final regression prediction. After 8, 16, and 32 times downsampling, three different scales of detection, P3, P4, and P5, are generated for identifying small, medium, and large targets of three different sizes.

[0065] As an optional implementation, the improved YOLOv5 network includes: a backbone network, a neck network and a detection network.

[0066] The backbone network is used to extract features from drone aerial images to be identified in complex weather conditions and obtain multiple feature maps to be fused.

[0067] The neck network is used to fuse the feature maps to be fused to obtain multiple fused feature maps.

[0068] The detection network is used to identify small targets based on the fused feature maps.

[0069] As an optional implementation, the backbone network includes: 5 CBS modules, 4 C3 modules and 1 SPPF module.

[0070] As an optional implementation, the backbone network of the neck network adopts a cascade of a weighted cross-layer feature pyramid network and PANet.

[0071] As an optional implementation, the neck network includes: 2 first feature extraction structures, 1 second feature extraction structure and 3 third feature extraction structures.

[0072] The first feature extraction structure includes: CBS module, upsampling module, splicing module and C3 module; the second feature extraction structure includes: CBS module, upsampling module, splicing module and RFFEB module; the third feature extraction structure includes: CBS module, splicing module and RFFEB module.

[0073] The RFFEB module includes: 5 1×1 convolutional layers, 2 3×3 convolutional layers, 1 5×5 convolutional layer, 1 7×7 convolutional layer, 1 3×3 convolutional layer with a dilation rate of 3, 1 3×3 convolutional layer with a dilation rate of 5, and 1 3×3 convolutional layer with a dilation rate of 7.

[0074] As an optional implementation, the detection network includes: 4 CBS modules and 4 variable detection heads.

[0075] Specifically, Figure 6-Figure 11 As shown in the figure, the structure of the improved YOLOv5 network (Diminutive Target-YOLOv5, DIT-YOLOv5) is as follows:

[0076] The framework structure of the DIT-YOLOv5 algorithm is as follows Figure 6 As shown in the figure, the Neck part adopts the weighted cross-layer feature pyramid network (Bi-directional Feature Pyramid Network, BiFPN) structure, which effectively fuses features at different levels through bidirectional cross connection and fast normalization methods. In addition, the RFFEB module is introduced, which introduces the attention mechanism to improve the detection accuracy of small targets. One of the reasons why small target detection does not work well in YOLOv5 is that the sample size of small targets is small. By default, the input resolution of the YOLOv5 algorithm is 640×640, and 8x, 16x, and 32x downsampling operations are used, which results in the generation of deep feature maps, making it difficult for the network to effectively learn the feature information of small targets. Therefore, the downsampling strategy needs to be adjusted accordingly to better adapt to the small object dataset under complex weather conditions. To solve this problem, DIT-YOLOv5 introduces a 4x downsampling small target detection layer, aiming to improve the recognition and detection efficiency of small targets, thereby solving the problem that the network is prone to missed detection when dealing with small targets in complex weather conditions. Therefore, the Head part adds a feature layer P2 obtained by 4x downsampling, and replaces the regression detection head of the YOLOv5 network with the variable detection head VA-Head proposed in this invention, thereby improving the network's positioning of dense small targets in complex weather conditions.

[0077] (1) Backbone network.

[0078] The backbone network includes: 5 CBS modules, 4 C3 modules and 1 SPPF module.

[0079] (2) Neck network.

[0080] During drone aerial photography in complex weather conditions, one of the difficulties in detecting small targets is how to effectively represent and process multi-scale feature fusion. The usual multi-scale feature fusion is to adjust all features to the same scale first, and then add the features. However, different input features have different effects on output features at different scales. In addition, during model training, due to the increase in the number of downsampling of deep feature maps, the receptive field becomes larger and the overlapping area between receptive fields increases, resulting in insufficient granularity of the extracted feature information, and thus ignoring some spatial position information that only shallow feature maps have, affecting the positioning accuracy and detection accuracy of small targets.

[0081] The key to solving this problem is how to effectively fuse the shallow feature map with the deep feature map to generate a final feature map with rich spatial location information and semantic information. The backbone network used by YOLOv5 is a cascade of FPN and PANet (such as Figure 7 As shown in Figure 1). FPN transfers and fuses feature maps of different levels by establishing contextual connections, while PANet further performs multi-scale context aggregation and selectively aggregates features of different levels through an adaptive feature aggregation module. However, during upsampling and downsampling and tensor concatenation, the weight distribution of fused features may be inconsistent, resulting in poor detection of small targets in complex weather conditions. Therefore, the improved YOLOv5 network uses a weighted cross-layer feature pyramid network (BiFPN) to replace the FPN in the YOLOv5 network, and the resulting backbone network structure is shown in Figure 1. Figure 8 As shown in the figure, learnable weights are introduced to distinguish the importance of different input features. This can enhance the influence of small target features in complex weather on the feature fusion network and improve the detection performance of small targets in complex weather.

[0082] BiFPN integrates bidirectional cross-scale connections and fast normalized fusion. The weighted fusion method it adopts is to fuse feature layers of different resolutions, add a weight to each input, and then the network adjusts the fusion weights of different inputs. The fast normalized feature fusion formula is as follows:

[0083] (1).

[0084] in, It is the feature after normalized feature fusion; A small additional value is used to keep the value stable. ; is the learnable weight of the input feature of the jth layer, which is ensured by the ReLU activation function ; is the learnable weight of the input feature of the i-th layer, which is ensured by the ReLU activation function ; is the input feature of the i-th layer.

[0085] During fast normalization fusion, the feature fusion between the upper and lower layers is implemented as follows (excluding and of fusion).

[0086] (2).

[0087] (3).

[0088] in, is the intermediate feature of the top-down path of the i-th layer; is the convolution operation; and are all learning weights associated with the input features; is the input feature of the i-th layer; Resize is the sampling operation; is the intermediate feature of the top-down path of the i+1th layer; is the output feature of the bottom-up path of the i-th layer; , and are all learning weights updated after calculation in the previous layer; is the output feature of the bottom-up path at layer i-1.

[0089] Compared with the node connection method of the feature pyramid network and path aggregation network cascade of the original YOLOv5, BiFPN has the following advantages: (1) Reduce computational complexity and improve feature transfer efficiency. Since the nodes without feature fusion have limited contribution to the feature network transfer calculation, the intermediate nodes of P2 and P5 can be removed to form a small-scale simplified bidirectional network, making the network transfer more efficient. (2) Realize high-dimensional feature fusion. Unlike the path aggregation network, which has only one top-down feature path and one bottom-up feature path, the weighted bidirectional feature pyramid network BiFPN regards each bidirectional path as a feature network layer. Repeating this feature network layer multiple times can achieve deeper feature fusion, thereby improving the network's expression ability and feature extraction ability. (3) Effectively fuse features of different resolutions and enhance the sensitivity of output features to small target detection. BiFPN cross-connects the features in the upper and lower directions, and through normalization operations, makes each input have different importance to the detection network, thereby increasing the weight of small targets.

[0090] In common drone aerial photography scenes, the scales of objects in the images vary. With the deepening of the network structure and multiple convolution operations, small targets in complex weather conditions often lose a lot of key feature information, making them difficult to detect and identify in high-level feature maps. Therefore, it is crucial to obtain feature information of various scales to improve the reliability of small target detection in complex weather conditions. Although the receptive field block (RFB) can capture image features of various scales and obtain different receptive field sizes, the extracted feature information is extensive and lacks attention to key details, resulting in unsatisfactory detection performance of small targets.

[0091] To solve this problem, the present invention introduces a receptive fields feature extraction block (RFFEB), which integrates channel and spatial attention mechanisms. The RFFEB module takes into account the feature variations between different receptive field channels to enhance the expression of feature information. The convolutional block attention module (CBAM) is an attention module that improves the convolutional neural network by combining channel and spatial attention, which helps to capture the importance of each channel in the input feature map and identify the importance of different positions on the feature map, such as Fig. 9 As shown in Figure 2, the introduction of the CBAM module not only allows RFB to cover a larger area to obtain rich feature information, but also adopts the attention mechanism to extract key features from the rich feature information. Therefore, it enhances the model's ability to detect multi-scale and dense small targets in complex backgrounds.

[0092] like Fig.10 As shown in the figure, the RFFEB structure consists of five parallel branches. The first branch consists of 1×1 and 3×3 convolutional layers, which aims to extract information from the input feature map without expansion. The three branches in the center use expansion rates of 3, 5, and 7, respectively, and each branch integrates a CBAM module to collect comprehensive feature information while emphasizing important details. Subsequently, the last branch contains only a 1×1 convolutional layer to reduce the number of channels. The feature maps extracted from the first four branches are concatenated and added to the original input feature information of the fifth branch to form a residual structure.

[0093] The implementation process of the RFFEB structure is as follows:

[0094] (4).

[0095] (5).

[0096] (6).

[0097] (7).

[0098] (8).

[0099] in, is the output of the branch consisting of 1×1 and 3×3 convolutional layers; is the input feature; is the output of the branch consisting of the CBAM module, 1×1, 3×3 convolutional layers, and a 3×3 convolutional layer with a dilation rate of 3; is the output of the branch consisting of the CBAM module, 1×1, 5×5 convolutional layers, and a 3×3 convolutional layer with a dilation rate of 5; is the output of the branch consisting of the CBAM module, 1×1, 7×7 convolutional layers, and a 3×3 convolutional layer with a dilation rate of 7; is the output of a branch consisting of a 1×1 convolutional layer;

[0100] (3) Detection network.

[0101] The complexity of positioning and classification in object detection essentially stems from the contradiction between translation invariance and image scale invariance in convolutional neural networks. Because real-world images usually contain multiple objects of different scales and sizes, and these objects may present completely different shapes and positions under different viewing angles. To solve this problem, the head part of the object detection should have a certain spatial perception ability.

[0102] The detection network proposes a new detection head called variable head (VA-Head). Fig.11 As shown, the detection head coherently combines multiple self-attention mechanisms to adapt diverse feature-level importances across scale-aware feature hierarchies and space-aware spatial locations, making it adaptive to the input data.

[0103] For a given feature pyramid 4D tensor ,in, represents the number of layers of the pyramid, , , and Represent the height, width and number of channels of the feature respectively. The self-attention expression is:

[0104] (9).

[0105] in, is the self-attention function; is the attention function.

[0106] The attention function is implemented through a fully connected layer, but learning the attention function directly in all dimensions will waste a lot of computing resources. Therefore, the present invention converts the attention function into three consecutive attentions, each of which focuses on only one angle. This decomposition helps to deal with the relationship between features at different levels and object scale differences, thereby improving the representation learning of features at different levels and helping to improve the scale perception of target detection. In addition, various geometric transformations of different object shapes are related to features at different spatial positions, so improving the representation learning of F at different spatial positions also helps to enhance the spatial perception of object detection.

[0107] According to the semantic importance of different scales, the dynamic fusion feature equation is as follows:

[0108] (10).

[0109] in, For task-aware attention function; is the scale-aware attention function; is the spatial-aware attention function.

[0110] Sparse spatial attention is learned through variable convolution, and spatial positions are sampled through additional self-learned offsets. This not only applies attention to each spatial position, but also adaptively aggregates multiple feature layers to learn more discriminative representations.

[0111] (11).

[0112] in, For the Layer pyramid; For the sparsely sampled locations; For the Layer Pyramid The weights of the values ​​at the sparsely sampled locations; It is used to use different variable convolutions according to feature maps of different levels, and unify feature maps of different levels into a function of the size of feature maps of the intermediate level; For the The self-learned spatial offset of the sparsely sampled locations; is the number of object classes; is the scalar of self-learning at position pk.

[0113] Dynamically fuse features according to their semantic importance at different scales.

[0114] (12).

[0115] in, Approximately represents the linear function of the 1×1 convolutional layer; is the Sigmoid function.

[0116] Use the dynamic ReLU activation function DYReLU-b. First, The network is then processed through two fully connected layers and a normalization layer, and finally the output is normalized to [-1, 1] using the Sigmoid function.

[0117] In order to verify the method in the embodiment, experiments and analysis were also conducted, and the specific contents are as follows.

[0118] 1. Dataset and Experimental Environment

[0119] The present invention selects the VisDrone2021 dataset and the self-made Foggy dataset for experiments. The VisDrone2021 dataset was collected by the AISKYEYE team of Tianjin University, covering images taken at different locations and altitudes using different models of drones under different scenes, weather and lighting conditions. The dataset includes a total of 8599 images and is divided into three subsets: training set (6471 images), validation set (548 images), and test set (1580 images). The Foggy dataset was collected by the team itself, covering more complex weather conditions, using images taken at different locations using the same model of drone. The dataset includes a total of 2066 images and is divided into three subsets: training set (1275 images), validation set (124 images), and test set (667 images).

[0120] Cars and pedestrians occupy the majority of the targets, while tricycles, buses, etc. account for a very small proportion of the targets, and the vast majority of the targets are small targets, resulting in an uneven distribution of target types, which brings significant challenges to the experiment.

[0121] The experimental environment is based on the Windows system. The model training and inference process are performed on the RTX 3080Ti GPU. The deep learning framework uses PyTorch 1.12 and the CUDA version number is 11.7. The total number of training rounds is set to 300 epochs, and the learning rate is 0.01. The optimizer uses the stochastic gradient descent method SGD (Stochastic Gradient Descent) with momentum, the momentum parameter is set to 0.937, and the weight decay coefficient is 0.0005.

[0122] 2. Experimental evaluation indicators

[0123] The recall rate R (recall), mean average precision (mAP), precision P (Precision) and frames per second (FPS) are used as evaluation indicators of the experiment.

[0124] Among them, the calculation expressions of P and R are as follows:

[0125] (13).

[0126] in, The number of true samples that are predicted and tested correctly; is the number of false positive samples that are incorrectly predicted as positive samples; is the number of false negative samples that were mistakenly missed.

[0127] The calculation formula of mAP is as follows:

[0128] (14).

[0129] in, is the ordinal number of the category; For the Average precision of the categories; is the total number of categories.

[0130] Average Precision (AP) is considered as the area under the PR curve of a certain class, and the formula is as follows:

[0131] (15).

[0132] 3. Ablation experiment and result analysis

[0133] In order to verify the impact of each module on the performance of the model, this experiment was based on the original YOLOv5 algorithm and conducted a series of ablation experiments on the VisDrone2021 dataset. These experiments added the improved modules and methods proposed in this invention one by one to evaluate their impact on the performance of the model. To ensure the accuracy of the experiment, no pre-trained model was used in the experimental setting, and the Batchsize was set to 32 and the training epochs were set to 300. First, the experiment added a new detection branch P2 on the basis of the original YOLOv5, and then replaced the structure in Neck with the BiFPN feature pyramid network structure. Subsequently, the feature fusion module RFFEB with attention was added, and finally the coupled prediction head was replaced with the detection head V-Head. To ensure the fairness of the comparison, different experiments only added the corresponding modules one by one without changing the optimization method or hyperparameters. The experimental results are shown in Table 1.

[0134] According to the ablation experiment results, the model improvement of the present invention shows a significant improvement in the accuracy of small target detection. First, the comparative experimental results show that the newly added detection branch P2 has the most obvious improvement in the mAP50 index, reaching 4.5 percentage points. This shows that the newly added detection branch is very effective for small target detection, and that setting the newly added P2 detection branch anchor frame to the size of the small target can greatly reduce the missed detection caused by the anchor frame being set too large. Secondly, the RFFEB structure improves the mAP50 index by 2.7 percentage points, indicating that the introduction of a feature fusion module with an attention mechanism can increase the attention to small targets, thereby improving the detection effect. In addition, the V-Head detection head has improved mAP50 by 1.6 percentage points, which shows that the introduction of attention mechanisms in the three dimensions of detection head scale, space and task can improve the accuracy of target detection. Finally, the BiFPN structure improves mAP50 by 0.6 percentage points, indicating that the detection effect of small targets can be improved by multi-level information fusion combined with shallow shape and size information. The detection effects of small targets such as awning tricycles, bicycles and vans in the table are significantly improved after each model modification, which shows that each added module has a positive impact on the small target detection effect.

[0135] Table 1 Ablation experiment results of VisDrone dataset

[0136] .

[0137] 4. Comparative experiment

[0138] In order to verify the performance of the DIT-YOLOv5 algorithm, this experiment compared the DIT-YOLOv5 algorithm with the mainstream target detection algorithms. The results are shown in Table 2, which shows the detection performance of the mainstream algorithms, including the classic two-stage target detection algorithm Light-RCNN and the more popular different branches of the single-stage target detection algorithm, namely the QueryDet algorithm and the TPH-YOLOv5 algorithm. When the AP value is 50%, DIT-YOLOv5 is 30.68 percentage points, 8.84 percentage points, and 24.51 percentage points higher than RetinaNet, YOLOX, and Cascade R-CNN, respectively. Compared with the TPH-YOLOv5 algorithm, it has achieved similar detection accuracy. When the AP is 75%, the mAP of DIT-YOLOv5 reaches 39.95%, which is 1.26 percentage points and 8.54 percentage points higher than TPH-YOLOv5 and YOLOX, respectively.

[0139] Table 2 Experimental results of DIT-YOLOv5 and mainstream methods

[0140] .

[0141] The experimental results show that the two different AP values ​​of DIT-YOLOv5 are better than the comparison algorithms, and at AP50, the improvement is more obvious than other algorithms, which proves that the method of the present invention has stronger ability in small target detection. In general, DIT-YOLOv5 shows certain advantages compared with other advanced algorithms and is suitable for drone aerial image target detection tasks.

[0142] 5. Detection effect analysis

[0143] In order to verify the detection effect of the DIT-YOLOv5 algorithm in actual scenes, this experiment studies its detection ability in special scenes. One drone aerial image each with dense distribution, complex background, extremely small target and dark background in the VisDrone2021 dataset is selected, and YOLOv5 and DIT-YOLOv5 are used for testing. The test results are compared. Figure 12-Figure 19 shown.

[0144] pass Fig.12 and Fig.13 It can be seen that in the case of dense distribution, YOLOv5 fails to identify pedestrians in densely packed vehicles in the distance, while DIT-YOLOv5 can accurately identify them. Fig.14 and Fig.15 It can be seen that in a complex background, YOLOv5 mistakenly identifies the eaves as a car due to the interference of complex background information, while DIT-YOLOv5 is not disturbed by complex information and correctly identifies the target in the picture. Fig.16 and Fig.17 It can be seen that under the condition of dark background and low light intensity at night, YOLOv5 failed to correctly identify the pedestrian next to the red vehicle, while DIT-YOLOv5 did not miss any. Fig.18 and Fig.19 It can be seen that for extremely small targets, DIT-YOLOv5 can learn features more fully, and compared with YOLOv5, it does not misdetect the black object on the roof as a motorcycle.

[0145] To further verify the detection effect of the DIT-YOLOv5 algorithm in complex weather scenes, this experiment studied its detection ability in different weather scenes. The drone aerial images in complex weather in the Foggy test set were selected for testing, including T-junctions in haze weather, roads in thick fog, country roads in light fog, and scenes with many vehicles. YOLOv5 and DIT-YOLOv5 were used for testing respectively, and the test results were compared. Figure 20-Figure 27 shown.

[0146] pass Fig. 20 and Fig.21 It can be seen that in the dense haze at the center of the T-junction, YOLOv5 failed to identify the vehicle with a smaller outline in the distance, while DIT-YOLOv5 was able to accurately identify it. Fig. 22 and Fig.23 It can be seen that on the road in dense fog, YOLOv5 mistakenly identified the antenna on the car as a pedestrian due to the interference of complex background information, and did not identify the pedestrian in the deep fog, while DIT-YOLOv5 was not disturbed by complex information and correctly identified the target in the picture. Fig.24 and Fig.25 It can be seen that on the foggy country road, YOLOv5 mistakenly identified the small objects on both sides of the country road as cars and motorcycles, while DIT-YOLOv5 did not make this recognition error. Fig.26 and Fig. 27 It can be seen that in foggy scenes with many targets to be detected, DIT-YOLOv5 can learn features more fully and recognize almost all the targets to be detected. On the contrary, YOLOv5 did not recognize the truck on the road.

[0147] By comparing the results, it can be seen that DIT-YOLOv5 obtains more sufficient feature information of small targets through the fusion of multi-level features, and extracts information that is beneficial to target positioning and classification from a large amount of multi-scale information. Compared with YOLOv5, it reduces missed detections and false alarms, and has stronger recognition capabilities for small targets in drone images under complex backgrounds.

[0148] Finally, this experiment tested DIT-YOLOv5 on drone aerial images in multiple complex weather scenarios in the public VisDrone test set and the self-made Foggy test set. The test set includes many scenes such as day and night scenes, thick fog and light fog, etc. For densely distributed small targets, such as cars and pedestrians, DIT-YOLOv5 can accurately locate the target location.

[0149] In the case of complex backgrounds, this method can eliminate the influence of interfering objects such as trees and buildings and correctly classify and locate targets. In general, the DIT-YOLOv5 algorithm has shown excellent detection results in actual scenes under different lighting conditions, backgrounds, and target distributions, meeting the needs of drone aerial image target detection tasks.

[0150] Example 2

[0151] The UAV aerial image small target detection system in this embodiment includes:

[0152] The image acquisition module is used to obtain the drone aerial images to be identified under complex weather conditions.

[0153] The small target detection module is used to input the drone aerial images to be identified under complex weather conditions into the small target recognition network to obtain the types of small targets in the drone aerial images to be identified under complex weather conditions; the types include: cars and pedestrians; the small target recognition network is obtained by training the improved YOLOv5 network using multiple training drone aerial images under complex weather conditions and the corresponding types of small targets. The improved YOLOv5 network is obtained by improving the YOLOv5 network through the attention mechanism, the weighted cross-layer feature pyramid network and the variable detection head.

[0154] Example 3

[0155] An electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the method for detecting small targets in drone aerial images in Example 1.

[0156] As an optional implementation, the memory is a readable storage medium.

[0157] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0158] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for detecting small targets in drone aerial images, characterized in that: The method comprises: Obtain drone aerial images to be identified in complex weather conditions; The unmanned aerial image to be identified under complex weather is input into a small target recognition network to obtain the types of small targets in the unmanned aerial image to be identified under complex weather; the types include: cars and pedestrians; the small target recognition network is obtained by training an improved YOLOv5 network using multiple training unmanned aerial images under complex weather and the corresponding types of small targets, and the improved YOLOv5 network is obtained by improving the YOLOv5 network through an attention mechanism, a weighted cross-layer feature pyramid network and a variable detection head; Improved YOLOv5 network includes: backbone network, neck network and detection network; The backbone network is used to extract features from drone aerial images to be identified in complex weather conditions, and obtain multiple feature maps to be fused; The neck network is used to fuse the feature maps to be fused to obtain multiple fused feature maps; The detection network is used to identify small targets based on the fused feature maps; The neck network includes: 2 first feature extraction structures, 1 second feature extraction structure and 3 third feature extraction structures; The first feature extraction structure includes: a CBS module, an upsampling module, a splicing module and a C3 module; the second feature extraction structure includes: a CBS module, an upsampling module, a splicing module and an RFFEB module; the third feature extraction structure includes: a CBS module, a splicing module and an RFFEB module; The RFFEB module includes: 5 1×1 convolutional layers, 2 3×3 convolutional layers, 1 5×5 convolutional layer, 1 7×7 convolutional layer, 1 3×3 convolutional layer with a dilation rate of 3, 1 3×3 convolutional layer with a dilation rate of 5, and 1 3×3 convolutional layer with a dilation rate of 7; The RFFEB module consists of five parallel branches; the first branch consists of a 1×1 convolutional layer and a 3×3 convolutional layer; the three central branches have dilation rates of 3, 5, and 7, respectively, and each branch integrates a CBAM module; the last branch contains only a 1×1 convolutional layer; The detection network includes: 4 CBS modules and 4 variable detection heads.

2. The method for detecting small targets in drone aerial images according to claim 1, characterized in that: The backbone network includes: 5 CBS modules, 4 C3 modules and 1 SPPF module.

3. The method for detecting small targets in drone aerial images according to claim 1, characterized in that: The backbone network of the neck network adopts a cascade of a weighted cross-layer feature pyramid network and PANet.

4. A small target detection system for drone aerial images, characterized in that: The system comprises: Image acquisition module, used to acquire the drone aerial images to be identified in complex weather conditions; A small target detection module is used to input the unmanned aerial image to be identified under complex weather conditions into a small target recognition network to obtain the types of small targets in the unmanned aerial image to be identified under complex weather conditions; the types include: cars and pedestrians; the small target recognition network is obtained by training an improved YOLOv5 network using multiple training unmanned aerial images under complex weather conditions and the corresponding types of small targets, and the improved YOLOv5 network is obtained by improving the YOLOv5 network through an attention mechanism, a weighted cross-layer feature pyramid network and a variable detection head; Improved YOLOv5 network includes: backbone network, neck network and detection network; The backbone network is used to extract features from drone aerial images to be identified in complex weather conditions, and obtain multiple feature maps to be fused; The neck network is used to fuse the feature maps to be fused to obtain multiple fused feature maps; The detection network is used to identify small targets based on the fused feature maps; The neck network includes: 2 first feature extraction structures, 1 second feature extraction structure and 3 third feature extraction structures; The first feature extraction structure includes: a CBS module, an upsampling module, a splicing module and a C3 module; the second feature extraction structure includes: a CBS module, an upsampling module, a splicing module and an RFFEB module; the third feature extraction structure includes: a CBS module, a splicing module and an RFFEB module; The RFFEB module includes: 5 1×1 convolutional layers, 2 3×3 convolutional layers, 1 5×5 convolutional layer, 1 7×7 convolutional layer, 1 3×3 convolutional layer with a dilation rate of 3, 1 3×3 convolutional layer with a dilation rate of 5, and 1 3×3 convolutional layer with a dilation rate of 7; The RFFEB module consists of five parallel branches; the first branch consists of a 1×1 convolutional layer and a 3×3 convolutional layer; the three central branches have dilation rates of 3, 5, and 7, respectively, and each branch integrates a CBAM module; the last branch contains only a 1×1 convolutional layer; The detection network includes: 4 CBS modules and 4 variable detection heads.

5. An electronic device, characterized in that: It includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the method for detecting small targets in drone aerial images according to any one of claims 1 to 3.

6. An electronic device according to claim 5, characterized in that: The memory is a readable storage medium.

Images