A method for identifying small targets of drones based on improved YOLOv4 network

Abstract

The present invention proposes a method for identifying small targets of unmanned aerial vehicles based on an improved YOLOv4 network, comprising the following steps: S1, preprocessing the collected graphics to generate a training data set and a test data set; S2, constructing a small target recognition model, the small target recognition model input is the training data and the test data, and the target recognition model output is a target recognition result; the training data is the data in the training data set, and the test data is the data in the test data set; S3, inputting the training data to train the small target recognition model; S4, inputting the test data to test the small target recognition model. The present invention can accurately identify small moving target objects, and improves the detection efficiency while ensuring the detection accuracy.

Description

Technical Field

[0001] The present invention relates to the field of target recognition, and in particular to a method for recognizing small targets of unmanned aerial vehicles based on an improved YOLOv4 network. Background Art

[0002] With the rapid development and widespread application of drones, drones have rapidly shifted from military to civilian use in recent years, and various aerial photography drones have come into the public eye. Due to the high degree of freedom, flexible operation and wide range of activities of drones, if they are not regulated, they will cause great harm to social security. In response to this, relevant departments have regulated important areas such as airports and railways. In addition, the technology of open source drones is now very mature, and drones can be assembled freely. Such drones often pose a greater threat to social security. Since self-assembled drones do not require user identity binding, illegal activities using drones have also occurred repeatedly. Since the abuse of drones will cause a large number of hidden dangers, research on anti-drones has also begun to rise, but since drones are small targets, have high degrees of freedom in flight and are fast, they are difficult to detect at the first time.

[0003] Drones present complex motion trajectories in the air. Due to the uncertainty of flight speed, scale changes, and complex backgrounds, it is difficult to monitor drones in real time in static scenes. Drone detection in dynamic scenes is still a very complex and challenging task. At present, neural networks have excellent performance in target detection. However, since drones can present multi-scale changes in images, small targets are relatively small in size in the original image. In the general target detection neural network model, the general basic backbone network has several downsampling processes, resulting in the size of small targets in the feature map is basically only a single-digit pixel size, resulting in poor classification effect of the designed target detection classifier for small targets. Summary of the invention

[0004] The present invention aims to at least solve the technical problems existing in the prior art, and in particular innovatively proposes a method for identifying small targets of unmanned aerial vehicles based on an improved YOLOv4 network.

[0005] In order to achieve the above-mentioned object of the present invention, the present invention provides a method for identifying small targets of unmanned aerial vehicles based on an improved YOLOv4 network, comprising the following steps:

[0006] S1, preprocessing the collected graphics to generate training data sets and test data sets;

[0007] S2, constructing a small target recognition model, wherein the small target recognition model input is the training data and the test data, and the target recognition model output is the target recognition result; the training data is the data in the training data set, and the test data is the data in the test data set;

[0008] S3, inputting the training data to train the small object recognition model;

[0009] S4, inputting the test data to test the small object recognition model.

[0010] Furthermore, the S1 includes:

[0011] S1-1, collecting drone images;

[0012] S1-2, use a labeling tool to label the drone location information of each image to obtain a labeling file;

[0013] S1-3, convert the format of the annotation file to obtain the data set; convert the LabelImg annotation .xml file into a .txt annotation file that can be used by YOLO;

[0014] S1-4, divide the data set into training data set and test data set.

[0015] Furthermore, the S1 further includes:

[0016] The image is first processed using an image processing algorithm to obtain a grayscale image containing edge features, wherein the image processing algorithm includes one or any combination of grayscale, binarization, corrosion, expansion, denoising, and edge detection;

[0017] Then, the channel image of the grayscale image containing edge features is fused with the original three-channel image to obtain a four-channel image containing edge features as an input of the small target recognition model.

[0018] Existing neural networks mainly extract semantic information and rarely use information about the target itself, such as edge features. The present invention combines edge information to facilitate target recognition.

[0019] Furthermore, the small target recognition model includes:

[0020] The backbone network, the feature extraction network, and the detection head network, the data output end of the backbone network is connected to the data input end of the feature extraction network, and the data output end of the feature extraction network is connected to the data input end of the detection head network.

[0021] The backbone network uses 2 CSP structures for feature extraction.

[0022] The CSP structure consists of 4 CBLs, a Res unit, and a concat layer, where the concat layer fuses the feature maps output by the first CBL and the third CBL;

[0023] The Res unit includes a depth-separable convolutional layer DW;

[0024] And / or using pruning technology to remove the number of layers of the feature fusion network in the feature extraction network so that the feature fusion network has 16 layers to reduce the amount of network calculation;

[0025] The feature fusion network includes: upsampling the output feature map of the backbone network and the feature map output by the feature extraction network and then performing feature fusion, thereby improving the utilization of shallow feature information;

[0026] The detection head network includes: using the output feature map resolution of Input image resolution detection head to improve the accuracy of small object detection.

[0027] The ratio of the feature map resolution to the input image resolution is n=1, 2, ..., N. The resolution of the input image in the specific embodiment of the present invention is 608×608, so the output feature map resolution is increased from the original 76×76 detection head to the 152×152 detection head.

[0028] Furthermore, the feature fusion network includes:

[0029] After pruning the YOLO v4 network, the output feature map of the nth layer backbone network and the feature map of the mth layer feature extraction network are downsampled and then feature fused. The specific fusion formula is as follows:

[0030]

[0031] L OUT It is the feature map output after concat feature fusion;

[0032] ⊙ represents the fusion symbol;

[0033] It refers to the i-dimensional feature map output by the n-th convolutional layer in the network;

[0034] The network is an improved YOLO v4 network, i.e., a small target recognition model.

[0035] It refers to the j-dimensional feature map output by the m-th convolutional layer in the network.

[0036] Furthermore, the small target recognition model also includes:

[0037] Use depth-separable convolution to replace the standard convolution operation in YOLO v4 to reduce the amount of convolution calculation and improve detection speed.

[0038] The depth-wise separable convolution includes channel-by-channel convolution and point-by-point convolution;

[0039] The channel-by-channel convolution means that each convolution kernel is only responsible for a single channel and is mainly responsible for filtering;

[0040] The point-by-point convolution performs a weighted combination of the feature map of the previous layer in the depth direction to generate a new feature map;

[0041] Compared with standard convolution, the depthwise separable convolution reduces the amount of computation of the network model during convolution operations. The specific calculation method is as follows:

[0042] Bflops dw =(W×H×C)×(F w ×F h )+C×F n

[0043] Where Bflops dw Represents the amount of computation of depth-wise separable convolution;

[0044] W represents the length of the feature map;

[0045] H represents the width of the feature map;

[0046] C represents the number of channels of the feature map;

[0047] F w Represents the length of the convolution kernel;

[0048] F h Indicates the width of the convolution kernel;

[0049] F n Indicates the number of convolution kernels.

[0050] Furthermore, the small target recognition model includes a loss function, and the loss function includes:

[0051] Loss=Loss cood +Loss conf +Loss cls

[0052] Among them, Loss cood represents coordinate loss;

[0053] Loss conf represents the confidence loss;

[0054] Loss cls represents the classification loss.

[0055] Furthermore, the coordinate loss includes:

[0056] Loss cood =1-IOU(A,B)+ρ 2(A ctr ,B ctr ) / c 2 +αυ

[0057]

[0058]

[0059] Where IOU(A,B) is the intersection-over-union ratio of A and B.

[0060] A is the annotation box;

[0061] B is the prediction box of the network;

[0062] ρ(A ctr ,B ctr ) represents the Euclidean distance between two points;

[0063] A ctr ,B ctr are the center point coordinates of A and B respectively;

[0064] c represents the diagonal distance between the minimum enclosing rectangle of the prediction box and the annotation box;

[0065] α represents the trade-off parameter;

[0066] υ represents the aspect ratio consistency parameter;

[0067] w gt 、h gt are the width and height of the annotation box respectively;

[0068] w and h are the width and height of the prediction box respectively.

[0069] In summary, due to the adoption of the above technical solution, the present invention can accurately identify small moving target objects and improve the detection efficiency while ensuring the detection accuracy.

[0070] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0072] Figure 1 It is a flow chart of the present invention.

[0073] Figure 2 It is a schematic diagram of input layer optimization of the present invention.

[0074] Figure 3It is a schematic diagram of the network structure design of the present invention.

[0075] Figure 4 It is a schematic diagram of the standard convolution and depth-wise separable convolution of the present invention.

[0076] Figure 5 It is a schematic diagram of data set preparation of the present invention.

[0077] Figure 6 It is a schematic diagram of the loss function curve of the present invention.

[0078] Figure 7 It is a schematic diagram of the recognition result of the present invention.

[0079] Figure 8 It is a schematic diagram comparing the CSP structure and Res unit structure of the present invention with those of YOLO v4. DETAILED DESCRIPTION

[0080] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0081] Figure 1 The overall flow chart of the method of the present invention mainly includes: data set preparation; network design and optimization; model training; target recognition.

[0082] Figure 2 To optimize the input layer, the original YOLO network input is a 3-channel RGB image. As the number of network layers increases, the network mainly extracts the semantic information of the image. In order to improve the network's use of shallow image features, the network input layer is optimized. The core idea is to use the original RGB three-channel image (such as Figure 2 (b) Perform an edge detection and convert the image after edge detection (such as Figure 2 (c) The feature map is fused with the original RGB image to form a four-channel image, and the four-channel image is used as data input.

[0083] The specific fusion process is: first divide the RGB channel of the image into three single-channel images. Since the single-channel image is a two-dimensional matrix, a container can be used to store these three two-dimensional matrices in sequence. Since the grayscale image is also a single-channel image, the grayscale image containing edge features is stored in the above container to form a four-channel image.

[0084] Figure 3The diagram is a schematic diagram of the optimized network structure. Compared with the original YOLO v4 network structure, the improved network model is reduced from 161 layers to 40 layers, and the backbone network is reduced from 53 layers to 12 layers, the feature fusion network is reduced from 105 layers to 16 layers, and the detection heads are changed from 3 (3 layers) to 2 (2 layers).

[0085] The reason why the feature fusion network has 16 layers is that, through experiments, it was found that the overall structure of 16 layers is the most streamlined number of network layers, and 16 layers have the best effect.

[0086] Feature fusion network: low-level features have high resolution and contain more location and detail information, while high-level features have stronger semantic information, but low resolution and poor perception of details. In order to improve the recognition ability of small targets, the present invention adopts an FPN feature pyramid structure in the feature extraction network, and the output feature map of the backbone network is subjected to a Resunit structure and a CSP structure to extract features at a deeper level and then input into the large target detection head Y1. At the same time, the shallow feature map output by the backbone network and the deep feature map obtained by the Res unit structure and the CSP structure are upsampled and then feature fused, that is, the two feature maps are connected through the concat layer, and the output feature map is sent to the small target detection head Y2.

[0087] Feature fusion: Using the skip connection method, the i-dimensional feature map output by the 9th convolutional layer and the j-dimensional feature map output by the 29th convolutional layer are downsampled, and then fused through the route layer to directly connect the two feature maps to form an i+j-dimensional feature map.

[0088] The SPP structure (spatial pyramid pooling) in the original YOLO v4 model is deleted in the improved model, but the feature pyramid structure is retained to detect objects of different scales. The main core idea is to fuse the shallow feature information with the deep feature information to make better use of the shallow features. The implementation process is as follows: In order to improve the utilization of the shallow network detail information, the feature layering method is used to use each layer of features to predict the detection results of the corresponding scale resolution. After the YOLO v4 network is pruned, the output feature map of the backbone network and the feature map output by the feature extraction network are downsampled for feature fusion as the input of the small target detection head. The pruning technology is used to remove the convolutional layers that have little effect on the output results. The pruning of the model is actually an iterative process. The convolutional layers of the model are continuously removed in the iteration, and the removed model is trained. The influence of the parameters of this convolutional layer on the model is evaluated based on the training results.

[0089]

[0090] LOUT It is the feature map output after concat feature fusion, ⊙ represents the fusion symbol, It refers to the i-dimensional feature map output by the n-th convolutional layer in the network. It refers to the j-dimensional feature map output by the m-th convolutional layer in the network. Therefore, after concat feature fusion, L OUT It is an i+j dimensional feature map.

[0091] In order to effectively improve the detection capability of small targets, the detection head is enlarged from 76*76 to 152*152, and the accuracy of small target detection is increased by increasing the calculation amount of the detection head. In order to better optimize the model, the loss function is changed from coordinate loss to loss. cood , confidence loss Loss conf And classification loss Loss cls It consists of three parts.

[0092] Loss=Loss cood +Loss conf +Loss cls

[0093] The coordinate loss Loss cood It is the core of the loss function. The coordinate loss adopts the CIOU loss function. The CIOU loss function takes into account the overlapping area, center point distance and aspect ratio of the predicted box and the real box to improve the convergence speed and regression accuracy.

[0094] Loss cood =1-IOU(A,B)+ρ 2 (A ctr ,B ctr ) / c 2 +αυ

[0095]

[0096]

[0097] Where A is the annotation box and B is the prediction box of the network. IOU(A,B) is the intersection-over-union ratio of A and B. ctr ,B ctr are the center point coordinates of A and B respectively, ρ(A ctr ,B ctr ) represents the Euclidean distance between two points, c represents the diagonal distance between the minimum bounding rectangle of the prediction box and the annotation box; α represents the trade-off parameter, υ represents the aspect ratio consistency parameter, and w gt 、h gt, w, h are the width and height of the annotation box and the prediction box respectively. α and υ are penalty items of loss. When the aspect ratio of the prediction box and the true box is close, υ approaches 0 and the penalty item becomes invalid. It can be seen that the penalty item is to accelerate the convergence of the network.

[0098] Figure 4 It is an optimization of the convolution method. Its main contents involve: replacing standard convolution with depthwise separable convolution. Depthwise separable convolution mainly consists of two parts: channel-by-channel convolution and point-by-point convolution. Channel-by-channel convolution means that each convolution kernel is only responsible for a single channel and is mainly responsible for filtering; point-by-point convolution is a weighted combination of the feature map of the previous layer in the depth direction to generate a new feature map. Compared with standard convolution, depthwise separable convolution reduces the amount of calculation of the network model during convolution operations. The specific calculation method is as follows:

[0099] The computational volume of standard convolution is Bflops conv for:

[0100] Bflops conv =(W×H×C)×(F w ×F h )×F n

[0101] Where W, H, C are the length, width and number of channels of the feature map, and F w 、F h is the length and width of the convolution kernel, F n is the number of convolution kernels.

[0102] When outputting feature maps with the same number of channels, the computational cost of depthwise separable convolution is Bflops dw for:

[0103] Bflops dw =(W×H×C)×(F w ×F h )+C×F n

[0104] It can be seen that the parameter calculation amount of depth-separable convolution is λ times that of standard convolution:

[0105]

[0106] When the concatenation of the convolution results is not considered, the computational complexity of the depthwise separable convolution is twice that of the standard convolution. It can be seen that as the number of convolution kernels increases, depthwise separable convolution can significantly reduce the computational complexity of the convolution operation.

[0107] Figure 5This is a dataset of drone images. Since there is no public drone dataset, a large number of drone images need to be collected and annotated manually. The dataset includes 9287 images containing drone information. Some of the images are as follows: Figure 5 As shown in the figure, the data set production process is as follows:

[0108] ① Collect drone images;

[0109] ② Use the LabelImg annotation tool to label the drone location information of each image. Figure 5 As shown in b;

[0110] ③Convert the LabelImg annotation .xml file into a .txt annotation file that can be used by YOLO;

[0111] ④ Divide the data set into train and test.

[0112] Figure 6 is the network model loss function curve, which can be used to understand whether the network model can converge robustly with the increase in the number of iterations, and is an important basis for determining the model convergence effect. Figure 6 It can be seen that the improved network model converges quickly, and the loss value at the 4000th iteration is 0.25, which shows a good convergence effect.

[0113] Figure 7 In order to verify the recognition effect, the improved network model was used in the drone data set for experimental verification. The 9287 drone images were randomly distributed in the ratio of 7232 training sets and 2055 test sets. During the training process, the number of iterations was set to 4000. When the maximum number of iterations was reached, the network model tended to be stable, and the detection effect of the network model on drones was as follows: Figure 7As shown, it can be seen that the model has a good recognition effect on drones of all scales. Table 2 is the detection results of four network models, among which YOLO v4 and YOLO v4-tiny networks are currently public network models; YOLO v4-UAV is an improved network model of the present invention, and YOLO v4-UAV_D is a network model after replacing the standard convolution with a depth-separable convolution on the basis of YOLO v4-UAV. In order to better judge the performance of these models when applied to drone detection, the present invention selects the currently commonly used image evaluation standards, which are: mean average precision m AP (mean Average Precision) and BFLOPS of the network model. mAP is an important indicator for measuring the quality of the target detection model. The higher the value of m AP, the higher the average accuracy of the model for object detection and the better the performance of the model. BFLOPS of the network model refers to the total amount of floating-point operations performed by the network each time (unit: billion floating-point operations). The smaller the total amount of floating-point operations, the more pictures the network processes per second. Therefore, the present invention comprehensively evaluates the advantages and disadvantages of the model from the two directions of model accuracy and real-time performance.

[0114] From the experimental results, we can see that the mAP of the YOLO v4-UAV model can reach 82.5%, which is a significant improvement compared to the YOLO v4-tiny model. In terms of detection speed, the floating-point operations of the improved YOLO v4-UAV model are 13.941, which is twice the floating-point operations of the YOLO v4-tiny, but it has been greatly improved compared to the YOLO v4 model, and the accuracy has not decreased significantly. After adding the depthwise separable convolution layer, although the mAP value obtained is always reduced by about 1%, the floating-point operations of adding the depthwise separable convolution have dropped from 13.941 to 5.485, and the amount of operations has been greatly reduced. It can be seen that the depthwise separable convolution can significantly reduce the number of network parameters and improve the detection speed.

[0115] The experimental environment is shown in Table 1:

[0116] Table 1 Experimental environment configuration

[0117] Configuration parameter Environmental Framework Darknet CPU Intel I7 10770K Memory 32G Video Memory 24G GPU NVIDIA 3090 operating system Ubuntu 18.04LTS Development Language C / C++ Using Modules CUDA11

[0118] The experimental results are shown in Table 2:

[0119] Table 2 Experimental results

[0120] Network Model mAP(%) BFLOPS YOLO v4 98.51 127.232 YOLO v4-tiny 68.16 6.787 YOLO v4-UAV 82.50 13.941 YOLO v4-UAV_D 81.87 5.485

[0121] Compared with the original YOLO v4, the CSP structure of the present invention removes a standard convolution layer CBL to reduce the amount of calculation, and adds a depth-separable convolution layer DW in the res unit. The specific structure is as follows: Figure 8 shown.

[0122] The CSP structure in the improved network of YOLO v4 is composed of 4 CBLs, one Res unit and one concat. The concat layer fuses the feature maps output by the first CBL and the third CBL. This connection method can make the network training easier to converge. In order to make the network more streamlined, compared with the CSP structure of YOLO v4, the CSP structure of the present invention removes one CBL and adds a depth-separable convolution layer in the Res unit, so that the computational complexity of a part of the network can be reduced while ensuring the number of convolutional layers.

[0123] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for identifying small targets of unmanned aerial vehicles based on an improved YOLOv4 network, characterized in that: The following steps are involved: S1, preprocessing the collected graphics to generate training data sets and test data sets; S2, constructing a small target recognition model, wherein the input of the small target recognition model is the training data and the test data, and the output of the small target recognition model is a target recognition result; the training data is the data in the training data set, and the test data is the data in the test data set; S3, inputting the training data to train the small object recognition model; S4, inputting the test data to test the small target recognition model; The small target recognition model includes: The backbone network, the feature extraction network, and the detection head network, the data output end of the backbone network is connected to the data input end of the feature extraction network, and the data output end of the feature extraction network is connected to the data input end of the detection head network. The backbone network uses 2 CSP structures for feature extraction. The CSP structure consists of 4 CBLs, a Res unit, and a concat layer, where the concat layer fuses the feature maps output by the first CBL and the third CBL; The Res unit includes a depth-separable convolutional layer DW; and using pruning technology to remove the number of layers of the feature fusion network in the feature extraction network so that the feature fusion network has 16 layers; The feature fusion network comprises: upsampling the output feature map of the backbone network and the feature map output by the feature extraction network and then performing feature fusion; The detection head network includes: using the output feature map resolution of Input image resolution to the detection head.

2. According to claim 1, a method for identifying small targets of unmanned aerial vehicles based on an improved YOLOv4 network is characterized in that: The S1 includes: S1-1, collecting drone images; S1-2, use a labeling tool to label the drone location information of each image to obtain a labeling file; S1-3, convert the format of the annotation file to obtain a data set; S1-4, divide the data set into training data set and test data set.

3. The method for identifying small targets of unmanned aerial vehicles based on an improved YOLOv4 network according to claim 1, characterized in that: The S1 further comprises: The image is first processed using an image processing algorithm to obtain a grayscale image containing edge features, wherein the image processing algorithm includes one or any combination of grayscale, binarization, corrosion, expansion, denoising, and edge detection; Then, the channel image of the grayscale image containing edge features is fused with the original three-channel image to obtain a four-channel image containing edge features as an input of the small target recognition model.

4. The method for identifying small targets of unmanned aerial vehicles based on an improved YOLOv4 network according to claim 1, characterized in that: The feature fusion network includes: After pruning the YOLO v4 network, the output feature map of the nth layer backbone network and the feature map of the mth layer feature extraction network are downsampled and then feature fused. The specific fusion formula is as follows: L OUT It is the feature map output after concat feature fusion; ⊙ represents the fusion symbol; It refers to the i-dimensional feature map output by the n-th convolutional layer in the network; It refers to the j-dimensional feature map output by the m-th convolutional layer in the network.

5. The method for identifying small targets of unmanned aerial vehicles based on an improved YOLOv4 network according to claim 1, characterized in that: The small target recognition model also includes: Use depth-wise separable convolution to replace the standard convolution operation in YOLO v4. The depth-wise separable convolution includes channel-by-channel convolution and point-by-point convolution; The channel-by-channel convolution means that each convolution kernel is only responsible for a single channel and is mainly responsible for filtering; The point-by-point convolution performs a weighted combination of the feature map of the previous layer in the depth direction to generate a new feature map; The specific calculation method of the depth-wise separable convolution is as follows: Bflops dw =(W×H×C)×(F w ×F h )+C×F n Where Bflops dw Represents the amount of computation of depth-wise separable convolution; W represents the length of the feature map; H represents the width of the feature map; C represents the number of channels of the feature map; F w Represents the length of the convolution kernel; F h Indicates the width of the convolution kernel; F n Indicates the number of convolution kernels.

6. The method for identifying small targets of unmanned aerial vehicles based on an improved YOLOv4 network according to claim 1, characterized in that: The small target recognition model includes a loss function, and the loss function includes: Loss=Loss cood +Loss conf +Loss cls Among them, Loss cood represents coordinate loss; Loss conf represents the confidence loss; Loss cls represents the classification loss.

7. The method for identifying small targets of unmanned aerial vehicles based on an improved YOLOv4 network according to claim 6, characterized in that: The coordinate loss includes: Loss cood =1-IOU(A,B)+ρ 2 (A ctr ,B ctr ) / c 2 +au Where IOU(A,B) is the intersection-over-union ratio of A and B. A is the annotation box; B is the prediction box of the network; ρ(A ctr ,B ctr ) represents the Euclidean distance between two points; A ctr ,B ctr are the center point coordinates of A and B respectively; c represents the diagonal distance between the minimum enclosing rectangle of the prediction box and the annotation box; α represents the trade-off parameter; υ represents the aspect ratio consistency parameter; w gt 、h gt are the width and height of the annotation box respectively; w and h are the width and height of the prediction box respectively.

Images