A drone target detection model based on YOLOv5 network

Abstract

An embodiment of the present invention provides a UAV target detection model based on a YOLOv5 network, and the model includes: a backbone module, which is used to perform multiple downsampling on an input image to obtain multiple feature maps of different scales; a neck module, which is used to perform multiple upsampling and feature fusion based on the feature maps of different scales obtained by the backbone module, and output multiple detection feature maps of different scales, wherein, compared with the original YOLOv5 network, the output of the neck module adds a new detection feature map that is larger than the maximum-scale detection feature map in the original YOLOv5 network, and reduces a minimum-scale detection feature map; a prediction module, which is used to perform UAV target prediction according to the multiple detection feature maps output by the neck module, and obtain predicted position information and classification results of the target, wherein the classification categories include UAV class and non-UAV class.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and video surveillance, and in particular to a drone target detection model based on a YOLOv5 network. Background Art

[0002] From 2015 to 2020, the overall scale of my country's civilian drone market is expected to grow from 2.4 billion yuan to 36.1 billion yuan, with a compound annual growth rate of 72%. Behind the rapid growth of the drone market, there are also increasing risks of low-altitude security.

[0003] Most of the existing low-altitude security solutions currently use radio detection technology to detect drones, that is, to detect the presence of drones through frequency bands, and then track and detect the target through optoelectronic systems. For drones that maintain silent flight, radio detection technology cannot detect them, and radar detection is usually required. If radar is used to detect silently flying drones, the cost is too high, and its large size limits the deployment environment. At the same time, the installation and use of these technical equipment have strict requirements and cannot meet the requirements of rapid deployment. In addition, the technical equipment cannot meet the real-time end-to-end output and needs to be tracked and detected in conjunction with optoelectronic equipment. These shortcomings cannot meet the requirements of real-time, fast and accurate drone detection.

[0004] However, with the development of artificial intelligence technology, the existing low-altitude security solutions can no longer meet the current growing needs of low-altitude security. The current low-altitude security urgently needs an end-to-end fast and real-time detection solution, but the existing detection equipment is bloated and has harsh deployment conditions. Especially in the complex electromagnetic environment of the city, the detection accuracy is greatly suppressed, and the operation is complicated, and it is impossible to achieve real-time end-to-end output. In addition, when the drone flies at high speed and low altitude, it will cause motion blur of dense targets. When the drone flies at different altitudes, the target scale changes greatly, resulting in inaccurate detection results. These problems have brought a great burden to the optimization of the detection equipment. In particular, the drone flying in the air is generally far away from the detection equipment, which makes the target obtained by the detection equipment too small, and the detection accuracy of the drone of the tiny target is not high. If the detection equipment is combined with artificial intelligence technology and deep learning methods, and optimized based on the target detection algorithm to detect the drone of the tiny target, it will increase the computing power and cost of the equipment.

[0005] Therefore, there is an urgent need for a target detection model that can not only reduce the size of the detection equipment and achieve real-time end-to-end output, but also improve the detection performance of small target drones while ensuring detection accuracy, while also maintaining or reducing the corresponding computational complexity and cost to a certain extent. Summary of the invention

[0006] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a drone target detection model based on the YOLOv5 network.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] According to a first aspect of the present invention, a UAV target detection model based on a YOLOv5 network is provided, the model comprising: a backbone module, used to perform multiple downsampling on an input image to obtain multiple feature maps of different scales; a neck module, used to perform multiple upsampling and feature fusion based on the feature maps of different scales obtained by the backbone module, and output multiple detection feature maps of different scales, wherein, compared with the original YOLOv5 network, the output of the neck module adds a new detection feature map that is larger than the maximum scale detection feature map in the original YOLOv5 network, and reduces a minimum scale detection feature map; a prediction module, used to perform UAV target prediction according to the multiple detection feature maps output by the neck module, and obtain predicted position information and classification results of the target, wherein the classification categories include UAV class and non-UAV class.

[0009] In some embodiments of the present invention, compared to the original YOLOv5, a branch network required to generate a new detection feature map is added to the neck module.

[0010] In some embodiments of the present invention, compared to the original YOLOv5 network, the input data of the first upsampling layer of the neck module of the model does not flow to the path aggregation network, and the detection feature map with the minimum scale generated is reduced.

[0011] In some embodiments of the present invention, the branch network is arranged between the final output end of the feature pyramid network of the neck module and the input end of the path aggregation network; wherein the branch network generates a new detection feature map based on the output of the final output end of the feature pyramid network and the feature map of the corresponding scale output by the first C3 network of the trunk module.

[0012] In some embodiments of the present invention, the branch network sequentially includes a convolution layer, an upsampling layer, a feature fusion layer and a C3 network for processing input data; the branch network performs convolution operations and upsampling on the output of the final output end of the input feature pyramid network through its convolution layer and upsampling layer, and the feature fusion layer obtains the feature map of the corresponding scale output by the first C3 network of the trunk module, and after feature fusion of the upsampled result with the feature map of the corresponding scale, it is input into the C3 network of the branch network to generate a new detection feature map.

[0013] In some embodiments of the present invention, compared with the original YOLOv5 network, a convolutional attention mechanism layer is introduced at the output end of the C3 network of the branch network and the output end of each C3 network of the feature pyramid network; the feature pyramid network is based on feature maps of multiple scales, performs upsampling, feature fusion, and is processed by the C3 network and the convolutional attention mechanism layer multiple times, and outputs multiple fused feature maps of different scales in sequence.

[0014] In some embodiments of the present invention, the path aggregation network includes a first path network, a second path network, and a convolutional attention mechanism layer arranged between the output end of the first path network and the input end of the second path network; the multiple detection feature maps of different scales include: the first detection feature map output by the first path network based on the data of the detection feature map generated by the branch network processed by the convolutional attention mechanism layer and the fused feature map of the corresponding size output by the final output end of the feature pyramid network; the second detection feature map output by the second path network based on the output obtained by inputting the detection feature map output by the first path network into the convolutional attention mechanism layer and the fused feature map of the corresponding size output by the feature pyramid network for the first time.

[0015] In some embodiments of the present invention, the prediction module includes multiple prediction heads, each prediction head is used to predict a detection feature map of a corresponding scale, wherein, compared with the original YOLOv5 network, it includes an additional prediction head for predicting a new detection feature map, and a reduced prediction head for predicting a detection feature map of the minimum scale.

[0016] In some embodiments of the present invention, compared with the original YOLOv5 network, multiple Ghost network layers and convolutional attention mechanism layers are added to the residual branch of the bottleneck structure with a step size of 1 of each C3 network in the trunk module, wherein the residual branch includes a Ghost network layer, a BN layer, a ReLU activation function layer, a convolutional attention mechanism layer, a Ghost network layer and a BN layer in sequence; and the standard convolutional network in the identity mapping branch of the bottleneck structure is replaced with a depth-separable convolutional network.

[0017] In some embodiments of the present invention, compared with the original YOLOv5 network, multiple Ghost network layers, depth-separable convolutional networks and convolutional attention mechanism layers are added to the residual branch of the bottleneck structure with a step size of 2 in the C3 network of the backbone module, wherein the residual branch includes a Ghost network layer, a BN layer, a ReLu activation function layer, a depth-separable convolutional network, a convolutional attention mechanism layer, a Ghost network layer and a BN layer in sequence; and a depth-separable convolutional network is added to the input end of the standard convolutional network in the identity mapping branch of the bottleneck structure.

[0018] In some embodiments of the present invention, the model also includes an image adjustment network, which is a convolutional neural network that can participate in training and is used to adjust the image to an input image of a predetermined scale size, and the output end of the image adjustment network is connected to the input end of the improved YOLOv5 network.

[0019] In some embodiments of the present invention, the image adjustment network includes a residual branch network and an identity mapping branch network, the residual branch network includes a convolution layer, a ReLu activation function layer, a convolution layer, a ReLu activation function layer, a BN layer, a bilinear adjustment network, multiple residual blocks, a convolution layer, a BN layer, a feature fusion layer and a convolution layer in sequence, and the identity mapping branch network includes a bilinear adjustment network.

[0020] According to a second aspect of the present invention, a training method for the drone target detection model of the first aspect of the present invention is provided, comprising training the model multiple times iteratively in the following manner:

[0021] A training set is obtained, wherein the samples in the training set consist of images with targets and label information of the targets in the images, wherein the label information includes location information and classification, and the classification categories include drone class and non-drone class; a drone target detection model is trained using the training set to obtain a prediction result of the target in each sample; a total loss is calculated based on the difference between the prediction results and label information of all samples in this round, and the parameters of the drone target detection model are updated based on the total loss.

[0022] In some embodiments of the present invention, the total loss is calculated according to the CIOU loss function.

[0023] According to a third aspect of the present invention, a drone detection method is provided, comprising: providing a drone target detection model obtained by the training method based on the second aspect of the present invention, detecting the drone target, and obtaining a detection result.

[0024] According to a fourth aspect of the present invention, there is provided an electronic device comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the method according to any one of the second and third aspects of the present invention by executing the executable instructions.

[0025] Compared with the prior art, the advantages of the present invention are:

[0026] 1. The model of the present invention performs multiple downsampling of the input image through the trunk module to obtain multiple feature maps of different scales; the neck module performs multiple upsampling and feature fusion based on the feature maps of different scales, outputs multiple detection feature maps of different scales, and outputs a new detection feature map that is larger than the detection feature map of the largest scale in the original YOLOv5 network, and reduces a detection feature map of the minimum scale; the multiple detection feature maps output by the prediction module are used to predict the target of the drone, and the predicted position information and classification results of the target are obtained. By outputting a larger new detection feature map and detecting it, the model can improve the detection performance of small target drones while ensuring accuracy; at the same time, by reducing the output of a detection feature map of the minimum scale, the corresponding calculation amount and storage cost can be maintained or reduced to a certain extent.

[0027] 2. The present invention introduces CBAM layers in multiple places of the model. The lightweight structure of the CBAM layer can be plug-and-play into the convolutional neural network to extract attention areas, so as to help the model resist misleading feature information and focus on useful target objects, reduce the amount of calculation, and can be trained in an end-to-end manner to achieve simple and effective feature extraction. Adding multiple Ghost network layers and deep separable convolutional networks in the residual branch of the bottleneck structure in the C3 network of the backbone module can reduce the amount of calculation to a certain extent while improving the detection performance of the model.

[0028] 3. The model of the present invention introduces an image adjustment network before the improved YOLOv5 network to replace the traditional adjustment method, so that the image adjustment operation is trained and adjusted together with the improved YOLOv5 network, so that the input image is more suitable for detection after entering the improved YOLOv5 network, thereby improving the detection performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0030] Figure 1 Schematic diagram of a drone target detection model according to an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of the structure of an improved YOLOv5 network according to an embodiment of the present invention;

[0032] Figure 3 A schematic diagram of the structure of a CBAM layer according to an embodiment of the present invention;

[0033] Figure 4 A schematic diagram of the structural principle of an image adjustment network according to an embodiment of the present invention;

[0034] Figure 5A schematic diagram of the structure of a C3 network according to an embodiment of the present invention;

[0035] Figure 6 A schematic diagram of the structure of an improved YOLOv5 network according to another embodiment of the present invention;

[0036] Figure 7 is a schematic diagram of a bottleneck structure with a step size of 1 in the first three C3G networks of a backbone module according to another embodiment of the present invention;

[0037] Figure 8 A schematic diagram of a bottleneck structure with a step length of 2 in the last C3G network of a backbone module according to another embodiment of the present invention;

[0038] Fig. 9 is a target statistical size histogram of a multimodal unmanned aerial vehicle tracking dataset (Anti-UAV) according to an embodiment of the present invention;

[0039] Fig.10 Schematic diagram of an image before and after cropping according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0041] As mentioned in the background technology section, with the development of artificial intelligence technology, the existing low-altitude security solutions can no longer meet the current growing low-altitude security needs. The current low-altitude security requirements urgently require an end-to-end fast and real-time detection solution, but the existing detection equipment is bloated and has harsh deployment conditions. Especially in the complex electromagnetic environment of the city, the detection accuracy is greatly suppressed, and the operation is complicated, and it is impossible to achieve real-time end-to-end output. In addition, when the drone flies at high speed and low altitude, it will cause motion blur of dense targets. When the drone flies at different altitudes, the target scale changes greatly, resulting in inaccurate detection effects, etc., which have brought a great burden to the optimization of the detection equipment. In particular, the drone flying in the air is generally far away from the detection equipment, causing the target obtained by the detection equipment to be too small, and it is impossible to detect the drone of the tiny target. If the detection equipment is combined with artificial intelligence technology and deep learning methods, and optimized based on the target detection algorithm to detect the drone of the tiny target, it will increase the computing power and cost of the equipment.

[0042] In view of the above problems, the present invention provides a drone target detection model based on the YOLOv5 network. Figure 1The model includes an image adjustment network and an improved YOLOv5 network. The image adjustment network adjusts the image to an input image of a predetermined scale based on convolutional neural network learning. The input image is downsampled multiple times through the backbone module of the improved YOLOv5 network to obtain multiple feature maps of different scales. The neck module performs multiple upsampling and feature fusion based on feature maps of different scales, outputs multiple detection feature maps of different scales, and compared with the original YOLOv5 network, the output adds a new detection feature map that is larger than the maximum scale detection feature map in the original YOLOv5 network, and reduces a minimum scale detection feature map. The multiple detection feature maps output by the prediction module are used to predict drone targets, and the predicted position information and classification results of the target are obtained, wherein the classification categories include drones and non-drone classes. By outputting a larger new detection feature map and detecting it, the model can improve the detection performance of small target drones while ensuring accuracy. At the same time, reducing the output of a minimum scale detection feature map can maintain or reduce the corresponding computational complexity and storage cost to a certain extent.

[0043] Furthermore, the model is iteratively trained and evaluated multiple times using the collected training sets to obtain a trained drone target detection model, which is then deployed on embedded devices to perform real-time detection of flying drones, achieving rapid deployment, reducing the size of the detection equipment, and realizing real-time end-to-end output with simple operation.

[0044] Based on the above research, according to one embodiment of the present invention, the drone target detection model is improved based on the YOLOv5 network. Specifically, the model is improved based on the 4.0 version of the YOLOv5 network. Figure 2 , including a trunk module, a neck module and a prediction module, the combination of the three modules is an improved YOLOv5 network. It should be understood that the improvement of the model of the YOLOv5 network-V4.0 version is only an illustrative embodiment. According to the guidance of the present invention, those skilled in the art can also make similar improvements to other versions of the YOLOv5 network to achieve the same or similar effects. In order to better understand the present invention, each module of the model is described in detail below in conjunction with specific embodiments.

[0045] According to one embodiment of the present invention, the backbone module is used to perform multiple downsampling on the input image to obtain multiple feature maps of different scales. Figure 2The main modules include Focus module, convolutional layer, C3 network, convolutional layer, C3 network, convolutional layer, C3 network, convolutional layer, SPP module and C3 network. SPP module is the spatial pyramid pooling module, and C3 network is the concentrated-comprehensive convolution block.

[0046] According to an embodiment of the present invention, for example, an input image of 608*608 scale is input into the backbone module to obtain multiple feature maps of different scales, including: a feature map of 304*304 scale is obtained after downsampling processing by the Focus module of the backbone module; then downsampling operation is performed through the first convolution layer of the backbone module, and then input into the first C3 network, and the size is reduced by half again. Similarly, the feature map passes through the second convolution layer, the second C3 network, the third convolution layer and the third C3 network in sequence, so that before reaching the SPP module, the feature map of 304*304 scale is subjected to corresponding operations through three groups of convolution layers + C3 networks, and corresponding feature maps of 152*152, 76*76 and 38*38 scales are obtained from top to bottom, and then downsampled through the last convolution layer process of the backbone module on the basis of the feature map of 38*38 scale to obtain a feature map of 19*19 scale. Finally, the trunk module processes the 19*19 feature map through the SPP module and the last C3 network, and inputs it to the input end of the neck module.

[0047] Since the captured images always contain geographical and background elements that can mislead the model, the Convolutional Block Attention Module (CBAM) layer can be used to extract attention regions to help the model resist misleading feature information and focus on useful target objects. It can be plugged and played into convolutional neural networks and can be trained in an end-to-end manner. Therefore, the CBAM layer is a simple, effective and lightweight convolutional attention mechanism layer, see Figure 3 , when an input feature map is given, based on the channel attention module and the spatial attention module, the input feature map is sequentially inferred along the two independent dimensions of channel and space to obtain the attention map, and then the attention map is multiplied with the input feature map to perform adaptive feature refinement to obtain the corresponding adaptive refined features. In the present invention, CBAM layers are introduced in multiple places of the model to extract attention areas to help the model focus on useful target objects.

[0048] According to the definition of target size during target detection, the present invention refers to targets with pixel sizes between 0*0 and 16*16 as tiny targets, targets with pixel sizes between 16*16 and 32*32 as small targets, targets with pixel sizes between 32*32 and 96*96 as medium targets, and targets with pixel sizes greater than 96*96 as large targets. Therefore, in order to increase the detection of drones flying farther away, it is necessary to output a new detection feature map that is larger than the maximum scale detection feature map in the original YOLOv5 network to increase the detection of tiny targets. For example, when the input image is 608*608 in scale, the maximum scale detection feature map in the original YOLOv5 network is a feature map of 76*76 in scale, which corresponds to 608 / 76=8, that is, a tiny target of 8*8 pixel size can be detected. When a new detection feature map with a scale larger than 76*76 is output, a tiny target smaller than 8*8 pixel size can be detected. According to one embodiment of the present invention, the neck module is used to perform multiple upsampling and feature fusion based on feature maps of different scales obtained by the trunk module, and output multiple detection feature maps of different scales, wherein, compared with the original YOLOv5 network, the output adds a new detection feature map that is larger than the detection feature map of the largest scale in the original YOLOv5 network, and reduces a detection feature map of the smallest scale, thereby reducing computational and storage costs.

[0049] According to one embodiment of the present invention, see Figure 2 Compared with the original YOLOv5 network, the CBAM layer is introduced at the output end of each C3 network of the feature pyramid network. The structure of the obtained feature pyramid network includes convolution layer, upsampling layer, feature fusion layer, C3 network, CBAM layer, convolution layer, upsampling layer, feature fusion layer, C3 network and CBAM layer in sequence. The feature pyramid network is based on feature maps of multiple scales, performs upsampling, feature fusion multiple times, and is processed by C3 network and convolution attention mechanism layer, and outputs multiple fused feature maps of different scales in sequence. For example, based on the backbone module, the input image of 608*608 is downsampled multiple times to obtain multiple feature maps of different scales. The feature pyramid network processes the 19*19 feature map through the convolution layer and the upsampling layer in sequence and inputs it into the feature fusion layer. The feature fusion layer fuses it with the 38*38 feature map obtained by the backbone module, and then processes it through the C3 network, CBAM layer and convolution layer in sequence to obtain a fused feature map of 38*38 scale. After the fused feature map of 38*38 scale is upsampled through the upsampling layer, the upsampled result is fused with the feature map of 76*76 scale through the feature fusion layer, and then processed through the C3 network and CBAM layer to obtain a fused feature map of 76*76 scale.

[0050] To output a new detection feature map that is larger than the maximum scale detection feature map in the original YOLOv5 network, according to one embodiment of the present invention, see Figure 2 Compared with the original YOLOv5, a branch network required for generating a new detection feature map is added to the neck module. The branch network is set between the final output end of the feature pyramid network of the neck module and the input end of the path aggregation network. The branch network generates a new detection feature map based on the output of the final output end of the feature pyramid network and the feature map of the corresponding scale output by the first C3 network YOLOv5 of the trunk module.

[0051] According to one embodiment of the present invention, the branch network sequentially includes a convolution layer, an upsampling layer, a feature fusion layer, a C3 network and a CBAM layer for processing input data; the branch network performs convolution processing and upsampling on the output of the final output end of the input feature pyramid network through its convolution layer and upsampling layer, and the feature fusion layer obtains the feature map of the corresponding scale output by the first C3 network of the model trunk module, and after feature fusion of the upsampled result with the feature map of the corresponding scale, it is input into the C3 network of the branch network to generate a new detection feature map. For example, if the input image is 608*608 in size, it is processed by the Focus module, convolution layer, and C3 network of the backbone module in sequence, and the first C3 network outputs a 152*152 scale feature map. The output of the final output of the feature pyramid network is a 76*76 scale fusion feature map. The branch network performs convolution processing and upsampling on the input 76*76 scale fusion feature map through its convolution layer and upsampling layer. After the feature fusion layer obtains the 152*152 scale feature map for feature fusion, the C3 network input to the branch network generates a 152*152 scale new detection feature map, and the output channels include 255. A 152*152 scale new detection feature map is generated on each channel, and the final result is represented by 152*152*255. The obtained new detection feature map is input to the prediction module for prediction. At the same time, the new detection feature map is processed by the CBAM layer and input into the path aggregation network.

[0052] According to one embodiment of the present invention, the path aggregation network includes a first path network 1, a second path network 2, and a convolutional attention mechanism layer arranged between the output end of the first path network 1 and the input end of the second path network 2; the multiple detection feature maps of different scales include: the first path network 1 outputs a first detection feature map based on the data of the detection feature map generated by the branch network processed by the convolutional attention mechanism layer and the fused feature map of the corresponding size output by the final output end of the feature pyramid network; the second path network 2 outputs a second detection feature map based on the output obtained by inputting the detection feature map output by the first path network 1 into the convolutional attention mechanism layer and the fused feature map of the corresponding size output by the feature pyramid network for the first time. For details, see Figure 2 , the first path network 1 and the second path network 2 have the same structure, and both include: convolution layer, feature fusion layer and C3 network in sequence. Multiple detection feature maps of different scales include: the first path network 1 outputs a first detection feature map of 76*76 scale based on the data of the new detection feature map of 152*152 scale processed by the CBAM layer and the fused feature map of 76*76 scale output by the feature pyramid network; the second path network 2 outputs a second detection feature map of 38*38 scale based on the output obtained by inputting the detection feature map of 76*76 scale into the convolution attention mechanism layer and the fused feature map of 38*38 scale output by the feature pyramid network for the first time. Similarly, its output channels include 255, and finally 255 first detection feature maps of 76*76 scale and 255 second detection feature maps of 38*38 scale are obtained, which are represented by 76*76*255 and 38*38*255 respectively.

[0053] In order to reduce the output of a minimum-scale detection feature map, according to an embodiment of the present invention, compared with the original YOLOv5 network, the input data of the first upsampling layer of the neck module of the model does not flow to the path aggregation network, and the generation of the minimum-scale detection feature map is reduced, thereby reducing the model calculation amount and storage cost.

[0054] According to one embodiment of the present invention, the prediction module is used to predict drone targets based on multiple detection feature maps output by the neck module to obtain predicted position information and classification results of the target, wherein the classification categories include drone class and non-drone class.

[0055] According to one embodiment of the present invention, see Figure 2The prediction module includes three prediction heads, each of which is used to predict a detection feature map of a corresponding scale. Compared with the original YOLOv5 network, it includes an additional prediction head for predicting a new detection feature map of a scale of 152*152, and a reduced prediction head for predicting a detection feature map of the smallest scale (i.e., a second detection feature map of a scale of 38*38).

[0056] According to one embodiment of the present invention, multiple prediction heads respectively perform drone target prediction on multiple feature maps generated by the neck module to obtain the predicted bounding box of the target in each detection feature map and the category of the target in the predicted bounding box; the confidence that the drone exists in the predicted bounding box is calculated based on the probability that the target in the predicted bounding box is a drone category and the ratio of the intersection area to the union area between the real bounding box and the predicted bounding box of the drone target; the confidences of multiple feature maps are compared, and the predicted bounding box corresponding to the highest confidence and the category of the target in the predicted bounding box are used as the predicted position information and classification result of the target.

[0057] Finally, an image adjustment network (resize model) can be introduced before the improved YOLOv5 network to replace the traditional adjustment method, so that the image adjustment operation can be trained and adjusted together with the improved YOLOv5 network, so that the input image is more suitable for detection after entering the improved YOLOv5 network, thereby improving the detection performance of the drone target detection model. According to one embodiment of the present invention, the model also includes an image adjustment network, and the image adjuster is a convolutional neural network that can participate in training, which is used to adjust the image to an input image of a predetermined scale size, and the output end of the image adjustment network is connected to the input end of the improved YOLOv5 network. Specifically, the image adjustment network can uniformly adjust all images of the original scale size to an input image of a predetermined scale size of 608*608.

[0058] According to one embodiment of the present invention, see Figure 4, the image adjustment network (Resizer Model) includes a residual branch network and an identity mapping branch network, the residual branch network includes a two-dimensional convolution layer, a ReLu activation function layer, a two-dimensional convolution layer, a ReLu activation function layer, a BN layer, a bilinear adjustment network (Bilnear Resizer), r Res residual blocks, a two-dimensional convolution layer, a BN layer, a feature fusion layer and a two-dimensional convolution layer, and the identity mapping branch network includes a bilinear adjustment network (Bilnear Resizer). Among them, all convolution layers in the image adjustment network have 16 kernels of size 3×3. The convolution kernel size of the first and last layers is 7×7, and the parameters introduced by the image adjustment network are very small. In addition, the r Res residual blocks in the image adjustment network constitute a residual network, so that the network structure can maintain good feature extraction capabilities at a deeper level.

[0059] According to another embodiment of the present invention, a CBAM layer is introduced into the C3 network of the backbone module of the original YOLOv5 network. At the same time, in order to reduce the computational complexity of the backbone module and improve the detection capability of the model, a Ghost network layer is also introduced. Figure 5 , one branch of the C3 network includes a convolution layer, and the other branch includes a convolution layer and n bottleneck structures (Bottleneck) in sequence, as well as a feature fusion layer and a convolution layer at the output of the two branches. In addition, in the 4.0 version of the YOLOv5 network, the step length of each bottleneck structure of the first three C3 networks in the backbone module is 1, the step length of each bottleneck structure of the last C3 network in the backbone module is 2, and the step length of each bottleneck structure of the C3 network in the neck module is 2. In the present invention, the C3 network of the backbone network can be improved accordingly, and the C3 network in the neck module can also be improved. The improved C3 network is represented by a C3G network. Among them, see Figure 6 , all C3 networks in the backbone module are improved, and the resulting backbone modules include: Focus module, convolutional layer, C3G network, convolutional layer, C3G network, convolutional layer, C3G network, convolutional layer, SPP module and C3G network, among which the first three C3G networks of the backbone module are improved based on the bottleneck structure of the C3 network with a step size of 1, and the last C3G network is improved based on the bottleneck structure of the C3 network with a step size of 2.

[0060] According to another embodiment of the present invention, another improvement of the C3 network in the backbone module is shown in Figure 7Compared with the original YOLOv5 network, multiple Ghost network layers and convolutional attention mechanism layers are added to the residual branch of the bottleneck structure with a step size of 1 in each C3 network in the backbone module, and the residual branch obtained includes a Ghost network layer, a BN layer, a ReLU activation function layer, a CBAM layer, a Ghost network layer and a BN layer in sequence; and the standard convolutional network in the identity mapping branch of the bottleneck structure is replaced with a depth-separable convolutional network. That is, the bottleneck structure with a step size of 1 in the backbone module of the original YOLOv5 network is improved in this way, and the improved C3 network is obtained as the first three C3G networks of the backbone module of the model.

[0061] According to another embodiment of the present invention, another improvement of the C3 network in the backbone module is shown in FIG. Figure 8 Compared with the original YOLOv5 network, multiple Ghost network layers, deep separable convolutional networks and convolutional attention mechanism layers are added to the residual branch of the bottleneck structure with a step size of 2 in the C3 network of the backbone module, and the residual branch obtained includes Ghost network layer, BN layer, ReLu activation function layer, deep separable convolutional network, CBAM layer, Ghost network layer and BN layer in sequence; and a deep separable convolutional network is added to the input end of the standard convolutional network in the identity mapping branch of the bottleneck structure. That is, the bottleneck structure with a step size of 2 in the backbone module of the original YOLOv5 network is improved in this way, and the improved C3 network is obtained as the last C3G network of the backbone module of the model.

[0062] According to one embodiment of the present invention, a training method for the drone target detection model of the first aspect of the present invention is provided, comprising training the model multiple times iteratively according to the following steps:

[0063] Step S1: Obtain a training set, wherein the samples in the training set are composed of images with targets and label information of the targets in the images, wherein the label information includes location information and classification, and the classification categories include drone and non-drone. The location information includes the real bounding box of the target and the center point position of the real bounding box.

[0064] According to one embodiment of the present invention, a training set generation method includes: determining the target category to be detected, and for this purpose, a crawler script can be written to collect public data sets of corresponding categories on the Internet, which may include data sets of drones flying in the air, flying birds that are easily confused with drones, kites, etc.; using a labeling tool to label each data set to obtain labeling information in XML format, and the labeling information mainly includes coordinate information and category information of the target in the image; using a programming language such as Python to respectively write a labeling information modification script, a labeling position information, and a statistical script to perform data analysis, and obtain the following: Figure 8The target size histogram of the multimodal UAV tracking dataset (Anti-UAV) shown in the figure shows that the horizontal axis represents the area pixel size (Area in pixel), and the vertical axis represents the frequency of occurrence (Frequency of area). It can be found from the figure that small targets account for a large proportion, and the number of medium targets and targets above medium targets is negligible compared with the number of small targets; by using the data enhancement method of cropping the dataset, the targets to be detected are randomly distributed in various positions of the picture, rather than concentrated in the center of the picture, as shown in Figure 1. Fig. 9 As shown in the figure, the target before cropping is distributed in the center of the picture, and the target after cropping is distributed on the right side of the picture. All annotation files are converted from XML format to TXT format and the coordinates are normalized. The completed TXT text contains the target category, upper left corner coordinates and lower right corner coordinates (the real bounding box information and center point coordinate information can be obtained based on the two coordinates). 90% of the data is used as a training set and 10% of the data is used as a test set to complete all preprocessing before the image is input into the drone target detection model.

[0065] Step S2: Use the training set to train the drone target detection model to obtain the prediction result of the target in each sample. The prediction result of the target includes the predicted bounding box of the target and the center point position of the predicted bounding box.

[0066] Step S3: Calculate the total loss based on the difference between the prediction results and label information of all samples in this round, and update the parameters of the drone target detection model based on the total loss.

[0067] According to one embodiment of the present invention, the total loss is calculated according to the CIOU loss function in the following manner:

[0068]

[0069] Among them, IOU is the ratio of the intersection area of ​​the predicted bounding box and the true bounding box to the area of ​​the union, ρ 2 (b,b gt ) represent the Euclidean distance between the center point of the predicted bounding box and the true bounding box, b represents the center point of the predicted bounding box, and b gt represents the center point of the true bounding box, C represents the diagonal distance of the minimum closure area that can contain both the predicted bounding box and the true bounding box. v is a parameter used to measure the consistency of aspect ratio, w gt represents the width of the true bounding box, h gt represents the height of the ground-truth bounding box, w p represents the width of the predicted bounding box, h pIndicates the height of the predicted bounding box. This loss function takes into account the aspect ratio of the predicted bounding box when calculating, effectively avoiding the problem that when the predicted bounding box is inside the real bounding box and the predicted bounding boxes are of the same size, it cannot distinguish the positional relationship of each predicted bounding box. Finally, according to the total loss CIOU_Loss, the parameters of the drone target detection model are updated.

[0070] According to one embodiment of the present invention, after the training is completed, an intermediate model file (drone target detection model file) is obtained, and then the intermediate model file (drone target detection model file) is tested and the model is evaluated using a test set. The evaluation indicators can comprehensively consider the mainstream MAP, Precision, Recall, Flops and other indicators in the deep learning model evaluation. It is judged whether each indicator meets the requirements of the technical specifications. If not, the training set and test set can be re-established by optimizing sample annotation, image data enhancement processing, etc. Then, through repeated iterative training and evaluation, the optimal basic model file and the corresponding training set are obtained.

[0071] According to one embodiment of the present invention, the intermediate model file (UAV target detection model file) is tested and the model is evaluated using a test set. Specifically, the UAV target detection model obtained after the training is deployed in an embedded device for testing and evaluation. For example, a data stream analysis toolkit (Deepstream) is installed on a development board (Jetson nano). The toolkit contains hardware acceleration building blocks that can bring deep neural networks and other complex processing tasks into the stream processing pipeline. At the same time, a file (AntiUAV-YOLOv5.wts) is generated on the server side using the neural network reasoning library (TensorRT), and computational reasoning acceleration can be achieved. The corresponding Engine file is then generated, and Deepstream deploys the Engine file to complete the model deployment. Finally, the deployed UAV target detection model is used to test and evaluate the test set.

[0072] According to one embodiment of the present invention, a drone detection method is provided, comprising: based on the drone target detection model obtained by the above-mentioned training method of the present invention, detecting the drone target and obtaining a detection result. Specifically, the trained drone target detection model can be deployed in an embedded device for real-time detection of flying drones.

[0073] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0074] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0075] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0076] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A UAV target detection system based on YOLOv5 network, characterized in that: The system comprises: The backbone module is used to downsample the input image multiple times to obtain feature maps of multiple scales; The neck module is used to perform multiple upsampling and feature fusion based on the feature maps of different scales obtained by the trunk module, and output multiple detection feature maps of different scales. Compared with the original YOLOv5 network, the output of the neck module adds a new detection feature map that is larger than the maximum scale detection feature map in the original YOLOv5 network, and reduces a minimum scale detection feature map; A prediction module is used to predict the drone target based on the multiple detection feature maps output by the neck module, and obtain the predicted position information and classification results of the target, wherein the classification categories include drone class and non-drone class; Among them, a branch network required for generating a new detection feature map is added to the neck module; the branch network is arranged between the final output end of the feature pyramid network of the neck module and the input end of the path aggregation network, and sequentially includes a convolution layer, an upsampling layer, a feature fusion layer, a C3 network and a CBAM layer for processing input data; The branch network performs convolution processing and upsampling on the output of the final output end of the input feature pyramid network through its convolution layer and upsampling layer. The feature fusion layer obtains the feature map of the corresponding scale output by the first C3 network of the model trunk module, and performs feature fusion on the up-sampled result and the feature map of the corresponding scale, and then inputs it into the C3 network of the branch network to generate a new detection feature map; the obtained new detection feature map is input into the prediction module for prediction, and at the same time, the new detection feature map is processed by the CBAM layer and input into the path aggregation network; The path aggregation network includes a first path network, a second path network, and a convolutional attention mechanism layer arranged between an output end of the first path network and an input end of the second path network; The detection feature maps of multiple different scales include: The first path network outputs a first detection feature map based on the data processed by the convolutional attention mechanism layer on the detection feature map generated by the branch network and the fused feature map of the corresponding size output by the final output end of the feature pyramid network; The second path network outputs a second detection feature map based on the output obtained by inputting the detection feature map output by the first path network into the convolutional attention mechanism layer and the fused feature map of the corresponding size first output by the feature pyramid network.

2. The system according to claim 1, characterized in that Compared with the original YOLOv5 network, the input data of the first upsampling layer of the neck module of this system does not flow to the path aggregation network, and reduces the detection feature map with the minimum scale generated.

3. The system according to claim 2, characterized in that The branch network generates a new detection feature map based on the output of the final output end of the feature pyramid network and the feature map of the corresponding scale output by the first C3 network of the trunk module.

4. The system according to claim 1, characterized in that Compared with the original YOLOv5 network, a convolutional attention mechanism layer is introduced at the output end of the C3 network of the branch network and the output end of each C3 network of the feature pyramid network; The feature pyramid network is based on feature maps of multiple scales, performs upsampling and feature fusion multiple times, and processes through a C3 network and a convolutional attention mechanism layer, and sequentially outputs multiple fused feature maps of different scales.

5. The system according to claim 1, characterized in that The prediction module includes multiple prediction heads, each prediction head is used to predict a detection feature map of a corresponding scale, wherein, compared with the original YOLOv5 network, it includes an additional prediction head for predicting a new detection feature map, and a reduced prediction head for predicting a detection feature map of a minimum scale.

6. The system according to claim 1, characterized in that Compared with the original YOLOv5 network, multiple Ghost network layers and convolutional attention mechanism layers are added to the residual branch of the bottleneck structure with a stride of 1 in each C3 network in the backbone module, where the residual branch includes a Ghost network layer, a BN layer, a ReLU activation function layer, a convolutional attention mechanism layer, a Ghost network layer, and a BN layer in sequence; and The standard convolutional network in the identity mapping branch of the bottleneck structure is replaced by a depthwise separable convolutional network.

7. The system according to claim 1, characterized in that Compared with the original YOLOv5 network, multiple Ghost network layers, depth-separable convolutional networks and convolutional attention mechanism layers are added to the residual branch of the bottleneck structure with a step size of 2 in the C3 network of the backbone module, wherein the residual branch includes a Ghost network layer, a BN layer, a ReLu activation function layer, a depth-separable convolutional network, a convolutional attention mechanism layer, a Ghost network layer and a BN layer in sequence; And adding a depthwise separable convolutional network to the input of the standard convolutional network in the identity mapping branch of the bottleneck structure.

8. The system according to any one of claims 1 to 7, characterized in that: It also includes an image adjustment network, which is a convolutional neural network that can participate in training and is used to adjust the image to an input image of a predetermined scale size. The output end of the image adjustment network is connected to the input end of the improved YOLOv5 network.

9. The system according to claim 8, characterized in that The image adjustment network includes a residual branch network and an identity mapping branch network, the residual branch network includes a convolution layer, a ReLu activation function layer, a convolution layer, a ReLu activation function layer, a BN layer, a bilinear adjustment network, multiple residual blocks, a convolution layer, a BN layer, a feature fusion layer and a convolution layer in sequence, and the identity mapping branch network includes a bilinear adjustment network.

10. A training method for a drone target detection system according to any one of claims 1 to 9, characterized in that: This involves training the system multiple times in the following ways: A training set is obtained, wherein the samples in the training set are composed of images with targets and label information of the targets in the images, wherein the label information includes location information and classification, and the classification categories include drone class and non-drone class; Use the training set to train the drone target detection system and obtain the prediction results of the target in each sample; According to the difference between the prediction results and label information of all samples in this round, the total loss is calculated, and the parameters of the drone target detection system are updated based on the total loss.

11. The method according to claim 10, characterized in that The total loss is calculated according to the CIOU loss function.

12. A method for detecting a drone, characterized in that: include: The UAV target detection system obtained based on the training method described in claim 10 detects the UAV target and obtains the detection result.

13. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 10 to 12.

14. An electronic device, characterized in that: include: one or more processors; as well as A memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 10 to 12 by executing the executable instructions.

Images