Target detection method, system and device based on unmanned aerial vehicle and nvidia development board

Abstract

The application discloses a target detection method, system and device based on a UAV and a NVIDIA development board, and the method comprises the following steps: constructing an improved YOLO v4-tiny model; the improved YOLO v4-tiny model comprises an initial convolution module, a first hourglass type pooling module, a second hourglass type pooling module, a third hourglass type pooling module, a first pooling feature enhancement module, a second pooling feature enhancement module, a splicing module, a first output module and a second output module; training the improved YOLO v4-tiny model by using a training set to obtain a final target detection model; acquiring a video stream collected by an on-board camera of the UAV by using a video acquisition card; inputting the video stream into the final target detection model deployed on the NVIDIA development board to perform target detection, and outputting a target detection result. The improved YOLO v4-tiny model realizes real-time detection of the UAV video stream, and the ground equipment solves the problem that the computing capacity of the target detection network is limited.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image target detection, and in particular to a target detection method, system and device based on a UAV and a NVIDIA development board. BACKGROUND

[0002] Target detection based on deep learning is an important research direction in the field of image processing, mainly including two directions, two-stage target detection and single-stage target detection.

[0003] Two-stage target detection has better accuracy and is suitable for scenes with high detection precision. However, its high accuracy is at the cost of large amount of calculation. Two-stage target detection networks usually have large network parameters and are more suitable for deployment on high-performance computers, and are not suitable for the present application.

[0004] Single-stage target detection sacrifices part of the accuracy to achieve a balance between accuracy and real-time performance. Current target detection based on UAVs mainly adopts single-stage target detection. However, current mainstream single-stage target detection algorithms such as YOLO, SSD, etc. still require large amount of calculation and cannot be directly applied to UAVs. Therefore, some current applications collect images through UAVs and then process the images on high-performance computing devices, which has low detection efficiency and greatly limits the application of UAVs in practice.

[0005] Currently, the mainstream single-stage target detection algorithm in the field of computer vision is YOLO v4. Compared with YOLO v3, YOLO v4 uses mish activation function, SPP (Spatial Pyramid Pooling)-YOLO, PANet (Path Aggregation Network) and other technologies, and the overall model performance is better than YOLO v3. YOLO v4-tiny is the latest lightweight target detection algorithm based on YOLO v4. This algorithm extracts features from the input image through a convolutional neural network to obtain a series of prediction boxes of different sizes and outputs the size of the corresponding prediction box. Finally, non-maximum suppression is performed according to the probability of the prediction box to draw the final detection box. Since this algorithm is designed for mobile devices, the neural network is designed to be shallow, and the parameter amount of the model is 6.817BFLOPs, and the weight file is 23.7Mb. Although this design scheme can greatly improve the real-time performance of the model, the accuracy of the model is greatly lost because it does not use any neural network training techniques. The model only cascades three different scale CSPNets, which makes the model have a large search range for targets of different scales and reduces the sensitivity of the model to targets.

[0006] The Chinese patent document "A Small Target Rapid Detection Method for Unmanned Aerial Vehicle Perspective Based on YOLO v3" discloses a method combining unmanned aerial vehicle and target detection, including four steps: image acquisition, small target detection network establishment, network training, and network pruning. This method provides a new idea for small target detection from the perspective of unmanned aerial vehicle, but the model uses YOLO v3-tiny, which has a gap in accuracy and real-time performance compared to YOLO v4-tiny. In addition, this method does not achieve real-time detection of unmanned aerial vehicle video streams, which limits the practical application of this method.

[0007] The Chinese patent document "A Low Altitude Target Precision Detection and Recognition Method for Unmanned Aerial Vehicle" proposes a target detection and recognition method for unmanned aerial vehicle images based on a fully convolutional network. This method combines unmanned aerial vehicles with the real world, establishing a camera imaging coordinate system, which can improve the recognition accuracy of targets of different scales. However, this method is not mounted on unmanned aerial vehicles and does not achieve real-time target detection.

[0008] The Chinese patent document "An Airborne Image Unmanned Aerial Vehicle Target Adaptive Detection Method" proposes a method of mounting a target detection network on an unmanned aerial vehicle computer for real-time detection. This method includes three steps: pre-training an initial model, pre-training a detection model, and loading the model onto the unmanned aerial vehicle computer. This method achieves real-time detection of unmanned aerial vehicle video streams, but the real-time performance of the target detection network may be affected due to the limited computing power of the onboard unmanned aerial vehicle. In addition, the onboard computer increases the load of the unmanned aerial vehicle, and the target detection computing power of the onboard unmanned aerial vehicle is large, which will affect the endurance of the unmanned aerial vehicle and limit its application in reality.

[0009] In addition, existing technologies for detecting unmanned aerial vehicle videos can be roughly divided into two categories. One is to use unmanned aerial vehicles for data collection and then input the video files into a target detection network on a computer. This method does not achieve real-time detection of video streams, has low timeliness, and has little practical significance. The other method is to use a lightweight target detection network to perform real-time detection of video streams through a microprocessor onboard the unmanned aerial vehicle. This method has high timeliness, but also has many problems. First, the computing power of the onboard microprocessor is a problem. Current target detection networks are based on deep convolutional neural networks (DNNs), which require high computing power of hardware devices. Limited by the low computing power of the onboard microprocessor, such target detection networks cannot be deep or large, which will directly affect the performance of the model. Second, the endurance of the unmanned aerial vehicle is a problem. If a microprocessor is used onboard, the weight of the development board and the inference process of the neural network will additionally increase the power consumption of the unmanned aerial vehicle system, thereby affecting the endurance time of the unmanned aerial vehicle.

[0010] Based on the defects of the above technical solutions, how to solve the problem of both real-time detection of UAV video streams and limited computing power of target detection networks has become a technical problem that needs to be solved in the field. SUMMARY

[0011] The purpose of the present application is to provide a target detection method, system and device based on a UAV and a NVIDIA development board to solve the problem of limited computing power of target detection networks.

[0012] To achieve the above purpose, the present application provides a target detection method based on a UAV and a NVIDIA development board, which comprises:

[0013] Step S1: constructing an improved YOLO v4-tiny model; the improved YOLO v4-tiny model comprises an initial convolution module, a first hourglass type pooling module, a second hourglass type pooling module, a third hourglass type pooling module, a first pooling feature enhancement module, a second pooling feature enhancement module, a splicing module, a first output module and a second output module;

[0014] Step S2: training the improved YOLO v4-tiny model using a training set to obtain a final target detection model;

[0015] Step S3: obtaining a video stream collected by a UAV on-board camera using a video capture card; the video stream comprises a plurality of continuous images;

[0016] Step S4: inputting the video stream into the final target detection model deployed on the NVIDIA development board for target detection and outputting a target detection result.

[0017] Optionally, the training of the improved YOLO v4-tiny model using the training set to obtain the final target detection model specifically comprises:

[0018] Step S21: when any one of the data set Pascal VOC, the data set MS COCO and the data set Visdron is used as the training set, the images in the training set are input into the improved YOLO v4-tiny model for training, the loss value is calculated using a total loss function, until the first termination condition is met, and the improved YOLO v4-tiny model corresponding to the minimum loss value or the maximum accuracy is selected as the final target detection model.

[0019] Optionally, the training of the improved YOLO v4-tiny model using the training set to obtain the final target detection model specifically comprises:

[0020] Step S21: When any two of the data set Pascal VOC, the data set MS COCO and the data set Visdron are taken as the training set, an image in any selected data set is input to the improved YOLO v4-tiny model for training, a loss value is calculated by using a total loss function, until a first termination condition is met, and an improved YOLO v4-tiny model corresponding to a minimum loss value or a maximum accuracy is taken as a first target detection model;

[0021] Step S22: An image in another data set is input to the first target detection model for training, a loss value is calculated by using the total loss function, until a second termination condition is met, and the first target detection model corresponding to the minimum loss value or the maximum accuracy is taken as a final target detection model.

[0022] Optionally, the improved YOLO v4-tiny model is trained by using the training set, and the final target detection model is obtained, and specifically includes the following steps.

[0023] Step S21: When the data set Pascal VOC, the data set MS COCO and the data set Visdron constitute the training set, an image in the data set Pascal VOC is input to the improved YOLO v4-tiny model for training, a loss value is calculated by using a total loss function, until a first termination condition is met, and an improved YOLO v4-tiny model corresponding to a minimum loss value or a maximum accuracy is taken as a first target detection model;

[0024] Step S22: An image in the data set MS COCO is input to the first target detection model for training, a loss value is calculated by using the total loss function, until a second termination condition is met, and the first target detection model corresponding to the minimum loss value or the maximum accuracy is taken as a second target detection model;

[0025] Step S23: An image in the data set Visdron is input to the second target detection model for training, a loss value is calculated by using the total loss function, until a third termination condition is met, and the second target detection model corresponding to the minimum loss value or the maximum accuracy is taken as a final target detection model.

[0026] Optionally, a specific calculation formula of the total loss function is as follows:

[0027]

[0028] Wherein, L is a total loss function, L DIOU is a positioning loss function, L cls is a target confidence loss function, L class is a classification loss function, ρ(B, B gt) is the Euclidean distance between the center point of the predicted bounding box B of the UAV image and the real bounding box B gt c is the distance between the predicted bounding box and the real bounding box of the UAV image, and IOU is the intersection over union of the predicted bounding box and the real bounding box.

[0029] Optionally, the first hourglass type pooling module comprises one hourglass type CSPNet module and one maximum pooling layer; wherein the hourglass type CSPNet module comprises: two dilated convolution layers with a dilated rate of 5, two convolution layers, a feature fusion layer add, and a feature fusion layer Concat; the second convolution layer adopts a convolution kernel of 1*1, a step length of 1, and a filter number of 32; the initial convolution module is connected with the first dilated convolution layer, the first dilated convolution layer is connected with the first convolution layer and the feature fusion layer Concat respectively, the first convolution layer is connected with the second convolution layer, the second convolution layer is connected with the feature fusion layer add respectively, the feature fusion layer add is connected with the second dilated convolution layer, the second dilated convolution layer is connected with the feature fusion layer Concat, and the feature fusion layer Concat is connected with the maximum pooling layer;

[0030] The second hourglass type pooling module comprises one hourglass type CSPNet module and one maximum pooling layer; wherein the hourglass type CSPNet module comprises: two dilated convolution layers with a dilated rate of 3, two convolution layers, a feature fusion layer add, and a feature fusion layer Concat; the second convolution layer adopts a convolution kernel of 1*1, a step length of 1, and a filter number of 64; the maximum pooling layer in the first hourglass type pooling module is connected with the first dilated convolution layer in the second hourglass type pooling module, the first dilated convolution layer is connected with the first convolution layer and the feature fusion layer Concat respectively, the first convolution layer is connected with the second convolution layer, the second convolution layer is connected with the feature fusion layer add respectively, the feature fusion layer add is connected with the second dilated convolution layer, the second dilated convolution layer is connected with the feature fusion layer Concat, and the feature fusion layer Concat is connected with the maximum pooling layer;

[0031] The third hourglass type pooling module comprises one hourglass type CSPNet module and one maximum pooling layer; wherein the hourglass type CSPNet module comprises two dilated convolution layers with a dilated rate of 1, two convolution layers, a feature fusion layer add and a feature fusion layer Concat; the second convolution layer adopts a convolution kernel of 1*1, a step of 1 and a filter number of 128; the maximum pooling layer in the second hourglass type pooling module is connected with the first dilated convolution layer in the third hourglass type pooling module, the first dilated convolution layer is connected with the first convolution layer and the feature fusion layer Concat respectively, the first convolution layer is connected with the second convolution layer, the second convolution layer is connected with the feature fusion layer add respectively, the feature fusion layer add is connected with the second dilated convolution layer, the second dilated convolution layer is connected with the feature fusion layer Concat, and the feature fusion layer Concat is connected with the maximum pooling layer;

[0032] The first pooling feature enhancement module comprises one convolution layer, one maximum pooling layer, one average pooling layer, one up-sampling layer and one superimposition layer; the convolution kernel of the convolution layer is 1*1, the step is 1 and the filter number is 128; the size of the maximum pooling layer is 3*3 and the step is 2; the convolution layer is connected with the maximum pooling layer in the first hourglass type pooling module, the convolution layer is connected with the maximum pooling layer and the average pooling layer respectively, the average pooling layer is connected with the up-sampling layer, the up-sampling layer and the maximum pooling layer are connected with the superimposition layer respectively, and the superimposition layer is connected with the splicing module;

[0033] The second pooling feature enhancement module comprises one convolution layer, one maximum pooling layer, one average pooling layer, one up-sampling layer and one superimposition layer; the convolution kernel of the convolution layer is 1*1, the step is 1 and the filter number is 256; the size of the maximum pooling layer is 3*3 and the step is 1; the convolution layer is connected with the maximum pooling layer in the second hourglass type pooling module, the convolution layer is connected with the maximum pooling layer and the average pooling layer respectively, the average pooling layer is connected with the up-sampling layer, the up-sampling layer and the maximum pooling layer are connected with the superimposition layer respectively, and the superimposition layer is connected with the splicing module.

[0034] The application further provides a target detection system based on a UAV and a NVIDIA development board.

[0035] The model construction module is used for constructing an improved YOLO v4-tiny model; the improved YOLO v4-tiny model comprises an initial convolution module, a first hourglass type pooling module, a second hourglass type pooling module, a third hourglass type pooling module, a first pooling feature enhancement module, a second pooling feature enhancement module, a splicing module, a first output module and a second output module.

[0036] The training module is configured to train the improved YOLO v4-tiny model by using a training set, and obtain a final target detection model.

[0037] The video stream acquisition module is configured to acquire a video stream collected by an onboard camera of the unmanned aerial vehicle by using a video capture card.

[0038] The target detection module is configured to input the video stream into the final target detection model deployed on the NVIDIA development board to perform target detection, and output a target detection result.

[0039] Optionally, the training module specifically comprises:

[0040] The final target detection model determination unit is configured to, when any one of the data set Pascal VOC, the data set MS COCO and the data set Visdron is used as the training set, input images in the training set into the improved YOLO v4-tiny model to perform training, calculate a loss value by using a total loss function, until a termination condition is met, and select an improved YOLO v4-tiny model corresponding to a minimum loss value or a maximum accuracy as the final target detection model.

[0041] Optionally, a specific calculation formula of the total loss function is as follows:

[0042]

[0043] wherein, L is the total loss function, L DIOU is a positioning loss function, L cls is a target confidence loss function, L class is a classification loss function, p(B, B gt ) is an Euclidean distance between a center point of a predicted bounding box B of the unmanned aerial vehicle image and a center point of a real bounding box B gt , c is a distance between a vertex of the predicted bounding box of the unmanned aerial vehicle image and a vertex of the real bounding box farthest from each other, and IOU is an intersection over union of the predicted bounding box and the real bounding box.

[0044] The application further provides a target detection device based on an unmanned aerial vehicle and an NVIDIA development board.

[0045] The onboard camera of the unmanned aerial vehicle is configured to collect a video stream.

[0046] The video capture card is connected with the onboard camera of the unmanned aerial vehicle, and is configured to perform format conversion on the collected image stream.

[0047] The internal deployment final target detection module is a NVIDIA development board, the video stream is input to the final target detection model for target detection, and the target detection result is output; the final target detection model comprises an initial convolution module, a first hourglass type pooling module, a second hourglass type pooling module, a third hourglass type pooling module, a first pooling feature enhancement module, a second pooling feature enhancement module, a splicing module, a first output module and a second output module.

[0048] According to the specific embodiments provided by the application, the following technical effects are disclosed.

[0049] The lightweight target detection network YOLO v4-tiny is redesigned, and is deployed on a ground device (i.e. a NVIDIA Jetson Xavier NX development board), and the video stream transmitted back by the unmanned aerial vehicle in real time is detected by the NVIDIA Jetson Xavier NX development board. The application realizes real-time detection of the unmanned aerial vehicle video stream, and solves the problem of limited computing power of the target detection network through the ground device. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0051] Figure 1 The flow chart of the target detection method based on the unmanned aerial vehicle and the NVIDIA development board of the present application;

[0052] Figure 2 The structure diagram of the pooling module in the original YOLO v4-tiny algorithm;

[0053] Figure 3 The schematic diagram of the original YOLO v4-tiny algorithm;

[0054] Figure 4 The structure diagram of the hourglass type CSPNet module in the improved YOLO v4-tiny algorithm of the present application;

[0055] Figure 5 The schematic diagram of the improved YOLO v4-tiny algorithm of the present application;

[0056] Figure 6 The structure diagram of the target detection system based on the unmanned aerial vehicle and the NVIDIA development board of the present application;

[0057] Figure 7 The comparison schematic diagram of the experimental results of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the protection scope of the present application.

[0059] The purpose of the present application is to provide a target detection method, system and device based on a UAV and a NVIDIA development board, to solve the problem of limited computing power of the target detection network.

[0060] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0061] Embodiment 1

[0062] As shown in Figure 1 The present application discloses a target detection method based on a UAV and a NVIDIA development board, which comprises:

[0063] Step S1: constructing an improved YOLO v4-tiny model; the improved YOLO v4-tiny model comprises an initial convolution module, a first hourglass type pooling module, a second hourglass type pooling module, a third hourglass type pooling module, a first pooling feature enhancement module, a second pooling feature enhancement module, a splicing module, a first output module and a second output module, as shown in Figure 5

[0064] Step S2: training the improved YOLO v4-tiny model using a training set to obtain a final target detection model.

[0065] Step S3: acquiring a video stream collected by a UAV on-board camera using a video capture card; the video stream comprises a plurality of continuous images.

[0066] Step S4: inputting the video stream into the final target detection model deployed on the NVIDIA development board for target detection, and outputting a target detection result.

[0067] The various steps will be described in detail below:

[0068] Without considering portability, if higher detection accuracy and higher performance models are required, the ground equipment (NVIDIA Jetson Xavier NX development board) can be replaced with a high-performance computer carrying a large target detection model.

[0069] ​As shown in Figure 3 The original YOLO v4-tiny algorithm is a convolutional neural network model with 6 cascade modules, and the structure is in turn: initial convolution module→first pooling module→second pooling module→third pooling module→first output module→second output module.

[0070] The initial convolution module contains three convolution kernels with a size of 3×3, a step of 2, and filter numbers of 32, 64, and 64, respectively, which performs preliminary feature extraction on the input image, and outputs a feature map with a size of 104×104×64.

[0071] The first pooling module contains a CSPNet module and a maximum pooling layer MaxPool. Figure 2 The specific structure of the CSPNet module is given, which contains 4 cascaded convolution layers Conv and 1 concatenation layer Concat. The first convolution layer has a convolution kernel size of 3×3, a step of 1, and a filter number of 64; the second convolution layer has a convolution kernel size of 3×3, a step of 1, and a filter number of 32; the third convolution layer has a convolution kernel size of 3×3, a step of 1, and a filter number of 32; the fourth convolution layer has a convolution kernel size of 3×3, a step of 1, and a filter number of 64. Among them, the 4 convolution layers of the CSPNet module are connected using a residual structure, the output feature map of the third convolution layer and the output feature map of the second convolution layer are output to the fourth convolution layer through the feature fusion layer Cocat using feature fusion; the output feature map of the fourth convolution layer and the output feature map of the first convolution layer are input to the concatenation layer for concatenation and then output to the maximum pooling layer MaxPool. The maximum pooling layer has a size of 2×2 and a step of 2, which can reduce the parameter quantity of the output feature and improve the processing speed of the model. The feature map size after passing through the maximum pooling layer is 52×52×128.

[0072] The second pooling module contains a CSPNet module and a maximum pooling layer MaxPool with a size of 2×2 and a step of 2. Among them, the overall structure of the CSPNet module is the same as that of the CSPNet module in the previous layer, only the filter number is different. The filter numbers of the 4 cascaded convolution layers of this layer of CSPNet module are 128, 64, 64, and 128, respectively, and the output feature map size is 26×26×256.

[0073] The third pooling module contains a CSPNet module and a maximum pooling layer MaxPool with a size of 2×2 and a step of 2. Among them, the overall structure of the CSPNet module is the same as that of the CSPNet module in the previous layer, only the filter number is different. The filter numbers of the 4 cascaded convolution layers of this layer of CSPNet module are 256, 128, 128, and 256, respectively, and the output feature map size is 13×13×512.

[0074] The first output module comprises four cascaded convolution layers Conv and one detection layer. The first convolution layer has a convolution kernel size of 3*3, a step of 1 and a filter number of 512; the second convolution layer has a convolution kernel size of 3*3, a step of 1 and a filter number of 256; the third convolution layer has a convolution kernel size of 3*3, a step of 1 and a filter number of 256; the fourth convolution layer has a convolution kernel size of 1*1, a step of 1 and a filter number of 18; and the detection layer outputs a feature map with a size of 13*13*18.

[0075] The second output module comprises three cascaded convolution layers Conv, one up-sampling layer Upsample and one detection layer. The first convolution layer has a convolution kernel size of 1*1, a step of 1 and a filter number of 128; the second convolution layer has a convolution kernel size of 3*3, a step of 1 and a filter number of 256; the third convolution layer has a convolution kernel size of 1*1, a step of 1 and a filter number of 18; and the up-sampling layer has an up-sampling multiple of 2. The input feature map of the first convolution layer is the feature map output by the second convolution layer in the first output module, and after the first convolution layer in the second output module, an up-sampling operation is performed, and the feature map is fused with the fourth convolution layer in the second pooling module and then output to the third convolution layer, and finally the feature map with a size of 26*26*18 is output by the detection layer.

[0076] The improved YOLO v4-tiny model constructed in the application is improved on the basis of the original YOLO v4-tiny algorithm, mainly including three aspects: 1. improvement of the loss function; 2. introduction of a different scale hourglass type CSPNet module; and 3. introduction of two pooling feature enhancement modules.

[0077] 1. Improvement of the loss function.

[0078] The loss function of YOLO v4-tiny has three parts, which can be expressed as: L tiny = L ciou + L cls + L class . Among them, L ciou is the positioning loss function, L cls is the target confidence loss function, and L class is the classification loss function. Among them, the positioning loss function can measure the error of the predicted bounding box position and the box size; the target confidence loss function can reflect whether the class is balanced in the detection process; and the classification loss function is the square error of the conditional class probability of each class. Since the feature scale of the unmanned aerial vehicle image changes greatly, false detection and missed detection are prone to occur in the detection process, therefore, the positioning loss function is improved in the application, and the improved formula L ciou is as follows:

[0079] L ciou The expression is:

[0080] Wherein, beta is a weight function, rho is the Euclidean distance between the prediction box and the real box center point of the unmanned aerial vehicle image, w gt , h gt The width and height of the real box, w, h is the width and height of the prediction box, c is the distance between the prediction box and the real box of the unmanned aerial vehicle image, B is the prediction box area B gt The intersection over union of the prediction box and the real box, The model compares the prediction box and the real box of the detection result of the first output module (i.e. the first YOLO module) and the second output module (i.e. the second YOLO module), and obtains the related parameters. Finally, the positioning loss of the algorithm is calculated by formula .

[0081] CIOU considers the Euclidean distance between the prediction box and the real box, the overlap rate, the scale feature, and the width-height ratio of the prediction box. When detecting targets with certain width-height ratio (such as people, cars, etc.), the width-height ratio of the real box of the target can be obtained, thereby improving the accuracy of target detection. However, the image features collected by the unmanned aerial vehicle are complex, and the scale changes greatly, so there is no specific rule for the width-height ratio. If CIOU is used to make the model learn the width-height ratio of some similar size real box, it may cause misjudgment of the model for other width-height ratio images, which will reduce the robustness of the model. The English corresponding to CIOU is shown in Table 1.

[0082] In summary, the DIOU (Distance-IoU) is used to improve the positioning loss function L DIOU of the application, and the specific formula is: In the formula, L DIOU is the positioning loss function, rho is the Euclidean distance between the prediction box and the real box center point, and c is the diagonal distance of the smallest rectangular box containing the prediction box and the real box. DIOU contains the Euclidean distance between the prediction box and the real box center point, the overlap degree and other features, but does not consider the width-height ratio of the real box. Therefore, for the width-height ratio of the image collected by the unmanned aerial vehicle, DIOU can better reflect the relationship between the prediction box and the real box, and can effectively enhance the generalization ability of the model. The English corresponding to DIOU is shown in Table 1.

[0083] In summary, the specific calculation formula of the improved total loss function of the application is:

[0084]

[0085] Wherein, L is the total loss function, L DIOU is the positioning loss function, Lcls L is a target confidence loss function class L is a classification loss function, and p(B, B gt ) is the Euclidean distance between the center point of the predicted frame B of the unmanned aerial vehicle image and the real frame B gt , c is the distance of the farthest vertex between the predicted frame and the real frame of the unmanned aerial vehicle image, and IOU is the intersection over union of the predicted frame and the real frame.

[0086] 2. Introduce different scale hourglass CSPNet modules.

[0087] The original YOLO v4-tiny algorithm adopts three cascaded pooling modules. In order to obtain more rich aerial image detail features, the CSPNet module adopts a residual structure for connection. YOLO v4-tiny only adopts a traditional convolution cascade structure, and compared with other target detection algorithms, it has a great advantage in detection speed, but due to the simple network structure, it cannot adapt to multi-scale unmanned aerial vehicle aerial detection objects. In view of the problem, the present application introduces an inflation convolution, and proposes the structure of the hourglass CSPNet module.

[0088] The inflation convolution can obtain higher resolution features while keeping the parameter amount and the same convolution kernel size. Let F be a discrete function satisfying Let Let k be a discrete filter with a size of (2r+1) 2 , satisfying Let l be the inflation coefficient of the inflation convolution, and the inflation convolution operator l can be defined as: The inflation convolution has both the convolution filtering function of the conventional convolution layer and the generalization effect of the pooling layer. In addition, the inflation convolution does not reduce the feature map size with the increase of stride, can expand the receptive field of the model, and obtain higher resolution feature maps. The inflation convolution is applied to the base layer and the partial transition of CSPNet, which can widen the receptive field of CSPResNet with a small calculation cost. Therefore, the inflation convolution with different inflation rates is adopted to widen the feature map size, relieve the gradient confusion problem and improve the model performance.

[0089] The improved algorithm of the present application adopts three hourglass CSPNet modules with different inflation rates to replace the CSPNet modules in the original algorithm, and the hourglass CSPNet module is as shown in Figure 4 The structure of the hourglass CSPNet module is the same as that of the CSPNet module, and adopts a residual structure. Since the feature map size of the shallow layer of the network is larger, and the feature map size of the deep layer of the network is smaller, the improved algorithm of the present application adopts a larger inflation rate in the shallow layer of the network, and adopts a smaller inflation rate in the deep layer of the network.

[0090] The first hourglass-shaped pooling module includes: one hourglass-shaped CSPNet module and one max-pooling layer (MaxPool); the hourglass-shaped CSPNet module includes: two dilated convolutional layers, two convolutional layers, and one feature fusion layer; the initial convolutional module is connected to the first dilated convolutional layer, the first dilated convolutional layer is connected to the first convolutional layer and the feature fusion layer (Concat), the first convolutional layer is connected to the second convolutional layer, the second convolutional layer is connected to the first convolutional layer and the feature fusion layer (Add), the feature fusion layer (Add) is connected to the second dilated convolutional layer, the second dilated convolutional layer is connected to the feature fusion layer (Concat), and the feature fusion layer (Concat) is connected to the max-pooling layer; the dilation rate of the first dilated convolutional layer (i.e., the first dilated convolutional layer) and the fourth dilated convolutional layer (i.e., the second dilated convolutional layer) is 5, and other parameters are consistent with the corresponding convolutional layers of the CSPNet module in the original model. The parameters of the second convolutional layer (i.e., the first convolutional layer) are consistent with the corresponding convolutional layer in the original CSPNet module. The third convolutional layer (i.e., the second convolutional layer) uses a 1*1 convolutional kernel with a stride of 1 and 32 filters. The four convolutional layers of the hourglass pooling module are connected using a residual structure. The output feature map of the third convolutional layer and the output feature map of the second convolutional layer are fused through the feature fusion layer `add` and output to the fourth dilated convolutional layer (i.e., 3*3 dilated Conv). The output feature map of the fourth dilated convolutional layer and the output feature map of the first dilated convolutional layer are fused through the feature fusion layer `cocat` and output to the max pooling layer `MaxPool`. The final output feature map size is 52×52×128.

[0091] The second hourglass type pooling module comprises one hourglass type CSPNet module and one maximum pooling layer MaxPool; wherein the hourglass type CSPNet module comprises two dilated convolution layers, two convolution layers and one feature fusion layer, wherein the dilated rate of the first dilated convolution layer (i.e. the first dilated convolution layer) and the fourth dilated convolution layer (i.e. the second dilated convolution layer) is 3. The maximum pooling layer in the first hourglass type pooling module is connected with the first dilated convolution layer in the second hourglass type pooling module, the first dilated convolution layer is connected with the first convolution layer and the feature fusion layer Concat respectively, the first convolution layer is connected with the second convolution layer, the second convolution layer is connected with the feature fusion layer add, the feature fusion layer add is connected with the second dilated convolution layer, the second dilated convolution layer is connected with the feature fusion layer Concat, and the feature fusion layer Concat is connected with the maximum pooling layer; the parameters of the second convolution layer (i.e. the first convolution layer) are consistent with the corresponding convolution layer of the CSPNet module of the original model, the third convolution layer (i.e. the second convolution layer) adopts a convolution kernel of 1*1, the step is 1, and the filter number is 64. The two convolution layers and the two dilated convolution layers of the hourglass type CSPNet module are connected by using a residual structure, and the output feature map size is 26*26*256.

[0092] The third hourglass type pooling module comprises one hourglass type CSPNet module and one maximum pooling layer MaxPool; wherein the hourglass type CSPNet module comprises two dilated convolution layers, two convolution layers and one feature fusion layer, wherein the dilated rate of the first dilated convolution layer (i.e. the first dilated convolution layer) and the fourth dilated convolution layer (i.e. the second dilated convolution layer) is 1. The maximum pooling layer in the second hourglass type pooling module is connected with the first dilated convolution layer in the third hourglass type pooling module, the first dilated convolution layer is connected with the first convolution layer and the feature fusion layer Concat respectively, the first convolution layer is connected with the second convolution layer, the second convolution layer is connected with the feature fusion layer add, the feature fusion layer add is connected with the second dilated convolution layer, the second dilated convolution layer is connected with the feature fusion layer Concat, and the feature fusion layer Concat is connected with the maximum pooling layer; the parameters of the second convolution layer (i.e. the first convolution layer) are consistent with the corresponding convolution layer of the CSPNet module of the original model, the third convolution layer (i.e. the second convolution layer) adopts a convolution kernel of 1*1, the step is 1, and the filter number is 128. The two convolution layers and the two dilated convolution layers of the hourglass type CSPNet module are connected by using a residual structure, and the output feature map size is 13*13*512.

[0093] 3. Two pooling feature enhancement modules are added.

[0094] The pooling feature enhancement module simultaneously uses a maximum pooling operation and an average pooling operation. The average pooling operation can aggregate spatial dimension information of channel features, the maximum pooling can refine channel attention, and simultaneously using the maximum pooling and the average pooling can improve network performance.

[0095] The pooling feature enhancement module first performs maximum pooling on the features and average pooling operations to preliminarily integrate the feature map and obtain spatial information thereof. In addition, to maintain consistency of the feature map dimension, the present inventors add an up-sampling layer after the average pooling layer. Then, the preliminarily extracted feature I' is transmitted to a weight-shared convolution layer, and after a pooling operation, output elements are combined to obtain output features The calculation process of the pooling feature enhancement module can be represented by the following formula:

[0096]

[0097] wherein sigmoid is a sigmoid function, Conv is a weight-shared convolution layer after a pooling operation, and W1 Conv are shared weights, a Relu activation function is used, I represents an input feature map, I Avg represents a feature map input to an average pooling layer, I Max represents a feature map input to a maximum pooling layer, AvgPool() represents an average pooling layer, and MaxPool() represents a maximum pooling layer.

[0098] As Figure 5As shown, the two pooling feature enhancement modules in the improved algorithm are respectively cascaded with the first hourglass type pooling module and the second hourglass type pooling module; wherein the first pooling feature enhancement module includes 1 convolution layer Conv, 1 maximum pooling layer MaxPool, 1 average pooling layer AvgPool, 1 up-sampling layer Upsample and 1 superimposition layer route. The convolution layer Conv is connected with the maximum pooling layer MaxPool in the first hourglass type pooling module, the convolution layer Conv is respectively connected with the maximum pooling layer MaxPool and the average pooling layer AvgPool, the average pooling layer AvgPool is connected with the up-sampling layer Upsample, the up-sampling layer Upsample and the maximum pooling layer MaxPool are respectively connected with the superimposition layer route, and the superimposition layer route is connected with the splicing module shortcut. Among them, the convolution kernel of the convolution layer is 1*1, the step is 1, and the number of filters is 128; the size of the maximum pooling layer is 3*3, the step is 2, and the average pooling layer and the up-sampling layer are connected in parallel; the up-sampling multiple of the up-sampling layer is 13. The input feature map size is 52*52*128, and the output feature map size is 13*13*256.

[0099] The second pooling feature enhancement module includes 1 convolution layer Conv, 1 maximum pooling layer MaxPool, 1 average pooling layer AvgPool, 1 up-sampling layer Upsample and 1 superimposition layer route. The convolution layer Conv is connected with the maximum pooling layer MaxPool in the second hourglass type pooling module, the convolution layer Conv is respectively connected with the maximum pooling layer MaxPool and the average pooling layer AvgPool, the average pooling layer AvgPool is connected with the up-sampling layer Upsample, the up-sampling layer Upsample and the maximum pooling layer MaxPool are respectively connected with the superimposition layer route, and the superimposition layer route is connected with the splicing module shortcut. Among them, the convolution kernel of the convolution layer is 1*1, the step is 1, and the number of filters is 256; the size of the maximum pooling layer is 3*3, the step is 1, and the average pooling layer and the up-sampling layer are connected in parallel; the up-sampling multiple of the up-sampling layer is 13. The input feature size is 26*26*256, and the output feature size is 13*13*512.

[0100] Finally, the output feature maps of the first pooling feature enhancement module, the second pooling feature enhancement module and the third hourglass type pooling module are spliced by the splicing layer shortcut and then input to the first output module.

[0101] In order to further reduce the calculation amount of the model, the filter number of the second 3*3 convolution layer of the first output module in the improved algorithm is adjusted from the original 512 to 256, and the filter number of the first 3*3 convolution layer of the second output module is adjusted from the original 512 to 256. In addition, the original algorithm is also compressed to improve the detection efficiency of the model.

[0102] Step S2: training the improved YOLO v4-tiny model using the training set to obtain the final target detection model, specifically including the following three ways.

[0103] The first way step is:

[0104] Step S21: when any one of the data set Pascal VOC, the data set MSCOCO and the data set Visdron is used as the training set, the images in the training set are input to the improved YOLO v4-tiny model for training, and the loss value is calculated using the total loss function until the first termination condition is met, and the improved YOLOv4-tiny model corresponding to the minimum loss value or the maximum accuracy is selected as the final target detection model.

[0105] The second way step is:

[0106] Step S21: when any two of the data set Pascal VOC, the data set MSCOCO and the data set Visdron are used as the training set, the images in any one of the data sets are input to the improved YOLO v4-tiny model for training, and the loss value is calculated using the total loss function until the first termination condition is met, and the improved YOLOv4-tiny model corresponding to the minimum loss value or the maximum accuracy is selected as the first target detection model.

[0107] Step S22: the images in the other data set are input to the first target detection model for training, and the loss value is calculated using the total loss function until the second termination condition is met, and the first target detection model corresponding to the minimum loss value or the maximum accuracy is selected as the final target detection model.

[0108] The third way step is:

[0109] Step S21: when the data set Pascal VOC, the data set MSCOCO and the data set Visdron constitute the training set, the images in the data set Pascal VOC are input to the improved YOLO v4-tiny model for training, and the loss value is calculated using the total loss function until the first termination condition is met, and the improved YOLOv4-tiny model corresponding to the minimum loss value or the maximum accuracy is selected as the first target detection model.

[0110] Step S22: inputting images in the data set MS COCO into the first target detection model for training, calculating a loss value by using a total loss function until a second termination condition is met, and selecting the first target detection model corresponding to the minimum loss value or the maximum accuracy as the second target detection model.

[0111] Step S23: inputting images in the data set Visdron into the second target detection model for training, calculating a loss value by using a total loss function until a third termination condition is met, and selecting the second target detection model corresponding to the minimum loss value or the maximum accuracy as the final target detection model.

[0112] The three mode models of the application are trained by using multiple data sets for transfer learning, and therefore the final model has strong robustness.

[0113] In the embodiment, the first termination condition, the second termination condition and the third termination condition can be the same or different, and the first termination condition can be that the number of iterations reaches a maximum number of iterations, the learning rate is less than a certain threshold, or the difference between adjacent two loss values is less than a certain threshold, or the loss values in continuous several times show a downward trend.

[0114] In the embodiment, the improved YOLO v4-tiny model is trained by using an RTX 2080Ti type training platform, and the training framework is Darknet, which is a target detection platform based on C++ language. The learning rate directly affects the performance of the gradient descent method, and therefore the learning rate is set to 0.00261 in the application. The maximum number of iterations is set to 80000 times. In the first 1000 training, the learning rate remains unchanged; when the training reaches 64000 times and 72000 times, the learning rate of the model is reduced by 10 times and 100 times, respectively. The momentum of the model can accelerate the learning speed of the model, especially for high curvature or noisy gradients. Due to the complexity of the unmanned aerial vehicle aerial image, the momentum of the model is set to 0.9 in the application. The input image size is 416*416.

[0115] In the embodiment, the improved YOLO v4-tiny model, the first target detection model and the second target detection model have the same structure and only different weights, and therefore the method for inputting images in the data set Pascal VOC, the data set MS COCO or the data set Visdron into the improved YOLO v4-tiny model, the first target detection model and the second target detection model for training is similar, and therefore the specific process of inputting images in the data set Pascal VOC into the improved YOLO v4-tiny model for training is shown in the following table: Figure 5

[0116] ​1、The image in the data set Pascal VOC is input to the preliminary convolution module, and the image is preliminarily extracted through three 3*3 convolution layers, and a feature map A is output; wherein the size of the feature map A is 104*104*64. The size of the input image is 416*416.

[0117] 2、The feature map A is input to the first hourglass type module, and the feature of the expanded receptive field is extracted through the dilated convolution with the dilated rate of 5, and the feature map output by the first hourglass type module is down-sampled through a 2*2 maximum pooling layer, and a feature map B is output, and the size of the feature map is 52*52*128.

[0118] 3、The feature map B is output to the first pooling feature enhancement module and the second hourglass type pooling module respectively. The feature output to the first pooling feature enhancement module is firstly preliminarily extracted through a 1*1 convolution layer with a step of 4, and then output to the maximum pooling layer and the average pooling layer respectively, and the feature map output by the up-sampling is fused with the feature map output by the maximum pooling layer, and a feature map C is output, and the size of the feature map C is 13*13*256. The feature map output to the second hourglass type pooling module is extracted through the dilated convolution with the dilated rate of 3, and the feature of the expanded receptive field is extracted, and the feature map is pooled through a 2*2 maximum pooling layer, and a feature map D is output, and the size of the feature map D is 26*26*256.

[0119] 4、The feature map D is output to the second pooling feature enhancement module and the third hourglass type pooling module respectively. The feature output to the second pooling feature enhancement module is firstly preliminarily extracted through a 1*1 convolution layer with a step of 2, and then output to the maximum pooling layer and the average pooling layer respectively, and the feature map obtained by the up-sampling is fused with the feature map output by the maximum pooling layer, and a feature map E is output, and the size of the feature map E is 13*13*512. The feature map output to the third hourglass type module is extracted through the dilated convolution with the dilated rate of 1, and the high-resolution deep feature is extracted, and the feature is pooled through a 2*2 maximum pooling layer, and a feature map F is output, and the size of the feature map F is 13*13*512.

[0120] 5、The feature map C, the feature map E and the feature map F are spliced through the Shortcut splicing layer, and the spliced feature G has the size of 13*13*512.

[0121] 6、The feature map G is output to the first YOLO module (i.e. the first output module), and the first detection result is output through four cascaded convolution layers and a detection layer, and the size of the first detection result is 13*13*18, which includes the prediction frame and the category of the input image. Then the prediction frame is compared with the real frame, and the improved positioning loss function The localization loss is calculated.

[0122] 7, the feature map G is output to the second output module through a 3*3 convolution layer and a 1*1 convolution layer. The feature map H is obtained through an up-sampling layer with a step of 2. Then the feature map H is fused with the output feature map of the second hourglass type pooling module, and is output to the detection layer through two cascaded convolution layers, with a size of 26*26*18. The detection layer is mainly responsible for classifying the output features and drawing prediction boxes. Finally, by comparing with the class and real box of the real image, the total loss of the model is obtained through the formula L=L DIOU +L cls +L class .

[0123] The trained final target detection model is deployed to the NVIDIA development board, the development board is equipped with an ubuntu operating system, and is compiled in a darknet environment, and the highest performance working mode of the development board is 15W 4Core, and the target detection algorithm is tested in this mode.

[0124] The video stream is transmitted to the NVIDIA development board through the HDMI connection line, and the highest output resolution of the video stream is 4k, the real-time detection of the unmanned aerial vehicle video is realized through the final target detection model carried by the development board, the real-time video stream is directly output to the display device of the development board, and the detection result can be directly observed. In addition, the algorithm is tested in three ways of pictures, video files and real-time pictures of a camera, and the power and working mode of the development board are adjusted according to the test result, so that the performance of the method disclosed in the application reaches the optimum in the hardware environment.

[0125] Table 1 English table

[0126]

[0127]

[0128] Example 2

[0129] As Figure 6 shown, the application also discloses a target detection system based on an unmanned aerial vehicle and an NVIDIA development board, the system comprises:

[0130] The model construction module 601 is used for constructing an improved YOLO v4-tiny model; the improved YOLO v4-tiny model comprises an initial convolution module, a first hourglass type pooling module, a second hourglass type pooling module, a third hourglass type pooling module, a first pooling feature enhancement module, a second pooling feature enhancement module, a splicing module, a first output module and a second output module.

[0131] The training module 602 is configured to train the improved YOLO v4-tiny model by using the training set, and obtain a final target detection model.

[0132] The video stream acquisition module 603 is configured to acquire a video stream collected by an unmanned aerial vehicle on-board camera by using a video capture card; the video stream includes a plurality of continuous images.

[0133] The target detection module 604 is configured to input the video stream into the final target detection model deployed on the NVIDIA development board to perform target detection, and output a target detection result.

[0134] As an optional implementation, the training module 602 includes the following components:

[0135] The final target detection model determination unit is configured to, when any one of the data set Pascal VOC, the data set MS COCO and the data set Visdron is used as the training set, input images in the training set into the improved YOLO v4-tiny model for training, calculate a loss value by using a total loss function, until a first termination condition is met, and select the improved YOLO v4-tiny model corresponding to a minimum loss value or a maximum accuracy as the final target detection model.

[0136] As an optional implementation, the training module 602 includes the following components:

[0137] The first training unit is configured to, when any two of the data set Pascal VOC, the data set MS COCO and the data set Visdron are used as the training set, input images in any selected data set into the improved YOLO v4-tiny model for training, calculate a loss value by using a total loss function, until a first termination condition is met, and select the improved YOLO v4-tiny model corresponding to a minimum loss value or a maximum accuracy as the first target detection model.

[0138] The final target detection model determination unit is configured to input images in another data set into the first target detection model for training, calculate a loss value by using a total loss function, until a second termination condition is met, and select the first target detection model corresponding to a minimum loss value or a maximum accuracy as the final target detection model.

[0139] As an optional implementation, the training module 602 includes the following components:

[0140] The first training unit is configured to input images in the data set Pascal VOC into the improved YOLO v4-tiny model for training, calculate a loss value by using a total loss function until a first termination condition is met, and select an improved YOLO v4-tiny model corresponding to a minimum loss value or a maximum accuracy as a first target detection model.

[0141] The second training unit is configured to input images in the data set MS COCO into the first target detection model for training, calculate a loss value by using a total loss function until a second termination condition is met, and select a first target detection model corresponding to a minimum loss value or a maximum accuracy as a second target detection model.

[0142] The final target detection model determination unit is configured to input images in the data set Visdron into the second target detection model for training, calculate a loss value by using a total loss function until a third termination condition is met, and select a second target detection model corresponding to a minimum loss value or a maximum accuracy as a final target detection model.

[0143] The same content as in Embodiment 1 will not be discussed one by one, and specific details can be found in Embodiment 1.

[0144] Embodiment 3

[0145] The application also provides a target detection device based on a UAV and a NVIDIA development board, the device comprising:

[0146] A UAV on-board camera is configured to collect a video stream.

[0147] A video capture card is connected to the UAV on-board camera and is configured to perform format conversion on the collected image stream.

[0148] A NVIDIA development board is internally deployed with a final target detection module, the video stream is input into the final target detection model for target detection, and a target detection result is output; the final target detection model comprises an initial convolution module, a first hourglass type pooling module, a second hourglass type pooling module, a third hourglass type pooling module, a first pooling feature enhancement module, a second pooling feature enhancement module, a splicing module, a first output module, and a second output module.

[0149] A display is configured to display the target detection result.

[0150] Embodiment 4

[0151] Table 2 is a comparison of the performance of the original YOLO v4-tiny algorithm and the improved YOLO v4-tiny algorithm on the Pascal VOC dataset. Among them, the parameter quantity is the calculation amount of all modules of the model, which can measure the complexity of the algorithm; the weight size is the size of the largest weight file of the model, which can reflect the amount of calculation required by the model and will directly affect the FPS (Frames Per Second) of the model; mAP (mean Average Precision) is the detection accuracy, which is a commonly used measurement index in the field of object detection, including the accuracy of identification and the matching degree of the predicted anchor box and the real anchor box.

[0152] Table 2 is a comparison of the performance of the original YOLO v4-tiny algorithm and the improved YOLO v4-tiny algorithm on the Pascal VOC dataset. Among them, the parameter quantity is the calculation amount of all modules of the model, which can measure the complexity of the algorithm; the weight size is the size of the largest weight file of the model, which can reflect the amount of calculation required by the model and will directly affect the FPS (Frames Per Second) of the model; mAP (mean Average Precision) is the detection accuracy, which is a commonly used measurement index in the field of object detection, including the accuracy of identification and the matching degree of the predicted anchor box and the real anchor box.

[0153] Parameter size Weight size mAP (%) Original algorithm 6.817 23.7 Mb 62.57 Improved algorithm 6.426 19 Mb 66.7

[0154] As shown in Table 2, due to the pruning of the deep convolution kernel of the model, the parameter quantity and weight size of the improved algorithm are lower than those of the original algorithm, and the improved algorithm has lower requirements for the computing power of the hardware device. The mAP of the improved algorithm is also better than that of the original algorithm, which proves that the improvement of the present application is effective. To further compare the performance of the improved algorithm, the present application further experiments on the Visdrone dataset of the unmanned aerial vehicle.

[0155] Table 3 is a comparison of the performance of the original YOLO v4-tiny algorithm and the improved YOLO v4-tiny algorithm on the Visdrone dataset. As shown in Table 3, the mAP and FPS of the improved algorithm are better than those of the original algorithm. In addition, the present application adds an improvement of the loss function. The loss function value of the improved algorithm is lower than that of the original algorithm, which further proves that the improvement of the loss function of the present application is effective.

[0156] Parameter size Weight size mAP (%) Loss FPS Original algorithm 6.817 23.7 Mb 13.5 1.95 25.6 Improved algorithm 6.426 19 Mb 14.2 1.52 28.1

[0157] Table 3 is a comparison of the performance of the original YOLO v4-tiny algorithm and the improved YOLO v4-tiny algorithm on the Visdrone dataset. As shown in Table 3, the mAP and FPS of the improved algorithm are better than those of the original algorithm. In addition, the present application adds an improvement of the loss function. The loss function value of the improved algorithm is lower than that of the original algorithm, which further proves that the improvement of the loss function of the present application is effective.

[0158] After deploying the trained model to the Nvidia development board, Figure 7 Fig. (a) is a detection effect diagram obtained by using the YOLO v4-tiny algorithm, Figure 7 Fig. (b) is a detection effect obtained by using the improved YOLO v4-tiny algorithm.

[0159] The various embodiments described in this specification are presented for the purpose of illustrating the principles of the application and its best mode of operation. Each of the embodiments described in this specification has been provided for the purpose of illustration and is not intended to limit the application. The same or similar reference numerals in different drawings represent the same or similar elements.

[0160] The principles and operation of the present application have been explained so far with the help of specific examples. The examples have been presented for the purpose of illustration and are not intended to limit the application. The application described in this specification can be implemented in hardware and / or software that is modified to operate in accordance with the principles set forth in this specification. Additionally, the description and drawings are to be regarded as illustrative in nature and their objects are to be understood not only to be the solution suggested by the inventor or inventors but to include any and all implementations within the scope of the claims.

Claims

1. A target detection method based on a UAV and a NVIDIA development board, characterized in that, The method comprises: Step S1: constructing an improved YOLO v4-tiny model; the improved YOLO v4-tiny model comprises an initial convolution module, a first hourglass type pooling module, a second hourglass type pooling module, a third hourglass type pooling module, a first pooling feature enhancement module, a second pooling feature enhancement module, a splicing module, a first output module and a second output module; The first hourglass type pooling module comprises one first hourglass type CSPNet module and one maximum pooling layer; wherein the first hourglass type CSPNet module comprises two dilated convolution layers with a dilated rate of 5, two convolution layers, a feature fusion layer add and a feature fusion layer Concat; The second hourglass type pooling module comprises one second hourglass type CSPNet module and one maximum pooling layer; wherein the second hourglass type CSPNet module comprises two dilated convolution layers with a dilated rate of 3, two convolution layers, a feature fusion layer add and a feature fusion layer Concat; The third hourglass type pooling module comprises one third hourglass type CSPNet module and one maximum pooling layer; wherein the third hourglass type CSPNet module comprises two dilated convolution layers with a dilated rate of 1, two convolution layers, a feature fusion layer add and a feature fusion layer Concat; The first pooling feature enhancement module comprises one convolution layer, one maximum pooling layer, one average pooling layer, one up-sampling layer and one superimposition layer; The second pooling feature enhancement module comprises one convolution layer, one maximum pooling layer, one average pooling layer, one up-sampling layer and one superimposition layer; Step S2: training the improved YOLO v4-tiny model using a training set to obtain a final target detection model; Step S3: acquiring a video stream collected by a UAV on-board camera using a video capture card; the video stream comprises a plurality of continuous images; Step S4: inputting the video stream into the final target detection model deployed on the NVIDIA development board to perform target detection and output a target detection result. 2.The lightweight target detection method for a UAV and a NVIDIA development board of claim 1, wherein The training of the improved YOLO v4-tiny model using a training set to obtain a final target detection model comprises: Step S21: when any one of a data set PascalVOC, a data set MS COCO and a data set Visdron is used as a training set, images in the training set are input into the improved YOLO v4-tiny model for training, a loss value is calculated using a total loss function until a first termination condition is met, and the improved YOLO v4-tiny model corresponding to the minimum loss value or the maximum accuracy is selected as the final target detection model. 3.The lightweight object detection method for a UAV and a NVIDIA development board of claim 1, wherein The training of the improved YOLO v4-tiny model using a training set to obtain a final target detection model comprises: Step S21: when any two of the data set PascalVOC, the data set MS COCO and the data set Visdron are taken as the training set, an image in any selected data set is input to the improved YOLO v4-tiny model for training, a loss value is calculated by using a total loss function, until a first termination condition is met, and an improved YOLO v4-tiny model corresponding to a minimum loss value or a maximum accuracy is taken as a first target detection model; Step S22: an image in another data set is input to the first target detection model for training, a loss value is calculated by using the total loss function, until a second termination condition is met, and the first target detection model corresponding to the minimum loss value or the maximum accuracy is taken as a final target detection model.

4. The lightweight object detection method for drone and NVIDIA development board of claim 1, wherein The improved YOLO v4-tiny model is trained by using the training set, and the final target detection model is obtained, and specifically includes: Step S21: when the data set PascalVOC, the data set MS COCO and the data set Visdron constitute the training set, an image in the data set PascalVOC is input to the improved YOLO v4-tiny model for training, a loss value is calculated by using a total loss function, until a first termination condition is met, and an improved YOLO v4-tiny model corresponding to a minimum loss value or a maximum accuracy is taken as a first target detection model; Step S22: an image in the data set MS COCO is input to the first target detection model for training, a loss value is calculated by using the total loss function, until a second termination condition is met, and the first target detection model corresponding to the minimum loss value or the maximum accuracy is taken as a second target detection model; Step S23: an image in the data set Visdron is input to the second target detection model for training, a loss value is calculated by using the total loss function, until a third termination condition is met, and the second target detection model corresponding to the minimum loss value or the maximum accuracy is taken as a final target detection model.

5. The lightweight object detection method for drone and NVIDIA development board according to any one of claims 2-4, characterized in that, The specific calculation formula of the total loss function is: wherein L is a total loss function, L DIOU is a positioning loss function, L cls is a target confidence loss function, L class is a classification loss function, p(B, B gt ) is the Euclidean distance between the center point of the predicted bounding box B of the UAV image and the center point of the real bounding box B gt , c is the distance of the farthest vertex between the predicted bounding box of the UAV image and the real bounding box, and IOU is the intersection over union of the predicted bounding box and the real bounding box.

6. The lightweight object detection method for drone and NVIDIA development board of claim 1, wherein, The 2nd convolutional layer of the first hourglass type pooling module adopts a 1*1 convolutional kernel, a step is 1, and the number of filters is 32; the initial convolutional module is connected with the 1st dilated convolutional layer, the 1st dilated convolutional layer is connected with the 1st convolutional layer and the feature fusion layer Concat, the 1st convolutional layer is connected with the 2nd convolutional layer, the 2nd convolutional layer is connected with the feature fusion layer add, the feature fusion layer add is connected with the 2nd dilated convolutional layer, the 2nd dilated convolutional layer is connected with the feature fusion layer Concat, and the feature fusion layer Concat is connected with the maximum pooling layer; The second hourglass type pooling module comprises one hourglass type CSPNet module and one maximum pooling layer; wherein the second convolutional layer of the hourglass type CSPNet module adopts a 1*1 convolution kernel, the step length is 1, and the filter quantity is 64; the maximum pooling layer in the first hourglass type pooling module is connected with the first dilated convolutional layer in the second hourglass type pooling module, the first dilated convolutional layer is connected with the first convolutional layer and the feature fusion layer Concat respectively, the first convolutional layer is connected with the second convolutional layer, the second convolutional layer is connected with the first convolutional layer respectively and the feature fusion layer add, the feature fusion layer add is connected with the second dilated convolutional layer, the second dilated convolutional layer is connected with the feature fusion layer Concat, and the feature fusion layer Concat is connected with the maximum pooling layer; The second convolutional layer of the third hourglass type pooling module adopts a 1*1 convolution kernel, the step length is 1, and the filter quantity is 128; the maximum pooling layer in the second hourglass type pooling module is connected with the first dilated convolutional layer in the third hourglass type pooling module, the first dilated convolutional layer is connected with the first convolutional layer and the feature fusion layer Concat respectively, the first convolutional layer is connected with the second convolutional layer, the second convolutional layer is connected with the first convolutional layer respectively and the feature fusion layer add, the feature fusion layer add is connected with the second dilated convolutional layer, the second dilated convolutional layer is connected with the feature fusion layer Concat, and the feature fusion layer Concat is connected with the maximum pooling layer; The convolution kernel of the convolutional layer of the first pooling feature enhancement module is 1*1, the step length is 1, and the filter quantity is 128; the size of the maximum pooling layer is 3*3, and the step length is 2; the convolutional layer is connected with the maximum pooling layer in the first hourglass type pooling module, the convolutional layer is connected with the maximum pooling layer and the average pooling layer respectively, the average pooling layer is connected with the up-sampling layer, the up-sampling layer and the maximum pooling layer are connected with the superposition layer respectively, and the superposition layer is connected with the splicing module; The convolution kernel of the convolutional layer of the second pooling feature enhancement module is 1*1, the step length is 1, and the filter quantity is 256; the size of the maximum pooling layer is 3*3, and the step length is 1; the convolutional layer is connected with the maximum pooling layer in the second hourglass type pooling module, the convolutional layer is connected with the maximum pooling layer and the average pooling layer respectively, the average pooling layer is connected with the up-sampling layer, the up-sampling layer and the maximum pooling layer are connected with the superposition layer respectively, and the superposition layer is connected with the splicing module.

7. A target detection system based on a drone and a NVIDIA development board, characterized in that, The system comprises: a model construction module configured to construct an improved YOLO v4-tiny model; the improved YOLO v4-tiny model comprises an initial convolutional module, a first hourglass type pooling module, a second hourglass type pooling module, a third hourglass type pooling module, a first pooling feature enhancement module, a second pooling feature enhancement module, a splicing module, a first output module, and a second output module; The first hourglass type pooling module comprises one first hourglass type CSPNet module and one maximum pooling layer; wherein the first hourglass type CSPNet module comprises two dilated convolution layers with a dilated rate of 5, two convolution layers, a feature fusion layer add, and a feature fusion layer Concat; The second hourglass type pooling module comprises one second hourglass type CSPNet module and one maximum pooling layer; wherein the second hourglass type CSPNet module comprises two dilated convolution layers with a dilated rate of 3, two convolution layers, a feature fusion layer add, and a feature fusion layer Concat; The third hourglass type pooling module comprises one third hourglass type CSPNet module and one maximum pooling layer; wherein the third hourglass type CSPNet module comprises two dilated convolution layers with a dilated rate of 1, two convolution layers, a feature fusion layer add, and a feature fusion layer Concat; The first pooling feature enhancement module comprises one convolution layer, one maximum pooling layer, one average pooling layer, one up-sampling layer, and one superimposition layer; The second pooling feature enhancement module comprises one convolution layer, one maximum pooling layer, one average pooling layer, one up-sampling layer, and one superimposition layer; The training module is configured to train the improved YOLO v4-tiny model using a training set to obtain a final target detection model. The video stream acquisition module is configured to acquire a video stream collected by an unmanned aerial vehicle on-board camera using a video capture card; the video stream comprises a plurality of continuous images. The target detection module is configured to input the video stream into the final target detection model deployed on the NVIDIA development board to perform target detection and output a target detection result. 8.The lightweight object detection system of claim 7, wherein The training module specifically comprises: When any one of the data set Pascal VOC, the data set MS COCO, and the data set Visdron is used as a training set, the images in the training set are input into the improved YOLO v4-tiny model for training, a loss value is calculated using a total loss function, until a first termination condition is met, and the improved YOLO v4-tiny model corresponding to the minimum loss value or the maximum accuracy is selected as the final target detection model. 9.The lightweight object detection system of claim 8, wherein The specific calculation formula of the total loss function is: wherein L is a total loss function, L DIOU is a positioning loss function, L cls is a target confidence loss function, L class is a classification loss function, p(B, B gt ) is the Euclidean distance between the center point of the predicted bounding box B of the UAV image and the center point of the real bounding box B gt , c is the distance of the farthest vertex between the predicted bounding box and the real bounding box of the UAV image, and IOU is the intersection over union of the predicted bounding box and the real bounding box.

10. A target detection device based on a drone and a NVIDIA development board, characterized in that, The device comprises: An unmanned aerial vehicle on-board camera configured to collect a video stream; A video capture card connected to the unmanned aerial vehicle on-board camera and configured to perform format conversion on the collected image stream; An internal deployment NVIDIA development board configured to input the video stream into a final target detection model to perform target detection and output a target detection result; the final target detection model comprises an initial convolution module, a first hourglass type pooling module, a second hourglass type pooling module, a third hourglass type pooling module, a first pooling feature enhancement module, a second pooling feature enhancement module, a splicing module, a first output module, and a second output module. The first hourglass type pooling module comprises one first hourglass type CSPNet module and one maximum pooling layer; wherein the first hourglass type CSPNet module comprises: two dilated convolution layers with a dilated rate of 5, two convolution layers, a feature fusion layer add, and a feature fusion layer Concat; The second hourglass type pooling module comprises one second hourglass type CSPNet module and one maximum pooling layer; wherein the second hourglass type CSPNet module comprises: two dilated convolution layers with a dilated rate of 3, two convolution layers, a feature fusion layer add, and a feature fusion layer Concat; The third hourglass type pooling module comprises one third hourglass type CSPNet module and one maximum pooling layer; wherein the third hourglass type CSPNet module comprises: two dilated convolution layers with a dilated rate of 1, two convolution layers, a feature fusion layer add, and a feature fusion layer Concat; The first pooling feature enhancement module comprises: one convolution layer, one maximum pooling layer, one average pooling layer, one up-sampling layer, and one superimposition layer; The second pooling feature enhancement module comprises: one convolution layer, one maximum pooling layer, one average pooling layer, one up-sampling layer, and one superimposition layer.

Images