A lightweight automatic detection method for UAV reconnaissance of ground targets

Abstract

The present invention relates to the field of image processing and machine vision technology, and provides a lightweight automatic detection method for drone reconnaissance of ground targets, including S1, model establishment and training: using an improved SSD algorithm to perform feature extraction, labeled box regression and category prediction on drone reconnaissance images to complete model training; the improved SSD algorithm uses Squeeze Net to replace VGG as the basic network, and Squeeze Net adds several convolutional layers; S2, using the trained model to perform target detection on drone reconnaissance images. The present invention uses the Squeeze‑net algorithm as a network pre-algorithm to extract features of the target, avoiding missed detection or false detection caused by the detection algorithm due to the inability to change the sliding window size and step size steplessly or the unclear target features, thereby increasing the detection accuracy; integrating ARM, GPU, and NPU for joint reasoning to achieve multi-hardware parallel computing and improve computing efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and machine vision, and in particular to a lightweight automatic detection method for unmanned aerial vehicle (UAV) to detect ground targets. Background Art

[0002] In recent years, as an emerging combat force, drones have played an irreplaceable role in intelligent combat conditions. Vigorously developing drone equipment technology is of great strategic significance for improving the combat capabilities of troops. As one of the key technologies for drones to perform reconnaissance and strike missions, automatic target detection technology can provide strong support for drone visual navigation, target tracking, target positioning and precision strike technologies. Efficient and accurate detection algorithms can effectively reduce the burden on ground operators, improve reconnaissance capabilities and rapid response combat effectiveness.

[0003] Traditional drone aerial image target detection algorithms require image preprocessing, and then perform sliding window traversal operations on the entire image. By preliminarily judging the possible location of the target, some features of the target are manually designed, such as the common histogram of oriented gradient (HOG) feature, scale-invariant feature transform (SIFT) feature, Haar feature (Haar-link feature) and speeded up robust feature (SURF), etc. Finally, the features are sent to the support vector machine (SVM) or Adaboost classifier for classification to complete the detection task. However, there are great disadvantages in manually designed features. In the design process, it will rely too much on past experience, and the algorithm will not perform well in unfamiliar scenes. The robustness of the detection algorithm is not strong, which greatly hinders the application of the detection algorithm.

[0004] Convolutional neural networks are deep in terms of network layers and the number of channels in a single-layer network, and both have a large amount of data that needs to be processed. More than 90% of the computational requirements of convolutional neural networks come from the stereo convolution operator (three-dimensional), which needs to perform stereo convolution operations on all data in the network point by point. When the CPU is subjected to such a huge amount of computation, the computational efficiency is very low.

[0005] In the UAV video image processing system, the automatic detection technology for detecting ground targets currently faces the following problems:

[0006] 1) Traditional target detection algorithms retrieve targets by traversing the entire image through a sliding window, which affects both detection efficiency and accuracy. Based on existing hardware processing, it is difficult to ensure real-time performance.

[0007] 2) Network training consumes too much energy and model training costs are too high;

[0008] 3) Due to the limitation of CPU hardware conditions, the algorithm advantages cannot be fully utilized under limited resource conditions, and the computing resources are allocated unreasonably, resulting in low computing efficiency. Summary of the invention

[0009] In view of the characteristics of UAV video images and the defects of existing domestic technologies in UAV reconnaissance ground target detection, the present invention comprehensively considers algorithm performance and deployment adaptability. The technical problems to be solved by the present invention are at least one of the following technical problems:

[0010] 1. The traditional target detection algorithm has problems with feature extraction and accuracy when sliding the window across the entire image;

[0011] 2. Traditional deep learning algorithm models are large and difficult to deploy in embedded devices;

[0012] 3. Irrational allocation of computing resources, low computing efficiency, and difficulty in performing multi-hardware parallel computing;

[0013] 4. The number of pixels of the target to be detected is small, and the texture is not obvious, which may easily lead to false detection or missed detection;

[0014] 5. Real-time issues in the detection process.

[0015] The present invention adopts the following technical solution:

[0016] A lightweight automatic detection method for unmanned aerial vehicle (UAV) to detect ground targets, the method comprising:

[0017] S1. Model establishment and training: The improved SSD algorithm is used to perform feature extraction, annotation box regression and category prediction on the drone reconnaissance image to complete the model training; the improved SSD algorithm uses Squeeze Net to replace VGG16 as the front-end basic network to extract features of the target image; Squeeze Net adds several convolutional layers;

[0018] S2. Use the trained model to perform target detection on drone reconnaissance images.

[0019] Furthermore, the specific steps of step S1 are as follows:

[0020] S1.1 The drone reconnaissance image is input and the SSD algorithm after replacing VGG16 with Squeeze net is used for feature extraction;

[0021] S1.2 Based on the multi-scale detection in the SSD algorithm, the target in the UAV reconnaissance image is detected and identified to obtain a candidate area;

[0022] S1.3 selects the final target region through non-maximum suppression based on the classification scores of all candidate regions.

[0023] Furthermore, during model training, multi-hardware parallel computing is adopted; the multi-hardware includes ARM, GPU, and NPU.

[0024] Further, in step S1, the NPU processes conventional convolution and full connection calculations, the ARU processes data scaling and data mapping, and the CPU processes the remaining data calculations.

[0025] Furthermore, integer quantization is performed during model training, fixed-point parameters are saved with floating points, and int8 is used instead of float32 used in the original weights.

[0026] Furthermore, in step S1, two convolutional layers fire10 and fire11 are added to Squeeze Net, and the feature maps output by four convolutional layers fire5, fire9, fire10, and fire11 are selected for classification and regression calculation of detection boxes.

[0027] An information data processing terminal for realizing the above-mentioned lightweight automatic detection method for unmanned aerial vehicle to detect ground targets.

[0028] A computer-readable storage medium includes instructions, which, when executed on a computer, enable the computer to execute the above-mentioned lightweight automatic detection method for unmanned aerial vehicle to detect ground targets.

[0029] The beneficial effects of the present invention are:

[0030] 1) The present invention has high operating efficiency. When only the graphics card GTX1050 is used, the operation time on the video image with a resolution of 1920*1080 is 20ms, and real-time processing can be performed.

[0031] 2) The present invention uses the Squeeze-net algorithm as a network pre-algorithm to extract target features, thereby avoiding missed detection or false detection caused by the inability of the sliding window size and step size to change steplessly or the unclear target features, thereby increasing the detection accuracy.

[0032] 3) The present invention performs integer quantization when training the network model, using int8 instead of float32 used in the original weights, thereby obtaining nearly 4 times the network acceleration and ensuring the detection accuracy while reducing the size.

[0033] 4) The present invention integrates ARM, GPU, and NPU for joint reasoning, rationally allocates computing resources, realizes multi-hardware parallel computing, and improves computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The figure shows a training flow chart of a lightweight automatic detection method for a UAV to detect ground targets according to an embodiment of the present invention.

[0035] Figure 2 Shown is the fire module structure diagram.

[0036] Figure 3 Shown is a schematic diagram of the Squeeze-SSD feature pyramid structure in an embodiment. DETAILED DESCRIPTION

[0037] The specific embodiments of the present invention will be described in detail below in conjunction with specific drawings. It should be noted that the technical features or combinations of technical features described in the following embodiments should not be considered isolated, and they can be combined with each other to achieve better technical effects.

[0038] The present invention uses a combination of the Squeeze-SSD target detection algorithm and multi-hardware collaborative quantization training to complete the automatic detection of ground targets in UAV reconnaissance. First, a target detection model training scheme based on integer quantization and multi-hardware collaborative computing during training is adopted, and then the trained Squeeze-SSD model is used to perform target detection on UAV reconnaissance video images. The overall algorithm training process is as follows: Figure 1 shown.

[0039] The algorithm can be summarized into the following steps:

[0040] 1) Use the SSD algorithm after replacing VGG16 with Squeeze net to extract features from the image input;

[0041] 2) Based on the multi-scale detection concept in the SSD algorithm, the targets in the drone reconnaissance images are detected and identified;

[0042] 3) Perform non-maximum suppression on the annotation boxes where targets may exist, eliminate redundant annotation boxes of the same target, and reduce the false detection rate of the algorithm;

[0043] 4) Use quantization during training to reduce the size of the model and the number of model parameters;

[0044] 5) Use NPU to process conventional convolution and full connection calculations without occupying CPU resources. Assign data scaling and data mapping related calculations to APU, and use CPU for other data shape transformations. Improve computing efficiency from the perspective of hardware utilization layout and resource allocation.

[0045] The key technologies involved in the algorithm implementation process are described in detail below with the help of flowchart.

[0046] 1. Squeeze Net Network Structure

[0047] The lightweight convolutional neural network SqueezeNet is characterized by its fire module structure. Figure 2 As shown, the structure consists of a squeeze layer and an expand layer. Among them, the squeeze layer is full of convolution kernels of size 1×1. After the feature map passes through the squeeze layer, the number of feature channels is reduced, and the number of parameters is reduced by 8 / 9 compared to the convolution kernel of size 3×3. At the same time, the output of the squeeze layer will be used as the input of the expand layer. The reduction in the number of channels will greatly reduce the number of parameters of the entire network; the expand layer contains 1×1 convolution kernels and 3×3 convolution kernels. The feature map passes through the two convolution kernels respectively and is output through the concat operation. After passing through the fire module structure, the size of the feature map will not change, but compared to the feature map that only passes through a 3×3 convolution kernel, its size remains unchanged but the number of parameters is greatly reduced. This is the fundamental reason why Squeeze Net is used as the front-end feature extraction network of the detection network in the present invention.

[0048] 2. Principle of Target Detection in UAV Reconnaissance Images

[0049] The SSD algorithm uses multi-scale feature maps for prediction in target detection tasks, and performs prediction box regression and category prediction in multiple feature layers. The core concepts of the SSD algorithm can be summarized as follows:

[0050] (1) Generate prior frames from feature maps of different scales

[0051] In the convolutional neural network structure, due to the convolution and pooling operations, the size of the feature map will continue to decrease as the network deepens. The shallow convolution layer has a high resolution, and the extracted features contain more detailed information, such as edges, shapes and other geometric information; the deep convolution layer has a low resolution, and the extracted features contain more semantic information, and the information expressed is more abstract. Due to the different sizes of targets in the detection task, if only the feature information of the shallow convolution layer is used for prediction, the detection effect of small targets will be poor. Therefore, the SSD algorithm uses multi-scale feature maps for prediction, that is, using deep convolution features with a larger receptive field to predict large targets, and using shallow convolution features with a smaller receptive field to predict small targets. This also happens to meet the characteristics of the present invention in detecting targets in drone aerial videos.

[0052] (2) Use convolutional layers to complete detection and discard fully connected layers

[0053] In the process of researching the target detection algorithm, the fully connected layer in the network has not been abandoned, which leads to a large number of parameters in the network. Too many parameters make it impossible to effectively transfer features backward and make the model training more difficult. Therefore, the SSD algorithm changes the original fully connected layer to a convolutional layer and adds several convolutional layers after the network, which is more conducive to the expression of features in image data.

[0054] (3) Set the prior frame according to the task

[0055] The SSD algorithm inherits the concept of anchor in Faster R-CNN. It sets a priori boxes of different scales and different aspect ratios for each feature layer. During training, it compares the position difference between the predicted bounding box and the priori box and calculates the loss function of the two. The rule for generating the priori box is: first, a rectangular box is generated based on the center of each unit, where the minimum side length of the square priori box is min_size and the maximum side length is Then configure a parameter in the network as the aspect ratio (aspect_ratio), and combine the above minimum and maximum values ​​to get the length and width of the two rectangular boxes, which are and The maximum and minimum values ​​in each feature map are determined by formula (1).

[0056]

[0057] Among them, m represents the number of feature maps, k ranges from 1 to m, and s k Indicates the ratio of the prior box size to the image, s min and max Represent the minimum and maximum values ​​of the ratio respectively.

[0058] The SSD algorithm generates prior boxes of different sizes on feature maps of multiple scales, matches and calculates these prior boxes and prediction boxes to achieve the division of positive and negative samples, and finally completes the training and updates the parameters in the model.

[0059] 3. Fusion of Squeeze Net and SSD Algorithms

[0060] The SSD algorithm consists of three parts: a basic network, a detection network, and a classification network. The fusion of Squeeze Net and the SSD algorithm first replaces the basic network of the detection algorithm from VGG (visual geometry group) to Squeeze Net, which greatly reduces the number of parameters in the entire detection architecture. In the previous section, it was mentioned that the SSD detection algorithm needs to detect the scale feature map, so the selection of the feature layer is critical. According to the design concept of the original SSD algorithm, the present invention adds two convolutional layers, fire10 and fire11, on the basis of Squeeze Net, and selects the feature maps output by the four convolutional layers of fire5, fire9, fire10 and fire11 for classification and regression calculation of the detection frame.

[0061] In a specific embodiment, the original squeezenet process is as follows (the sequence number represents the order of the process):

[0062] (1) Convolutional layer conv1: The input is 224×224×3, the output is 111×111×96, the convolution kernel is 7×7, the step size is 2, and it is used for feature extraction, local perception, and parameter sharing.

[0063] (2) maxpool1 pooling layer: input 111×111×96, output 55×55×96, convolution kernel 3×3, step size 2. The pooling layer is also called the subsampling layer, which is used for feature selection, reducing the number of features and thus reducing the number of parameters. Maxpool is the maximum pooling, taking the maximum value of the four eigenvalues ​​as the new eigenvalue.

[0064] (3) Fire2: Input 55×55×96, output 55×55×128. It consists of a squeeze part and an expand part. The squeeze part is composed of a group of continuous convolutions, and the expand part is composed of a group of continuous convolutions and a group of continuous convolutions.

[0065] (4)fire3: Input 55×55×128, output 55×55×128.

[0066] (5)fire4: Input 55×55×128, output 55×55×256.

[0067] (6) maxpool4 pooling layer: input 55×55×256, output 27×27×256, convolution kernel 3×3, stride 2.

[0068] (7)fire5: Input 27×27×256, output 27×27×256.

[0069] (8)fire6: Input 27×27×256, output 27×27×384.

[0070] (9)fire7: Input 27×27×384, output 27×27×384.

[0071] (10)fire8: Input 27×27×384, output 27×27×512.

[0072] (11) maxpool8 pooling layer: input 27×27×512, output 13×12×512, convolution kernel 3×3, stride 2.

[0073] (12)fire9: Input 13×12×512, output 13×13×512.

[0074] (13)conv10 convolutional layer: the input is 13×13×512, the output is 13×13×1000, the convolution kernel is 1×1, and the stride is 1.

[0075] (14) avgpool10 pooling layer: the input is 13×13×1000, the output is 1×1×1000, the convolution kernel is 13×13, and the step size is 1. The average of the four eigenvalues ​​is taken as the new eigenvalue.

[0076] The squeeze-SSD process after using squeezenet instead of vgg16 is modified as follows:

[0077] The original network structure after fire9 of squeezenet is deleted, and maxpool10, fire10 and two convolutional layers (conv12 and 13) are added to connect the fourth layer after the fifth block of the original SSD network. The features extracted by the conv10 convolutional layer are further subjected to the maximum pooling and convolution operations.

[0078] Among them, the fire layers 4.5.8.9.10 and the second layers of the convolutional layers conv12 and 13 are connected as a multi-level structure for generating prediction frames. The number of prior frames in each layer is 6, and the size of the convolution kernel is 3×3. After generating the prior frames and 4 parameters (x, y, w, h) of each level, the results are stacked. The center point of each grid is added with its corresponding x_offset and y_offset. The result after adding is the center of the prediction frame, and then the length and width of the prediction frame are calculated by combining the prior frame with h and w. Finally, the position and confidence of the entire prediction frame are output, and then the redundant detection frames are filtered out through NMS.

[0079] Although several embodiments of the present invention have been given herein, those skilled in the art should understand that the embodiments of the present invention may be modified without departing from the spirit of the present invention. The above embodiments are merely exemplary and should not be used as a limitation on the scope of the present invention.

Claims

1. A lightweight automatic detection method for unmanned aerial vehicle reconnaissance of ground targets, characterized in that: The method comprises: S1. Model establishment and training: The improved SSD algorithm is used to perform feature extraction, annotation box regression and category prediction on the drone reconnaissance image to complete the model training; the improved SSD algorithm uses Squeeze Net to replace VGG16 as the front-end basic network to extract features of the target image; Squeeze Net adds several convolutional layers; The squeeze-SSD process after using squeezenet instead of vgg16 is modified as follows: The original network structure after fire9 of squeezenet is deleted, and maxpool10, fire10 and two convolutional layers conv12 and conv13 are added to connect the fourth layer after the fifth block of the original SSD network; the features extracted by the conv10 convolutional layer are further subjected to maximum pooling and convolution operations; Among them, fire layers 4.5.8.9.10 and the second layers of convolutional layers conv12 and 13 are connected as a multi-level structure for generating prediction frames. The number of prior frames in each layer is 6, and the size of the convolution kernel is 3×3. After generating the prior frames and 4 parameters (x, y, w, h) of each level, the results are stacked. The center point of each grid is added with its corresponding x_offset and y_offset. The result after adding is the center of the prediction frame, and then the length and width of the prediction frame are calculated by combining the prior frame with h and w. Finally, the position and confidence of the entire prediction frame are output, and then the redundant detection frames are filtered out through NMS. S2, use the trained model to perform target detection on drone reconnaissance images; During model training, multi-hardware parallel computing is used; the multi-hardware includes ARM, GPU, and NPU; The NPU handles conventional convolution and full connection calculations, the ARU handles data scaling and data mapping, and the CPU handles the remaining data calculations; Integer quantization is performed during model training, fixed-point parameters are saved with floating points, and int8 is used instead of float32 used in the original weights.

2. The lightweight automatic detection method for unmanned aerial vehicle reconnaissance of ground targets as claimed in claim 1, characterized in that: The specific steps of step S1 are as follows: S1.1 The drone reconnaissance image is input and the SSD algorithm after replacing VGG16 with Squeeze net is used for feature extraction; S1.2 Based on the multi-scale detection in the SSD algorithm, the target in the UAV reconnaissance image is detected and identified to obtain a candidate area; S1.3 selects the final target region through non-maximum suppression based on the classification scores of all candidate regions.

3. The lightweight automatic detection method for unmanned aerial vehicle reconnaissance of ground targets as claimed in claim 1, characterized in that: In step S1, two convolutional layers fire10 and fire11 are added to Squeeze Net, and the feature maps output by four convolutional layers fire5, fire9, fire10, and fire11 are selected for classification and regression calculation of detection boxes.

4. A computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute the lightweight automatic detection method for unmanned aerial vehicle reconnaissance of ground targets as described in any one of claims 1 to 3.

Images