A method for detecting small targets in aerial images of a drone for embedded devices

Abstract

The application discloses a UAV aerial image small target detection method for embedded equipment. The application designs a high-efficiency computing aggregation network to extract the multi-scale features of an input image, adopts an efficient one-time aggregation structure, and introduces partial convolution, thereby improving the parameter utilization rate, reducing the calculation redundancy, memory access, and energy consumption; a multi-level adjacent feature fusion pyramid is adopted to perform feature fusion, that is, the semantic information of a lower level is fused first, then the semantic information of a higher level is introduced gradually, different weights are given to features of different levels through an adaptive spatial fusion structure, the information fusion degree between adjacent levels is improved, the calculation amount is reduced, and high accuracy is achieved; network training adopts a task alignment index based on a center prior cost as a standard for positive and negative sample distribution, simultaneously considers a classification score, a positioning IOU score, and a center distance between a predicted frame and a real frame, convergence stability is improved, and the matched positive sample region is more flexible.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a method for small target detection in drone aerial images for embedded devices. Background Technology

[0002] Due to their high mobility, flexibility, and low cost, unmanned aerial vehicles (UAVs) have become an indispensable part of many fields. Especially in military, agriculture, environmental protection, geographic mapping, communications, and rescue, UAV technology has played a crucial role. To further improve the functionality and practicality of UAVs, target detection technology has become one of the key research areas.

[0003] Object detection is a core task in computer vision, aiming to automatically identify and locate multiple objects of interest in an image. This task has wide applications in various real-world scenarios, such as autonomous driving, video surveillance, robot visual navigation, and drone aerial object detection. In recent years, breakthroughs in computer vision and artificial intelligence have provided new impetus for the development of object detection technology. The development of deep learning technology has enabled computers to understand and process image data in unprecedented ways. In particular, the emergence of Convolutional Neural Networks (CNNs) has achieved significant performance improvements in object detection tasks, demonstrating high accuracy in handling complex scenes. For example, the YOLO series of algorithms is one of the most representative real-time object detection algorithms. Proposed by J. Redmon et al. in 2016, the YOLO algorithm transforms the object detection problem into a regression problem. YOLO divides the input image into multiple grids, with each grid responsible for predicting the target object it contains and its bounding box. YOLO boasts a fast detection speed, achieving real-time object detection. YOLO achieves a good balance between speed and accuracy, performing exceptionally well in real-time applications.

[0004] Research on target detection in UAV aerial images has significant theoretical and practical implications. Target detection technology based on convolutional neural networks can enhance the autonomous flight and navigation capabilities of UAVs in complex environments. However, in practical applications, target detection in UAV aerial images faces numerous challenges. On the one hand, the visual features of target objects in the image may change significantly when the UAV is flying at high altitudes. Simultaneously, factors such as lighting, occlusion, and motion blur can also affect the performance of target detection algorithms. On the other hand, the embedded devices on UAVs have limited computing power, placing high demands on the number of parameters and computational efficiency of the target detection network.

[0005] Existing drone aerial target detection algorithms face multiple challenges, one of the main problems being the high computational cost and poor efficiency of deep neural networks. Current algorithms often struggle to meet real-time requirements when processing large-scale image data due to the massive number of parameters and complex computational processes of deep neural network models, especially when running on embedded devices.

[0006] Another problem is that the detection of target features changes drastically, affecting the detection results. When a drone flies at high altitude, the visual features of the target object in the image may change significantly. Factors such as lighting and occlusion can also affect the performance of the target detection algorithm. This makes it difficult for the algorithm to detect small targets in a timely and accurate manner in rapidly changing environments, limiting its practicality in real-world applications. Summary of the Invention

[0007] In view of this, the present invention provides a method for small target detection in UAV aerial images for embedded devices. It consumes little computing resources and has good detection results. It can be applied to target detection in UAV aerial images that runs efficiently on embedded devices with limited computing power, thereby improving the reliability and practicality of target detection in UAV aerial photography.

[0008] The present invention provides a method for small target detection in drone aerial images for embedded devices, comprising:

[0009] Step 1: Obtain the drone aerial image dataset and divide it into training and testing sets;

[0010] Step 2: Construct an object detection network based on the YOLOv7 network; the object detection network includes a feature extraction subnetwork, a feature fusion subnetwork, and an object prediction subnetwork;

[0011] The feature extraction subnetwork employs an efficient computational aggregation network to extract multi-scale features from the input image. The efficient computational aggregation network consists of multiple convolutional layers, an efficient computational aggregation module, and pooling layers stacked together. The efficient computational aggregation module consists of multiple CBS convolutional modules and a portion of the convolutional PConv module stacked together in a single aggregation structure. The convolutional results of the CBS convolutional layers and the portion of the convolutional PConv module are superimposed as the scale features extracted by the efficient computational aggregation module.

[0012] The feature fusion subnetwork adopts a multi-level neighbor feature fusion pyramid to gradually fuse the features of different scales obtained in step two from low to high levels, resulting in feature maps of different levels after fusion. During the fusion process, an adaptive spatial fusion structure is used to assign different weights to features of different scales.

[0013] The target prediction sub-network uses detection heads of different scales to extract feature information from the fused feature maps at different levels for target prediction.

[0014] Step 3: Set the network hyperparameters and train the network constructed in Step 2 to obtain the trained network;

[0015] During training, a label assignment method based on an improved center prior cost is used to assign positive and negative samples. Specifically, the m largest values ​​of t are selected as positive samples; t = s α ×u β ×r v Where s is the classification score, u is the location IOU score, and v = |x pred -x gt |The a priori cost centered on x pred x is the center point location of the predicted bounding box. gt Let α, β, and r be the center point of the true bounding box, and let r be the hyperparameters for adjusting the importance of the indicators.

[0016] Step 4: Target detection is performed on the drone aerial images to be detected based on the trained network.

[0017] Preferably, step one further includes data augmentation of the training set: adjusting the images in the training set to the same size, randomly selecting multiple images from the training set, merging parts of them into a large image, and adjusting the labels accordingly; randomly selecting two images from two different batches of training sets, mixing them according to certain weights, and interpolating the corresponding labels according to the same weights; finally, performing color gamut transformation on the images to enhance the training sample set and obtain the final training samples.

[0018] Preferably, the CBS convolution module consists of a two-dimensional convolution, a BN layer, and a silu activation function.

[0019] Preferably, the high-efficiency computation aggregation module consists of five convolutional layers connected together, wherein the first, third, and fifth layers are CBS convolutional modules, and the second and fourth layers are partially convolutional PConv modules; after the features extracted by the first, third, and fifth layers are superimposed using a concat layer, the number of channels is adjusted using the CBS convolutional modules to obtain the final feature output of the high-efficiency computation aggregation module.

[0020] Preferably, the feature extraction subnetwork extracts m features at different scales, namely p0, p1, p2, ..., p m In the feature fusion subnetwork, the lower-level p is first processed. m and p m-1 Feature fusion is performed to obtain the second-level fused feature c. m and c m-1 Then, higher-level features p are introduced. m-2 Feature fusion is performed to obtain the third-level fused feature s. m sm-1 and s m-2 Similarly, the highest-level feature p0 is introduced for feature fusion to obtain the m-th level fused features a0, a1, a2, ..., a m ;

[0021] During feature fusion, an adaptive spatial fusion structure is used to assign different weights to the features to be fused. Specifically, this involves using... This represents the eigenvector at position (i,j) from level n to level l. The resulting eigenvector is expressed as... but in, and Let represent the weights of the two features in the l-th level space, and These are learnable parameters, learned through convolution using 1x1 kernels. The obtained parameters To adjust oneself.

[0022] Preferably, in step four, the target prediction subnetwork further includes a small target detection head0, which performs small target detection on the highest-level feature map obtained by the feature fusion subnetwork.

[0023] Beneficial effects:

[0024] (1) When extracting multi-scale features of images, this invention adopts an efficient computational aggregation network. This network adopts an efficient one-time aggregation structure and introduces partial convolution. Both reduce memory access and computational redundancy, make full and effective use of information from all channels, improve parameter utilization, and reduce memory access and energy consumption. When feature fusion, a multi-level neighbor feature fusion pyramid is adopted. First, the semantic information of adjacent levels is fused in the region, and then higher-level semantic information is gradually introduced to highlight the feature expression of adjacent levels, reduce the semantic gap in feature fusion, and use an adaptive spatial fusion structure to perform weighted fusion of features at different levels, give higher weight to the feature fusion of adjacent levels, improve the degree of information fusion between adjacent levels, retain more useful information, avoid large semantic gaps between non-adjacent levels, improve the information utilization in multi-level transmission, reduce the amount of computation, and have high accuracy. During network training, a task alignment index based on center prior cost is used as the standard for positive and negative sample allocation. At the same time, the softened center prior cost, classification score and localization IOU score are considered, and their product is used as the basis for label allocation. An exponential factor is used to adjust the importance of the index. On the one hand, it can achieve stable convergence, and on the other hand, it can make the matched positive sample region more flexible. This invention achieves better real-time performance on embedded devices by reducing computational complexity and employing computationally efficient network structures and optimization strategies.

[0025] (2) The present invention adds a small target detection head0, which enhances the network’s detection effect on small targets and has a strong small target detection capability. Attached Figure Description

[0026] Figure 1 This is a flowchart of the method of the present invention;

[0027] Figure 2 This is a diagram of the overall network framework of the present invention;

[0028] Figure 3 This is a schematic diagram of the feature fusion neck network framework of the present invention;

[0029] Figure 4 This is a structural framework diagram of the feature extraction of the present invention;

[0030] Figure 5 This is a flowchart illustrating the deployment of a model using TensorRT in this invention;

[0031] Figure 6 This is the test set result output by this invention. Detailed Implementation

[0032] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] This invention provides a method for small target detection in drone aerial images for embedded devices, addressing the problems of low target detection accuracy, large model parameter count, and unsuitability for deployment on embedded devices. It provides aerial images under different lighting conditions, shooting angles, and scenes, trains a detection model through deep learning, and transforms it into a form easily deployable on embedded devices before performing target detection in drone aerial images. This method improves the accuracy of small target detection, reduces the number of model parameters, and enhances computational capabilities on embedded devices.

[0034] This invention is based on the YOLOv7 network. First, an efficient computational aggregation network is designed as the backbone network. This network adopts a single-aggregation network structure and introduces partial convolutions. Both reduce memory access and computational redundancy, effectively reducing the computational complexity and number of parameters of the network. Second, a multi-level proximity feature fusion pyramid is designed as the neck network. This neck structure first fuses feature information from adjacent levels and gradually introduces higher-level feature information, reducing semantic gaps in feature fusion. It also uses an adaptive spatial fusion structure to weightedly fuse features from different levels, resulting in low computational cost and high accuracy. Simultaneously, a label allocation method based on center prior cost is proposed for training to obtain the original object detection model, which makes training convergence more stable and faster. Finally, a transformation suitable for deployment on embedded devices is performed to obtain the final object detection model.

[0035] The flowchart of the method of this invention is as follows Figure 1 As shown, the specific steps include the following:

[0036] Step 1: Download the publicly available drone aerial image dataset and divide it into training and test sets according to the dataset's specifications;

[0037] The drone aerial image dataset used in this embodiment is VisDrone2019; the training set VisDrone2019-DET-train and the test set VisDrone2019-DET-test are divided according to the specified method of the dataset.

[0038] Step 2: Perform data augmentation on the training set within the divided dataset: resize the images in the training set to 640×640 pixels. Randomly select four images from the training set, merge portions of them into a single large image, and adjust the labels accordingly. Then, randomly select two images from two different batches of the training set, blend them according to certain weights, and interpolate the corresponding labels with the same weights. Finally, perform operations such as color gamut transformation on the images to augment the training sample set, obtaining the final training samples.

[0039] Step 3: Perform multi-scale feature extraction on the images in the training set. This invention employs an Efficient Computational Aggregation block (ECAblock) for feature extraction. The ECAblock backbone consists of multiple stacked ECAblock modules, convolutional layers, and pooling layers, as shown below. Figure 2 As shown.

[0040] In this embodiment, the input image first passes through four CBS convolutional layer modules to obtain a 160×160×128 feature map. Then, four efficient computational aggregation modules ECAblock1, ECAblock2, ECAblock3, and ECAblock4, and four MPConv max pooling layer modules are used to extract image information features, resulting in feature maps p0, p1, p2, and p3 with 128, 256, 512, and 1024 channels, respectively.

[0041] Among them, the feature extraction structure of the high-efficiency computation aggregation module is as follows: Figure 4 As shown, the high-efficiency computational aggregation module adopts an efficient single-pass aggregation structure in its network architecture. It only superimposes the features extracted from the preceding feature layers in the last layer. Its main structure consists of five convolutional layers, with highly computationally efficient partial convolutional structures introduced in the second and fourth layers to improve the computational efficiency of the feature extraction module. In the last layer, the features extracted by the branch structure are superimposed with the features extracted from the first, third, and fifth layers to obtain the final result. The high-efficiency computational aggregation module not only has the advantage of representing multiple features with multiple receptive fields, maintaining good feature extraction capabilities and thus good detection accuracy, but also avoids the problem of low efficiency associated with dense connections.

[0042] Specifically, such as Figure 4 As shown, this embodiment's high-efficiency computation aggregation module stacks multiple CBS modules and partially convolutional PConv. This stacking method reduces memory access cost (MAC) and benefits GPU computing. Partially convolutional PConv can simultaneously reduce computational redundancy and memory access, applying regular convolutions only on a portion of the input channels for spatial feature extraction without affecting the remaining channels. PConv makes better use of the device's computing power and is also very effective in extracting spatial features. The FLOPs of PConv are: In the commonly used part, r=c p With / c=1 / 4, PConv's FLOPs are only 1 / 16 of those of regular convolutions. PConv has lower memory access costs, i.e. For r = 1 / 4, it is only 1 / 4 of the value of a regular convolution.

[0043] The CBS convolutional layer module consists of two-dimensional convolution, BN layers, and silu activation functions, while the PConv module consists of max pooling layers and two-dimensional convolution.

[0044] Step 4: Fuse features at different scales obtained in Step 3 using a Multi-level Neighbor Feature Fusion Pyramid Network (MN-FPN). Its structure is as follows: Figure 3 As shown, the neck structure integrates feature information from adjacent levels and gradually introduces higher-level feature information, reducing the semantic gap in feature fusion. Furthermore, it uses an adaptive spatial feature fusion (ASFF) structure to perform weighted fusion of features from different levels.

[0045] The specific process of feature fusion in the multi-level neighbor feature fusion pyramid is as follows: First, the lower-level features p2 and p3 are fused at adjacent levels. The second-level fused features c2 and c3 are obtained through an adaptive spatial fusion structure. The feature map size is transformed by upsampling and downsampling, and then the channels are transformed by 1×1 convolution. The higher-level feature p1 is then introduced for adaptive spatial fusion to obtain the third-level fused features s3, s2 and s1. The feature map size is transformed by upsampling and downsampling, and then the channels are transformed by 1×1 convolution. Finally, the highest-level feature p0 is introduced for adaptive spatial fusion to obtain the output features a3, a2, a1 and a0, with 64, 128, 256 and 512 channels, respectively.

[0046] To suppress information contradictions between features at different levels, an adaptive spatial fusion structure is introduced in the multi-level feature fusion process. ASFF is used to assign different spatial weights to features at different levels, enhancing the importance of key levels and mitigating the impact of contradictory information from different objectives. This represents the eigenvector at position (i,j) from level n to level l. The resulting eigenvector is expressed as... Obtained through adaptive spatial fusion of multi-level features, and derived from a linear combination of feature vectors: in, and Let represent the weights of the two features in the l-th level space, and These are learnable parameters, learned through convolution using 1x1 kernels. The obtained parameters To adjust oneself;

[0047] Step 5: After extracting features from the feature map obtained in Step 4, input the extracted feature information into the corresponding size detection head for classification and regression prediction to achieve small target prediction.

[0048] Specifically, the feature maps a3, a2, and a1 obtained in step 4 are passed through feature extraction modules with different inputs. The extracted feature information is then input into the detection head Head1 (80×80×256), the detection head Head2 (40×40×512), and the detection head Head3 (20×20×1024) to complete the target prediction for small target detection.

[0049] To address the problem of insufficient feature learning for small targets and enhance the detection performance of small targets, this invention adds a small-sized (160×160×128) detection head Head0. The feature map a0 obtained in step 4 is processed through a feature extraction and enhancement module to obtain a refined feature map. This feature map is then input into Head0 to improve the detection performance of small targets.

[0050] Step 6: Set network hyperparameters and train the network. During network training, this invention adopts a label allocation method based on center prior cost to allocate positive and negative samples. It proposes a task alignment index based on center prior cost as the standard for positive and negative sample allocation, while also considering softened center prior cost, classification score, and localization IOU score. Their product is used as the basis for label allocation, and an exponential factor is used to adjust the importance of the index. This ensures stable convergence and makes the matched positive sample regions more flexible. The formula is as follows: t = s α ×u β ×r v Where s is the classification score, u is the location IOU score, and v = |x pred -x gt | represents the center prior cost, and α, β, r are hyperparameters that adjust the importance of the indicators. When assigning labels, the m largest t samples are selected as positive samples.

[0051] Step 7: Export the trained model file as an ONNX file and use TensorRT to deploy and generate the Engine model on an embedded device, NVIDIA Jetson NX.

[0052] The TensorRT inference engine generation process is as follows: Figure 5As shown. After the model trained in step 6 is ported to an edge device with a configured development environment, it is compiled using GCC and then processed by TensorRT to perform inter-layer fusion, tensor fusion, model pruning, and weight quantization to obtain the TensorRT inference engine. Inter-layer fusion is the vertical integration of the network structure, combining Conv, BN, and SiLU into a single layer, namely the CBS layer; tensor fusion is the horizontal integration of the network structure, combining layers with the same input tensors and performing the same operations into a single layer; model pruning reduces concat layers, directly feeding the input of concat layers into the operation, reducing transmission throughput. Quantization refers to converting FP32 data into FP16 or INT8 data for processing, which improves speed during inference. After this series of optimization processes, the TensorRT inference engine is obtained.

[0053] Step 8: Input the test set from Step 1 into the Engine model from Step 7 to obtain the test results of UAV small target detection.

[0054] Furthermore, the test set is used to perform inference calculations using the Engine model generated by TenosrRT to generate detection boxes for targets in drone aerial images.

[0055] Figure 6 The test set results output in this embodiment show that the target detection algorithm proposed in this invention has a good detection effect on UAV aerial images and also has a relatively good detection effect on dense small targets.

[0056] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for small target detection in drone aerial images for embedded devices, characterized in that, include: Step 1: Obtain the drone aerial image dataset and divide it into training and testing sets; Step 2: Construct an object detection network based on the YOLOv7 network; The target detection network includes a feature extraction subnetwork, a feature fusion subnetwork, and a target prediction subnetwork; The feature extraction subnetwork employs an efficient computational aggregation network to extract multi-scale features from the input image. The efficient computational aggregation network consists of multiple convolutional layers, an efficient computational aggregation module, and pooling layers stacked together. The efficient computational aggregation module consists of multiple CBS convolutional modules and a portion of the convolutional PConv module stacked together in a single aggregation structure. The convolutional results of the CBS convolutional layers and the portion of the convolutional PConv module are superimposed as the scale features extracted by the efficient computational aggregation module. The feature fusion subnetwork adopts a multi-level neighbor feature fusion pyramid to gradually fuse the features of different scales obtained in step two from low to high levels, resulting in feature maps of different levels after fusion. During the fusion process, an adaptive spatial fusion structure is used to assign different weights to features of different scales. The target prediction sub-network uses detection heads of different scales to extract feature information from the fused feature maps at different levels for target prediction. Step 3: Set the network hyperparameters and train the network constructed in Step 2 to obtain the trained network; During training, a label assignment method based on an improved center prior cost is used to assign positive and negative samples. Specifically, the m largest values ​​of t are selected as positive samples; t = s α ×u β ×r v Where s is the classification score, u is the location IOU score, and v = |x pred -x gt |The a priori cost centered on x pred x is the center point location of the predicted bounding box. gt Let α, β, and r be the center point of the true bounding box, and let r be the hyperparameters for adjusting the importance of the indicators. Step 4: Target detection is performed on the drone aerial images to be detected based on the trained network.

2. The method as described in claim 1, characterized in that, Step one also includes data augmentation of the training set: adjusting the images in the training set to the same size, randomly selecting multiple images from the training set, merging parts of them into a large image, and adjusting the labels accordingly; randomly selecting two images from two different batches of training sets, mixing them according to certain weights, and interpolating the corresponding labels according to the same weights; finally, performing color gamut transformation on the images to augment the training sample set and obtain the final training samples.

3. The method as described in claim 1, characterized in that, The CBS convolution module consists of two-dimensional convolution, BN layers, and a silu activation function.

4. The method as described in claim 1 or 3, characterized in that, The efficient computing aggregation module consists of five convolutional layers connected together. The first, third, and fifth layers are CBS convolutional modules, and the second and fourth layers are partially convolutional PConv modules. The features extracted by the first, third, and fifth layers are superimposed using a concat layer, and the number of channels is adjusted using the CBS convolutional modules to obtain the final feature output of the efficient computing aggregation module.

5. The method as described in claim 1, characterized in that, The feature extraction subnetwork extracts m features at different scales, namely p0, p1, p2, ..., p m In the feature fusion subnetwork, the lower-level p is first processed. m and p m-1 Feature fusion is performed to obtain the second-level fused feature c. m and c m-1 Then, higher-level features p are introduced. m-2 Feature fusion is performed to obtain the third-level fused feature s. m s m-1 and s m-2 Similarly, the highest-level feature p0 is introduced for feature fusion to obtain the m-th level fused features a0, a1, a2, ..., a m ; During feature fusion, an adaptive spatial fusion structure is used to assign different weights to the features to be fused. Specifically, this involves using... This represents the eigenvector at position (i,j) from level n to level l. The resulting eigenvector is expressed as... but in, and Let represent the weights of the two features in the l-th level space, and These are learnable parameters, learned through convolution using 1x1 kernels. The obtained parameters To adjust oneself.

6. The method as described in claim 1, characterized in that, In step four, the target prediction sub-network further includes a small target detection head0, which performs small target detection on the highest-level feature map obtained by the feature fusion sub-network.

Images