Unmanned aerial vehicle image-based small object detection method for target areas

Abstract

The present invention relates to the technical field of deep learning and computer vision. Disclosed is an unmanned aerial vehicle image-based small object detection method for target areas. The present invention crops images of obvious small objects in certain target areas, and annotates the small objects of different categories to form a raw training and testing dataset, so as to ensure the accuracy of data required in the early stage of the algorithm and further ensure the scientificity of the algorithm; uses the computing capability of an improved YOLOv7 detection model to collect image features of different degrees in the dataset, the improved YOLOv7 detection model using YOLOv7 as a basic model and adding to a neck network an MS-CET module, which is constituted by an improved self-attention mechanism and convolution module SPPCSP, and a BHC-FB module, which is constituted by bidirectional mixed convolution modules NConv and RPConv connected in parallel; and finally fuses different feature layers as a final judgment basis of an unmanned aerial vehicle for small object detection in the target areas, to further check the accuracy of the algorithm and criteria for dataset selection, thereby improving recognition accuracy.

Description

A target region small target detection method based on unmanned aerial vehicle images TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and computer vision, and particularly relates to a target region small target detection method based on unmanned aerial vehicle images. BACKGROUND

[0002] In recent years, it has important theoretical and practical significance in various scenes such as daily monitoring, unmanned aerial vehicle scene analysis, pedestrian detection, and traffic sign detection in automatic driving. In particular, with the wide application of unmanned aerial vehicle aerial photography technology, it is a challenging task to efficiently and accurately identify small target objects in the aerial view and realize the landing of small target detection algorithms in aerial scenes combined with unmanned aerial vehicle hardware devices. The application potential of the application is extraordinary. However, some small target detection images in difficult unmanned aerial vehicle scenes, such as small targets in scale imbalance scenes, small targets with sparse spatial distribution characteristics, dim scenes, blurred targets, small targets with dense occlusion and cluttered backgrounds, and the like, are difficult to be accurately detected by traditional conventional target detection methods. Therefore, it is necessary to develop an effective small target detection method.

[0003] At the same time, YOLOv7 is one of the representative algorithms for target detection, and good results have been achieved in target recognition in ordinary scenes, but the effect is not ideal in the above-mentioned complex scenes.

[0004] Therefore, in order to solve the above technical problems, it is necessary to provide a target region small target detection method based on unmanned aerial vehicle images, which improves the structure of YOLOv7, makes up for the shortcomings of traditional YOLOv7 in complex scenes, and greatly improves the feature extraction ability and detection accuracy. SUMMARY

[0005] To solve the above technical problems, the present application provides a target area small target detection method based on unmanned aerial vehicle images, by designing a novel compact network, innovatively introducing Dilated Transfomer, by utilizing multi-head self-attention mechanism, we design a new module MS-CET that focuses on the spatial distribution sparsity of small targets, realizing the capture of global sparsity features in the spatial distribution of small targets in the field of view of the unmanned aerial vehicle; in order to handle the large-scale variance of the target, the present application additionally designs a multi-level cross-feature structure BHC-FB, which faces the large-scale variance of the target, and performs feature fusion of the shallow high-resolution feature map in the multi-layer pyramid structure, enhancing the local feature perception ability of the model for small targets; considering the long-range dependence of CNN, the present application proposes NConv and RPConv modules in the BHC-FB structure, performs bidirectional mixed convolution precision compensation, and effectively reduces redundant features and reduces the computational amount of the model; we pay attention to the fact that the loss function in the existing network has a penalty difference for small target samples of different qualities, therefore the present application proposes a WIoU loss function as a substitute, processes the sample learning imbalance caused by the extremely small target, reduces the false detection and missed detection phenomenon in the dense scene, and improves the prediction accuracy of the model.

[0006] The target area small target detection method based on unmanned aerial vehicle images provided by the present application comprises:

[0007] S1, acquiring an unmanned aerial vehicle image;

[0008] S2, improving the Y0L0v7 model;

[0009] S3, training the improved Y0L0v7 model;

[0010] Specifically, the image is intercepted from the video picture collected by the unmanned aerial vehicle and input to the improved YOLOv7 detection model for target detection, so as to obtain the category of the target, wherein the improved YOLOv7 detection model comprises a backbone network, a neck network and a prediction network;

[0011] The intercepted image is first input to the backbone network for feature extraction, then all the extracted features are fused in the neck network, and finally the fused features are input to the detection layers of large, medium and small scales of the prediction network for detection, so as to obtain the detection result;

[0012] The improved YOLOv7 detection model takes YOLOv7 as a basic model, adds MS-CET modules and BHC-FB modules in the neck network, the MS-CET module is composed of an improved self-attention mechanism and a convolution module SPPCSP, the BHC-FB module is composed of a bidirectional mixed convolution module NConv and an RPConv in parallel, and finally different feature layers are fused, then a first two-dimensional convolution layer, a batch normalization layer, a nonlinear function, a channel shuffle module and two two-dimensional convolution layers are sequentially included, and a Sigmoid activation function is used.

[0013] Preferably, the MS-CET module is combined with the multi-scale convolution module SPPCSP of YOLOv7 through a multi-head self-attention mechanism, after the image is input into the MS-CET module and passes through a CBS module, it is divided into two branches (CBS, CBS+DilatedBlock) for processing, concentrating on learning the spatial distribution characteristics of small targets, and realizing the global sparseness feature area attention of small targets.

[0014] Preferably, the DilatedBlock module includes:

[0015] The input feature map is linearly projected to three different spaces, a sliding window with a size of w x w is used, and a self-attention mechanism operation is performed on different heads in different spaces with different scales, and finally the three spaces are input into a linear layer for feature aggregation.

[0016] Preferably, the CBS module is composed of Conv2d, BatchNorm2d and SiLU in series.

[0017] Preferably, the MS-CET module includes:

[0018] The high-resolution feature layer with a size of 160 x 160 is output in the Backbone stage of YOLOv7, on the basis of retaining more small target information in high resolution, the bidirectional mixed convolution module NConv and the RPConv are introduced, the low-resolution features with strong semantic information and the high-resolution features with high texture information are fused and complementary, the multi-level structure is used to continuously aggregate and abstract local features, so that the model effectively focuses on the features in the center position of the feature map.

[0019] Preferably, the NConv module is a normal convolution with a convolution kernel of 3 x 3.

[0020] Preferably, the RPConv module includes:

[0021] Through 3X3 two-dimensional convolution, through 1X1 two-dimensional convolution kernel, the scale is kept invariable, the network depth is increased from the perspective of full connection layer, the information interaction is realized, more nonlinearity is introduced, and the model feature representation ability is enhanced, then the feature map X is subjected to forward_split_cat operation to obtain X.

[0022] Preferably, the improved YOLOv7 detection model replaces the complete intersection ratio loss function into the WIoU loss function.

[0023] Compared with the related art, the target region small target detection method based on the unmanned aerial vehicle image has the following beneficial effects:

[0024] The application provides a target region small target detection method based on an unmanned aerial vehicle image.

[0025] By using the public data set, the accuracy of the data required by the early algorithm is ensured, and the scientificity of the algorithm is ensured, the improved YOLOv7 computing power is used, different degrees of image features in the data set are collected, finally different feature layers are fused as the judgment basis of the final target recognition, and the accuracy of the algorithm and the standardization of the data set are further tested, so as to improve the recognition precision. BRIEF DESCRIPTION OF DRAWINGS

[0026] Fig. 1 is a flowchart of the target region small target detection method based on the unmanned aerial vehicle image provided by the application;

[0027] Fig. 2 is the architecture of the analysis framework proposed by the application;

[0028] Fig. 3 is a diagram for visualizing the sparse attention mechanism of the network using GradCAM technology;

[0029] Fig. 4 is a flowchart of the image input into the MS-CET module;

[0030] Fig. 5 is a BHC-FB structure diagram, P3, P4 and P5 represent feature maps of different resolution levels, (·) represents an activation function, the upper branch module is represented by NConv, and the lower branch module is represented by RPConv;

[0031] Fig. 6 is a feature level visualization of the ATBHC-YOLO and YOLOv7 neck network with the same input: (a) and (d) are RGB inputs; (b) and (c) are features of YOLOv7; (e) and (f) are features of ATBHC-YOLO; the features are up-sampled to the same scale as the input image for comparison, (b) and (e) are low-level features, and (c) and (f) are high-level features in a deeper level;

[0032] Fig. 7 is a schematic diagram of a loss function WloU;

[0033] Fig. 8 is a comparison diagram of WloU visualization small target detection;

[0034] Fig. 9 is a detection effect diagram of four typical problem scenes in the Vis-Drone2021 dataset;

[0035] Fig. 10 is a visualization test result diagram of the ATBHC-YOLO algorithm based on the VEDAI dataset;

[0036] Fig. 11 is a visualization test result diagram of the ATBHC-YOLO algorithm on the DIOR dataset. DETAILED DESCRIPTION

[0037] The application will be further described below in conjunction with the drawings and embodiments.

[0038] As shown in Fig. 1, it is a target detection determination method based on unmanned aerial vehicle images in the embodiments of the present application, which comprises the following steps:

[0039] S1, acquiring an unmanned aerial vehicle image;

[0040] S2, improving the YOLOv7 model;

[0041] S3, training the improved YOLOv7 model.

[0042] Specifically, an image is intercepted from a video picture collected by an unmanned aerial vehicle and input to an improved YOLOv7 detection model for target detection to obtain the category of the target, wherein the improved YOLOv7 detection model comprises a backbone network, a neck network and a prediction network;

[0043] The intercepted image is first input to the backbone network for feature extraction, then all the extracted features are fused in the neck network, and finally the fused features are input to the detection layers of large, medium and small scales of the prediction network for detection to obtain the detection result.

[0044] The improved YOLOv7 detection model takes YOLOv7 as a basic model, adds MS-CET modules and BHC-FB modules to the neck network, the MS-CET module is composed of an improved self-attention mechanism and a convolution module SPPCSP, the BHC-FB module is composed of parallelly connected bidirectional hybrid convolution modules NConv and RPConv, different feature layers are finally fused, then a first two-dimensional convolution layer, a batch normalization layer, a nonlinear function, a channel shuffle module and two two-dimensional convolution layers are sequentially included, and a Sigmoid activation function is adopted.

[0045] In the present application, the small target detection model based on deep learning is called ATBHC-YOLO, which can detect difficult small targets in the multi-scale field of view of the unmanned aerial vehicle, and it includes three main improvements: (1) MS-CET; (2) BHC-FB; (3) WIoU loss function optimization. The structure of the proposed model is shown in Figure 2, and the input image is fed into the backbone network for feature learning. In the Neck part, features are extracted from different resolution levels of the backbone network, and more distribution information conducive to small target detection is captured through the BHC-FB and MS-CET modules designed by us. Finally, the whole system is trained in an end-to-end manner through WIoU loss function learning loss.

[0046] The baseline of the small target detection model based on deep learning consists of three parts: Backbone, Neck and Head. In this model, first, we extract the features of the initial input image through Backbone, and output feature maps of different resolution sizes; second, in the top part of Neck, we design the MS-CET module to aggregate more semantic features conducive to small target detection according to the different sizes of the input feature map, fully extract features of different granularities and resolutions of the spatial distribution sparsity of small targets, and in the middle part of Neck, we construct the BHC-FB structure, additionally increase the high-resolution feature map branch for small targets, realize the introduction of the bidirectional precision convolution module structure focusing on the features of the center position of the feature map through multi-level cross-feature layer fusion, and reduce redundant target features. At the same time, we pay attention to the loss of small target texture information with the deepening of network training, and use the residual structure to solve the feature loss problem using shallow feature maps. Finally, we input the enhanced features to the Head part, use the adaptive optimization loss function WIoU for the small target difficult detection samples of the data set, further improve the sensitivity of small target detection based on the dynamic non-monotonic focusing mechanism bounding box regression principle, and reasonably measure the scale to reduce the information error caused by the limited features of small targets, and solve the false detection and missed detection phenomenon in dense scenes. Through the above way, our network can better extract the features and position information of small targets for efficient detection, and remove redundant features in the training process.

[0047] The improved aspects are described below:

[0048] 1、MS-CET

[0049] We designed the MS-CET module before the convolution module in the Neck part of the Neck part to better learn the global sparse features of small targets in the spatial distribution. As shown in the heat map comparison in FIG. 3, the effective receptive field range obtained by adding the MS-CET module is more comprehensive, which can capture the context semantic information, and the self-attention mechanism can dynamically generate corresponding attention weights according to the input image. However, DilatedTransformer does not have multi-scale feature interaction capability, and has certain limitations in small target detection, and the effect is not satisfactory. Although ordinary CNN has been widely applied in multi-scale feature learning, CNN relies on convolution, pooling, alignment and other mechanisms to learn local features, and is limited by the constraint of inductive bias, but the area that really needs to be focused on in our task is small and sparse, which leads to insufficient detection capability of the model for small targets.

[0050] Based on the above shortcomings, we combined the multi-head self-attention mechanism of DilatedTransformer with the multi-scale convolution module SPPCSP of YOLOv7. As shown in the heat map visualization in FIG. 3, the former can model the long-range dependence of the context, effectively aggregate the semantic information of different scales within the attention receptive field, and the latter uses the translational invariance and local sensitivity of the convolution kernel to extract local features. After adjusting the resolution of the output feature map, the two are aggregated to enhance the small target deep feature representation capability.

[0051] Design of MS-CET.SPPCSP module is located at the top of the Neck part of the YOLOv7 model, as shown in FIG. 4. First, we input the feature map X of the Backbone, which is processed through a CBS (composed of Conv2d, BatchNorm2d and SiLU) module, and then use the C3_DilateBlock module designed by us. Through the following equation 1, the channels of the feature map are divided into 3 heads with different sparsity, that is, we select the appropriate sparsity head n value as 2, 3, and 4 through experiments. By obtaining the corresponding queries, keys and values by linear projection. X = LP(Concat[h1,...,hi]), 1≤i≤n, (1)

[0052] where LP represents linear projection of the input feature map X to three different spaces to obtain Q, K and V matrices as shown in equation 2. Each row of the three matrices represents a single query / key / value feature vector. Q = [q1,...,qn], K = [k1,...,kn], V = [v1,..., vn], (2)

[0053] Then we define the query centering at the original feature map (m, n) is selected using a sliding window of size w x w, and the sparse nature of different scales is utilized in different heads with different dilation rates to model the long-rangedependencies in the feature map X. The above process is described by Equation 3 as follows. i = SD(Q i , K i , V i , r i ), 1≤i≤n, (3)

[0054] Here hi is the i-th selected head, and Qi, Ki, Vi are the corresponding slices of the feature map sent to the i-th head for SD (sliding window) operation. ri is the dilation rate of the corresponding head.

[0055] xn is the corresponding total amount of output from DilatedAttention of different scales, and Att is the DilatedAttention operation. H, W are the height and width of the feature map, respectively, and Ki, Vi are the selected keys and values from the feature map. qnis all the query vectors. KTQ represents the input vector correlation calculated using Q and K. T is the transpose operation, which facilitates vector dot multiplication to obtain the weight based on input data relationship modeling, and the similarity is calculated by querying the key set.

[0056] Finally, all hi are concatenated together and input to a linear layer for feature aggregation. As shown in Equation (4).

[0057] Finally, the feature aggregation of the ordinary convolution block and the Dilateformer is realized through the residual structure, and then we use four different scale sizes of maximum pooling to realize the fusion of different feature scale information, and the pooling kernel size is set to 9x9, 5x5, 3x3, 1x1. In addition, we use the CBS module to adjust the resolution of the output feature map to match the feature map after the Dilateformer, so as to perform Concat operation again through the residual structure for output. The whole MS-CET module aggregates the self-attention mechanism of the Dilateformer and the convolution module SPPCSP, learns more small target global sparse features through multi-scale method, reduces the weight bias learning of non-small target samples, and uses the sparse nature of different scales to perform multi-head self-attention mechanism operation, so that the feature extraction is more concentrated and in-depth, in order to cope with the challenge of spatial distribution sparsity in small target detection.

[0058] 2、BHC-FB

[0059] The main idea of this branch is to detect targets in higher resolution feature maps, combined with the multi-level cross-feature layer fusion branch we proposed to learn more information about small targets. In this invention, we are inspired by the above to detect small targets in the large field of view image of the UAV. We find that the feature layer resolution size of the general branch BPF-N used by this model for ordinary size targets is: 80x80, 40x40, 20x20. For ordinary targets, through these three feature detection layers, suitable feature information can be extracted, but for small targets, the spatial distribution on the UAV image has the characteristic of locality, which leads to the fact that low resolution feature layers cannot meet the requirements of detecting small targets.

[0060] Based on the above shortcomings, as shown in FIG. 5, we use the high-resolution feature layer of 160x160 output by the Backbone stage in YOLOv7. On the basis of retaining more small target information in high resolution, we introduce the bidirectional mixed convolution module NConv and RPConv, which realizes the fusion and complementation of low-resolution features with strong semantic information and high-resolution features with high texture information, and uses the multi-level structure to continuously aggregate and abstract local features, so that the model can effectively focus on the features in the center position of the feature map. On the basis of not destroying the spatial structure and local features of the image, redundant features are discarded, so as to guide the model to better train and enhance the local feature sensitivity of the model to small target detection.

[0061] Design of BHC-FB. As we analyzed above, high-resolution feature maps can extract more texture features of small targets, as shown in the heat map visualization of FIG. 6. The BHC-FB module focuses on the key areas in the image channel through the above cross-branch feature sharing structure, refines local features, and helps us to more accurately locate and detect small targets in the UAV image. First, we use a multi-level way to realize the bidirectional bridging structure of the convolution mixing module. The bidirectional mixing structure further enhances the feature refinement in the model training process by fusing two convolution modules with different advantages. The NConv module is a normal convolution with a 3x3 convolution kernel. Combined with our small target detection task, NConv focuses on more local features of small targets. This is because NConv has a small convolution kernel that can extract more fine-grained features, and the resulting computational load is also relatively small.

[0062] Then, we designed a special RPConv module, as shown in the Partical_Conv structure diagram in FIG. 5, and next we will introduce the advantages of this module applied to the BHC-FB module in detail. First, we pass through the high-resolution feature map output by the Backbone, and increase the network depth in the perspective of the fully connected layer while introducing more nonlinearity to enhance the model feature representation ability under the condition of keeping the feature map scale unchanged through Conv1x1. Then we perform the forward_split_cat operation on the feature map X to obtain X1, as shown in equation (6),

[0063] where n_div represents the selected part of the feature map channel, and X1 is the corresponding part of the channel for convolution. The main purpose of this operation is to select only a part of the feature map X1 using a Conv3x3 convolution operation, and the input and output dimensions remain unchanged, to obtain the partially convolved feature map X1. The reason for this is that there are a lot of redundant feature maps in all channels of the feature map, and if a part of the feature map is convolved, the computational complexity will be effectively reduced, as shown in equation (7):

[0064] Here, h and w are the height and width of the input feature map, respectively, k is the number of convolution kernels, and cx1 is the part of the channel to be convolved. Finally, we splice the feature map channels of the two branches to obtain the output result as shown in equation (8),

[0065] At this time, we obtain the feature map U after Partial_Conv, and use two Conv1x1 in series to fully utilize the channel information, and use the skip connection to reintroduce the unprocessed feature map X. Through Partial_Conv, more local non-redundant feature information is extracted for Upsample operation, and the feature fusion of the two branches is performed. This compensates for the feature information that may be lost by partial channel convolution, and the BN layer and the activation function SiLu are placed at the back to ensure feature diversity and reduce model complexity. In addition, we also introduce feature maps of different resolutions at each bidirectional branch, and adjust the resolution to 160x160, which is consistent with the high-resolution feature map in the bidirectional branch, for feature fusion. The advantage of this is to supplement the insufficient semantic information in the high-resolution feature map, which is beneficial to our bidirectional branch to balance local feature extraction and feature refinement enhancement.

[0066] To refine the local feature extraction of small targets, we designed the RPConv module, which focuses on the local characteristics of small targets and only convolves part of the feature map channels to reduce computational complexity. Finally, through the bidirectional bridging of RPConv and NConv, we implemented the multi-level cross-feature fusion operation of BHC-FB combining different resolution feature maps, further enhancing the feature refinement and extraction capability, while reducing the model's computational complexity without compromising the model's feature extraction capability.

[0067] 3、WIoU loss function optimization

[0068] To address the problem of low-quality labeled data in the selected dataset, which exacerbates the model's punishment of low-quality samples, we replaced the CIoU loss function of YOLOv7 with the WIoU loss function for bounding box regression prediction. Conventional bounding box loss functions continuously fit the optimal result by defining the calculation method based on the geometric elements of the predicted box and the true box, such as the CIoU used by YOLOv7. However, in the dataset, we noticed that the anchor box annotations in the dataset have differences, especially the insensitivity of the efficient fitting loss function to small target data annotations, which greatly interferes with model convergence, resulting in poor feature learning effect. Therefore, we chose the WIoU loss function based on the dynamic non-monotonic focusing mechanism for bounding box regression, as shown in Figure 7. The WIoU calculation formula is:

[0069] WIoU assigns a reasonable gradient gain by designing a non-monotonic change in gradient gain r under increasing loss value. α and δ are hyperparameters. Specifically, the gradient gain is dynamically allocated based on the anchor box's outlier factor. The outlier calculation formula is:

[0070] where β is the outlier factor value, L*IoU is the gradient gain value, LIoU

[0071] is the momentum moving average.

[0072] Then, R WIoU ∈[1,e) as a penalty term to strengthen the loss of ordinary quality anchor boxes, L IoU ∈[0,1] as the IoU loss, used to weaken the penalty term of high-quality anchor boxes, which can strengthen the focus on the center distance when the anchor box and the predicted box are highly overlapped; in addition, * indicates that it does not participate in backpropagation, effectively preventing the network model from producing gradients that are not easy to converge. The loss function parameter visualization is shown in Figure X.

[0073] Nw and Nh are the width and height of the smallest rectangle formed by the true and predicted boxes, respectively; Wgt and Hgt are the width and height of the true box, respectively; w and h are the width and height of the predicted box, respectively.

[0074] This loss method pays more attention to samples of ordinary quality, thereby improving the model's ability to locate objects. For the target detection task in the UAV aerial scene, small objects with high proportions increase the difficulty of detection. WIoU can dynamically optimize the balance of weighting small targets, solve the BBR balance problem between samples of good quality and poor quality, and improve the detection performance of the model on ordinary quality samples.

[0075] The performance of the method proposed in the application will be evaluated through experiments.

[0076] The data set used in this experiment is Vis-Drone2021. The VisDrone data set is collected by the AISKYEYE team of the Machine Learning and Data Mining Laboratory of Tianjin University in China, and is suitable for target detection tasks under the perspective of unmanned aerial vehicles. The data set consists of 10209 images with a resolution of 2000 pixels x 1500 pixels, covering multiple scenes under different weather and lighting conditions. These images are captured by different unmanned aerial vehicle cameras from different places in China, containing rich urban and rural environments, capturing a large number of objects such as pedestrians, vehicles, tricycles, etc. The advantage of this data set is worth attention, that is, during the process of collecting pictures by unmanned aerial vehicles, the influence of different scenes and weather is considered, which greatly approximates the real environment. Specifically, the data set is divided into 6471 training images, 548 validation images, and 1610 test images, with 10 categories: pedestrians, people, bicycles, cars, vans, trucks, tricycles, sunshade tricycles, buses, and electric machines. In this experiment, we analyzed that the data set has target size imbalance, that is, less than 16x16 pixels accounts for 12.05%, greater than 16x16 and less than 32x32 pixels accounts for 32.65%, and greater than 32x32 pixels accounts for 55.3%. In addition, due to the characteristics of unmanned aerial vehicle shooting, there are difficulties such as sparse distribution of small targets in space, local dense occlusion, and different lighting conditions. Implementation and hyperarameters: In the experiment, we use the PyTorch framework to implement our method, use CUDA11.5 for operation acceleration, the GPU is NVIDIA A6000 with 80GB of graphics memory, and YOLOv7 is used as the baseline. In order to compare the performance of the algorithm fairly, all methods use the same hyperparameter settings for training and validation, as shown in Table 1, and the early stopping strategy is used in the training process to avoid model overfitting.

[0077] Table 1: Experimental detial of the ATBHC-YOLO algorithm

[0078] To evaluate the performance of our ATBHC-YOLO model on UAV aerial images, we use the evaluation indicators commonly used to evaluate the performance of target detection, Precision, Recall, GFLOPs, mAP@.5. Precision represents the proportion of correctly identified parts among all identified parts, reflecting the model's ability to correctly classify; Recall is the ratio of correctly identified parts to the total number of parts in the test set, reflecting the model's ability to comprehensively detect targets; GFLOPs is the number of floating-point operations of the network model; mAP is the average of AP for each class in the data set, and the larger the mAP, the better the detection accuracy of the model. mAP is calculated based on the intersection over union (IoU) of the predicted box and the labeled box, and the calculation formula is as follows:

[0079] mAP is divided into mAP0.5, mAP0.5:0.95, mAP0.5 according to different IoU thresholds. mAP0.5 is the mAP when IoU is 0.5. Under the condition that IoU is 0.5, the average precision (AP) of each class is calculated, and then the average of all classes is calculated.

[0080] To verify the performance of different modules proposed in this paper under the same experimental conditions, we use the test set of VisDrone2021 to further verify the relationship between the designed modules and the influence of a single module on the algorithm. In this section, we conduct ablation experiments, and the evaluation results of each module of the UAV small target detection network are shown in Table 2. Compared with the basic model, the detection accuracy is improved by 3.1 percentage points after adding MS-CET and BHC-FB. The subsequent improvement of the loss function, Wiou, also increases the detection accuracy, and the modules can achieve a good balance in detection accuracy, showing good detection effect. MS-CET captures the sparse characteristics of small targets in the spatial distribution of the UAV field of view, effectively modeling the long-range dependencies of small targets in the sparse distribution, and using DilatedTransformer as the main guide, supplemented by some ordinary CNN modules, using the sparsity of small targets in different scales to perform self-attention mechanism operations, thereby making small targets more visible, and the detection accuracy of small targets is effectively improved by 1.7%.

[0081] Table 2: For ablation experiments on the Visdrone dataset,↑indicates higher results,↓indicates lower results,and the best results have been are shown in bold.

[0082] The BHC-FB module reduces the bias of the model for large and medium-sized target detection by introducing high-resolution feature maps for small targets, starting from locality, designing a bidirectional mixed convolution cascade structure for small target feature fusion, enhancing the sensitivity of small target fine-grained feature information, and focusing on refining redundant small target features. The small target detection accuracy is further improved to 3.1% and the computational complexity is reduced. This method extracts more detailed small target local feature information such as texture and edge features of small targets by additionally increasing high-resolution branches for feature fusion. This method focuses on the local information of small targets, fully balances the spatial and semantic information of small targets, and is suitable for small target detection tasks under the field of view of unmanned aerial vehicles.

[0083] Finally, we found a more suitable WIoU loss function for small target detection through experiments. Based on the quality gap between samples, we dynamically allocate gradient gain based on the outlying factor of anchor boxes to dynamically optimize the BBR balance problem between poor samples and good samples, so that the anchor box can more accurately and quickly locate the target, further improving the small target detection accuracy to 3.5%, thereby improving the model generalization ability and robustness.

[0084] Overall, through ablation experiments, we improved the model compared to the basic model, and the detection accuracy improved by 3.5 percentage points. This method starts from the characteristics of small targets in the field of view of unmanned aerial vehicles, which are sparse and local in space distribution, and can capture small target features with more spatial details, achieving good small target detection results.

[0085] In this section, we compare our proposed method with the state-of-the-art algorithms. We compare the ATBHC-YOLO detection model with one-stage algorithms CDNet, TPH-YOLOv5, UFPMP-Det, DCRFF, Yolov5l, Yolov8l, and two-stage algorithms FasterRCNN, Cascade-RCNN on the dataset Visdrone-test-dev.

[0086] The compared experimental models are currently more advanced small target detection models. Through experimental data comparison, the model of the present application is obviously superior to other several most advanced models in the small target detection index mAP50. The main reason is that in most small target detection categories, the detection accuracy of the ATBHC-YOLO proposed by us is better than other detection models with the same input resolution. The research results show that the MS-CET module designed by us is based on spatial distribution sparse features, and performs multi-scale feature learning on high-resolution feature maps, extracts more sparse features through self-attention mechanism, and enhances the feature bias learning of small targets. At the same time, BPF-RB also helps our model to process more local feature loss problems caused by the dramatic change of target scale, and the improved network and algorithm improve the ability of small target feature representation, which demonstrates the influence of local features on small target detection accuracy. In addition, the appropriate loss function is more accurate and rapid to locate the detection target we need. The experimental results are shown in Table 3.

[0087] Table 3: Performance of ATBHC-YOLO and other algorithms on the dataset Visdrone-test-dev for different categories

[0088] In the detection of each category of the data set, the method of the present application has obtained good competitive performance, especially on the van, tricycle and awning-tricycle with unbalanced data categories and scarce samples, the mAP50 index reaches a high level, and in general, under the complex unmanned aerial vehicle field of view, ATBHC-YOLO significantly improves the small target detection performance, and more clearly and quickly captures small targets, which has certain competitive performance compared with other algorithms.

[0089] In order to further prove the effectiveness and robustness of the algorithm ATBHC-YOLO of the present application, the present application specially selects two mainstream data sets VEDAI[] and DIOR[] similar to VisDrone, and proposes the advantages of the algorithm based on qualitative angle analysis, as shown in Figures 10 and 11. Among them, the VEDAI data set is used for multi-class vehicle detection in aerial images, which contains vehicles, and these vehicles, in addition to small size, also show different changes such as multi-directionality, light / shadow changes, reflection or occlusion. DIOR is a large benchmark data set for optical remote sensing image target detection, containing 23463 images and 192472 instances, covering 20 object classes. Through the analysis of Tables 4 and 5, it is found that the method proposed by ATBHC-YOLO has certain generalization ability and robustness in small target recognition. In the unknown data set, its performance is better than that of other models, showing obvious accuracy advantage.

[0090] Table 4: Performance of ATBHC-YOLO and other algorithms on the dataset VEDAI-test for different categories

[0091] Table 5: of ATBHC-YOLO and other algorithms on the dataset DIOR-test for different categories.

[0092] Result analysis:

[0093] The experimental results fully verify the effectiveness of each module proposed by us, and the comparability of the overall method proposed by us and other most advanced algorithms.

[0094] Next, we will detect small targets in different UAV visual field scenes and visualize the key experimental results. Fig. 9 is the detection effect diagram of four typical problem scenes in the Vis-Drone2021 dataset. In the scale imbalance scene, (a) compared with (b), the occlusion and scale change are severe, not only most of the small targets can be detected, but also the occluded small targets can be captured. Especially in the pedestrian gathering area, the occluded targets can still be accurately detected. In the high-altitude sparse distribution scene, since the UAV visual field is at high altitude, (d) the baseline cannot detect the extremely small and sparse targets in the playground, and the overall method proposed in (c) can accurately detect the small and sparse targets and has better accuracy. In addition, the accuracy of other targets is also improved. In the dim and blurred scene, the target to be detected is limited by the night light and buildings, (f) uses the baseline detection, and the vehicle parking area in the dark light will have the problem of missed detection and low accuracy, and the method proposed in (e) improves the light quality, false detection and missed detection, and two trucks in the baseline detection are detected in the method, and the accuracy of the surrounding vehicles is improved. In the crowd dense scene, the most prominent problem is boundary confusion and body blur, and the method proposed in the application can better handle the problems of incomplete target, false detection and low target confidence compared with the baseline. In (h), most of the targets in the red circle are occluded and difficult to identify, and after improvement in (g), such problems can be better solved.

[0095] The model and algorithm proposed in the application can reduce the small targets that are not detected or predicted as false categories or have uncertainty, improve the detection accuracy, and reduce the probability of poor detection in many different UAV scenes, improve the characteristics of local and sparseness of small targets in spatial distribution, and select a suitable loss function to reduce the insensitivity of small target detection.

[0096] In summary, compared with the prior art, the method and module proposed in the application can effectively improve the small target detection accuracy in the UAV visual field, solve the problems of small target characteristics, such as few pixels, local blur, occlusion, boundary confusion, etc.

[0097] The application has strong competitiveness compared with the current most advanced network, effectively reduces the characteristics of local and sparseness of small targets in spatial distribution, and reduces the probability of problems such as missed detection, false detection, low detection accuracy, etc.

[0098] The present application is based on the spatial distribution sparsity of small targets, attempts to add a multi-head self-attention mechanism, sets different scales of sparsity, carries out long-range dependence of context, and models more effective features; secondly, based on the spatial distribution locality of small targets, by increasing the cascade branch of higher resolution, the more refined features of small targets are extracted, and the representation ability of small targets is further enhanced; finally, in order to reduce the insensitivity of the model to small target detection, the present application adds a loss function Wiou, optimizes the loss of most redundant features, reduces the calculation amount, and learns more bias for small targets.

[0099] Through the above work, experiments prove that when the input image resolution is 640*640 pixels, the detection accuracy mAP50 and mAP of the ATBHC-YOLO algorithm are improved by 3.5 percentage points and 3.2 percentage points respectively relative to the benchmark algorithm YOLOv7, although the detection speed decreases and the calculation amount increases, the detection accuracy is greatly improved, and the model has good performance.

[0100] The above is only an embodiment of the present application, and does not limit the patent scope of the present application, any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1.A method for detecting small targets in a target area based on images of a UAV, characterized in that, The method comprises: S1, acquiring an image photographed by a UAV; S2, improving a Y0L0v7 model: S3, training the improved Y0L0v7 model; Specifically, an image is intercepted from a video picture collected by a UAV, and is input to an improved YOLOv7 detection model for target detection to obtain the category of the target, wherein the improved YOLOv7 detection model comprises a backbone network, a neck network and a prediction network; The intercepted image is first input to the backbone network for feature extraction, then all the extracted features are fused in the neck network, and finally the fused features are input to detection layers of three scales of large, medium and small in the prediction network for detection to obtain a detection result. The improved YOLOv7 detection model, that is, taking YOLOv7 as a basic model, adds an MS-CET module and a BHC-FB module to the neck network, the MS-CET module is composed of an improved self-attention mechanism and a convolution module SPPCSP, the BHC-FB module is composed of a bidirectional hybrid convolution module NConv and an RPConv in parallel, and finally different feature layers are fused, then a first two-dimensional convolution layer, a batch normalization layer, a nonlinear function, a channel shuffle module and two two-dimensional convolution layers are sequentially included, and a Sigmoid activation function is used. 2.The target region small target detection method based on UAV images according to claim 1, characterized in that, The MS-CET module is combined by a multi-head self-attention mechanism and a multi-scale convolution module SPPCSP of YOLOv7, after the image is input to the MS-CET module and passes through a CBS module, it is processed by two branches (CBS, CBS+DilatedBlock), which concentrates on learning the spatial distribution features of small targets and realizes the global sparsity feature region attention of small targets. 3.The target region small target detection method based on UAV images according to claim 1, characterized in that, The DilatedBlock module comprises: The input feature map is linearly projected to three different spaces, a sliding window with a size of w x w is used, and a self-attention mechanism operation is performed on different heads in different spaces by using the sparsity in different scales, and finally the three spaces are input to a linear layer for feature aggregation. 4.The method of claim 1, wherein, The CBS module is composed of Conv2d, BatchNorm2d and SiLU in series. 5.The target region small target detection method based on UAV images according to claim 1, characterized in that, The MS-CET module comprises: A high-resolution feature layer with a size of 160 x 160 is output in the Backbone stage of YOLOv7, on the basis of retaining more small target information in high resolution, a bidirectional hybrid convolution module NConv and an RPConv are introduced, the low-resolution features with strong semantic information and the high-resolution features with high texture information are fused and complementary, the local features are continuously aggregated and abstracted by using a multi-level structure, and the model effectively focuses on the features in the center position of the feature map. 6.The target region small target detection method based on UAV images according to claim 1, characterized in that, The NConv module is a normal convolution with a convolution kernel of 3 x 3. 7.The target region small target detection method based on UAV images according to claim 1, characterized in that, The RPConv module comprises: Through 3X3 two-dimensional convolution, through 1X1 two-dimensional convolution kernel, the scale is kept invariable, the network depth is increased from the perspective of full connection layer, the information interaction is realized, more nonlinearity is introduced, and the model feature representation ability is enhanced, then the feature map X is subjected to forward_split_cat operation to obtain X. 8.The target region small target detection method based on UAV images of claim 1, wherein, The improved YOLOv7 detection model replaces the complete intersection ratio loss function with the WIoU loss function.

Images