A method for detecting small targets in target areas based on UAV images

Abstract

The invention relates to the technical fields of deep learning and computer vision, and discloses a method for detecting small targets in a target area based on unmanned aerial vehicle images. The method ensures the accuracy of data required by an early algorithm by cropping images of obvious small targets in certain target areas and marking small targets of different categories, thereby ensuring the scientific nature of the algorithm. The computing power of an improved YOLOv7 detection model is used to collect image features of different degrees in a data set. The improved YOLOv7 detection model uses YOLOv7 as a basic model, and adds an MS-CET module and a BHC-FB module to a neck network. The MS-CET module is composed of an improved self-attention mechanism and a convolution module SPPCSP. The BHC-FB module is composed of a bidirectional mixed convolution module NConv and a RPConv connected in parallel. Finally, different feature layers are fused as a judgment basis for the final detection of small targets in the target area of ​​the unmanned aerial vehicle, and the accuracy of the algorithm and the standardization of the data set selection are further verified, thereby improving the recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and computer vision technology, and in particular to a method for detecting small targets in a target area based on drone images. Background Art

[0002] In recent years, it has important theoretical and practical significance in various scenarios such as daily monitoring, drone scene analysis, pedestrian detection, and traffic sign detection in autonomous driving. In particular, with the widespread application of drone aerial photography technology, it is a challenging task to efficiently and accurately identify small target objects in the aerial photography field of view and implement small target detection algorithms in aerial photography scenes in combination with drone hardware equipment. Its application results have extraordinary potential. However, some images of small target detection in difficult drone scenes, such as small targets in scenes with uneven scales, small targets with sparse spatial distribution characteristics, dim scenes, and blurred targets, small targets with dense occlusion and cluttered backgrounds, and other typical small targets in drone images are difficult to be accurately detected by traditional conventional target detection methods. Therefore, it is necessary to develop effective small target detection methods.

[0003] At the same time, YOLOv7 is one of the representative algorithms for target detection. It has achieved good results in target recognition in ordinary scenarios, but the effect is not ideal in the above-mentioned complex scenarios.

[0004] Therefore, in order to solve the above technical problems, it is necessary to provide a small target detection method in the target area based on UAV images. By improving the YOLOv7 structure, the shortcomings of traditional YOLOv7 in complex scenes are made up, and the feature extraction capability and detection accuracy are greatly improved. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a method for detecting small targets in target areas based on drone images. By designing a novel and compact network, Dilated Transformer is innovatively introduced. By utilizing the multi-head self-attention mechanism, we design a new module MS-CET that focuses on the sparsity of the spatial distribution of small targets, so as to capture the global sparsity features in the spatial distribution of small targets in the drone field of view. In order to deal with the drastic change of large-scale variance of the target, the present invention additionally designs a multi-level cross-feature structure BHC-FB. Facing the large-scale variance of the target, the shallow high-resolution feature map is feature-fused in a multi-layer pyramid structure to enhance the model's local feature perception ability of small targets. Considering the remote dependence of CNN, the present invention proposes NConv and RPConv modules in the BHC-FB structure to perform bidirectional mixed convolution accuracy compensation, effectively reduce redundant features, and reduce the amount of model calculation. We note that the loss function in the existing network has different penalties for the different quality of small target samples, so the present invention proposes the WIoU loss function as a substitute to deal with the sample learning imbalance caused by extremely small targets, reduce the phenomenon of false detection and missed detection in dense scenes, and improve the prediction accuracy of the model.

[0006] The present invention provides a method for detecting small targets in a target area based on drone images, the method comprising:

[0007] S1, obtain the image taken by the drone;

[0008] S2, Improvement of Y0L0v7 model;

[0009] S3, training improved Y0L0v7 model;

[0010] Specifically, images are captured from video images captured by the drone and input into an improved YOLOv7 detection model for target detection to obtain the category of the target. The improved YOLOv7 detection model includes a backbone network, a neck network, and a prediction network.

[0011] The captured image is first input into the backbone network for feature extraction, then all the extracted features are fused in the neck network, and finally the fused features are input into the large, medium and small scale detection layers of the prediction network for detection to obtain the detection results;

[0012] The improved YOLOv7 detection model takes YOLOv7 as the basic model, adds MS-CET module and BHC-FB module to the neck network, the MS-CET module consists of an improved self-attention mechanism and a convolution module SPPCSP, the BHC-FB module consists of a bidirectional mixed convolution module NConv and RPConv in parallel, and finally fuses different feature layers, and then sequentially includes a first two-dimensional convolution layer, a batch normalization layer, a nonlinear function, a channel shuffle module and two two-dimensional convolution layers, and adopts a Sigmoid activation function.

[0013] Preferably, the MS-CET module is a combination of a multi-head self-attention mechanism and a multi-scale convolution module SPPCSP of YOLOv7. After the image is input into the MS-CET module and passes through a CBS module, it is divided into two branches (CBS, CBS+DilatedBlock) for processing, focusing on in-depth learning of the spatial distribution characteristics of small targets and realizing global sparsity feature area attention of small targets.

[0014] Preferably, the DilatedBlock module includes:

[0015] The input feature map is linearly projected into three different spaces. A sliding window of size w×w is used to perform self-attention mechanism operations on different heads in different spaces using sparsity at different scales. Finally, the three spaces are input into the linear layer for feature aggregation.

[0016] Preferably, the CBS module is composed of Conv2d, BatchNorm2d and SiLU connected in series.

[0017] Preferably, the BHC-FB module comprises:

[0018] By utilizing the Backbone stage in YOLOv7 to output a high-resolution feature layer of size 160×160, and on the basis of retaining more small target information at high resolution, the bidirectional hybrid convolution modules NConv and RPConv are introduced. By realizing the fusion and complementation of low-resolution features with strong semantic information and high-resolution features with high texture information, the multi-level structure is used to continuously aggregate and abstract local features, so that the model can effectively focus on the features at the center of the featuremap.

[0019] Preferably, the NConv module is a common convolution with a convolution kernel of 3×3.

[0020] Preferably, the RPConv module includes:

[0021] Through 3X3 two-dimensional convolution and 1×1 two-dimensional convolution kernel, the scale is kept unchanged, and the network depth is increased from the perspective of the fully connected layer to achieve information interaction while introducing more nonlinearity and enhancing the model feature representation capability. Then, the feature map X is forward_split_cat operated to obtain X.

[0022] Preferably, the improved YOLOv7 detection model replaces the complete intersection-over-union loss function with a WIoU loss function.

[0023] Compared with the related art, the target area small target detection method based on drone images provided by the present invention has the following beneficial effects:

[0024] The present invention provides a method for detecting small targets in target areas based on drone images:

[0025] By using public data sets, the accuracy of the data required for the early algorithm is guaranteed, thereby ensuring the scientific nature of the algorithm. By utilizing the computing power of the improved YOLOv7, image features of varying degrees in the data set are collected, and finally different feature layers are fused as the basis for final target recognition. This further verifies the accuracy of the algorithm and the standardization of the data set, thereby improving recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flowchart of a method for detecting small targets in target areas based on drone images provided by the present invention;

[0027] Figure 2 The architecture of the analytical framework proposed for this invention;

[0028] Figure 3 To visualize the sparse attention mechanism of the network using GradCAM technology; Figure

[0029] Figure 4 This is the flow chart after the image is input into the MS-CET module;

[0030] Figure 5 is the structure diagram of BHC-FB, P3, P4, and P5 represent feature maps of different resolution levels, (·) represents the activation function, the upper branch module is represented by NConv, and the lower branch module is represented by RPConv;

[0031] Figure 6Feature-level visualization of the ATBHC-YOLO and YOLOv7 neck networks with the same input: (a) and (d) are RGB inputs; (b) and (c) are features of YOLOv7; (e) and (f) are features of ATBHC-YOLO; the features are upsampled to the same scale as the input image for comparison, (b) and (e) are low-level features, (c) and (f) are high-level features in deeper layers;

[0032] Figure 7 Schematic diagram of the loss function WIoU;

[0033] Figure 8 Comparison chart of small object detection for WIoU visualization;

[0034] Fig. 9 The following are the detection effect diagrams of four typical problem scenarios in the Vis-Drone2021 dataset;

[0035] Fig.10 This is a visualization test result chart of the ATBHC-YOLO algorithm based on the VEDAI dataset;

[0036] Fig.11 This is a visualization test result diagram of the ATBHC-YOLO algorithm on the DIOR dataset. DETAILED DESCRIPTION

[0037] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0038] like Figure 1 As shown in the figure, a method for detecting and determining a target based on a drone image is shown in an embodiment of the present application, and the method comprises the following steps:

[0039] S1, obtain the image taken by the drone;

[0040] S2, Improvement of Y0L0v7 model;

[0041] S3. Train the improved Y0L0v7 model.

[0042] Specifically, images are captured from video images captured by the drone and input into an improved YOLOv7 detection model for target detection to obtain the category of the target. The improved YOLOv7 detection model includes a backbone network, a neck network, and a prediction network.

[0043] The captured image is first input into the backbone network for feature extraction, then all the extracted features are fused in the neck network, and finally the fused features are input into the large, medium and small scale detection layers of the prediction network for detection to obtain the detection results;

[0044] The improved YOLOv7 detection model takes YOLOv7 as the basic model, adds MS-CET module and BHC-FB module to the neck network, the MS-CET module consists of an improved self-attention mechanism and a convolution module SPPCSP, the BHC-FB module consists of a bidirectional mixed convolution module NConv and RPConv in parallel, and finally fuses different feature layers, and then sequentially includes a first two-dimensional convolution layer, a batch normalization layer, a nonlinear function, a channel shuffle module and two two-dimensional convolution layers, and adopts a Sigmoid activation function.

[0045] In this paper, a small target detection model based on deep learning is called ATBHC-YOLO, which can detect difficult small targets in the multi-scale field of view of drones. It includes three main improvements: (1) MS-CET; (2) BHC-FB; (3) WIoU loss function optimization. The construction of the proposed model is as follows Figure 2 As shown in Figure 1, the input image is fed into the backbone network for feature learning. In the Neck part, features are extracted from different resolution levels of the backbone network, and more distribution information that is beneficial to small object detection is captured through our designed BHC-FB and MS-CET modules. Finally, the loss is learned through the WIoU loss function, and the entire system is trained in an end-to-end manner.

[0046] The baseline of the small target detection model based on deep learning consists of three parts: Backbone, Neck, and Head. In this model, first, we extract the features of the initial input image through the Backbone and output feature maps of different resolution sizes; secondly, at the top of the Neck, we designed the MS-CET module to aggregate more semantic features that are conducive to small target detection at multiple scales according to the different sizes of the input feature map feature map, and fully extract the features of different granularities and resolutions on the sparsity of the small target spatial distribution. In the middle of the Neck, we constructed the BHC-FB structure, and added a high-resolution feature map branch for small targets. Through multi-level cross-feature layer fusion, the bidirectional precision convolution module structure is introduced to focus on the features at the center of the feature map and reduce redundant target features. At the same time, we noticed that as the network training deepens, the texture information of small targets is lost, and the residual structure is used to use the shallow feature map to solve the feature loss problem. Finally, we input the enhanced extracted features into the Head part, and use the adaptive optimization loss function WIoU for the difficult detection samples of small targets in this dataset. Based on the principle of dynamic non-monotonic focusing mechanism border regression, we further improve the sensitivity of small target detection. Reasonable scale measurement can reduce the information error caused by the limited features of small targets and solve the problem of false detection and missed detection in dense scenes. In this way, our network can better extract the features and location information of small targets for efficient detection, and remove redundant features during training.

[0047] The following is an explanation of the improvements:

[0048] 1. MS-CET

[0049] We designed the MS-CET module before the convolutional module in the Neck part to better learn the global sparse features of small objects in spatial distribution. Figure 3 As shown in the heat map comparison, the effective receptive field range obtained by adding the MS-CET module is more comprehensive and can capture contextual semantic information. The self-attention mechanism will dynamically generate corresponding attention weights according to the different input images. However, DilatedTransformer does not have the ability to interact with multi-scale features, and has certain limitations in small target detection, and the effect is not satisfactory. Although ordinary CNNs have been widely used in multi-scale feature learning, CNNs rely on convolution, pooling, alignment and other mechanisms to learn local features, which is limited to the constraint of inductive bias. However, the areas that really need to be paid attention to in our tasks are often small and sparse, which leads to the model's insufficient ability to detect small targets.

[0050] Based on the above shortcomings, we combine the multi-head self-attention mechanism of DilatedTransformer with the multi-scale convolution module SPPCSP of YOLOv7. Figure 3 As shown in the heat map visualization, the former can model long-range context dependencies and effectively aggregate semantic information of different scales within the attention receptive field. The latter uses the translation invariance and local sensitivity of the convolution kernel to extract local features. Both adjust the resolution of the output feature map and then aggregate it to enhance the deep feature representation capability of small targets.

[0051] Design of MS-CET.SPPCSP module is located at the top of the Neck part of the YOLOv7 model, such as Figure 4 As shown, first, we input the characteristics of Backbone Figure X , after a CBS module (composed of Conv2d, BatchNorm2d and SiLU in series), and then using the C3_DilateBlock module we designed, the channels of the feature map are divided into three heads with different sparsity through the following equation 1, that is, we experimentally select the appropriate sparsity head n as 2, 3, and 4. Obtain the corresponding queries, keys and values ​​by linear projection.

[0052] X=LP(Concat[h1,...,hi]), 1≤i≤n, (1)

[0053] Among them, LP represents the input feature Figure X Linear projection to three different spaces gives the three matrices Q, K and V as shown in Equation 2. Each row of the three matrices represents a single query / key / value feature vector.

[0054] Q=[q1,...,qn], K=[k1,...,kn], V=[v1,...,vn], (2)

[0055] Then we define the query centered at the original feature map (m, n), use a sliding window of size w×w, and use the sparsity of different scales to perform self-attention mechanism operations on different heads with different dilation rates. Figure X The above process is described by the following equation 3.

[0056] h i =SD(Q i , K i , Vi , r i ), 1≤i≤n, (3)

[0057]

[0058] Here hi is the selected i-th head, Qi, Ki, Vi are the corresponding slices of the feature map sent to the i-th head for SD (sliding window) operation, and ri is the dilation rate of the corresponding head.

[0059] xn is the corresponding total amount of DilatedAttention output at different scales, Att is the DilatedAttention operation. H, W are the height and width of the feature map, Ki, Vi are the keys and values ​​selected from the feature map. qn represents all query vectors. KTQ represents the correlation of the input vector calculated using Q and K, T is the transpose operation, which facilitates the dot multiplication of the vector to obtain the weight of the relationship modeling based on the input data, which is calculated by similarity on the key set through the query.

[0060] Finally, all hi are concatenated together and input into the linear layer for feature aggregation, as shown in equation (4).

[0061]

[0062] Finally, the residual structure is used to achieve feature aggregation of ordinary convolution blocks and Dilateformer. Then we use four different scales of maximum pooling to achieve the fusion of information of different feature scales. The pooling kernel size is set to 9×9, 5×5, 3×3, and 1×1. In addition, we use the CBS module to adjust the resolution of the output feature map to match the feature map after the Dilateformer, so as to superimpose the residual structure for Concat operation and output. The entire MS-CET module aggregates the self-attention mechanism of the Dilateformer and the convolution module SPPCSP. Through multi-scale training, more global sparsity features of small targets are learned, and the weight bias learning of non-small target samples is reduced to utilize the sparsity at different scales for multi-head self-attention mechanism operations, so that feature extraction is more concentrated and in-depth to cope with the challenges of spatial distribution sparsity in small target detection.

[0063] 2. BHC-FB

[0064] The main idea of ​​this branch is to detect targets in higher resolution feature maps, and learn more information about small targets in combination with the multi-level cross-feature layer fusion branch we proposed. In this invention, we are inspired by the above to detect small targets in drone images with a large field of view. We found that the feature layer resolution sizes in the general branch BPF-N of this model for targets of ordinary sizes are: 80×80, 40×40, and 20×20. For ordinary targets, these three feature detection layers can enhance the extraction of appropriate feature information, but for small targets, the spatial distribution in drone images is localized, resulting in the low-resolution feature layer being unable to meet the requirements for detecting small targets.

[0065] Based on the above shortcomings, Figure 5 As shown in the figure, we use the feature layer with a high resolution of 160×160 output by the Backbone stage in YOLOv7, and introduce the bidirectional hybrid convolution modules NConv and RPConv on the basis of retaining more small target information at high resolution. By realizing the fusion and complementation of low-resolution features with strong semantic information and high-resolution features with high-texture information, the multi-level structure is used to continuously aggregate and abstract local features, so that the model can effectively focus on the features at the center of the featuremap. Without destroying the spatial structure and local features of the image, redundant features are discarded, thereby guiding the model to better train and enhancing the model's sensitivity to local features for small target detection.

[0066] Design of BHC-FB. As we analyzed above, high-resolution feature maps can extract more texture features of small objects, such as Figure 6 As shown in the heat map visualization, the BHC-FB module uses the above-mentioned cross-branch feature sharing structure to focus on the key areas in the image channel and refine the local features, which helps us locate and detect small targets in drone images more accurately. First, we use a multi-level approach to implement the bidirectional bridging structure of the convolutional hybrid module. The bidirectional hybrid structure further enhances the feature refinement during the model training process by fusing two convolutional modules with different advantages. Among them, the NConv module is a common convolution with a convolution kernel of 3×3. Combined with our small target detection task, NConv is used to focus on more local features of small targets. This is because NConv has a smaller convolution kernel, which can extract more fine-grained features, and the amount of computation it brings is relatively small.

[0067] Then, we designed a special RPConv module such as Figure 5As shown in the Partical_Conv structure diagram in the figure, we will introduce in detail the advantages of this module in BHC-FB module. First, we use the high-resolution feature map output by Backbone, and increase the network depth from the perspective of the fully connected layer while keeping the featuremap scale unchanged through Conv1×1, so as to achieve information interaction and introduce more nonlinearity to enhance the model feature representation ability. Then we perform orward_split_cat operation on featuremapX to obtain X1, as shown in equation (6),

[0068]

[0069] Among them, n_div represents the partial channels of the featuremap we select, and X1 is the partial channels corresponding to the convolution. The main purpose of this operation is to select only a part of the channels X1 in the feature map and use a Conv3×3 convolution operation. The input and output dimensions remain unchanged, and the feature map X1 after partial convolution is obtained. The reason for this is that there are many redundant and repeated featuremaps in all channels of the featuremap. If the convolution operation is performed on some featuremaps, the amount of calculation will be effectively reduced, as shown in equation (7):

[0070]

[0071] Here h and w are the height and width of the input feature map, k is the number of convolution kernels, and cx1 is the part of the channel to be convolved. Finally, we concatenate the feature map channels of these two branches to get the output result As shown in equation (8),

[0072]

[0073] At this point, we get featuremapU after Partial_Conv. We use two series-connected Conv1×1 to fully utilize the channel information, and use skip connections to reintroduce the unprocessed featuremapX. We extract more local non-redundant feature information through Partial_Conv for Upsample operation and perform feature fusion of the two branches. To make up for the feature information that may be lost by partial channel convolution, the BN layer and activation function SiLu are placed at the end to ensure feature diversity and reduce model complexity. In addition, we also introduce feature maps of different resolution sizes at each bidirectional branch, and adjust the resolution to match the high-resolution feature map in the bidirectional branch. Figure 1The advantage of this is that it supplements the insufficient semantic information in the high-resolution feature map, which is beneficial for our two-way branch to take into account both local feature extraction and feature refinement and enhancement.

[0074] In order to refine and enhance the local feature extraction of small objects, we designed the RPConv module, which pays attention to the local characteristics of small objects and only performs convolution on some featuremap channels from the perspective of reducing the amount of calculation. Finally, through the bidirectional bridging method of RPConv and NConv, BHC-FB combines multi-level cross-feature fusion operations of feature maps with different resolutions to further enhance the feature refinement extraction capability and reduce the model calculation amount without reducing the model feature extraction capability.

[0075] 3. WIoU loss function optimization

[0076] Since the dataset we selected contains some low-quality labeled data, which causes the model to intensify the penalty for low-quality samples, the loss function of our network is replaced by WIoU from CIoU of YOLOv7 for bounding box regression prediction. Conventional bounding box loss functions define calculation methods based on the geometric elements of the predicted box and the true box to continuously fit to obtain the optimal result, such as CIoU used by YOLOv7. However, in the dataset, we noticed that there are differences in the annotations of anchor boxes in the dataset, especially the insensitivity of the efficient fitting loss function to the data annotations of very small targets, which will greatly interfere with the convergence of the model and lead to poor feature learning effects. For this reason, we chose WIoU for bounding box regression based on a dynamic non-monotonic focusing mechanism such as Figure 7 As shown, the WIoU calculation formula is:

[0077]

[0078] WIoU designs a reasonable gradient gain allocation, that is, as the loss value increases, the gradient gain r shows a non-monotonic change. α and δ are hyperparameters. Specifically, the gradient gain is dynamically allocated according to the outlier factor of the anchor frame. The outlier calculation formula is:

[0079]

[0080] Among them, β is the outlier factor value, L*IoU is the gradient gain value, LIoU

[0081] is the momentum moving average.

[0082]

[0083] Then, R WIoU ∈[1, e) as a penalty term to strengthen the loss of common quality anchor boxes, L IoU∈[0, 1] is used as the IoU loss to reduce the penalty term of high-quality anchor boxes. When the anchor box and the prediction box are highly overlapped, the center distance can be emphasized. In addition, * indicates that it does not participate in back propagation, which effectively prevents the network model from generating gradients that are difficult to converge. The loss function parameters are visualized as follows: Figure X shown.

[0084] Nw and Nh are the width and height of the smallest rectangle formed by the real box and the predicted box, respectively; Wgt and Wgt are the width and height of the real box, respectively; w and h are the width and height of the predicted box, respectively.

[0085] The loss method calculation pays more attention to samples of normal quality, thereby improving the model's ability to locate objects. For target detection tasks in drone aerial photography scenes, a high proportion of small objects increases the difficulty of detection. WIoU can dynamically optimize the balance to weight small targets and solve the BBR balance problem between samples of good quality and poor quality, so as to improve the model's detection performance for samples of normal quality.

[0086] The performance of the method proposed in the present invention will be evaluated through experiments.

[0087] The dataset used in this experiment is Vis-Drone2021. The VisDrone dataset was collected by the AISKYEYE team of the Machine Learning and Data Mining Laboratory of Tianjin University, China, and is suitable for target detection tasks from the perspective of drones. The dataset consists of 10,209 images with a resolution of 2000 pixels × 1500 pixels, covering multiple scenes under different weather and lighting conditions. These images were captured from different places in China by different drone cameras, including rich urban and rural environments, and captured a large number of objects such as pedestrians, vehicles, tricycles, etc. The advantage of this dataset is that in the process of drone image collection, the influence of different scenes and weather is considered, which is very close to the real environment. Specifically, the dataset is divided into 6471 training sets, 548 validation sets, and 1610 test sets, with 10 categories: pedestrians, people, bicycles, cars, vans, trucks, tricycles, awning tricycles, buses, and motors. In this experiment, we analyzed that the dataset has an imbalanced target scale, that is, 12.05% of the targets are smaller than 16×16 pixels, 32.65% are larger than 16×16 and smaller than 32×32 pixels, and 55.3% are larger than 32×32 pixels. In addition, due to the characteristics of drone photography, there are difficulties such as sparse spatial distribution of small targets, local dense occlusion, and different lighting conditions. Implementation and hyperparameters: In the experiment, we use the PyTorch framework to implement our method, use CUDA11.5 for computational acceleration, the GPU is NVIDIA A6000 with 80GB of graphics memory, and YOLOv7 is used as the baseline. In order to fairly compare the performance of the algorithms, all methods use the same hyperparameter settings for training and verification, as shown in Table 1. The early stopping strategy is used during training to avoid model overfitting.

[0088] Table 1: Experimental detialoftheATBHC-YOLO algorithm

[0089]

[0090] In order to evaluate the performance of our ATBHC-YOLO model on drone aerial images, we used the evaluation indicators commonly used in target detection to evaluate model performance, namely Precision, Recall, GFLOPs, and mAP@.5. Precision represents the proportion of correctly identified parts among all identified parts, reflecting the model's ability to correctly classify; Recall is the ratio of the number of correctly identified parts to the number of all parts in the test set, reflecting the model's ability to comprehensively detect targets; GFLOPs is the number of floating-point operations of the network model; mAP is the average value of AP of each category in the data set. The larger the mAP, the better the model's detection accuracy. The calculation formula of mAP is as follows:

[0091]

[0092] mAP is divided into mAP0.5 and mAP0.5:0.95 according to different IoU thresholds. mAP0.5 is the mAP when IoU is 0.5. Under the condition of IoU of 0.5, the average precision (AP) of each category of mAP50 is calculated, and then the average is taken for all categories.

[0093] In order to verify the performance impact of the different modules we proposed on the small target detection task in the field of view of the drone under the same experimental conditions. We use the test set of VisDrone2021 to further verify the relationship between the design modules and the impact of a single module on the algorithm. In this section, we conduct ablation experiments. The evaluation results of each module of the drone small target detection network are shown in Table 2. Compared with the basic model, after adding MS-CET and BHC-FB, the detection accuracy is improved by 3.1 percentage points. The subsequent improvement of WIoU on the loss function also increases the detection accuracy. The modules can achieve a good balance in detection accuracy, showing a good detection effect. MS-CET captures the sparse characteristics of the spatial distribution of small targets in the field of view of the drone. In order to better learn the globality of small targets, it effectively models the long-range dependency of the small target context on the sparse distribution. It is dominated by DilatedTransformer and assisted by some ordinary CNN modules. The sparsity of small targets at different scales is used to perform self-attention mechanism operations, thereby generating more significant visibility for small targets. The detection accuracy of small targets is effectively improved by 1.7%.

[0094] Table 2: Forablationexperiments on theVisdrone dataset,↑indicates higherresults,↓indicates lowerresults,andthebestresults have been shown in bold.

[0095]

[0096] The BHC-FB module introduces high-resolution feature maps for small targets, and designs a bidirectional hybrid convolutional cascade structure from the perspective of locality to fuse features of small targets, reduce the bias of the model for large and medium-sized target detection, enhance the sensitivity to fine-grained feature information of small targets, and pay attention to the refinement of redundant small target features. The accuracy of small target detection is further improved to 3.1% and the amount of calculation is reduced. This method adds additional high-resolution branches for feature fusion to extract more detailed local feature information of small targets, such as texture and edge features of small targets. This method pays attention to the local information of small targets, fully balances the spatial and semantic information of small targets, and is suitable for small target detection tasks under the field of view of drones.

[0097] Finally, through experimental comparison, we found a WIoU loss function that is more suitable for small target detection. Based on the quality gap between samples, on the basis of improving the small target positioning ability in the UAV scene, we dynamically allocate gradient gains according to the outlier factor of the anchor frame, and dynamically optimize to solve the BBR balance problem between poor samples and good samples. Therefore, the anchor frame can locate the target more accurately and quickly, and further improve the small target detection accuracy to 3.5%, thereby improving the generalization ability and robustness of the model.

[0098] In general, the ablation experiment proves that the detection accuracy of our improved model is 3.5 percentage points higher than that of the basic model. Based on the sparse and local characteristics of small targets in the field of view of drones, this method can capture small target features with more spatial details and has a better small target detection effect.

[0099] In this section, we compare our proposed method with the state-of-the-art algorithms. We compare the detection performance of the ATBHC-YOLO detection model with the one-stage algorithms CDNet, TPH-YOLOv5, UFPMP-Det, DCRFF, Yolov5l, Yolov8l and the two-stage algorithms FasterRCNN and Cascade-RCNN on the dataset Visdrone-test-dev.

[0100] The compared experimental models are all relatively advanced small target detection models. Through experimental data comparison, the model of the present invention is significantly better than several other state-of-the-art models in terms of mAP50, an indicator of small target detection. The main reason is that in most small target detection categories, the detection accuracy of the ATBHC-YOLO proposed by us is better than other detection models with the same input resolution. The research results show that the MS-CET module designed by us performs multi-scale feature learning on high-resolution feature maps based on spatially distributed sparse features, extracts more sparse features through the self-attention mechanism, and enhances the feature bias learning of small targets. At the same time, BPF-RB also helps our model deal with more local feature loss problems caused by drastic changes in target scale. The improved network and algorithm have improved the ability to represent small target features, demonstrating the influence of local features on the accuracy of small target detection. In addition, the appropriate loss function can more accurately and quickly locate the target we need to detect. The experimental results are shown in Table 3. Table 3: Performance of ATBHC-YOLO and other algorithms on the dataset Visdrone-test-dev for different categories

[0101]

[0102] In each category detection of the data set, the method of the present invention has achieved good competitive performance, especially in van, tricycle, and awning-tricycle where the data categories are unbalanced and the samples are relatively scarce, reaching a high level of mAP50 index. In general, under the complex UAV field of view, ATBHC-YOLO significantly improves the small target detection performance, and captures small targets more clearly and quickly, and has certain competitive performance compared with other algorithms.

[0103] In order to further demonstrate the effectiveness and robustness of the ATBHC-YOLO algorithm of the present invention, the present invention specifically selects two mainstream datasets VEDAI[] and DIOR[] that are the same as the VisDrone class, and analyzes the advantages of the proposed algorithm based on a qualitative perspective, such as Fig.10 and Fig.11As shown in Figure 4. Among them, the VEDAI dataset is used for multi-class vehicle detection in aerial images. It contains vehicles that, in addition to their small size, also exhibit different changes such as multi-directionality, light / shadow changes, reflection or occlusion. DIOR is a large benchmark dataset for target detection in optical remote sensing images, containing 23,463 images and 192,472 instances, covering 20 object classes. By analyzing Tables 4 and 5, it is found that the method proposed by ATBHC-YOLO has certain generalization ability and robustness in small target recognition. In unknown datasets, its performance is better than other models, showing a clear accuracy advantage.

[0104] Table 4: Performance of ATBHC-YOLO and other algorithms on the datasetVEDAI-test for different categories

[0105]

[0106] Table 5: of ATBHC-YOLO and other algorithms on the dataset DIOR-test for different categories.

[0107]

[0108] Result analysis:

[0109] The experimental results fully verify the effectiveness of each module we proposed, as well as the comparability of the proposed overall method with other state-of-the-art algorithms.

[0110] Next, we will detect small targets in different drone vision scenarios and visualize key experimental results. Fig. 9The following are detection effect diagrams of four typical problem scenes in the Vis-Drone2021 dataset. In the scale-unbalanced scene, compared with (b), (a) can not only detect most small targets in the case of occlusion and drastic scale changes, but also capture targets with smaller occlusion scales. Especially in pedestrian gathering areas, occluded targets can still be accurately detected. In the sparse high-altitude distribution scene, since the drone's shooting field of view is at high altitude, (d), i.e., the baseline, cannot detect extremely small sparse targets appearing in the playground. The overall method we proposed can accurately detect (c) with good accuracy by improving the sparse distribution problem of small targets. In addition, the accuracy of other targets has also been improved. In dim and blurry scenes, the targets to be detected are limited by night light and buildings. (f) The baseline detection in the darker light parking area will cause missed detection and low accuracy. Our proposed method improves the poor light quality, false detection, and missed detection. In (e), two trucks that were not detected in the baseline detection were detected by our method, and the accuracy of surrounding vehicles was improved. In crowded scenes, the most prominent problems are boundary confusion and human body blur. Compared with the baseline, our proposed method can better handle the problems of incomplete target appearance, false detection, and low target confidence. In (h), it can be seen that most of the targets marked in the red circle are occluded and difficult to identify. After our improvement, such problems can be better solved in (g).

[0111] The model and algorithm we proposed can reduce the number of small targets that are not detected or predicted as the wrong category or with uncertainty, improve detection accuracy, and reduce the probability of poor detection in many different drone scenarios. We improve the detector's characteristics of locality and sparsity in spatial distribution of small targets, and select a suitable loss function to reduce the insensitivity of detecting small targets.

[0112] In summary, compared with the prior art, the method and module proposed in the present invention can effectively improve the accuracy of small target detection in the field of view of drones, and solve the problems of small target feature pixels being few, local blur, dense occlusion, and boundary confusion;

[0113] The present invention is highly competitive with the most advanced networks currently available, and effectively reduces the probability of problems such as missed detection, false detection, and low detection accuracy caused by the local and sparse characteristics of small targets in spatial distribution.

[0114] Based on the sparsity of the spatial distribution of small targets, the present invention attempts to add a multi-head self-attention mechanism, set the sparsity at different scales, perform contextual long-range dependency, and model more effective features; secondly, based on the locality of the spatial distribution of small targets, by adding higher-resolution cascade branches, more detailed features of small targets are extracted, and the representation ability of small targets is further enhanced; finally, in order to reduce the insensitivity of the model to small target detection, the present invention adds the loss function WIoU to optimize the loss of most redundant features, reduce the amount of calculation, and perform more bias learning on small targets.

[0115] Through the above work, experiments have proved that when the input image resolution is 640×640 pixels, the detection accuracy mAP50 and mAP of the ATBHC-YOLO algorithm are 3.5 percentage points and 3.2 percentage points higher than the benchmark algorithm YOLOv7, respectively. Although the detection speed has decreased and the computational complexity has increased, the detection accuracy has been greatly improved, and our model has a good performance.

[0116] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for detecting small targets in target areas based on drone images, characterized in that: The method comprises: S1, obtain the image taken by the drone; S2. Improvements to the YOLOv7 model: S3, training improved YOLOv7 model; Specifically, images are captured from the video footage captured by the drone and input into the improved YOLOv7 detection model for target detection to obtain the target category; The improved YOLOv7 detection model, that is, taking YOLOv7 as the basic model, adds an MS-CET module and a BHC-FB module to the neck network, wherein the MS-CET module is composed of an improved self-attention mechanism and a convolution module SPPCSP, and the BHC-FB module is composed of a bidirectional hybrid convolution module NConv and a RPConv in parallel, and finally the different feature layers are fused, and then the first two-dimensional convolution layer, a batch normalization layer, a nonlinear function, a channel shuffle module and two two-dimensional convolution layers are sequentially included, and a Sigmoid activation function is used; The MS-CET module combines the multi-head self-attention mechanism with the multi-scale convolution module SPPCSP of YOLOv7. After the image is input to the MS-CET module and passes through a CBS module, it is divided into two branches CBS and CBS+DilatedBlock for processing; The DilatedBlock module includes: linearly projecting the input feature map into three different spaces, using a sliding window of size w×w to perform a self-attention mechanism operation on different heads in different spaces using sparsity at different scales, and finally inputting the three spaces into a linear layer for feature aggregation; The BHC-FB module includes: using the Backbone stage in YOLOv7 to output a high-resolution feature layer of size 160×160, and introducing bidirectional hybrid convolution modules NConv and RPConv on the basis of retaining more small target information at high resolution; The NConv module is a common convolution with a convolution kernel of 3×3; The RPConv module includes: through 3×3 two-dimensional convolution, through 1×1 convolution kernel two-dimensional convolution to keep the scale unchanged, increase the network depth from the perspective of the fully connected layer, realize information interaction while introducing more nonlinearity, enhance the model feature representation ability, and then operate the feature map X to obtain X1, the formula is as follows: Among them, n_div represents the partial channels of the featuremap we select, X1 is the partial channels corresponding to the convolution, H and W are the height and width of the input featuremap respectively, and C is the number of channels; this operation is to select only a part of the channels X1 in the feature map and use a Conv3×3 for convolution operation. The input and output dimensions remain unchanged to obtain the feature map X1 after partial convolution; it is concatenated with the remaining channel feature map X2 to obtain the feature map U, and two series-connected Conv1×1 are used for it. The unprocessed feature map X is reintroduced using jump connections for feature fusion.

2. The method for detecting small targets in target areas based on drone images according to claim 1 is characterized in that: The improved YOLOv7 detection model replaces the complete intersection-over-union loss function with the WIoU loss function.

Images