Small target detection method based on multi-branch enhancement and multi-layer feature fusion

Abstract

This paper discloses a small target detection method based on multi-branch reinforcement and multi-layer feature fusion. The method includes obtaining an image dataset containing target images and preparing training samples. Using the YOLOv10-S network as a base network, a multi-framework (MFR-YOLO) network model for target recognition is constructed. The backbone of the MFR-YOLO network model includes a MECA module, and the core of the MFR-YOLO network model includes a multi-layer feature fusion module. The training samples are input into the MFR-YOLO network model for training. Finally, the image to be detected is input into the trained MFR-YOLO network model for target detection. The method significantly improves the stability of target detection, reduces the missed detection rate, and improves the accuracy of target detection.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a small target detection method based on multi-branch enhancement and multi-layer feature fusion. Background Art

[0002] The rapid development of unmanned aerial vehicle (UAV) technology has driven the application of UAV-based object detection in a variety of fields, including public safety monitoring, wildlife conservation, and military operations. However, compared to ground-based object detection tasks, aerial UAVs present more challenges, such as image quality degradation, uneven target distribution, small objects, and strong background noise. Furthermore, the limited computing power of UAV embedded devices necessitates the deployment of lightweight models, which limits object detection performance.

[0003] In the field of object detection, there are two main approaches: traditional algorithms and deep learning-based algorithms. Traditional algorithms combine manually designed target features with machine learning classifiers, such as Haar features. [1] and histogram of oriented gradients [2] . These methods often cannot meet actual needs when dealing with complex small target detection tasks. With the development of convolutional neural networks (CNN), target detection algorithms based on deep learning provide more effective solutions for practical applications. At present, target detection algorithms based on deep learning are mainly divided into two categories: one is a two-stage detection algorithm based on region proposal, such as R-CNN, FastR-CNN and FasterR-CNN; the other is a single-stage detection algorithm based on regression, such as SSD and YOLO series. Although the two-stage detection algorithm is accurate, it has a high computational cost, is sensitive to the design of anchor boxes, and is not suitable for real-time small target detection. In contrast, the single-stage detection algorithm is fast and easy to deploy, but there is still room for improvement in the detection accuracy of small targets in complex environments.

[0004] Currently, a real-time, densely packed small target detection algorithm for drones has been proposed based on an improved YOLOv5 (Feng Zhiqiang, Xie Zhijun, Bao Zhengwei, Chen Kewei. Acta Aeronautica Sinica, 2023, 44(7):327106-327106.). This algorithm is based on the YOLOv5 model and integrates the attention mechanism with the self-attention mechanism to improve the network architecture. This algorithm can deeply mine the hidden features of densely packed small target areas, effectively suppress the interference of complex background noise, and improve detection accuracy. Its core innovations include the optimization of the attention module, the improvement of the feature fusion method, and the upgrade of the backbone network performance, achieving effective real-time, densely packed small target detection for drones.

[0005] However, the above method has the following disadvantages:

[0006] 1. The backbone network has shortcomings and is easily interfered with by densely packed targets, reducing the stability of target detection. This technical solution uses YOLOv5 as the backbone network. YOLOv5 is designed to divide the entire image into different regions and independently predict the target in each region. In the case of densely packed targets, mutual interference between targets may lead to reduced detection performance. This results in high false detection and missed detection rates when using YOLOv5 for drone aerial photography target detection.

[0007] 2. The attention mechanism has deficiencies and easily overlooks small target features, leading to missed detection of small targets. This technical solution uses a spatial-channel attention module, but this attention mechanism only utilizes the horizontal and vertical information of the feature map during the attention weight generation process, ignoring the channel information. Furthermore, the horizontal and vertical information is average-pooled to generate the location of interest with a global receptive field. However, the low-level feature maps of drone aerial images contain a large amount of target detail information and background redundant information, which makes the spatial-channel attention module of this technical solution easily overlook small target features.

[0008] 3. The feature transfer mechanism is inadequate, resulting in insufficient utilization of effective information and impacting target detection accuracy. This technical solution uses a unidirectional feature transfer mechanism, which results in insufficient utilization of the rich information in feature maps of different scales. This makes it difficult to effectively fuse low-resolution features with strong semantic information and high-resolution features with strong detail information, affecting the accuracy of target detection in drone aerial imagery. Summary of the Invention

[0009] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a small target detection method based on multi-branch enhancement and multi-layer feature fusion. Based on the YOLOv10-S network, an MFR-YOLO network with multi-block coordinate feature enhancement and adaptive weight feature reconstruction for multi-scale target detection is proposed, which can effectively overcome the various defects of the existing technology.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0011] A small target detection method based on multi-branch enhancement and multi-layer feature fusion, the key of which is to include the following steps:

[0012] Step 1: Obtain an image dataset containing the target image and create training samples;

[0013] Step 2: Using the YOLOv10-S network as the base network, an MFR-YOLO network model for target recognition is constructed. The backbone of the MFR-YOLO network model includes a MECA module, and the neck of the MFR-YOLO network model includes a multi-layer feature fusion module.

[0014] The MECA module is used to divide the input feature map into blocks, generate an attention weight matrix through the attention mechanism according to the characteristics of the feature maps of different layers, and assign different weights to the information in the feature map;

[0015] The multi-layer feature fusion module is used to perform downsampling, channel information processing, and upsampling operations on the feature maps output by the backbone of the MFR-YOLO network, and then perform feature fusion, and recalibrate the weights of the fused features before outputting them;

[0016] Step 3: Input the training samples into the MFR-YOLO network model for training;

[0017] Step 4: Input the image to be detected into the trained MFR-YOLO network model for target detection.

[0018] Furthermore, the process of preparing the training samples is as follows:

[0019] Step 1.1: Select a data source and obtain a sample set from it.

[0020] Step 1.2: Label the images in the sample set;

[0021] Step 1.3: Cut the marked image to the required size;

[0022] Step 1.4: Convert the label format to the required format to obtain training samples.

[0023] Furthermore, the process of the MECA module assigning different weights to the information in the low-level feature map is as follows:

[0024] The input low-level feature map is cut into pieces to obtain multiple low-level feature sub-maps of the same size;

[0025] For each low-level feature sub-image, weight features are extracted using average pooling and maximum pooling operations along the horizontal X direction and vertical Y direction, respectively, to obtain weight features in the X direction and Y direction;

[0026] The extracted weight features in the X and Y directions are fused to obtain fused features;

[0027] The fused features are divided into fused features in the X direction and Y direction, and convolution and activation operations are performed respectively to obtain the attention weight matrices in the X direction and Y direction;

[0028] The obtained attention weight matrices in the X and Y directions are multiplied by the input features to complete the recalibration of the feature weights of the low-level feature sub-graphs, and all the low-level feature sub-graphs after feature recalibration are spliced ​​to obtain the low-level feature graph.

[0029] Furthermore, the calculation formulas for extracting weight features along the horizontal X direction and the vertical Y direction of the MECA module are as follows:

[0030]

[0031] Among them, X(f ij ) is the low-level feature subgraph f ij The weight feature of the X direction, Y(f ij ) is the low-level feature subgraph f ij The weight features in the Y direction, MaxPool is the maximum pooling operation, AvgPool is the average pooling operation, H and W are the height and width of the low-level feature map respectively, and n is the number of low-level feature sub-maps.

[0032] Furthermore, the specific process of the MECA module assigning different weights to the information in the high-level feature map is as follows:

[0033] The input high-level feature map is cut into blocks to obtain multiple high-level feature sub-maps of the same size;

[0034] Extract weight features along the horizontal X direction, vertical Y direction and channel Z direction for each high-level feature subgraph, and obtain weight features in the X direction, Y direction and Z direction respectively;

[0035] The weighted features in the Z direction are fused with the weighted features in the X direction and the Y direction respectively to obtain the fused features in the X direction and the Y direction, and then the fused features in the X direction and the Y direction are spliced ​​together to obtain the fused features;

[0036] The fused features are divided into fused features in the X direction and Y direction, and convolution and activation operations are performed respectively to obtain the attention weight matrices in the X direction and Y direction;

[0037] The obtained attention weight matrices in the X and Y directions are multiplied by the input features to complete the recalibration of the feature weights of the high-level feature subgraphs. All the high-level feature subgraphs after feature recalibration are spliced ​​to obtain the high-level feature graph.

[0038] Furthermore, the calculation formula for the weight features extracted by the MECA module in the X, Y and Z directions is:

[0039]

[0040] Among them, X(f ij ) is the high-level feature subgraph f ij The weight feature of the X direction, Y(f ij ) is the high-level feature subgraph f ij The weight feature in the Y direction, Z(fij ) is the high-level feature subgraph f ij The weight features in the Z direction, AvgPool is the average pooling, GAP is the global average pooling, H and W are the height and width of the high-level feature map, and n is the number of high-level feature sub-maps.

[0041] Furthermore, the neck part of the MFR-YOLO network model includes two multi-layer feature fusion modules, five C2f modules, one UPSample module, four Concat modules, two convolution modules, one SCDown module, and one C2fCIB module, where:

[0042] The three input ends of the first multi-layer feature fusion module respectively input the remaining feature maps output by the backbone part of the MFR-YOLO network model except the lowest layer feature map, and the features output by the first multi-layer feature fusion module are output to the first C2f module;

[0043] One output end of the first C2f module is connected to the input end of the second multi-layer feature fusion module, the other two input ends of the second multi-layer feature fusion module input the lowest two layers of feature maps output by the backbone of the MFR-YOLO network model, and the output end of the second multi-layer feature fusion module is connected to the input end of the second C2f module;

[0044] An output end of the second C2f module is connected to an input end of the first Concat module via the UPSample module, another input end of the first Concat module inputs the lowest layer feature map output by the backbone part of the MFR-YOLO network model, and the output end of the first Concat module is connected to the input end of the third C2f module;

[0045] An output end of the third C2f module is connected to an input end of the second Concat module via the first convolution module, another input end of the second Concat module is connected to another output end of the second C2f module, and the output end of the second Concat module is connected to an input end of the third Concat module after passing through the fourth C2f module and the second convolution module; another input end of the third Concat module is connected to another output end of the first C2f module, the output end of the third Concat module is connected to an input end of the fourth Concat module after passing through the fifth C2f module and the SCDown module, and another input end of the fourth Concat module is connected to the highest-level feature map output by the backbone of the MFR-YOLO network model, and the output end of the fourth Concat module is connected to the input end of the C2fCIB module;

[0046] Another output end of the third C2f module, an output end of the fourth C2f module, an output end of the fifth C2f module, and an output end of the C2fCIB module are all connected to the head part of the MFR-YOLO network model.

[0047] Furthermore, the multi-layer feature fusion module processes the three scale feature maps input as follows:

[0048] Upsample the small-scale feature map to obtain the intermediate feature map of the small-scale feature;

[0049] The large-scale feature map is compressed by 3×3 convolution operation, normalized by BN operation, and linearly activated by ReLU operation to obtain the intermediate feature map of the large-scale feature;

[0050] The intermediate feature map of small-scale features, the intermediate feature map of large-scale features and the feature map of intermediate scale are spliced ​​in the channel dimension to obtain the fused feature map;

[0051] The fused feature map is compressed using a 1×1 convolution operation, normalized using a BN operation, and linearly activated using a ReLU operation. It is then processed by MECA to obtain a feature map with recalibrated feature weights.

[0052] Furthermore, when the multi-layer feature fusion module performs MECA processing, its processing process is consistent with the processing process of MECA assigning different weights to information in the high-level feature map.

[0053] Furthermore, the loss function of the MFR-YOLO network model is:

[0054]

[0055] Among them, L inner-MPDIoU is the loss function, IoU inner is the inner intersection and union ratio, d1 is the distance between the upper left corner coordinates of the true bounding box and the predicted bounding box, d2 is the distance between the lower right corner coordinates of the true bounding box and the predicted bounding box, and w and h are the width and height of the input feature map respectively.

[0056] This paper proposes an MFR-YOLO network model for multi-scale target detection based on the YOLOv10-S network, which features multi-block coordinate feature enhancement and adaptive weight feature reconstruction. In the backbone of the network model, a MECA module is constructed to recalibrate the weights to enhance useful information in the feature map and suppress useless information, enabling the network to better capture the important features of the target and provide richer feature information for the neck and head. In the neck of the network model, an MFR module is constructed. By fusing multi-level feature maps, the module fully integrates low-resolution features containing strong semantic information with high-resolution features containing detailed information. The MECA module adaptively adjusts the information importance weights to highlight important target information, thereby enhancing the algorithm's target detection capabilities at different scales. On this basis, the MFR-Neck is further designed to fuse multi-scale features and enhance the network's perception of targets of different scales. In the head of the network model, the detection head design of YOLOv10 is adopted.

[0057] The remarkable effects of the present invention are:

[0058] 1. Solved the problem of weak stability in dense target detection and improved the stability of target detection. In order to solve the problem of low dense target detection capability in the technical solutions of the existing technology, the backbone network was improved, and YOLOv5 was replaced with YOLOv10-S, which was used as the backbone network of the target detection model. YOLOv10 is a new generation of real-time end-to-end target detection model proposed in May 2024, which takes into account both accuracy and efficiency. It is designed based on the advantages of the previous version of the YOLO algorithm, introduces an overall efficiency-accuracy driven model design strategy, includes a compact inverted block (CIB) structure, and designs a rank-guided adaptive block allocation strategy. It uses large kernel depth convolution to increase the receptive field of the deep stage, avoiding the overhead of the shallow stage, enabling the model to better capture the global information and contextual information in the image, enhancing the perception of the target, and improving the model's detection stability and accuracy in complex scenarios.

[0059] 2. Solved the problem of missed detection of small targets and reduced the missed detection rate of target detection. To address the problem of missed detection of small targets in the existing technical solutions, a MECA module was constructed. By dividing the feature map into blocks and designing two attention weight generation methods for feature maps of different layers, the average pooling operation is used to aggregate the information of the width and height directions of the feature map, and the channel direction information mining is added. At the same time, by integrating multiple directional information, it can not only capture direction and position information, but also capture cross-channel information, so that the model can more accurately locate and identify targets of interest, improve the model's feature extraction ability for small targets, and thus enhance the detection ability of small targets.

[0060] 3. Solved the problem of low detection accuracy for small targets and improved target detection accuracy. In order to solve the problem of missed detection of small targets in the technical solutions of the existing technology, an adaptive weight fusion mechanism based on the MECA module is proposed. In the feature extraction part, the feature enhancement mechanism is used to enhance the network's ability to extract important features. In the feature fusion part, three feature maps of different scales are selected in sequence, and the downsampling operation, channel information processing and upsampling operation are performed respectively before fusion. The fused feature weights are recalibrated by the MECA module, and the integration of large-scale features is used to facilitate the detection of small targets. The detailed features contained in the large-scale feature map and the semantic features contained in the small-scale feature map are fully integrated to generate high-quality features after fusion of multiple scales, which is conducive to the detection of occluded targets and effectively improves the detection accuracy of dense UAV aerial photography targets under complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flow chart of the method of the present invention;

[0062] Figure 2 This is the overall structure diagram of the MFR-YOLO network model;

[0063] Figure 3 This is the structural diagram of the MECA module;

[0064] Figure 4 This is the structural diagram of the MFR module. DETAILED DESCRIPTION

[0065] The specific implementation manner and working principle of the present invention will be further described in detail below with reference to the accompanying drawings.

[0066] like Figure 1 As shown, this embodiment addresses various defects in the prior art and provides a small target detection method based on multi-branch enhancement and multi-layer feature fusion. The specific steps are as follows:

[0067] Step 1: Obtain an image dataset containing the target image and create training samples;

[0068] In this example, the process of preparing the training samples is as follows:

[0069] Step 1.1: Select a data source and obtain a sample set from it.

[0070] Step 1.2: Label the images in the sample set;

[0071] Step 1.3: Cut the marked image to the required size;

[0072] Step 1.4: Convert the label format to the required format, obtain training samples, and prepare the training samples into a training set and a test set in an 8:2 ratio.

[0073] Step 2: Based on the YOLOv10-S network, we build the MFR-YOLO network model for target recognition. The overall network structure is as follows: Figure 2 As shown:

[0074] The backbone of the MFR-YOLO network model includes sequentially connected ConvModule modules ( Figure 2 Module 0 in), ConvModule module ( Figure 2 Module 1), MECA module ( Figure 2 Module 2 in), C2f module ( Figure 2 Module 3 in), ConvModule module ( Figure 2 Module 4 in), C2f module ( Figure 2 Module 5 in), SCDown module ( Figure 2 Module 6 in), C2f module ( Figure 2 Module 7 in), SCDown module ( Figure 2 Module 8 in), MECA module ( Figure 2 Module 9 in), C2f module ( Figure 2 Module 10 in), SPPF module ( Figure 2 Module 11 in), PSA module ( Figure 2 Module 12 in the embodiment, wherein the first MECA module, the second C2f module, the third C2f module, and the PSA module respectively output feature maps of four different scales from low to high to the neck part; the MECA module is used to block the input feature map, generate an attention weight matrix through the attention mechanism according to the characteristics of the feature maps of different layers, and assign different weights to the information in the feature map;

[0075] Because targets in drone aerial photography vary widely in scale, are densely distributed, and feature complex backgrounds, the target feature extraction process is easily affected by the complex background, resulting in reduced detection accuracy. In the drone aerial object detection task, introducing an attention mechanism during the target feature extraction stage allows the model to focus on key areas and reduce background interference. The CA attention mechanism, proposed by Hou et al., is a highly efficient attention mechanism that combines spatial attention and channel attention to capture both horizontal and vertical positional information in feature maps. This positional information is used to generate attention weights, enhancing target features and helping the network more accurately locate targets of interest. However, the CA attention mechanism only utilizes the horizontal and vertical information of the feature map during the attention weight generation process, ignoring channel information. Furthermore, the horizontal and vertical information is average-pooled to generate the location of interest with a global receptive field. Low-level feature maps in drone aerial images contain a wealth of target details and redundant background information. Directly applying the CA attention mechanism can easily overlook small target features. Furthermore, high-level feature maps have low resolution but many channels. Directly applying the CA attention mechanism can lead to insufficient utilization of channel information during weight generation.

[0076] Therefore, this embodiment aims at the problem of feature extraction in the task of detecting aerial targets in UAVs, and constructs a MECA module. By dividing the feature map into blocks and designing two attention weight generation methods for feature maps of different layers, one is Low layer Coordinate Attention (LLCA), which is used for the low-level features of the feature extraction network, and its function is to highlight the target feature information, reduce background interference information, and reduce information redundancy; the other is High layer Coordinate Attention (HLCA), which is used for the high-level features of the feature extraction network. Compared with LLCA, it adds channel attention information, makes full use of the multi-channel characteristics of high-level feature maps, and focuses on the characteristics of important channels to improve the model's feature extraction ability for the target. The structure of the MECA module is as follows: Figure 3 shown.

[0077] The key idea of ​​the MECA module is to divide the feature map into blocks, generate an attention weight matrix through the attention mechanism according to the characteristics of the feature maps of different layers, and assign different weights to the information in the feature map to enhance the target information for better feature extraction.

[0078] For low-level feature maps, the image resolution is high, the number of channels is small, and the information of small targets and other target details is complete. The LLCA part splits the feature map output by the second convolution module along the height and width directions into multiple sub-maps. Each sub-map uses average pooling and maximum pooling to aggregate information along the horizontal X and vertical Y directions respectively. The maximum pooling operation collects texture, contour and other detailed information in the low-level feature map. The average pooling operation reduces information redundancy and suppresses noise. The two pooling operations are used to process the sub-maps at the same time. The underlying detailed information can be highlighted while suppressing background noise. The information processed in the horizontal X and vertical Y directions is further integrated. The information weight of the feature map is generated through channel transformation and activation function, so that the network can better capture the detailed features of small targets and other targets in complex backgrounds.

[0079] For high-level feature maps, the image resolution is low and the number of channels is large. The feature maps lack detailed information but contain rich semantic information. In particular, as the number of channels increases, the channel dimension contains rich channel information. To fully utilize the channel information, the HLCA part will aggregate the horizontal X, vertical Y, and channel Z direction information. Based on the low-level feature map operation, it adds channel direction weight calculation, assigns weights to different channels, and guides the model to focus more on learning important channel features, thereby better extracting features under limited computing resources. The main processing process of the MECA module is as follows:

[0080] like Figure 3 As shown in FIG, the LLCA part in the MECA module assigns different weights to the information in the low-level feature map output by the second convolutional module of the backbone as follows:

[0081] A1. Input feature map Perform block processing to obtain n 2 A low-level feature subgraph f ij , i=0,...,n-1;j=0,...,n-1, the size of each low-level feature subgraph is

[0082] In A2 and LLCA, weight features are extracted using average pooling and maximum pooling along the horizontal X direction and vertical Y direction for each low-level feature sub-image, respectively. The weight features in the X direction and Y direction are obtained respectively. The calculation formula is as follows:

[0083]

[0084] Among them, X(f ij ) is the output of the attention module in the X direction, which is also the low-level feature subgraph f ij The weight feature of the X direction, Y(f ij ) is the output of the attention module in the Y direction, which is also the low-level feature subgraph f ijThe weight features in the Y direction, MaxPool is the maximum pooling operation, AvgPool is the average pooling operation, H and W are the height and width of the low-level feature map respectively, and n is the number of low-level feature sub-maps.

[0085] A3, LLCA part fuses the extracted X-direction and Y-direction weight features to obtain the fused feature M ij , the calculation formula is as follows:

[0086]

[0087] Among them, M ij is the fused feature, Represents the concatenation operation, and ConvModule represents a module containing a convolutional layer, a BN layer, and a SiLu activation function.

[0088] A4. The fused feature M ij Segmented into fusion features in the X direction and the fusion features in the Y direction And perform convolution and activation operations respectively to obtain the attention weight matrix in the X direction and Y direction;

[0089]

[0090] in, is the attention weight in the X direction, is the attention weight in the Y direction, conv is a 1×1 convolution that does not change the image size but only changes the number of channels, and σ(·) is the activation function.

[0091] A5. Obtain the attention weight matrix in the X direction Attention weight matrix in the Y direction Perform dot multiplication with the input features to complete the low-level feature subgraph f ij The feature weights are recalibrated, and the low-level feature subgraph f after all feature recalibration is ij The output of MECA is obtained by splicing, that is, the low-level feature map F′. The output low-level feature map F′ is consistent with the input low-level feature map F in size and number of channels. The output formula is as follows:

[0092]

[0093] Among them, f′ ij f ij The output after feature weight recalibration, is the output of the MECA structure.

[0094] from Figure 3It can also be seen that the HLCA part aggregates the horizontal X, vertical Y, and channel Z direction information. On the basis of aggregating the horizontal X and vertical Y information, it adds the weight calculation of the channel direction (the difference between the two modes is that the LLCA part only aggregates the horizontal X and vertical Y direction information, while the HLCA part not only aggregates the horizontal X and vertical Y but also the Z direction). This assigns weights to different channels to guide the model to focus more on learning important channel features. Therefore, the specific process of the HLCA part in the MECA module described in this example assigning different weights to the information in the high-level feature map output by the backbone SCDown module is as follows:

[0095] B1. HLCA part of the input high-level feature map Perform block processing to obtain n 2 A high-level feature subgraph f ij , i=0,…,n-1;j=0,…,n-1, the size of each high-level feature subgraph is

[0096] B2. For each high-level feature subgraph f ij The weight features are extracted along the horizontal X direction, the vertical Y direction and the channel Z direction respectively, and the weight features X(f ij )、Y(f ij )、Z(f ij ), the calculation formula is:

[0097]

[0098] Among them, X(f ij ) is the output of the attention module in the X direction, which is also the high-level feature subgraph f ij The weight feature of the X direction, Y(f ij ) is the output of the attention module in the Y direction, which is also the high-level feature subgraph f ij The weight feature in the Y direction, Z(f ij ) is the output of the attention module in the Z direction, which is also the high-level feature subgraph f ij The weight features in the Z direction, AvgPool is the average pooling, GAP is the global average pooling, H and W are the height and width of the high-level feature map, and n is the number of high-level feature sub-maps.

[0099] B3, the weight feature Z(f ij ) and the weight features X(f ij )、Y(f ij ) to obtain the fused X-direction and Y-direction features, and then splice the fused X-direction and Y-direction features together to obtain the fused feature M ij, the calculation process is as follows:

[0100]

[0101] Among them, M ij is the fused feature, Z′ ij is the channel weight in the Z direction, MLP represents the multi-layer perceptron, σ represents the sigmoid activation function, To fuse the X-direction features of the Z-direction channel weights, To fuse the Y-direction features of the Z-direction channel weights, represents element-wise multiplication, Represents the concatenation operation, and ConvModule represents a module containing a convolutional layer, a BN layer, and a SiLu activation function.

[0102] B4, the fused feature M ij Segmented into fusion features in the X direction and the fusion features in the Y direction And perform convolution and activation operations respectively to obtain the attention weight matrix in the X direction and Y direction;

[0103]

[0104] in, is the attention weight in the X direction, is the attention weight in the Y direction, conv is a 1×1 convolution that does not change the image size but only changes the number of channels, and σ(·) is the activation function.

[0105] B5. The obtained attention weight matrix in the X direction Attention weight matrix in the Y direction Perform dot multiplication with the input features to complete the high-level feature subgraph f ij The feature weights are recalibrated, and the high-level feature subgraph f after all the features are recalibrated ij The output of MECA is obtained by splicing, which is the high-level feature map F′. The output high-level feature map F′ is consistent with the input high-level feature map F in size and number of channels. The output formula is as follows:

[0106]

[0107] Among them, f′ ij f ij The output after feature weight recalibration, is the output of the MECA structure.

[0108] In this embodiment, the MECA module designs an effective way to capture direction perception information and position perception information based on the characteristics of feature maps at different layers, so that the model can more accurately locate and identify targets of interest, extract image features more specifically, and improve image recognition effects.

[0109] During the specific implementation process, the feature maps at different positions of the YOLOv10s network backbone contain different types of rich information. However, since it only adopts a unidirectional feature transfer mechanism, it does not fully utilize the rich information of feature maps of different scales, making it difficult to effectively fuse low-resolution features with strong semantic information and high-resolution features with strong detail information, affecting the accuracy of target detection in drone aerial images.

[0110] In response to the above problems, this embodiment proposes a neck part of an adaptive weight fusion mechanism based on the MECA module to improve the fusion efficiency of features of different scales. The neck part includes a multi-layer feature fusion module; wherein the multi-layer feature fusion module is used to perform downsampling operations, channel information processing and upsampling operations on the feature maps output by the backbone of the MFR-YOLO network, and then perform feature fusion, and recalibrate the weights of the fused features before output. Its specific structure is as follows Figure 2 As shown in the figure, it includes two multilayer feature fusion (MFR) modules, five C2f modules, one UPSample module, four Concat modules, two convolution modules, one SCDown module, and one C2fCIB module. Specifically:

[0111] The first multi-layer feature fusion module ( Figure 2 The three input ends of the module 13 in the MFR-YOLO network model are respectively input with the remaining feature maps except the lowest layer feature map output by the backbone part of the MFR-YOLO network model, and the feature output of the first multi-layer feature fusion module is output to the first C2f module ( Figure 2 Module 14 in

[0112] The first C2f module has an output terminal and the second multi-layer feature fusion module ( Figure 2 The other two input ends of the second multi-layer feature fusion module input the lowest two layers of feature maps output by the backbone of the MFR-YOLO network model, and the output end of the second multi-layer feature fusion module is connected to the second C2f module ( Figure 2 The input terminal of the module 16) is connected;

[0113] One output of the second C2f module is connected to the UPSample module ( Figure 2 Module 17) and the first Concat module ( Figure 2 The other input end of the first Concat module inputs the lowest layer feature map output by the backbone of the MFR-YOLO network model, and the output end of the first Concat module is connected to the third C2f module ( Figure 2 The input terminal of the module 19) is connected;

[0114] One output of the third C2f module is convolutionally connected to the first ConvModule ( Figure 2 Module 20 in ) and the second Concat module ( Figure 2 One input end of the module 21 in the second Concat module is connected to the other input end of the second C2f module ( Figure 2 The output of the second Concat module is connected to the other output of the module 16 in the fourth C2f module ( Figure 2 Module 22 in ) and the second convolutional module ( Figure 2 Module 23) and the third Concat module ( Figure 2 The other input end of the third Concat module is connected to the first C2f module ( Figure 2 The output of the third Concat module is connected to the other output of the module 14 in the figure, and the output of the third Concat module is connected to the output of the fifth C2f module ( Figure 2 Module 25) and SCDown module ( Figure 2 Module 26) and the fourth Concat module ( Figure 2 The output of the fourth Concat module is connected to the C2fCIB module ( Figure 2 The input terminal of the module 28) is connected;

[0115] Another output end of the third C2f module, an output end of the fourth C2f module, an output end of the fifth C2f module, and an output end of the C2fCIB module are all connected to the head part of the MFR-YOLO network model.

[0116] From the above structure diagram, we can see that the MFR module takes three scale feature maps as input, selects three different scale feature maps in turn, performs downsampling operation, channel information processing and upsampling operation and then fuses them. The MECA module recalibrates the weight of the fused features to generate high-quality features after fusion of multiple scales, so as to improve the detection accuracy of dense UAV aerial photography targets under complex background. The structure of the MFR module is as follows: Figure 4 shown.

[0117] The MFR module takes three scale feature maps as input. For large-scale feature maps, Depthwise Separable Conv (DSConv) is used, which includes Depthwise Convolution, Pointwise Convolution, BN layer and SiLu activation function to halve the height and width of the feature map and double the number of channels. For small-scale feature maps, Nearest Neighbor Interpolation is used for upsampling to maintain the rich local features of the low-resolution image. The processed feature maps of three different scales are spliced ​​in the spatial dimension to fully fuse the low-resolution features with strong semantic information with the high-resolution features with weak semantic information. Then, the MECA module generates adaptive feature weights, and the generated adaptive feature weights are used to perform attention weighting on the fused features. By capturing direction, position and channel information, the fused target features are strengthened to improve the model's ability to detect targets. The main processing steps of the MFR module are as follows:

[0118] C1. Small-scale feature map F s Perform upsampling to obtain the small-scale feature intermediate feature map F s ';

[0119] C2, for large-scale feature map F l After 3×3 convolution operation to compress height and width, BN operation to normalize, and ReLU operation to linearly activate, the large-scale feature intermediate feature map F is obtained. l ';

[0120] C3, small-scale feature intermediate feature mapping F s ′, large-scale feature intermediate feature map F l ′ and the intermediate scale feature map F m Splicing in the channel dimension to obtain the fused feature map F f , the calculation formula is:

[0121]

[0122] C4, the fused feature map F f After using 1×1 convolution operation to compress the channel, BN operation to normalize, and ReLU operation to perform linear activation processing, and then MECA processing, the feature map F after feature weight recalibration is obtained. f ′, the calculation formula is:

[0123] F f ′=MECA(ConvModule(F f )) (14)

[0124] Since the feature fusion stage aggregates feature maps of three scales, the number of channels is large and the channel dimension also contains a lot of information, the MECA processing here adopts the HLCA mode, which can fully exploit the channel information on the basis of making full use of the position information and focus on the channels containing more target information, which is conducive to improving the feature expression ability of the network.

[0125] Step 3: Input the training samples into the MFR-YOLO network model for training;

[0126] During training, traditional YOLOv10s uses CIoU Loss to calculate bounding box regression. CIoU Loss considers the overlap area, center point distance, and aspect ratio of the predicted and ground-truth boxes, resulting in faster convergence and an effective description of the bounding box regression state. In the bounding box regression process for drone aerial object detection, the same image contains numerous objects, including large, medium, and small objects, which may overlap. When the predicted and ground-truth boxes overlap, the CIoU loss function optimizes the shape, position, and size deviations between the predicted and ground-truth boxes through a penalty term. However, when the predicted and ground-truth boxes have the same aspect ratio but different height and width values, the CIoU loss function loses its effectiveness in calculating bounding box regression, limiting the model's convergence speed and accuracy. MPDIoU Loss, on the other hand, effectively distinguishes between predicted and ground-truth boxes with the same aspect ratio but different height and width values. It directly calculates the distance between the upper left and lower right corners of the predicted box and the true box, which simplifies the calculation process and can more accurately reflect the difference between the predicted box and the true box. It is suitable for the bounding box regression loss calculation of targets of various sizes in UAV aerial photography target detection tasks.

[0127] Therefore, this embodiment adopts MPDIoU and combines it with Inner-IoU based on auxiliary bounding boxes to calculate the regression loss of the bounding box, targeting the high density and mutual overlap of drone aerial photography targets. Compared with MPDIoU, this method uses auxiliary bounding boxes of different scales to calculate the loss, focusing more on the core part of the bounding box, making it possible to make more accurate judgments on overlapping areas and effectively accelerating the bounding box regression process. The calculation formula of the Inner-MPDIoU loss function is as follows:

[0128]

[0129] Among them, L inner-MPDIoU is the loss function, IoU inner is the inner intersection over union (Inner-IoU), d1 is the true bounding box Β gt and the predicted bounding box Βprd The distance between the upper left corner coordinates, d2 is the real bounding box Β gt and the predicted bounding box Β prd The distance between the lower right corner coordinates of the predicted bounding box Β prd The width and height are w prd 、h prd , the coordinates of its upper left and lower right corners are The coordinates of its center point are The ground-truth bounding box Β gt The width and height are w gt 、h gt , the coordinates of its upper left and lower right corners are The coordinates of its center point are w and h are the width and height of the input feature map respectively.

[0130] In this embodiment, the calculation formula of the Inner-IoU is as follows:

[0131]

[0132] uinon=(w gt *h gt )*ratio 2 +(w prd *h prd )*ratio 2 -inter (18)

[0133]

[0134] Among them, ratio is the scale factor, and its value range is [0.5, 1.5].

[0135] This example uses a Windows 10 system with 16GB of memory, an Intel(R) Core(TM) i7-13700KF CPU, and the experimental environment is Python 3.11.9, PyTorch 2.0.1, and CUDA 11.8. All models are trained and tested on an NVIDIA RTX 4080 GPU.

[0136] Training and evaluation process of the MFR-YOLO network model:

[0137] ①Use the conda create-n yolov10 python=3.11.9 command to create a virtual environment named yolov10 and activate the environment;

[0138] ②Then install PyTorch and related dependent libraries;

[0139] ③ According to the specific situation of the dataset, modify the corresponding yolov10.yaml configuration file to specify the number of categories, the paths of the training set and the test set, and other information;

[0140] ④ Use the training command to start training. After training is completed, the model is evaluated on the test set and the evaluation indicators such as accuracy, recall rate, mAP and other test results are output.

[0141] Step 4: Input the image to be detected into the trained MFR-YOLO network model for target detection.

[0142] To sum up, in order to solve the defects of low target detection stability, missed detection of small targets and low target detection accuracy in the background technology, the present invention improves the backbone network and adopts YOLOv10 as the backbone network of the detection model to better capture the global information and context information in the image, enhances the perception ability of the target, and improves the model's detection stability and accuracy of the target in complex scenes; and constructs a MECA module in the backbone part, by dividing the feature map into blocks and designing two attention weight generation methods for feature maps of different layers to improve the model's feature extraction ability for small targets, thereby enhancing the detection ability of small targets; at the same time, an adaptive weight fusion mechanism based on the MECA module is proposed in the neck part, by sequentially selecting feature maps of three different scales, performing downsampling operations, channel information processing and upsampling operations and then fusing them, and recalibrating the fused feature weights through the MECA module to generate high-quality features after fusion of multiple scales, thereby effectively improving the detection accuracy of dense UAV aerial photography targets under complex backgrounds.

[0143] The technical solution provided by the present invention is introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A small target detection method based on multi-branch enhancement and multi-layer feature fusion, characterized in that: The steps include: Step 1: Obtain an image dataset containing the target image and create training samples; Step 2: Using the YOLOv10-S network as the base network, an MFR-YOLO network model for target recognition is constructed. The backbone of the MFR-YOLO network model includes a MECA module, and the neck of the MFR-YOLO network model includes a multi-layer feature fusion module. The MECA module is used to divide the input feature map into blocks, generate an attention weight matrix through the attention mechanism according to the characteristics of the feature maps of different layers, and assign different weights to the information in the feature map; The MECA module designs two attention weight generation methods for feature maps at different layers. One is LLCA, which is used to extract low-level features of the feature extraction network; The other is HLCA, which is used for high-level features of feature extraction network; LLCA splits the input low-level feature map into multiple sub-maps along both the height and width directions. Each sub-map aggregates information using average pooling and maximum pooling along the horizontal X and vertical Y directions, respectively. The maximum pooling operation collects detailed information from the low-level feature map, while the average pooling operation reduces information redundancy and suppresses noise. Simultaneously processing the sub-maps using the two pooling operations can highlight the underlying detailed information while suppressing background noise. Further fusing the processed information in the horizontal X and vertical Y directions, and generating the information weight of the feature map through channel transformation and activation function, the network model can better capture the detailed features of small targets and other targets in complex backgrounds. The HLCA part slices the input high-level feature map and aggregates the horizontal X, vertical Y, and channel Z information of each sub-map. Based on the low-level feature map operation, it adds channel-wise weight calculation, assigns weights to different channels, and guides the network model to focus more on learning important channel features. The multi-layer feature fusion module is used to perform downsampling, channel information processing, and upsampling operations on the feature maps output by the backbone of the MFR-YOLO network, and then perform feature fusion, and recalibrate the weights of the fused features before outputting them; Step 3: Input the training samples into the MFR-YOLO network model for training; Step 4: Input the image to be detected into the trained MFR-YOLO network model for target detection.

2. The small target detection method based on multi-branch enhancement and multi-layer feature fusion according to claim 1, characterized in that: The process of making the training samples is as follows: Step 1.1: Select a data source and obtain a sample set from it. Step 1.2: Label the images in the sample set; Step 1.3: Cut the marked image to the required size; Step 1.4: Convert the label format to the required format to obtain training samples.

3. The small target detection method based on multi-branch enhancement and multi-layer feature fusion according to claim 1, characterized in that: The process of the MECA module assigning different weights to the information in the low-level feature map is as follows: The input low-level feature map is cut into pieces to obtain multiple low-level feature sub-maps of the same size; For each low-level feature sub-graph, weight features are extracted using average pooling and maximum pooling operations along the horizontal X direction and vertical Y direction, respectively, to obtain weight features in the X direction and Y direction; The extracted weight features in the X and Y directions are fused to obtain fused features; The fused features are divided into fused features in the X direction and the Y direction, and convolution and activation operations are performed respectively to obtain the attention weight matrices in the X direction and the Y direction; The obtained attention weight matrices in the X and Y directions are multiplied by the input features to complete the recalibration of the feature weights of the low-level feature sub-graphs, and all the low-level feature sub-graphs after feature recalibration are spliced ​​to obtain the low-level feature graph.

4. The small target detection method based on multi-branch enhancement and multi-layer feature fusion according to claim 3, characterized in that: The calculation formula for extracting weight features along the horizontal X direction and the vertical Y direction of the MECA module is: Among them, X(f ij ) is the low-level feature subgraph f ij The weight feature in the X direction, Y(f ij ) is the low-level feature subgraph f ij The weight features in the Y direction, MaxPool is the maximum pooling operation, AvgPool is the average pooling operation, H and W are the height and width of the low-level feature map respectively, and n is the number of low-level feature sub-maps.

5. The small target detection method based on multi-branch enhancement and multi-layer feature fusion according to claim 1, characterized in that: The specific process of the MECA module assigning different weights to the information in the high-level feature map is as follows: The input high-level feature map is cut into blocks to obtain multiple high-level feature sub-maps of the same size; Extract weight features along the horizontal X direction, vertical Y direction and channel Z direction for each high-level feature subgraph, and obtain weight features in the X direction, Y direction and Z direction respectively; The weighted features in the Z direction are fused with the weighted features in the X direction and the Y direction respectively to obtain the fused features in the X direction and the Y direction, and then the fused features in the X direction and the Y direction are spliced ​​together to obtain the fused features; The fused features are divided into fused features in the X direction and the Y direction, and convolution and activation operations are performed respectively to obtain the attention weight matrices in the X direction and the Y direction; The obtained attention weight matrices in the X and Y directions are multiplied by the input features to complete the recalibration of the feature weights of the high-level feature subgraphs. All the high-level feature subgraphs after feature recalibration are spliced ​​to obtain the high-level feature graph.

6. The small target detection method based on multi-branch enhancement and multi-layer feature fusion according to claim 5, characterized in that: The calculation formula for the weight features extracted by the MECA module in the X, Y and Z directions is: Among them, X(f ij ) is the high-level feature subgraph f ij The weight feature in the X direction, Y(f ij ) is the high-level feature subgraph f ij The weight feature in the Y direction, Z(f ij ) is the high-level feature subgraph f ij The weight features in the Z direction, AvgPool is the average pooling, GAP is the global average pooling, H and W are the height and width of the high-level feature map, and n is the number of high-level feature sub-maps.

7. The small target detection method based on multi-branch enhancement and multi-layer feature fusion according to claim 1, characterized in that: The neck part of the MFR-YOLO network model includes two multi-layer feature fusion modules, five C2f modules, one UPSample module, four Concat modules, two convolution modules, one SCDown module, and one C2fCIB module, among which: The three input ends of the first multi-layer feature fusion module respectively input the remaining feature maps output by the backbone part of the MFR-YOLO network model except the lowest layer feature map, and the features output by the first multi-layer feature fusion module are output to the first C2f module; One output end of the first C2f module is connected to the input end of the second multi-layer feature fusion module, the other two input ends of the second multi-layer feature fusion module input the lowest two layers of feature maps output by the backbone of the MFR-YOLO network model, and the output end of the second multi-layer feature fusion module is connected to the input end of the second C2f module; An output end of the second C2f module is connected to an input end of the first Concat module via the UPSample module, another input end of the first Concat module inputs the lowest layer feature map output by the backbone part of the MFR-YOLO network model, and the output end of the first Concat module is connected to the input end of the third C2f module; An output end of the third C2f module is connected to an input end of the second Concat module via the first convolution module, another input end of the second Concat module is connected to another output end of the second C2f module, and the output end of the second Concat module is connected to an input end of the third Concat module after passing through the fourth C2f module and the second convolution module; another input end of the third Concat module is connected to another output end of the first C2f module, the output end of the third Concat module is connected to an input end of the fourth Concat module after passing through the fifth C2f module and the SCDown module, and another input end of the fourth Concat module is connected to the highest-level feature map output by the backbone of the MFR-YOLO network model, and the output end of the fourth Concat module is connected to the input end of the C2fCIB module; Another output end of the third C2f module, an output end of the fourth C2f module, an output end of the fifth C2f module, and an output end of the C2fCIB module are all connected to the head part of the MFR-YOLO network model.

8. The small target detection method based on multi-branch enhancement and multi-layer feature fusion according to claim 7, characterized in that: The multi-layer feature fusion module processes the three scale feature maps input as follows: Upsample the small-scale feature map to obtain the intermediate feature map of the small-scale feature; The large-scale feature map is compressed by 3×3 convolution operation, normalized by BN operation, and linearly activated by ReLU operation to obtain the intermediate feature map of the large-scale feature; The intermediate feature map of small-scale features, the intermediate feature map of large-scale features and the feature map of intermediate scale are spliced ​​in the channel dimension to obtain the fused feature map; The fused feature map is compressed using a 1×1 convolution operation, normalized using a BN operation, and linearly activated using a ReLU operation. It is then processed by MECA to obtain a feature map with recalibrated feature weights.

9. The small target detection method based on multi-branch enhancement and multi-layer feature fusion according to claim 8, characterized in that: When the multi-layer feature fusion module performs MECA processing, its processing process is consistent with the processing process of MECA assigning different weights to information in the high-level feature map.

10. The small target detection method based on multi-branch enhancement and multi-layer feature fusion according to any one of claims 1 to 9, characterized in that: The loss function of the MFR-YOLO network model is: Among them, L inner-MPDIoU is the loss function, IoU inner is the inner intersection and union ratio, d1 is the distance between the upper left corner coordinates of the true bounding box and the predicted bounding box, d2 is the distance between the lower right corner coordinates of the true bounding box and the predicted bounding box, and w and h are the width and height of the input feature map respectively.

Images