Aerial photography dense small target detection method based on spatial feature alignment and multi-scale feature aggregation

Abstract

The invention provides an aerial photography dense small target detection method based on spatial feature alignment and multi-scale feature aggregation, and the method comprises the steps: collecting an image containing a target, inputting the image into a DSOD-SM network model, and outputting a target detection result through the DSOD-SM network model; the DSOD-SM network model comprises a backbone network, a feature fusion module and a detection head; the data output end of the backbone network is connected with the data input end of the feature fusion module, and the data output end of the feature fusion module is connected with the data input end of the detection head; the method is effective for dense small target detection, and can effectively improve the reconnaissance capability, enhance the strike precision and assist battlefield situation awareness.

Description

Technical Field

[0001] The present invention relates to the field of small target detection, and in particular to a method for detecting dense small targets in aerial photography based on spatial feature alignment and multi-scale feature aggregation. Background Art

[0002] In future intelligent wars, as the hardware performance of drones improves, drones will play an important role in battlefield reconnaissance, small target detection and tracking, military-assisted precision strikes, etc., and become one of the important means of perceiving battlefield situations. At the same time, as deep learning technology has achieved great success in the field of computer vision, drone-based visual target detection has a pivotal position in the military field. In recent years, convolution-based target detection networks have gradually matured. Compared with target detection algorithms such as Faster RCNN, SSD (Single Shot Multi-Box Detector) and RetinaNet, the YOLO series has become one of the most popular single-stage target detection networks due to its simple structure, especially its ability to balance accuracy and detection speed.

[0003] Although the above algorithm can obtain satisfactory detection results in common application scenarios, it still has shortcomings when dealing with detection tasks in dense small target scenes. Compared with conventional target detection problems, small targets in aerial photography have smaller pixels and dense distribution. Their image features are easily misaligned during the transmission process (such as scale reduction, splicing, etc.), which will have an adverse effect on subsequent detection tasks. In addition, the multi-scale feature fusion mechanism of the existing model is single, and it is not possible to perform adaptive feature weight allocation according to different scenes where the features are located. Summary of the invention

[0004] The present invention aims to at least solve the problem that, under drone aerial photography, dense small targets in the existing technology are prone to feature offset and have weak multi-scale feature fusion capabilities. In particular, an innovative method for detecting dense small targets in aerial photography based on spatial feature alignment and multi-scale feature aggregation is proposed.

[0005] In order to achieve the above-mentioned object of the present invention, the present invention provides an aerial photography dense small target detection method based on spatial feature alignment and multi-scale feature aggregation, comprising: collecting an image containing a target, then inputting the image into a DSOD-SM network model, and the DSOD-SM network model outputs a target detection result;

[0006] The DSOD-SM network model includes a backbone network, a feature fusion module and a detection head; the data output end of the backbone network is connected to the data input end of the feature fusion module, and the data output end of the feature fusion module is connected to the data input end of the detection head;

[0007] The backbone network adopts the CSPDarknet network as a feature extractor, including a first Conv module, a second Conv module, a first C2f module, a third Conv module, a second C2f module, a fourth Conv module, a third C2f module, a fifth Conv module, a fourth C2f module and an SPPF module connected in sequence; wherein the second C2f module, the third C2f module and the fourth C2f module output the first primary feature, the second primary feature and the third primary feature respectively;

[0008] The feature fusion module includes three feature fusion branches, namely, the first feature fusion branch, the second feature fusion branch, and the third feature fusion branch, and its inputs are respectively the first primary feature, the second primary feature, and the third primary feature;

[0009] The first feature fusion branch includes a first MFC module, a fifth C2f module, and a first MFA module connected in sequence; the second feature fusion branch includes a second MFC module, a sixth C2f module, a third MFC module, a seventh C2f module, and a second MFA module connected in sequence, and the data output end of the sixth C2f module is connected to the data input end of the second MFA module; the third feature fusion branch includes a fourth MFC module, an eighth C2f module, and a third MFA module connected in sequence;

[0010] The connection relationship between the first feature fusion branch and the second feature fusion branch is: the data input end of the first MFC module is connected to the data output end of the first upsampling module, and the data input end of the first upsampling module is connected to the data output end of the second MFC module; the data output end of the fifth C2f module is connected to the data input end of the second MFA module;

[0011] The connection relationship between the second feature fusion branch and the third feature fusion branch is: the data input end of the second MFC module is connected to the data output end of the second up-sampling module, and the input of the data input end of the second up-sampling module is the third primary feature; the data input end of the fourth MFC module is connected to the data output end of the second down-sampling module, and the data input end of the second down-sampling module is connected to the data output end of the third MFC module;

[0012] The MFC module includes:

[0013]

[0014] F mfc It is the output feature that is fused in a cascade manner;

[0015] Conv() is a standard convolution.

[0016] Indicates pixel-by-pixel addition;

[0017] L1, L2, and L3 represent the first resolution feature, the second resolution feature, and the third resolution feature, respectively;

[0018] ω1, ω2, ω3 represent the weight parameter of the first branch, the weight parameter of the second branch, and the weight parameter of the third branch respectively;

[0019] The MFA module includes:

[0020] F mfc =Conv(Concat[ω1·L1,ω2·L2,ω3·L3])

[0021] Among them, F mfa It is the output feature fused by matrix and method;

[0022] Concat represents a concatenation operation;

[0023] The detection head includes three sub-detection heads, which are respectively used for feature detection of three feature fusion branches.

[0024] Preferably, the DSOD-SM network model further includes a feature alignment module (SFA module). At this time, the data output end of the backbone network is connected to the data input end of the feature alignment module, and the data output end of the feature alignment module is connected to the data input end of the feature fusion module:

[0025] The data output ends of the second C2f module, the third C2f module, and the fourth C2f module are respectively connected to the data input ends of the first feature alignment module, the second feature alignment module, and the third feature alignment module, and the data output ends of the first feature alignment module, the second feature alignment module, and the third feature alignment module are respectively connected to the data input ends of the first MFC module, the second MFC module, and the fourth MFC module; in addition, the data output end of the third feature alignment module is also connected to the data input end of the second up-sampling module;

[0026] The feature alignment module (SFA) comprises:

[0027] First, perform global average pooling on the input feature X, and obtain the feature vector F GAP , the global average pooling is expressed as:

[0028]

[0029] Among them, G d Represents the output value of the dth (1≤d≤c) channel feature map of the input feature F after global average pooling;

[0030] h and w represent the size of the feature map respectively;

[0031] F d (i,j) represents the element at (i,j) in the d-th channel feature map region.

[0032] Then F GAP As input, use the fully connected layer f c / 32 and f 32 / c Learning feature alignment coefficients As shown in the following formula:

[0033]

[0034] Among them, f c / 32 Represents the first fully connected layer (input dimension is c, output dimension is 32);

[0035] f 32 / c Represents the second fully connected layer (input dimension is 32, output dimension is c);

[0036] σ represents the sigmoid activation function, and its formula is:

[0037]

[0038] Among them, exp() is the natural exponential function;

[0039] x represents the output vector of the fully connected layer.

[0040] Then the spatial alignment correction coefficient is applied to the transposed feature matrix F T , and get the aligned spatial attention map M:

[0041]

[0042] in, represents the feature alignment coefficient;

[0043] · T represents transpose;

[0044] F represents the input feature;

[0045] Finally, the obtained spatial attention map is fused to the input feature map to obtain the final output features:

[0046] F"=reshape(F T ·M),

[0047] Among them, reshape( ) indicates matrix dimension reconstruction.

[0048] Preferably, it also includes: the data output end of the first feature alignment module is connected to the data input end of the first MFA module, the data output end of the second feature alignment module is connected to the data input end of the second MFA module, and the data output end of the third feature alignment module is connected to the data input end of the third MFA module.

[0049] Preferably, the backbone network adopts the CSPDarknet network as a feature extractor, and replaces the third and subsequent Conv modules with a scale scaling module (ST module) for preliminary feature extraction of the collected images; by introducing the scale scaling module, the performance of the model in the feature extraction stage is enhanced.

[0050] The scale scaling module includes:

[0051] First, the input feature X is spatially downsampled using stride convolution with a convolution kernel of 2 and maximum pooling to obtain two feature maps F of the same size. c and F m :

[0052] F c =Conv 2×2 (X),

[0053] F m =MaxPool(X),

[0054] Then the two feature maps are added pixel by pixel, and the final sampling information can be expressed as

[0055] F'=F c +F m .

[0056] The above processing reduces the computational cost and the number of parameters; and retains the effective feature information in the downsampling process to the greatest extent, which helps to capture important features in the image, especially significant features such as edges and textures.

[0057] Preferably, ω1, ω2, ω3 are obtained through a weight learner:

[0058] First, the weight learner cascades the different resolution features {L1, L2, L3} at the same level to obtain the feature The implementation is as follows:

[0059] F cat =Concat(L1,L2,L3),

[0060] Next, the feature F cat Perform a global average pooling operation to obtain F g, and then use the multi-layer perceptron MLP to learn the feature branch weight parameter W, W = {W1, W2, W3}. The learning process is:

[0061] W=MLP(F g ),

[0062] in, MLP is a multi-layer perceptron. Considering the stability of the weight learner, it is also necessary to normalize the output of the multi-layer perceptron to obtain the normalized branch weight parameter ω={ω1,ω2,ω3}. The normalization operation of the i-th weight is:

[0063]

[0064] where ω i represents the normalized weight of the i-th branch, ω i ={ω1,ω2,ω3};

[0065] W i is the weight information of the i-th branch learned by the multilayer perceptron;

[0066] ε represents the smoothness term.

[0067] Preferably, the F obtained for the MFC module mfc , F obtained by MFA module mfa Use convolution for feature fusion, and add batch normalization and sigmoid activation function after convolution.

[0068] Preferably, the method further includes: using the NAM channel attention mechanism to perform in-depth analysis on the adjusted features.

[0069] In summary, due to the adoption of the above technical solution, the present invention not only solves the shortcomings of traditional networks in feature fusion, but also significantly improves the detection performance of dense small targets. The target detection model designed by the present invention integrates a scale scaling module (ST). By optimizing the downsampling strategy, it not only improves the representation ability of the model, but also reduces unnecessary parameters in the network. At the same time, it ensures that when the resolution of the feature map is reduced, the important feature information of small targets in the detection task can be retained to the maximum extent, providing a richer and more effective feature representation for subsequent detection tasks.

[0070] In order to alleviate the problem of feature offset that is prone to occur in feature transfer of dense small objects, a novel spatial feature alignment attention mechanism (Spatial Feature Alignment Attention, SFA) is proposed. This mechanism uses channel information and spatial information to correct features, allowing the model to focus more keenly on subtle spatial feature details, effectively reducing false detection and missed detection of dense small-sized objects.

[0071] In addition, in order to more effectively fuse the aligned multi-scale features, a multi-scale feature aggregation module based on multi-branch dynamic weights is designed. This module can significantly enhance the model's ability to fuse features of different scales, thereby improving detection accuracy in complex scenarios.

[0072] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0074] Figure 1 It is a structural schematic diagram of the DSOD-SM framework of the present invention.

[0075] Figure 2 It is a schematic diagram of the ST structure of the present invention.

[0076] Figure 3 It is a schematic diagram of the SFA structure of the present invention.

[0077] Figure 4 It is a schematic diagram of the structure of MFC and MFA of the present invention.

[0078] Figure 5 Schematic diagram of the NAM attention mechanism of the present invention.

[0079] Figure 6 It is a schematic diagram comparing the detection results of DSOD-SM of the present invention and the baseline model.

[0080] Figure 7 It is a training comparison diagram of the DSOD-SM of the present invention and the baseline model.

[0081] Figure 8 It is a vehicle detection effect diagram specifically implemented in the present invention.

[0082] Fig. 9 It is a characteristic heat map of each attention mechanism implemented in the present invention. DETAILED DESCRIPTION

[0083] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0084] In order to improve the detection performance of UAVs for dense small targets, this paper uses YOLOv8 as the baseline model and proposes a detection algorithm for dense small targets in aerial photography. The overall framework of DSOD-SM is as follows: Figure 1 As shown in the figure, it consists of four core modules: backbone network, feature alignment module, feature fusion module and detection head. Specifically, the backbone network uses the CSPDarknet network as the feature extractor, retains the SPPF module, and introduces the scale scaling module (ScaleTransformation, ST) to enhance the performance of the model in the feature extraction stage. After the backbone network completes feature extraction, in order to alleviate the feature offset phenomenon during the feature transmission process, the spatial alignment attention mechanism layer is integrated in the specific layer (the feature sizes of each layer are 80×80, 40×40, and 20×20 respectively). Subsequently, the aligned features are sent to the Neck part for deep fusion. In the Neck part, the multi-scale feature connection aggregation module (Multi-scale Feature Connection Aggregation, MFC) based on multi-branch weights and the multi-scale feature addition aggregation module (Multi-scale Feature Addition Aggregation, MFA) based on multi-branch weights are introduced. These two modules work together to effectively fuse the feature information of different scales in the same level, greatly enhancing the network's perception of the scale changes of dense small targets. Finally, these fused features are passed to the detection head as input for the classification and regression detection box position tasks. Through this series of optimization measures, DSOD-SM not only solves the shortcomings of traditional networks in feature fusion, but also significantly improves the detection performance of dense small objects.

[0085] 1.1 Lightweight Scaling Module

[0086] Traditional target detection models generally use pooling operations and 3×3 stride convolution as a means of scale scaling, but this approach often leads to the loss of feature map information, especially when dealing with dense small targets, the feature information loss is serious, and the number of convolution parameters will double with the increase of the convolution kernel.

[0087] The baseline model uses a 3×3 standard convolution with a stride of 2, which simultaneously implements spatial downsampling (from h×w to h / 2×w / 2) and channel conversion (from c to 2c). This introduces a non-negligible computational cost O(9 / 2hwc 2 ) and the number of parameters O(18c 2 ). The combination of convolution with a convolution kernel of 2 and maximum pooling for scale scaling and channel addition operations can not only improve the representation ability of the model, but also greatly reduce the amount of calculation, thereby achieving more efficient sampling. The specific operations are as follows Figure 2As shown in Figure 2, we first use stride convolution with a convolution kernel of 2 and maximum pooling to perform spatial downsampling, and obtain two feature maps F of the same size. c and F m , the implementation methods are

[0088] F c =Conv 2×2 (X),

[0089] F m =MaxPool(X),

[0090] Where X represents the input feature;

[0091] Then the two feature maps are added pixel by pixel to obtain the final sampling information, which can be expressed as F':

[0092] F'=F c +F m ,

[0093] The above process reduces the computational cost to O(7 / 2hwc 2 ), the number of parameters is reduced to O(16c 2 ), while retaining the effective feature information in the downsampling process to the greatest extent, it helps to capture important features in the image, especially significant features such as edges and textures. Among them, O( ) is the time complexity function of the computational amount (cost) and the space complexity function of the parameter amount.

[0094] 1.2 Spatial Feature Alignment Attention Mechanism Module

[0095] This paper designs a spatial feature alignment (SFA) attention module. The basic idea is to first extract and learn the alignment correction coefficients through feature information, and then integrate these coefficients into the original features to correct the features, thereby guiding the model to focus more on the target feature area. Figure 3 As shown, the feature alignment correction coefficient is learned through full connection without introducing learning parameters. It is mainly obtained through the following two steps of learning: the first step is to perform global average pooling (GAP) on the input feature X, and the obtained feature vector The implementation of GAP is:

[0096]

[0097] Among them, G d Represents the output value of the dth (1≤d≤c) channel feature map of the input feature F after global average pooling;

[0098] h and w represent the size of the feature map respectively;

[0099] F d (i,j) represents the element at (i,j) in the d-th channel feature map region.

[0100] The second step is to GAP As input, use the fully connected layer f c / 32 and f 32 / c Learning feature alignment coefficients As shown in the following formula:

[0101]

[0102] Among them, f c / 32 Represents the first fully connected layer (input dimension is c, output dimension is 32);

[0103] f 32 / c Represents the second fully connected layer (input dimension is 32, output dimension is c);

[0104] σ represents the sigmoid activation function, and its formula is:

[0105]

[0106] Among them, exp() is the natural exponential function;

[0107] x represents the output vector of the fully connected layer.

[0108] Then the spatial alignment correction coefficient is applied to the transposed feature matrix F T , and get the aligned spatial attention map:

[0109]

[0110] here,

[0111] represents the feature alignment coefficient;

[0112] · T represents transpose;

[0113] F represents the input feature;

[0114] Finally, the obtained spatial attention map is fused to the input feature map to obtain the final output feature:

[0115] F"=reshape(F T ·M),

[0116] reshape() indicates matrix dimension reshaping.

[0117] 1.3 Multi-scale Weighted Feature Aggregation Module

[0118] Traditional multi-scale feature fusion methods are often limited by unidirectional information flow, which is particularly evident when fusing features of different resolutions. The traditional approach is usually to adjust these features to a uniform resolution and then fuse them directly by channel connection or feature information addition. However, since the input branch features come from different resolutions, their contributions to the final output features are often not equal. In order to overcome the above limitations, this paper proposes a new scheme. Its basic idea is as follows: Figure 4 As shown, by introducing Figure 1 The curved path from left to right in the figure enhances the network aggregation ability and dynamically assigns weights to each branch feature, so that the network can autonomously learn the importance of each input branch feature.

[0119] Considering that only adjusting branch weights can give features global importance, it may ignore the specific contributions of certain key channels within the branch. In order to focus more finely on these important channels, a channel attention mechanism, NAM, is integrated at the end of the module. This mechanism can further refine the weight distribution of features and ensure that the network can capture both global and local key information. The specific implementation process is as follows:

[0120] First, through the multi-branch aggregation network, the weights of each branch feature are calculated to achieve preliminary importance adjustment, and the input features of different scales are subjected to multi-branch weighted feature fusion. The purpose of multi-scale feature fusion is to aggregate features of different scales. Formally, given a multi-scale feature list P of the same size in =(L1,L2,...,L n ), where L i represents the i-th branch feature of the same layer. The weight parameters W of different branches are learned by the weight learner, and then the importance of each branch is calculated respectively. Assume that the number of branches is n = 3. First, the weight learner cascades the different resolution features {L1, L2, L3} of the same layer to obtain the feature The implementation is as follows:

[0121] F cat =Concat(L1,L2,L3),

[0122] Next, the feature F cat Perform a global average pooling operation (GAP) to obtain Then, the feature branch weight parameter W = {W1, W2, W3} is learned through the multi-layer perceptron MLP. The learning process is:

[0123] W=MLP(F g ),

[0124] in, MLP is a multi-layer perceptron. Considering the stability of the weight learner, it is also necessary to normalize the output of the multi-layer perceptron to obtain the normalized branch weight ω = {ω1, ω2, ω3}. The normalization operation of the i-th weight is:

[0125]

[0126] where ω i represents the normalized weight of the i-th branch;

[0127] W i is the weight information of the i-th branch learned by the multilayer perceptron, W i ∈W, feature branch weight parameter W = {W1, W2, W3};

[0128] ε represents a smoothing term (constant). To avoid numerical instability, ε=0.0001.

[0129] Then, the weighted features of different branches learned are fused. To this end, two methods are used for the final fusion, one is feature addition and the other is feature channel connection. Their fusion operations are:

[0130]

[0131] F mfc =Conv(Concat[ω1·L1,ω2·L2,ω3·L3])

[0132] Where L1, L2, and L3 represent the first resolution feature, the second resolution feature, and the third resolution feature, respectively;

[0133] ω1, ω2, ω3 represent the weight parameter of the first branch, the weight parameter of the second branch, and the weight parameter of the third branch respectively;

[0134] Indicates pixel-by-pixel addition;

[0135] Concat represents concatenation operation;

[0136] F mfc and F mfa They are the output features of the two feature fusion methods. It is integrated through cascading; It is integrated through matrix and method.

[0137] Subsequently, the NAM channel attention mechanism is used to perform in-depth analysis on the adjusted features to identify and enhance channel information that is critical to the task. NAM is an efficient and lightweight attention mechanism, such as Figure 5As shown. Using the scaling factor of batch normalization, the importance of weights is represented by standard deviation, as shown in the formula.

[0138]

[0139] Among them, B out is the output of the batch normalization operation;

[0140] BN() represents batch normalization operation;

[0141] F in It is the output feature after multi-scale feature fusion;

[0142] μ B and σ B are the mean and standard deviation of the training batches respectively;

[0143] γ and β are trainable affine transformation parameters,

[0144] Calculate the channel attention weight W through γ γ , and its calculation formula is:

[0145]

[0146] γ k ,γ j represents the channel scale factor, W γ represents the attention weight;

[0147] Finally, the weighted feature fusion mechanism and the NAM attention mechanism are combined to improve the model's global and local perception capabilities. The final output is as follows:

[0148] Z=Sigmoid(W γ ·B out ),

[0149] Among them, Sigmoid() is the activation function;

[0150] This process not only circumvents the one-way information flow limitation of traditional multi-scale feature fusion, but also significantly improves the effect of feature fusion through the dual effects of dynamic weight allocation and channel attention mechanism.

[0151] Experiment and result analysis

[0152] 2.1 Experimental Environment

[0153] All experiments are performed within a unified hardware and software environment framework. Specifically, the experimental environment is equipped with a high-performance NVIDIA GeForce RTX 3060Ti graphics processing unit (GPU) and an Ubuntu 22.04 operating system with 16GB of memory. The network architecture is built based on the PyTorch deep learning framework, and CUDA11.6 is used to achieve GPU accelerated computing.

[0154] To ensure the comparability and consistency of the experiment, the configuration standards of the baseline model were strictly followed. The training uniformly used the stochastic gradient descent optimizer SGD (Stochastic Gradient Descent), with a momentum of 0.937 and a weight decay of 0.0005. In the initial three model warm-ups, the learning rate was linearly increased from 0.0034 to 0.01, and then gradually decreased to 0.001 using the cosine annealing decay strategy. Table 1 lists some of the key hyperparameter configurations of the model in detail to provide a reference for the reproducibility of the experiment. All training was uniformly performed for 300 rounds.

[0155] Table 1. Some hyperparameter settings of the model

[0156] Hyperparameters value Hyperparameters value lr0 0.01 hsv_h 0.015 lrf 0.01 hsv_s 0.7 momentum 0.937 hsv_v 0.4 weight_decay 0.0005 degrees 0.0 warmup_epochs 3.0 translate 0.1 warmup_momentum 0.8 scale 0.5 warmup_bais_lr 0.1 shear 0.0 box 7.5 perspective 0.0 cls 0.5 flipud 0.0 dfl 1.5 fliplr 0.5 Mixup 0.0 mosaic 1.0

[0157] 2.2 Experimental data

[0158] The experiments used three aerial target detection datasets, VisDrone2019, CARPK, and PUCPR+, to verify the performance of the proposed algorithm. The VisDrone2019 dataset is a dataset acquired by Tianjin University using drone cameras in different scenes, weather, and lighting conditions. It covers a wide range and has rich detection targets. As shown in Table 2, there are not only many small targets in the dataset, but also densely distributed targets. The CARPK dataset contains images of nearly 90,000 cars from 4 different parking lots, which were collected from drone views at a height of about 40m. The image set is annotated by the bounding box of each car, and all marked bounding boxes record the upper left corner and lower right corner. In addition, the PUCPR+ dataset contains information on 16,456 cars in the same parking lot.

[0159] The algorithm evaluation indicators selected are precision, recall, F1-Score and mean average precision (mAP). The formulas of the above indicators are expressed as follows:

[0160]

[0161] Among them, TP is the number of correctly predicted positive samples, FP is the number of incorrectly predicted positive samples, FN is the number of incorrectly predicted negative samples, N is the number of detection categories in the training set, and when N = 1, mAP = AP. At the same time, the number of model parameters is selected to evaluate the complexity of the model.

[0162] 2.3 Ablation Experiment

[0163] 2.3.1 Overall ablation experiment

[0164] In order to fully verify the effectiveness of the proposed module, a series of ablation experiments were designed on the VisDrone-2019 dataset. The experimental design is divided into two stages: First, the newly proposed modules ST, MFC, MFA and SFA in DSOD-SM are introduced into the baseline method one by one to independently evaluate the improvement effect of each module on the performance of dense small object detection. The experimental results are shown in Table 3, which clearly shows the performance changes after the introduction of each module. Specifically, the baseline method has a mAP@0.5 of 40.5% on the VisDrone-2019 dataset. After the ST, MFC, MFA and SFA modules were introduced separately, they all had a positive impact on the detection ability of dense small objects, and the mAP@0.5 indicators were increased to 40.8%, 41.1%, 40.7% and 41.0% respectively.

[0165] Subsequently, combined experiments between modules were conducted to fully verify

[0166] The overall detection performance of these modules after being integrated into the DSOD-SM framework. It is mainly pointed out that the combined effect of the ST and MFC modules is quite significant, further improving mAP@0.5 to 41.3%. On this basis, the MFA module is introduced again, and the mAP@0.5 jumps to 41.8%; finally, when the SFA module is also included in the combination, the mAP@0.5 index rises to 42.3%, achieving a 0.5% improvement. And the precision reaches 54.4%, achieving a 3 percentage point increase, which fully demonstrates the significant advantage of the SFA module in enhancing the model's ability to accurately locate dense and small objects.

[0167] Table 2 Statistics of all target bounding boxes in the VisDrone2019 training set

[0168] Pixel size <![CDATA[<200 2 ]]> <![CDATA[200 2 ~400 2 ]]> <![CDATA[>400 2 ]]> <![CDATA[<32 2 ]]> <![CDATA[32 2 ~96 2 ]]> <![CDATA[>96 2 ]]> quantity 487887 2035 42 3006262 159999 23703

[0169] In summary, the above experiments not only verify the independent contribution of each module, but also deeply reveal the powerful performance gain of dense small target detection when they work together.

[0170] 2.3.2 Ablation experiment of feature aggregation unit

[0171] In order to further verify the effectiveness of NAM (asymmetric normalized attention module) in the feature aggregation units MFC and MFA, two groups of ablation experiments were carried out on the basis of the baseline model: one group is to integrate only MFC and MFA without channel attention, and the other group is to integrate MFC and MFA with complete functions (including NAM). The experimental results are shown in Table 4. When the standard MFC and MFA are introduced into the baseline method, the overall performance of the system is significantly improved, specifically, the mAP@0.5 jumps from 40.7% and 39.9% to 41.1% and 40.7% respectively, and the mAP@0.75 gains of 0.4% are obtained respectively.

[0172] Further analysis shows that when the channel attention mechanism is not introduced in MFC and MFA, its multi-scale feature fusion module based on multi-branch weights mainly focuses on the single relationship between global channels, while ignoring the more complex and critical interdependence between channels. This limitation restricts the model's ability to capture feature details. However, by integrating the NAM module, the importance of each channel in the feature map can be captured. This improvement not only enhances the model's perception of the importance of local features, but also promotes the overall optimization of the overall metric.

[0173] The above experimental results demonstrate the effectiveness of NAM in MFC and MFA. It successfully makes up for the shortcomings of the original feature aggregation unit in processing inter-channel correlation, significantly improves the overall performance of the model, and further verifies the key role of NAM in focusing on and strengthening the importance of local features.

[0174] Table 3 Ablation experiments on VisDrone2019

[0175]

[0176]

[0177] Table 4 Ablation experiments of MFC / MFA with and without NAM

[0178]

[0179] 2.4 Comparative Experiment

[0180] 2.4.1 Comparative Experiments on the Visdrone-2019 Dataset

[0181] In order to comprehensively evaluate the performance of the model, the following comparison networks are selected: YOLOv5s (6.1), YOLOv7-Tiny, YOLOXs and YOLOv8s. The experiments were carried out on the validation set (VisDrone2019-val) and test set (VisDrone2019-test) of the VisDrone2019 dataset, and the complete experimental results are summarized in Table 5. Among them, DSOD-SM achieved 42.3% and 34.6% mAP@0.5 on VisDrone2019-val and VisDrone2019-test, respectively. At the same time, it also achieved 26.2% and 20.0% detection accuracy on mAP@0.75, and 25.5% and 19.7% on mAP@0.5:0.95. Although DSOD-SM has an increase of about 1.4M parameters compared to the baseline model, this small increase has resulted in a significant improvement of 1.8% and 1.4% in mAP@0.5 and mAP@0.75 on the validation set. The above results show that DSOD-SM has shown obvious advantages over other mainstream methods in recent years. Figure 6 The comparison visualization of the detection results of the proposed algorithm and the baseline model is intuitively shown. Figure 6 (a) is the detection result of the baseline model. The black box in the figure represents the small target missed by the baseline model. Figure 6 (b) is the detection result of DSOD-SM. It can be clearly observed from the figure that the proposed algorithm significantly reduces missed detection and false detection when dealing with dense small targets, thereby greatly improving the accuracy and reliability of detection. In addition, Figure 7 The training advantages of the proposed algorithm are further demonstrated through the comparison of the training process. During the training process, all its metrics show faster convergence speed and higher final accuracy than the baseline model, which not only reflects the superiority of the DSOD-SM network structure, but also proves the effectiveness of its optimization algorithm and strategy.

[0182] In summary, through the above innovative design and experiments, DSOD-SM can successfully improve the performance of target detection significantly, especially in complex scenes and dense small target detection, and has good detection effect.

[0183] Table 5 Comparison of experimental results of each model on VisDrone2019

[0184]

[0185] 2.4.2 Comparative Experiments on Vehicle Detection Dataset

[0186] In order to further verify the effectiveness of the proposed DSOD-SM, the public CARPK dataset and PUCPR+ dataset were used to conduct comparative experiments on the model. 640×640 images were used as the input of the target detection model. The models included MobileNetv3-L, GhostNetv2, YOLOv5s (6.1), YOLOv7-Tiny, YOLOXs and YOLOv8. The algorithms were executed for 300 rounds. The experimental results are shown in Table 6, where the bold value represents the highest accuracy value in the column. Compared with the mainstream dense small target detection model, although the proposed algorithm has a large number of parameters, it shows a relatively superior performance in mAP. Compared with the baseline model, the Precision, mAP@0.5 and mAP@0.5:0.95 on the CARPK dataset and PUCPR+ dataset have been significantly improved, with an increase of 2.0 percentage points and 2.2 percentage points in mAP@0.5 respectively. Figure 8 The actual effect of DSOD-SM in vehicle detection tasks is demonstrated. Figure 8 (a) is the original image, Figure 8 (b) is the detection effect diagram of DSOD-SM. Faced with a large number of small aerial targets, the algorithm can effectively identify the vast majority of small targets while maintaining a high detection accuracy, which fully proves that the detection method proposed in this paper has strong generalization and robustness for densely packed small aerial targets.

[0187] Table 6 Comparison of experimental results of each model on the vehicle detection dataset

[0188]

[0189] 2.4.3 Comparative Experiments on Feature Alignment Attention Mechanism

[0190] In order to further evaluate the actual performance of the spatial feature alignment attention mechanism, a variety of currently popular attention mechanisms, such as SE, CBMA, CA and GC, are embedded at the end of the backbone network, and the experimental results are systematically recorded in Table 7. Compared with other attention mechanisms, the proposed attention mechanism is almost the same in terms of computational complexity and parameter quantity. Although CBMA performs more prominently in mAP@0.5, reaching 32.2%, the mAP@0.5:0.95 of SFA is 0.3% higher than that of CBMA, and the parameter quantity of CBMA is higher. Fig. 9The responses of different attention mechanisms to key features are presented intuitively and visualized in the form of heat maps. Figure (a) is the feature heat map of the last layer of the baseline backbone, which is used as a benchmark reference; Figures (b), (c), and (d) are the feature heat maps after adding SE, CBAM, and SFA to the last layer of the baseline backbone, respectively. It can be seen that the proposed SFA attention mechanism has a higher sensitivity to key feature information, and its focus area is wider and more accurate, which reduces the model's missed detection rate when dealing with dense small targets, laying a good foundation for improving the overall performance of detection tasks.

[0191] Table 7 Comparison of attention mechanism results on the VisDrone2019 dataset

[0192]

[0193] A large number of experimental results on the above classic aerial photography and vehicle detection datasets show that the mAP@50 of the algorithm proposed in this paper has increased by 1.8%, 2.0% and 2.2% respectively, and has better dense small target detection performance. It can effectively improve the efficiency and accuracy of military reconnaissance and enhance battlefield situation awareness, and will provide strong support for military reconnaissance, target recognition and tracking.

[0194] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for detecting dense small targets in aerial photography based on spatial feature alignment and multi-scale feature aggregation, characterized in that: include: Collect an image containing the target, then input the image into the DSOD-SM network model, and the DSOD-SM network model outputs the target detection result; The DSOD-SM network model includes a backbone network, a feature fusion module and a detection head; the data output end of the backbone network is connected to the data input end of the feature fusion module, and the data output end of the feature fusion module is connected to the data input end of the detection head; The backbone network adopts the CSPDarknet network as a feature extractor, including a first Conv module, a second Conv module, a first C2f module, a third Conv module, a second C2f module, a fourth Conv module, a third C2f module, a fifth Conv module, a fourth C2f module and an SPPF module connected in sequence; wherein the second C2f module, the third C2f module and the fourth C2f module output the first primary feature, the second primary feature and the third primary feature respectively; The feature fusion module includes three feature fusion branches, namely, the first feature fusion branch, the second feature fusion branch, and the third feature fusion branch, and its inputs are respectively the first primary feature, the second primary feature, and the third primary feature; The first feature fusion branch includes a first MFC module, a fifth C2f module, and a first MFA module connected in sequence; the second feature fusion branch includes a second MFC module, a sixth C2f module, a third MFC module, a seventh C2f module, and a second MFA module connected in sequence, and the data output end of the sixth C2f module is connected to the data input end of the second MFA module; the third feature fusion branch includes a fourth MFC module, an eighth C2f module, and a third MFA module connected in sequence; The connection relationship between the first feature fusion branch and the second feature fusion branch is: the data input end of the first MFC module is connected to the data output end of the first upsampling module, and the data input end of the first upsampling module is connected to the data output end of the second MFC module; the data output end of the fifth C2f module is connected to the data input end of the second MFA module; The connection relationship between the second feature fusion branch and the third feature fusion branch is: the data input end of the second MFC module is connected to the data output end of the second up-sampling module, and the input of the data input end of the second up-sampling module is the third primary feature; the data input end of the fourth MFC module is connected to the data output end of the second down-sampling module, and the data input end of the second down-sampling module is connected to the data output end of the third MFC module; The MFC module includes: F mfc It is the output feature that is fused in a cascade manner; Conv() is a normal standard convolution; Indicates pixel-by-pixel addition; L1, L2, and L3 represent the first resolution feature, the second resolution feature, and the third resolution feature, respectively; ω1, ω2, ω3 represent the weight parameter of the first branch, the weight parameter of the second branch, and the weight parameter of the third branch respectively; The MFA module includes: F mfc =Conv(Concat[ω1·L1,ω2·L2,ω3·L3]) where F mfa It is the output feature fused by matrix and method; Concat represents a concatenation operation; The detection head includes three sub-detection heads, which are respectively used for feature detection of three feature fusion branches.

2. According to claim 1, a method for detecting dense small targets in aerial photography based on spatial feature alignment and multi-scale feature aggregation is characterized in that: The DSOD-SM network model also includes a feature alignment module. At this time, the data output end of the backbone network is connected to the data input end of the feature alignment module, and the data output end of the feature alignment module is connected to the data input end of the feature fusion module: The data output ends of the second C2f module, the third C2f module, and the fourth C2f module are respectively connected to the data input ends of the first feature alignment module, the second feature alignment module, and the third feature alignment module, and the data output ends of the first feature alignment module, the second feature alignment module, and the third feature alignment module are respectively connected to the data input ends of the first MFC module, the second MFC module, and the fourth MFC module; In addition, the data output terminal of the third feature alignment module is also connected to the data input terminal of the second up-sampling module; The feature alignment module comprises: First, perform global average pooling on the input feature X, and obtain the feature vector F GAP , the global average pooling is expressed as: Among them, G d Represents the output value of the d-th channel feature map of the input feature F after global average pooling; h and w represent the size of the feature map respectively; F d (i, j) represents the element at (i, j) in the d-th channel feature map region; Then F GAP As input, use the fully connected layer f c / 32 and f 32 / c Learning feature alignment coefficients As shown in the following formula: Among them, f c / 32 Represents the first fully connected layer; f 32 / c represents the second fully connected layer; σ represents the sigmoid activation function, and its formula is: Among them, exp() is the natural exponential function; x represents the output vector of the fully connected layer; Then the spatial alignment correction coefficient is applied to the transposed feature matrix F T , and get the aligned spatial attention map M: in, represents the feature alignment coefficient; · T represents transpose; F represents the input feature; Finally, the obtained spatial attention map is fused to the input feature map to obtain the final output features: F"=reshape(F T ·M), Among them, reshape() means matrix dimension reconstruction.

3. The method for detecting dense small targets in aerial photography based on spatial feature alignment and multi-scale feature aggregation according to claim 2, characterized in that: It also includes: the data output end of the first feature alignment module is connected to the data input end of the first MFA module, the data output end of the second feature alignment module is connected to the data input end of the second MFA module, and the data output end of the third feature alignment module is connected to the data input end of the third MFA module.

4. The method for detecting dense small targets in aerial photography based on spatial feature alignment and multi-scale feature aggregation according to claim 1, characterized in that: The backbone network uses the CSPDarknet network as a feature extractor, and replaces the third and subsequent Conv modules with scale scaling modules to perform preliminary feature extraction on the collected images; The scale scaling module includes: First, the input feature X is spatially downsampled using stride convolution with a convolution kernel of 2 and maximum pooling to obtain two feature maps F of the same size. c and F m : F c =Conv 2×2 (X), F m =MaxPool(X), Then the two feature maps are added pixel by pixel, and the final sampling information can be expressed as F'=F c +F m 。 5. The method for detecting dense small targets in aerial photography based on spatial feature alignment and multi-scale feature aggregation according to claim 1, characterized in that: ω1, ω2, ω3 are obtained through the weight learner: First, the weight learner concatenates the different resolution features {L1, L2, L3} at the same level to obtain the feature F cat The implementation is as follows: F cat =Concat(L1,L2,L3), Next, the feature F cat Perform a global average pooling operation to obtain F g , and then use the multi-layer perceptron MLP to learn the feature branch weight parameter W, W = {W1, W2, W3}. The learning process is: W=MLP(F g ), Among them, MLP is a multi-layer perceptron; Normalize the output of the multilayer perceptron to obtain the branch weight parameter ω={ω1,ω2,ω3}: where ω i Represents the normalized weight parameter of the i-th branch; W i is the weight information of the i-th branch learned by the multilayer perceptron; ε represents the smoothness term.

6. The method for detecting dense small targets in aerial photography based on spatial feature alignment and multi-scale feature aggregation according to claim 1, characterized in that: F obtained for the MFC module mfc 、F obtained by MFA module mfa Use convolution for feature fusion, and add batch normalization and sigmoid activation function after convolution.

7. The method for detecting dense small targets in aerial photography based on spatial feature alignment and multi-scale feature aggregation according to claim 6, characterized in that: Also includes: Then use the NAM channel attention mechanism to perform in-depth analysis on the adjusted features.

Images