Aerial image small target detection method based on Yolov7 and Transform

Abstract

The invention belongs to the technical field of unmanned aerial vehicle image processing, and provides a Yolov7 and Transformer-based aerial image small target detection method, which comprises three components of a YOLOv7-based framework construction feature extraction network, a feature fusion network and a prediction layer, and is characterized in that the feature extraction network adopts a CSP-Darknets3 framework and is added with a neighborhood attention encoder NAT, and the feature fusion network adopts a CSP-Darknets3 framework; transmitting the output of the feature extraction network to a feature fusion network, and carrying out multi-scale detection; wherein the Neck part firstly performs Gaussian scaling mitigation scale processing on outputs of the first two NAT modules after the third down-sampling module and splicing, and then inputs the outputs into a subsequent network, and the outputs of the second two NAT modules are not subjected to Gaussian scaling operation; according to the method, a convolutional neural network is combined with an EdgeNeXt hybrid architecture, a lightweight convolutional coding network LCEN is proposed, and an embedded hierarchical attention network is introduced in a feature extraction stage to balance the detection accuracy and reasoning speed of a model.

Description

Technical Field

[0001] The invention belongs to the technical field of unmanned aerial vehicle image processing, and in particular to an aerial image small target detection method based on Yolov7 and Transformer. Background Art

[0002] Object detection is a challenging task in the field of computer vision, and it is one of the foundations of other complex visual tasks. Compared with simple image classification tasks, object detection tasks require not only classifying and marking multiple objects in an image, but also accurately locating the rectangular box where the object is located. In object detection, small object detection is a challenging subtask that involves finding objects with small size and few visual features in the image. Aerial small object detection is the application of this challenging task to aerial images. Images acquired from drones usually have the characteristics of large frames, more noise interference, a wide range of object scale variations, and a high proportion of small objects;

[0003] There are many challenges in object detection in aerial images: compared with natural scene images, aerial images are usually larger in size and have more redundant background information due to their different imaging perspectives; the objects are sparsely distributed and contain a large number of small objects; dense objects are easily occluded, while edge objects are easily truncated; the object sizes vary widely, and the distribution of object categories is uneven, resulting in a long-tail effect; in addition, the shooting angle of the object is not fixed.

[0004] Deep learning-based methods have significant advantages in improving the speed and accuracy of small target detection. However, the currently used general target detection networks, such as FasterR-CNN and YOLO, have some limitations. These networks lack efficient mechanisms in feature extraction and fusion, especially in the fusion and utilization of features of different scales. Therefore, although these methods have improved the performance of small target detection to a certain extent, there is still room for improvement, especially the need for more flexible and efficient feature extraction and fusion strategies to better cope with the challenges of different scenarios and target scales. The limitations of existing aerial photography small target detection algorithms in processing complex images, sparse distribution and a large number of small targets, as well as the deficiencies in extracting target semantic information, retaining spatial information, detection speed and accuracy, lead to common false detection and missed detection problems. At the same time, it is difficult to achieve an ideal balance between speed and accuracy when deploying models to resource-constrained edge devices.

[0005] To this end, the present invention proposes a small target detection method based on YOLOv7 and Transformer in aerial photography, aiming to improve the accuracy and efficiency of target detection. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a small target detection method for aerial images based on Yolov7 and Transformer to solve the problems raised in the background technology.

[0007] A small target detection method for aerial images based on Yolov7 and Transformer includes three components: feature extraction network, feature fusion network and prediction layer based on the YOLOv7 framework. The feature extraction network adopts the CSP-Darknets3 framework and adds a neighborhood attention encoder NAT. The output of the feature extraction network is passed to the feature fusion network for multi-scale detection. Among them, the Neck part, after the third downsampling module and splicing, first performs Gaussian scaling and scale processing on the outputs of the first two NAT modules, and then inputs them into the subsequent network, while the outputs of the last two NAT modules are not subjected to Gaussian scaling operation.

[0008] Preferably, in the feature extraction network, the initial input is firstly subjected to lossless double downsampling through the Focus structure to concentrate the image feature information into the channel dimension; then the feature information is extracted through the CSP structure, and the output obtained through the spatial pyramid pooling SPP processing is sent to the neighborhood attention encoder module;

[0009] NAT adopts a multi-level hierarchical design and uses overlapping convolutions with small kernels for downsampling. At the same time, after each level, overlapping convolutions with small kernels are applied for embedding and downsampling. First, the input is layer-normalized LN according to the dimension of the hidden layer, and is adjacent to a multi-head neighborhood attention module MHNA, which uses skip connection operations to connect features at different levels together. At the same time, it is processed by a multi-layer perceptron before output. The perception range of each image block is limited to its neighborhood through the neighborhood attention Transformer. By introducing a local inductive bias similar to convolution, the model can retain local detail information while extracting global feature information.

[0010] When selecting the key vector and value vector, NA requires selection from a neighborhood centered on the query vector. The calculation formula for NA on a single pixel is as follows:

[0011]

[0012] Among them, the input vector is represented by X, the query vector Q, the key vector K and the value vector V, the relative position deviation B, is added to each attention weight according to the relative position, and scale represents the normalized scaling factor; p(i,j) represents the neighborhood space of the pixel at the (i,j) position, the size of L*L, when L=3, the neighborhood space corresponds to a 9*9 grid; when L is extended to include all pixels, the neighborhood attention is degenerated into self-attention.

[0013] Preferably, after convolution processing, each position in the input feature map captures the information of the local area of ​​the original image. CNN integrates feature information from different levels and adopts a dual-space aggregation pyramid TSAP module. Its dual-space aggregation pyramid network integrates the feature pyramid network FPN and the pyramid aggregation network PAN, and introduces a coordinate attention mechanism to obtain a wider range of dependency information.

[0014] The channel attention is decomposed into two parallel one-dimensional feature codes by the coordinate attention mechanism. The two feature maps have a specific direction through the encoded information. The longer-range dependency information in the input feature map can be obtained along a spatial direction through each feature map. The two pooled vectors are spliced ​​together along the spatial dimension, and then the number of channels is reduced through a point convolution operation. The result is regularized and a nonlinear activation function is applied to finally generate a feature map. The generated feature map is divided into multiple parts along the spatial dimension, and then each part is subjected to a point convolution operation to restore it to the same number of channels as the input feature; at the same time, each part is sigmoid normalized and weighted to obtain the final output; the specific calculation formula is as follows:

[0015]

[0016] f=δ(F1([z h ,z w ]))

[0017] g h =σ(F h (f h ))

[0018] g h =σ(F w (f w ))

[0019] y c (i,j)=x c (i,j)*g c h (i)*g c w(j)

[0020] in, Respectively represent the feature vectors after global pooling of spatial information on the channel in the horizontal and vertical directions, and input feature vector x c , input feature width W, the height of the input feature is H, the number of channels is represented by C, f represents the output feature map after regularization and nonlinear activation, δ(·) represents the hard_sigmoid nonlinear activation function, F1 represents regularization, and the encoded vector is represented by z along the horizontal coordinate and vertical left. h 、z w It means that, with the input x c The horizontal component with the same number of channels is represented by g h Indicates that F h represents the regularization in the horizontal direction, f h is the horizontal component of f, g w Represents the same as input x c The vertical component of the same number of channels, F w represents the regularization in the vertical direction, f w is the vertical component of f, σ(·) represents the ReLu6 nonlinear activation function, y c (i,j) represents the coordinate attention at pixel (i,j), represents the equivalent weight in the horizontal direction, Represents the equivalent weight in the vertical direction.

[0021] Preferably, the activation function in the original network convolution module is replaced with MetaACON-C, so that its activation function can adaptively select whether to activate neurons as needed, expressed as:

[0022] ACON-C(x)=S β (p1x,p2x)=(p1-p2)x*σ[β(p1-p2)x]+p2x

[0023]

[0024] In the formula, S β (·) represents the smooth differentiable variant of the MAX number, β represents the smoothing factor, when β→∞, S β (·) is the standard MAX function. p1 and p2 use two learnable parameters for adaptive adjustment. c (·)∈{0,1} represents the adaptive function of the channel dimension, H represents the height of the feature map, W represents the width of the feature map, and x c,h,w represents the input feature vector with c channels, h height and w width. σ represents the sigmoid activation function. W1=(C,C / r) and W2=(C / r,C) represent two 1*1 convolutional layers, which are used to change the number of channels. r is the scaling factor. During the forward propagation, the β value is used to control whether the neuron is activated, that is, it is not activated when β is 0.

[0025] Preferably, the convolutional neural network is integrated with the EdgeNeXt hybrid architecture based on YOLOV7-NAT to launch the lightweight convolutional coding network LCEN, and the embedding layer attention network EHAN is introduced in the feature extraction part; its EdgeNeXt adopts the segmentation depth transposed attention coding SDTA mechanism;

[0026] The lightweight convolutional coding network LCEN is based on the YOLOv7-NAT framework and has carried out a lightweight transformation of the feature extraction network. It effectively extracts image features by embedding the hierarchical attention network EHAN. In the feature fusion link, the network continues to use the dual spatial aggregation pyramid TSAP for multi-scale feature fusion. At the same time, the prediction layer uses transposed encoding TE as the core component.

[0027] Preferably, a dynamic convolutional coding DCE module is specially introduced in the EHAN part to optimize the feature extraction process; in the shallow layer of the network, a small convolution kernel is used to capture the detailed information of the image, while in the deep network, a large convolution kernel is chosen to be used; and the DCE module is combined with the transpose encoder TE to enhance the network's ability to extract features and effectively fuse local information with global information.

[0028] Preferably, the transposed encoder TE is combined with CNN and DCE in the EHAN part. CNN and DCE can extract local detail information of the image, while the introduction of the TE module enables the network to capture global information.

[0029] Preferably, the complete intersection-over-union loss, binary cross entropy loss, and balanced focus loss of YOLOv7 are combined, supplemented by confidence loss, to form a total loss function;

[0030] When positioning the loss, CIoU is used to optimize the bounding box regression. The minimum bounding box C containing the real box and the predicted box is introduced, so that the center point of the predicted box can be close to the center point of the real box. CIoU limits the aspect ratio of the predicted box and the real box. By introducing the weight factor a, CIoU balances the IoU part loss and the aspect ratio loss. The specific calculation formula is as follows:

[0031]

[0032] L CIoU (b,b gt )=1-CIoU(b,b gt )

[0033] In the formula, b represents the prediction box, and b gt represents the true box, the variable d represents the straight-line distance between the center point of the predicted box and the center point of the true box, and c is the diagonal distance of the bounding box C. α is used as a weight factor to balance the loss calculation.gt and h gt Represent the width and height of the real box, while w and h represent the width and height of the predicted box respectively;

[0034] Balanced focus loss is improved on the basis of Focal Loss. By introducing focus factor and weight factor, it solves the imbalance between foreground and background and the imbalance within foreground, which can be expressed as:

[0035]

[0036] In the formula, p t ∈[0,1] represents the prediction confidence score of a candidate target, reflecting the model's judgment on whether the target exists or not; α t As a balance parameter between foreground and background, by combining the prediction score p t and focusing factor γ, redistribute the loss and focus on learning the foreground information; the more serious the imbalance between the foreground and the background, the larger the γ value; j represents the focusing factor of the jth class, which adjusts the weight of the loss function according to the distribution of different categories, and γ b and Represent the factors that are irrelevant to the category and relevant to the category respectively. Through their collaboration, Focus the model’s learning attention on the imbalance between positive and negative samples; j represents the weight factor of the jth class, which is used to assign a larger weight value to the rare class; for the class with a high proportion, its corresponding w j The value is close to 1;

[0037] The balanced focus loss is used as the classification loss function, which is expressed as:

[0038]

[0039] In the formula, N represents the total number of categories, α t is the equilibrium parameter, w i represents the weight factor of the i-th category, γ i represents the focusing factor of the i-th category, p t is the confidence score of the predicted box.

[0040] Preferably, confidence is an indicator used to measure the credibility of the prediction box. The confidence loss is calculated by using binary cross entropy loss and combining it with the fused classification and positioning loss to form a total loss function; it is expressed as:

[0041]

[0042] In the formula, N represents the number of detection layers, L cls represents the classification loss, L loc represents the bounding box regression loss, L obj is the confidence loss, and λ1, λ2, and λ3 represent the weights of the three losses.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention combines the target detection algorithm YOLOv7 and introduces the neighborhood attention Transformer (NAT) module on its basis. NAT retains the advantages of the visual Transformer by introducing position offset information, and further improves the effect of local feature extraction on this basis. At the same time, a lightweight convolutional coding network LCEN is adopted, and an embedded hierarchical attention network is introduced in the feature extraction stage to balance the detection accuracy and reasoning speed of the model. The experimental data fully proves that the method proposed by the present invention has a significant effect in solving the problem of small target detection in aerial images. In addition, when compared with other mainstream aerial small target detection models on two data sets, the method of the present invention shows a higher average accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The overall network structure diagram of the improved YOLOv7-NAT of the present invention;

[0046] Figure 2 Note the encoder structure diagram for the field of the present invention;

[0047] Figure 3 Note the Transformer structure diagram for the field of the present invention;

[0048] Figure 4 It is the coordinate attention structure diagram of the present invention;

[0049] Figure 5 The EdgeNeXt network structure diagram of the present invention;

[0050] Figure 6 The overall structure diagram of the lightweight convolutional coding network of the present invention;

[0051] Figure 7 It is a structural diagram of a dynamic convolutional coding module of the present invention;

[0052] Figure 8 This is a structural diagram of the transposition encoder of the present invention. DETAILED DESCRIPTION

[0053] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0054] As attached Figure 1 To Attachment Figure 8 As shown:

[0055] Embodiment: The present invention provides a method for detecting small targets in aerial images based on Yolov7 and Transformer, including three components: a feature extraction network, a feature fusion network and a prediction layer based on the YOLOv7 framework. The feature extraction network adopts the CSP-Darknets3 framework, and adds a neighborhood attention encoder NAT. The output of the feature extraction network is passed to the feature fusion network for multi-scale detection. The Neck part, after the third downsampling module and splicing, first performs Gaussian scaling and scale processing on the outputs of the first two NAT modules, and then inputs them into the subsequent network, while the outputs of the last two NAT modules do not undergo Gaussian scaling operations. The overall network structure of the improved YOLOv7-NAT is as follows: Figure 1 shown.

[0056] In the feature extraction network, the initial input first passes through the Focus structure for lossless double downsampling, which helps to concentrate the image feature information into the channel dimension; then the feature information is extracted through the CSP structure, and the output obtained through spatial pyramid pooling SPP processing is sent to the neighborhood attention encoder module; the purpose of this step is to effectively obtain global information, thereby improving the detection ability of small targets in aerial images. The structure of NAT is as follows Figure 2 shown.

[0057] NAT adopts a multi-level hierarchical design and uses overlapping convolutions with small kernels for downsampling instead of conventional overlapping convolutions. At the same time, after each level, overlapping convolutions with small kernels are applied for embedding and downsampling. This method can not only reduce the spatial size while improving the speed and accuracy of model detection, but also effectively improve the reasoning speed. First, the input will be layer-normalized LN according to the dimension of the hidden layer, and adjacent to a multi-head neighborhood attention module MHNA, using jump connection operations to connect features of different levels together, and at the same time, it is processed by a multi-layer perceptron before output. The perception range of each image block is limited to its neighborhood through the neighborhood attention Transformer. By introducing a local inductive bias similar to convolution, the model can retain local detail information while extracting global feature information. The structure of the domain attention Transformer is as follows: Figure 3 shown.

[0058] When selecting the key vector and value vector, NA requires selection from a neighborhood centered on the query vector. The calculation formula for NA on a single pixel is as follows:

[0059]

[0060] Among them, the input vector is represented by X, the query vector Q, the key vector K and the value vector V, the relative position deviation B, is added to each attention weight according to the relative position, and scale represents the normalized scaling factor; p(i,j) represents the neighborhood space of the pixel at the (i,j) position, the size of L*L, when L=3, the neighborhood space corresponds to a 9*9 grid; when L is extended to include all pixels, the neighborhood attention is degenerated into self-attention.

[0061] After convolution processing, each position in the input feature map captures the information of the local area of ​​the original image. CNN integrates feature information from different levels and adopts a dual-space aggregation pyramid TSAP module. Its dual-space aggregation pyramid network integrates the feature pyramid network FPN and the pyramid aggregation network PAN, and introduces a coordinate attention mechanism to obtain a wider range of dependency information; the coordinate attention structure is as follows Figure 4 shown.

[0062] The channel attention is decomposed into two parallel and one-dimensional feature encodings by the coordinate attention mechanism. The two feature maps have a specific direction through the encoded information. The long-range dependency information in the input feature map can be obtained along a spatial direction through each feature map. The two pooled vectors are spliced ​​together along the spatial dimension, and then the number of channels is reduced through a point convolution operation. The result is regularized and a nonlinear activation function is applied to finally generate a feature map. The generated feature map is divided into multiple parts along the spatial dimension, and then each part is subjected to a point convolution operation to restore it to the same number of channels as the input feature; at the same time, each part is sigmoid normalized and weighted to obtain the final output; in the deep feature space, through the coordinate attention mechanism, the network can pay more attention to the target area, obtain a larger receptive field and more accurate spatial position encoding information, so that the positioning accuracy of the model is further improved. The specific calculation formula is as follows:

[0063]

[0064] f=δ(F1([z h ,z w ]))

[0065] g h =σ(F h (f h ))

[0066] gh =σ(F w (f w ))

[0067]

[0068] in, Respectively represent the feature vectors after global pooling of spatial information on the channel in the horizontal and vertical directions, and input feature vector x c , input feature width W , the height of the input feature is H, the number of channels is represented by C, f represents the output feature map after regularization and nonlinear activation, δ(·) represents the hard_sigmoid nonlinear activation function, F1 represents regularization, and the encoded vector is represented by z along the horizontal coordinate and vertical left. h 、z w It means that, with the input x c The horizontal component with the same number of channels is represented by g h Indicates that F h represents the regularization in the horizontal direction, f h is the horizontal component of f, g w Represents the same as input x c The vertical component of the same number of channels, F w represents the regularization in the vertical direction, f w is the vertical component of f, σ(·) represents the ReLu6 nonlinear activation function, y c (i,j) represents the coordinate attention at pixel (i,j), represents the equivalent weight in the horizontal direction, Represents the equivalent weight in the vertical direction.

[0069] The activation function in the original network convolution module is replaced with MetaACON-C, so that its activation function can adaptively choose whether to activate neurons as needed, thereby further improving the accuracy of the network, which is expressed as:

[0070] ACON-C(x)=S β (p1x,p2x)=(p1-p2)x*σ[β(p1-p2)x]+p2x

[0071]

[0072] In the formula, S β (·) represents the smooth differentiable variant of the MAX number, β represents the smoothing factor, when β→∞, S β (·) is the standard MAX function. p1 and p2 use two learnable parameters for adaptive adjustment. c(·)∈{0,1} represents the adaptive function of the channel dimension, H represents the height of the feature map, W represents the width of the feature map, and x c,h,w represents the input feature vector with c channels, h height and w width. σ represents the sigmoid activation function. W1=(C,C / r) and W2=(C / r,C) represent two 1*1 convolutional layers, which are used to change the number of channels. r is the scaling factor. During the forward propagation, the β value is used to control whether the neuron is activated, that is, it is not activated when β is 0.

[0073] Based on YOLOV7-NAT, the convolutional neural network and EdgeNeXt hybrid architecture are integrated to launch the lightweight convolutional coding network LCEN, and the embedded layer attention network EHAN is introduced in the feature extraction part, aiming to ensure the detection accuracy and reasoning speed of the model at the same time, so as to meet the high-efficiency performance requirements in practical applications; its EdgeNeXt adopts the segmentation depth transposed attention encoding SDTA mechanism, which enables the model to efficiently integrate local and global information, while significantly reducing the complexity of the model, thereby improving the reasoning speed of the model; the overall network structure of EdgeNeXt is as follows Figure 5 shown.

[0074] The lightweight convolutional coding network LCEN is based on the YOLOv7-NAT framework. It has lightened the feature extraction network and effectively extracted image features by embedding the hierarchical attention network EHAN, so that the model can still maintain good performance under limited resources. In the feature fusion link, the network continues to use the dual spatial aggregation pyramid TSAP for multi-scale feature fusion to meet the detection needs of targets of different sizes. At the same time, in order to improve the detection performance, the prediction layer uses transposed encoding TE as the core component, aiming to ensure that LCEN can achieve satisfactory detection accuracy while maintaining efficient inference speed. The overall network architecture of the lightweight convolutional coding network LCEN is shown below. Figure 6 shown.

[0075] In the EHAN part, the dynamic convolutional coding DCE module is specially introduced to optimize the feature extraction process; in the shallow layer of the network, a small convolution kernel is used to capture the detailed information of the image, while in the deep network, a large convolution kernel is used to ensure a larger receptive field while reducing the computational complexity; and the DCE module is combined with the transposed encoder TE to enhance the network's ability to extract features, effectively integrate local information with global information, and further improve the detection performance. The specific structure of the dynamic convolutional coding module is as follows Figure 7 shown.

[0076] In the EHAN part, the transpose encoder TE is combined with CNN and DCE. CNN and DCE can extract local detail information of the image, while the introduction of the TE module enables the network to capture global information. This combination enables the network to make fuller use of the feature information of small targets, thereby significantly improving the detection performance. The specific structure of the transpose encoder is as follows Figure 8 shown.

[0077] The complete intersection-over-union loss, binary cross entropy loss, and balanced focus loss of YOLOv7 are combined with confidence loss to form the total loss function; the loss function can not only stabilize the training process of the network, but also effectively solve the imbalance problem between foreground categories, thereby improving the performance of the model.

[0078] When the positioning loss is calculated, CIoU is used to introduce the minimum bounding box C containing the real box and the predicted box in the optimized bounding box regression, so that the center point of the predicted box can be close to the center point of the real box. This is achieved by considering the straight-line distance d between the predicted box and the center point of the bounding box; and CIoU also limits the aspect ratio of the predicted box and the real box to further improve the positioning accuracy. By introducing the weight factor a, CIoU balances the I oU partial loss and the aspect ratio loss, thereby achieving accurate calculation of multi-dimensional losses. This method not only improves the positioning effect, but also makes the training process more stable and converges faster. The specific calculation formula is as follows:

[0079]

[0080] L CIoU (b,b gt )=1-CIoU(b,b gt )

[0081] In the formula, b represents the prediction box, and b gt represents the true box, the variable d represents the straight-line distance between the center point of the predicted box and the center point of the true box, and c is the diagonal distance of the bounding box C. α is used as a weight factor to balance the loss calculation. gt and h gt Represent the width and height of the real box, while w and h represent the width and height of the predicted box respectively; these variables together constitute the calculation basis of the CI oU loss function, enabling the model to locate the target more accurately and optimize the match between the predicted box and the real box.

[0082] Balanced focus loss is improved on the basis of Focal Loss. By introducing focus factor and weight factor, it solves the imbalance between foreground and background and the imbalance within foreground, which can be expressed as:

[0083]

[0084] In the formula, p t ∈[0,1] represents the prediction confidence score of a candidate target, reflecting the model's judgment on whether the target exists or not; α t As a balance parameter between foreground and background, by combining the prediction score p t and focusing factor γ, redistribute the loss and focus on learning foreground information; when the imbalance between foreground and background is more serious, the larger the γ value is, which helps the model better cope with the challenge of imbalanced data; γ j represents the focusing factor of the jth class, which adjusts the weight of the loss function according to the distribution of different categories, and γ b and Represent the factors that are irrelevant to the category and relevant to the category respectively. Through their collaboration, Focus the model’s learning attention on the imbalance between positive and negative samples; j represents the weight factor of the jth class, which is used to assign a larger weight value to rare classes, thereby increasing the contribution of these classes in the loss function and improving the model's ability to recognize them; for classes with a high proportion, the corresponding w j The value is close to 1, which keeps the loss function stable on these categories;

[0085] In order to improve the detection accuracy of the model and reduce the negative impact of uneven data distribution on the training process, balanced focus loss is used as the classification loss function, which is expressed as:

[0086]

[0087] In the formula, N represents the total number of categories, α t is the equilibrium parameter, w i represents the weight factor of the i-th category, γ i represents the focusing factor of the i-th category, p t is the confidence score of the predicted box.

[0088] Confidence is an indicator used to measure the credibility of the prediction box. The confidence loss is calculated by using binary cross entropy loss and combining it with the fused classification and positioning loss to form a total loss function; it is expressed as:

[0089]

[0090] In the formula, N represents the number of detection layers, L cls represents the classification loss, L loc represents the bounding box regression loss, L obj is the confidence loss, and λ1, λ2, and λ3 represent the weights of the three losses.

[0091] Experimental example: In order to verify the performance of the improved YOLOv7 combined with the domain attention transformer algorithm in aerial photography detection, this experiment selected the aerial photography dataset VisDrone-Det for training and testing. In order to further verify the practicality and generalization ability of this method, the method was also applied to the UVADT dataset, and a comparative analysis of the detection performance was performed.

[0092] During the training of LCEN, EHAN was selected as the feature extraction network, TSAP as the feature fusion network, and TE as the prediction layer. Since VisDrone-Det is a small dataset, a total of 100 iterations were performed during the entire training process. At the beginning of the training, three rounds of warm-up training phases were set to gradually adapt the model to the learning data distribution. During this process, the SGD optimizer was used, and the momentum was set to the default value of 0.937, and the weight decay was set to 5e-4 to prevent overfitting. During the warm-up training phase, the momentum of the SGD optimizer was specifically adjusted to 0.8, and the learning rate of each iteration was gradually adjusted by one-dimensional linear interpolation to help the model transition smoothly. After the warm-up training, the initial learning rate was set to l e-2, and in the subsequent training process, the learning rate will gradually decay to 0.2 of the initial learning rate according to the law of cosine change to ensure that the model can converge stably and achieve the best performance.

[0093] The model performance was quantitatively evaluated using the mAP (mean average precision) and AP50 indicators.

[0094] In order to verify the effectiveness of the various improvement measures proposed in this paper, this experiment carried out a detailed ablation experiment, specifically involving the removal or replacement of the NAT module, TSAP module, DCE module and TE module to form different variant networks. Subsequently, these variant networks were quantitatively compared with the method in this paper on the VisDrone-Det dataset, aiming to deeply analyze the specific contribution of each module to the detection of small aerial targets. The results are shown in Table 1 below:

[0095] Table 1 Quantitative comparison of aerial images before and after loss ablation

[0096]

[0097] As can be seen from Table 1, the present invention replaces the original NAT with three C3 structures in the last three layers of the feature extraction network of YOLOv7 to form the network structure model Our-NFAT. In the Neck part, the CSP structure is used to replace NAT, thereby obtaining a new network structure. Subsequently, Our-NFAT is compared with the YOLOv7-NAT proposed in this paper. The results show that compared with the basic network without the NAT module, the detection performance of the model is significantly improved after the introduction of the NAT module, which is specifically manifested in an increase of 3.3% in the mAP index. In the feature fusion network part, the FPN+PAN structure is used to replace TSAP to construct the network structure model Our-TSAP. Compared with YOLOv7-NAT, after the TSAP module is integrated, the detection accuracy of the model has been slightly improved, which is specifically reflected in an increase of 1.5% in the mAP index. The activation function of the feature extraction network is replaced with the SiLU activation function, and the network structure Our-ACON is constructed, which is compared with YOLOv7-NAT. The experimental results show that after the fusion of Meta ACON-C, the detection accuracy is slightly improved, which is specifically manifested in an increase of 0.9% in the mAP index. Next, the C3 module was used to replace the DCE module in the feature extraction network to construct the network structure model Our-DCE, and compared with the second network LCEN proposed in this paper. The experimental results show that after integrating the DCE module, the mAP index of the model is improved by 3.6%. In addition, we also tried to replace the TE module with three convolutional layers in the feature extraction network, and replaced the prediction layer with the CSP structure to obtain the network structure model Our-TE. Compared with the LCEN network, after integrating the TE module, the mAP index of the model is improved by 2.5%. Finally, the loss function proposed in this paper is adopted, which further improves the mAP index of the model by 2.1%.

[0098] In order to comprehensively evaluate the performance of the proposed method, a detailed quantitative comparison was conducted between the proposed method and the current mainstream aerial small target detection methods on the VisDrone-Det dataset. These mainstream methods include VistrongerDet

[23] , ViT-YOLO, etc. At the same time, TPH-YOLOv5 was used as a benchmark to compare the GFLOPs values ​​of the proposed model and its model. Table 2 clearly shows the quantitative comparison of different aerial small target detection models on the VisDrone dataset. By comparison, it can be found that the proposed method shows a better performance in detection performance and has significant advantages over other mainstream aerial small target detection models. In addition, in terms of model complexity, compared with the TPH-YOLOv5 detection model, NATCA-Greater YOLO is comparable in complexity, while the LCEN model achieves a significant reduction in complexity. This result further verifies that the proposed method can effectively reduce the complexity of the model while improving the detection performance.

[0099] Table 2 Quantitative comparison of detection results of different models on VisDrone dataset

[0100]

[0101] In order to verify the generalization ability of the model proposed in this paper, a comparative analysis was conducted with other aerial photography detection models, and the comparison results are summarized in Table 3. Through the comparison, it can be found that the method of the present invention shows superiority in detection accuracy compared with other mainstream aerial photography small target detection models. This result fully proves that the algorithm designed by the present invention has strong applicability and practicality, and can achieve excellent performance under different environments and different data sets.

[0102] Table 3 Quantitative comparison of detection results of different models on UAVDT dataset

[0103]

[0104] As can be seen from the above, the present invention proposes a new solution for detecting small targets in aerial images by integrating YOLOv7 and Transformer. Based on the general target detection network YOLOv7, the present invention makes improvements and introduces a neighborhood attention Transformer module, which is integrated with the CSP-Darknet53 backbone network to capture more extensive contextual information and rich global information. In addition, the idea of ​​coordinate attention is combined to propose a high-performance dual-space aggregation pyramid (TSAP) module, which generates position-sensitive feature maps by aggregating features in the horizontal and vertical spaces respectively, thereby significantly enhancing the network's attention to the target area and its learning ability for targets of different sizes. These improvements jointly improve the detection performance of small targets in aerial images. In view of the limitations of drones in computing resources and spatial resources, the present invention further optimizes YOLOv7-NAT and designs a lightweight network structure. The structure cleverly combines the convolutional neural network with the EdgeNeXt hybrid architecture, and innovatively proposes a lightweight convolutional coding network (LCEN). LCEN not only maintains excellent model performance, but also achieves significant improvements in inference speed, making it easier to deploy on edge devices.

[0105] Additionally, in order to provide a concise description of exemplary embodiments, all features of an actual embodiment (ie, those features that are not relevant to the best mode presently contemplated for carrying out the invention or those that are not relevant to implementing the invention) may not be described.

[0106] It should be understood that in the development of any actual implementation, as in any engineering or design project, numerous implementation-specific decisions may be made. Such a development effort may be complex and time-consuming, but for those of ordinary skill having the benefit of this disclosure, without undue experimentation, the development effort will be a routine task of design, fabrication, and production.

[0107] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. Aerial image small target detection method based on Yolov7 and Transformer, characterized by: It includes three components: feature extraction network, feature fusion network and prediction layer, which are constructed based on the YOLOv7 framework. The feature extraction network adopts the CSP-Darknets3 framework, and adds the neighborhood attention encoder NAT. The output of the feature extraction network is passed to the feature fusion network for multi-scale detection. Among them, after the third downsampling module and splicing, the Neck part first performs Gaussian scaling and scale processing on the outputs of the first two NAT modules, and then inputs them into the subsequent network, while the outputs of the last two NAT modules do not undergo Gaussian scaling operation.

2. The method for detecting small targets in aerial images based on Yolov7 and Transformer as claimed in claim 1, characterized in that: In the feature extraction network, the initial input first passes through the Focus structure for lossless double downsampling to concentrate the image feature information into the channel dimension; then the feature information is extracted through the CSP structure, and the output obtained through the spatial pyramid pooling SPP processing is sent to the neighborhood attention encoder module; NAT adopts a multi-level hierarchical design and uses overlapping convolutions with small kernels for downsampling. At the same time, after each level, overlapping convolutions with small kernels are applied for embedding and downsampling. First, the input is layer-normalized LN according to the dimension of the hidden layer, and is adjacent to a multi-head neighborhood attention module MHNA, which uses skip connection operations to connect features at different levels together. At the same time, it is processed by a multi-layer perceptron before output. The perception range of each image block is limited to its neighborhood through the neighborhood attention Transformer. By introducing a local inductive bias similar to convolution, the model can retain local detail information while extracting global feature information. When selecting the key vector and value vector, NA requires selection from a neighborhood centered on the query vector. The calculation formula for NA on a single pixel is as follows: Among them, the input vector is represented by X, the query vector Q, the key vector K and the value vector V, the relative position deviation B, is added to each attention weight according to the relative position, and scale represents the normalized scaling factor; p(i,j) represents the neighborhood space of the pixel at the (i,j) position, the size of L*L, when L=3, the neighborhood space corresponds to a 9*9 grid; when L is extended to include all pixels, the neighborhood attention is degenerated into self-attention.

3. The method for detecting small targets in aerial images based on Yolov7 and Transformer as claimed in claim 2, characterized in that: After convolution processing, each position in the input feature map captures the information of the local area of ​​the original image. CNN integrates feature information from different levels and adopts the dual-space aggregation pyramid TSAP module. Its dual-space aggregation pyramid network integrates the feature pyramid network FPN and the pyramid aggregation network PAN, and introduces the coordinate attention mechanism to obtain a wider range of dependency information. The channel attention is decomposed into two parallel one-dimensional feature codes by the coordinate attention mechanism. The two feature maps have a specific direction through the encoded information. The longer-range dependency information in the input feature map can be obtained along a spatial direction through each feature map. The two pooled vectors are spliced ​​together along the spatial dimension, and then the number of channels is reduced through a point convolution operation. The result is regularized and a nonlinear activation function is applied to finally generate a feature map. The generated feature map is divided into multiple parts along the spatial dimension, and then each part is subjected to a point convolution operation to restore it to the same number of channels as the input feature; at the same time, each part is sigmoid normalized and weighted to obtain the final output; the specific calculation formula is as follows: f=δ(F1([z h ,z w ])) g h =σ(F h (f h )) g h =σ(F w (f w )) in, Respectively represent the feature vectors after global pooling of spatial information on the channel in the horizontal and vertical directions, and input feature vector x c , input feature width W , the height of the input feature is H, the number of channels is represented by C, f represents the output feature map after regularization and nonlinear activation, δ(·) represents the hard_sigmoid nonlinear activation function, F1 represents regularization, and the encoded vector is represented by z along the horizontal coordinate and vertical left. h 、z w It means that, with the input x c The horizontal component with the same number of channels is represented by g h Indicates that F h represents the regularization in the horizontal direction, f h is the horizontal component of f, g w Represents the same as input x c The vertical component of the same number of channels, F w represents the regularization in the vertical direction, f w is the vertical component of f, σ(·) represents the ReLu6 nonlinear activation function, y c (i,j) represents the coordinate attention at pixel (i,j), represents the equivalent weight in the horizontal direction, Represents the equivalent weight in the vertical direction.

4. The method for detecting small targets in aerial images based on Yolov7 and Transformer as claimed in claim 1, characterized in that: The activation function in the original network convolution module is replaced with MetaACON-C, so that its activation function can adaptively choose whether to activate neurons as needed, expressed as: ACON-C(x)=S β (p1x,p2x)=(p1-p2)x*σ[β(p1-p2)x]+p2x In the formula, S β (·) represents the smooth differentiable variant of the MAX number, β represents the smoothing factor, when β→∞, S β (·) is the standard MAX function. p1 and p2 use two learnable parameters for adaptive adjustment. c (·)∈{0,1} represents the adaptive function of the channel dimension, H represents the height of the feature map, W represents the width of the feature map, and x c,h,w represents the input feature vector with c channels, h height and w width. σ represents the sigmoid activation function. W1=(C,C / r) and W2=(C / r,C) represent two 1*1 convolutional layers, which are used to change the number of channels. r is the scaling factor. During the forward propagation, the β value is used to control whether the neuron is activated, that is, it is not activated when β is 0.

5. The method for detecting small targets in aerial images based on Yolov7 and Transformer as claimed in claim 1, characterized in that: Based on YOLOV7-NAT, the convolutional neural network and EdgeNeXt hybrid architecture are integrated to launch the lightweight convolutional coding network LCEN, and the embedding layer attention network EHAN is introduced in the feature extraction part; its EdgeNeXt adopts the segmentation deep transposition attention coding SDTA mechanism; The lightweight convolutional coding network LCEN is based on the YOLOv7-NAT framework and has carried out a lightweight transformation of the feature extraction network. It effectively extracts image features by embedding the hierarchical attention network EHAN. In the feature fusion link, the network continues to use the dual spatial aggregation pyramid TSAP for multi-scale feature fusion. At the same time, the prediction layer uses transposed encoding TE as the core component.

6. The method for detecting small targets in aerial images based on Yolov7 and Transformer as claimed in claim 5, characterized in that: In the EHAN part, the dynamic convolutional coding DCE module is specially introduced to optimize the feature extraction process; in the shallow layer of the network, small convolution kernels are used to capture the detailed information of the image, while in the deep network, large convolution kernels are chosen to be used; and the DCE module is combined with the transpose encoder TE to enhance the network's ability to extract features and effectively integrate local information with global information.

7. The method for detecting small targets in aerial images based on Yolov7 and Transformer as claimed in claim 6, characterized in that: In the EHAN part, the transposed encoder TE is combined with CNN and DCE. CNN and DCE can extract local detail information of the image, while the introduction of the TE module enables the network to capture global information.

8. The method for detecting small targets in aerial images based on Yolov7 and Transformer as claimed in claim 1, characterized in that: The complete intersection-over-union loss, binary cross entropy loss, and balanced focus loss of YOLOv7 are combined with confidence loss to form the total loss function. When positioning the loss, CIoU is used to optimize the bounding box regression. The minimum bounding box C containing the real box and the predicted box is introduced so that the center point of the predicted box can be close to the center point of the real box. CIoU limits the aspect ratio of the predicted box to the real box. By introducing the weight factor a, CIoU balances the IoU partial loss and the aspect ratio loss. The specific calculation formula is as follows: L CIoU (b,b gt )=1-CIoU(b,b gt ) In the formula, b represents the prediction box, and b gt represents the true box, the variable d represents the straight-line distance between the center point of the predicted box and the center point of the true box, and c is the diagonal distance of the bounding box C. α is used as a weight factor to balance the loss calculation. gt and h gt Represent the width and height of the real box, while w and h represent the width and height of the predicted box respectively; Balanced focus loss is improved on the basis of Focal Loss. By introducing focus factor and weight factor, it solves the imbalance between foreground and background and the imbalance within foreground, which can be expressed as: In the formula, p t ∈[0,1] represents the prediction confidence score of a candidate target, reflecting the model's judgment on whether the target exists or not; α t As a balance parameter between foreground and background, by combining the prediction score p t and focusing factor γ, redistribute the loss and focus on learning the foreground information; the more serious the imbalance between the foreground and the background, the larger the γ value; j represents the focusing factor of the jth class, which adjusts the weight of the loss function according to the distribution of different categories, and γ b and Represent the factors that are irrelevant to the category and relevant to the category respectively. Through their collaboration, Focus the model’s learning attention on the imbalance between positive and negative samples; j represents the weight factor of the jth class, which is used to assign a larger weight value to the rare class; for the class with a high proportion, its corresponding w j The value is close to 1; The balanced focus loss is used as the classification loss function, which is expressed as: In the formula, N represents the total number of categories, α t is the equilibrium parameter, w i represents the weight factor of the i-th category, γ i represents the focusing factor of the i-th category, p t is the confidence score of the predicted box.

9. The method for detecting small targets in aerial images based on Yolov7 and Transformer as claimed in claim 8, characterized in that: Confidence is an indicator used to measure the credibility of the prediction box. The confidence loss is calculated by using binary cross entropy loss and combining it with the fused classification and positioning loss to form a total loss function; it is expressed as: In the formula, N represents the number of detection layers, L cls represents the classification loss, L loc represents the bounding box regression loss, L obj is the confidence loss, and λ1, λ2, and λ3 represent the weights of the three losses.

Images