Light unmanned aerial vehicle aerial image small target detection method and system based on TFA-YOLO11

Abstract

The invention discloses a light unmanned aerial vehicle aerial image small target detection method and system based on TFA-YOLO11, and relates to the field of target detection. The method comprises the steps of data set acquisition and data enhancement; a TFA-YOLO11 network model is constructed; training the neural network by using the data set after data enhancement to obtain a trained neural network; and detecting an image to be detected by using the trained model. According to the method, on the basis of a multi-head self-attention mechanism, complementary information between adjacent features is emphatically mined, a feature aggregation module is designed, and space and semantic information of feature maps with different scales can be better fused; according to the invention, a minimum target detection module is designed based on a high-resolution feature map, so that the sensing capability of a network to a minimum target can be enhanced; according to the method, lightweight trunk and neck networks are designed based on SPD convolution, and downsampling can be realized with smaller parameter quantity and calculation quantity under the condition of no information loss.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a method and system for detecting small targets in lightweight unmanned aerial vehicle aerial images based on TFA-YOLO11. Background Art

[0002] With the rapid development of drone technology, drones have become an important tool in modern monitoring, investigation, rescue and other fields. Drones are flexible and efficient, and can obtain large-scale, high-resolution image data, providing valuable information for various applications. Small targets refer to targets with a size less than 32*32 pixels in the image. Small target detection refers to the identification and location of small objects in the image, such as pedestrians and vehicles. In drone aerial images containing rich environmental information, small targets are often difficult to detect due to their small size, low contrast between the target and the background, and severe occlusion.

[0003] With the rapid development of deep learning, neural network-based target detection algorithms have become the mainstream methods in the field of target detection. Target detection technologies based on deep learning can generally be divided into two categories: one-stage and two-stage. The two-stage target detection algorithm first uses the candidate region generation technology to generate candidate regions that may contain targets, and then regresses the position of the target and predicts the category of these candidate regions. Representative methods include Faster R-CNN, FPN, etc. One-stage target detection does not require the extraction of candidate regions, and only requires one forward pass to obtain the target category and position. Representative methods include SSD, YOLO, etc. Two-stage target detection follows a coarse-to-fine processing mode, with high detection accuracy but slow speed, and is rarely used in engineering. In contrast, one-stage target detection is usually slightly less accurate, but has better real-time performance. YOLO has become the preferred target detection algorithm at this stage due to its faster inference speed and smaller model size.

[0004] However, these algorithms mainly focus on the detection of targets of regular size and regular perspective, and are often difficult to deploy in drone target detection tasks. There are two main reasons for this: on the one hand, ground target detection based on drone videos faces more problems such as small target size, complex and changeable background, overlap, occlusion, scale change, etc., and conventional target detection algorithms are difficult to accurately detect specific objects; on the other hand, since the reasoning process needs to be implemented in drone equipment, it is often difficult to balance high detection accuracy and lightweight models. These factors have hindered the development of deep learning methods in the field of small target detection from the perspective of drones. Summary of the invention

[0005] In order to solve the problems in the prior art, the present invention proposes a small target detection method and system for lightweight UAV aerial images based on TFA-YOLO11.

[0006] The technical solution adopted by the present invention is as follows:

[0007] In a first aspect, the present invention discloses a method for detecting small targets in lightweight UAV aerial images based on TFA-YOLO11, comprising the following steps:

[0008] Step 1): Obtain a drone aerial image dataset and perform data enhancement;

[0009] Step 2): Build the TFA-YOLO11 network model;

[0010] The TFA-YOLO11 network model includes a backbone network, a neck network, a Transformer-based feature aggregation module, a very small target detection module and a detection head module; the backbone network is used to perform multi-level downsampling and feature extraction on the input image to obtain a multi-layer feature map; the feature aggregation module is used to perform feature aggregation of adjacent layer feature maps on the multi-layer feature map to obtain high-quality aggregated feature maps of multiple different scales; the neck network and the very small target detection module together constitute a feature pyramid structure, and feature fusion is performed on high-quality aggregated feature maps of different scales through the feature pyramid structure, so that the obtained final feature map has feature information of aggregated feature maps of other scales; in addition to the part that constitutes the feature pyramid structure, the very small target detection module also includes a very small target detection head, which is used to detect small targets on the final feature map with the highest spatial resolution; except for the final feature map with the highest spatial resolution, other final feature maps are subjected to target detection through the detection head module;

[0011] Step 3): Use the data augmented data set to train the TFA-YOLO11 network model to obtain a trained TFA-YOLO11 network model;

[0012] Step 4): Use the trained TFA-YOLO11 network model to detect the drone aerial image to be detected, and obtain the position and category of all targets on the image to be detected. In a second aspect, the present invention discloses a lightweight drone aerial image small target detection system based on TFA-YOLO11 of the method, comprising:

[0013] Data enhancement module, which is used to perform data enhancement on drone aerial image datasets;

[0014] Network model building module, which is used to construct the TFA-YOLO11 network model;

[0015] A neural network training module is used to train the TFA-YOLO11 network model using the data augmented data set to obtain a trained TFA-YOLO11 network model;

[0016] The drone aerial image detection module is used to detect the drone aerial images to be detected using the trained TFA-YOLO11 network model to obtain the positions and categories of all targets on the images to be detected.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1) This paper proposes a Transformer-based feature aggregation module, which uses a multi-head attention mechanism to realize information interaction between features aggregated from different layers of the network, effectively analyzes the correlation between feature maps of different scales, fully integrates the spatial and semantic information of feature maps of different scales, and enhances the representation ability of the model. And for the first time, a layer feature aggregation module (Layer Feature Aggregation, LFA) is proposed to aggregate the features of each layer to obtain the aggregated feature vector for cross-layer information interaction.

[0019] 2) The present invention proposes a very small target detection module, expands the neck network, and adds a very small target detection head, which can enhance the very small target perception ability of the network model;

[0020] 3) The present invention proposes a backbone network and a neck network for downsampling based on SPD convolution, and uses SPD convolution with smaller parameters and computational complexity to achieve feature map downsampling without information loss by converting the spatial dimension to the channel dimension, thereby achieving lightweight model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A structural diagram of a network model for small target detection in lightweight UAV aerial images based on TFA-YOLO11 provided by the present invention;

[0022] Figure 2 This is a schematic diagram of the structure of a small target detection device for drone aerial images of the present invention;

[0023] Figure 3 This is a structural diagram of the TFA module of the present invention;

[0024] Figure 4 for Figure 1 Schematic diagram of SPD convolution structure;

[0025] Figure 5 The VisDrone2021 drone aerial image dataset used in this embodiment;

[0026] Figure 6 This is a comparison chart of the detection effects of the VisDrone2021 drone aerial image dataset using an embodiment of the present invention and the YOLO11s model. DETAILED DESCRIPTION

[0027] The present invention is further described and illustrated below in conjunction with specific embodiments. The embodiments are merely exemplary of the present disclosure and do not define the scope of limitation. The technical features of each embodiment of the present invention may be combined accordingly without conflicting with each other.

[0028] Example 1

[0029] like Figure 1 As shown, it is a structural diagram of the network model for small target detection in lightweight UAV aerial images based on TFA-YOLO11 (Transformer-based Feature Aggregation-YOLO11) of the present invention, which mainly includes:

[0030] Step 1: Get the dataset and perform data augmentation.

[0031] First, we obtain a drone aerial image dataset. For the drone aerial images in the dataset, we first randomly transform the hue, saturation, and brightness to increase color disturbance. Then, we randomly translate, scale, and flip the images horizontally with a certain probability, which helps the model detect partially visible targets and multi-scale targets and increase the diversity of the dataset. On this basis, we use the mosaic data augmentation method to further enhance the diversity of the dataset and prevent the model from overfitting.

[0032] The mosaic data enhancement method specifically includes:

[0033] (1) Randomly select four images from the dataset for mosaic data augmentation;

[0034] (2) Randomly generate the coordinates of the intersection points of the four images;

[0035] (3) Each image is randomly scaled to maintain its aspect ratio, and then stitched together to form a new image, with the portion that exceeds the size boundary cropped off;

[0036] (4) The annotations of each image are also processed accordingly to obtain new training samples with annotations.

[0037] In this embodiment, the maximum ratios of random transformation of hue, saturation and brightness are 0.015, 0.7 and 0.4, respectively; the maximum ratios of random translation, scaling and horizontal flipping are 0.1, 0.5 and 0.5, respectively.

[0038] Step 2: Build the TFA-YOLO11 network model.

[0039] The TFA-YOLO11 network model uses the YOLO11 network as the basic network. The traditional YOLO11 network is divided into three parts: the backbone network, the neck network and the detection head. Among them, the backbone network extracts multi-level features from the input image through a series of convolution, pooling and other operations to obtain multi-layer feature maps from large to small. Subsequently, the neck network adopts the Feature Pyramid Network (FPN) structure to transfer feature information from bottom to top and top to bottom for feature maps of different scales, transfer the semantic information of the deep feature map to the shallow feature map, and transfer the spatial position information of the shallow feature map to the deep feature map, thereby improving the representation ability of the model. The detection head predicts the target position and category of the feature map processed by the neck network.

[0040] The TFA-YOLO11 network model of the present invention is an improvement based on the traditional YOLO11 network. Figure 1 As shown, it includes a backbone network and a neck network for downsampling based on SPD convolution, a feature aggregation module based on Transformer, a very small target detection module and a standard detection head. The present invention adopts SPD convolution to replace the standard convolution in the backbone network and neck network of YOLO11. The backbone network and neck network for downsampling based on SPD convolution realize downsampling with less parameters and calculation amount without information loss by converting the spatial dimension to the channel dimension, thereby realizing the lightweight of the model. The present invention adds a feature aggregation module based on Transformer between the backbone network and the neck network of YOLO11. The feature aggregation module based on Transformer is used to perform feature aggregation on adjacent three-layer feature maps extracted by the backbone network, effectively analyze the correlation between feature maps of different scales, fully integrate the spatial and semantic information of feature maps of different scales, and enhance the representation ability of the model. The present invention also adds a very small target detection module after one of the feature aggregation modules. The very small target detection module adds a feature map with higher resolution to the feature aggregation and feature fusion of the neck network, thereby increasing the model's perception ability of very small targets and effectively improving the model's small target detection performance. A very small target detection head (Tiny Head) is introduced into the very small target detection module, which performs prediction based on a feature map with higher spatial resolution. Compared with the standard detection head, the very small target detection head has a stronger perception ability for small targets, thereby effectively improving the detection accuracy of the network model for small targets.

[0041] Furthermore, the backbone network of YOLO11 includes multiple standard convolutional layers, multiple C3k2 layers, an SPPF layer and a C2PSA layer. The structural layers are connected in series and numbered from layer 0 to layer 10 in sequence according to the process, where layers 0, 1, 3, 5, and 7 are standard convolutional layers, layers 2, 4, 6, and 8 are C3k2 layers, and layer 9 is an SPPF layer. The backbone network of the present invention replaces layers 1, 3, 5, and 7 with SPD convolutional layers on this basis. The SPD convolutional layer can downsample the feature map to reduce the resolution of the feature map, and at the same time transfer the spatial information reduced by downsampling to the channel dimension to increase the depth of the feature map. The output of the 0th convolutional layer, each C3k2 layer, and the C2PSA layer is used as the output of the backbone network, that is, a multi-layer feature map, specifically, a 6-layer feature map P={P1,...,P i ,...,P6}.

[0042] Further, such as Figure 3 The specific working process of the Transformer-based feature aggregation module is as follows:

[0043] According to the order of feature maps in the multi-layer feature map output by the backbone network, the two feature maps before and after the feature map are its adjacent feature maps. The multi-layer feature map is input into the feature aggregation module based on Transformer, and the feature aggregation module performs the same feature aggregation operation on the feature maps output by the four C3k2 layers of the backbone network and their adjacent feature maps. Specifically, for each feature map output by the C3k2 layer, the feature aggregation module based on Transformer first aligns the spatial resolution of the feature map and its adjacent feature maps, and makes the resolution of the three feature maps the same by upsampling the smaller feature map and downsampling the larger feature map, and keeps the number of channels of the three feature maps consistent. i Taking the feature aggregation module as an example, the scale alignment process can be expressed as:

[0044] X i-1 =DownSample(P i-1 )

[0045] X i+1 =UpSample(P i+1 )

[0046] Among them, X i-1 , X i+1 They represent the feature maps after scale alignment, DownSample and UpSample represent the downsampling and upsampling functions respectively.

[0047] After the spatial resolution adjustment is completed, the feature maps after each three-scale alignment are sent to the layer feature aggregation module for feature aggregation of the feature map of this layer. The layer feature aggregation module first extracts significant features from the feature maps after each scale alignment through global maximum pooling and global average pooling, and obtains two feature vectors for each feature map; then the two feature vectors are processed by a shared multi-layer perceptron (Shared MLP) to interact with information and obtain higher-level features. After adding the two feature vectors output by the shared multi-layer perceptron, nonlinear characteristics are given through the SiLU activation function to obtain the aggregated feature vector of the feature map of this layer. The process can be expressed as:

[0048] M i =SiLU(MLP(X i,max )+MLP(X i,avg ))

[0049] Among them, X i,max , X i,avg Represents X i The global maximum pooling and global average pooling features, M i Represents X i The operation on adjacent feature maps is consistent with the above formula.

[0050] After the three adjacent layers of feature maps obtain the aggregated feature vector in the same way, the three aggregated feature vectors are sent to the Transformer encoder to realize cross-layer information interaction. The Transformer encoder weights the effective information according to the correlation between features in different layers, thereby giving the aggregated features of each level the ability to perceive the semantic information contained in the aggregated features of the adjacent layers, solving the deficiency that only the features of the previous level can be perceived in the bottom-up path of the feature pyramid. The three aggregated feature vectors after cross-layer information interaction are used as channel attention weights and applied to the feature maps after scale alignment of the corresponding levels, and the feature maps after cross-layer information interaction are obtained by combining residual connections. Finally, the group of three feature maps after cross-layer information interaction are connected according to the channel dimension, and the channel dimension feature fusion is realized through 1×1 convolution and the dimension is reduced to the original number of channels, so as to obtain the high-quality aggregated feature map finally output by the Transformer-based feature aggregation module. The above operation can be expressed as:

[0051] [M i ' -1 ,M i ′,M i ' +1 ]=Transformer([M i-1 ,M i ,M i+1 ])

[0052] X i′=M i ′⊙X i +X i

[0053] X fuse =Conv(Concat(X i ' -1 ,X i ′,X i ' +1 ))

[0054] Among them, M i ' -1 、M i ′、M i ' +1 They represent the aggregated feature vectors after cross-layer interaction of feature maps at all levels; ⊙ is the Hadamard product, which realizes channel-by-channel weighting; X i ' -1 , X i ′、X i ' +1 They represent the feature maps after information interaction; X fuse A high-quality aggregated feature map representing the output of the model.

[0055] Furthermore, the neck network includes two upsampling modules and two downsampling modules, forming a feature pyramid structure, the feature maps in the two upsampling modules are transmitted from bottom to top, and the feature maps in the two downsampling modules are transmitted from top to bottom. The upsampling module includes an upsampling layer, a concatenation layer, and a C3k2 layer. The downsampling module includes an SPD convolution layer, a concatenation layer, and a C3k2 layer.

[0056] The Transformer-based feature aggregation module will output four aggregated feature maps with different spatial resolutions. The aggregated feature map with the lowest resolution first enters the first upsampling module. After passing through the first upsampling layer, it is spliced ​​with the aggregated feature map with the second lowest resolution in the first splicing layer in the channel dimension. Then it passes through the first C3k2 layer and the second upsampling layer in the second upsampling module. The feature map obtained is spliced ​​with the aggregated feature map with the second highest resolution in the channel dimension in the second splicing layer. The obtained feature map is the output of the bottom-to-top path.

[0057] In the top-to-bottom path, the feature map that enters the top of the path for information transmission first passes through the SPD convolution layer of the second downsampling module, and is spliced ​​with the feature map output by the first C3k2 layer at the splicing layer, and then passes through the C3k2 layer of the second downsampling module, the SPD convolution layer of the first downsampling module, and is spliced ​​with the lowest resolution aggregate feature map at the splicing layer, and then passes through the C3k2 layer of the first downsampling module. The feature map obtained is the output of the top-to-bottom path. The outputs of the C3k2 layers of the two downsampling modules are respectively input into a standard detection head. The standard detection head is the original detection head of the YOLO11 network.

[0058] The very small target detection module includes a third upsampling module, a third downsampling module and a very small target detection head. The structure is the same as the upsampling module and downsampling module in the neck network. The upsampling module and downsampling module of the very small target detection module are added to the feature pyramid of the neck network to form a three-level feature pyramid. At the same time, the feature map with the highest spatial resolution in the process of feature pyramid information transmission is passed through the very small target detection head for detection tasks. The third upsampling module receives the aggregated feature map with the highest spatial resolution as input 1, and receives the feature map transmitted from the bottom-to-top path in the feature pyramid of the neck network as input 2. After upsampling input 2, it is spliced ​​with input 1, and after feature fusion through the C3k2 layer of YOLO11, the P2 feature map with the highest resolution is output. P2 feature Figure 1 On the one hand, it enters the third downsampling module, and on the other hand, it is input into the Tiny Head. In the third downsampling module, the P2 feature map is spliced ​​with the output of the bottom-to-top path of the neck network after passing through the SPD convolution layer, and then enters the top-to-bottom path of the neck network after being processed by the C3k2 layer for information transmission. The output after being processed by the C3k2 layer of the third downsampling module will also be input into the third standard detection head for detection tasks.

[0059] The very small target detection head is a decoupled head, which consists of two branches. The first branch is responsible for the target position regression task, and further features are extracted by two standard convolution blocks, and then the number of channels is adjusted by convolution with a convolution kernel size of 1, which is used to predict the target's bounding box position and calculate the positioning loss; the second branch is responsible for the target category prediction task, which is simpler than the position regression task, so the second branch consists of two depth-separable convolutions with smaller parameters and calculations, and the number of channels is adjusted by convolution with a convolution kernel size of 1, which is used to predict the target category and calculate the category loss. The other detection heads have the same structure as the very small target detection head.

[0060] Furthermore, the structure of the SPD convolutional layer is as follows: Figure 4 As shown, the specific working process is as follows:

[0061] Assume an input feature map X of size H×W×C, where H and W represent the height and width of the feature map, and C represents the number of channels. Given a downsampling factor s, SPD convolution divides X into s 2 subgraphs, and connect all subgraphs along the channel dimension, and then use a 1×1 convolution to adjust the number of channels to get the final output. It can be simplified to the following equation:

[0062] X i,j =X[i:H:s,j:W:s],

[0063] Where X i,j Represents the subgraph with starting coordinates i,j, i,j∈{0,1,···,s-1}.

[0064] X′=Conv 1×1 [Concat(X i∈[0,s-1],j∈[0,s-1] )],

[0065] Among them, X′ represents the output feature map obtained after applying SPD convolution to the input feature map X.

[0066] It can be understood that SPD convolution first transfers spatial information to depth information, halving the spatial resolution of the feature map without losing information, and then maintains the spatial resolution and reduces the number of channels through a 1×1 convolution, thereby achieving feature extraction and reducing the number of parameters and calculations. Using SPD convolution to replace standard convolution can retain key detail information that may be lost in standard convolution and improve the perception of small targets.

[0067] Step 3: Use the data augmented dataset to train the TFA-YOLO11 network model to obtain a trained network model.

[0068] Specifically, in this embodiment, the data augmented data set is divided into a training set and a validation set in a ratio of 8:2; 8 pictures are input for each training; the input picture size is uniformly scaled to 800×800 pixels; this embodiment iterates the training for 500 rounds; CIoU loss and distributed focus loss (DFL) are used as positioning losses, binary cross entropy loss BCE is used as classification loss, and the network weights are updated using stochastic gradient descent, with an initial learning rate of 0.01 and a momentum of 0.937; after the training, the trained network weights are saved.

[0069] Step 4: Use the trained network model to detect drone aerial images.

[0070] Specifically, based on the TFA-YOLO11 network model and the trained network weights, the network performs target detection on the drone aerial images to be detected. The network performs a forward inference and the detection heads (including three standard detection heads and a very small target detection head) output the bounding box coordinates and categories of the detected targets.

[0071] Example 2

[0072] In this embodiment, a lightweight UAV aerial image small target detection system based on TFA-YOLO11 is also provided, which is used to implement the above embodiment. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the system described in the following embodiments is preferably implemented in software, it is also possible to implement hardware, or a combination of software and hardware.

[0073] Figure 2 FIG. 1 is a block diagram of a lightweight UAV aerial image small target detection system based on TFA-YOLO11 according to an embodiment. Figure 2 As shown, the system includes:

[0074] The data enhancement module is used to enhance the data of drone aerial image datasets, increase data diversity, and improve network robustness;

[0075] Network model building module, which is used to construct the TFA-YOLO11 network model;

[0076] A neural network training module is used to train the TFA-YOLO11 network model using the data augmented data set to obtain a trained TFA-YOLO11 network model;

[0077] The drone aerial image detection module is used to detect the drone aerial images to be detected using the trained TFA-YOLO11 network model to obtain the positions and categories of all targets on the images to be detected.

[0078] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0079] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic. For example, the data enhancement module can be divided into a logical function. There may be other division methods in actual implementation. For example, multiple modules can be combined or integrated into another unit. Another point is that the connection between the modules shown or discussed can be a communication connection through some interfaces, which can be electrical or other forms. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present application. Ordinary technicians in this field can understand and implement it without creative work. The following takes the target detection in real drone aerial images as an example to illustrate the specific implementation method to reflect the technical effect of the present invention. The specific steps in the embodiment will not be repeated.

[0080] Example 3

[0081] Next, the VisDrone2021 drone aerial image dataset is used as the research object to verify the target detection method of the present invention. In order to comprehensively compare the target detection performance, the target detection results are displayed from the perspective of visualization and quantification. The mean average precision (mAP), the mean average precision (AP) of small target detection, and the average precision (mAP) of small target detection are used to compare the target detection performance. S The proposed object detection method is evaluated. The mAP here refers to taking 10 IoU thresholds from 0.5 to 0.95 with a step size of 0.05, and then calculating the average precision. This paper uses mAP 50-95 The calculation process of mAP is as follows:

[0082]

[0083]

[0084]

[0085]

[0086] Where P is precision, R is recall; TP represents true positive, FP represents false positive, and FN represents false negative; J(P,R) k represents the average precision function, and n represents the total number of categories.

[0087] According to the guidelines of the COCO dataset, targets with a pixel size less than 32×32 pixels are defined as small targets, and AP S Represents the average precision of small targets, which is used to evaluate the detection performance of small targets. S The calculation method is the same as mAP, but different from mAP is that AP is calculated SOnly small objects with a size less than 32×32 pixels are considered.

[0088] In addition, model parameters Params (Parameters) and computing amount GFLOPs (Giga-Floating-Point Operations Per seconds) are used as model lightweight evaluation indicators.

[0089] The VisDrone2021 dataset consists of drone aerial images taken at different locations and altitudes, with 10 target categories (pedestrians, cars, bicycles, etc.). The VisDrone2021 dataset has a total of 6471 training set images and 548 validation set images. Figure 5 These are some typical images in the VisDrone2021 drone aerial image dataset. Figure 6 : This is a comparison chart of target detection results obtained using the embodiment of the present invention and the YOLO11s algorithm, where (a) is the detection result of the YOLO11s algorithm and (b) is the detection result of the embodiment of the present invention. Figure 6 It can be seen that the embodiment of the present invention has stronger extremely small target perception capability and better extremely small target detection performance.

[0090] Table 1 Evaluation indicators of target detection results of VisDrone2021 drone aerial image dataset

[0091]

[0092] The comparative method Faster R-CNN comes from Ren S, He K, Girshick R, et al. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks[J]. IEEETransactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137–1149.

[0093] The comparison method RT-DETR-L comes from Zhao Y, Lv WY, Xu SL, et al. DETRs Beat YOLOs on Real-time Object Detection [C]. 2024 IEEE / CVF Conference on ComputerVision andPattern Recognition (CVPR), Seattle, WA, USA, 2024, pp.16965-16974.

[0094] The comparison method YOLOv10s comes from Wang A, Chen H, Liu L, et al. YOLOv10: Real-Time End-to-End Object Detection: arXiv: 2405.14458 [Z / OL]. arXiv, 2024.

[0095] Comparison methods YOLO11s and YOLO11m are from Jocher G, Qiu J. Ultralytics YOLO11[Z]2024.https: / / github.com / ultralytics / ultralytics.

[0096] The target detection visualization results of the VisDrone2021 dataset are as follows Figure 6 The quantification results are shown in Table 1. In the quantification results, the mAP 50-95 The present invention achieved the best performance in AP S The present invention has achieved suboptimal performance, only slightly lower than YOLO11m, but the number of parameters and the amount of calculation are much smaller than YOLO11m. It can be seen that the present invention has obtained stronger extremely small target perception ability based on the extremely small target detection module, and improved the small target detection accuracy; at the same time, the feature aggregation module based on Transformer improves the representation ability of the model, bringing about an improvement in the overall detection accuracy; in addition, SPD convolution replaces standard convolution to achieve downsampling, reduces the number of parameters and the amount of calculation of the model, and realizes the lightweight of the model.

[0097] The above-mentioned embodiments only express several implementation modes of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. For those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A lightweight UAV aerial image small target detection method based on TFA-YOLO11, characterized in that: The following steps are involved: Step 1): Obtain a drone aerial image dataset and perform data enhancement; Step 2): Build the TFA-YOLO11 network model; The TFA-YOLO11 network model includes a backbone network, a neck network, a Transformer-based feature aggregation module, a very small target detection module and a detection head module; the backbone network is used to perform multi-level downsampling and feature extraction on the input image to obtain a multi-layer feature map; the feature aggregation module is used to perform feature aggregation of adjacent layer feature maps on the multi-layer feature map to obtain high-quality aggregated feature maps of multiple scales; the neck network and the very small target detection module together constitute a feature pyramid structure, and feature fusion is performed on high-quality aggregated feature maps of different scales through the feature pyramid structure, so that the final feature map obtained has feature information of aggregated feature maps of other scales; in addition to the part constituting the feature pyramid structure, the very small target detection module also includes a very small target detection head, which is used to detect small targets on the final feature map with the highest spatial resolution; Except for the final feature map with the highest spatial resolution, other final feature maps are used for object detection through the detection head module; Step 3): Use the data augmented data set to train the TFA-YOLO11 network model to obtain a trained TFA-YOLO11 network model; Step 4): Use the trained TFA-YOLO11 network model to detect the drone aerial images to obtain the positions and categories of all targets on the images to be detected.

2. The method for detecting small targets in lightweight UAV aerial images based on TFA-YOLO11 according to claim 1 is characterized in that: The data enhancement described in step 1) specifically includes: The hue, saturation and brightness of the images in the drone aerial image dataset are randomly transformed within a certain range of proportions; the images in the dataset are randomly translated, scaled and horizontally flipped; and the mosaic data enhancement method is used to combine the four images in the dataset into a new image.

3. The method for detecting small targets in lightweight UAV aerial images based on TFA-YOLO11 according to claim 1, characterized in that: The backbone network uses SPD convolution for downsampling.

4. The method for detecting small targets in lightweight UAV aerial images based on TFA-YOLO11 according to claim 1, characterized in that: The step of performing feature aggregation of adjacent layer feature maps on the multi-layer feature maps specifically includes: For every three adjacent feature maps P in the multi-layer feature map i-1 , P i and P i+1 Follow steps a)-d): a) Through upsampling and downsampling, the feature map P with smaller resolution among the three adjacent feature maps is i+1 and a feature map P with larger resolution i-1 The scale is towards the middle feature map P i Align to obtain three feature maps X after scale alignment i-1 , X i and X i+1 ; b) Three feature maps X after scale alignment i-1 , X i and X i+1 The features are aggregated through the layer feature aggregation module to obtain their respective aggregated feature vectors M i-1 、M i 、M i+1 ; c) The three aggregated feature vectors are passed through the Transformer encoder to realize cross-layer information interaction, and the aggregated feature vector M after cross-layer interaction is obtained i ' -1 、M i ′、M i ' +1 ; The aggregated feature vector after cross-layer information interaction is used as the channel attention weight, which is applied to the feature map after the level scale is aligned, and the feature map X after cross-layer information interaction is obtained by combining the residual connection i ' -1 , X i ′、X i ' +1 , the expression is as follows: X i ′ -1 =M i ′ -1 ⊙X i-1 +X i-1 X i ′=M i ′⊙X i +X i X i ′ +1 =M i ′ +1 ⊙X i+1 +X i+1 d) The feature maps after the three cross-layer information interactions are connected according to the channel dimension, and the channel dimension feature fusion is realized through 1×1 convolution and the dimension is reduced to the original number of channels, so as to obtain the high-quality aggregated feature map X finally output by the Transformer-based feature aggregation module. fuse ; After performing steps a)-d) on all feature maps in the multi-layer feature map, multiple high-quality aggregated feature maps with different resolutions are obtained.

5. The method for detecting small targets in lightweight UAV aerial images based on TFA-YOLO11 according to claim 4 is characterized in that: The specific method for the layer feature aggregation module to perform feature aggregation is: extracting significant features from the feature map sent to the layer feature aggregation module through global maximum pooling and global average pooling to obtain global maximum pooling features and global average pooling features; then using a shared multi-layer perceptron to process the two global maximum pooling features and global average pooling features respectively, adding the two feature vectors output by the shared multi-layer perceptron and giving them nonlinear characteristics through the SiLU activation function to obtain the aggregated feature vector of the feature map of this layer.

6. The method for detecting small targets in lightweight UAV aerial images based on TFA-YOLO11 according to claim 4 is characterized in that: The neck network includes a first upsampling module, a second upsampling module, a first downsampling module and a second downsampling module. The very small target detection module includes a third upsampling module, a third downsampling module and a very small target detection head. The three upsampling modules and the three downsampling modules together constitute a feature pyramid structure. The three upsampling modules constitute a bottom-to-top path of the feature pyramid structure, which is used to transfer the semantic information of the deep feature map to the shallow feature map. The three downsampling modules constitute a top-to-bottom path of the feature pyramid structure, which is used to transfer the spatial position information of the shallow feature map to the deep feature map. The upsampling modules all include an upsampling layer, a splicing layer and a C3k2C3k2 layer, which are used to upsample, splice and fuse the high-quality aggregated feature maps output by the feature aggregation module; the downsampling modules all include an SPD convolution layer, a splicing layer and a C3k2C3k2 layer, which are used to downsample, splice and fuse the feature maps output by the upsampling modules.

7. The method for detecting small targets in lightweight UAV aerial images based on TFA-YOLO11 according to claim 6 is characterized in that: The feature aggregation module outputs four high-quality aggregated feature maps in total. The feature fusion of high-quality aggregated feature maps of different scales through the feature pyramid structure is specifically as follows: The high-quality aggregated feature map with the lowest resolution among the multiple high-quality aggregated feature maps output by the feature aggregation module first enters the first upsampling module, is upsampled in the first upsampling module, and is spliced ​​with the high-quality aggregated feature map with the second lowest resolution and then fused with features through the C3k2C3k2 layer; The feature map output by the first upsampling module enters the second upsampling module, where it is upsampled and concatenated with the high-quality aggregate feature map with the second highest resolution, and then features are fused through the C3k2C3k2 layer; The feature map output by the second upsampling module enters the third upsampling module, where it is upsampled and concatenated with the highest-resolution high-quality aggregate feature map, and then features are fused through the C3k2C3k2 layer; The final feature map output by the third upsampling module is fed into the very small target detection head on the one hand, and is fed into the third downsampling module on the other hand; in the third downsampling module, the feature map output by the third upsampling module is downsampled, and is concatenated with the feature map output by the second upsampling module, and features are fused through the C3k2C3k2 layer; The feature map output by the third downsampling module enters the second downsampling module, is downsampled in the second downsampling module, and is concatenated with the feature map output by the first upsampling module and fused through the C3k2C3k2 layer; The feature map output by the second downsampling module enters the first downsampling module, is downsampled in the first downsampling module, and is concatenated with the high-quality aggregated feature map with the lowest resolution and then fused through the C3k2 layer; the final feature maps output by the three downsampling modules are all sent to the detection head module.

8. The method for detecting small targets in lightweight UAV aerial images based on TFA-YOLO11 according to claim 1, characterized in that: The detection head module includes three detection heads. The detection heads in the detection head module and the extremely small target detection head are both decoupled heads, which are used to perform target detection on the input feature map.

9. The method for detecting small targets in lightweight UAV aerial images based on TFA-YOLO11 according to claim 1, characterized in that: In the step 3), the training of the TFA-YOLO11 network model uses CIoU loss and distributed focal loss as positioning losses, uses binary cross entropy loss as classification loss, and uses stochastic gradient descent to update the weights of the network model.

10. A lightweight UAV aerial image small target detection system based on TFA-YOLO11 that implements the method described in claim 1, characterized in that: include: Data enhancement module, which is used to enhance the data of drone aerial image dataset; Network model building module, which is used to construct the TFA-YOLO11 network model; A neural network training module is used to train the TFA-YOLO11 network model using the data augmented data set to obtain a trained TFA-YOLO11 network model; The drone aerial image detection module is used to detect the drone aerial images to be detected using the trained TFA-YOLO11 network model to obtain the positions and categories of all targets on the images to be detected.

Images