An aerial small target detection method based on multi-feature parallel hybrid

Abstract

This invention discloses an aerial small target detection method based on multi-feature parallel hybridization. The method includes: introducing max pooling into self-calibrating convolution to enhance the recognition of local features and contextual information; leveraging quantum principles, in the field of visual multilayer perceptrons, treating the relationship between segmented tokens as an aggregation of wave functions, using max pooling to generate amplitude and phase, achieving better aggregation results, and strengthening local feature extraction while extracting global features; utilizing max pooling, depthwise separable convolution, maximizing self-calibrating convolution, and wave-based visual multilayer perceptron in parallel to obtain a multi-branch module, realizing the gradual outward expansion extraction of local features, obtaining more contextual features of small targets, and fusing global and local features; using the selected module in combination with the YOLOX network, the method is tested on an aerial small target dataset. Our model achieves high-precision small target detection results even with low input resolution.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to an aerial small target detection method based on multi-feature parallel hybridization. Background Technology

[0002] Object detection is fundamental to various industrial computer vision applications. While object detection algorithms have made significant progress, most focus on targets in general scenarios, particularly those with relatively large sizes. In the field of aerial small object detection, there is still considerable room for improvement. Firstly, datasets for small object detection are relatively limited. Secondly, small objects contain fewer pixels, resulting in insufficient feature representation. Thirdly, due to their small size, occlusion or other background interference can severely impact detection. Finally, drones or high-definition cameras are often used to capture images of small objects, resulting in high resolution images containing limited information about the small objects. Downsampling and other operations can further interfere with this information, affecting detection performance. Therefore, small object detection demands relatively high computing power and is quite challenging. Improving the performance of small object detection has broader implications in many practical applications, such as autonomous vehicles, drone technology, national security, forestry, and transportation.

[0003] In the field of aerial small target detection, most methods have improved convolutional neural networks (CNNs), including multi-layer feature fusion, the introduction of attention mechanisms, and image preprocessing to increase the proportion of small targets. CNNs still dominate the field, and there is room for further improvement in detection accuracy. While CNNs can detect small targets, they have limitations in modeling global contextual information. The emergence of the Vision Transformer structure can extract global features, but its attention mechanism is relatively complex. Recently, MLP structures have replaced the attention mechanism in Transformers and been applied to the vision domain, achieving global feature extraction and comparable performance to Transformers. However, such structures are computationally expensive, and their processing of local features needs improvement. Currently, research on applying Transformers and MLPs to small target detection is relatively limited. Combining them and integrating their advantages is a key research direction. Summary of the Invention

[0004] To address the aforementioned issues, this invention discloses a method for small target detection in aerial photography based on multi-feature parallel hybridization, specifically for small target detection scenarios captured by drones. It designs a maximized self-correcting convolution to enhance the extraction of local and contextual features. Based on quantum principles, it improves the structure of the visual multilayer perceptron, optimizing the focus on local features while maintaining its global feature extraction capabilities. Finally, a multi-branch parallel structure is designed to achieve extended extraction and feature fusion of target features across different ranges. Combining this structure with YOLOX achieves high-precision aerial small target detection results.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A method for detecting small targets in aerial photography based on parallel fusion of multiple features includes the following steps:

[0007] Step 1: Download the VisDrone public dataset for small drone targets, which includes training and testing sets, and adjust the image resolution of the dataset;

[0008] Step 2: Design a self-correcting convolution to enhance the extraction of local and contextual features of small targets;

[0009] Step 3: Using quantum thinking, construct a wave-based visual multilayer perceptron structure (MWaveMLP), treat tokens as waves, and use max pooling to generate amplitude and phase. Treat the aggregation between tokens as the superposition of wave functions to achieve better aggregation effect and enhance local feature extraction while extracting global features.

[0010] Step 4: Combine max pooling, depthwise separable convolution, maximizing self-correcting convolution and wave-based visual multilayer perceptron in parallel to obtain a multi-branch module, realize the gradual outward extraction of local features, obtain more contextual features of small targets, and realize global and local feature extraction and fusion.

[0011] Step 5: Using the selected modules and the YOLOX network, establish the entire aerial small target detection model based on multi-feature parallel hybrid convolutional neural network and visual multilayer perceptron;

[0012] Step 6: Load the training set and pre-trained weights, train the network, and save the optimal weight model based on the test accuracy;

[0013] Step 7: Use the network trained in Step 6 to detect small targets in the aerial photography dataset.

[0014] Furthermore, in step 1, the VisDrone public dataset for small drone targets is downloaded, and the image resolution of the training and test sets is adjusted to 640×640.

[0015] Furthermore, the maximizing self-correcting convolutional structure designed in step 2, through convolutional feature transformation in two different scale spaces, extracts local salient features using max pooling downsampling, obtaining features with a smaller resolution and a larger receptive field. This feature is then combined with the feature transformation result obtained from the original feature space to achieve a corrective effect, enriching the contextual information at each spatial location. The specific calculation method is as follows:

[0016] X1'=Act(X+Up(F2(MaxPool(X))))

[0017] Y = F3(F1(X)·X1')

[0018] In this algorithm, MaxPool() represents max pooling, F represents convolution, UP() represents upsampling, and Act() represents the activation function. First, max pooling is used for downsampling to extract local features. Then, a 3×3 convolution feature transformation is applied, followed by an activation function to add non-linearity. Finally, bilinear interpolation is used for upsampling to restore the original feature space size. To preserve the original input features, identity mapping is used for superposition, and then a sigmoid activation function is applied to add non-linearity. After feature transformation in the original feature space using convolution, features obtained from the small-scale latent space are used for self-calibration. Then, convolution is used for feature fusion to obtain the final target features containing rich contextual information. This approach enables communication between different convolution kernels and different scale spaces, expanding the receptive field and facilitating the detection of small targets.

[0019] Furthermore, the wave-based visual multilayer perceptron structure in step 3 specifically includes the following process:

[0020] Step 301: For the VisionMLP structure, the image is first segmented into multiple tokens. Each token is treated as a wave function containing amplitude and phase, as shown below:

[0021] t j =|t j |⊙cosθ j +i|t j |⊙sinθ j j = 1, 2, ..., n

[0022] The result of token aggregation is affected by the amplitude and phase difference between two tokens. The relationship between tokens and fixed weights is dynamically adjusted by the aggregation of wave functions, thereby achieving a better token aggregation effect.

[0023] In the token aggregation module (MPATM), max pooling is used to generate amplitude and phase, which are then aggregated using TokenFC. ChannelFC is then used to enhance feature representation, yielding the module's final output. The formulas for TokenFC and ChannelFC are shown below:

[0024] ChannelFC(t j W c ) = W c t j j = 1, 2, ..., n

[0025]

[0026] Among them W c The weights, W, have learnable parameters. t It represents the token-mixing weight, where ⊙ indicates element-wise multiplication, and the index j represents the j-th output token.

[0027] Step 302: The wave-based visual multilayer perceptron (MWaveMLP) consists of Token-MLP and Channel-MLP. Token-MLP uses two parallel MPATM modules and additional ChannelFC connections to preserve the original information, then performs a weighted sum of their outputs. Channel-MLP is implemented by stacking two ChannelFCs and a non-linear activation function. This achieves both global feature extraction and enhanced local feature extraction.

[0028] Furthermore, step 4 incorporates a multi-branch parallel feature extraction module (MBblock). For local components, one branch uses max pooling to calculate local maxima; another branch uses depthwise separable convolution to extract local features while reducing the number of parameters; and a third branch uses our designed maximizing self-correcting convolution to enhance local feature extraction while simultaneously strengthening the extraction of contextual information for small targets, thus obtaining more information to assist in small target feature detection. For global components, MWaveMLP is used to enhance local feature extraction while simultaneously extracting global features. To reduce computational burden, the input features are first dimensionality reduced, then feature extraction is performed through the four branches, and finally, the feature components are concatenated together. Convolution and activation functions are then used to fuse the features and restore them to the input feature dimension. By gradually expanding outward from local features and fusing them, the advantages of convolution and Vision MLP structures are fully utilized to achieve optimal extraction of small target features.

[0029] Furthermore, in step 5, our designed multi-branch parallel feature extraction module (MBblock) is combined with YOLOX to build our small object detection network. Since shallow layers contain rich positional information, while deeper layers have more semantic information to aid classification, and small object information accounts for a small proportion of the feature map with limited information, downsampling will gradually weaken the information of small objects. Therefore, the extraction of key features mainly focuses on shallow features. In the first two layers of the backbone network, we enhance the feature extraction of small objects by stacking MBblocks. In the last two layers, the original module CSPblock is used to replace the YOLOX backbone network to construct the small object detection model.

[0030] Furthermore, in step 6, the model is first trained using the COCO dataset to obtain pre-trained weights, and then the pre-trained weights are loaded onto the VisDrone dataset for training, for a total of 100 training cycles.

[0031] Furthermore, in step 7, after obtaining the model trained on the VisDrone dataset, detection is performed on the test set, achieving high-precision aerial small target detection results.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. This invention employs max pooling to improve self-correcting convolution and enhance the extraction of local features and contextual information. Based on quantum thinking, max pooling is used to generate amplitude and phase to construct wave functions to represent tokens, realizing dynamic aggregation between tokens and capturing more feature information. This forms a wave-based multilayer perceptron (MWaveMLP) structure, which enhances local feature extraction on the basis of global feature extraction.

[0034] 2. A multi-branch structure based on the parallel combination of MLP and CNN is proposed, which includes max pooling, depthwise separable convolution, MSCConv, and MWaveMLP. It gradually expands from local features to contextual features and then to global features to extract fused features, obtain more features to assist in the detection of small targets, and achieve higher detection accuracy.

[0035] 3. The proposed multi-branch module (MBblock) is used in conjunction with YOLOX to build the backbone network. Experiments are conducted on the VisDrone dataset to verify that when the input image resolution is low, the method proposed in this invention can achieve high accuracy in small target detection. Attached Figure Description

[0036] Figure 1 A flowchart of an aerial small target detection method based on multi-feature parallel hybridization provided by the present invention;

[0037] Figure 2 This is a schematic diagram of the maximized self-correcting convolution (MSCConv) structure of the present invention;

[0038] Figure 3 This is a schematic diagram of the wave-based multilayer perceptron (MWaveMLP) structure of the present invention;

[0039] Figure 4 This is a schematic diagram of the multi-branch structure block (MBblock) based on convolutional neural networks and MLP of the present invention. Detailed Implementation

[0040] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0041] This invention provides an aerial small target detection method based on multi-feature parallel hybridization. For local features, a maximized self-correcting convolution is designed, and max pooling is introduced to enhance the extraction of local features and contextual information. For global features, in the Vision MLP structure, the token is treated as a wave, and max pooling is used to generate amplitude and phase, achieving global feature extraction while strengthening local feature extraction. This is combined in parallel with max pooling and depthwise separable convolution to achieve progressive outward feature extraction from the local to the surrounding area, enhancing the feature acquisition capability for small targets. Using the designed multi-branch structure block and YOLOX network, the method is tested on the VisDrone dataset. Even with low input image resolution, our model achieves high-precision small target detection results.

[0042] The following is in conjunction with the appendix Figure 1 The aerial small target detection method based on multi-feature parallel fusion provided by this invention further includes the following steps:

[0043] Step 1: Download the VisDrone public dataset for small drone targets, and adjust the image resolution of the training and test sets to 640×640.

[0044] Step 2: Design a self-correcting convolutional structure. This involves performing convolutional feature transformations in two different scale spaces, extracting local salient features using max pooling downsampling, and obtaining features with a smaller resolution and larger receptive field. These features are then combined with the feature transformation results obtained from the original feature space to provide correction and enrich the contextual information at each spatial location. The specific calculation method is as follows:

[0045] X1'=Act(X+Up(F2(MaxPool(X))))

[0046] Y = F3(F1(X)·X1')

[0047] In this algorithm, MaxPool() represents max pooling, F represents convolution, UP() represents upsampling, and Act() represents the activation function. First, max pooling is used for downsampling to extract local features. Then, a 3×3 convolution feature transformation is applied, followed by an activation function to add non-linearity. Finally, bilinear interpolation is used for upsampling to restore the original feature space size. To preserve the original input features, identity mapping is used for superposition, and then a sigmoid activation function is applied to add non-linearity. After feature transformation in the original feature space using convolution, features obtained from the small-scale latent space are used for self-calibration. Then, convolution is used for feature fusion to obtain the final target features containing rich contextual information. This approach enables communication between different convolution kernels and different scale spaces, expanding the receptive field and facilitating the detection of small targets.

[0048] Step 3: Design a wave-based visual multilayer perceptron structure, which includes the following process:

[0049] Step 301: For the VisionMLP structure, the image is first segmented into multiple tokens. Each token is treated as a wave function containing amplitude and phase, as shown below:

[0050] t j =|t j |⊙cosθ j +i|t j |⊙sinθ j j = 1, 2, ..., n

[0051] The result of token aggregation is affected by the amplitude and phase difference between two tokens. The relationship between tokens and fixed weights is dynamically adjusted by the aggregation of wave functions, thereby achieving a better token aggregation effect.

[0052] In the token aggregation module (MPATM), max pooling is used to generate amplitude and phase, which are then aggregated using TokenFC. ChannelFC is then used to enhance feature representation, yielding the module's final output. The formulas for TokenFC and ChannelFC are shown below:

[0053] ChannelFC(t j W c ) = W c t j j = 1, 2, ..., n

[0054]

[0055] Among them W c The weights, W, have learnable parameters.t It represents the token-mixing weight, where ⊙ indicates element-wise multiplication, and the index j represents the j-th output token.

[0056] Step 302: MWaveMLP consists of Token-MLP and Channel-MLP. Token-MLP uses two parallel MPATM modules and additional ChannelFC connections to preserve the original information, then performs a weighted sum of their outputs. Channel-MLP is implemented by stacking two ChannelFCs and a non-linear activation function. This achieves both global feature extraction and enhanced local feature extraction.

[0057] Step 4: Design a multi-branch parallel feature extraction module (MBblock). For local components, one branch uses max pooling to calculate local maxima; another branch uses depthwise separable convolution to extract local features while reducing the number of parameters; and a third branch uses our designed maximizing self-calibrating convolution to enhance local feature extraction while strengthening the extraction of contextual information for small targets, obtaining more information to assist in small target feature detection. For global components, MWaveMLP is used to enhance local feature extraction while extracting global features. To reduce computational burden, the input features are first reduced in dimensionality, then features are extracted through the four branches, and finally the feature components are concatenated together. Convolution and activation functions are then used to fuse the features and restore the input feature dimension. By gradually expanding outward from local features and fusing them, the advantages of convolution and VisionMLP structures are fully utilized to achieve optimal extraction of small target features.

[0058] Step 5: Combine our designed multi-branch parallel feature extraction module (MBblock) with YOLOX to build our small object detection network. Since shallow layers contain rich positional information, while deeper layers have more semantic information to aid classification, and small object information accounts for a small proportion of the feature map with less information, downsampling will gradually weaken the information of small objects. Therefore, the extraction of key features mainly focuses on shallow features. In the first two layers of the backbone network, we enhance the feature extraction of small objects by stacking MBblocks. In the last two layers, we use the original module CSPblock to replace the YOLOX backbone network to construct the small object detection model.

[0059] Step 6: First, train the model using the COCO dataset to obtain pre-trained weights, then load the pre-trained weights onto the VisDrone dataset for training, for a total of 100 training cycles.

[0060] Step 7: After obtaining the model trained on the VisDrone dataset, detection was performed on the test set, achieving high-precision aerial small target detection results.

[0061] The aerial small target detection method based on multi-feature parallel hybridization adopted in this invention combines CNN structure and Vision MLP in parallel. It utilizes the local feature extraction capability of convolutional structure and realizes context information extraction. Combined with Vision MLP structure, it enhances local feature extraction while ensuring global feature extraction. Finally, through feature fusion, it obtains richer small target feature information to assist detection. It achieves high-precision detection effect on aerial small target images when the input resolution is reduced, and has important application value.

[0062] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.

Claims

1. A method for detecting small targets in aerial photography based on parallel hybrid multi-feature analysis, characterized in that, Includes the following steps: Step 1: Download the VisDrone public dataset for small drone targets, which includes training and testing sets, and adjust the image resolution of the dataset; Step 2: Design a maximizing self-correcting convolutional structure to enhance the extraction of local and contextual features of small targets; the designed maximizing self-correcting convolutional structure performs convolutional feature transformation in two different scale spaces, uses max pooling downsampling to extract local salient features, and obtains features with smaller resolution and larger receptive fields. This feature is then combined with the feature transformation results obtained from the original feature space to play a corrective role and enrich the contextual information of each spatial location. Step 3: Leveraging quantum principles, construct a wave-based visual multilayer perceptron structure, MWaveMLP. Treat tokens as waves and use max pooling to generate amplitude and phase. View the aggregation of tokens as the superposition of wave functions to achieve better aggregation results, enhancing local feature extraction while simultaneously improving global feature extraction. The wave-based visual multilayer perceptron structure specifically includes the following processes: Step 301: For the vision MLP structure, the image is first cut into multiple tokens, and each token is regarded as a wave function containing amplitude and phase. In the token aggregation module MPATM, max pooling is used to generate amplitude and phase respectively, then TokenFC is used for aggregation, and ChannelFC is used to enhance the feature representation capability to obtain the final output of the module; the formulas of TokenFC and ChannelFC are shown below: ； in Weights are weights with learnable parameters. It is token-mixing weight. This indicates element-wise multiplication, where the index j represents the j-th output token; Step 302: The wave-based visual multilayer perceptron structure MWaveMLP consists of Token-MLP and Channel-MLP. Token-MLP uses two parallel MPATM modules and additional ChannelFC connections to preserve the original information, and then performs a weighted sum of their outputs. Channel-MLP is implemented by stacking two ChannelFCs and a non-linear activation function; this achieves both global feature extraction and enhanced local feature extraction. Step 4: Combine max pooling, depthwise separable convolution, maximizing self-correcting convolution and wave-based visual multilayer perceptron in parallel to obtain a multi-branch parallel feature extraction module, realize the gradual outward extraction of local features, obtain more contextual features of small targets, and realize global and local feature extraction and fusion. Step 5: Using a combination of a multi-branch parallel feature extraction module and the YOLOX network, establish the entire aerial small target detection model based on convolutional neural networks and visual multilayer perceptrons with multiple features in parallel. Step 6: Load the training set and pre-trained weights, train the network, and save the optimal weight model based on the test accuracy; Step 7: Use the network trained in Step 6 to detect small targets in the aerial photography dataset.

2. The aerial small target detection method based on multi-feature parallel hybridization according to claim 1, characterized in that, In step 1, download the VisDrone public dataset for small drone targets and adjust the image resolution of the training and test sets to 640×640.

3. The aerial small target detection method based on multi-feature parallel hybridization according to claim 1, characterized in that, The specific calculation method for step 2 is as follows: ； ； In this approach, MaxPool() represents max pooling, F represents convolution, UP() is upsampling, and Act() is activation. First, max pooling is used for downsampling to extract local features. Then, a 3×3 convolution feature transformation is applied, followed by an activation function to add non-linearity. Finally, bilinear interpolation is used for upsampling to restore the original feature space size. To preserve the original input features, identity mapping is used for superposition, and then a sigmoid activation function is applied to add non-linearity. After feature transformation in the original feature space using convolution, features obtained from the small-scale latent space are used for self-calibration. Then, convolution is used for feature fusion to obtain the final target features containing rich contextual information. This method enables communication between different convolution kernels and different scale spaces, expanding the receptive field and facilitating the detection of small targets.

4. The aerial small target detection method based on multi-feature parallel hybridization according to claim 1, characterized in that, In step 301, for the vision MLP structure, the image is segmented into multiple tokens, and each token is considered as a wave function containing amplitude and phase, as shown below: ； The result of token aggregation is affected by the amplitude and phase difference between two tokens. The relationship between tokens and fixed weights is dynamically adjusted by the aggregation of wave functions, thereby achieving a better token aggregation effect.

5. The aerial small target detection method based on multi-feature parallel hybridization according to claim 1, characterized in that, In step 4, a multi-branch parallel feature extraction module MBblock is designed. For local components, one branch uses max pooling to calculate the local maximum value; one branch uses depthwise separable convolution to extract local features while reducing the number of parameters; and one branch uses our designed maximizing self-correcting convolution to enhance the extraction of local features and strengthen the extraction of contextual information of small targets, thereby obtaining more information to assist in the detection of small target features. For the global component, MWaveMLP is used to enhance local feature extraction while extracting global features. To reduce the computational burden, the input features are first reduced in dimensionality, then features are extracted through four branches, and finally the feature components are connected together. Convolution and activation functions are used to fuse the features and restore them to the input feature dimension. By gradually expanding the extraction of features from local to external regions and fusing them, the advantages of convolution and vision MLP structures are fully utilized to achieve optimal extraction of features for small targets.

6. The aerial small target detection method based on multi-feature parallel hybridization according to claim 1, characterized in that, In step 5, we combine our designed multi-branch parallel feature extraction module MBblock with YOLOX to build our small target detection network. Since the shallow layers contain rich positional information and the deep layers have more semantic information to assist classification, and the information of small targets accounts for a small proportion in the feature map and has less information, downsampling will gradually weaken the information of small targets. Therefore, the extraction of key features is mainly in the shallow features. In the first two layers of the backbone network, we strengthen the feature extraction of small targets by stacking MBblock. In the last two layers, we use the original module CSPblock to replace the backbone network of YOLOX to build a small target detection model.

7. The aerial small target detection method based on multi-feature parallel hybridization according to claim 1, characterized in that, In step 6, the model is first trained using the COCO dataset to obtain pre-trained weights, and then the pre-trained weights are loaded to train the VisDrone dataset for a total of 100 training cycles.

8. The aerial small target detection method based on multi-feature parallel hybridization according to claim 1, characterized in that, In step 7, after obtaining the model trained on the VisDrone dataset, detection is performed on the test set, achieving high-precision aerial small target detection results.

Images