A small target detection method from UAV perspective based on multi-scale fusion dynamic convolution

Abstract

The present invention belongs to the field of deep learning and target detection, and discloses a small target detection method from the perspective of a drone based on multi-scale fusion dynamic convolution. First, the detection algorithm significantly improves the efficiency and speed of feature extraction by proposing a multi-scale fusion dynamic convolution module. Secondly, a multi-level structure is designed for small target detection, and the detection accuracy of small targets is significantly improved by introducing additional prediction heads on a larger feature map, which is crucial for drones to accurately identify small targets in complex environments. Finally, the algorithm has strong adaptability. Both the dynamic characteristics of the multi-scale fusion dynamic convolution module and the flexibility of the small target detection structure can enable the algorithm to adapt to different data sets and task requirements, providing wide compatibility and reliability for the application of drones in different scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and target detection, and relates to a target detection algorithm in the field of computer vision. Specifically, it is a small target detection method from the perspective of an unmanned aerial vehicle (UAV) based on multi-scale fusion dynamic convolution. Background Art

[0002] Small target detection is a crucial topic in computer vision, playing a key role in numerous fields, including maritime rescue, environmental monitoring, defect detection, and aerial image analysis. The application of small target detection technology is particularly important when viewed from the perspective of drones. For example, in maritime rescue operations, accurately identifying small targets from a drone's perspective is crucial for promptly locating people in distress. In environmental monitoring, small target detection technology onboard drones helps detect ecological changes in monitored areas. And in remote sensing image analysis, small target detection from a drone's perspective facilitates surveillance and security tasks. Advances in small target detection technology in drone applications have not only improved the efficiency of surveillance and reconnaissance but have also, to a certain extent, driven the development of computer vision, particularly visual processing from drones. Dynamic convolution is a key innovation in deep learning. It enables convolutional neural networks (CNNs) to dynamically adjust the weights of the convolution kernels based on the characteristics of the input data, thereby improving the model's adaptability and performance. ODConv (Omni-Dimensional Dynamic Convolution) is a novel dynamic convolution design (Li C, Zhou A, Yao A. Omni-dimensional dynamic convolution [J]. arXiv preprint arXiv:2209.07947, 2022.). It significantly enhances feature extraction capabilities by learning the attention of convolution kernels across multiple dimensions, including spatial size, number of input and output channels, and the number of kernels. ODConv utilizes a multi-dimensional attention module to focus on different dimensions of the kernel, enabling fine-tuning of features. This helps the model more comprehensively capture image context and improve detection accuracy. ODConv's design enables it to match or exceed the performance of multi-kernel methods, even when using only a single kernel, while maintaining model compactness. This flexible design allows ODConv to easily replace the convolutional layers in traditional CNNs and has been widely used in tasks such as image classification, object detection, and semantic segmentation.

[0003] In the field of small target detection, especially from drone perspectives, the long shooting distance, wide viewing angle, and small object size within the image present unique challenges, including small object size, sparse feature information, large scale variations, diverse viewpoints, and complex backgrounds. ODConv's multi-dimensional attention mechanism is particularly effective for handling small-scale objects because it specifically enhances the feature representation of small objects from drone perspectives. However, existing small target detection algorithms based on dynamic convolution focus primarily on detecting small targets in the small-scale feature dimension of dynamic convolution, while neglecting the processing of medium- and high-scale features. To address this technical limitation, we propose a multi-scale fusion dynamic convolution module. This module's multi-scale feature fusion capability enables it to adapt to the wide range of object scales and viewpoints observed at different drone flight altitudes and shooting angles. Furthermore, this multi-scale fusion dynamic convolution module is unique in its ability to process high-frequency detail information in drone imagery, which is also crucial for small target detection. Summary of the Invention

[0004] The present invention aims to provide a small target detection method from the perspective of a drone based on dynamic convolution, which solves the problem that existing target detection algorithms often ignore the high-scale features of small targets. At the same time, it utilizes the efficient inference speed of dynamic convolution to improve the running speed of the target detection algorithm.

[0005] The method of the present invention can be deployed in a cloud computing cluster or on a drone equipped with a computing device.

[0006] The technical solution of the present invention:

[0007] A small target detection method from the perspective of a UAV based on multi-scale fusion dynamic convolution, the steps are as follows:

[0008] Step 1: Collect image dataset for model training

[0009] We collected urban aerial image datasets and collected aerial images corresponding to common visual tasks in cities. We used the stable diffusion model to augment data to address the small sample size problem caused by some scarce samples. The main process is as follows:

[0010] Classify different types of scarce samples into datasets D of different domains k , the domain features of each sample are extracted through the pre-trained ResNet50 model. The formula is as follows:

[0011]

[0012] where N k represents the number of samples, φ represents the ResNet50 model, Θ is its pre-training weight, Represents domain D k A sample in is the weight coefficient set according to the importance of a scarce sample, Indicates D k In-domain features of the dataset. When using stable diffusion for data augmentation, the original generation method that uses random noise for initialization cannot generate images with the characteristics of scarce samples. It is improved to use the in-domain features extracted by the pre-trained model. The generated method is initialized to generate similar images with features in a certain type of scarce sample domain.

[0013] Step 2: Data preprocessing and data enhancement

[0014] We preprocess and enhance the image dataset by rotating, flipping, and scaling the image. Instead of simply stacking these data enhancement methods, we assign a learnable constant coefficient to each enhancement method, which can be optimized during model training. The process is as follows:

[0015] I aug =A(I,θ,φ,σ)=S(F(R(I,θ),φ),σ)

[0016] Where I is the input image; R is the image rotation, θ is the rotation coefficient; F is the image inversion, φ is the flip coefficient; S is the image scaling, σ is the scaling coefficient. A is the composite enhancement method, I aug The image after data augmentation. During model training, we can use the difference between the model's predicted labels and the true labels to adjust the coefficients of the composite data augmentation using gradient descent. To ensure the stability of the adjustment, we introduce a regularization term to constrain it.

[0017]

[0018] Where L is the mean square error loss function, y is the true label, is the predicted label. λ is the l1 regularization coefficient, and L′ is the constrained loss function. Ultimately, we constructed a dynamic, composite data augmentation strategy that adaptively adjusts the proportions of various augmentation methods during training. This not only increases the complexity and diversity of the dataset, but also significantly improves the model's generalization capabilities.

[0019] Step 3: Feature Extraction Network

[0020] To adapt to the complex backgrounds and multi-scale characteristics of small target detection from a drone's perspective, a new feature extraction network structure based on the fusion of dynamic convolution and convolutional networks was designed. For targets of different scales, the feature extraction process considers fusing features extracted from different receptive fields. The specific process is as follows:

[0021] The input image I is subjected to preliminary feature extraction using a traditional convolution module to obtain features at the P1 level. The features at the P1 level are processed using our dynamic convolution feature fusion module, which extracts features of different scales under different receptive fields through three dynamic convolution modules with different step sizes (1, 2, and 4, respectively). The features extracted under these three step sizes are recorded as o1, o2, and o3, respectively. Features o2 and o3 are upsampled to the same size o′2 and o′3 as feature o1 through nearest neighbor interpolation. The obtained o1, o′2, and o′3 are subjected to feature fusion using the channel splicing method. The formula is as follows:

[0022]

[0023] Where C1, C2 and C3 are the number of channels of features o1, o2 and o3, o concat (i, j, k) represents the feature value of the kth channel at position (i, j). The concatenated features are the features at the P2 level. This combines traditional convolution with the multi-scale fusion dynamic convolution module to form a feature extraction process. Similarly, based on the P2 level features, features are extracted step by step to obtain multi-scale fused image features at multiple levels, ultimately resulting in features at the P1, P2, P3, P4, and P5 levels, with the feature sizes of these five levels decreasing in sequence. The P5 level features are processed using the SPPF module in YOLOV8 to obtain the final features Fb.

[0024] Step 4: Feature Fusion Network

[0025] In step 3, five levels of features were extracted. Now, these five levels of features need to be fused to enhance the model's perception capabilities. Based on the framework of the Neck part of the YOLOV8 model, an upsampling process is added. This operation achieves the effect of fusing P2-level features with a larger feature scale. In addition, the original feature extraction module C2f of the Neck part of YOLOV8 is replaced by the dynamic convolution feature fusion module proposed in step 3. The extracted features are fused so that the model can better adapt to the complexity and multi-scale characteristics of aerial images in urban environments. The features fused at each level are divided into four levels according to their size and input into the prediction head, corresponding to the P2, P3, P4, and P5 levels in step 3.

[0026] Step 5: Small object detection

[0027] Use the prediction head of the YOLOV8 model for target detection. Before detection, all four-level features input in step 4 are pre-processed. The pre-processing process is as follows:

[0028] For the feature X input in step 4 i First, channel compression is performed to compress the two-dimensional feature map of each channel into a real number.

[0029]

[0030] Where S is the compressed feature vector, X ic is feature X i The features of the c-th channel, H and W represent the features X i The height and width of the .

[0031] Then the feature vector S is passed through two fully connected layers and an activation function to learn the dependencies between channels. The formula is as follows:

[0032] E=σ(W2δ(W1S))

[0033] Where W1 is the weight of the first fully connected layer, W2 is the weight of the second fully connected layer, S is the feature vector obtained in the previous step, δ and σ are the Relu function and Sigmoid function respectively, and E is the learned channel weight.

[0034] The feature weight E is combined with the original feature X according to a specific rule. i Multiply to get the pre-processed feature X i ′, the formula is as follows:

[0035]

[0036] The final feature X i ’ is input to the prediction head of the corresponding level for target detection.

[0037] Beneficial effects of the present invention:

[0038] First, the improved algorithm significantly improves the efficiency and speed of feature extraction by introducing a multi-scale fusion dynamic convolution module. This enables drones to process image data more quickly when performing small target detection tasks, ensuring the feasibility of real-time detection. The design of the multi-scale fusion dynamic convolution module reduces the model's computational complexity and resource consumption, making the algorithm more suitable for running on drone platforms with limited computing power and improving its practicality.

[0039] Secondly, a multi-level structure specifically for small target detection is designed. By introducing additional prediction heads on larger feature maps, the detection accuracy of small targets is significantly improved, which is crucial for drones to accurately identify small targets in complex environments.

[0040] Finally, the algorithm is highly adaptable. Both the dynamic characteristics of the multi-scale fusion dynamic convolution module and the flexibility of the small target detection structure enable the algorithm to adapt to different data sets and task requirements, providing wide compatibility and reliability for the application of drones in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of the overall structure of the algorithm.

[0042] Figure 2 Schematic diagram of dynamic convolution.

[0043] Figure 3 Schematic diagram of the dynamic convolution module structure for multi-scale fusion. DETAILED DESCRIPTION

[0044] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0045] like Figure 1 As shown in the figure, the present invention provides a small target detection model from the perspective of a drone based on dynamic convolution, and its specific structure is as follows:

[0046] (1) Feature extraction network

[0047] The feature extraction network is responsible for progressively extracting features from the input data. The network begins with the input layer, which receives the raw image. It first passes through one or more convolutional modules, which perform feature mapping. Next, a multi-scale fusion dynamic convolution module is used to enhance feature representation. These convolutional modules and multi-scale fusion dynamic convolution modules are repeated multiple times, each round increasing the level of feature abstraction and capturing increasingly complex patterns.

[0048] (2) Feature Fusion Network

[0049] The feature fusion network integrates feature maps from different scales to produce a multi-scale feature representation. Each multi-scale fusion dynamic convolution module in the feature extraction network is connected to a multi-scale feature extraction module in the feature fusion network. This means that features extracted at different levels are combined. The feature fusion network includes an upsampling step, which allows the network to process features at different resolutions, helping to capture multi-scale information. Before and after the upsampling, convolution modules continue to transform and combine features, further enhancing the feature representation capability.

[0050] (3) Detection head

[0051] The primary function of the detection head is to transform multi-scale features into final detection results. Each branch has a prediction head that maps features into outputs: classification probabilities and bounding box coordinates. Prior to each prediction head, a multi-scale feature fusion module and a convolution module further adjust and optimize the features for more accurate detection.

[0052] like Figure 2 As shown in the figure, the dynamic convolution operation used in the multi-scale fusion dynamic convolution module works as follows: After the input features undergo a single average pooling, they enter a linear layer with a ReLU activation function, and then enter a linear layer with a softmax layer to obtain the pre-processed features. Next, the convolution kernel Wi is multiplied by the attention of the four directions of the pre-processed features. These results are summed and multiplied with the original features to obtain the output.

[0053] like Figure 3 As shown in the figure, the multi-scale fusion dynamic convolution distributes the input data to three parallel dynamic convolution modules through a split operation for feature extraction and enhancement. The step sizes of these three dynamic convolutions are 1, 2, and 4, respectively, and the extracted features are of decreasing size. The three features of different sizes are upsampled to the same size through nearest neighbor interpolation for feature fusion.

Claims

1. A small target detection method from the perspective of a UAV based on multi-scale fusion dynamic convolution, characterized by: Here are the steps: Step 1: Collect image dataset for model training We collected urban aerial image datasets and collected aerial images of corresponding targets for urban visual tasks. We used the stable diffusion model to augment data to address the small sample size problem caused by the scarcity of some samples in the urban aerial image dataset. The main process is as follows: Classify different types of scarce samples into datasets D of different domains k , the domain features of each sample are extracted through the pre-trained ResNet50 model. The formula is as follows: Among them, N k represents the number of samples, φ represents the ResNet50 model, Θ is its pre-training weight, Represents domain D k A sample in is the weight coefficient set according to the importance of a scarce sample, Indicates D k Intra-domain features of the dataset; when using stable diffusion for data augmentation, the original generation method that uses random noise for initialization cannot generate images with the characteristics of scarce samples. It is improved to extract in-domain features using the pre-trained ResNet50 model An initialization generation method is used to generate similar images with features in a certain type of scarce sample domain; Step 2: Data preprocessing and data enhancement The image dataset is preprocessed and enhanced by image rotation, flipping, and scaling. Each enhancement method is assigned a learnable constant coefficient, which is optimized during the ResNet50 model training process. The process is as follows: I aug =A(I,θ,φ,σ)=S(F(R(I,θ),φ),σ) Where I is the input image; R is the image rotation, θ is the rotation coefficient; F is the image inversion, φ is the flip coefficient; S is the image scaling, σ is the scaling coefficient; A is the composite enhancement method, I aug The image after data enhancement. During the training of the ResNet50 model, the difference between the predicted label and the true label of the ResNet50 model is used to adjust the coefficients of the composite data enhancement using the gradient descent method. In order to ensure the stability of the adjustment, the regularization term is introduced for constraint. Among them, L is the mean square error loss function, y is the true label, is the predicted label; λ is the l1 regularization coefficient, and L′ is the constrained loss function; finally, a dynamic composite data augmentation strategy is constructed, which can adaptively adjust the proportion of various augmentation methods during the training process; Step 3: Feature Extraction Network A new feature extraction network structure based on the fusion of dynamic convolution and convolutional network is designed. For targets of different scales, the features extracted under different receptive fields are considered to be fused during the feature extraction process. The specific process is as follows: The input image I is subjected to preliminary feature extraction using a convolution module to obtain features at the P1 level. The P1 level features are processed using a dynamic convolution module, which extracts features of different scales under different receptive fields using dynamic convolution modules with step sizes of 1, 2, and 4. The features extracted under these three step sizes are denoted as o1, o2, and o3 respectively. Features o2 and o3 are upsampled to the same size o′2 and o′3 as feature o1 through nearest neighbor interpolation. The obtained o1, o′2, and o′3 are subjected to feature fusion using the channel splicing method. The formula is as follows: Among them, C1, C2 and C3 are the number of channels of features o1, o2 and o3, o concat (i, j, k) represents the feature value of the kth channel at position (i, j); the concatenated features are the features of the P2 level; the convolution module plus the multi-scale fusion dynamic convolution module constitute a feature extraction process; and so on, based on the features of the P2 level, features are extracted step by step to obtain multi-scale fused image features at multiple levels, and finally the features of the five levels P1, P2, P3, P4, and P5 are obtained. The feature sizes of these five levels decrease in sequence; the SPPF module in the YOLOV8 model is used to process the features of the P5 level to obtain the final feature Fb; Step 4: Feature Fusion Network In step 3, the features of five levels are extracted and fused to enhance the perception ability of the model. An upsampling process is added to the framework of the Neck part of the YOLOV8 model to achieve the effect of fusing the P2 level features with a larger feature scale. In addition, the feature extraction module C2f of the Neck part of the original YOLOV8 model is replaced by the dynamic convolution module proposed in step 3. The extracted features are fused. The fused features are divided into four levels according to their size and input into the prediction head, corresponding to the P2, P3, P4 and P5 levels in step 3. Step 5: Small object detection Use the prediction head of the YOLOV8 model for target detection. Before detection, all four-level features input in step 4 are pre-processed. The process of pre-processing operation is as follows: For the feature X input in step 4 i First, channel compression is performed to compress the two-dimensional feature map of each channel into a real number; Among them, S is the compressed feature vector, X ic is feature X i The features of the c-th channel, H and W represent the features X i height and width; Then the feature vector S is passed through two fully connected layers and an activation function to learn the dependencies between channels. The formula is as follows: E=σ(W2δ(W1S)) Where W1 is the weight of the first fully connected layer, W2 is the weight of the second fully connected layer, S is the feature vector obtained in the previous step, δ and σ are the ReLU function and Sigmoid function respectively, and E is the learned channel weight; The feature weight E is combined with the original feature X according to a specific rule. i Multiply to get the pre-processed feature X i ′, the formula is as follows: The final feature X i ’ is input to the prediction head of the corresponding level for target detection.

Images