Unmanned aerial vehicle view angle small target detection method based on multi-scale fusion dynamic convolution

Abstract

The invention belongs to the field of deep learning and target detection, and discloses an unmanned aerial vehicle visual angle small target detection method based on multi-scale fusion dynamic convolution. Firstly, according to the detection algorithm, the efficiency and speed of feature extraction are remarkably improved by proposing a multi-scale fusion dynamic convolution module; secondly, a multi-level structure for small target detection is designed, and an additional prediction head is introduced to a larger feature map, so that the detection precision of the small target is remarkably improved, which is crucial for the unmanned aerial vehicle to accurately identify the small target in a complex environment. Finally, the algorithm is high in adaptability, the algorithm can adapt to different data sets and task requirements regardless of the dynamic characteristic of a multi-scale fusion dynamic convolution module or the flexibility of a small target detection structure, and wide compatibility and reliability are provided for application of the unmanned aerial vehicle in different scenes.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and target detection, and relates to a target detection algorithm in the field of computer vision, specifically a small target detection method from the perspective of an unmanned aerial vehicle based on multi-scale fusion dynamic convolution. Background Art

[0002] Small target detection is an important topic in the field of computer vision. It plays a key role in many fields such as maritime rescue, environmental monitoring, defect detection and aerial image analysis. Especially from the perspective of drones, the application of small target detection technology is particularly important. For example, in maritime rescue operations, accurate identification of small targets from the perspective of drones is crucial to timely finding people in distress; in the field of environmental monitoring, small target detection technology carried by drones helps to discover ecological changes in the monitored area; and in remote sensing image analysis, small target detection from the perspective of drones helps to perform monitoring and security tasks. The advancement of small target detection technology in drone applications has not only improved the efficiency of monitoring and reconnaissance, but also promoted the development of computer vision technology to a certain extent, especially visual processing technology from the perspective of drones. Dynamic convolution technology is an important innovation in deep learning. It enables the Convolutional Neural Network (CNN) to dynamically adjust the weights of the convolution kernel according to the characteristics of the input data, thereby improving the adaptability and performance of the model. ODConv (Omni-Dimensional Dynamic Convolution) is a new type of dynamic convolution design (Li C, Zhou A, Yao A. Omni-dimensional dynamic convolution [J]. arXiv preprintarXiv: 2209.07947, 2022.), which significantly enhances the ability of feature extraction by learning the attention of convolution kernels in multiple dimensions such as spatial size, number of input and output channels, and number of convolution kernels. ODConv uses a multi-dimensional attention module to focus on different dimensions of the convolution kernel, thereby achieving fine-tuning of features, which helps the model to more comprehensively capture the contextual information of the image and improve detection accuracy. The design of ODConv enables it to achieve or exceed the performance of multi-convolution kernel methods even when using only a single convolution kernel, while maintaining the compactness of the model. This flexible design allows ODConv to easily replace the convolution layer in traditional CNN and is widely used in tasks such as image classification, object detection, and semantic segmentation.

[0003] In the field of small target detection, especially from the perspective of drones, due to the long shooting distance, wide viewing angle and small proportion of the target in the image, small target detection faces unique challenges such as small target size, sparse feature information, large scale changes, diverse perspectives and complex background. The multi-dimensional attention mechanism of ODConv is particularly effective in processing small-scale targets because it can specifically enhance the feature representation of small targets from the perspective of drones. However, existing small target detection algorithms based on dynamic convolution focus more on small target detection in the small-scale feature dimension of dynamic convolution, while ignoring the processing of medium and high-scale features. To address this technical limitation, we propose a multi-scale fusion dynamic convolution module. The multi-scale feature fusion capability of this module enables it to adapt to the large scale differences and perspective diversity of targets obtained at different flight altitudes and shooting angles of drones. In addition, the uniqueness of this multi-scale fusion dynamic convolution module is that it can process high-frequency detail information in drone images, which is also crucial for small target detection. Summary of the invention

[0004] The present invention aims to provide a small target detection method from the perspective of a drone based on dynamic convolution, which solves the problem that the existing target detection algorithms often ignore the high-scale features of small targets, and at the same time utilizes the efficient reasoning speed of dynamic convolution to improve the running speed of the target detection algorithm.

[0005] The method of the present invention can be deployed in a cloud computing cluster or on a drone equipped with a computing device.

[0006] The technical solution of the present invention:

[0007] A small target detection method from the perspective of a UAV based on multi-scale fusion dynamic convolution, the steps are as follows:

[0008] Step 1: Collect image dataset for model training

[0009] Collect urban aerial image datasets, collect aerial images of corresponding targets for common visual tasks in cities; use the stable diffusion model to augment data to address the small sample problem caused by some scarce samples. The main process is as follows:

[0010] Classify different types of scarce samples into datasets of different domains D k , the domain features of each sample are extracted through the pre-trained ResNet50 model. The formula is as follows:

[0011]

[0012] Where N k represents the number of samples, φ represents the ResNet50 model, Θ represents its pre-training weight, Indicates domain D k A sample in is the weight coefficient set according to the importance of a scarce sample, Indicates D k In-domain features of the dataset. When using stable diffusion for data augmentation, the original generation method that uses random noise for initialization cannot generate images with the characteristics of scarce samples. It is improved to extract in-domain features using the pre-trained model. The generative method is initialized to generate similar images with features in a certain type of scarce sample domain.

[0013] Step 2: Data preprocessing and data enhancement

[0014] The image dataset is preprocessed and enhanced by image rotation, flipping and scaling. Instead of simply superimposing these data enhancement methods, we assign learnable constant coefficients to each enhancement method, which can be optimized during model training. The process is as follows:

[0015] I aug =A(I,θ,φ,σ)=S(F(R(I,θ),φ),σ)

[0016] Where I is the input image; R is the image rotation, θ is the rotation coefficient; F is the image inversion, φ is the flip coefficient; S is the image scaling, σ is the scaling coefficient. A is the composite enhancement method, I aug is the image after data enhancement. During the model training process, we can use the difference between the model's predicted label and the true label to adjust the coefficients of the composite data enhancement using the gradient descent method. In order to ensure the stability of the adjustment, we introduce a regularization term for constraint.

[0017]

[0018] Where L is the mean square error loss function, y is the true label, is the predicted label. 1 Regularization coefficient, L′ is the loss function after constraint. Finally, we constructed a dynamic composite data augmentation strategy, which can adaptively adjust the proportion of various augmentation methods during training, thereby not only enhancing the complexity and diversity of the data set, but also significantly improving the generalization ability of the model.

[0019] Step 3: Feature Extraction Network

[0020] In order to adapt to the small target detection task from the perspective of drones, where the background of the picture is complex and various targets present multiple scales, a new feature extraction network structure based on the fusion of dynamic convolution and convolution network is designed. For targets of different scales, the features extracted under different receptive fields are considered to be fused during the feature extraction process. The specific process is as follows:

[0021] The input image I is subjected to preliminary feature extraction using a traditional convolution module to obtain features at the P1 level. The features at the P1 level are processed using our dynamic convolution feature fusion module, which extracts features of different scales under different receptive fields through three dynamic convolution modules with different step sizes (1, 2, and 4, respectively). The features extracted under these three step sizes are denoted as o 1 , o 2 , o 3 . Set the feature o 2 With o 3 Upsample to feature o through nearest neighbor interpolation 1 Same size o' 2 and o′ 3 . To get o 1 , o′ 2 and o′ 3 Use channel splicing method for feature fusion. The formula is as follows:

[0022]

[0023] Among them C 1 , C 2 and C 3 For the characteristics of 1 , o 2 and 3 The number of channels, o concat (i,j,k) represents the feature value of the kth channel at position (i,j). The concatenated features are the features of the P2 level. In this way, the traditional convolution plus the multi-scale fusion dynamic convolution module constitute a feature extraction process. Similarly, based on the features of the P2 level, features are extracted step by step to obtain multi-scale fused image features at multiple levels, and finally the features of the five levels of P1, P2, P3, P4, and P5 are obtained. The feature sizes of these five levels decrease in sequence. The SPPF module in YOLOV8 is used to process the features of the P5 level to obtain the final feature Fb.

[0024] Step 4: Feature Fusion Network

[0025] In step 3, five levels of features are extracted. Now these five levels of features need to be fused to enhance the perception ability of the model. Based on the framework of the Neck part of the YOLOV8 model, an upsampling process is added again. This operation achieves the effect of fusing P2-level features with a larger feature scale. In addition, the feature extraction module C2f of the original Neck part of YOLOV8 is replaced by the dynamic convolution feature fusion module proposed in step 3. The extracted features are fused so that the model can better adapt to the complexity and multi-scale characteristics of aerial photos in urban environments. The features fused step by step are divided into four levels according to their size and input into the prediction head, corresponding to the P2, P3, P4 and P5 levels in step 3.

[0026] Step 5: Small object detection

[0027] Use the prediction head of the YOLOV8 model to perform target detection. Before detection, all the 4-level features input in step 4 are pre-processed. The process of pre-processing operation is as follows:

[0028] For the feature X input in step 4 i First, channel compression is performed to compress the two-dimensional feature map of each channel into a real number.

[0029]

[0030] Where S is the compressed feature vector, X ic It is feature X i The features of the cth channel, H and W represent the features X i The height and width of the

[0031] Then the feature vector S is passed through two fully connected layers and an activation function to learn the dependencies between channels. The formula is as follows:

[0032] E=σ(W 2 δ(W 1 S))

[0033] Where W 1 is the weight of the first fully connected layer, W 2 is the weight of the second fully connected layer, S is the feature vector obtained in the previous step, δ and σ are the Relu function and Sigmoid function respectively, and E is the learned channel weight.

[0034] The feature weight E is combined with the original feature X according to a specific rule. i Multiply to get the pre-processed feature X i ′, the formula is as follows:

[0035]

[0036] The final feature X i ’ is input into the prediction head of the corresponding level for target detection.

[0037] Beneficial effects of the present invention:

[0038] First, the improved algorithm significantly improves the efficiency and speed of feature extraction by proposing a multi-scale fusion dynamic convolution module, which enables drones to process image data faster when performing small target detection tasks, ensuring the feasibility of real-time detection. The design of the multi-scale fusion dynamic convolution module reduces the computational complexity of the model and reduces resource consumption, which makes the algorithm more suitable for running on drone platforms with limited computing power and improves the practicality of the algorithm.

[0039] Secondly, a multi-level structure specifically designed for small target detection is used to significantly improve the detection accuracy of small targets by introducing additional prediction heads on larger feature maps, which is crucial for drones to accurately identify small targets in complex environments.

[0040] Finally, the algorithm has strong adaptability. Whether it is the dynamic characteristics of the multi-scale fusion dynamic convolution module or the flexibility of the small target detection structure, the algorithm can adapt to different data sets and task requirements, providing wide compatibility and reliability for the application of UAVs in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of the overall structure of the algorithm.

[0042] Figure 2 Schematic diagram of dynamic convolution.

[0043] Figure 3 Schematic diagram of the dynamic convolution module structure for multi-scale fusion. DETAILED DESCRIPTION

[0044] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0045] like Figure 1 As shown, the present invention provides a small target detection model from the perspective of a drone based on dynamic convolution, and its specific structure is as follows:

[0046] (1) Feature extraction network

[0047] The feature extraction network is responsible for gradually extracting features from the input data. The network starts at the input layer, which accepts the original image. It first passes through one or more convolution modules, which are used for feature mapping. Next is the multi-scale fusion dynamic convolution module, which aims to enhance the representation ability of features. The above convolution modules and multi-scale fusion dynamic convolution modules will be repeated many times, and each round will increase the level of abstraction of the features and gradually capture more complex patterns.

[0048] (2) Feature Fusion Network

[0049] The role of the feature fusion network is to integrate feature maps from different scales to produce a multi-scale feature representation. Each multi-scale fused dynamic convolution module in the feature extraction network is connected to the multi-scale feature extraction module in the feature fusion network. This shows that the features extracted at different levels are combined together. The feature fusion network includes an upsampling step, which allows the network to process features at different resolutions, helping to capture multi-scale information. Before and after upsampling, there are convolution modules that continue to transform and combine features to further enhance the representation ability of the features.

[0050] (3) Detection head

[0051] The main function of the detection head is to convert multi-scale features into the final detection results. Each branch has a prediction head that maps features to output results, namely classification probabilities and bounding box coordinates. Before each prediction head, there are multi-scale feature fusion modules and convolution modules, which further adjust and optimize the features to perform detection tasks more accurately.

[0052] like Figure 2 As shown in the figure, the dynamic convolution operation principle used in the dynamic convolution module of multi-scale fusion is as follows: after the input features are averaged and pooled once, they enter a linear layer + relu activation function structure, and then enter a linear layer + softmax layer to obtain the pre-processed features. Then, the convolution kernel Wi is multiplied by the attention in four directions of the pre-processed features, and these results are summed and multiplied with the original features to obtain the output.

[0053] like Figure 3 As shown in the figure, the multi-scale fused dynamic convolution distributes the input data to three parallel dynamic convolution modules through the shunt operation for feature extraction and enhancement. The step sizes of the three dynamic convolutions are 1, 2, and 4 respectively, and the extracted feature sizes decrease in sequence. The three features of different sizes are upsampled to the same size through nearest neighbor interpolation for feature fusion.

Claims

1. A small target detection method from the perspective of a drone based on multi-scale fusion dynamic convolution, characterized in that: Here are the steps: Step 1: Collect image dataset for model training Collect urban aerial image datasets, and collect aerial images of corresponding targets for visual tasks in the city; use the stable diffusion model to augment data to address the small sample problem caused by some scarce samples in the urban aerial image dataset; the main process is as follows: Classify different types of scarce samples into datasets of different domains D k , the domain features of each sample are extracted through the pre-trained ResNet50 model. The formula is as follows: Among them, N k represents the number of samples, φ represents the ResNet50 model, Θ represents its pre-training weight, Indicates domain D k A sample in is the weight coefficient set according to the importance of a scarce sample, Indicates D k In-domain features of the dataset; when using stable diffusion for data augmentation, the original generation method that uses random noise for initialization cannot generate images with scarce sample characteristics. It is improved to extract in-domain features using the pre-trained ResNet50 model An initialization generation method is used to generate similar images with features in a certain type of scarce sample domain; Step 2: Data preprocessing and data enhancement The image dataset is preprocessed and enhanced by image rotation, flipping and scaling. A learnable constant coefficient is assigned to each enhancement method. These constant coefficients are optimized during the ResNet50 model training process. The process is as follows: I aug =A(I,θ,φ,σ)=S(F(R(I,θ),φ),σ) Where I is the input image; R is the image rotation, θ is the rotation coefficient; F is the image inversion, φ is the flip coefficient; S is the image scaling, σ is the scaling coefficient; A is the composite enhancement method, I aug The image is after data enhancement. During the training of the ResNet50 model, the difference between the predicted label and the real label of the ResNet50 model is used to adjust the coefficients of the composite data enhancement using the gradient descent method. In order to ensure the stability of the adjustment, the regularization term is introduced for constraint. Among them, L is the mean square error loss function, y is the true label, is the predicted label; λ is the l1 regularization coefficient, and L′ is the loss function after constraint; finally, a dynamic composite data enhancement strategy is constructed, which can adaptively adjust the proportion of various enhancement methods during the training process; Step 3: Feature Extraction Network A new feature extraction network structure based on the fusion of dynamic convolution and convolutional network is designed; for targets of different scales, the features extracted under different receptive fields are considered to be fused during the feature extraction process; the specific process is as follows: The input image I is subjected to preliminary feature extraction using a convolution module to obtain features at the P1 level. The features at the P1 level are processed using a dynamic convolution module, which extracts features of different scales under different receptive fields through dynamic convolution modules with step sizes of 1, 2, and 4. The features extracted under these three step sizes are recorded as o1, o2, and o3 respectively. Features o2 and o3 are upsampled to the same size o′2 and o′3 as feature o1 through nearest neighbor interpolation. The obtained o1, o′2, and o′3 are subjected to feature fusion using the channel splicing method. The formula is as follows: Among them, C1, C2 and C3 are the number of channels of features o1, o2 and o3, o concat (i,j,k) represents the feature value of the kth channel at position (i,j); the concatenated features are the features of the P2 level; the convolution module plus the multi-scale fusion dynamic convolution module constitute a feature extraction process; by analogy, based on the features of the P2 level, features are extracted step by step to obtain multi-scale fused image features at multiple levels, and finally the features of the five levels P1, P2, P3, P4, and P5 are obtained, and the feature sizes of these five levels decrease in sequence; the SPPF module in the YOLOV8 model is used to process the features of the P5 level to obtain the final feature Fb; Step 4: Feature Fusion Network In step 3, the features of five levels are extracted, and the features of the five levels are fused to enhance the perception ability of the model; based on the framework of the Neck part of the YOLOV8 model, an upsampling process is added again, so that the P2 level features with a larger feature scale can be fused; in addition, the feature extraction module C2f of the Neck part of the original YOLOV8 model is replaced by the dynamic convolution module proposed in step 3; the extracted features are fused; the features fused step by step are divided into four levels according to their size and input into the prediction head, corresponding to the P2, P3, P4 and P5 levels in step 3; Step 5: Small object detection The prediction head of the YOLOV8 model is used for target detection. Before detection, the four-level features input in step 4 are pre-processed. The process of the pre-processing operation is as follows: For the feature X input in step 4 i First, channel compression is performed to compress the two-dimensional feature map of each channel into a real number; Among them, S is the compressed feature vector, X ic It is feature X i The features of the cth channel, H and W represent the features X i The height and width of Then the feature vector S is passed through two fully connected layers and an activation function to learn the dependencies between channels. The formula is as follows: E=σ(W2δ(W1S)) Among them, W1 is the weight of the first fully connected layer, W2 is the weight of the second fully connected layer, S is the feature vector obtained in the previous step, δ and σ are the Relu function and Sigmoid function respectively, and E is the learned channel weight; The feature weight E is combined with the original feature X according to a specific rule. i Multiply to get the pre-processed feature X i ′, the formula is as follows: The final feature X i ’ is input into the prediction head of the corresponding level for target detection.

Images