Optical remote sensing target detection method based on YOLO11 and fused with multi-scale attention

Abstract

The invention relates to an optical remote sensing target detection method based on YOLO11 and fused with multi-scale attention, and solves the problems existing in the current method. An improved Backbone and Neck network is adopted to perform feature extraction of a small target, a mosaic data enhancement method is adopted in an input part, and a backbone network comprises Conv, C3K2, SPPF and C3k2StarCAA modules; the neck network combines a multi-scale attention fusion module and a context information fusion module and enhances the expression ability of features, a C3K2 module serves as a main improvement part of YOLO11, dynamic switching of parameters is achieved through a C3K structure, and a deep separable convolution (DWConv) and a self-adaptive anchor frame mechanism are introduced, so that the parameter quantity is remarkably reduced, and the fusion efficiency is improved. The module improves a feature extraction module in a YOLO series model, so that the YOLO series model has higher adaptability when processing a complex scene, in addition, the multi-scale attention fusion and context information fusion module further improves the robustness of features, and thus the detection performance of small targets in remote sensing images is improved.

Description

Technical Field

[0001] This invention belongs to the field of military intelligent testing technology and is mainly used for testing and improving autonomous target recognition algorithms. Specifically, it relates to an optical image detection method based on YOLO11 and a fusion attention mechanism. Background Technology

[0002] With the rapid development of remote sensing technology, optical remote sensing imagery has been widely applied in disaster relief, traffic flow monitoring, and military fields. The continuous advancement of these technologies has also injected strong momentum into the development of optical remote sensing image detection technology. The core task of optical remote sensing image detection is to accurately identify the location and category of target objects in remote sensing images, such as the detection of aircraft and drones. Optical remote sensing cameras are typically mounted on flight platforms such as satellites and aircraft, possessing a wide field of view and capable of acquiring image data over large areas. When aircraft are chosen as the remote sensing platform, even though their flight altitude is usually lower than that of satellites, they can still reach altitudes of several kilometers, covering areas of several square kilometers, thereby achieving rapid imaging of specific target areas and providing a comprehensive perspective for regional planning and environmental monitoring.

[0003] Furthermore, remote sensing images taken from different angles can provide three-dimensional spatial information of ground features, offering fundamental data for geological hazard assessment. However, the detection of targets in optical remote sensing images faces numerous challenges, such as complex terrain and lighting conditions. These issues make target detection more challenging and highlight the importance of target detection algorithms as a core technology in optical remote sensing image analysis.

[0004] Traditional object detection algorithms typically rely on manual feature extraction, such as Scale Invariant Feature Transform (SIFT) and Histogram of Oriented Gradients (HOG). These methods are sensitive to occlusion or noise and have limited ability to capture high-order semantic information. While machine learning algorithms such as Support Vector Machines (SVM), K-Nearest Neighbors (KNN), and Random Forests have shown some effectiveness in improving classification efficiency, their accuracy drops significantly when faced with complex backgrounds. With the development of deep learning technology, neural network-based optical remote sensing image detection methods have gradually become the mainstream. Deep learning possesses end-to-end learning capabilities, enabling it to automatically learn and extract high-order semantic features, thereby replacing traditional feature extraction methods.

[0005] Despite the significant advantages that deep learning technology has brought to object detection, object detection in remote sensing images still faces unique challenges. Remote sensing targets often exhibit characteristics such as low resolution and severe occlusion, making it difficult for traditional detection models to effectively identify them. Accurate detection of small targets, in particular, remains a key challenge in remote sensing image detection. Since subsequent applications heavily rely on the accuracy of recognition, improving the precision of small target detection is crucial for remote sensing image detection.

[0006] Based on the above analysis, how to achieve accurate detection and classification of targets in remote sensing images has become a critical issue that urgently needs to be addressed. Summary of the Invention

[0007] This invention addresses the aforementioned problems and needs by proposing an optical remote sensing image detection method based on an improved YOLOv11. It employs an improved Backbone and Neck network for small target feature extraction, and utilizes a mosaic data augmentation method in the input portion. The Backbone network includes Conv, C3K2, SPPF, and C3k2_Star_CAA modules; the Neck network combines multi-scale attention fusion and contextual information fusion modules to enhance feature expressiveness. The C3K2 module, a major improvement on YOLOv11, achieves dynamic parameter switching through the C3K structure and introduces depthwise separable convolution (DWConv) and adaptive anchor box mechanisms, significantly reducing the number of parameters. This module improves the feature extraction module in the YOLO series models, making it more adaptable to complex scenes. Furthermore, the multi-scale attention fusion and contextual information fusion modules further enhance feature robustness, thereby improving the detection performance of small targets in remote sensing images.

[0008] To achieve the above objectives, the present invention provides the following technical solution: An optical remote sensing target detection method based on YOLO11 and multi-scale attention fusion, the specific steps of which are as follows: Step 1: Use the publicly available SIMD remote sensing dataset to convert the labels in the dataset to YOLO format; Furthermore, step 1 specifically includes: Step 1.1: Use a Python script to convert the dataset format. Convert from PASCAL VOC or COCO format to YOLO format; Step 2: Construct an optical remote sensing image detection model based on an improved YOLO11 network. First, the public dataset needs to be divided proportionally. Second, data augmentation techniques are used to process the training set images. Then, the augmented images are input into the improved YOLO11 network for training, during which the model performance is optimized by adjusting parameters such as the learning rate and the number of iterations. Finally, the optimal model weights are obtained after training, achieving accurate detection of targets in optical remote sensing images. Furthermore, step 2 specifically includes: Step 2.1: Divide the acquired images into training set, validation set and test set in a ratio of 8:1:1. There is no overlap between training samples, validation samples and test samples to avoid indirect correlation between network training and testing processes. Step 2.2: Enhance the images in the training set using the mosaic data augmentation method; The mosaic data augmentation method randomly crops, scales, and stitches four images together to form a new image. The specific steps are as follows: Select center point: Determine the center point of the mosaic image for subsequent stitching tasks; Random image selection: Images are randomly selected from the training set; Cropping and scaling: Performs cropping and scaling operations on each image; Stitching: Stitching together the processed images; Adjust bounding box: Adjust the size of the target bounding box; Generate enhanced image: Generate a new image after adjusting the bounding box; Step 2.3: Set the hyperparameters of the model, including Epoch, Batch size, Learning Rate, and Momentum; Step 3: In the improved YOLO11 network, the C3K2 module is replaced by C3k2_Star_CAA in the Backbone part to improve the ability to extract small targets and training efficiency. Meanwhile, multi-scale attention fusion and context information fusion modules are introduced in the Neck section to address the problem of target information being gradually lost during multi-layer propagation. Introducing the above two modules enables the model to better understand multi-scale target information during training, and ultimately obtains the optimal model parameters through training iterations.

[0009] Step 4: Input the test set images into the detection model with optimal weights for testing, thereby obtaining the detection results of targets in each optical remote sensing image.

[0010] As can be seen from the above technical solution, the beneficial effects of the present invention are: 1. To address the issue of poor performance in extracting small targets using the YOLO11 Backbone network, C3k2_Star_CAA was introduced to replace the original C3K2 module, thereby improving the detection accuracy and efficiency for small targets.

[0011] 2. To address the issue of YOLO11's Neck network losing detailed information when fusing shallow features, a multi-scale attention fusion and context information fusion module was designed to enhance the representation ability of important features while suppressing unimportant features, allowing the network to focus on the small target features to be detected.

[0012] 3. The efficient detection model proposed in this invention achieves accurate detection and classification of optical remote sensing images, with an mAP@0.5 (mean accuracy at the 0.5 threshold) of 82.09% and an mAP@0.5:0.95 (mean accuracy at the 0.5-0.95 threshold) of 64.37%.

[0013] In addition to the objectives, features, and advantages described above, the preferred embodiments of the invention will be described in more detail below with reference to the accompanying drawings, so as to facilitate a better understanding of the features and advantages of the invention. Attached Figure Description

[0014] Figure 1 This is a schematic diagram illustrating the specific steps of an optical remote sensing image detection method based on an improved YOLO11 according to the present invention. Figure 2 This is a diagram showing the original image and its enhanced effect in this example; Figure 3 This is a schematic diagram of the structure of the improved YOLO11 network in this example; Figure 4 This is a schematic diagram of the structure of C3k2_Star_CAA in this example; Figure 5 This is a schematic diagram of the context information fusion structure in this example; Figure 6 This is a schematic diagram of the scale attention fusion structure in this example; Figure 7 This is a schematic diagram illustrating the results of optical remote sensing image detection using the improved YOLO11 network in this example. Detailed Implementation

[0015] To more clearly illustrate the purpose, technical solution, and advantages of this invention, the technical solution will be described in detail below with reference to the accompanying drawings of specific embodiments of this invention. It should be noted that the described embodiments are only some examples of this invention, and not all examples. All other embodiments derived by those skilled in the art based on these examples without creative effort are included within the protection scope of this invention.

[0016] The YOLO11 network structure inherits the advantages of the YOLO series, consisting of four parts: input, backbone, neck, and output. The input part of YOLO11 mainly consists of mosaic data augmentation and cropping, uniformly scaling or grayscale filling the input size to 640×640.

[0017] The Backbone network consists of Conv, C3K2, SPPF, and C3k2_Star_CAA modules, while the Neck network integrates multi-scale attention and contextual information fusion modules to enhance feature representation. The C3K2 module, a key improvement in YOLO11, utilizes the C3K structure to achieve dynamic parameter switching, while incorporating depthwise separable convolution (DWConv) and adaptive anchor box mechanisms, significantly reducing the number of parameters. This improvement optimizes the feature extraction module of the YOLO series models, making them more adaptable to complex scenarios. Furthermore, the use of multi-scale attention fusion and contextual information fusion modules further enhances the robustness of features.

[0018] like Figures 1 to 6 As shown, the present invention proposes an optical remote sensing image detection method based on an improved YOLO11, and the specific implementation steps are as follows: Step 1: Divide the acquired images into training, validation, and test sets in an 8:1:1 ratio. The training, validation, and test samples must be completely separate to prevent indirect correlations during network training and testing.

[0019] Step 2: Enhance the image using the mosaic data augmentation method to improve the model's stability. Figure 2 The image shows the original dataset and its enhanced version.

[0020] Figure 3 This is a schematic diagram of the improved YOLO11 network structure. The improved detection network consists of four parts: input, backbone, neck, and head.

[0021] C3K2 is the core module of the backbone network, using smaller 3×3 convolutional kernels instead of traditional larger ones. By segmenting feature maps, performing multiple convolutions, and finally merging them, it effectively improves computational efficiency and optimizes internal information flow.

[0022] The SPPF and C2PSA modules are key components of the neck network. By adopting the concept of multi-branch feature extraction, feature representation and detection accuracy are enhanced.

[0023] The CBS Blocks module is an important component of the detection head, focusing on further refining the feature map, accurately extracting key features, and ensuring high-precision target detection.

[0024] Step 3: Input the training set images from the SIMD dataset into the improved YOLO11 network for training. In the improved network, the Backbone part replaces the C3K2 module with C3k2_Star_CAA to improve the ability to extract small targets and the training speed.

[0025] StarBlock-CAA is a lightweight model architecture that combines StarBlock with the Context Anchor Attention (CAA) attention mechanism, such as... Figure 4 As shown, integrating StarBlock-CAA into the backbone network can effectively reduce the model's complexity and the number of parameters.

[0026] StarBlock is a simplified star-shaped network developed by Microsoft that can achieve high-precision feature extraction without relying on multi-branch structures.

[0027] In StarBlock's architecture, feature extraction and channel reassembly of the feature map are first performed using depthwise separable convolution (DWConv). Subsequently, the feature map is split, and the number of channels is increased through two parallel convolutions, while a ReLU6 activation function is used to introduce non-linearity. Next, element-wise multiplication is used to promote feature interaction and fusion. Finally, context anchor attention (CAA) and depthwise separable convolution (DWConv) are used again to further optimize the feature extraction performance.

[0028] By assigning different weights to each channel, the feature response is adaptively recalibrated to highlight key features, thereby significantly enhancing the discriminative power of the feature map. Furthermore, StarBlock employs a residual structure, fusing input features with the output features after a series of operations during feature extraction. It utilizes identity mapping to directly pass the input to subsequent layers, achieving efficient information transfer and effectively solving the gradient vanishing problem in deep network training. The symbol "*" indicates summation or a star-shaped operation on the input features.

[0029] Star diagrams perform element-wise multiplication to non-linearly combine different channels of the input features, avoiding the introduction of additional computational resources while mapping the input to a higher-dimensional non-linear feature space. In a single-layer neural network, a star diagram can be represented by the following formula.

[0030] in, The blocks represent feature inputs, where W represents the weight term and B represents the bias term. These blocks facilitate the progressive extraction and processing of feature maps, thereby transforming low-level feature images into high-level features.

[0031] The Context Anchor Attention (CAA) module integrates local features with global contextual information awareness, enhancing the expressive power of features during the feature extraction stage, especially when dealing with low-resolution targets. Context Anchor Attention can accurately capture and strengthen key features, effectively compensating for feature loss caused by insufficient resolution.

[0032] The CAA module first extracts local region features through global average pooling. Then, it uses 1... 1. Convolution changes the size of the feature map, as shown in the following formula.

[0033] To effectively capture long-range contextual information, two depthwise separable convolutions are used to effectively extract spatial and channel features, as shown in the following formula.

[0034] Finally, through 1 1. Convolution and the Sigmoid function generate attention weights, as shown in the following formula.

[0035] in, Indicates the feature input; Avg represents average pooling; 1 1. Convolution; and For depthwise separable convolution, and This represents the kernel size.

[0036] The features extracted by the Backbone are input into the Neck part, where they are deeply fused. The Neck network introduces multi-scale attention fusion and contextual information fusion modules, effectively avoiding the loss of target information during multi-layer propagation.

[0037] Figure 5 It is a Contextual Information Fusion Block structure used to guide and adaptively adjust contextual information during multi-scale feature fusion.

[0038] The context information fusion module receives feature inputs from two branches and generates corresponding attention weights using the SE attention mechanism. These weights are then applied element-wise to the corresponding original features to calibrate them. The calibrated features are then fused with features from the other branch, thereby enhancing the representation of important features while suppressing unimportant ones.

[0039] The attention mechanism of the context information fusion module recalibrates the channel feature responses by adaptively learning weights. This mechanism enables the module to automatically adapt to different feature fusion scenarios, highlighting feature channels with high information content, thereby providing more discriminative feature representations for object detection.

[0040] Suppose the two input tensors are respectively and First, through 1 1 convolution The channel was adjusted to be consistent with The same, as shown in the following formula.

[0041] Adjusted features and The concatenation is performed along the channel dimension, as shown in the following formula.

[0042] The spliced ​​features The attention weight for each channel is calculated using the SEAttention module, as shown in the formula below.

[0043] In the SE attention module, global average pooling and fully connected layers generate channel weights as shown in the following formula.

[0044] in, It is the Sigmoid activation function. and It is a fully connected layer.

[0045] Attention weights are applied to the original features, and feature fusion is performed as shown in the following formula.

[0046] Figure 6This is a schematic diagram of the Multi-Scale Attention Fusion Block. This module achieves adaptive integration of local and global features through an attention mechanism that fuses local and global features, thereby effectively handling multi-scale input scenes.

[0047] The innovation of the multi-scale attention fusion module lies in fusing inputs at different scales and refining the input features through a local-global attention branch. This design includes three parallel branches: a local branch, a global branch, and a fusion branch. Each branch is responsible for extracting features at different scales, fully utilizing global contextual information while preserving local details.

[0048] For a given input feature tensor In the fusion branch The number of channels is adjusted during convolution, and then... The features are integrated during convolution, and the output feature tensor is: The feature tensors obtained through local and global branches are as follows: and The formula for merging the three branches is shown below.

[0049] In the local-global attention branch, the difference between the local and global branches is controlled by the patch size parameter. This is achieved through the parameters. This is achieved through the aggregation and displacement of non-overlapping patches in the spatial dimension. Among them... Indicates the local branch. This indicates the global branch.

[0050] First, Divided into patches that are continuous in spatial dimensions ( Secondly, channel averaging is performed to obtain ( Subsequently, a feedforward neural network (FFN) and activation functions are used to obtain the probability distribution of linearly computed features in the spatial dimension, and the feature weights are dynamically adjusted accordingly. Next, feature selection is performed on the weighted features to filter out features highly relevant to the task in both the label and channel dimensions.

[0051] Specifically, for the feature matrix Perform a linear transformation, select the optimal channel for each marker, and finally perform feature reshaping and bilinear interpolation to obtain the desired result. and .

[0052] Feature fusion of all three branches passed Convolution and RepConv Implementation. The first convolution is used for dimensionality reduction, decreasing the computational complexity of subsequent steps. Here, reparameterizable convolutions are employed for feature recombination to improve parameter utilization and more efficiently integrate multi-branch features. The formula is shown below.

[0053] By introducing the two modules mentioned above, the model's ability to capture and understand multi-scale target information is significantly enhanced, ensuring the accuracy of feature representation during training, thereby obtaining the optimal model parameters through iterative training.

[0054] Step 4: Input the test set images into the trained detection model with optimal weights for performance evaluation, thereby obtaining accurate detection results of target objects in each optical remote sensing image.

[0055] Table 1. Metrics results of the improved YOLO11 algorithm for each category on the SIMD test set. Category Name accuracy Recall rate F1 score mAP50 mAP75 mAP50:95 car 85.89% 86.31% 86.10% 92.99% 88.31% 72.69% truck 79.99% 76.15% 78.02% 84.62% 82.14% 67.41% van 83.79% 78.33% 80.97% 87.99% 85.28% 70.90% long vehicle 72.98% 72.32% 72.65% 81.95% 73.79% 62.63% bus 84.79% 90.87% 87.72% 93.99% 89.46% 74.84% airliner 90.95% 98.81% 94.72% 98.77% 98.77% 91.23% propeller aircraft 95.62% 92.86% 94.22% 96.27% 96.27% 83.74% trainer aircraft 81.08% 91.67% 86.05% 88.65% 82.89% 70.93% chartered aircraft 89.07% 92.27% 90.64% 95.18% 91.73% 83.05% fighter aircraft 94.64% 71.43% 81.41% 86.90% 86.90% 74.02% others 45.91% 30.59% 36.72% 36.26% 33.30% 28.70% stair truck 72.73% 40.82% 52.29% 57.22% 42.38% 37.07% pushback truck 47.70% 24.00% 31.93% 38.21% 32.64% 25.01% helicopter 72.70% 100% 84.19% 95.50% 25.36% 46.58% boat 86.81% 97.10% 91.67% 96.80% 93.61% 76.68% all 78.98% 76.23% 76.62% 82.09% 73.52% 64.37% Table 2. Comparison results of the improved YOLO11 and the original algorithm on the SIMD test set. method accuracy Recall rate F1 score mAP50 mAP75 mAP50:95 YOLO11 71.68% 75.03% 73.32% 79.98% 73.41% 62.68% Ours 78.98% 76.23% 76.62% 82.09% 73.52% 64.37% In this embodiment, the abbreviations / symbols used or appearing above are explained as follows: In this invention: SE (Squeeze-and-Excitation): a channel attention mechanism; StarBlock: a star-shaped computational network; C3k2_Star_CAA: an innovative fusion module; CAA (Context Anchor Attention): a context anchor attention module; DWConv (Depth-wise Convolution): a depthwise separable convolution; ContextualInformation Fusion Block: a context information fusion module; Multi-Scale Attention Fusion Block: a multi-scale attention fusion module.

Claims

1. An optical remote sensing target detection method based on YOLO11 and multi-scale attention fusion, characterized by: Detailed steps as follows: Step 1: Use the publicly available SIMD remote sensing dataset to convert the labels in the dataset to YOLO format; Step 2: To construct an optical remote sensing image detection model based on the improved YOLO11 network, the public dataset needs to be divided proportionally. Secondly, data augmentation techniques are used to process the training set images; then, the augmented images are input into the improved YOLO11 network for training, during which the model performance is optimized by adjusting the learning rate and the number of iterations; finally, the optimal model weights are obtained after training, so as to achieve accurate detection of targets in optical remote sensing images. Step 3: In the improved YOLO11 network, the C3K2 module is replaced by C3k2_Star_CAA in the Backbone part to improve the ability to extract small targets and training efficiency. Meanwhile, multi-scale attention fusion and context information fusion modules are introduced in the Neck section to solve the problem of target information being gradually lost in multi-layer propagation; Introducing the above two modules enables the model to better understand multi-scale target information during training, and ultimately obtains the optimal model parameters through training iterations; Step 4: Input the test set images into the detection model with optimal weights for testing, thereby obtaining the detection results of targets in each optical remote sensing image.

2. The optical remote sensing target detection method based on YOLO11 and multi-scale attention fusion as described in claim 1, characterized in that: Step 1 specifically includes: using a Python script to convert the dataset format from PASCALVOC or COCO format to YOLO format.

3. The optical remote sensing target detection method based on YOLO11 and multi-scale attention fusion as described in claim 1, characterized in that: Step 2 specifically includes: Step 2.1: Divide the acquired images into training set, validation set and test set in a ratio of 8:1:

1. There is no overlap between training samples, validation samples and test samples to avoid indirect correlation between network training and testing processes. Step 2.2: Enhance the images in the training set using the mosaic data augmentation method. Step 2.3: Set the hyperparameters of the model, including Epoch, Batch size, Learning Rate, and Momentum.

4. The optical remote sensing target detection method based on YOLO11 and multi-scale attention fusion as described in claim 3, characterized in that: The specific method for enhancing images in the training set using the mosaic data augmentation method is as follows: The mosaic data augmentation method randomly crops, scales, and stitches four images together to form a new image. The specific steps are as follows: Select center point: Determine the center point of the mosaic image for subsequent stitching tasks; Random image selection: Images are randomly selected from the training set; Cropping and scaling: Performs cropping and scaling operations on each image; Stitching: Stitching together the processed images; Adjust bounding box: Adjust the size of the target bounding box; Generate an enhanced image: A new image is generated after adjusting the bounding box.

Images