A remote sensing small target detection method and system based on multi-modal fusion
By introducing a multimodal fusion method that combines a hybrid aggregation network and a spatial enhancement module into the YOLO11 model, the problem of unstable accuracy in the detection of small targets in remote sensing is solved, achieving higher detection accuracy and robustness, and making it suitable for complex remote sensing scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- DONGHUA UNIV
- Filing Date
- 2026-03-09
- Publication Date
- 2026-06-12
AI Technical Summary
Existing remote sensing small target detection technologies suffer from unstable accuracy and insufficient robustness in multimodal fusion applications. Traditional YOLO models cannot effectively utilize the complementary advantages of multimodal data, and existing multimodal fusion methods fail to fully explore the intrinsic correlations between modalities, resulting in limited detection performance.
A YOLO11-based multimodal fusion method is adopted, introducing a hybrid aggregation network module and a spatial enhancement module to construct feature extraction, feature fusion, and target detection modules. Through cross-scale and cross-modal feature fusion of multi-scale feature maps, a multi-scale feature fusion map is generated to enhance the small target detection capability.
It significantly improves the accuracy and robustness of remote sensing small target detection, with a 3.95 percentage point improvement in detection accuracy at mAP@0.5 and a 1.6 percentage point improvement at mAP@0.5:0.95, demonstrating good generalization ability and robustness in complex remote sensing scenarios.
Smart Images
Figure CN122200337A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of remote sensing target detection, and in particular to a method and system for detecting small remote sensing targets based on multimodal fusion. Background Technology
[0002] With the continuous development of remote sensing technology, remote sensing images have been widely used in various fields such as national defense, transportation, environmental protection, and disaster prevention and control. As one of the core technologies of remote sensing image processing, remote sensing target detection analyzes remote sensing images acquired by satellites or airborne platforms to achieve automatic identification and precise positioning of specific targets in the images. It has important practical application value in scenarios such as national defense surveillance, traffic management, disaster assessment, and environmental monitoring.
[0003] Remote sensing targets are diverse in type and vary in detection difficulty. Among them, small target detection, due to its inherent characteristics, has become a key technical challenge in the field of remote sensing target detection. In recent years, with the popularization and application of high-resolution remote sensing imagery, the research focus of remote sensing target detection has gradually shifted from traditional large-scale target detection to small and dense target detection, placing higher demands on detection accuracy and robustness.
[0004] Given the increased detection requirements and the inherent characteristics of small remote sensing targets, their detection difficulty has significantly increased, with the following specific characteristics:
[0005] 1. Small pixel size and lack of semantic information: The pixel ratio of small targets in remote sensing is extremely low, and features such as texture and contour are scarce, resulting in insufficient semantic information. This makes it difficult for traditional detection methods to accurately extract features and identify targets.
[0006] 2. High similarity between target and background, weak distinguishability: The grayscale and texture of small targets are highly similar to the surrounding buildings, roads, vegetation and other backgrounds, resulting in low distinguishability and easy confusion, which further increases the difficulty of detection.
[0007] 3. Complex imaging conditions and significant differences between different modalities: Remote sensing images are easily interfered with by factors such as illumination and weather. Moreover, the imaging mechanisms of different modalities, such as optical and SAR, are very different, requiring complex processing techniques to effectively fuse multimodal information to meet detection requirements.
[0008] To address the aforementioned challenges in detection, existing remote sensing small target detection technologies often focus on single-modal processing. However, due to insufficient data information, their detection results fall short of high-precision requirements. Therefore, multimodal remote sensing data fusion has become a key direction for improving detection accuracy. The complementarity of optical and SAR multimodal data can provide richer features, enhancing detection robustness and accuracy.
[0009] In existing technologies, the YOLO series of CNN-based algorithms are widely used for small target detection in remote sensing due to their high efficiency in end-to-end detection. However, traditional YOLO models have inherent limitations in handling small targets, and their performance is limited in multimodal fusion applications, with accuracy and robustness needing improvement. Based on the direction of multimodal fusion, relevant existing technologies can be mainly divided into four categories, as follows:
[0010] 1. Scheme based on improved YOLO model: By optimizing the YOLO backbone network, reducing downsampling, and adopting multi-scale fusion, the detection capability of small targets is improved. The effect is acceptable in single-modal images, but in multimodal applications, there are defects such as difficulty in fusion and unstable accuracy, which cannot give full play to the complementary advantages of multimodal models.
[0011] 2. Multimodal data fusion-based solutions: Introducing multimodal fusion strategies to achieve joint detection through data splicing or simple feature fusion can initially compensate for the shortcomings of a single modality. However, the fusion method is crude, fails to explore the intrinsic correlation between modalities, has limited fusion effect, and does not significantly improve accuracy.
[0012] 3. Attention-based approach: Introducing an attention mechanism into multimodal fusion can focus on key features and suppress interference, which can improve detection accuracy to some extent. However, the attention weights are difficult to adjust dynamically, and the detection performance is not stable enough in scenarios with large modal differences.
[0013] 4. Deep learning model solutions based on fusion strategies: Through optimization strategies such as weighted concatenation and FPN hierarchical fusion, multi-scale feature representation is enhanced and background interference is alleviated. However, there are still problems such as ineffective resolution of background noise, insufficient modality fusion, and easy missed detection and false detection of small targets. Summary of the Invention
[0014] In view of the aforementioned shortcomings of current remote sensing target detection technologies, this invention provides a method and system for detecting small remote sensing targets based on multimodal fusion. Based on YOLO11 fusion of multimodal and enhancement modules, it can significantly improve the detection accuracy of small remote sensing targets. To achieve the above objectives, the embodiments of this invention adopt the following technical solutions:
[0015] A remote sensing small target detection method based on multimodal fusion, the method comprising:
[0016] Acquire multimodal remote sensing image data and preprocess it;
[0017] A remote sensing small target detection model based on YOLO11 is constructed, which includes a feature extraction module, a feature fusion module and a target detection module. The feature extraction module is based on the original YOLO11 backbone network architecture and further introduces a hybrid aggregation network module. The feature fusion module is based on the original YOLO11 neck network architecture and replaces its built-in convolutional fusion layer with a spatial enhancement module.
[0018] The preprocessed multimodal remote sensing image data is input into the remote sensing small target detection model to obtain the remote sensing small target detection results.
[0019] According to one aspect of the present invention, the preprocessing of multimodal remote sensing image data specifically includes multimodal data alignment, format conversion, size adjustment, and standardization.
[0020] According to one aspect of the present invention, in the remote sensing small target detection model: the feature extraction module includes the Conv module, C3k2 module, and SPPF module of the original YOLO11 backbone network architecture, and a hybrid aggregation network module further introduced between the C3k2 module and the SPPF module. Preprocessed multimodal remote sensing image data is input into the feature extraction module and processed sequentially by the Conv module, C3k2 module, hybrid aggregation network module, and SPPF module to generate a multi-scale feature map. The multi-scale feature map is input into the feature fusion module, and cross-scale and cross-modal feature fusion is achieved by the spatial enhancement module to generate a multi-scale feature fusion map. The multi-scale feature fusion map is input into the target detection module to generate a remote sensing small target detection result.
[0021] According to one aspect of the present invention, the preprocessed multimodal remote sensing image data is input into the feature extraction module, and then processed sequentially by the Conv module, C3k2 module, hybrid aggregation network module, and SPPF module to generate a multi-scale feature map, as follows:
[0022] The preprocessed multimodal remote sensing image data is downsampled by the Conv module to generate initial feature maps at five scales: P1, P2, P3, P4, and P5.
[0023] For the initial feature maps at both scales P2 and P3, the C3k2 module is used for feature enhancement.
[0024] For the initial feature maps at both P4 and P5 scales, a hybrid aggregation network module is used for feature enhancement.
[0025] At the end of the feature extraction module, the SPPF module is used to perform multi-scale context modeling and output multi-scale feature maps.
[0026] According to one aspect of the present invention, the hybrid aggregation network module is used to simultaneously model global semantic context, local spatial texture information, and multi-scale feature dependencies, which can be represented as:
[0027]
[0028] in, For input features, C is the number of channels, and H and W are the feature map dimensions; For channel attention; Spatial attention; It is a multi-scale aggregation function; Enhance operations on a channel-by-channel basis.
[0029] According to one aspect of the present invention, the multi-scale feature map includes three scales, P3, P4, and P5, which respectively correspond to the preprocessed multimodal remote sensing image data. , , size.
[0030] According to one aspect of the present invention, the multi-scale feature map is input to the feature fusion module, and cross-scale and cross-modal feature fusion is achieved through the spatial enhancement module to generate a multi-scale feature fusion map, specifically as follows:
[0031] Channel adjustment is performed on the multi-scale feature maps P3, P4, and P5 to obtain feature maps P3', P4', and P5', respectively.
[0032] After upsampling P4', it is concatenated with P3' and then fused using the spatial enhancement module to obtain the fused feature map F1;
[0033] F1 is downsampled and then concatenated with P4', and then fused using the spatial enhancement module to obtain the fused feature map F2;
[0034] After downsampling F2, it is concatenated with P5' and then fused using the spatial enhancement module to obtain the fused feature map F3.
[0035] According to one aspect of the present invention, the multi-scale feature fusion map is input into the target detection module to generate remote sensing small target detection results specifically as follows: based on the fused feature maps F1, F2, and F3, prediction results corresponding to targets at different scales are generated respectively; wherein, the target detection module adopts a decoupled design including mutually independent classification branches and regression branches, the classification branch outputs the remote sensing small target category probability, and the regression branch outputs the bounding box coordinates.
[0036] According to one aspect of the present invention, the remote sensing small target detection method further includes training a remote sensing small target detection model, wherein the training parameters are as follows: preprocessed multimodal remote sensing image data is used as training data, and its size is adjusted to 640×640; Adam is used as the optimizer, and the initial learning rate is set to... The training batch size is 16, and the training rounds are 500 epochs.
[0037] A remote sensing small target detection system based on multimodal fusion for the above-mentioned remote sensing small target detection method, the system comprising:
[0038] The data acquisition module acquires multimodal remote sensing image data and preprocesses it.
[0039] The model building module constructs a remote sensing small target detection model based on YOLO11, including a feature extraction module, a feature fusion module, and a target detection module. The feature extraction module is based on the original YOLO11 backbone network architecture and further introduces a hybrid aggregation network module. The feature fusion module is based on the original YOLO11 neck network architecture and replaces its built-in convolutional fusion layer with a spatial enhancement module.
[0040] The result generation module inputs the preprocessed multimodal remote sensing image data into the remote sensing small target detection model to obtain the remote sensing small target detection results.
[0041] The advantages of this invention are: by introducing a hybrid aggregation network module and replacing the convolutional fusion layer with a spatial enhancement module, the model's ability to represent and locate the features of small remote sensing targets is effectively enhanced, and the detection performance in complex remote sensing scenarios is significantly improved. Attached Figure Description
[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0043] Figure 1 This is a schematic flowchart of a remote sensing small target detection method based on multimodal fusion as described in this invention;
[0044] Figure 2 This is a structural diagram of a remote sensing small target detection system based on multimodal fusion as described in this invention;
[0045] Figure 3 This is a dynamic monitoring curve of a remote sensing small target detection method based on multimodal fusion as described in this invention. Detailed Implementation
[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0047] Example 1
[0048] like Figure 1As shown, a remote sensing small target detection method based on multimodal fusion is described, the method comprising:
[0049] Step S1: Acquire multimodal remote sensing image data and preprocess it;
[0050] Furthermore, acquiring multimodal remote sensing image data specifically involves:
[0051] The multimodal remote sensing image data includes optical remote sensing images and synthetic aperture radar (SAR) images. These two modalities complement each other to adapt to complex remote sensing scenarios. The multimodal remote sensing image data can be acquired via satellite or aerial platforms using satellite imagery or aerial imagery of the actual scene, or by directly accessing the AI-TOD dataset. The AI-TOD dataset includes small target types such as vehicles, ships, and aircraft, and covers complex backgrounds such as tree shadows, building obstructions, and cloud cover, realistically simulating actual remote sensing detection scenarios.
[0052] Furthermore, the acquired multimodal remote sensing image data undergoes preprocessing, including multimodal data alignment, format conversion, resizing, and standardization, specifically:
[0053] (1) Multimodal data alignment: The optical remote sensing image and the SAR image are spatially aligned by combining coordinate calibration and feature point matching to ensure the pixel-level spatial correspondence between the two modal data and eliminate the positional shift caused by the difference in imaging viewpoint.
[0054] (2) Format conversion: All modal remote sensing image data are uniformly converted into RGB format, and finally image data with dimensions H×W×3 is formed, where H is the image height and W is the image width;
[0055] (3) Size adjustment: The image was uniformly adjusted to 640×640 pixels using bilinear interpolation to meet the input requirements of the remote sensing small target detection model;
[0056] (4) Standardization processing: The adjusted image pixel values are normalized and mapped to the [0,1] interval;
[0057] Optionally, it also includes data augmentation: using the Mosaic data augmentation strategy, four remote sensing images of different modalities are randomly selected, randomly cropped, flipped and stitched together into a 640×640 pixel image to improve the model’s robustness and generalization ability to small targets.
[0058] In summary, the remote sensing image data, after preprocessing, forms an input tensor with a format of 640×640×3.
[0059] Step S2: Construct a remote sensing small target detection model based on YOLO11, including a feature extraction module, a feature fusion module, and a target detection module. The feature extraction module is based on the original YOLO11 backbone network architecture and further introduces a hybrid aggregation network module. The feature fusion module is based on the original YOLO11 neck network architecture and replaces its built-in convolutional fusion layer with a spatial enhancement module.
[0060] Furthermore, in the remote sensing small target detection model: the feature extraction module includes the Conv module, C3k2 module, and SPPF module of the original YOLO11 backbone network architecture, and a hybrid aggregation network module further introduced between the C3k2 module and the SPPF module. The preprocessed multimodal remote sensing image data is input into the feature extraction module, and is processed sequentially by the Conv module, C3k2 module, hybrid aggregation network module, and SPPF module to generate a multi-scale feature map; the multi-scale feature map is input into the feature fusion module, and cross-scale and cross-modal feature fusion is achieved by the spatial enhancement module to generate a multi-scale feature fusion map; the multi-scale feature fusion map is input into the target detection module to generate the remote sensing small target detection result.
[0061] Furthermore, the preprocessed multimodal remote sensing image data is input into the feature extraction module, and then processed sequentially through the Conv module, C3k2 module, hybrid aggregation network module, and SPPF module to generate multi-scale feature maps as follows:
[0062] Step S211: The preprocessed multimodal remote sensing image data is downsampled by the Conv module to generate initial feature maps at five scales: P1, P2, P3, P4, and P5; specifically:
[0063] The preprocessed multimodal remote sensing image data (size 640×640) is input into the Conv module. Downsampling is performed using a convolution operation with a kernel size of 3×3 and a stride of 2 to generate initial feature maps of five scales: P1 (640×640), P2 (320×320), P3 (160×160), P4 (80×80), and P5 (40×40), thus achieving preliminary feature extraction and scale compression.
[0064] Step S212: For the initial feature maps at scales P2 and P3, feature enhancement is performed using the C3k2 module; specifically:
[0065] For the P2 and P3 scale feature maps in the initial feature map, the C3k2 module is introduced for feature enhancement. This module adopts a two-branch and residual connection structure, where branch one performs two 3×3 convolutions sequentially, and branch two performs one 3×3 convolution.
[0066] Step S213: For the initial feature maps at scales P4 and P5, a hybrid aggregation network module is used for feature enhancement; specifically:
[0067] For the P4 and P5 feature maps in the initial feature map, a Mixed Aggregation Network module is introduced for feature enhancement. This module is used to simultaneously model global semantic context, local spatial texture information, and multi-scale feature dependencies, and can be represented as:
[0068]
[0069] in, Here, C represents the input features for P4 and P5, and H and W represent the feature map dimensions. Channel attention (capturing global semantics); Spatial attention (emphasizing local texture); It is a multi-scale aggregation function; For channel-by-channel enhancement operations, it is used to jointly enhance the channel and spatial response of the target area and suppress interference from complex remote sensing backgrounds.
[0070] Step S214: At the end of the feature extraction module, the SPPF module is used to perform multi-scale context modeling and output multi-scale feature maps.
[0071] At the end of the feature extraction module, an SPPF module is introduced for multi-scale context modeling. Without significantly increasing computational cost, this expands the receptive field of high-level features and enhances the global perception capability of targets at different scales.
[0072] Finally, after the preprocessed multimodal remote sensing image data is input into the feature extraction module, it is processed sequentially by the modules mentioned above, ultimately outputting multi-scale feature maps at three scales: P3 (160×160), P4 (80×80), and P5 (40×40), corresponding to the preprocessed multimodal remote sensing image data. , , size.
[0073] Furthermore, the multi-scale feature map is input to the feature fusion module, and the spatial enhancement module (MANet_FasterBlock) achieves cross-scale and cross-modal feature fusion to generate a multi-scale feature fusion map as follows:
[0074] Step S221: Channel adjustment is performed on the multi-scale feature maps P3, P4, and P5 to obtain feature maps P3', P4', and P5' respectively;
[0075] Perform a 1×1 convolution operation on the multi-scale feature maps P3, P4, and P5 respectively, and adjust the number of channels to 256 to obtain feature maps P3' (160×160, 256 channels), P4' (80×80, 256 channels), and P5' (40×40, 256 channels).
[0076] Step S222: Upsample P4' and concatenate it with P3', then fuse them using the spatial enhancement module to obtain the fused feature map F1;
[0077] The feature map P4' is upsampled using bilinear interpolation to make its size consistent with P3'; then, the upsampled P4' and P3' are concatenated by channels (the number of channels after concatenation is 512); the concatenated features are input into the spatial enhancement module, which performs feature recombination and detail enhancement through channel attention, spatial attention, and multi-scale aggregation functions, and finally outputs a fused feature map F1 (160×160) with 256 channels, which is suitable for small-scale remote sensing target detection.
[0078] Step S223: After downsampling F1, it is concatenated with P4' and then fused by the spatial enhancement module to obtain the fused feature map F2;
[0079] The fused feature F1 is downsampled using a 3×3 convolution with a stride of 2 to make its size consistent with P4'; then the downsampled F1 and P4' are concatenated (the number of channels after concatenation is 512); the concatenated feature is input into the spatial enhancement module for feature integration, and a fused feature map F2 (80×80) with 256 channels is output to adapt to mesoscale remote sensing target detection.
[0080] Step S224: After downsampling F2, it is concatenated with P5' and then fused by the spatial enhancement module to obtain the fused feature map F3.
[0081] The fused feature F2 is downsampled using a 3×3 convolution with a stride of 2 to make its size consistent with P5'; then the downsampled F2 and P5' are concatenated (the number of channels after concatenation is 512); the concatenated feature is input into the spatial enhancement module for feature integration, and a fused feature map F3 (40×40) with 256 channels is output to adapt to large-scale remote sensing target detection.
[0082] Furthermore, the multi-scale feature fusion map is input into the target detection module to generate remote sensing small target detection results specifically as follows: based on the fused feature maps F1, F2, and F3, prediction results corresponding to targets at different scales are generated respectively; wherein, the target detection module adopts a decoupled design including mutually independent classification branches and regression branches, the classification branch outputs the remote sensing small target category probability, and the regression branch outputs the bounding box coordinates.
[0083] Furthermore, the remote sensing small target detection method also includes training a remote sensing small target detection model, specifically:
[0084] (1) Training data configuration: The AI-TOD dataset is used as the training data. After preprocessing, it is divided into training set and validation set;
[0085] (2) Training parameter settings: Adam is used as the optimizer, and the initial learning rate is set to The training batch size is 16, the training epochs are 500, and the training data is trained iteratively.
[0086] (3) Loss function configuration: The multi-task weighted loss function is adopted, and the total loss is the weighted sum of the BCE loss of the classification branch and the DFL (distribution focus loss) and CIoU loss of the regression branch;
[0087] (4) Training evaluation: After each epoch of training, the model accuracy is evaluated using the validation set with mAP@0.5 as the evaluation index. When the validation set accuracy does not improve for three consecutive epochs, training is stopped and the optimal model parameters are saved.
[0088] Step S3: Input the preprocessed multimodal remote sensing image data into the remote sensing small target detection model to obtain the remote sensing small target detection results. Specifically:
[0089] The preprocessed multimodal remote sensing image data is input into the constructed remote sensing small target detection model. Multi-scale features are extracted by the feature extraction module, and then fused by the feature fusion module. Finally, the target detection module outputs two types of results: class probability and bounding box coordinates. Among them:
[0090] Bounding box coordinates: format is Where x1 and y1 are the coordinates of the top left corner, and x2 and y2 are the coordinates of the bottom right corner, with the coordinate values being pixel values relative to the image.
[0091] Category probability: Calculated using the softmax function, outputting the probability that each predicted box belongs to the target, and taking the category corresponding to the maximum probability as the predicted category;
[0092] The final output detection results include: the category name and original coordinates of the bounding box for each target in the image. These can be saved as a JSON file or the bounding box and category label can be drawn directly on the image and then a visualization can be output.
[0093] Furthermore, this embodiment also includes experimental verification and performance comparison. Under the same experimental conditions, the remote sensing small target detection model was compared with the YOLO11 baseline model and other mainstream target detection models. The specific details are as follows:
[0094] (1) Experimental setup
[0095] Experimental dataset: The AI-TOD dataset is used, which contains multimodal remote sensing image data, covering typical features of remote sensing scenes such as small targets and complex backgrounds. The AI-TOD dataset is divided into a training set (train) and a validation set (val).
[0096] Input and training parameters: The input image resolution was uniformly adjusted to 640×640, the training epochs were 500, the optimizer was Adam, and the initial learning rate was... The training batch size is 16, which is exactly the same as the parameters during the model training phase.
[0097] Evaluation metrics: Average Precision (AP) and mean Average Precision (mAP), which are commonly used in the field of object detection, are adopted as evaluation metrics. Among them, mAP@0.5 represents the mAP when the Intersection over Union (IoU) threshold is fixed at 0.5. IoU is defined as the ratio of the overlap area of the predicted box and the ground truth box to the union area. mAP@0.5:0.95 represents the average mAP after multiple evaluations at intervals of 0.05 when the IoU threshold is in the range of 0.5 to 0.95, which more comprehensively reflects the model's ability to detect targets with different degrees of overlap.
[0098] (2) Experimental results and analysis
[0099] Table 1 shows the comparison of the detection performance of each model on the AI-TOD dataset:
[0100] Table 1. Results of the comparative test on detection performance
[0101]
[0102] The experimental results show that:
[0103] Compared to the YOLO11 baseline model, the remote sensing small target detection model in this embodiment improves the mAP@0.5 metric from 44.05% to 48.00%, an improvement of 3.95 percentage points; and the model performance improves from 20.1% to 21.7% at the mAP@0.5:0.95 metric, an improvement of 1.6 percentage points. Compared with other mainstream target detection models (such as SSD-512, CenterNet, etc.), the remote sensing small target detection model in this embodiment also has a significant advantage in detection accuracy.
[0104] (3) Dynamic monitoring of the training process
[0105] like Figure 3As shown, this embodiment dynamically monitors the training process of the remote sensing small target detection model on the AI-TOD dataset, specifically as follows:
[0106] The bounding box regression loss (box_loss), classification loss (cls_loss), and distribution focus loss (dfl_loss) curves of the training set (train) decrease rapidly with the increase of training epochs and tend to stabilize after 200 epochs, indicating that the model's feature learning process for small remote sensing targets is stable and has good convergence. The loss curve of the validation set (val) is highly consistent with the trend of the training set and there is no obvious overfitting fluctuation, indicating that the model has good generalization ability on unseen data and strong robustness to complex remote sensing scenarios.
[0107] During training, the detection precision and recall gradually increased and converged, indicating that the model's ability to identify and locate small remote sensing targets continued to improve. The average precision (mAP@0.5, mAP@0.5:0.95) steadily increased with the number of training rounds and tended to stabilize, eventually reaching 48.0% and 21.7%, respectively.
[0108] Example 2
[0109] like Figure 2 As shown, a remote sensing small target detection system based on multimodal fusion for use in the remote sensing small target detection method described in Embodiment 1, the system comprising:
[0110] The data acquisition module M1 acquires multimodal remote sensing image data and preprocesses it;
[0111] The model building module M2 constructs a remote sensing small target detection model based on YOLO11, including a feature extraction module, a feature fusion module, and a target detection module. The feature extraction module is based on the original YOLO11 backbone network architecture and further introduces a hybrid aggregation network module. The feature fusion module is based on the original YOLO11 neck network architecture and replaces its built-in convolutional fusion layer with a spatial enhancement module.
[0112] The result generation module M3 inputs the preprocessed multimodal remote sensing image data into the remote sensing small target detection model to obtain the remote sensing small target detection results.
[0113] The advantages of this invention are: by introducing a hybrid aggregation network module and replacing the convolutional fusion layer with a spatial enhancement module, the model's ability to represent and locate the features of small remote sensing targets is effectively enhanced, and the detection performance in complex remote sensing scenarios is significantly improved.
[0114] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A remote sensing small target detection method based on multimodal fusion, characterized in that, The method includes: Acquire multimodal remote sensing image data and preprocess it; A remote sensing small target detection model based on YOLO11 is constructed, which includes a feature extraction module, a feature fusion module and a target detection module. The feature extraction module is based on the original YOLO11 backbone network architecture and further introduces a hybrid aggregation network module. The feature fusion module is based on the original YOLO11 neck network architecture and replaces its built-in convolutional fusion layer with a spatial enhancement module. The preprocessed multimodal remote sensing image data is input into the remote sensing small target detection model to obtain the remote sensing small target detection results.
2. The remote sensing small target detection method according to claim 1, characterized in that, The preprocessing of multimodal remote sensing image data specifically includes multimodal data alignment, format conversion, size adjustment, and standardization.
3. The remote sensing small target detection method according to claim 1, characterized in that, In the remote sensing small target detection model: the feature extraction module includes the Conv module, C3k2 module, and SPPF module of the original YOLO11 backbone network architecture, and a hybrid aggregation network module further introduced between the C3k2 module and the SPPF module. The preprocessed multimodal remote sensing image data is input into the feature extraction module, and is processed sequentially by the Conv module, C3k2 module, hybrid aggregation network module, and SPPF module to generate a multi-scale feature map. The multi-scale feature map is input into the feature fusion module, and cross-scale and cross-modal feature fusion is achieved by the spatial enhancement module to generate a multi-scale feature fusion map. The multi-scale feature fusion map is input into the target detection module to generate the remote sensing small target detection result.
4. The remote sensing small target detection method according to claim 3, characterized in that, The preprocessed multimodal remote sensing image data is input into the feature extraction module, and then processed sequentially by the Conv module, C3k2 module, hybrid aggregation network module, and SPPF module to generate multi-scale feature maps as follows: The preprocessed multimodal remote sensing image data is downsampled by the Conv module to generate initial feature maps at five scales: P1, P2, P3, P4, and P5. For the initial feature maps at both scales P2 and P3, the C3k2 module is used for feature enhancement. For the initial feature maps at both P4 and P5 scales, a hybrid aggregation network module is used for feature enhancement. At the end of the feature extraction module, the SPPF module is used to perform multi-scale context modeling and output multi-scale feature maps.
5. The remote sensing small target detection method according to claim 4, characterized in that, The hybrid aggregation network module is used to simultaneously model global semantic context, local spatial texture information, and multi-scale feature dependencies, which can be represented as: in, For input features, C is the number of channels, and H and W are the feature map dimensions; For channel attention; Spatial attention; It is a multi-scale aggregation function; Enhance operations on a channel-by-channel basis.
6. The remote sensing small target detection method according to claim 4, characterized in that, The multi-scale feature map includes three scales: P3, P4, and P5, which correspond to the preprocessed multimodal remote sensing image data, respectively. , , size.
7. The remote sensing small target detection method according to claim 6, characterized in that, The multi-scale feature map is input into the feature fusion module, and the spatial enhancement module achieves cross-scale and cross-modal feature fusion to generate a multi-scale feature fusion map as follows: Channel adjustment is performed on the multi-scale feature maps P3, P4, and P5 to obtain feature maps P3', P4', and P5', respectively. After upsampling P4', it is concatenated with P3' and then fused using the spatial enhancement module to obtain the fused feature map F1; F1 is downsampled and then concatenated with P4', and then fused using the spatial enhancement module to obtain the fused feature map F2; After downsampling F2, it is concatenated with P5' and then fused using the spatial enhancement module to obtain the fused feature map F3.
8. The remote sensing small target detection method according to claim 7, characterized in that, The multi-scale feature fusion map is input into the target detection module to generate remote sensing small target detection results. Specifically, based on the fused feature maps F1, F2, and F3, prediction results corresponding to targets at different scales are generated respectively. The target detection module adopts a decoupled design including mutually independent classification branches and regression branches. The classification branch outputs the probability of remote sensing small target category, and the regression branch outputs the bounding box coordinates.
9. The remote sensing small target detection method according to claim 1, characterized in that, The remote sensing small target detection method further includes training a remote sensing small target detection model, wherein the training parameters are as follows: preprocessed multimodal remote sensing image data is used as training data, and its size is adjusted to 640×640; Adam is used as the optimizer, and the initial learning rate is set to... The training batch size is 16, and the training rounds are 500 epochs.
10. A remote sensing small target detection system based on multimodal fusion for use in any one of the remote sensing small target detection methods according to claims 1 to 9, characterized in that, The system includes: The data acquisition module acquires multimodal remote sensing image data and preprocesses it. The model building module constructs a remote sensing small target detection model based on YOLO11, including a feature extraction module, a feature fusion module, and a target detection module. The feature extraction module is based on the original YOLO11 backbone network architecture and further introduces a hybrid aggregation network module. The feature fusion module is based on the original YOLO11 neck network architecture and replaces its built-in convolutional fusion layer with a spatial enhancement module. The result generation module inputs the preprocessed multimodal remote sensing image data into the remote sensing small target detection model to obtain the remote sensing small target detection results.