Aerial small target image detection method based on multi-modal progressive fusion

Abstract

The application relates to the technical field of computer vision, and discloses a multi-modal progressive fusion aerial small target image detection method, which comprises the following steps: collecting an aerial small target image; inputting the aerial small target image into an improved cross-modal target detection model to output the position, category and confidence of a predicted small target in the aerial small target image; the cross-modal target detection model comprises a backbone network, a neck network and a detection head; the backbone network comprises a visible light branch, an infrared branch, four dynamic enhancement modules and a multi-scale small target cross screening mechanism module which fuses the visible light branch and the infrared branch; and the neck network replaces a C2k3 module in a YOLO11 network with the multi-scale small target cross screening mechanism module. Under the condition of simultaneously inputting bimodal data, the cross-modal target detection model is constructed, bistream features are extracted and fused, and the same-dimension feature enhancement and cross-dimension feature fusion in different stages are realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for detecting small targets in aerial images using multimodal progressive fusion. Background Technology

[0002] Target detection technology, as a cutting-edge field in computer vision, has significant potential in tracking, positioning, emergency rescue, and other areas. With the development of the low-altitude economy, the fusion application of remote sensing aerial imagery and target detection algorithms is rapidly developing. However, due to differences in the formation and source methods of image sensors and data sources, most detection technology networks are designed and constructed based on single-modal imagery (such as visible light images, infrared thermal imaging, etc.). Although they have achieved good results in some scenarios, their ability to extract information under scene migration is limited. Due to the lack of multimodal feature complementarity, target features are severely lost when performing downstream detection tasks. Small targets in aerial small target detection are defined as targets with extremely low pixel proportions and blurred visual features in aerial images or videos (industry-standard threshold: pixel side length <32×32, or accounting for <0.1% of the entire image). These targets appear as only a few pixels or small color blocks in a single frame due to the high viewing angle and long shooting distance of aerial photography, without obvious contour details. Currently, remote sensing aerial images of small targets face many challenges in the context of target detection: (1) The aerial perspective is unique and the environment is relatively complex, which often results in multiple occlusions of the target. Static occlusions are mainly caused by structures such as buildings, bridges, and trees, while dynamic occlusions are caused by moving objects such as vehicles and pedestrians; (2) Extreme lighting conditions such as strong light and shadow, as well as complex weather such as fog, haze, rain, and snow, increase the difficulty of target detection; (3) The targets are numerous and small, and as the fusion layer and detection layer are deepened, edge information features are more easily lost. Therefore, the engineering recognition capability of remote sensing aerial images of small targets still needs to be optimized.

[0003] Visible light images (RGB) excel at capturing color and detail, and are excellent at representing target edge information and detailed characteristics while preserving visual features. However, their performance significantly degrades in low-light, nighttime, and complex weather scenarios. Infrared thermal imaging (IR) is unaffected by illumination and is highly capable of preserving the visible features of targets when sensing temperature differences, but it lacks feature texture, color representation, and various background information, making it more suitable for nighttime surveillance scenarios. In the development of multimodal aerial target detection, features from visible light images and infrared thermal imaging are extracted and then fused. However, this is merely a simple overlay on the feature maps. While this achieves a certain degree of feature complementarity, it fails to fully consider issues such as semantic alignment and weight allocation between modalities, resulting in unstable fusion results in complex scenes. Summary of the Invention

[0004] The purpose of this invention is to construct a cross-modal target detection model under the condition of simultaneous input of dual-modal data, extract and fuse dual-stream features, and realize the same-dimensional feature enhancement and cross-dimensional feature fusion at different stages. To this end, a multimodal progressive fusion aerial small target image detection method is provided.

[0005] To achieve the above-mentioned objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A multimodal progressive fusion method for detecting small targets in aerial images includes the following steps:

[0007] Acquire aerial images of small targets; input the aerial images of small targets into a cross-modal target detection model based on an improved YOLO11 network, and output the predicted location, category, and confidence score of the small targets in the aerial images;

[0008] The cross-modal target detection model includes a backbone network, a neck network, and a detection head. The backbone network includes a visible light branch, an infrared branch, four dynamic enhancement modules, and a multi-scale small target cross-selection mechanism module that fuses the visible light branch and the infrared branch. The neck network replaces the C2k3 module in the YOLO11 network with the multi-scale small target cross-selection mechanism module.

[0009] In the above scheme, the cross-modal target detection model based on the YOLO11 network improves upon the introduction of the Dynamic Enhancement Module (DRFA) and the Multi-Scale Small Target Cross-Filtering Mechanism Module (MS_CSM), which solves the core defects of "coarse fusion operation" and "inefficient feature selection" in simple image fusion detection, thereby achieving a balance between accuracy and efficiency.

[0010] The four dynamic enhancement modules in the backbone network are DRFA-1, DRFA-2, DRFA-3, and DRFA-4.

[0011] The multi-scale small target cross-selection mechanism module is represented by MS_CSM;

[0012] The visible light branch includes IN-1, Multiin-1, Conv-1, Conv-2, C3k2-1, Conv-3, C3k2-2, Conv-4, C3k2-3, Conv-5, C3k2-4, SPPF-1, and C2PSA-1 connected in sequence.

[0013] The infrared branch includes IN-2, Multiin-2, Conv-6, Conv-7, C3k2-5, Conv-8, C3k2-6, MS_CSM-1, Conv-9, C3k2-7, MS_CSM-2, Conv-10, C3k2-8, SPPF-2, and C2PSA-2 connected in sequence.

[0014] The outputs of C3k2-1 and C3k2-5 are connected to the inputs of DRFA-1, and the outputs of DRFA-1 are connected to the inputs of Conv-3 and Conv-8, respectively. The outputs of C3k2-2 and C3k2-6 are connected to the inputs of DRFA-2, and the outputs of DRFA-2 are connected to the inputs of Conv-4 and MS_CSM-1, respectively. The output of C3k2-2 is also connected to the input of MS_CSM-1. The outputs of C3k2-3 and C3k2-5 are connected to the inputs of DRFA-1, respectively. The output of C3k2-3 is connected to the input of DRFA-3. The output of DRFA-3 is connected to the input of Conv-5 and the input of MS_CSM-2. The output of C3k2-3 is also connected to the input of MS_CSM-2. The output of C2PSA-1 is connected to the input of DRFA-4 and the input of MS_CSM-3. The output of C2PSA-2 is connected to the input of DRFA-4 and the input of MS_CSM-3. The output of DRFA-4 is connected to the input of MS_CSM-3.

[0015] In the above scheme, under the condition of simultaneous input of dual-modal data (RGB image and IR image), dual-stream features are extracted and fused to achieve same-dimensional feature enhancement and cross-dimensional feature fusion at different stages. Specifically, the Dynamic Enhancement Module (DRFA) introduced in the backbone network first uses dynamic kernel parameter learning and cross-attention mechanisms to capture global contextual information and long-range semantic dependencies between the visible light and infrared modalities, and performs adaptive dynamic aggregation interaction on cross-modal features, effectively enhancing the expressive power of complementary information between the two modalities in the feature space. Subsequently, the generated shared dynamic features are mapped back to the visible light branch and the infrared branch respectively through a spatial gate mechanism and residual connections, thereby achieving precise complementary enhancement and adaptive modulation of visible light and infrared features, providing robust fusion information with both global vision and anti-interference capabilities for subsequent networks. To make the mission more suitable for aerial photography of small targets, in actual operation, the Dynamic Enhancement Module (DRFA) is progressively embedded into the backbone network in four stages, which respectively act on the scale-level cross-modal fusion of P2 (high resolution shallow features), P3 (medium resolution shallow features), P4 (medium and low resolution medium and deep features) and P5 (high resolution deep features) at the feature pyramid level.

[0016] The processing flow for each of the aforementioned dynamic enhancement modules is as follows:

[0017] The characteristic F of the visible light branch of the input dynamic enhancement module rgb and the characteristic F of the infrared branch ir They are respectively:

[0018] ;

[0019] ;

[0020] in, Let B be the set of real numbers, C be the number of channels, H be the height, and W be the width.

[0021] Generate dynamic convolutional kernel weights W k :

[0022] ;

[0023] Among them, W k The dimension corresponds to the size of the dynamic convolution kernel; Softmax is the Softmax activation function; Conv2 is the second layer of convolution processing; ReLU activation function; Conv1 is the first layer convolution process; AvgPool is the global pooling operation;

[0024] Utilizing a multi-head cross-attention mechanism to extract the feature F of visible light branches rgb As query Q, the feature F of the infrared branch ir As keys K and values ​​V, calculate the similarity between Q and K, and aggregate V using weighted averages. After calculation, rearrange the results back to the spatial dimension [B,C,H,W], specifically:

[0025] ;

[0026] Among them, F att For interactive features; Attention is a multi-head cross-attention mechanism operation;

[0027] Using dynamic convolution kernel weights W k For interaction features F att Feature enhancement is performed, specifically as follows:

[0028] ;

[0029] Among them, F dyn To share dynamic features; This is a dynamic convolution operation;

[0030] Calculate the weights:

[0031] ;

[0032] Among them, G rgb G represents the weight of the visible light branch. ir The weights for the infrared branch; The Sigmoid activation function is used; Concat is the concatenation operation; Conv is the convolution operation.

[0033] Using gated adjustment of shared dynamic features as residuals added back to the original features, specifically:

[0034] ;

[0035] ;

[0036] in, Features of the visible light branch output by the dynamic enhancement module; This refers to the characteristics of the infrared branch output by the dynamic enhancement module.

[0037] In the above scheme, the Dynamic Enhancement Module (DRFA) first utilizes dynamic kernel parameter learning and a multi-head cross-attention mechanism to capture global contextual information and long-range semantic dependencies between visible light and infrared features. It then adaptively aggregates and semantically interacts with cross-modal features, effectively enhancing the expressive power of complementary information between the two modalities in the feature space. Subsequently, the generated shared dynamic features are mapped back to the dual-stream backbone network through spatial gating and residual connections, thereby achieving precise complementary enhancement and adaptive modulation of visible light and infrared features. This provides robust fusion information with both global perspective and anti-interference capabilities for subsequent networks.

[0038] The processing flow of each of the multi-scale small target cross-selection mechanism modules is as follows:

[0039] The image features F input to the multi-scale small target cross-selection mechanism module are divided into mainstream features Fmain. pri and complementary features F comp Calculate the mainstream feature F pri and complementary features F comp Dedicated feature matrix:

[0040] ;

[0041] ;

[0042] Among them, M(F) pri The aggregated features are the main features; n=3,5,9 indicates that the convolution kernel size is n×1; W n×1 This represents a vertical strip convolution with a kernel size of n×1; LN represents layer normalization; Q pri K priV pri Representing the mainstream features F respectively pri Dedicated query matrix, key matrix, and value matrix; It is a 1×1 convolutional linear mapping;

[0043] ;

[0044] ;

[0045] Among them, M(F) comp ) represents the aggregated features of complementary features; n=3,5,9 indicates that the convolution kernel size is 1×n; W 1×n This represents a horizontal bar convolution with a kernel size of 1×n; Q comp K comp V comp F represents complementary features respectively comp Dedicated query matrix, key matrix, and value matrix;

[0046] Calculate the main attention map A1 and the complementary attention map A2:

[0047] ;

[0048] ;

[0049] Where Softmax is the Softmax activation function; d n This is the scaling factor; This is the matrix transpose.

[0050] Calculate the dominant feature subset Z1 and the complementary feature subset Z2:

[0051] ;

[0052] ;

[0053] in, Indicates through the mapping layer Reduce the channel dimension from C to C / 2;

[0054] ;

[0055] Among them, O final This represents the tensor output by the multi-scale small target cross-selection mechanism module.

[0056] In the above scheme, the multi-scale small target cross-screening mechanism module (MS_CSM) acts as a bridge between the backbone network and the neck network. Under the secondary screening and fusion operation, through multi-level precision filtering, it realizes mutual query and calibration of visible light and infrared semantic information, preserves the accurate contour of multimodal targets, outputs clear fusion information, and finally enables the detection head to accurately capture small targets and predict and output the target position, type and confidence level.

[0057] Compared with existing technologies, this invention constructs a cross-modal target detection model with a dual-stream detection architecture, and introduces a four-stage progressive dynamic enhancement (DRFA) and a multi-scale small target cross-screening mechanism module (MS_CSM), which brings the following beneficial effects:

[0058] (1) Significantly improve the detection accuracy of small targets: By progressively embedding the four-stage DRFA module in the multi-scale feature layers (P2, P3, P4, P5), the shallow detail information and deep semantic information of the visible light mode and infrared mode are effectively enhanced, which significantly alleviates the feature loss problem of small targets in deep networks.

[0059] (2) Enhance cross-modal feature complementarity: The DRFA module enhances the expressive power of the feature space between modalities by using mechanisms such as multi-head cross attention, and improves the robustness of the model in complex scenes (such as occlusion and extreme lighting).

[0060] (3) Optimize the efficiency of multi-scale feature fusion: MS-CSM effectively suppresses noise interference by filtering and fusing different semantics, improves the accuracy of simple feature addition fusion for detection tasks, and enhances the expressive power and positioning accuracy of feature pyramid in aerial small target detection tasks.

[0061] (4) Balancing detection accuracy and computational efficiency: Without excessively increasing the computational burden, the model achieves significant improvements in precision, recall and mAP across multiple input sizes. Attached Figure Description

[0062] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 These are schematic diagrams of four existing multimodal target detection network structures. Figure 1 (a) in the diagram is a schematic diagram of the multimodal target detection network structure fused in the early stage. Figure 1(b) in the diagram is a schematic diagram of the multimodal target detection network structure with mid-term fusion. Figure 1 (c) in the diagram is a schematic diagram of the multimodal target detection network structure with mid-to-late stage fusion. Figure 1 (d) in the diagram is a schematic diagram of the multimodal target detection network structure for later fusion;

[0064] Figure 2 This is a schematic diagram of the structure of a traditional YOLO11 network;

[0065] Figure 3 This is a schematic diagram of the cross-modal target detection model of the present invention;

[0066] Figure 4 This is a schematic diagram of the processing flow of the dynamic enhancement module of the present invention;

[0067] Figure 5 This is a schematic diagram of the processing flow of the multi-scale small target cross-screening mechanism module of the present invention. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0069] It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance, or suggesting any such actual relationship or order between these entities or operations. Additionally, the terms "connected," "linked," etc., can refer to a direct connection between elements, components, modules, etc., or an indirect connection via other elements, components, modules, etc.

[0070] Most current target detection algorithms rely primarily on visible light images. However, due to various environmental interferences, such as occlusion, complex weather conditions, and uneven illumination, visible light sensors often struggle to acquire complete target information, thus failing to meet the requirements for accurate detection. To address this, a fusion technique combining visible light images (RGB) and infrared thermal imaging (IR) has been proposed. Infrared thermal imaging relies mainly on the thermal radiation energy emitted by the target and is insensitive to lighting conditions, providing clear target outlines even in low-light environments. By effectively integrating the complementary information of these two modalities, not only can target detection performance be improved, but its application in real-world scenarios can also be expanded. Therefore, constructing target detection techniques based on dual-modal feature fusion has gradually become a research hotspot.

[0071] In the field of dual-modal aerial target detection technology, most studies are based on classic detection frameworks such as the YOLO model or the Faster R-CNN model. These frameworks construct single-backbone networks to extract visible light image features and infrared thermal imaging features separately, and then fuse them at specific levels. For example... Figure 1 The diagram shows four typical existing multimodal target detection network architectures.

[0072] Figure 1 (a) in the diagram represents the initial fusion stage. At the very beginning of the data input, the visible light image (RGB) and infrared thermal image (IR) are stitched together along the channel dimension and then fed into a backbone network for feature extraction. The extracted features are then processed by a neck network and a head network for target localization and classification. This method has a simple structure and relatively low computational cost, but it has the shallowest fusion level, performing fusion only at the pixel level. It cannot handle the huge differences in feature distribution between different modalities. The backbone network with shared weights cannot adapt to the characteristics of two modalities simultaneously, which may lead to insufficient feature extraction and failure to effectively utilize the unique advantages of each modality.

[0073] Figure 1 (b) in the diagram represents intermediate-level fusion. Visible light images (RGB) and infrared thermal images (IR) are input into two independent backbone networks. These two backbone networks can have different structures but do not share parameters, allowing them to specifically learn features of their respective modalities. After complete feature extraction from the two backbone networks, the high-level feature maps output are fused at a specific level. The fusion method can be channel concatenation, element-wise addition, or attention weighting. The fused feature map is then integrated into a neck network and a detection head for target localization and classification. This strategy balances modality-specific feature extraction and feature complementarity, but the fusion location is singular and fixed, typically only fusing deep features. This results in the loss of shallow details crucial for small targets, such as edges and textures, making the fusion strategy often rather coarse.

[0074] Figure 1(c) represents mid-to-late stage fusion. Similar to mid-stage fusion, visible light images (RGB) and infrared thermal images (IR) first pass through two independent backbone networks. Feature map fusion occurs not only at the end of the backbone network output but also extends to the neck network. For example, bimodal feature interaction and fusion are performed at different scale layers of the feature pyramid (FPN / PANet). The features after multi-level fusion are then fed into a detector head for target localization and classification. Compared to mid-stage fusion, mid-to-late stage fusion performs fusion at multiple scales, better combining shallow details and deep semantic information, making it a more advanced four-way approach. Although the fusion layers are deeper, the fusion layers themselves are relatively simple, failing to achieve deep, adaptive cross-modal feature interaction; furthermore, this structure is typically complex to design and computationally expensive.

[0075] Figure 1 In step (d), the fusion process is performed later. Visible light images (RGB) and infrared thermal images (IR) are processed by two independent target detection networks, each consisting of a backbone network, a neck network, and a detection head. Each network outputs its own detection results, including bounding boxes, categories, and confidence scores. In the final stage, these results are fused, using methods such as weighted average coordinates and confidence scores, or non-maximum suppression to select the optimal bounding box. This strategy offers high flexibility, allowing each network to optimize based on its own modal characteristics. However, if one modality fails completely while the other still provides results, the feature complementarity is completely abandoned, and the fusion effect heavily depends on the design of the later fusion strategy. This approach has the highest computational cost and inference time due to the need to run two complete target detection networks. Furthermore, if one modality has a low signal-to-noise ratio, its erroneous results can interfere with the final fusion.

[0076] This invention is achieved through the following technical solution: a multimodal progressive fusion method for detecting small targets in aerial images. Figure 1 Improvements were made to the intermediate fusion approach in (b) by introducing the “DRFA module” and the “MS_CSM module”, which solved the core defects of “coarse fusion operation” and “inefficient feature selection” in simple image fusion detection, thereby achieving a balance between accuracy and efficiency.

[0077] The cross-modal target detection model is an improvement based on the mainstream YOLO11 network, such as... Figure 2 As shown, the mainstream YOLO11 network is a single-branch architecture that supports single-modal data detection, including a backbone network, a neck network, and a head. The backbone network is responsible for feature extraction and uses a series of convolutional and deconvolutional layers, mainly including modules such as SPPF, C2PSA, C3K2, and CBS, adopting a modular and progressive downsampling design.

[0078] The proposed cross-modal target detection model is an improvement upon the mainstream YOLO11 network, such as... Figure 3 As shown, the cross-modal target detection model includes a backbone network, a neck network, and a head. The backbone network comprises a visible light branch, an infrared branch, four dynamic enhancement modules (DRFA, Dynamic Reinforcement Fusion Attention), and a multi-scale small target cross-selection mechanism module (MS_CSM-3) that fuses the visible light and infrared branches. The four dynamic enhancement modules are designated as the first dynamic enhancement module (DRFA-1), the second dynamic enhancement module (DRFA-2), the third dynamic enhancement module (DRFA-3), and the fourth dynamic enhancement module (DRFA-4). Visible light images (RGB) are input to the visible light branch, and infrared thermal imaging (IR) is input to the infrared branch.

[0079] Please continue reading Figure 3The hierarchical connection relationship within the backbone network is as follows: The visible light branch includes IN-1, Multiin-1, Conv-1, Conv-2, C3k2-1, Conv-3, C3k2-2, Conv-4, C3k2-3, Conv-5, C3k2-4, SPPF-1, and C2PSA-1 connected in sequence; the infrared branch includes IN-2, Multiin-2, Conv-6, Conv-7, C3k2-5, Conv-8, C3k2-6, MS_CSM-1, Conv-9, C3k2-7, MS_CSM-2, Conv-10, C3k2-8, SPPF-2, and C2PSA-2 connected in sequence; simultaneously, the output terminals of C3k2-1 and C3k2-5 are respectively connected to the input terminal of DRFA-1, and the output terminal of DRFA-1 is respectively connected to the input terminal of Conv-3, Conv-4, Conv-5, C3k2-4, SPPF-1, and C2PSA-2. The input terminal of 8 is connected; the output terminals of C3k2-2 and C3k2-6 are connected to the input terminal of DRFA-2, respectively; the output terminal of DRFA-2 is connected to the input terminal of Conv-4 and the input terminal of MS_CSM-1, respectively; the output terminal of C3k2-2 is also connected to the input terminal of MS_CSM-1; the output terminals of C3k2-3 and C3k2-7 are connected to the input terminal of DRFA-3, respectively; the output terminal of DRFA-3 is connected to the input terminal of Conv-5 and the input terminal of MS_CSM-2, respectively; the output terminal of C3k2-3 is also connected to the input terminal of MS_CSM-2; the output terminal of C2PSA-1 is connected to the input terminal of DRFA-4 and the input terminal of MS_CSM-3, respectively; the output terminal of C2PSA-2 is connected to the input terminal of DRFA-4 and the input terminal of MS_CSM-3, respectively; the output terminal of DRFA-4 is connected to the input terminal of MS_CSM-3.

[0080] Please continue reading Figure 3The hierarchical connection relationship within the neck network is as follows: The neck network includes MS_CSM-4, MS_CSM-5, Conv-11, MS_CSM-6, Conv-12, and MS_CSM-7; the feature map F3 output by MS_CSM-3 is upsampled and fused with the feature map F2 output by MS_CSM-2 to obtain feature map F4; feature map F4 is processed by MS_CSM-4 to obtain feature map F5; feature map F5 is upsampled and fused with the feature map F2 output by MS_CSM-2. Feature map F1 output by SM-1 is fused to obtain feature map F6. Feature map F6 is processed by MS_CSM-5 to obtain feature map F7. Feature map F7 is processed by Conv-11 and fused with feature map F5 to obtain feature map F8. Feature map F8 is processed by MS_CSM-6 to obtain feature map F9. Feature map F9 is processed by Conv-12 and fused with feature map F3 to obtain feature map F10. Feature map F10 is processed by MS_CSM-7 to obtain feature map F11.

[0081] Please continue reading Figure 3 The hierarchical connection relationship inside the detection head is as follows: The detection head includes Detect-1, Detect-2, and Detect-3. The output terminal of MS_CSM-5 is connected to the input terminal of Detect-1, the output terminal of MS_CSM-6 is connected to the input terminal of Detect-2, and the output terminal of MS_CSM-7 is connected to the input terminal of Detect-3.

[0082] It should be noted that the "-1", "-2", etc. at the end of each layer in the cross-modal target detection model are only used to distinguish layers with the same structure. For example, Conv-1 and Conv-2 belong to convolutional layers with the same structure.

[0083] The C3k2 architecture comprises several layers: IN (Instance Normalization) is a normalization layer that independently calculates the mean and variance for each channel of the RGB and IR images and performs standardization. Multiin is a preprocessing layer that receives the input tensors from the RGB and IR images, segments them by channel, and extracts data from different modalities. Conv (Convolutional Layer) is a convolutional layer that extracts local features from the input and generates feature maps. C3k2 optimizes information flow by segmenting feature maps and applying different convolutional layers, serving as both a flexible feature extraction module and a fast feature fusion module. SPPF (Spatial Pyramid Pooling Fast) is a fast spatial pyramid pooling module used to improve the computational efficiency of convolutional neural networks when processing multi-scale features. C2PSA (C2 Position-Sensitive Attention) is a feature extraction module that introduces a position-sensitive attention mechanism to dynamically adjust the importance of features at different positions, thereby better capturing global and local features.

[0084] This scheme improves upon the traditional YOLO11 network, constructing a novel cross-modal target detection model. Under the condition of simultaneous input of dual-modal data (RGB and IR images), it extracts and fuses dual-stream features, achieving same-dimensional feature enhancement and cross-dimensional feature fusion at different stages. Specifically, the Dynamic Enhancement Module (DRFA) introduced into the backbone network first utilizes dynamic kernel parameter learning and cross-attention mechanisms to capture global contextual information and long-range semantic dependencies between the visible and infrared modalities. It adaptively and dynamically aggregates cross-modal features, effectively enhancing the expressive power of complementary information between the two modalities in the feature space. Subsequently, the generated shared dynamic features are mapped back to the visible and infrared branches through spatial gating and residual connections, respectively. This achieves precise complementary enhancement and adaptive modulation of visible and infrared features, providing robust fusion information with both global perspective and anti-interference capabilities for subsequent networks. To make the mission more suitable for aerial photography of small targets, in actual operation, the Dynamic Enhancement Module (DRFA) is progressively embedded into the backbone network in four stages, which respectively act on the scale-level cross-modal fusion of P2 (high resolution shallow features), P3 (medium resolution shallow features), P4 (medium and low resolution medium and deep features) and P5 (high resolution deep features) at the feature pyramid level.

[0085] The core idea of ​​the Dynamic Enhancement Module (DRFA) is to utilize the multi-head attention mechanism of the Transformer and the features processed by dynamic convolution to generate enhanced residual terms, and then add them back to the original visible light branch and infrared branch respectively through a gating mechanism. The overall flowchart is as follows. Figure 4 As shown, since the four dynamic enhancement modules have the same structure, their input processing flow is also the same. This embodiment takes any one of the dynamic enhancement modules (DRFA) as an example, and its processing flow is as follows:

[0086] (1) Dynamic Branch.

[0087] The characteristic F of visible light branching rgb and the characteristic F of the infrared branch ir They are represented as follows:

[0088] ;

[0089] ;

[0090] Among them, F rgb To input the visible light branch characteristics of the dynamic enhancement module, F ir The infrared branch features are input to the dynamic enhancement module. Since the input visible light and infrared images have already undergone normalization and preprocessing in the earlier IN and Multiin stages, the feature maps transmitted in the same dimension of the visible light and infrared branches have the same scale. Let B be the set of real numbers, C be the number of channels, H be the height, and W be the width.

[0091] To adaptively perceive global context information, the weights of the dynamic convolutional kernel are first generated based on the sum of the inputs from the two streams (visible light branch and infrared branch). This is achieved by adding the features from both branches, and then generating the dynamic convolutional kernel weights W using global average pooling, two convolutional layers (MLP and Conv Block), and a softmax activation function. k Specifically:

[0092] ;

[0093] Among them, W k The dimension corresponds to the size of the dynamic convolution kernel (k×k); Softmax is the Softmax activation function; Conv2 is the second layer of convolution processing; is the ReLU activation function; Conv1 is the first layer convolution processing; AvgPool is the global pooling operation.

[0094] (2) Feature interaction computation (Cross-Attention Branch).

[0095] The feature interaction stage utilizes the transformer's multi-head cross-attention mechanism to capture long-range dependencies between the two modalities. After the multi-head cross-attention mechanism, the features F of the visible light branch are... rgb As a query (Q), the characteristic F of the infrared branch ir As keys (K) and values ​​(V), infrared information is used to enhance the semantic representation of visible light features. The similarity between Q and K is calculated, and V is weighted and aggregated. After calculation, the results are rearranged back to the spatial dimension [B,C,H,W], specifically:

[0096] ;

[0097] Among them, F att For interactive features; Attention is a multi-head cross-attention mechanism operation.

[0098] (3) Dynamic feature aggregation (Dynamic Convolution / Aggregation).

[0099] Using dynamic convolution kernel weights W k For interaction features F att Feature enhancement is performed to complete dynamic feature aggregation. The calculation operation is to combine F... att Expand by sliding window according to kernel size, and W k Einstein summed the values, then folded them back to their original size, specifically:

[0100] ;

[0101] Among them, F dyn To share dynamic features; This is a dynamic convolution operation.

[0102] (4) Gating Branch.

[0103] To control the intensity of dynamic feature injection into different modalities, gating computation is used to transform the original input F... rgb and F ir The weights are concatenated along the channel dimension (Concat), and local spatial information is perceived through convolutional layers (Conv). The sigmoid activation function is then used to map the weights to the (0,1) interval, resulting in two output weights G. rgb and G ir These are used to adjust the information content of the visible light branch and the infrared branch, respectively, as follows:

[0104] ;

[0105] Among them, G rgbG represents the weight of the visible light branch. ir The weights for the infrared branch; is the Sigmoid activation function; Concat is the concatenation operation; Conv is the convolution operation.

[0106] (5) Dual-branch residual enhancement output.

[0107] The output sequence is used to adjust the shared dynamic features through gating, and then added back to the original features as residuals. Specifically:

[0108] ;

[0109] ;

[0110] in, Characteristics of the visible light branch output by the Dynamic Enhancement Module (DRFA); This describes the characteristics of the infrared branch output by the Dynamic Radiation Enhancement Module (DRFA). Figure 3 Taking DRFA-1 as an example, DRFA-1 will generate Output to Conv-3, DRFA-1 will generate Output to Conv-8.

[0111] In the fusion stage of the cross-modal target detection model, the Dynamic Enhancement Module (DRFA) first utilizes dynamic kernel parameter learning and a multi-head cross-attention mechanism to capture global contextual information and long-range semantic dependencies between visible light and infrared features. It adaptively aggregates and semantically interacts with cross-modal features, effectively enhancing the expressive power of complementary information between the two modalities in the feature space. Subsequently, the generated shared dynamic features are mapped back to the dual-stream backbone network through spatial gating and residual connections, achieving precise complementary enhancement and adaptive modulation of visible light and infrared features. This provides robust fusion information with both global perspective and anti-interference capabilities for subsequent networks.

[0112] In neck network architectures, traditional feature pyramids (such as FPN) often employ direct feature aggregation when dealing with cross-modal semantic gaps and complex background noise. However, for small target detection from an aerial perspective, indiscriminate fusion can lead to the semantic information in deep features being diluted by a large amount of background texture in shallow features. In extreme cases, the fusion may introduce noise from the RGB image and blur the boundary information of the IR image, resulting in a decrease in the ability to discriminate small targets. Therefore, ensuring that the features input to the detection head are focused and well-defined is crucial for detecting small targets with extremely low pixel counts. In multimodal image fusion, the IR image has clearer target contour information, while the RGB image has richer target detail information. In the detection process, using complementary features across modalities for fusion and filtering can effectively achieve the above results.

[0113] To address the shortcomings of traditional feature pyramid fusion methods, this scheme introduces a multi-scale small target cross-selection mechanism module (MS_CSM). This module aims to establish a feature fusion mechanism for small aerial targets, eliminating semantic differences between mainstream and complementary feature representations. Semantic alignment and spatial selection are achieved through multi-scale bar convolution. It's important to note that in images, mainstream features refer to the original geometric topology of the target, which can be understood as the target contour; complementary features refer to the image's frequency detail information, i.e., the target detail information. The structure and computational flow of the multi-scale small target cross-selection mechanism module (MS_CSM) are consistent in both the backbone and neck networks, except for the different features represented. Since infrared images have strong contour information and visible light images have strong detail features, the multi-scale small target cross-selection mechanism module (MS_CSM) in the backbone network directly uses infrared features (possessing geometric topology and target contour) as mainstream features and visible light features (possessing image frequency detail and target detail) as complementary features for fusion. In the neck network, the multi-scale small target cross-selection mechanism module (MS_CSM) replaces the traditional C3k2 module. Therefore, the multi-scale small target cross-selection mechanism module (MS_CSM) divides the input feature F into two streams for computation. The mainstream feature is the contour information in F, and the complementary feature is the image frequency detail information in F. The rest of the computation process is the same.

[0114] Since the multi-scale small target cross-selection mechanism modules (MS_CSM) in the cross-modal target detection model all have the same structure, this embodiment takes any one of the multi-scale small target cross-selection mechanism modules (MS_CSM) as an example. Figure 5 As shown, its processing flow can be divided into three stages, as follows:

[0115] (1) Multiseale Mapping.

[0116] First, a multi-scale semantic projection space is constructed, and target features at different levels are captured through orthogonal decomposition convolution operators. The input image features F are divided into main features Fi. pri and complementary features F comp Mainstream feature F pri It carries the main geometric and topological structure of the target, i.e., the target contour information; complementary features F comp It carries image frequency details, i.e., target detail information. In the improvement of the cross-modal target detection model, MS_CSM replaces the Concat module of the backbone network and the C3k2 module of the neck network in the traditional YOLO11 network, respectively.

[0117] To overcome feature diffusion of dense small targets, a set of convolution kernels S={3,5,9} adapted to small targets is introduced in the selection of orthogonal convolution kernels. Among them, K=3 focuses on the micro scale, responds to the extreme points of pixels, and retains the significant features of targets with a small proportion of pixels in the high altitude, such as vehicles and pedestrians; K=5 focuses on matching the physical boundaries of single instances and separating densely arranged targets; K=9 focuses on capturing the group dependence of local areas, while limiting the receptive field to suppress large-scale background noise and reduce the false detection rate.

[0118] Mainstream feature F pri After layer normalization, depthwise separable convolution aggregation in orthogonal directions is performed, specifically vertical strip convolutions (3×1, 5×1, 9×1) to gather texture details. Then, the enhanced multi-scale features are mapped to the mainstream features F through a 1×1 convolutional linear mapping. pri Dedicated feature matrix Q pri K pri V pri , is represented as:

[0119] ;

[0120] ;

[0121] Among them, M(F) pri The aggregated features are the main features; n=3,5,9 indicates that the convolution kernel size is n×1; W n×1 This represents a vertical strip convolution with a kernel size of n×1; LN represents layer normalization; Q pri K pri V pri Representing the mainstream features F respectively pri Dedicated query matrix, key matrix, and value matrix; It is a 1×1 convolutional linear mapping.

[0122] Complementary features F compAfter layer normalization, depthwise separable convolution aggregation in orthogonal directions is performed, specifically horizontal strip convolutions (1×3, 1×5, 1×9) to gather texture details. Then, the enhanced multi-scale features are mapped to complementary features F through a 1×1 convolutional linear mapping. comp Dedicated feature matrix Q comp K comp V comp , is represented as:

[0123] ;

[0124] ;

[0125] Among them, M(F) comp ) represents the aggregated features of complementary features; n=3,5,9 indicates that the convolution kernel size is 1×n; W 1×n This represents a horizontal bar convolution with a kernel size of 1×n; Q comp K comp V comp F represents complementary features respectively comp Dedicated query matrix, key matrix, and value matrix.

[0126] (2) Dual-path cross-semantic calibration.

[0127] Next, a cross-attention mechanism is used to achieve mutual activation and noise suppression of heterogeneous feature flows, including mutual attention graph calculation and feature selection and compression. The cross-attention mechanism is expressed as:

[0128] ;

[0129] ;

[0130] Where A1 is the mainstream attention map, A2 is the complementary attention map; Softmax is the Softmax activation function; d n This is the scaling factor; This is the matrix transpose.

[0131] The value matrix is ​​weighted using both the mainstream attention map and the complementary attention map, and then projected through a projection layer. Reducing the channel dimension from C to C / 2 yields the mainstream feature subset and the complementary feature subset, represented as:

[0132] ;

[0133] ;

[0134] Among them, Z1 is the mainstream feature subset, and Z2 is the complementary feature subset; Indicates through the mapping layer Reduce the channel dimension from C to C / 2.

[0135] (3) Fusion and Output.

[0136] Finally, the two feature subspaces are concatenated and merged. The activated mainstream feature subset Z1 and the selected complementary feature subset Z2 are concatenated along the channel dimension to form the final aligned output tensor, represented as:

[0137]

[0138] Among them, O final This represents the tensor output by the multi-scale small target cross-selection mechanism module (MS_CSM).

[0139] The multi-scale small target cross-screening mechanism module (MS_CSM) acts as a bridge between the backbone network and the neck network. Under the secondary screening and fusion operation, it realizes mutual query and calibration of visible light and infrared semantic information through multi-level precision filtering, preserves the accurate contour of multimodal targets, outputs clear fusion information, and ultimately enables the detection head to accurately capture small targets and predict and output the target position, type and confidence level.

[0140] To systematically evaluate the performance advantages of the proposed cross-modal target detection model, this embodiment conducts comparative experiments on the integrated VEDAI dataset based on the stepwise optimization merging criterion. Each optimization module is introduced sequentially, and qualitative performance analysis is performed on the enhanced detection network. The experiments focus on examining multiple evaluation metrics, including precision, recall, and mAP at different IoU thresholds. Due to the subtle features and extremely small overall proportion of target data in the dataset, a 640×640 input size was used for comparative experiments to further verify the model's performance at different input scales. Detailed experimental results are summarized in Table 1.

[0141] Table 1 Experimental Results

[0142]

[0143] The experimental results in Table 1 show that the model proposed in this patent significantly outperforms the traditional model in terms of precision (P), recall (R), and the highest mean precision, improving by 12.4%, 6.5%, 10.7%, and 8.9%, respectively. This verifies that the improved cross-modal target detection model proposed in this scheme has good multimodal image extraction and fusion detection capabilities.

[0144] Compared with the results of cross-sectional studies, the model proposed in this patent can significantly improve model accuracy and maintain high inference efficiency without increasing the computational burden too much. The simultaneous improvement in precision and recall indicates that the model improves the coverage of the target while reducing false positives. The increase in mAP values ​​under the two thresholds further confirms that the model exhibits stronger robustness under different IoU thresholds. Under larger input conditions, the model improves steadily and has good adaptability.

[0145] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting small targets in aerial images using multimodal progressive fusion, characterized in that, Includes the following steps: Acquire aerial images of small targets; input the aerial images of small targets into a cross-modal target detection model based on an improved YOLO11 network, and output the predicted location, category, and confidence score of the small targets in the aerial images; The cross-modal target detection model includes a backbone network, a neck network, and a detection head. The backbone network includes a visible light branch, an infrared branch, four dynamic enhancement modules, and a multi-scale small target cross-selection mechanism module that fuses the visible light branch and the infrared branch. The neck network replaces the C2k3 module in the YOLO11 network with the multi-scale small target cross-selection mechanism module. The processing flow for each of the aforementioned dynamic enhancement modules is as follows: The characteristic F of the visible light branch of the input dynamic enhancement module rgb and the characteristic F of the infrared branch ir They are respectively: ； ； in, Let B be the set of real numbers, C be the number of channels, H be the height, and W be the width. Generate dynamic convolutional kernel weights W k : ； Among them, W k The dimension corresponds to the size of the dynamic convolution kernel; Softmax is the Softmax activation function; Conv2 is the second layer of convolution processing; ReLU activation function; Conv1 is the first layer convolution process; AvgPool is the global pooling operation; Utilizing a multi-head cross-attention mechanism to extract the feature F of visible light branches rgb As query Q, the feature F of the infrared branch ir As keys K and values ​​V, calculate the similarity between Q and K, and aggregate V using weighted averages. After calculation, rearrange the results back to the spatial dimension [B,C,H,W], specifically: ； Among them, F att For interactive features; Attention is a multi-head cross-attention mechanism operation; Using dynamic convolution kernel weights W k For interaction features F att Feature enhancement is performed, specifically as follows: ； Among them, F dyn To share dynamic features; This is a dynamic convolution operation; Calculate the weights: ； Among them, G rgb G represents the weight of the visible light branch. ir The weights for the infrared branch; The Sigmoid activation function is used; Concat is the concatenation operation; Conv is the convolution operation. Using gated adjustment of shared dynamic features as residuals added back to the original features, specifically: ； ； in, Features of the visible light branch output by the dynamic enhancement module; Features of the infrared branch output by the dynamic enhancement module; The processing flow of each of the multi-scale small target cross-selection mechanism modules is as follows: The image features F input to the multi-scale small target cross-selection mechanism module are divided into mainstream features Fmain. pri and complementary features F comp Calculate the mainstream feature F pri and complementary features F comp Dedicated feature matrix: ； ； Among them, M(F) pri The aggregated features are the main features; n=3,5,9 indicates that the convolution kernel size is n×1; W n×1 This represents a vertical strip convolution with a kernel size of n×1; LN represents layer normalization; Q pri K pri V pri Representing the mainstream features F respectively pri Dedicated query matrix, key matrix, and value matrix; It is a 1×1 convolutional linear mapping; ； ； Among them, M(F) comp ) represents the aggregated features of complementary features; n=3,5,9 indicates that the convolution kernel size is 1×n; W 1×n This represents a horizontal bar convolution with a kernel size of 1×n; Q comp K comp V comp F represents complementary features respectively comp Dedicated query matrix, key matrix, and value matrix; Calculate the main attention map A1 and the complementary attention map A2: ； ； Where Softmax is the Softmax activation function; d n This is the scaling factor; This is the matrix transpose. Calculate the dominant feature subset Z1 and the complementary feature subset Z2: ； ； in, Indicates through the mapping layer Reduce the channel dimension from C to C / 2; ； Among them, O final This represents the tensor output by the multi-scale small target cross-selection mechanism module.

2. The multimodal progressive fusion aerial small target image detection method according to claim 1, characterized in that, The four dynamic enhancement modules in the backbone network are DRFA-1, DRFA-2, DRFA-3, and DRFA-4. The multi-scale small target cross-selection mechanism module is represented by MS_CSM; The visible light branch includes IN-1, Multiin-1, Conv-1, Conv-2, C3k2-1, Conv-3, C3k2-2, Conv-4, C3k2-3, Conv-5, C3k2-4, SPPF-1, and C2PSA-1 connected in sequence. The infrared branch includes IN-2, Multiin-2, Conv-6, Conv-7, C3k2-5, Conv-8, C3k2-6, MS_CSM-1, Conv-9, C3k2-7, MS_CSM-2, Conv-10, C3k2-8, SPPF-2, and C2PSA-2 connected in sequence. The outputs of C3k2-1 and C3k2-5 are connected to the inputs of DRFA-1, and the outputs of DRFA-1 are connected to the inputs of Conv-3 and Conv-8, respectively. The outputs of C3k2-2 and C3k2-6 are connected to the inputs of DRFA-2, and the outputs of DRFA-2 are connected to the inputs of Conv-4 and MS_CSM-1, respectively. The output of C3k2-2 is also connected to the input of MS_CSM-1. The outputs of C3k2-3 and C3k2-5 are connected to the inputs of DRFA-1, respectively. The output of C3k2-3 is connected to the input of DRFA-3. The output of DRFA-3 is connected to the input of Conv-5 and the input of MS_CSM-2. The output of C3k2-3 is also connected to the input of MS_CSM-2. The output of C2PSA-1 is connected to the input of DRFA-4 and the input of MS_CSM-3. The output of C2PSA-2 is connected to the input of DRFA-4 and the input of MS_CSM-3. The output of DRFA-4 is connected to the input of MS_CSM-3.

Images