A multispectral small target detection method based on a multi-level double-flow fusion network

Abstract

The present application relates to a kind of multispectral small target detection methods based on multi-level double-flow fusion network, the method includes: obtaining multispectral image pair, input double-branch extraction network based on improved YOLOv5 to image pair, respectively extract the shallow feature, middle feature and deep feature corresponding to RGB mode and IR mode;Through the light multi-layer fusion module, the shallow, middle, deep features of each mode are processed, and the final shallow, middle, deep fusion features are output;The final shallow fusion feature, middle fusion feature, deep fusion feature is input into feature pyramid network for multiscale integration, and the obtained pyramid fusion feature is input into detection head, and the small target detection result is output.Compared with prior art, the present application has the advantages of hierarchical fusion, focusing on details, accurate and fast multispectral small target detection, etc.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and multimodal image processing technology, and in particular to a multispectral small target detection method based on a multi-level dual-stream fusion network. Background Technology

[0002] Multispectral small target detection, especially the fusion detection of visible light (RGB) and infrared (IR) images, has significant application value in fields such as autonomous driving, intelligent surveillance, nighttime target recognition, and military reconnaissance. In the complex and ever-changing conditions of the real world, single-modal visual information often falls short of the demands for high-precision detection. Visible light images provide rich texture, color, and edge information, which is beneficial for accurate target localization, but their performance degrades sharply in low-light, severely occluded, or nighttime environments. Conversely, infrared images capture the thermal radiation of objects, making them highly adaptable to low-visibility and low-light scenes, but they typically have lower resolution and lack fine-grained texture, making it difficult to accurately locate small targets in cluttered backgrounds. Therefore, how to efficiently fuse information from both RGB and IR modalities to achieve complementary feature representation has become a core challenge in multispectral target detection research.

[0003] Existing multispectral image fusion strategies are mainly classified into three categories: pixel-level, feature-level, and decision-level fusion. Pixel-level fusion methods (such as weighted averaging, principal component analysis (PCA), and multi-scale wavelet transform) are simple and easy to implement, but they only perform simple combinations at the pixel level, making it difficult to capture complex semantic and spatial relationships between modalities, resulting in insufficient detail preservation and poor semantic consistency in complex scenes. Decision-level fusion is more efficient, but it cannot utilize fine-grained cross-modal complementary information, limiting performance improvement. Feature-level fusion achieves a good balance between performance and efficiency and has become the mainstream method. Early dual-stream convolutional neural network (CNN) architectures could independently extract dual-modal features and fuse them in the mid-to-late stages, but their ability to extract and align shallow edge and texture features was limited, leading to unclear boundaries of small targets and limited detection accuracy. Deep global modeling often introduces a heavy computational burden, increasing the false detection rate in complex backgrounds.

[0004] Following the success of the Transformer architecture in vision tasks, cross-modal Transformers have been introduced into the field of multispectral fusion, achieving excellent semantic alignment and global context modeling capabilities. However, its enormous computational cost limits its direct deployment in lightweight, real-time scenarios. Furthermore, the pure Transformer architecture has limited ability to perceive shallow details, which may lead to blurred boundaries of small objects.

[0005] In recent years, hierarchical fusion strategies have gained attention, aiming to focus on spatial details and edges at the shallow level to enhance local perception of small targets, emphasize cross-modal semantic alignment at the mid-level, and integrate global context at the deep level. However, most existing methods use a uniform fusion module across different levels, ignoring the differences in feature distribution, modal complementarity, and receptive fields at different levels. This may lead to redundant computation or information loss, thus affecting the final performance and efficiency.

[0006] In summary, existing technologies face three core challenges in multispectral small target detection: (1) how to design an efficient multi-level fusion architecture that can fully explore and utilize low-level texture details, mid-level semantic information and high-level global context, while avoiding redundant computation and excessive inference costs; (2) how to effectively suppress false detections caused by background clutter, low resolution and bright IR regions in complex environments, and ensure the clarity of small target boundaries; and (3) how to improve detection accuracy while strictly controlling the complexity and computational load of the model to meet the real-time requirements of practical applications.

[0007] A recent proposal for a cross-modality feature fusion scheme based on the Transformer architecture (Cross-Modality Fusion Transformer for Multispectral Object Detection, CFT) introduces a global attention mechanism between the visible light (RGB) and infrared (IR) modes to achieve alignment and complementarity of cross-modal features. This represents the current state-of-the-art level of global feature fusion using Transformers in the field of multispectral object detection. It includes the CFT cross-modality fusion backbone network and the cross-modality fusion Transformer, such as... Figure 1 As shown. Although the CFT scheme achieves excellent performance through its powerful Transformer model, it suffers from the following inherent problems and drawbacks, which limit its application in real-world scenarios, especially in resource-constrained environments: 1) extremely high computational complexity and low inference efficiency; 2) coarse-grained fusion strategy, lacking hierarchy and specificity; 3) insufficient perception of shallow details and local structures, making it susceptible to interference, the root cause of which lies in the difficulty of a unified global attention mechanism to focus on local, detailed key areas. These drawbacks collectively lead to existing technologies sacrificing model efficiency and practicality in the pursuit of high accuracy, failing to adequately meet the comprehensive requirements of accuracy, speed, and resource consumption in industrial applications.

[0008] A search revealed Chinese invention patent application publication number CN119091122B, which discloses a multi-spectral feature fusion multi-scale target detection method based on a general attention mechanism. This method utilizes a backbone network to read visible light and infrared images, followed by parallel dual-stream network feature extraction. A GFT feature fusion network based on Transformer, incorporating surrogate attention, skip attention structures, and cross attention, is constructed to fuse features extracted by the dual-stream network at different stages. The neck network incorporates BiFPN, VoVGSCSP, and GSConv modules, using the fused features from different stages as input for feature enhancement. The enhanced features from each stage are then processed by the RepConv module to capture diverse features. Finally, the detection head and output layer yield the final target detection result. This existing patent application suffers from difficulty in focusing on localized, detailed key regions, resulting in low detection accuracy.

[0009] Achieving both accuracy and efficiency in multispectral small target detection has become a technical problem that needs to be solved. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a multispectral small target detection method based on a multi-level dual-stream fusion network.

[0011] The objective of this invention can be achieved through the following technical solutions: According to one aspect of the present invention, a multispectral small target detection method based on a multi-level dual-stream fusion network is provided, the method comprising: Multi-layer feature extraction: Obtain multispectral image pairs composed of RGB and IR images, input the image pairs into a dual-branch extraction network based on improved YOLOv5, and extract shallow, mid-level and deep features corresponding to RGB and IR modes respectively; Multi-layer feature fusion processing: The shallow fusion module processes the shallow features of each modality and outputs the final shallow fused features; the mid-layer fusion module processes the mid-layer features and shallow fused features of each modality and outputs the final mid-layer fused features; the deep fusion module processes the deep features and mid-layer fused features of each modality and outputs the final deep fused features. Multi-scale integration and detection: The final shallow fusion features, medium fusion features, and deep fusion features are input into the feature pyramid network for multi-scale integration. The resulting pyramid fusion features are then input into the detection head to output small target detection results.

[0012] As a preferred technical solution, the shallow fusion module sequentially performs edge extraction, block-level edge-guided attention enhancement, and gating modulation to output shallow fusion features of each modality, and integrates the two across modalities into the final shallow fusion features.

[0013] As a preferred technical solution, the block-level edge-guided attention enhancement process includes: dividing the input shallow feature map and the corresponding edge map into multiple non-overlapping local blocks, corresponding to feature blocks and edge blocks respectively; A query vector is generated for each feature block, and a key vector and a value vector are generated for each edge block. After linear mapping, the attention weights within the block are calculated to achieve edge-guided feature enhancement. After block reconstruction, an enhanced shallow feature map is output.

[0014] As a preferred technical solution, the edge map is obtained by performing Sobel edge extraction on the multispectral image pair to obtain the RGB edge map and the IR edge map.

[0015] As a preferred technical solution, the mid-layer fusion module sequentially performs channel dimensionality reduction, spatial location encoding injection, bidirectional cross-modal attention interaction, channel gating screening, and residual fusion to output the mid-layer fusion features of each modality, and integrates the two cross-modal features into the final mid-layer fusion features.

[0016] As a preferred technical solution, the bidirectional cross-modal attention interaction specifically involves mapping the mid-level features of RGB and IR modalities to the sequence space, and calculating the bidirectional cross-modal enhancement features of IR and RGB through a multi-head attention mechanism to achieve mid-level semantic interaction between modalities, wherein IR provides thermally significant cues for RGB; and RGB provides texture and structural supplements for IR.

[0017] As a preferred technical solution, the channel gating screening is characterized by: introducing the mid-layer features of the RGB and IR modes of the bidirectional cross-modal attention output into the channel gating module respectively, and generating channel attention weights through global average pooling, channel compression and Sigmoid activation to achieve the screening and modulation of each modal feature in order to suppress modal noise.

[0018] As a preferred technical solution, the deep fusion module sequentially performs channel compression, CBAM attention enhancement, bidirectional cross-modal Transformer global interaction, channel dimensionality enhancement, and residual fusion to output deep fusion features of each modality, and integrates the two cross-modal features into the final deep fusion feature.

[0019] As a preferred technical solution, the process of bidirectional cross-modal Transformer global interaction includes: inputting the deep features enhanced by CBAM attention into the bidirectional cross-modal Transformer module, and realizing bidirectional global semantic interaction between RGB and IR through a dual-branch design, specifically: The first branch takes the CBAM enhanced features of the RGB modality as the main input and introduces the CBAM enhanced features of the IR modality as auxiliary interaction features. Through the self-attention mechanism of the Transformer module, the semantic association between the RGB and IR modalities is established globally, and finally the deep features of the RGB modality after global cross-modal interaction are output. The second branch takes the CBAM-enhanced features of the IR modality as the main input and introduces the CBAM-enhanced features of the RGB modality as auxiliary interaction features. It adopts the same Transformer processing logic as the first branch to establish the global semantic dependency between the IR and RGB modalities and finally outputs the deep features of the IR modality after global cross-modal interaction.

[0020] As a preferred technical solution, the loss function used to train the feature pyramid network includes classification error, bounding box bias, and modality consistency.

[0021] Compared with the prior art, the present invention has the following beneficial effects: 1) This invention extracts and fuses shallow, medium and deep features of multispectral image pairs in a layered manner, and combines multi-scale integration of feature pyramids. This not only captures shallow details, medium semantics and deep global features of small targets in a layered manner, but also makes full use of the complementary information of RGB and IR modes, effectively improving the accuracy of small target detection in complex scenes. At the same time, the parallel feature extraction and layered processing flow optimizes the logical link of feature calculation, reduces redundant calculations, and significantly improves the overall efficiency of small target detection.

[0022] 2) The shallow fusion module of this invention accurately captures the edge details of small targets through Sobel edge extraction (solving the problem of blurred edges of small targets and easy missed detection). Combined with block-level edge-guided attention enhancement block division, attention calculation and block reconstruction operations, it not only focuses on feature enhancement of small target regions to improve their edge detection accuracy, but also avoids the high complexity of full-image attention calculation, reducing the computational cost of shallow feature processing. While strengthening the distinguishability of small target edge features, it also takes into account detection efficiency.

[0023] 3) The mid-layer fusion module of this invention simplifies the feature dimension through channel dimensionality reduction and introduces two-dimensional sinusoidal position coding to supplement spatial position information, enabling the mid-layer attention to have a stronger spatial structure perception ability while maintaining low computational complexity; the lightweight bidirectional cross-modal attention interaction realizes the complementarity of mid-layer semantics between RGB and IR modes (such as the thermal display cues of IR assisting the texture features of RGB to identify small targets). Compared with the unidirectional or asymmetric interaction of CFT, it realizes more sufficient modal information exchange and semantic alignment, and improves the semantic matching accuracy of small targets.

[0024] 4) The deep fusion module of this invention reduces the computational complexity of deep feature processing through channel compression, thereby improving detection efficiency; CBAM attention enhancement can accurately highlight the key regions of small targets, improving the feature response accuracy of small targets; the global interaction design of bidirectional cross-modal Transformer can capture the long-range dependencies of sparse, distant small targets, significantly improving the detection accuracy of such small targets in complex scenes. At the same time, this global interaction is based on the enhanced feature expansion, avoiding redundant calculations, and balancing the improvement in accuracy with the loss of detection efficiency while improving the global perception capability of small targets. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the structure of a CFT cross-modal feature fusion network in the prior art; Figure 2 This is a schematic diagram of the structure of the multi-level dual-stream fusion network in this invention; Figure 3 This is a schematic diagram of the shallow fusion module in this invention; Figure 4 This is a schematic diagram of the structure of the middle-layer fusion module in this invention; Figure 5 This is a schematic diagram of the deep fusion module in this invention; Figure 6 This is a schematic diagram of the multispectral small target detection method in this invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0027] In view of the three major drawbacks of the aforementioned background technology, namely high computational complexity, lack of hierarchical fusion strategy, and insufficient suppression of shallow details and noise, the present invention aims to provide a novel multispectral small target detection method to solve the above technical problems.

[0028] This embodiment relates to a multispectral small target detection method based on a multi-level dual-stream fusion network (MLDSFusion). This method utilizes a finer-grained shallow, medium, and deep hierarchical fusion strategy to achieve efficient, hierarchical, and robust cross-modal feature integration. Without significantly increasing computational overhead, it improves the accuracy, feature alignment capability, and robustness in complex scenarios for multispectral small target detection, thereby addressing the performance bottlenecks faced by existing technologies in practical applications and enhancing their usability and deployment friendliness in scenarios such as autonomous driving and intelligent monitoring.

[0029] like Figure 6 The method includes the following steps: Step 1: Multimodal Feature Extraction Visible light (RGB) and infrared (IR) images are input into a dual-branch extraction network, with both branches using a YOLOv5 backbone network that has the same structure but independent parameters. Each backbone layer of the YOLOv5 backbone network outputs a feature map with the same resolution, denoted as shallow features. Mid-layer features and deep features .

[0030] Step 2: Shallow Fusion For local detail features in shallow layers, a fusion mechanism based on block-level edge-guided attention and gated inhibition is adopted, such as... Figure 3 As shown: Sobel edge extraction was performed on both RGB and IR features to obtain edge maps. This is achieved through the EEM module.

[0031] Subsequently, the feature map is divided into local blocks using the edge map, and edge-guided attention is calculated within the block-level features to enhance the target edge, which is achieved through the EGAM module.

[0032] Meanwhile, gating mechanisms are introduced for RGB and IR features respectively to filter and modulate their respective features, suppressing irrelevant or over-response regions while retaining effective information, thereby improving the discriminative power and stability of multimodal fusion features. This is achieved through the GFM module.

[0033] Finally, residual fusion is used to combine the gated features with the original shallow features to output shallow fused features for each modality. This integrates the two across modalities into a final shallow fusion feature. .

[0034] This invention simultaneously enhances detail perception and improves the suppression of irrelevant responses in shallow fusion. Specifically, compared to CFT, which relies solely on global attention in shallow layers and struggles to fully capture the local structural features of small targets, this invention introduces a block-level edge-guided attention module (such as...). Figure 3 This allows the model to more keenly focus on the edges and fine-grained texture information of the target. Simultaneously, by setting an adaptive gating module to filter and weight RGB and IR features, it strengthens effective target information while suppressing irrelevant responses caused by redundant backgrounds and modal differences, thereby significantly improving the discriminative power and stability of shallow fusion features.

[0035] The processing steps of the block-level edge-guided attention-guided module include: 1) Feature block partitioning: Divide the input feature map into multiple local blocks of a fixed size (e.g., 2×2 or 4×4), and process each block separately.

[0036] 2) Edge block synchronous partitioning: Divide the corresponding edge map (from EEM) according to the same block size and position, so that each feature block corresponds to an edge block.

[0037] 3) Intra-block feature extraction: Generate query (Q), key (K) and value (V) features for each feature block, which are used for attention calculation in subsequent blocks.

[0038] 4) Use intra-block edge information for attention modulation: Modulate the K feature of the block by using the pixel values ​​in the corresponding edge block, so that the attention calculation within the block is more focused on the salient edge area.

[0039] 5) Intra-block attention calculation and feature update: Calculate attention weights based on Q, K, and V within each block, and obtain the updated block-level edge-guided enhancement features.

[0040] 6) Block reconstruction: All updated edge-guided enhancement feature blocks are reassembled according to their original positions to restore the enhancement feature map with the same size as the input.

[0041] 7) Output Enhanced Features: The module finally outputs the feature map after block-level processing, which is used for subsequent shallow fusion operations.

[0042] The block-level edge-guided attention module combines local block partitioning with edge guidance, enabling shallow features to precisely perceive target edge information within small blocks. Compared to traditional global attention, this module restricts attention computation to local blocks, significantly reducing computational complexity. Simultaneously, it focuses on significant edge regions through edge block modulation, effectively suppressing irrelevant background and noise. After block-level attention computation, the complete feature map is reconstructed and fused with the original feature residuals, ensuring preservation of detailed information and training stability. This design not only enhances the edge perception capability of small targets and improves the discriminative power and stability of shallow features but also balances lightweight design and computational efficiency, making it particularly suitable for embedded and real-time multispectral small target detection scenarios.

[0043] Step 3: Middle Layer Blending A lightweight cross-modal fusion module is designed to address semantic alignment and modal interaction of mid-level features. The structure of the mid-level fusion module is as follows: Figure 4 As shown.

[0044] The Conv↓ module (dimensionality reduction convolution) performs channel dimensionality reduction on two-modal features; To compensate for the insufficient sensitivity of cross-modal attention to spatial location information, this invention is based on a two-dimensional sinusoidal positional encoding function, which injects spatial positional encoding into the dimensionality-reduced RGB and IR features respectively, so that the network considers spatial structure information simultaneously during the attention calculation process, thereby improving the consistency of small target localization and modal alignment capability; the spatial positional encoding is implemented through two-dimensional positional encoding (PE).

[0045] After introducing positional encoding, RGB and IR features are mapped to the sequence space, and cross-modal enhanced features of RGB←IR and IR←RGB are calculated respectively through a lightweight multi-head attention mechanism to achieve mid-level semantic interaction between modalities: IR provides hot saliency cues for RGB; RGB provides texture and structure supplements for IR; this mechanism effectively improves cross-modal consistency at the mid-level semantic level.

[0046] After obtaining the interaction features through cross-modal attention calculation, they are restored from sequence form to the original spatial dimension (B×C×H×W), and then channel-up convolution is performed through the Conv↑ module to restore the original number of channels, so that the attention-enhanced features regain complete semantic expression and provide a consistent feature dimension for subsequent modal fusion. Lightweight channel gating (GFM) modules are introduced into the RGB and IR features after bidirectional cross-modal attention output. Channel attention weights are generated through global average pooling, channel compression and sigmoid activation to filter and modulate the features of each modality, thereby suppressing modal noise (such as the spurious response of the bright background in IR and the low-light blurring area in RGB) and enhancing the expression of the effective region. Finally, the gated modal features are residually fused with the original input features, and the feature distribution is stabilized by grouping and normalizing to obtain the mid-level fused features. , This integrates the two into the final mid-layer fusion feature. This ensures that the fusion process enhances modal complementarity while maintaining training stability and semantic continuity. This functionality is achieved through... Figure 4 The Add&normalize module is used for implementation.

[0047] Compared to CFT, which relies solely on a global Transformer for cross-modal interaction in the middle layer without specifically optimizing feature dimensions, spatial structure, and computational overhead, the lightweight middle-layer fusion module proposed in this invention makes key improvements in three aspects: First, it significantly reduces the computational cost of attention through channel dimensionality reduction and introduces two-dimensional sinusoidal positional encoding to supplement spatial positional information, enabling the middle-layer attention to have stronger spatial structure perception capabilities while maintaining low computational complexity; Second, this invention uses cross-modal multi-head attention to calculate the bidirectional semantic associations of RGB→IR and IR→RGB respectively, achieving more sufficient modal information exchange and semantic alignment compared to the unidirectional or asymmetric interaction of CFT; Finally, it restores semantic expression through channel dimensionality enhancement after attention interaction and introduces independent RGB and IR gating modulation mechanisms to effectively filter and emphasize key semantic regions, suppress redundant or conflicting information, and retain residuals to ensure feature stability and training convergence, thereby significantly improving the robustness and cross-modal consistency of the middle-layer fusion.

[0048] Step 4: Deep Fusion To address the global semantic modeling and cross-modal information integration of deep features, this invention proposes a hybrid global fusion mechanism, the structure and process of which are as follows: First, the Conv↓ module (dimensionality reduction convolution) performs channel compression on deep RGB and IR features respectively, reducing the original high-dimensional features to a smaller number of intermediate channels, thereby reducing the computational cost of global interaction; Subsequently, a CBAM (Convolutional Block Attention Module) is introduced onto the compressed features to significantly enhance the channel and spatial dimensions, thereby highlighting the target-related regions and suppressing background interference. Then, after attention enhancement, the two-branch cross-modal Transformer module (Cross-ModalTransformer module) is used to perform cross-modal semantic modeling of RGB→IR and IR→RGB respectively, so that the two modalities can achieve global semantic alignment and information complementarity at a deep level. Finally, the original number of channels is restored through channel-level upscaling convolution (Conv↑ module), and then fused with the original deep features using a residual method to obtain the deep fused features. This integrates the two into the final deep fusion feature. .

[0049] Compared to CFT, which directly uses a single Transformer to globally model high-dimensional features at deep layers without combining local saliency and channel filtering, the hybrid global fusion module proposed in this invention makes key improvements in three aspects: First, it reduces the number of deep feature channels through convolutional compression, thereby reducing the computational cost of global interactions. A CBAM module is introduced on the compressed features to enhance the saliency of channels and spatial dimensions, enabling deep attention to strengthen the response of small target regions while maintaining global semantic modeling capabilities. Second, this invention uses a bidirectional cross-modal Transformer module to calculate the global interactions between RGB→IR and IR→RGB respectively. Compared to the unidirectional or asymmetric modeling of CFT, this achieves sufficient semantic alignment and complementary information fusion of the two modalities at deep layers. Finally, after global interactions, the original feature dimensions are restored through channel dimensionality upscaling, and a residual fusion strategy is combined to preserve the high-level semantic structure of the backbone, thereby enhancing the stability, robustness, and semantic continuity of small targets in the deep fused features.

[0050] Step 5: Multi-scale feature integration and pyramid output The final shallow, middle, and deep layer fusion features The input is fed into a feature pyramid network for multi-scale integration, achieving hierarchical fusion of high and low-level features. This preserves information about small targets (from shallow layers) and large targets (from deep layers) at different scales, and outputs the pyramid fused features. The data is then transmitted to the detection head module for target classification and localization prediction.

[0051] Step 6: Detection Head Output and Loss Calculation The pyramid fusion feature is input into the detection head to complete bounding box regression and class prediction. The loss function comprehensively considers classification error, bounding box bias, and modality consistency constraints to achieve end-to-end training.

[0052] This embodiment also relates to a multispectral small target detection method based on a multi-level dual-stream fusion network, combined with... Figures 2-5 The method will be explained in detail.

[0053] Multi-level dual-stream fusion network (MLDSFusion) includes a dual-branch feature extraction network and a three-level hierarchical fusion module, such as... Figure 2 As shown, the three-level hierarchical fusion module consists of a shallow fusion network, a mid-level fusion network, and a deep fusion network. The input visible light image (RGB) and infrared image (IR) are respectively processed by the backbone network of the dual-branch feature extraction network for feature extraction, yielding shallow features of the visible light image (RGB) and infrared image (IR), respectively. Mid-layer features and deep features The superscripts (1), (2), and (3) represent the shallow, middle, and deep layers, respectively.

[0054] 1. Shallow Fusion: In the shallow layer stage, to fully utilize the local structural information contained in RGB and IR images while suppressing the high-brightness noise commonly found in infrared images, this method first performs Sobel operations on the shallow features of the two modalities respectively to extract their edge maps. The obtained edge maps can highlight the contours and detailed changes of the target, providing structural priors for subsequent local attention calculations.

[0055] Based on this, the feature map is divided into several local blocks of a fixed size, and the edge map is also divided into blocks in the same way, so that each feature block corresponds to a local edge region. Within each block, the local features are transformed into a sequence form through an unfolding operation, and a query (Q), key (K), and value (V) are constructed respectively. The key and value come directly from the corresponding edge block, which makes the attention modeling process explicitly guided by the local structure. Subsequently, the attention input is obtained through linear mapping, and edge-based attention weights are further calculated within each block, and these weights are used to enhance the local features. All blocks are then concatenated back into the original space after enhancement to form an enhanced feature map with edge saliency.

[0056] After structural enhancement, shallow fusion further introduces a gated modulation mechanism to generate gate coefficients for the shallow features of RGB and IR respectively. These coefficients extract local statistical information through convolution and are normalized by Sigmoid to form pixel-level adjustment weights ranging from 0 to 1. The gate coefficients are element-wise multiplied with the original features of the corresponding modality to achieve adaptive control of the response intensity at each location, thereby suppressing invalid textures in RGB and highlight interference caused by heat sources in IR.

[0057] Finally, the shallow fusion module integrates edge enhancement features with gated RGB and IR features to output shallow fusion features with clearer structure, lower noise, and greater sensitivity to targets, laying a more stable feature foundation for cross-modal modeling in subsequent middle and deep stages.

[0058] 1-1, Sobel edge extraction is implemented through the EEM (Edge Extraction Module) shown in Figure 3. Specifically, the Sobel edge extraction operator is applied to the shallow feature maps of the RGB and IR modes respectively, and edge response calculations are performed to obtain the corresponding edge feature maps (referred to as edge maps). The calculation process is as follows: (1) in, and These represent the edge feature maps of the RGB and IR modes, respectively. This represents the shallow feature mapping of the corresponding modality. This indicates the Sobel edge extraction operation.

[0059] 1-2, Block-level edge guidance of attention via Figure 3 The EGAM module (Edge-Guided Attention Module) shown is implemented. This module first divides the input features into several local blocks, and then uses the corresponding edge feature maps to guide the feature interaction process within the blocks, thereby enhancing the feature representation ability of structurally related regions within a local scope.

[0060] 1-2-1, Local block expansion: Input feature map Edge graph from another modality ,according to The fixed size is divided into several non-overlapping local blocks. An Unfold operation is performed on each local block, unfolding the features within the block into a serialized representation based on their spatial location, and constructing query, key, and value matrices respectively, as shown below: (2) in, This represents the input feature mapping of the current modality. This represents an edge feature map from another modality. , and These are the query, key, and value feature sequences obtained from the expansion of local blocks, respectively. The Unfold operation represents the spatial size of a local block and is used to unfold each local block from the 2D feature map into a sequence based on its spatial location, so that attention computation can be performed within the block.

[0061] 1-2-2, perform linear mapping on the query, key, and value respectively to obtain the attention input: (3) in, , , The linear projection weight matrix is ​​learnable. , , These are the projected query matrix, key matrix, and value matrix, respectively.

[0062] 1-2-3, Intra-block attention calculation: A weight matrix is ​​generated by scaling the dot product attention, and then multiplied by the value matrix to obtain the enhanced feature sequence: (4) Where A is the attention weight matrix, d Let T be the feature dimension and T be the transpose operation. Softmax This is a normalized activation function to ensure weight normalization.

[0063] 1-2-4, Intra-block Enhancement Feature Reconstruction: The enhanced feature sequence obtained by intra-block attention calculation is rearranged according to its corresponding spatial position, and then restored to a two-dimensional feature map with the same spatial size as the original input feature map through the Fold operation, thus obtaining the edge-guided enhanced features, which are represented as follows: (5) in, This represents the feature map after attention enhancement guided by block-level edges and spatial reconstruction. This represents the enhanced feature sequence obtained by attention weighting within the block. Fold The operation is used to reconstruct the intra-block features of the serialized representation into a two-dimensional feature map according to the original block partitioning method.

[0064] 1-3, Gated modulation is implemented through the GFM module (Gated Fusion Module). To balance the original modal information and cross-modal supplementary information in the shallow fusion stage, a gated fusion module (GFM) is introduced to adaptively modulate and reconstruct the original shallow features and the cross-modal enhancement features generated by block-level edge-guided attention. Specifically, let... and These represent the original RGB and IR shallow features extracted by the backbone network, respectively. and This represents the cross-modal enhancement feature obtained after block-level edge-guided attention processing. The GFM module then performs the following operations on this feature: 1-3-1, the original shallow features and their corresponding cross-modal enhancement features are concatenated along the channel dimension. Local joint statistical information is extracted through a convolutional layer and normalized using a sigmoid activation function to generate a spatially correlated gating coefficient matrix. (6) in, This represents the Sigmoid activation function. For learnable convolution weights, This represents the convolution operation. This indicates a channel splicing operation. This is the gating coefficient matrix for the corresponding modality, used to characterize the dependence of different spatial locations on the original features and cross-modal enhancement information.

[0065] 1-3-2, Gated Modulation Features: Based on the learned gating coefficient matrix, the original shallow features and cross-modal enhancement features are weighted and reconstructed to generate edge enhancement features of the corresponding modality.

[0066] (7) in, This represents element-wise multiplication. This is a gated edge enhancement feature. This gating mechanism enables the network to preferentially retain the original modal information in regions with clear structure and reliable semantics, while adaptively introducing cross-modal supplementary features in regions with weak texture, occlusion, or strong interference. This improves the stability and discriminative ability of shallow features while suppressing redundant responses.

[0067] In this way, the gating mechanism applies to both RGB and IR features simultaneously, preserving effective information while suppressing irrelevant regions or over-response, thereby improving the discriminative power and stability of shallow fusion features.

[0068] 1-4, Dual-branch output of the shallow fusion module: The shallow module does not output a single fused feature. Instead, it integrates the edge enhancement feature with the gated modulation feature, and outputs the enhanced RGB and IR features respectively for subsequent mid-layer cross-modal fusion. (8) in, This represents the final shallow-enhanced output features of the RGB modality. This represents the final shallow enhancement output characteristics of the IR mode. Represents edge enhancement features of the RGB modality. This represents the edge enhancement features of the IR mode.

[0069] Shallow outputs not only provide more reliable low-level structural information but also significantly reduce the risk of false detections caused by IR highlighting regions. Since the feature quality at this stage is passed as input to the mid-level fusion module, the effective feature reconstruction and noise suppression of the shallow layer can decisively affect the robustness and discriminative power of the mid-level cross-modal attention, laying a higher quality foundation for subsequent semantic-level cross-modal fusion.

[0070] 2. Mid-level Fusion: In the mid-level stage, the main objective of the model shifts from shallow local detail enhancement to cross-modal semantic alignment and information interaction. To this end, this method designs a lightweight cross-modal fusion structure, which follows the processing flow of channel dimensionality reduction, bidirectional cross-modal attention, channel gating, and residual fusion, thereby improving semantic interaction capabilities while maintaining low computational overhead.

[0071] First, convolutional processing is applied to the mid-level features of RGB and IR respectively to compress channels, mapping high-dimensional features to a more compact representation space. The dimensionality-reduced features significantly reduce the computational cost of subsequent attention modules while preserving key semantic information. Then, a cross-modal multi-head attention mechanism is used to achieve bidirectional semantic alignment: using RGB features as queries and IR features as keys and values, complementary information extracted from IR by RGB is calculated; simultaneously, using IR as a query and RGB as keys and values, supplementary information obtained from RGB by IR is calculated. Through this bidirectional interaction mechanism, the two modalities can establish a consistent semantic understanding for the same spatial location, enabling the texture structure of RGB and the thermal radiation characteristics of IR to complement each other semantically.

[0072] After cross-modal attention updates, the fused features are fed into a channel-gated module. This module predicts channel-level weights using lightweight convolutions and employs sigmoid activation, allowing the response intensity of each channel to adaptively adjust according to its importance. The gated RGB and IR features are then reweighted along the channel dimension, highlighting semantically effective features, suppressing redundant or noisy features, and further improving the discriminative power of the fused features.

[0073] Finally, the gated features are upscaled to restore them to their original channel scale, and the residuals are added to the original RGB mid-level features. Through this design, the model introduces cross-modal enhancement information while maintaining stable inheritance of the original semantic structure, making the output mid-level fused features both cross-modal consistent and possessing good trainability and stability.

[0074] 2-1, Channel Dimensionality Reduction , (9) in, and These are the mid-level features of RGB and IR after dimensionality reduction, respectively; and These are the channel-reduced convolution weights for the RGB and IR modes, respectively.

[0075] 2-2, Bidirectional Cross-Modal Multi-Head Attention Using RGB as the query and IR as the key and value: , (10) in, For cross-modal attention output in the RGB→IR direction, For the RGB modal query matrix, Let T be the transpose of the bond matrix of the IR modes. The feature dimension of the key vector. This is the value matrix for the IR modes.

[0076] Using IR as the query term and RGB as the key and value: , (11) in, This represents the cross-modal attention output in the IR→RGB direction. For the query matrix of IR modes, Here is the bond matrix for the IR modes, and T denotes transpose. The feature dimension of the key vector. This is the value matrix for the RGB modes.

[0077] 2-3, Dual-modal interaction fusion , (12) in, This represents the cross-modal fusion feature of RGB modes, which is the output of RGB→IR attention and contains complementary information of IR modes. The superscript cm indicates cross-modal. It is an IR modal cross-modal fusion feature, which is the output of IR→RGB attention and contains complementary information of RGB modalities.

[0078] 2-4, Channel Gating Screening , (13) in, This represents the Sigmoid activation function. These are the convolution weights for channel gating.

[0079] Enhanced features and : , (14) in, and These are the final features of the RGB and IR modes after gating enhancement, respectively.

[0080] 2-5, Channel Upgrading and Residual Output (15) in, and These are the mid-level fusion features of the output RGB and IR modes, respectively. and These are the channel-upgrading convolution weights for the RGB and IR modes, respectively.

[0081] The mid-level fusion module achieves deep interaction between RGB and IR features through bidirectional cross-modal multi-head attention. Channel gating further filters key channel information, and channel dimensionality enhancement and residual output ensure feature scale and modal integrity. This stage not only enhances the expressive power of complementary features between modalities but also suppresses the impact of redundant or interfering channels on target detection. Since the mid-level output serves as the input to the deep global fusion module, the effective cross-modal interaction and discriminative feature construction in the mid-level module decisively improve the robustness of deep semantic-level fusion and target recognition accuracy, providing more reliable semantic and structural information for the final small target detection.

[0082] 3. Deep Fusion: The deep fusion stage aims to integrate global semantic information and enhance long-distance dependencies and regional saliency across modalities. Therefore, a hybrid global fusion structure of "convolutional dimensionality reduction - CBAM attention enhancement - Transformer global interaction - channel dimensionality increase and residual output" is adopted. The results of the deep fusion module are as follows: Figure 5 As shown, the specific process includes: First, convolutional dimensionality reduction is performed on the deep RGB and IR features respectively, mapping the high-dimensional semantic features to a more compact channel space. This process effectively reduces the computational cost of subsequent Transformers while preserving deep semantic representation, laying a solid input foundation for global modeling.

[0083] After dimensionality reduction, the fusion module enhances feature saliency using CBAM. This module first generates channel attention weights using global average pooling and global max pooling, adaptively highlighting key semantic channels; then, it constructs spatial attention through 7×7 convolutions, guiding the model to focus on more important spatial regions. Through the cascading effect of channel and spatial attention, feature representations enhanced in both semantic and spatial dimensions are obtained, making deep features more sensitive to target regions.

[0084] Subsequently, the cross-modal features enhanced by CBAM are input into the Transformer module. With the help of the self-attention mechanism, the Transformer can model the long-range dependency between RGB and IR on a global scale and jointly encode cross-modal semantics, thereby improving the model's global perception ability of distant, sparse, small targets in complex scenes.

[0085] After completing the global interaction, the Transformer's output is restored to its original channel dimension through convolution or linear mapping to ensure consistency with the deep features of the backbone. Finally, the dimensionality-upgraded fused features are added to the original deep RGB features through residual connections, preserving the original semantic structure while introducing global enhancement information, thereby obtaining a more stable and globally discriminative deep fused feature.

[0086] 3-1, Convolution Dimensionality Reduction Convolutional dimensionality reduction is performed on the deep RGB and IR features respectively, making the high-dimensional semantic features more compact.

[0087] (16) in, and These are the dimensionality-reduced deep RGB and IR features, respectively. and These are the original deep RGB features and IR features, respectively. and These are the convolutional dimensionality reduction weights for each modality.

[0088] This process can effectively reduce the computational cost of subsequent Transformers while preserving deep semantic representations, laying a solid input foundation for global modeling.

[0089] 3-2, CBAM Attention Enhancement Channel attention and spatial attention are calculated sequentially on the dimensionality-reduced features to enhance the salient regions of the target: 3-2-1, Channel Attention: (17) in, For the deep features after dimensionality reduction of the input The generated channel-level weight matrix, MLP(⋅) represents the multilayer perceptron, and GAP and GMP represent global average pooling and global max pooling, respectively.

[0090] 3-2-2, Spatial Attention: (18) in, Represents the deep features after dimensionality reduction of the input. The generated spatial weight matrix, and These represent the spatial average feature and spatial maximum feature of a single channel, respectively. This represents a 7×7 convolution operation.

[0091] 3-2-3, CBAM Output: (19) in, This is a deep feature enhanced by CBAM.

[0092] By cascading channel and spatial attention, feature representations enhanced in both semantic and spatial dimensions are obtained, making deep features more sensitive to the target region.

[0093] 3-3, Bidirectional Cross-Modal Transformer Global Interaction The features enhanced by CBAM attention are input into the Transformer module to model cross-modal global dependencies:

[0094] (20) in, and These represent the deep features of the RGB and IR modes after global interaction with the Transformer, respectively. and These are the deep features of the RGB and IR modes after CBAM enhancement, respectively, with subscripts... tr This indicates the output of the Transformer.

[0095] This operation enables joint encoding of long-range dependencies of RGB and IR modes globally, thereby improving the model's ability to perceive distant, sparse, small targets in complex scenes.

[0096] 3-4, Dimensional Upgrading and Residual Output 3-4-1, Recover the original channel dimensions through convolution or linear mapping: , (twenty one) in, and These represent the deep features of the RGB and IR modes after dimensionality-upgrading convolution, respectively. and These are the up-dimensional convolution weights for the RGB and IR modes, respectively.

[0097] 3-4-2, Add residual join: , (twenty two) in, and These are the deep fusion features of the final output RGB and IR modes, respectively.

[0098] By using residual connections, the semantic structure of the original deep modalities is preserved while global enhancement information is introduced, thereby obtaining more stable deep fusion features with global discriminative power.

[0099] 3-5. Shallow, medium, and deep fused features are integrated at multiple scales through a Feature Pyramid Network (FPN) to form pyramid fused features. : , (twenty three) in, The fusion outputs from the shallow, medium, and deep fusion modules correspond to the final shallow, medium, and deep fusion features, respectively.

[0100] The pyramid fusion feature retains information about small and large targets at different scales, which is then passed to the detection head for target classification and bounding box regression. The loss function comprehensively considers classification error. Boundary box deviation and modal consistency constraints To achieve end-to-end training: , (twenty four) in, This is the modal consistency weight.

[0101] The model in this invention is implemented in PyTorch, trained using a GTX 3080ti graphics card with a batch size of 4, an initial learning rate of 0.001, and an Adam optimizer. Data augmentation includes random pruning, horizontal flipping, color normalization, and multi-scale scaling. Experiments were conducted to validate the model on four public datasets (FLIR, VEDAI, KAIST, and M3FD).

[0102] The performance comparison results of different multimodal methods on mAP50, mAP75, and mAP50:95 metrics (as shown in Table 1) demonstrate that the proposed multi-level dual-stream fusion method (MLDSF) achieves optimal detection performance on four multispectral datasets: FLIR, VEDAI, KAIST, and M3FD. Specifically, the mAP50 metric, a key indicator of detection accuracy, reaches 79.0%, 71.2%, 78.5%, and 89.0%, respectively, representing a performance improvement of 1.3%–5.3% compared to mainstream CFT methods. Particularly noteworthy is the 5.3% improvement in mAP50 on the VEDAI small target dataset, fully validating the significant advantages of this invention in small target boundary perception and adaptability to complex scenes. These results indicate that the proposed multi-level fusion strategy effectively suppresses false detections in infrared bright areas and enhances the discriminative ability of modal complementary features, thereby achieving more stable and robust detection performance.

[0103] Table 1

[0104] Furthermore, while maintaining leading detection accuracy, this invention also demonstrates a significant advantage in model complexity. As shown in Table 2, the MLDSFusion model has 75.94M parameters and a model size of 151.06MB, representing reductions of 63.14% and 61.70% respectively compared to the CFT method; its FLOPs are 203.72G, only 7.13% higher than the lightweight Two-Stream baseline, and far lower than CFT's 224.58G. These results demonstrate that this invention, through its innovative multi-level fusion design, significantly optimizes model complexity while maintaining high accuracy, combining lightweight efficiency with cross-modal fusion capabilities, providing excellent performance for engineering deployment and embedded applications.

[0105] Table 2

[0106] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multispectral small target detection method based on a multi-level dual-stream fusion network, characterized in that, The method includes: Multi-layer feature extraction: Obtain multispectral image pairs composed of RGB and IR images, input the image pairs into a dual-branch extraction network based on improved YOLOv5, and extract shallow, mid-level and deep features corresponding to RGB and IR modes respectively; Multi-layer feature fusion processing: The shallow fusion module processes the shallow features of each modality and outputs the final shallow fused features; the mid-layer fusion module processes the mid-layer features and shallow fused features of each modality and outputs the final mid-layer fused features; the deep fusion module processes the deep features and mid-layer fused features of each modality and outputs the final deep fused features. Multi-scale integration and detection: The final shallow fusion features, medium fusion features, and deep fusion features are input into the feature pyramid network for multi-scale integration. The resulting pyramid fusion features are then input into the detection head to output small target detection results.

2. The multispectral small target detection method based on a multi-level dual-stream fusion network according to claim 1, characterized in that, The shallow fusion module sequentially extracts edge features, enhances attention by block-level edge guidance, and modulates gating to output shallow fusion features for each modality, and integrates the two across modalities into the final shallow fusion features.

3. The multispectral small target detection method based on a multi-level dual-stream fusion network according to claim 2, characterized in that, The block-level edge-guided attention enhancement process includes: dividing the input shallow feature map and corresponding edge map into multiple non-overlapping local blocks, corresponding to feature blocks and edge blocks respectively; A query vector is generated for each feature block, and a key vector and a value vector are generated for each edge block. After linear mapping, the attention weights within the block are calculated to achieve edge-guided feature enhancement. After block reconstruction, an enhanced shallow feature map is output.

4. The multispectral small target detection method based on a multi-level dual-stream fusion network according to claim 3, characterized in that, The edge map is obtained by performing Sobel edge extraction on the multispectral image pair to obtain the RGB edge map and the IR edge map.

5. The multispectral small target detection method based on a multi-level dual-stream fusion network according to claim 1, characterized in that, The mid-level fusion module sequentially performs channel dimensionality reduction, spatial location encoding injection, bidirectional cross-modal attention interaction, channel gating filtering, and residual fusion to output mid-level fusion features for each modality, and integrates the two cross-modal features into the final mid-level fusion feature.

6. The multispectral small target detection method based on a multi-level dual-stream fusion network according to claim 5, characterized in that, The bidirectional cross-modal attention interaction specifically involves mapping the mid-level features of RGB and IR modalities to the sequence space, and calculating bidirectional cross-modal enhancement features of IR and RGB through a multi-head attention mechanism to achieve mid-level semantic interaction between modalities, wherein IR provides hot salient cues for RGB. RGB provides texture and structure complement to IR.

7. The multispectral small target detection method based on a multi-level dual-stream fusion network according to claim 5, characterized in that, The channel gating screening specifically involves introducing the mid-level features of the RGB and IR modes of the bidirectional cross-modal attention output into the channel gating module, generating channel attention weights through global average pooling, channel compression, and Sigmoid activation, thereby achieving the screening and modulation of each modal feature to suppress modal noise.

8. The multispectral small target detection method based on a multi-level dual-stream fusion network according to claim 1, characterized in that, The deep fusion module sequentially performs channel compression, CBAM attention enhancement, bidirectional cross-modal Transformer global interaction, channel dimensionality upscaling, and residual fusion to output deep fusion features for each modality, and integrates the two across modalities into the final deep fusion feature.

9. The multispectral small target detection method based on a multi-level dual-stream fusion network according to claim 8, characterized in that, The process of bidirectional cross-modal Transformer global interaction includes: inputting the deep features enhanced by CBAM attention into the bidirectional cross-modal Transformer module, and realizing bidirectional global semantic interaction between RGB and IR through a dual-branch design, specifically: The first branch takes the CBAM-enhanced features of the RGB modality as the main input and introduces the CBAM-enhanced features of the IR modality as auxiliary interaction features. Through the self-attention mechanism of the Transformer module, the semantic association between the RGB and IR modalities is established globally, and finally the deep features of the RGB modality after global cross-modal interaction are output. The second branch takes the CBAM-enhanced features of the IR modality as the main input and introduces the CBAM-enhanced features of the RGB modality as auxiliary interaction features. It adopts the same Transformer processing logic as the first branch to establish the global semantic dependency between the IR and RGB modalities and finally outputs the deep features of the IR modality after global cross-modal interaction.

10. The multispectral small target detection method based on a multi-level dual-stream fusion network according to claim 1, characterized in that, The loss function used to train the feature pyramid network includes classification error, bounding box bias, and modality consistency.

Images