Rgb-ir target detection method based on multi-modal feature fusion and frequency domain attention enhancement

Abstract

The application discloses an RGB-IR target detection method based on multi-modal feature fusion and frequency domain attention enhancement, adopts a double-flow YOLO main network to extract visible light and infrared multi-scale features; introduces a feature gate fusion module at each scale, combines modal difference guidance, and generates dynamic fusion weights through space-channel double gating to realize adaptive feature fusion; adds a multi-modal DCT frequency cross attention module to the minimum scale feature, converts the feature to the frequency domain through DCT transformation, uses multi-head cross attention to mine complementary information of the dual modal frequency domain, inversely transforms back to the spatial domain to complete feature enhancement; and finally outputs the detection result through the FPN and the detection head. The application can adaptively cope with the modal quality fluctuation caused by light changes, considers the detection accuracy and reasoning overhead, can be transferred to various double-flow detection frameworks, and is suitable for target detection tasks in complex environments.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an RGB-IR target detection method based on multimodal feature fusion and frequency domain attention enhancement. Background Technology

[0002] Object detection is one of the core tasks in computer vision, widely used in scenarios such as autonomous driving, intelligent security, and drone inspection. In complex environments such as low light, inclement weather, and target occlusion, the object detection performance of a single visible light (RGB) image deteriorates significantly. RGB images are rich in texture and color information but are sensitive to changes in lighting; infrared (IR) images, based on thermal radiation imaging, are unaffected by ambient light and can effectively detect low-contrast targets, but lack texture details. Therefore, fusing RGB and IR dual-modal information to achieve complementary advantages has become an effective way to improve object detection performance in complex scenes.

[0003] Currently, RGBIR multimodal object detection methods can be mainly divided into two categories: early fusion and mid-stage fusion. Early fusion concatenates RGB and IR images at the input layer before feeding them into a single-stream network. This method ignores the feature differences between modalities, resulting in limited fusion performance. Mid-stage fusion introduces cross-modal interaction during the feature extraction stage. A typical example is attention-based fusion methods, such as the Channel Attention Network (SENet), which generates channel weights through global pooling, or the Spatial Attention Module (CBMA), which generates spatial weights to perform weighted fusion of bimodal features. In addition, some works use a dual-stream network to extract RGB and IR features separately, and then fuse them through element-wise addition or concatenation followed by convolution.

[0004] Currently, relevant technologies exist. For example, Chinese patent application CN116630608A discloses a multimodal target detection method for complex scenes. It constructs a dual-input dual-stream backbone network based on the YOLOv5 model to acquire multi-scale features of infrared and visible light images. Then, based on attention mechanisms and intermediate fusion strategies, a cross-modal feature fusion module is built to fuse multimodal features and solve the modality balance problem. Afterwards, captured, registered, and segmented training set images are input into the constructed detection network for training to obtain the optimal weight file. Finally, the weight file and test set images are loaded into the model to obtain infrared and visible light image pairs with detection boxes and confidence scores. Chinese patent application CN117710899A discloses a multispectral pedestrian detection method based on differential attention and frequency domain fusion. Based on the VGG-16 dual-stream feature extraction network, it captures features of both modalities separately and constructs a cross-modal feature fusion module based on differential attention enhancement and frequency domain fusion. This fully fuses cross-modal features, and the fused feature set is fed into an SSD-based detection head to output pedestrian target location information. Chinese patent application CN120279004A discloses a multimodal remote sensing small target detection method across the space-frequency domain. It designs a dual-modal feature-level fusion target detection architecture, extracts features from each channel of visible light and infrared in a distributed manner, and proposes a cross-domain gated self-attention fusion module to perform interaction and fusion of dual-modal features. Finally, the multi-scale features are fused and input into the detection head to obtain multimodal target detection results.

[0005] However, the aforementioned existing technologies still have the following shortcomings:

[0006] First, there are significant differences in features between modalities. RGB and IR images have inherent differences in data distribution and feature representation. Existing methods, which use fixed fusion weights or simple channel / spatial attention, struggle to adaptively handle the dynamic quality differences between modalities. For example, in scenarios where RGB dominates during the day and IR dominates at night, fixed-weight fusion can introduce noise or lose key information from the dominant modality.

[0007] Second, there is a lack of adaptive mechanisms for modal quality fluctuations. In real-world scenarios, the quality of each modal image is dynamically affected by environmental factors. Existing methods fail to adjust the fusion strategy according to the real-time quality of the modalities, resulting in insufficient discriminative power of the fusion features.

[0008] Third, existing methods are limited to spatial domain feature fusion. Most methods perform attention weighting or feature concatenation in the spatial domain, failing to fully exploit the structural complementarity between RGB and IR in the frequency domain. Specifically, low-frequency components carry global structural information, while high-frequency components encode edge texture details; however, existing techniques do not utilize frequency domain information to enhance the fused features. Summary of the Invention

[0009] Purpose of the invention: The purpose of this invention is to address the shortcomings of existing technologies. This invention proposes an RGBIR target detection method based on multimodal feature fusion and frequency domain attention enhancement. It achieves adaptive weighted fusion through a feature-gated fusion module and introduces a DCT frequency cross-attention module to mine modal complementary information from the frequency domain dimension, which significantly improves the detection performance in complex environments.

[0010] Technical solution: The present invention provides an RGB-IR target detection method based on multimodal feature fusion and frequency domain attention enhancement, comprising the following steps:

[0011] Step 1: Obtain paired RGB and IR images to construct bimodal input data;

[0012] Step 2: Construct a two-stream YOLO backbone network and extract RGB three-scale feature maps respectively. and IR three-scale feature map The dual-stream YOLO backbone network includes parallel RGB feature extraction branches and IR feature extraction branches. Both branches use the same network structure (e.g., the CSPDarkNet network structure), perform independent forward propagation, and output feature maps at three scales respectively. This refers to the feature layer level. , The corresponding feature map spatial dimensions are respectively ;

[0013] Step 3: Convert the RGB three-scale feature maps and IR three-scale feature map The features are input into the feature-gated fusion module GFFM and subjected to adaptive gated weighted fusion to obtain preliminary fused features at the corresponding scales. ;

[0014] Step 4: Initially fuse the minimum-scale features obtained in Step 3. (size The input is a multimodal DCT frequency cross-attention module (MDFCA), which performs cross-modal attention enhancement in the frequency domain to obtain enhanced fusion features. ;

[0015] Step 5: Initial fusion features , and enhanced fusion features The input is fed into the FPN detection network, where multi-scale feature fusion is performed, and then the target detection bounding box is output by the detection head, along with its position, category, and confidence level.

[0016] This invention provides comprehensive enhancement across both channel and spatial dimensions. The Feature Gated Fusion (GFFM) module operates in the channel dimension, filtering key semantic channels through channel gating and highlighting significantly different regions by combining modal differences to enhance the semantic discriminative power of the fused features. The Multimodal DCT Frequency Cross-Attention (MDFCA) module operates in the spatial dimension, integrating global contextual information into each spatial location through frequency domain cross-attention, strengthening the response of the target region and effectively suppressing background noise. The two modules have clear division of labor and complement each other.

[0017] Furthermore, the feature-gated fusion module GFFM processes RGB feature maps of the same scale. and IR feature map Perform adaptive weighted fusion to output preliminary fused features. C represents the number of channels in the feature map, and H and W represent the rows and columns of the feature map, respectively; similar feature gating fusion modules exist in the prior art.

[0018] The Feature-Gated Fusion Module (GFFM) comprises a modal difference extraction submodule, a spatial gating submodule, a channel gating submodule, a gating interaction and adaptive selection submodule, and a dynamic residual connection submodule. GFFM achieves adaptive fusion between modalities by coordinating spatial gating, channel gating, and modal difference guidance through these three mechanisms. The specific steps are as follows:

[0019] Step 3.1: The modal difference extraction submodule explicitly calculates the difference information between the RGB mode and the IR mode to guide the subsequent generation of gating weights; firstly, it calculates the pixel-by-pixel absolute difference map. :

[0020] ;

[0021] Then, the difference diagram Channel-level global average pooling is performed, and channel difference weights are generated through a two-layer multilayer perceptron (MLP). :

[0022] ;

[0023] Simultaneously, compute spatial difference significance maps. :right Take the L2 norm along the channel dimension, and then normalize using Sigmoid:

[0024] ;

[0025] ;

[0026] Disparity diagram The number of channels, The difference diagrams are shown separately. Width and height, Disparity diagram In the Specific pixel values ​​on each channel; This refers to a spatial difference amplitude map;

[0027] Broadcast the channel difference weights and spatial difference saliency maps separately, and multiply them with the difference map to obtain the difference guidance map. : ;

[0028] This refers to element-wise multiplication; the difference guide diagram In subsequent fusion, it is used to enhance or suppress regions of significant difference;

[0029] Step 3.2: The spatial gating submodule generates fusion weights in spatial location; firstly, the RGB and IR features are concatenated along the channels to obtain... Then through two layers convolution (First layer output channel) Second layer output channel ), and then through the Sigmoid function Activate to obtain the spatial gating map :

[0030] ;

[0031] This space gating diagram The range of values ​​is This indicates the importance of each spatial location; a larger value means that the features of that location are more worth preserving. The fusion features after splicing This refers to the channel-dimensional splicing operation;

[0032] Step 3.3: The channel gating submodule generates fusion weights along the channel dimension; that is, it performs global average pooling on the RGB and IR features respectively. The two pooling vectors are added together, and then channel weights are generated through a two-layer MLP. :

[0033] ;

[0034] Channel weight vector Expanded into a channel-gated map with the same spatial size as the feature map, where each spatial location shares the same channel weights;

[0035] Step 3.4: The gating interaction and adaptive selection submodule merges the spatial gating map and the channel gating map;

[0036] First, the spatial gating graph is broadcast to the same dimension as the channel gating graph, and then multiplied element-wise with the channel gating graph to obtain the joint gating graph. : ;

[0037] in Indicates broadcast multiplication;

[0038] Then, the joint gating graph With difference guidance map By adding them together, we obtain the preliminary gating fusion features. : ;

[0039] This operation allows the fusion process to simultaneously consider both the joint importance of space and channel and the significance of modal differences;

[0040] Step 3.5: To adaptively preserve the original splicing features, the dynamic residual connection submodule introduces dynamic residual coefficients. Dynamic residual coefficient By analyzing the difference diagram After global average pooling, the result is obtained through a two-layer MLP: ;

[0041] Ultimately, the initial gating fusion features will be... pass The convolution operation is performed and concatenated with the original spliced ​​feature residuals to obtain preliminary fused features:

[0042] ;

[0043] This refers to normalizing the convolution result; This refers to the ReLU activation operation;

[0044] The aforementioned feature-gated fusion module can adaptively adjust the fusion weights in space and channels based on the dynamic quality differences between RGB and IR modes, effectively suppressing noise and highlighting complementary information. The resulting preliminary gated fusion features can adaptively reweight modes, adjusting the fusion intensity according to the real-time quality differences between RGB and IR modes, effectively suppressing low-quality mode noise and overcoming the shortcomings of fixed-weight fusion. Furthermore, through coordinated selection of spatial and channel gating, it balances target saliency localization and semantic channel filtering, improving the discriminative power of the fusion features in terms of spatial accuracy and semantic representation.

[0045] Existing technologies use Fourier transform and frequency decomposition-guided methods for multimodal frequency cross-enhancement. However, the standard Fourier transform cannot adaptively adjust the weights of different frequency components based on feature content, limiting the flexibility of fusion. Furthermore, global modeling at full image resolution incurs significant overhead. Frequency decomposition-guided methods ignore mid-frequency information between low and high frequencies. Mid-frequency components typically carry crucial structural cues such as target contours and edge transitions, playing a vital role in target recognition and localization. Categorizing them as either low or high frequencies leads to loss of structural information and noise amplification. To address this, the Multimodal DCT Frequency Cross-Attention Module (MDFCA) achieves cross-modal information enhancement in the frequency domain through frequency domain transformation, pseudo-modal branch replication, frequency domain cross-attention, and inverse transformation. The specific processing steps are as follows:

[0046] Step 4.1: Frequency domain transformation, i.e., preliminary fusion of features at the smallest scale. Perform a two-dimensional discrete cosine transform (2D-DCT) to obtain the frequency domain feature representation. : ;

[0047] Two-dimensional DCT decomposes spatial domain signals into the sum of cosine basis functions of different frequencies. Low-frequency coefficients correspond to the global structure of the image, while high-frequency coefficients correspond to edge and texture details.

[0048] Step 4.2, Modal branch replication: To achieve interaction between the RGB and IR modes in the frequency domain, the frequency domain features are copied. The copy is split into two independent branches, denoted as follows: and Both have the same initial value;

[0049] Subsequently, cross-attention is used to differentiate the two branches, thereby extracting complementary information;

[0050] Step 4.3: Frequency domain cross-attention, using the frequency domain features of the two branches as the query, key, and value, respectively; first, flatten the spatial dimensions of the feature map into a sequence form: and The shape from Transform into ;

[0051] Then, the query is obtained through linear transformation. ,key Sum ;

[0052] ;

[0053] in, It is a learnable linear transformation matrix. This refers to the flattening operation;

[0054] Then a multi-head attention mechanism is adopted (number of heads) ): ;

[0055] in Dimensions for each head; This refers to attention operations. This refers to the normalization function;

[0056] The output of multi-head attention, after linear transformation, is restored to... The shape is used to obtain the frequency domain cross-attention output. ;

[0057] ;

[0058] in, ;

[0059] This refers to the reshaping and restoration operation. This refers to the output of the i-th attention head. This refers to the projection weight matrix of the query at the i-th head. This refers to the key projection weight matrix of the i-th head. The value of the i-th head is projected onto the weight matrix;

[0060] Step 4.4: Inverse transform and residual connection, ... By mapping back to the spatial domain using the two-dimensional inverse discrete cosine transform (2D-IDCT), the corresponding feature map is obtained. :

[0061] ;

[0062] Finally, the features Connection with original feature F residual and Convolutional output enhancement features : ;

[0063] The MDFCA module uses DCT transform to map fused features to the frequency domain and achieves information interaction between the two modal branches through cross attention, which can effectively enhance the structural complementarity in the frequency domain, especially the balanced utilization of low-frequency global information and high-frequency detail information.

[0064] This invention employs a multi-scale feature pyramid structure. GFFM modules are set up for fusion at the P3, P4, and P5 scales of the YOLO dual-stream backbone network; the P5 scale (… The fused features are further enhanced in the frequency domain by the MDFCA module; additionally, the features at three scales (P3 fused features, P4 fused features, and P5 enhanced features) are input into the YOLO FPN neckline for top-down and bottom-up multi-scale feature fusion, and finally the detection head outputs the prediction result. This invention adopts an end-to-end training method, and the loss function is... Including target classification loss Bounding box regression loss and focus loss Loss function The expression is as follows:

[0065] ;

[0066] Each loss weight coefficient can be set according to the actual training situation, for example , , .

[0067] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0068] 1. Strong adaptive fusion capability, effectively addressing modal quality fluctuations.

[0069] The Feature Gated Fusion Module (GFFM) designed in this invention dynamically generates fusion weights through the collaboration of spatial gating, channel gating, and modal difference guidance. It can adaptively adjust the fusion strategy based on the real-time quality differences between RGB and IR modes under different environments. For example, during the day when RGB is dominant, channel gating automatically enhances the contribution of RGB features; at night when IR is dominant, spatial gating highlights infrared thermal target regions. Compared to existing methods using fixed weights or simple attention mechanisms, this invention significantly improves the discriminative power and robustness of the fused features under complex lighting conditions.

[0070] 2. For the first time, frequency domain cross-attention is introduced to fully explore modal complementarity information.

[0071] The proposed Multimodal DCT Frequency Cross-Attention Module (MDFCA) maps fused features to the frequency domain and achieves structural interaction between RGB and IR in the frequency domain through pseudomodal branch replication and cross-attention mechanisms. Low-frequency components enhance global structural consistency, while high-frequency components enhance edge texture details, effectively compensating for the frequency domain complementary information that is difficult to capture in spatial domain fusion. Experimental results demonstrate that this module significantly improves performance compared to existing spatial domain fusion methods in scenarios such as low contrast and small target detection.

[0072] 3. Multi-scale fusion and frequency domain enhancement work together to accommodate both small and large targets.

[0073] This invention applies the GFFM module to three scales (P3, P4, and P5) of the YOLO dual-stream backbone to ensure sufficient spatial domain fusion for targets at different scales. Simultaneously, the MDFCA module is applied only to the smallest scale (P5), leveraging the rich structural information of large targets in the frequency domain while controlling computational overhead. This strategy of "spatial domain full-scale fusion + frequency domain single-scale enhancement" achieves a good balance between detection accuracy and real-time performance.

[0074] 4. Highly versatile and adaptable to various dual-flow detection frameworks.

[0075] Although this invention is illustrated using the YOLO dual-stream backbone as an example, the proposed GFFM and MDFCA modules are universal and can be migrated to other dual-stream detection networks (such as dual-stream Faster R-CNN, dual-stream Transformer, etc.), exhibiting good scalability and portability. Attached Figure Description

[0076] Figure 1 This is the overall flowchart of the present invention.

[0077] Figure 2 This is a schematic diagram of the overall network model structure of the present invention.

[0078] Figure 3 This is a schematic diagram of the feature-gated fusion module GFFM of the present invention.

[0079] Figure 4 This is a schematic diagram of the multimodal DCT frequency cross-attention module (MDFCA) structure of the present invention.

[0080] Figure 5 This is a comparison of the detection results for Scenario 1 in the embodiment.

[0081] Figure 6 This is a comparison of the detection results for scenario two in the embodiment. Detailed Implementation

[0082] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0083] like Figure 1 and Figure 2 As shown, the RGB-IR target detection method based on multimodal feature fusion and frequency domain attention enhancement of the present invention includes the following steps:

[0084] Step 1: Obtain paired RGB and IR images to construct bimodal input data;

[0085] Step 2: Construct a two-stream YOLO backbone network and extract RGB three-scale feature maps respectively. and IR three-scale feature map The dual-stream YOLO backbone network includes parallel RGB feature extraction branches and IR feature extraction branches. Both branches use the same network structure, perform independent forward propagation, and output feature maps at three scales respectively. ;

[0086] Step 3: Convert the RGB three-scale feature maps and IR three-scale feature map The features are input into the feature-gated fusion module GFFM and subjected to adaptive gated weighted fusion to obtain preliminary fused features at the corresponding scales. ;

[0087] Step 4: Initially fuse the minimum-scale features obtained in Step 3. Input the multimodal DCT frequency cross-attention module MDFCA, perform cross-modal attention enhancement in the frequency domain, and obtain the enhanced fusion features. ;

[0088] Step 5: Initial fusion features , and enhanced fusion features The input is fed into the FPN detection network, where multi-scale feature fusion is performed, and then the target detection bounding box is output by the detection head, along with its position, category, and confidence level.

[0089] This invention relates to the following main components: a dual-stream YOLO backbone network, a feature-gated fusion module GFFM, a multimodal DCT frequency cross-attention module MDFCA, and a detection head. The dual-stream YOLO backbone network adopts a parallel CSPDarknet structure to extract multi-scale features from RGB and infrared images, providing feature maps of different resolutions for subsequent fusion. The feature-gated fusion module GFFM is inserted at the P3, P4, and P5 scale outputs of the backbone network. Through spatial gating, channel gating, and modal difference guidance, it guides the three to work together and adaptively generates dynamic fusion weights, realizing spatial domain weighted fusion of RGB and IR features, effectively suppressing noise and highlighting complementary information. The multimodal DCT frequency cross-attention module MDFCA only operates on the smallest scale (20×20) fusion features. First, it maps the features to the frequency domain through two-dimensional DCT, then uses cross-attention to realize the interaction between RGB and IR modes in the frequency domain, and finally outputs enhanced features through inverse DCT and residual connections, thereby mining the complementarity of low-frequency global structure and high-frequency detail information. The detection head predicts the position, category, and confidence of the target box from the features fused by the FPN neck and outputs the final detection result. The detection neck (FPN / PANet) and detection head receive the preliminary fusion features from P3 and P4 and the enhanced fusion features from P5. They perform multi-scale feature fusion from top to bottom and bottom to top, and finally the detection head outputs the position and category confidence of the target detection box.

[0090] like Figure 3 As shown, the feature-gated fusion module GFFM in this embodiment processes RGB feature maps of the same scale. and IR feature map Perform adaptive weighted fusion to output preliminary fused features. C represents the number of channels in the feature map, and H and W represent the rows and columns of the feature map, respectively. The specific steps are as follows:

[0091] Step 3.1: The modal difference extraction submodule explicitly calculates the difference information between the RGB mode and the IR mode to guide the subsequent generation of gating weights; firstly, it calculates the pixel-by-pixel absolute difference map. :

[0092] ;

[0093] Then, the difference diagram Channel-level global average pooling is performed, and channel difference weights are generated through a two-layer multilayer perceptron (MLP). :

[0094] ;

[0095] Simultaneously, calculate the spatial difference significance map: for the difference map Take the L2 norm along the channel dimension, and then normalize using Sigmoid:

[0096] ;

[0097] ;

[0098] Disparity diagram The number of channels, The difference diagrams are shown separately. Width and height, Disparity diagram In the Specific pixel values ​​on each channel;

[0099] Broadcast the channel difference weights and spatial difference saliency maps separately, and multiply them with the difference map to obtain the difference guidance map. :

[0100] ;

[0101] Step 3.2: The spatial gating submodule generates fusion weights in spatial location; firstly, the RGB features and IR features are concatenated along the channels to obtain... Then through two layers Convolution followed by Sigmoid activation yields a spatially gated graph. :

[0102] ;

[0103] This space gating diagram The range of values ​​is This indicates the importance of each spatial location; the larger the value, the more worthwhile the features of that spatial location are to be preserved.

[0104] Step 3.3: The channel gating submodule generates fusion weights along the channel dimension; that is, it performs global average pooling on the RGB and IR features respectively, adds the two pooling vectors, and then generates channel weights through a two-layer MLP. :

[0105] ;

[0106] Channel weight vector Expanded into a channel-gated map with the same spatial size as the feature map, where each spatial location shares the same channel weights;

[0107] Step 3.4: The gating interaction and adaptive selection submodule merges the spatial gating map and the channel gating map;

[0108] First, the spatial gating graph is broadcast to the same dimension as the channel gating graph, and then multiplied element-wise with the channel gating graph to obtain the joint gating graph. : ;

[0109] in Indicates broadcast multiplication;

[0110] Then, the joint gating graph With difference guidance map By adding them together, we obtain the preliminary gating fusion features. :

[0111] ;

[0112] This operation allows the fusion process to simultaneously consider both the joint importance of space and channel and the significance of modal differences;

[0113] Step 3.5: To adaptively preserve the original splicing features, the dynamic residual connection submodule introduces dynamic residual coefficients. Dynamic residual coefficient By analyzing the difference diagram After global average pooling, the result is obtained through a two-layer MLP: ;

[0114] Ultimately, the initial gating fusion features will be... pass The convolution operation is performed and concatenated with the original spliced ​​feature residuals to obtain preliminary fused features:

[0115] ;

[0116] The aforementioned feature-gated fusion module can adaptively adjust the fusion weights in space and channels based on the dynamic quality differences between RGB and IR modes, effectively suppressing noise and highlighting complementary information.

[0117] like Figure 4 As shown, the Multimodal DCT Frequency Cross-Attention Module (MDFCA) in this embodiment enhances cross-modal information in the frequency domain through frequency domain transformation, pseudo-modal branch replication, frequency domain cross-attention, and inverse transformation. The specific processing steps are as follows:

[0118] Step 4.1: Frequency domain transformation, i.e., preliminary fusion of features at the smallest scale. Perform a two-dimensional discrete cosine transform to obtain the frequency domain features. : To realize the transformation of spatial features into frequency domain features;

[0119] ;

[0120] Two-dimensional DCT decomposes spatial domain signals into the sum of cosine basis functions of different frequencies. Low-frequency coefficients correspond to the global structure of the image, while high-frequency coefficients correspond to edge and texture details.

[0121] Step 4.2, Modal branch replication: To achieve interaction between the RGB and IR modes in the frequency domain, the frequency domain features are copied. The copy is split into two independent branches, denoted as follows: and Both have the same initial value;

[0122] Subsequently, cross-attention is used to differentiate the two branches, thereby extracting complementary information;

[0123] Step 4.3: Frequency domain cross-attention, using the frequency domain features of the two branches as the query, key, and value, respectively; first, flatten the spatial dimensions of the feature map into a sequence form: and The shape from Transform into Then, through linear transformation, we obtain... :

[0124] ;

[0125] in, The transformation matrix is ​​a learnable linear transformation matrix, and then a multi-head attention mechanism is employed (number of heads). ): ;

[0126] in Dimensions for each head;

[0127] The output of multi-head attention, after linear transformation, is restored to... The shape is used to obtain the frequency domain cross-attention output. :

[0128] ;

[0129] in, ;

[0130] Step 4.4: Inverse transform and residual connection, ... Mapping back to the spatial domain via two-dimensional inverse discrete cosine transform (2D-IDCT):

[0131] ;

[0132] Finally, through residual connections and Convolutional output enhancement features :

[0133] .

[0134] This embodiment uses an end-to-end training method to train the network, and the loss function includes target classification loss. Bounding box regression loss and focus loss :

[0135] ;

[0136] in, , , These are the corresponding weights; , , .

[0137] To verify the technical effectiveness of this invention, the invention and a baseline model were compared on the DroneVehicle dataset. The results... Figure 5 , Figure 6 As shown in Table 1.

[0138] Figure 5 The red dashed area indicates that the baseline model (e.g., the YOLOv8-M model) has missed detections of targets; Figure 6The yellow dashed line area indicates that the baseline model misclassifies the target. In contrast, the present invention can accurately detect and correctly classify the target in the above scenarios, and the detection results are highly consistent with the ground truth (GT).

[0139] Table 1 Evaluation results of the present invention and the baseline model on the dataset.

[0140] Table 1 also evaluates the detection results and the baseline model at mAP@50 (mean accuracy), including the detection accuracy for different categories. The tracking performance of this invention is improved to some extent. In summary, this invention can effectively overcome the problems of missed detections and false positives of the baseline model, and has higher detection accuracy and robustness.

Claims

1. An RGB-IR target detection method based on multimodal feature fusion and frequency domain attention enhancement, characterized in that, Includes the following steps: Step 1: Obtain paired RGB and IR images to construct bimodal input data; Step 2: Construct a two-stream YOLO backbone network and extract RGB three-scale feature maps respectively. and IR three-scale feature map The dual-stream YOLO backbone network includes parallel RGB feature extraction branches and IR feature extraction branches. Both branches use the same network structure, perform independent forward propagation, and output feature maps at three scales respectively. This refers to the feature layer level; Step 3: Convert the RGB three-scale feature maps and IR three-scale feature map The features are input into the feature-gated fusion module GFFM and subjected to adaptive gated weighted fusion to obtain preliminary fused features at the corresponding scales. ; Step 4: Initially fuse the minimum-scale features obtained in Step 3. Input the multimodal DCT frequency cross-attention module MDFCA, perform cross-modal attention enhancement in the frequency domain, and obtain the enhanced fusion features. ; Step 5: Initial fusion features , and enhanced fusion features The input is fed into the FPN detection network, where multi-scale feature fusion is performed, and then the target detection bounding box is output by the detection head, along with its position, category, and confidence level.

2. The RGB-IR target detection method based on multimodal feature fusion and frequency domain attention enhancement according to claim 1, characterized in that: The feature-gated fusion module GFFM performs RGB feature maps of the same scale. and IR feature map Perform adaptive weighted fusion to output preliminary fused features. C represents the number of channels in the feature map, and H and W represent the rows and columns of the feature map, respectively. The specific steps are as follows: Step 3.1: The modal difference extraction submodule explicitly calculates the difference information between the RGB mode and the IR mode to guide the subsequent generation of gating weights; firstly, it calculates the pixel-by-pixel absolute difference map. : ； Then, the difference diagram Channel-level global average pooling is performed, and channel difference weights are generated through a two-layer multilayer perceptron (MLP). : ； Simultaneously, compute spatial difference significance maps. : Difference diagram Take the L2 norm along the channel dimension, and then normalize using Sigmoid: ； ； This represents the number of channels in the difference plot D. These represent the width and height of the difference graph D, respectively. The difference graph D represents the first time. Specific pixel values ​​on each channel; This refers to a spatial difference amplitude map; Broadcast the channel difference weights and spatial difference saliency maps separately, and multiply them with the difference map to obtain the difference guidance map. : ； This refers to element-wise multiplication; Step 3.2: The spatial gating submodule generates fusion weights in spatial location; firstly, the RGB features and IR features are concatenated along the channels to obtain... Then through two layers convolution Then through the Sigmoid function Activate to obtain the spatial gating map : ； This space gating diagram The range of values ​​is This indicates the importance of each spatial location; the larger the value, the more worthwhile the features of that spatial location are to be preserved. The fusion features after splicing This refers to the channel-dimensional splicing operation; Step 3.3: The channel gating submodule generates fusion weights along the channel dimension; that is, it performs global average pooling on the RGB and IR features respectively. The two pooling vectors are added together, and then channel weights are generated through a two-layer MLP. : ； Channel weight vector Expanded into a channel-gated map with the same spatial size as the feature map, where each spatial location shares the same channel weights; Step 3.4: The gating interaction and adaptive selection submodule merges the spatial gating map and the channel gating map; First, the spatial gating graph is broadcast to the same dimension as the channel gating graph, and then multiplied element-wise with the channel gating graph to obtain the joint gating graph. : ; in Indicates broadcast multiplication; Then, the joint gating graph With difference guidance map By adding them together, we obtain the preliminary gating fusion features. : ； Step 3.5: To adaptively preserve the original splicing features, the dynamic residual connection submodule introduces dynamic residual coefficients. Dynamic residual coefficient By analyzing the difference diagram After global average pooling, the result is obtained through a two-layer MLP: ; Ultimately, the initial gating fusion features will be... pass The convolution operation is performed and concatenated with the original spliced ​​feature residuals to obtain preliminary fused features: ； This refers to normalizing the convolution result; This refers to the ReLU activation operation; The aforementioned feature-gated fusion module adaptively adjusts the fusion weights in space and channels based on the dynamic quality differences between the RGB and IR modes.

3. The RGB-IR target detection method based on multimodal feature fusion and frequency domain attention enhancement according to claim 1, characterized in that, The Multimodal DCT Frequency Cross-Attention Module (MDFCA) enhances cross-modal information in the frequency domain through frequency domain transformation, pseudo-modal branch replication, frequency domain cross-attention, and inverse transformation. The specific processing steps are as follows: Step 4.1: Frequency domain transformation, i.e., preliminary fusion of features at the smallest scale. Perform a two-dimensional discrete cosine transform to obtain the frequency domain features. : ； Two-dimensional DCT decomposes spatial domain signals into the sum of cosine basis functions of different frequencies. Low-frequency coefficients correspond to the global structure of the image, while high-frequency coefficients correspond to edge and texture details. Step 4.2, Modal branch replication: To achieve interaction between the RGB and IR modes in the frequency domain, the frequency domain features are copied. The copy is split into two independent branches, denoted as follows: and Both have the same initial value; Subsequently, cross-attention is used to differentiate the two branches, thereby extracting complementary information; Step 4.3: Frequency domain cross-attention, using the frequency domain features of the two branches as the query, key, and value, respectively; first, flatten the spatial dimensions of the feature map into a sequence form: and The shape from Transform into Then, the query is obtained through linear transformation. ,key Sum ; ； in, It is a learnable linear transformation matrix. This refers to the flattening operation; Then, a multi-head attention mechanism is employed: ; in Dimensions for each head; This refers to attention operations. This refers to the normalization function; The output of multi-head attention, after linear transformation, is restored to... The shape is used to obtain the frequency domain cross-attention output. : ； in, ; This refers to the reshaping and restoration operation; This refers to the output of the i-th attention head. This refers to the query projection weight matrix of the i-th head. This refers to the key projection weight matrix of the i-th head. The value of the i-th head is projected onto the weight matrix; Step 4.4: Inverse transform and residual connection, ... By mapping back to the spatial domain using the two-dimensional inverse discrete cosine transform, the corresponding feature map is obtained. : ; Finally, the features Connection with original feature F residual and Convolutional output enhancement features : 。 4. The RGB-IR target detection method based on multimodal feature fusion and frequency domain attention enhancement according to claim 1, characterized in that, The network is trained using an end-to-end training method, and the loss function includes object classification loss. Bounding box regression loss and focus loss : ； in, , , These are the corresponding weights.

Images