Infrared and visible light image fusion method and system based on dual-channel frequency domain guidance

Abstract

The invention discloses an infrared and visible light image fusion method and system based on dual-channel frequency domain guidance, and the method comprises the following steps: firstly, extracting the shallow spatial domain features of infrared and visible light images through a convolution layer; secondly, converting the image to a frequency domain by using fast Fourier transform (FFT), respectively fusing amplitude and phase components, and recovering the image to a spatial domain through inverse FFT (IFFT); thirdly, inputting the shallow layer features into a module based on Swin Transform to extract deep global features; then, frequency domain and space domain information is integrated through a cross-domain fusion module, and intra-domain and cross-domain interaction is achieved in combination with multi-head self-attention (MSA) and cross-attention (MCA) mechanisms; and finally, reconstructing a fusion image based on a CNN-Transform hybrid architecture, and reserving the target saliency of the infrared image and the texture details of the visible light image. According to the method, through a frequency domain and space domain double-path fusion strategy, the quality of the fused image is remarkably improved, the recognition precision can be effectively improved in a target detection task, and the practical value is high.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a dual-path frequency domain guided network (DFDGFusion) for infrared and visible light image fusion, which aims to generate high-quality fused images with enhanced target saliency and rich texture details by fusing image information of two modalities. Background Art

[0002] The fusion technology of infrared and visible light images has important application value in many fields, such as military surveillance, autonomous driving, and medical imaging. Infrared images can highlight warm targets hidden in complex backgrounds based on the thermal radiation of objects, and are highly robust to lighting changes and adverse weather conditions, but they often lack rich texture details. Visible light images, on the other hand, show delicate texture details and high contrast by detecting reflected or emitted light, which is more in line with human visual perception. However, due to the significant differences in the spectral characteristics of the two modalities, how to effectively extract and integrate the feature information from the two modalities during the fusion process to generate a high-quality fused image is a key challenge. Most existing fusion methods focus on pixel-level or feature-level fusion in the spatial domain, and often ignore the value of frequency domain information, resulting in deficiencies in the fused image in terms of target saliency and texture detail preservation. Summary of the Invention

[0003] The purpose of the present invention is to address the defects and shortcomings of the existing technology, effectively integrate spatial and frequency domain information, and process their complementary characteristics in a parallel manner. In order to achieve this, it is ensured that spatial and frequency domain features are seamlessly integrated at multiple levels in three key stages (feature extraction, feature fusion and feature reconstruction). The present invention fully captures the hidden representations of the two domains through a parallel guided structure, thereby realizing comprehensive learning and integration of multi-level feature information to achieve better image fusion effects. The present invention creatively proposes a dual-path frequency domain guided infrared and visible light image fusion network (DFDGFusion), the core of which is to combine the information fusion strategies of the spatial domain and the frequency domain, and realize efficient fusion of infrared and visible light images through cross-domain long-distance learning and dual-path frequency domain guided spatial fusion strategy.

[0004] To achieve the above objectives, the present invention adopts the following technical solutions.

[0005] Step 1: Extract shallow spatial domain features from the input image.

[0006] Among them, multiple convolutional layers are used to extract features from the input infrared and visible light images to obtain low-level detail information.

[0007] Step 2: Extract shallow frequency domain features from the input image.

[0008] Among them, the input image is converted from the spatial domain to the frequency domain through FFT, the amplitude and phase information in the frequency domain are extracted and fused separately, and then converted back to the spatial domain through inverse FFT (IFFT).

[0009] Step 3: Extract deep features from the obtained shallow spatial domain and frequency domain feature maps.

[0010] Among them, a Transformer-based module is used for deep feature extraction. The module consists of multiple SwinTransformer layers and can capture the global dependencies of the image.

[0011] Step 4: Perform cross-domain fusion module on the extracted deep features.

[0012] The attention-guided cross-domain module enables information interaction and fusion between the frequency and spatial domains. This module includes multi-head self-attention (MSA) and multi-head cross-attention (MCA) mechanisms to achieve information integration within and between domains.

[0013] Step 5: Reconstruct the image based on the fused feature map.

[0014] Among them, a CNN and Transformer-based image reconstruction unit is used to map the fused deep features back to the image space to generate the final fused image.

[0015] To achieve the purpose of the present invention, based on the above method, the present invention proposes a dual-path frequency domain guided infrared and visible light image fusion system, including a shallow spatial domain feature module, a shallow frequency domain feature extraction module, a deep feature extraction module, a cross-domain fusion module, and an image reconstruction module.

[0016] Among them, the shallow spatial domain feature module is used to extract shallow spatial domain features from the input image;

[0017] Shallow frequency domain feature extraction module, used to extract shallow frequency domain features from the input image;

[0018] Deep feature extraction module, used to extract deep features from input images

[0019] Cross-domain fusion module, used to fuse the extracted deep features;

[0020] The image reconstruction module is used to reconstruct the image based on the fused feature map.

[0021] Beneficial effects

[0022] Through deep fusion in the spatial domain and frequency domain, the present invention not only retains the target saliency in the infrared image, but also inherits the rich texture details of the visible light image, significantly improving the quality of the fused image and making it more consistent with human visual perception. In downstream target detection tasks, the fused image can effectively improve the accuracy of target recognition. Since more key information is retained during the fusion process, the target detection model can more accurately identify and locate the target, thus having higher practical value in practical applications. The balance between computational efficiency and fusion performance is fully considered in the design. By reasonably setting the number and structure of the cross-domain fusion modules and adopting efficient self-attention and cross-attention mechanisms, the high performance of the model is guaranteed while avoiding excessive computational costs, making it more feasible and scalable in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is the overall flow chart of the method of the present invention;

[0024] Figure 2 This is a module composition diagram of the system of the present invention. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0026] like Figure 1 As shown, the dual-path frequency domain guided infrared and visible light image fusion method includes the following steps:

[0027] Step S10: Extract shallow spatial domain features from the input image.

[0028] Furthermore, multiple convolutional layers are used to extract features from the input infrared and visible light images to obtain shallow spatial domain features.

[0029] Step S20: Extract shallow frequency domain features from the input image.

[0030] Furthermore, the input image is converted from the spatial domain to the frequency domain by FFT, the amplitude and phase information in the frequency domain are extracted and fused separately, and then converted back to the spatial domain by inverse FFT (IFFT).

[0031] Specifically, let the input infrared image be I ir , visible light image is I vis , the infrared and visible light images are converted from the spatial domain to the frequency domain by applying the Fast Fourier Transform (FFT). Subsequently, the amplitude and phase components are extracted from the spectra of the infrared and visible light images as shown below

[0032] {AMP ir ,AMP vis}={|FFT(I ir )|,|FFT(I vis )|},

[0033] {PHA ir ,PHA vis}={θ(FFT(I ir )),θ(FFT(I vis ))},

[0034] Here, |·| represents the absolute value function, θ(·) represents the angle, AMP represents the amplitude, and PHA represents the phase.

[0035] Then, two independent branches are established to fuse the amplitude and phase components independently. Each branch consists of a guided enhancement block (GEB), which includes two 3×3 convolutional layers, followed by instance normalization and leaky relu, with a 3×3 convolutional layer in the middle. After fusion in the frequency domain, an inverse fast Fourier transform (IFFT) is applied to restore the fusion result from the frequency domain back to the spatial domain. The whole process can be mathematically expressed by the following formula,

[0036]

[0037] in, Indicates a channel connection.

[0038] Step S30: extracting deep features from the obtained shallow spatial domain and frequency domain feature maps.

[0039] Furthermore, deep feature extraction is performed using a Transformer-based module, which consists of multiple SwinTransformer layers and can capture the global dependencies of the image.

[0040] Specifically, let the shallow features of the visible light image be The shallow features of infrared images are The IFFT result is combined with and Connect them together as input for the next step of deep feature extraction.

[0041]

[0042] Then extract the deep features.

[0043]

[0044] Among them, H DE(·) represents the deep feature extraction unit, which consists of N Swin Transformer layers, where N is set to 4. The basic structure of the Swin Transformer layer is consistent with the inter-domain fusion unit.

[0045] Step S40: Perform a cross-domain fusion module on the extracted deep features.

[0046] Furthermore, an attention-guided cross-domain module is used to achieve information interaction and fusion between the frequency and spatial domains. This module includes multi-head self-attention (MSA) and multi-head cross-attention (MCA) mechanisms to achieve information integration within and between domains.

[0047] The intra-domain fusion unit performs the attention function h times in parallel and concatenates the results for multi-head self-attention, where h is set to 6. Subsequently, a feedforward network (FFN) consisting of two multi-layer perceptron (MLP) layers with GELU activation is applied to refine the feature labels generated by multi-head self-attention (MSA). Layer normalization (LN) is performed after both MSA and FFN, and residual connections are used in both modules.

[0048] Specifically, the complete process of the in-domain fusion unit of the local window feature X can be expressed as

[0049] {Q,K,V}={XW Q ,XW K ,XW V},

[0050]

[0051] Where Z represents the output of the intra-domain fusion unit with X as input, Q represents the query vector, and K represents the key vector.

[0052] Among them, the inter-domain fusion unit based on cross-attention further promotes the integration of global interactions between different domains. Both intra-domain and inter-domain fusion units are built on similar principles, with the main difference being that the inter-domain fusion unit uses multi-head cross attention (MCA) instead of multi-head self-attention (MSA) to achieve global context exchange between domains.

[0053] Specifically, given two local window features X1 and X2 from different domains, the entire process of the inter-domain fusion unit is defined as

[0054]

[0055] Among them, Z i Indicates that X iis the output of the fusion unit in the input domain, Q i Represents X i The corresponding query vector, K i Represents X i The corresponding key vector. For Q1 in domain 1, cross-domain information is integrated by using K2 and V2 in domain 2 for attention weighting, while retaining the information in domain 1 through residual connections, and vice versa. Our model adopts L attention-guided cross-domain fusion modules, each of which consists of a cascade of intra-domain fusion units and inter-domain fusion units, alternately integrating global inter-domain and cross-domain interactions. To strike a balance between computational efficiency and fusion performance, L is set to 2.

[0056] Step S50: reconstructing the image based on the fused feature map.

[0057] Furthermore, through an image reconstruction unit based on CNN and Transformer, the fused deep features are mapped back to the image space to generate the final fused image.

[0058] Specifically, after the attention-guided cross-domain fusion module, a convolutional layer with spatially invariant filters is applied to aggregate local information between different domains and improve the translation equivariance of DFDGFusion.

[0059] Specifically, and is the output feature of the cross-domain fusion module, is the deep frequency domain feature, the process can be expressed as

[0060]

[0061] Among them, Concat(·) represents the connection along the channel dimension, H Conv (·) denotes a convolutional layer with spatially invariant filters, F FDF Represents the fused deep features, which serve as the input of the feature reconstruction module.

[0062] After fully integrating the complementary information from different domains, a CNN-Transformer-based image reconstruction unit is designed to map the fused deep features back to the image space. The image reconstruction unit starts with three convolutional layers, followed by a Leaky ReLU activation function to reduce the number of channels. Next, P Swin Transformer layers are used to recover the deep fusion features from a global perspective and generate the fused image I f The value of P is set to 4, which is expressed as

[0063] I f =H IR (F FDF ).

[0064] Among them, H IR (·) denotes a reconstruction module with P Swin Transformer layers.

[0065] Figure 2 Schematic diagram of the structure of the system of the present invention.

[0066] like Figure 2 As shown, the dual-path frequency domain guided infrared and visible light image fusion system includes a shallow spatial domain feature module M10, a shallow frequency domain feature extraction module M20, a deep feature extraction module M30, a cross-domain fusion module M40 and an image reconstruction module M50.

[0067] Among them, the shallow spatial domain feature module M10 is used to extract shallow spatial domain features from the input image;

[0068] Shallow frequency domain feature extraction module M20, used to extract shallow frequency domain features from the input image;

[0069] Deep feature extraction module M30, used to extract deep features from the input image

[0070] Cross-domain fusion module M40, used to fuse the extracted deep features;

[0071] The image reconstruction module M50 is used to reconstruct the image based on the fused feature map.

[0072] The connection relationship between the above modules is:

[0073] The infrared input image and the visible light input image are connected to the input end of the shallow spatial domain feature module M10, and the infrared input image and the visible light input image are connected to the input end of the shallow frequency domain feature extraction module M20; the output end of the shallow spatial domain feature module M10 and the output end of the shallow frequency domain feature extraction module M20 are connected to the input end of the deep feature extraction module M30 after channel splicing in the spatial domain fusion path, and the output end of the shallow frequency domain feature extraction module M20 is connected to the input end of the deep feature extraction module M30 in the frequency domain guidance path; in the spatial domain fusion path, the output end of the deep feature extraction module M30 is connected to the input end of the cross-domain fusion module M40; in the frequency domain guidance path, the output end of the cross-domain fusion module M40 and the output end of the deep feature extraction module M30 are connected to the input end of the image reconstruction module M50 after channel splicing.

[0074] Therefore, the invented dual-pathway frequency-domain-guided infrared and visible image fusion method and system introduces a CNN-Transformer hybrid framework. A CNN-driven shallow feature extraction module captures low-level details in the spatial and frequency domains, while a Transformer-based deep feature extraction module mines deeper representations to exploit local and global dependencies. This design ensures more effective preservation of texture information in the source images. In the frequency domain path, a fast Fourier transform (FFT) is used to convert spatial information into the frequency domain for fusion, and then an inverse fast Fourier transform (IFFT) is applied to restore it back to the spatial domain. Finally, a Transformer-driven deep feature reconstruction module works together with a CNN-based fusion reconstruction module to synthesize the fused image, maintaining excellent visual perception by integrating global and local details. Using self-attention and cross-attention mechanisms implemented through a shifting window strategy, this design choice enables the model to efficiently process images of arbitrary size. Extensive experiments on multiple datasets validate the effectiveness of this method in generating high-quality fused images. The generated image preserves key information from both source modalities while maintaining high consistency with human visual perception. Furthermore, our approach significantly improves the accuracy of object detection, demonstrating its practical applicability.

Claims

1. A dual-path frequency domain guided infrared and visible light image fusion method, characterized in that: The following steps are involved: Step 1: Extract shallow spatial domain features from the input infrared image and visible light image; Step 2: extracting shallow frequency domain features from the infrared image and the visible light image; Step 3: Extracting deep features based on the shallow spatial domain features and shallow frequency domain features; Step 4: Perform cross-domain information fusion on the deep features through the cross-domain fusion module; Step 5: Reconstruct the final fused image based on the fused feature maps.

2. The method according to claim 1, wherein In step 1, extracting shallow spatial domain features includes: The input infrared image and visible light image are respectively subjected to feature extraction through multiple convolutional layers to obtain low-level spatial domain detail information.

3. The method according to claim 1, wherein In step 2, extracting shallow frequency domain features includes: The input infrared image and visible light image are converted to the frequency domain through fast Fourier transform (FFT), and the amplitude and phase components of the frequency domain are extracted; the amplitude and phase components are fused separately, and the fusion results are converted back to the spatial domain through inverse fast Fourier transform (IFFT).

4. The method according to claim 3, wherein The shallow frequency domain feature extraction formula is: {AMP ir ,AMP vis }={|FFT(I ir )|,|FFT(I vis )|}, {PHA ir ,PHA vis }={θ1FFT(I ir )),θ(FFT(I vis ))}, Here, |·| represents the absolute value function, and θ(·) represents the angle. in, Indicates a channel connection.

5. The method according to claim 1, characterized in that In step 3, extracting deep features includes: The shallow spatial domain features and shallow frequency domain features are channel-concatenated and input into a deep feature extraction module based on Swin Transformer to capture global dependencies.

6. The method according to claim 1, wherein In step 4, the cross-domain fusion module includes: The intra-domain fusion unit based on multi-head self-attention (MSA) is used to integrate global information within the same domain; the inter-domain fusion unit based on multi-head cross-attention (MCA) is used to realize cross-domain information interaction between frequency domain and spatial domain.

7. The method according to claim 5, wherein The process formula of the cross-domain fusion module is: Where X1 and X2 are two local window features from different domains.

8. The method according to claim 1, wherein: In step 5, reconstructing the fused image includes: The cross-domain fused feature map is input into the image reconstruction unit based on CNN and Transformer, and the final fused image is generated through the convolution layer and Swin Transformer layer.

9. A dual-path frequency domain guided infrared and visible light image fusion system, characterized in that: include: A shallow spatial domain feature module, configured to perform step 1 in claim 1; A shallow frequency domain feature extraction module, configured to perform step 2 in claim 1; A deep feature extraction module, configured to perform step 3 of claim 1; A cross-domain fusion module, configured to execute step 4 in claim 1; An image reconstruction module, configured to perform step 5 in claim 1.

Images