An infrared-visible light image fusion method based on frequency domain high-order interaction

Abstract

The application discloses an infrared-visible light image fusion method based on frequency domain high-order interaction, acquires an infrared image and a visible light image; initial features of the infrared image and the visible light image are respectively extracted through a cross center difference convolution operator and are input into an FDB module to obtain frequency domain primary features, and meanwhile, the initial features of the infrared image are input into an ATB module to obtain a weight matrix; Hadamard product processing is used to obtain visible light frequency domain features, and the initial features of the infrared image are gradually refined under the guidance of the weight matrix through convolution to obtain infrared frequency domain features; the obtained visible light and infrared frequency domain features are input into a channel interaction module to obtain primary visible light features; after the primary visible light feature output is aggregated with the initial features of the infrared image, the output is used as the input of the FDB module of the next layer, the above process is iterated, and finally, the fusion features are output. The method is used for generating a fusion image with delicate texture, higher contrast and better compliance with human visual perception.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically to an infrared-visible light image fusion method based on high-order frequency domain interaction. Background Technology

[0002] The primary goal of multimodal image fusion (MMIF) is to integrate single-modal image information from different optical sensors to generate multimodal images. This technology has been widely applied in various fields, providing strong support for downstream tasks such as autonomous driving, remote sensing, and medical image diagnosis. Infrared and visible light image fusion (IVF) is an important task within the MMIF field. On the one hand, infrared images excel at capturing thermal information, exhibiting higher contrast in low-light or foggy conditions, and effectively highlighting key targets such as pedestrians and vehicles. However, they are less effective at capturing accurate edge textures. On the other hand, visible light images excel at capturing texture information but are highly susceptible to environmental factors and face difficulties in capturing the structure of thermal targets. Therefore, researching how to integrate complementary information from these different modalities is of practical need. However, most existing methods suffer from problems such as image texture blurring and loss of original modal information. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of existing technologies and propose an infrared-visible image fusion method based on high-order frequency domain interaction. This method generates fused images with finer textures, higher contrast, better conformity to human visual perception, and better preservation of thermal information from the source images. The framework aims to address the problems of image texture blurring and loss of original modal information by improving spatial and channel interactions to enhance interaction performance. This invention designs a novel Frequency-Spatial Interaction Module (FSIM), which utilizes Fast Fourier Transform (FFT) to transform information from the spatial domain to the frequency domain, thereby achieving cross-modal information integration. This information is then transformed back to the spatial domain, and an Attention Transfer Block (ATB) is used to retain and integrate information from the original infrared image, guiding further feature interactions. To avoid losing important features during channel interactions, a gating mechanism is employed to prevent the loss of positive features.

[0004] To achieve the above objectives, the technical solution specifically adopted by the present invention is as follows:

[0005] An infrared-visible image fusion method based on high-order frequency domain interaction includes the following steps:

[0006] Step 1: Acquire the corresponding infrared and visible light images;

[0007] Step 2: Using the infrared image and the visible light image as input, extract the initial features of the infrared image and the visible light image respectively through the cross-center difference convolution operator;

[0008] Step 3: Input the initial features of the infrared image and the visible light image into the FDB module to obtain the primary features in the frequency domain, and simultaneously input the initial features of the infrared image into the ATB module to obtain the weight matrix;

[0009] Step 4: Use the Hadamard product to interact the weight matrix with the initial features of the infrared image to obtain the infrared weight matrix. At the same time, normalize the primary features in the frequency domain, and then use the Hadamard product to obtain the initial fused features under the guidance of the infrared weight matrix. The visible light frequency domain features are obtained by element-wise summing the initial fused features with the initial features of the visible light image.

[0010] The initial features of the infrared image are gradually refined into infrared frequency domain features through convolution guided by a weight matrix.

[0011] Step 5: Input the obtained visible light and infrared frequency domain features into the channel interaction module, and use a gated weighted W. G The GM module adjusts channel interactions to generate mixed features, resulting in primary visible light features.

[0012] Step 6: Aggregate the primary visible light feature output with the initial features of the infrared image and use the output as the input to the next layer's FDB module. Iterate the above process, and finally, through stitching and convolution, obtain multi-level visible light features from the accumulated primary visible light features, and project them back into the image space to output the final fused feature F. fuse The fusion feature F fuse The corresponding characteristic matrix I fuse The fused image is obtained by restoring the RGB channels.

[0013] Preferably, in step 2, the method for extracting initial features using the cross-center difference convolution operator is as follows:

[0014] For infrared images, initial features are extracted through convolution operations;

[0015] For visible light images, the initial features of the visible light image are obtained by processing the Y channel in the YCbCr color space.

[0016] Preferably, in step 3, the primary feature extraction method in the frequency domain is as follows:

[0017] The extracted primary features of infrared and visible light images are converted to the frequency domain using Fourier transform by the FBD module, thereby extracting the amplitude and phase components.

[0018] The amplitude and phase components are processed through two convolutional branches. First, they are connected along the channel dimension, then processed through a convolutional module consisting of two convolutional layers and a LeakyReLU activation function. Finally, the output is mapped back to the spatial domain by inverse Fourier transform to obtain the primary features in the frequency domain.

[0019] Preferably, the method for obtaining the weight matrix is ​​as follows:

[0020] The initial attention weights of the initial features of the infrared image are calculated through a convolutional layer with a Sigmoid activation function, and the weight matrix is ​​obtained through the Hadamard product, as shown in the following expression:

[0021] W R =F R ⊙σ(Conv(F R ))

[0022] Among them, F R This refers to the initial features of an infrared image, W. R This refers to the weight matrix, where ⊙ represents the Hadamard product and σ is the Sigmoid function.

[0023] Preferably, in step 4, the infrared weight matrix is ​​used to replace the weight matrix and interact with the primary features in the visible light frequency domain. The infrared-visible light fusion features output from the last interaction are then element-wise summed with the initial features of the visible light image to obtain the visible light frequency domain features.

[0024] Preferably, the gating weights are obtained by flattening the initial features of the infrared and visible light images and stitching them together along the channel dimension. After stitching, the images are processed by a multilayer perceptron, and the gating weights are obtained by activating the Sigmoid function.

[0025] Preferably, the method for obtaining the primary visible light features is as follows:

[0026]

[0027] in, Indicates multi-level visible light characteristics, This represents one of the branch features obtained by splicing visible light frequency domain features and infrared frequency domain features. This represents the matrix obtained by concatenating visible light frequency domain features and infrared frequency domain features, followed by global average pooling activation, and is used to interact with another branch. This represents the gating weight, where N refers to the number of iterations and L refers to the number of interactions.

[0028] Preferably, the frequency domain high-order interactive fusion model is constructed by steps 2-6, and the loss function used to train the frequency domain high-order interactive fusion model is the total loss function composed of content loss and saliency loss.

[0029] Preferably, the content loss is a combination of pixel intensity loss and gradient loss, expressed as follows:

[0030] L content =αL int +βL grad

[0031] Among them, L content Indicates content loss, L saliency L represents the saliency loss, α and β are the weights of the pixel intensity loss and gradient loss, respectively. int L represents pixel intensity loss. grad Indicates gradient loss;

[0032] The pixel intensity loss is calculated as follows:

[0033]

[0034] Where H and W represent the height and width of the image, ||·||1 represents the L1 norm, and I v I represents the feature matrix of a visible light image. R The infrared image feature matrix is ​​obtained by performing shallow feature extraction and normalization on the visible light image and the infrared image respectively using the cross-center difference convolution operator;

[0035] The gradient loss is calculated as follows:

[0036]

[0037] in, It is the Sobe l gradient operator, I fuse This represents the feature matrix of the final fused image.

[0038] Preferably, the significance loss expression is as follows:

[0039]

[0040] in, This indicates the mask weight.

[0041] This invention has the following characteristics and beneficial effects:

[0042] First, this invention proposes a novel Frequency-Spatial-Domain Interaction Module (FSIM) that thoroughly extracts global information from frequency domain features and dynamically guides feature aggregation through a specially designed attention module, solving the problem of long-distance dependency perception. Second, this invention develops a Gated-Channel Interaction Module (GCIM), whose gating mechanism can manage feature scoring allocation in channel interactions, effectively preventing the loss of positive features. Third, extensive experiments on three public datasets demonstrate that the algorithm of this invention outperforms state-of-the-art methods and significantly improves fusion performance. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating an infrared-visible image fusion method based on high-order frequency domain interaction according to an embodiment of the present invention. Detailed Implementation

[0044] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0045] An infrared-visible image fusion method based on high-order frequency domain interaction, such as Figure 1 As shown, it includes the following steps:

[0046] Step 1: Obtain the corresponding infrared and visible light image datasets.

[0047] In this embodiment, three important infrared and visible light image fusion datasets are used: M3FD, RoadScene, and TNO. The M3FD dataset contains 4200 image pairs, ideal for fusion and detection tasks. These images typically have a resolution of 1024×768 pixels and primarily depict university campuses and urban roads. In this study, 275 pairs were specifically selected for training, and 25 pairs were allocated for testing. The RoadScene dataset includes 221 pairs of aligned visible and infrared images, covering various scenes such as roads, vehicles, and pedestrians. 198 pairs were used for training, and 23 pairs were reserved for testing. The TNO dataset contains single-channel multispectral nighttime images of various military-related scenes. 53 pairs were selected for training, and 13 pairs for testing. These datasets have significantly advanced the field of infrared-visible light image fusion in recent years.

[0048] Step 2: Construct and train a high-order frequency domain interactive fusion model consisting of a frequency domain-spatial domain interaction module (FSIM) and a gated-channel interaction module (GCIM).

[0049] During training, a high-order interactive fusion model in the frequency domain was trained using a single Nvidia Titan Xp GPU and the PyTorch framework. To optimize model parameters, the Adam optimizer was used in this embodiment, with the learning rate set to 1.0 × 10⁻⁶. -4 The parameters β1 and β2 were set to 0.9 and 0.999, respectively, to ensure precise tuning during training. In each training epoch, the dataset was randomized and expanded tenfold to enhance the model's generalization ability. A batch size of 4 was configured for each GPU, and 8 worker threads were allocated to each GPU, resulting in a total of 100,000 iterations.

[0050] Furthermore, the loss function used is the total loss function, which consists of content loss and saliency loss.

[0051] L total =L content +λ s L sliency

[0052] Among them, L content Indicates content loss. λ represents the significant loss. s λ represents the hyperparameter, which serves as a balancing coefficient. In this embodiment, λ s =3;

[0053] Content loss is a combination of pixel intensity loss and gradient loss, expressed as follows:

[0054] L content =α Lint +βL grad

[0055] Among them, L content Indicates content loss, L saliency Representing the saliency loss, α and β are the weights of the pixel intensity loss and gradient loss, respectively. In this embodiment, α = 10, β = 10, and L... int L represents pixel intensity loss. grad Indicates gradient loss;

[0056] The pixel intensity loss is calculated as follows:

[0057]

[0058] Where H and W represent the height and width of the image, ||·||1 represents the L1 norm, and I v I represents the feature matrix of a visible light image. R The infrared image feature matrix is ​​obtained by performing shallow feature extraction and normalization on the visible light image and the infrared image respectively using the cross-center difference convolution operator;

[0059] The gradient loss is calculated as follows:

[0060]

[0061] in, It is the Sobe l gradient operator, I fuse This represents the feature matrix of the final fused image.

[0062] The above training was used to achieve optimal performance. The patch size was set to 128×128, and a manual seed of 10 was used. Furthermore, the model was validated every 1,000 iterations to ensure consistency in performance evaluation.

[0063] After training is complete, in this embodiment, a set of six evaluation metrics is used to comprehensively evaluate the effectiveness of image fusion: entropy (EN), mutual information (MI), spatial frequency (SF), average gradient (AG), visual information fidelity (VIF), and fusion-based quality assessment (Qabf). Generally, higher scores on these metrics indicate better image fusion performance.

[0064] Step 3: Apply the trained high-order interactive fusion model in the frequency domain to obtain the final fused image.

[0065] Specifically, visible light frequency domain features and infrared frequency domain features are extracted using FSIM:

[0066] A1. Using infrared and visible light images as input, the initial features of the infrared and visible light images are extracted respectively through the cross-center difference convolution operator;

[0067] For infrared images, initial features are extracted through convolution operations;

[0068] For visible light images, the initial features of the visible light image are obtained by processing the Y channel in the YCbCr color space.

[0069] A2. Input the initial features of the infrared and visible light images into the FDB module to obtain the primary features in the frequency domain.

[0070] Specifically, the extracted primary features of the infrared and visible light images are transformed to the frequency domain using the Fourier transform module, and then the amplitude and phase components are extracted, as shown below:

[0071] A V ,Φ V =FFT(F V )

[0072] A R ,ΦR =FFT(F R )

[0073] Among them, A V This refers to the amplitude component of a visible light image, Φ V This refers to the phase velocity of a visible light image, F. V This refers to the initial features of a visible light image, F. R This refers to the initial features of an infrared image, A R This refers to the amplitude component of the infrared image, Φ R This refers to the phase wind speed of the infrared image, and FFT(·) refers to the Fourier transform.

[0074] The amplitude and phase components are processed separately through two convolutions and concatenated along the channel dimension. Then, a convolutional module consisting of two convolutional layers and a LeakyReLU activation function is used to process the output. Finally, an inverse Fourier transform maps the output back to the spatial domain to obtain the primary frequency domain features, as shown in the following expression:

[0075] F freq =IFFT(Conv(C(A) V A R )),Conv(C(Φ V ,Φ R )))

[0076] F freq 1 refers to the primary features in the frequency domain, IFFT(·) refers to the inverse Fourier transform, and C refers to the parameters.

[0077] Simultaneously, the initial features of the infrared image are input into the ATB module, and the weight matrix is ​​calculated using a convolutional layer with a Sigmoid activation function, as shown in the following expression:

[0078] W R =F R ⊙σ(Conv(F R ))

[0079] Among them, F R This refers to the initial features of an infrared image, W. R This refers to the weight matrix, where ⊙ represents the Hadamard product and σ is the Sigmoid function.

[0080] A3. The infrared weight matrix is ​​obtained by interacting the weight matrix with the initial features of the infrared image using the Hadamard product. Simultaneously, the primary frequency domain features are normalized. Then, the normalized primary frequency domain features are used with the Hadamard product to obtain the initial fused features under the guidance of the infrared weight matrix. The infrared weight matrix is ​​used to replace the weight matrix and interact with the primary features in the visible light frequency domain. The infrared-visible fusion features output from the last interaction are then element-wise summed with the initial features of the visible light image to obtain the visible light frequency domain features.

[0081] The expression is as follows:

[0082]

[0083] Where ⊕ represents element-wise addition, and N and L are the number of iterations and the number of interactions, respectively. Represents the infrared weight matrix. Indicates the initial fusion features, To represent the frequency domain characteristics of visible light, in this embodiment, the iteration is performed 4 times, i.e., N=4.

[0084] For infrared images, the initial features of the infrared image are gradually refined through convolution under the guidance of a weight matrix to obtain infrared frequency domain features.

[0085] Specifically, the final fusion feature F is obtained through GCIM. fuse The fusion feature F fuse The corresponding characteristic matrix I fuse The fused image is obtained by restoring the RGB channels:

[0086] B1. Input the obtained visible light and infrared frequency domain features into the channel interaction module, and use a gated weighted W... G The GM module adjusts channel interactions to generate mixed features, resulting in primary visible light features.

[0087] Understandably, to address the issue of key features being unrepresentable when positive features are surrounded by negative features, the obtained visible light frequency domain features and infrared frequency domain features are input into GCIM, where the GM module aims to adjust channel interactions to generate hybrid features. By flattening the visible light and infrared features in FSIM, stitching them along the channel dimension, and processing them through a multilayer perceptron (MLP), the gated weights W are obtained. G It is activated by the Sigmoid function, the formula of which is as follows:

[0088] W G =σ(MLP(C(FL(F) V ),FL(F R ))))

[0089] During channel interaction, infrared and visible light features are concatenated and split into two branches. One branch serves as the primary input, while the other branch undergoes global average pooling followed by Sigmoid activation to obtain weights. These weights facilitate interaction through Hadamard product. The interaction result is then convolved and multiplied by gated weights to complete the initial fusion of infrared and visible light features, preparing for subsequent iterations. The process is as follows:

[0090]

[0091] in, Indicates multi-level visible light characteristics, This represents one of the branch features obtained by splicing visible light frequency domain features and infrared frequency domain features. This represents the matrix obtained by concatenating visible light frequency domain features and infrared frequency domain features, followed by global average pooling activation, and is used to interact with another branch. This represents the gating weight, where N refers to the number of iterations and L refers to the number of interactions.

[0092] B3. The initial visible light feature output is aggregated with the initial features of the infrared image and then used as the input to the next layer's FDB module. This process is iterated, and finally, the accumulated visible light features are projected back into the image space through a convolutional layer to output the final fused feature F. fuse The expression is as follows:

[0093]

[0094] Finally, the fused feature F fuse The corresponding characteristic matrix I fuse The fused image is obtained by restoring the RGB channels.

[0095] Based on the infrared-visible image fusion method based on high-order frequency domain interaction provided in this embodiment, the following comparative experiments are presented:

[0096] The method in this embodiment is compared with methods based on DenseNet (such as U2Fusion and DenseFuse), methods based on generative adversarial networks (such as TarDal), methods based on spatial-frequency domain fusion (such as SFDFasion), and methods based on hybrid attention (such as HAIAFusion). Furthermore, it is compared with the collaborative high-order interaction method SHIP. The table shows a comparison of various methods with the FDHIFusion framework proposed in this invention.

[0097] Table 1: Comparative Experiment Results

[0098] M3FD EN MI SF AG VIF Qabf U2Fusion 6.653 2.839 9.008 3.105 0.613 0.468 SDNet 6.533 3.026 10.314 3.623 0.678 0.406 TarDal 6.817 3.786 12.561 4.148 0.581 0.409 DenseFuse 6.648 2.838 8.777 3.048 0.632 0.449 SFDFusion 6.809 3.815 14.778 4.967 0.556 0.478 SHIP 6.859 4.845 14.764 4.942 0.801 0.628 HAIAFusion 6.855 4.963 14.300 4.725 0.806 0.614 FDHIFusion(Ours) 6.894 5.020 14.906 5.016 0.807 0.636

[0099] RoadScene EN MI SF AG VIF Qabf U2Fusion 7.029 2.628 13.952 5.586 0.559 0.501 SDNet 7.262 3.421 15.708 6.140 0.656 0.559 TarDal 7.196 3.247 10.448 4.259 0.553 0.403 DenseFuse 7.208 2.870 12.289 4.796 0.585 0.470 SFDFusion 7.246 3.982 14.889 6.045 0.641 0.472 SHIP 7.213 3.960 17.109 6.439 0.690 0.572 HAIAFusion 7.111 3.349 17.039 6.240 0.604 0.534 FDHIFusion(Ours) 7.274 4.004 17.704 6.676 0.679 0.593

[0100] TNO EN MI SF AG VIF Qabf U2Fusion 6.624 2.055 11.013 4.017 0.630 0.428 SDNet 6.623 2.824 9.184 3.352 0.402 0.383 TarDal 6.679 3.216 10.689 3.826 0.565 0.386 DenseFuse 6.696 2.676 9.490 3.720 0.564 0.366 SFDFusion 6.741 2.828 10.749 4.043 0.594 0.386 SHIP 6.667 4.054 11.089 4.168 0.761 0.536 HAIAFusion 6.726 3.952 10.664 4.152 0.786 0.536 FDHIFusion(Ours) 6.763 4.199 11.131 4.201 0.786 0.543

[0101] Experimental results on three datasets demonstrate that the model of this invention performs exceptionally well overall, achieving the highest scores on multiple key metrics. Table 1 presents a quantitative comparison of the method of this invention with state-of-the-art methods on the three datasets using six metrics. Compared to SHIP, the MI and AG metrics are improved by 3.61% / 1.50%, 1.11% / 3.68%, and 3.58% / 0.79% respectively on the three datasets. MI reflects the degree of source information retention in the fused image, while AG measures the sharpness of the fused image. These results indicate that, compared to basic high-order interaction methods, the method of this invention effectively preserves modality-specific features and enhances image texture. Although the use of high-exposure visible light images leads to a slight decrease in VIF on the RoadScene dataset, the method of this invention compensates for this by preserving richer modality information, reducing its negative impact on visual fidelity.

[0102] Furthermore, compared to six other state-of-the-art methods (i.e., U2Fusion, Densefuse, SDNet, TarDal, SDFusion, and HAIAFusion), the method of this invention achieves improvements of 1.15%, 0.55%, and 6.25% in the MI metric, and improvements of 0.99%, 6.99%, and 1.18% in the AG metric, respectively. Quantitative results demonstrate that the framework of this invention can create fused images that retain richer information, produce clearer textures, and better conform to human visual perception, meeting the requirements for natural and realistic images.

[0103] Furthermore, to verify the effectiveness of the proposed method, a comprehensive ablation study was conducted. These experiments aimed to evaluate key aspects of the framework, including the effectiveness of interactions and iterations, component ablation, and the analysis of higher-order interaction modules. To ensure fair and consistent comparisons, all tests were performed on three datasets under the same settings.

[0104] Table 2: Results of the experiment exploring the effectiveness of iterative interaction times

[0105]

[0106]

[0107] Table 3: Experimental Results of Modular Ablation

[0108] M3FD EN MI SF AG VIF Qabf Ablation of FDB 6.871 4.858 14.730 4.928 0.785 0.627 Ablation of ATB 6.854 4.757 14.823 4.945 0.788 0.627 ablation GM 6.849 4.770 14.854 4.986 0.779 0.621 FDHIFusion(Ours) 6.894 5.020 14.906 5.016 0.807 0.636

[0109]

[0110]

[0111] TNO EN MI SF AG VIF Qabf Ablation of FDB 6.717 4.131 10.960 3.794 0.746 0.504 Ablation of ATB 6.722 3.990 10.899 3.811 0.744 0.502 ablation GM 6.716 4.039 11.070 3.913 0.723 0.523 FDHIFusion(Ours) 6.763 4.199 11.131 4.201 0.786 0.543

[0112] Table 4: Experiment on High-Order Interaction Module Structure

[0113] M3FD EN MI SF AG VIF Qabf FSIM 6.840 4.490 14.766 5.204 0.698 0.603 GCIM 6.869 4.765 14.267 5.119 0.789 0.617 GCIM->FSIM 6.848 4.895 14.487 4.965 0.768 0.621 FDHIFusion(Ours) 6.894 5.020 14.906 5.016 0.807 0.636

[0114] RoadScene EN MI SF AG VIF Qabf FSIM 7.221 3.954 17.548 6.813 0.644 0.578 GCIM 7.211 3.980 17.670 6.595 0.663 0.581 GCIM->FSIM 7.200 3.993 17.353 6.648 0.646 0.572 FDHIFusion(Ours) 7.274 4.004 17.704 6.676 0.679 0.593

[0115]

[0116]

[0117] To evaluate the impact of interaction and iteration count on model performance, ablation experiments were conducted within the range of 3-5 interactions and 2-4 iterations. The results are shown in Table 2, where L represents the Lth interaction and N represents the Nth iteration. Experimental results show that when the number of iterations is fixed, 4 interactions achieve the best performance, improving the MI and AG metrics by 5.46% / 0.56%, 0.78% / 1.31%, and 1.06% / 1.79%, respectively, on the three test datasets. Conversely, increasing the number of interactions to 5 results in a decrease in MI and AG metrics of 0.34% / 0.63%, 0.79% / 1.51%, and 2.69% / 2.61%, respectively, compared to 3 interactions. These results indicate that too few interactions cannot adequately integrate modal information, while too many interactions may lead to information loss. Similarly, with a fixed number of interactions, 3 iterations outperformed other iteration counts, achieving improvements of 1.97% / 1.39%, 1.32% / 2.69%, and 1.30% / 1.92% in the MI and AG metrics, respectively. In contrast, compared to 4 iterations, 2 iterations resulted in a decrease of 3.42% / 1.41% / 2.65% in the MI metric, while slightly improving the AG metric by 2.36% / 2.01% / 0.19%. This indicates that while more iterations can enhance modal information integration, they may also negatively impact edge texture due to overprocessing. Based on these results, this invention selected 4 interactions and 3 iterations as the default settings for all experiments, achieving an optimal balance between modal integration, edge texture preservation, and computational efficiency.

[0118] This invention scientifically evaluated the effectiveness of the FDB, ATB, and GM modules, and detailed ablation experiment results are shown in Table 3. In the three datasets, FDB significantly enhanced the model's texture refinement capability, increasing AG by 1.79%, 3.76%, and 10.73%, respectively. ATB increased MI by 5.53%, 3.36%, and 5.24%, respectively, while preserving more original modal information. GM improved VIF by 3.59%, 5.93%, and 8.71%, respectively, significantly enhancing visual quality. This effectively demonstrates the importance of FDB for texture extraction, the necessity of ATB for preserving specific modal features, and the effectiveness of GM in enhancing visual quality and aligning with human visual perception.

[0119] In addition, this invention compared the fusion results using FSIM and GCIM in different orders, as summarized in Table 4. Using a single module significantly increased AG by 3.75%, 2.05%, and 4.45%, but other metrics (especially MI) decreased significantly, indicating that imperfect single interactions can lead to extreme fusion results, thus reducing visual quality. Conversely, the original module sequence outperformed the inverted module sequence in key metrics, with improvements in MI, AG, and VIF of 2.55% / 1.03% / 5.08%, 0.28% / 0.42% / 5.11%, and 1.84% / 1.77% / 1.55%, respectively. These results highlight the importance of well-ordered sequences in enhancing fusion results by enriching modal information, highlighting edge details, and more closely conforming to human visual perception.

[0120] The experiments described above demonstrate that the method provided in this embodiment suffers from severe limitations due to the loss of channel interaction features and original modal information, hindering the full fusion of cross-modal features. To address this issue, this paper proposes the FDHIFusion method, based on a high-order frequency-domain interactive fusion framework, for infrared and visible light image fusion. Key components of FDHIFusion include: an FDB module for cross-modal feature interaction integration; an ATB module for attention cross-modal transfer, guiding progressively refined feature interactions; and a GM module to prevent feature loss. Through this high-order frequency-domain interactive fusion framework, FDHIFusion can capture more comprehensive inter-modal relationships, thereby significantly improving fusion performance. Experimental results show that, compared to existing methods, FDHIFusion exhibits significant advantages in multiple fusion evaluation metrics and visual effects.

[0121] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An infrared-visible image fusion method based on high-order frequency domain interaction, characterized in that, Includes the following steps: Step 1: Acquire the corresponding infrared and visible light images; Step 2: Using the infrared image and the visible light image as input, extract the initial features of the infrared image and the visible light image respectively using the cross-center difference convolution operator; Step 3: Input the initial features of the infrared image and the visible light image into the FDB module to obtain the primary features in the frequency domain, and simultaneously input the initial features of the infrared image into the ATB module to obtain the weight matrix; The frequency domain primary feature extraction method is as follows: The extracted primary features of infrared and visible light images are converted to the frequency domain using Fourier transform by the FBD module, thereby extracting the amplitude and phase components. The amplitude and phase components are processed through two convolutional branches. First, they are connected along the channel dimension, then processed through a convolutional module consisting of two convolutional layers and a LeakyReLU activation function. Finally, the output is mapped back to the spatial domain by inverse Fourier transform to obtain the primary features in the frequency domain. Step 4: Use the Hadamard product to interact the weight matrix with the initial features of the infrared image to obtain the infrared weight matrix. At the same time, normalize the primary features in the frequency domain, and then use the Hadamard product to obtain the initial fused features under the guidance of the infrared weight matrix. The visible light frequency domain features are obtained by element-wise summing the initial fused features with the initial features of the visible light image. , The initial features of the infrared image are gradually refined into infrared frequency domain features through convolution guided by a weight matrix. ; Step 5: Input the obtained visible light and infrared frequency domain features into the channel interaction module, and use gated weights. The GM module adjusts channel interactions to generate mixed features, resulting in primary visible light features. ; Step 6: Aggregate the primary visible light feature output with the initial features of the infrared image and use the output as the input to the next layer's FDB module. Iterate the above process, and finally obtain multi-level visible light features from the accumulated primary visible light features through stitching and convolution, and project them back into the image space to output the final fused features. The fusion feature The corresponding characteristic matrix The fused image is obtained by restoring the RGB channels.

2. The infrared-visible image fusion method based on high-order frequency domain interaction according to claim 1, characterized in that, In step 2, the method for extracting initial features using the cross-center difference convolution operator is as follows: For infrared images, initial features are extracted through convolution operations; For visible light images, the initial features of the visible light image are obtained by processing the Y channel in the YCbCr color space.

3. The infrared-visible image fusion method based on high-order frequency domain interaction according to claim 1, characterized in that, The method for obtaining the weight matrix is ​​as follows: The initial attention weights of the initial features of the infrared image are calculated through a convolutional layer with a Sigmoid activation function, and the weight matrix is ​​obtained through the Hadamard product, as shown in the following expression: in, This refers to the initial features of an infrared image. This refers to the weight matrix, ⊙ represents the Hadamard product, and 𝜎 is the Sigmoid function.

4. The infrared-visible image fusion method based on high-order frequency domain interaction according to claim 1, characterized in that, In step 4, the infrared weight matrix is ​​replaced by the weight matrix and interacted with the primary features in the visible light frequency domain. The infrared-visible light fusion features output from the last interaction are element-wise accumulated with the initial features of the visible light image to obtain the visible light frequency domain features.

5. The infrared-visible image fusion method based on high-order frequency domain interaction according to claim 1, characterized in that, The method for obtaining the gating weights is as follows: flatten the initial features of the infrared image and the visible light image, and stitch them together along the channel dimension. After stitching, the images are processed by a multilayer perceptron, and finally the gating weights are obtained by activation through the Sigmoid function.

6. The infrared-visible image fusion method based on high-order frequency domain interaction according to claim 1, characterized in that, The method for obtaining the primary visible light features is as follows: in, Indicates multi-level visible light characteristics, This represents one of the branch features obtained by splicing visible light frequency domain features and infrared frequency domain features. This represents the matrix obtained by concatenating visible light frequency domain features and infrared frequency domain features, followed by global average pooling activation, and is used to interact with another branch. Represents the gating weight, where "L" refers to the number of iterations, and "L" refers to the number of interactions.

7. The infrared-visible image fusion method based on high-order frequency domain interaction according to claim 1, characterized in that, The frequency domain high-order interactive fusion model is constructed by steps 2-6. The loss function used to train the frequency domain high-order interactive fusion model is the total loss function composed of content loss and saliency loss.

8. The infrared-visible image fusion method based on high-order frequency domain interaction according to claim 7, characterized in that, The content loss is a combination of pixel intensity loss and gradient loss, expressed as follows: in, Indicates content loss. Indicates significant loss. and These are the weights for pixel intensity loss and gradient loss. Indicates pixel intensity loss. Indicates gradient loss; The pixel intensity loss is calculated as follows: in, and Indicates the height and width of the image. Describing the L1-norm, Represents the feature matrix of a visible light image. The infrared image feature matrix is ​​obtained by performing shallow feature extraction and normalization on the visible light image and the infrared image respectively using the cross-center difference convolution operator; The gradient loss is calculated as follows: Where ∇(·) is the Sobel gradient operator, This represents the feature matrix of the final fused image.

9. The infrared-visible image fusion method based on high-order frequency domain interaction according to claim 8, characterized in that, The significance loss expression is as follows: in, Indicates the mask weight, This indicates a significant loss.

Images