Multimodal image fusion method based on diffusion model-convolutional neural network

Abstract

The invention discloses a multi-modal image fusion method based on a diffusion model-convolutional neural network. The method comprises the steps of global information modeling, local detail feature extraction, design of a complementary feature aggregation module and design of a diversified loss function. The network model is composed of a diffusion-based branch and a CNN-based branch, in the diffusion branch, noise is introduced into original input X0 according to a proposed time step selection strategy, and intermediate features are extracted in the denoising process to realize global information modeling; in a CNN branch, using a multi-scale convolution kernel and an attention module to extract and integrate local detailed features; then, generating an attention map based on coordinate attention by utilizing a proposed complementary feature aggregation module (CFAM) to capture a long-distance dependency relationship among the features of each branch so as to realize effective aggregation; finally, the aggregation result is input to a fusion head to obtain a fusion result.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and image fusion, and specifically relates to a multimodal image fusion method based on a diffusion model-convolutional neural network. Background Art

[0002] Multimodal image fusion (MMIF) is a technology that integrates images from different sensors or imaging modalities to obtain a richer and more informative fused image. Among the multimodal image fusion methods based on deep learning, convolutional neural networks (CNN) and generative adversarial networks (GAN) are the most commonly used architectures. This technology is widely used in many fields, including but not limited to infrared and visible light fusion (IVF) and medical image fusion (MIF). In IVF, infrared images provide the thermal signature of the target, while visible light images capture rich texture and color information. This fusion technology is crucial in autonomous driving, drone night monitoring, and video surveillance because it can improve the understanding of the scene and enhance the accuracy of target detection and recognition. For example, in autonomous driving, fused images can provide clearer views in low-light environments, helping the driving system make safer decisions. In MIF, different medical imaging technologies (such as MRI, CT, and ultrasound) can provide complementary information. By fusing these images, doctors can obtain a more comprehensive picture of the patient, which is helpful for early diagnosis and personalized treatment. This fusion not only improves the clarity and contrast of the image, but also helps doctors identify tiny lesions, thereby improving the accuracy and efficiency of diagnosis. However, existing technologies still have the following shortcomings: 1) CNN methods are limited by local receptive fields and cannot fully capture cross-modal information and long-range dependencies, resulting in reduced information fidelity; 2) Technology based on generative adversarial networks (GANs) is difficult to train, resulting in modal imbalance and affecting fusion quality.

[0003] In order to achieve higher quality fusion effects and better application value, methods based on diffusion models have been proposed. Diffusion models have demonstrated excellent capabilities in a variety of generation tasks, including text-to-image generation, image-to-image generation, and image restoration. In recent years, these models have gradually been applied to higher-level visual tasks, such as semantic segmentation and object detection, and representative models such as DDPM have made significant progress in this field. At the same time, the application of diffusion models in low-level visual tasks has also received increasing attention, involving techniques such as image restoration, super-resolution, and image fusion. When performing these tasks, a common approach is to first introduce noise to the source image and then use the denoised U-Net structure to extract latent features to support subsequent processing. Although the diffusion model can generate visually attractive fused images, its lack of local perception ability and inherent noise mechanism lead to a significant loss of detail information and loss of original information; and the exploration of effective time step combinations is insufficient, and the features of different denoising time steps are not fully utilized.

[0004] Dif-Fusion improves the color fidelity of infrared and visible light image fusion through a diffusion model, which mainly includes constructing a multi-channel data distribution, extracting multi-channel diffusion features, directly generating a three-channel fused image, and using multi-channel gradient loss and intensity loss to retain texture and intensity information. However, this method has limitations: on the one hand, there is no compensation or restoration method designed for noisy images, resulting in irreversible information loss; on the other hand, although the diffusion model has the ability to capture global macro information, it is not effective in extracting local detail features. Therefore, some subtle local features are omitted in the fused image obtained by the Dif-Fusion method, which affects the overall quality of the fused image.

[0005] DDFM achieves multimodal image fusion by combining the denoising diffusion probability model (DDPM) with hierarchical Bayesian inference. This method first uses the pre-trained DDPM model for unconditional image generation, and then performs likelihood correction through the EM algorithm to retain the cross-modal information in the source image. However, it shows insufficient generalization ability when processing image fusion with complex backgrounds and subtle features, and because the entire back diffusion process needs to be completed, the computing resource requirements are high, which limits its application in resource-constrained environments. Summary of the invention

[0006] In order to make up for the shortcomings of the prior art, the present invention provides a multimodal image fusion method based on diffusion model-convolutional neural network, which aims to extract complementary features through diffusion and CNN branches and effectively aggregate these features.

[0007] The technical problem solved by the present invention can be achieved through the following technical solutions:

[0008] The multimodal image fusion method based on diffusion model-convolutional neural network comprises the following steps:

[0009] S1. Global Information Modeling: By introducing noise and using denoising U-Net to extract multi-scale features, the global information of the image is captured and modeled using a network branch based on a diffusion model;

[0010] S2. Local detail feature extraction: extract multi-scale local detail features through multi-scale convolutional layers and hybrid attention modules, and use multi-scale feature aggregators to enhance hierarchical feature representation;

[0011] S3. Design a complementary feature aggregation module: By calculating the intensity tensor in multiple one-dimensional spatial directions, a weight map based on the coordinate attention mechanism is obtained to capture the long-range dependency between global information and local detail features, and feature aggregation is performed using branch complementarity;

[0012] S4. Design diverse loss functions: Explore the complementarity between features by combining intensity loss, gradient loss and cosine similarity loss.

[0013] Furthermore, in step S1, a single-step denoising-denoising process is performed on the image at several time steps, and the intermediate features of the diffusion model are extracted to characterize the global information. The specific operations are:

[0014] First, in the diffusion branch, Gaussian noise is introduced to the original input X0 through a single-step process to obtain the noisy image X at a specific time step t. t ,Right now:

[0015]

[0016] Among them, ∈ is the noise that satisfies independent Gaussian distribution, is a set of predefined variance parameters;

[0017] Then, the noisy image X t Input into the diffusion model for single-step denoising, the calculation formula is as follows:

[0018]

[0019] Among them, X t-1 is the noise image corresponding to time step t-1, z is a sample from the standard normal distribution, ∈ θ (X t ,t) is the prediction of the real noise, σ t and α tis a predefined parameter; that is, when predicting the image with diffusion time step t-1 based on the image with diffusion time step t, the model captures the global feature information of the image.

[0020] Furthermore, in step S2, the features of each scale are gradually merged through a multi-scale feature aggregator, as follows:

[0021] First, the features of the previous layer are upsampled to match the size of the next layer, and then the scaling factor γ and bias β are generated through the multi-layer perception mechanism to adjust the multi-scale feature ratio. The calculation formula is as follows:

[0022]

[0023] in, represents the feature corresponding to the k+1th convolution layer, ⊙ represents element-by-element multiplication, and γ k is the scaling parameter, β k is the offset parameter. Through this process, the multi-scale local detail features are effectively integrated into the local detail features FC to obtain the multi-scale detail features of the original pure image, which complements the diffusion branch.

[0024] Furthermore, in step S3, the specific workflow of the complementary feature aggregation module is as follows:

[0025] First, extract the one-dimensional orthogonal direction attention tensor F according to the following formula x and F y :

[0026] F x ,F y =XAP(F),YAP(F),

[0027] Among them, XAP and YAP represent average pooling of features along the horizontal and vertical directions, respectively, to obtain the feature saliency information in the corresponding directions;

[0028] Then, the two one-dimensional intensity tensors of size h×1 and 1×w are multiplied to obtain the coordinate attention map matrix W of size h×w, which reflects the long-range dependencies in all directions. CFAM aggregates the global information F through the following formula D and local detail features F C :

[0029] F R =W⊙F C +(1-W)☉F D ,

[0030] Among them, F R Represents the aggregated features, and obtains the coordinate attention map through network training;

[0031] Finally, the aggregated features are fed into the fusion head to generate multimodal image fusion results.

[0032] Furthermore, the loss function in step S4 includes the strength loss L int , gradient loss L grad And cosine similarity loss L CD ,Right now:

[0033] L=L int +αL grad +βL CD ,

[0034] Among them, α and β are the balance factors between the loss terms, and the intensity loss L int Calculate the fusion result image I fused The pixel-by-pixel intensity difference between the two input images I1 and I2 is defined as:

[0035]

[0036] Similarly, the gradient loss L grad It is used to measure the difference in the gradient of the input and output images and is defined as:

[0037]

[0038] In addition, the cosine divergence loss L is introduced CD , to measure the structural differences between branch features, its value range is between [-1,1], 1 means complete similarity, -1 means complete dissimilarity, defined as:

[0039]

[0040] where · represents the vector dot product and ∈ is a small constant to avoid the denominator being zero; minimizing L CD To enlarge F C and F D The structural differences between them.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] (1) The dual-branch framework for extracting diversified image features in the present invention combines the diffusion model and the convolutional neural network. By utilizing the diffusion model's modeling ability for global information and the CNN's sensitivity to local details, a more sophisticated and comprehensive image fusion is achieved, which effectively solves the limitations of a single model in feature extraction and improves the fusion performance of multimodal images.

[0043] (2) The adaptive aggregation module of the present invention selects complementary features and dynamically adjusts the fusion weights of global features and local features by generating a coordinate attention map; it captures long-range dependencies in different directions, thereby optimizing the combination of features and ensuring that the fusion result retains global information while taking into account detail performance, thereby improving the quality and visual effect of the fused image.

[0044] (3) The present invention designs a loss function for unsupervised learning and introduces three loss functions: intensity loss, gradient loss, and cosine similarity loss. The purpose is to comprehensively optimize the image fusion process. By evaluating the consistency between the fused image and the input image in intensity and gradient, as well as the complementarity between branch features, the model effect is effectively optimized without labeled data. This is a key technology for solving the unsupervised image fusion task. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of the multimodal image fusion method of the present invention;

[0046] Figure 2 A flow chart of global information modeling of the present invention;

[0047] Figure 3 It is a flow chart of local detail feature extraction of the present invention;

[0048] Figure 4 A complementary feature aggregation module of the present invention;

[0049] Figure 5 This is a comparison diagram of infrared-visible light fusion visualization results in Example 1 of the present invention;

[0050] Figure 6 This is a comparison chart of the medical image fusion visualization results in Example 2 of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0052] like Figure 1 As shown, a multimodal image fusion method based on diffusion model-convolutional neural network includes global information modeling, local detail feature extraction, design of complementary feature aggregation module, and design of diversified loss function. This embodiment adopts a dual-branch diffusion-convolutional neural network (CNN) framework to achieve comprehensive multimodal image fusion, with infrared visible image fusion (I VF ) task as an example, the visible image V of the RGB channel IS (size is h×w×3) and infrared image I R(size is h×w×1) combined to form the original input X0 (size is h×w×4). The network model consists of two branches, one based on diffusion and the other based on CNN. In the diffusion branch, noise is introduced to the original input X0 according to the proposed time step selection strategy, and intermediate features are extracted during the denoising process to achieve global information modeling; in the CNN branch, multi-scale convolution kernels and attention modules are used to extract and integrate local detailed features; then, the proposed complementary feature aggregation module (CFAM) is used to generate an attention map based on coordinate attention to capture the long-distance dependencies between the features of each branch, thereby achieving effective aggregation; finally, the aggregation result is input to the fusion head to obtain the fusion result. The specific details are as follows:

[0053] (i) Global information modeling: By introducing noise and using denoising U-Net to extract multi-scale features, the global information of the image is captured and modeled using a network branch based on a diffusion model.

[0054] The denoising process of the diffusion model in image fusion tasks is usually very time-consuming, which makes it less than ideal in application scenarios that require efficient processing; and each denoising step involves complex calculations and multiple iterations, which significantly increases the overall processing time. Therefore, for fusion tasks that require fast response and real-time performance, the traditional full-process diffusion model method may not meet the requirements. The high time consumption problem limits the application potential of the diffusion model in real-time image fusion and dynamic scene processing.

[0055] In the denoising process, the diffusion model can model the global information of the image and encapsulate it into the intermediate features. Figure 2 As shown in FIG. 1 , the present invention performs a single-step denoising-denoising process on the image at several time steps, and extracts the intermediate features of the diffusion model to characterize the global information. First, in the diffusion branch, we introduce Gaussian noise to the original input X0 through a single-step process to obtain the noisy image X at a specific time step t. t ,Right now:

[0056]

[0057] Among them, ∈ is the noise that satisfies independent Gaussian distribution, is a set of predefined variance parameters. Next, the noisy image X t Input into the diffusion model and perform single-step denoising (inverse diffusion) processing. The calculation formula is as follows:

[0058]

[0059] Among them, X t-1 is the noise image corresponding to time step t-1, z is a sample from the standard normal distribution, ∈ θ (X t,t) is the prediction of the real noise, σ t and α t is a predefined parameter. When predicting the image with diffusion time step t-1 based on the image with diffusion time step t, the model captures the global feature information of the image. The following cross-time step aggregation module is adopted, using multiple convolution and attention modules to aggregate the intermediate features from different time steps to form the final output of the diffusion branch, which is represented as F D , providing rich feature representation for global information modeling.

[0060] (ii) Local detail feature extraction: Multi-scale local detail features are extracted through multi-scale convolutional layers and hybrid attention modules, and the hierarchical feature representation is enhanced using a multi-scale feature aggregator.

[0061] In the existing fusion methods based on diffusion models, the image is damaged by adding noise, resulting in the loss of detail information and affecting the fusion quality; and the diffusion model does not have a fine receptive field like the convolution kernel, so it is relatively weak in detail extraction ability and cannot fully capture and reconstruct the tiny features in the image.

[0062] The present invention utilizes the excellent local perception ability of convolutional neural network (CNN) to capture detailed features, thus effectively supplementing the global information constructed by diffusion model (DDPM). Figure 3 As shown in the figure, in the CNN branch, multi-scale convolution kernels and hybrid attention modules (MABs) are used to extract multi-scale local detail features, and the features of each scale are gradually merged through the multi-scale feature aggregator (MSFA).

[0063] First, the features of the previous layer are upsampled to match the size of the next layer, and then the scaling factor γ and bias β are generated by the multi-layer perceptron (MLP) to adjust the multi-scale feature ratio. The calculation formula is as follows:

[0064]

[0065] in, represents the feature corresponding to the k+1th convolution layer, ⊙ represents element-by-element multiplication, and γ k is the scaling parameter, β k is the offset parameter. Through this process, multi-scale local detail features are effectively aggregated to F C In the CNN-based network branch, the multi-scale detail features of the original pure image are obtained, which complements the diffusion branch.

[0066] (III) Propose a complementary feature aggregation module: By calculating the intensity tensor in multiple one-dimensional spatial directions, a weight map based on the coordinate attention mechanism is obtained to capture the long-distance dependency between global information and local detail features, and to perform feature aggregation by utilizing branch complementarity.

[0067] The features extracted by the above two branches are complementary, including common features and personalized features. How to effectively mine and make full use of the relationship between them remains a challenge. Among the two common ideas, the method of manually setting thresholds lacks generalization and is difficult to adapt to different application scenarios; while deep learning methods may cause significant changes in the distribution of original feature values, resulting in distortion or overexposure. These problems seriously affect the fusion effect. Therefore, it is necessary to design a method that can adaptively calculate the dynamic proportion of each branch aggregation according to the original attributes of the features, so as to take into account the quality and robustness of multimodal image fusion.

[0068] This paper designs a novel complementary feature aggregation module (CFAM) to effectively aggregate global information F D and local detail features F C CFAM can capture multi-directional long-distance dependencies by generating a coordinate attention map, thereby dynamically adjusting the aggregation weights of the two branch features.

[0069] like Figure 4 As shown in Figure 2, the specific workflow of the complementary feature aggregation module is as follows:

[0070] First, extract the one-dimensional orthogonal direction attention tensor F according to the following formula x and F y :

[0071] F x , F y =XAP(F),YAP(F),

[0072] Among them, XAP and YAP represent the average pooling of features along the horizontal and vertical directions, respectively, so as to obtain the feature saliency information of the corresponding directions; then, the two one-dimensional intensity tensors of size h×1 and 1×w are multiplied to obtain the coordinate attention map matrix of size h×w, which reflects the long-distance dependencies in all directions. Therefore, CFAM aggregates the global information F through the following formula D and local detail features F C :

[0073] F R =W⊙F C +(1-W)☉F D ,

[0074] Among them, F R Represents the aggregated features. The coordinate attention map is obtained through network training. CFAM makes full use of the complementary properties of the dual-branch features and effectively aggregates the global information F D and local detail features F C .

[0075] Finally, the aggregated features are fed into the fusion head to generate multimodal image fusion results.

[0076] (iv) Design diverse loss functions: Explore the complementarity between features by combining strength loss, gradient loss and cosine similarity loss.

[0077] Multimodal image fusion is an unsupervised task, and there is no clear supervisory signal to guide the optimization of the model. Therefore, common loss functions such as mean square error and cross entropy cannot be used for this task, so it is necessary to design a new loss function that takes into account both information fidelity and model working mechanism.

[0078] The diversified loss functions designed by the present invention include: strength loss L int , gradient loss L grad And cosine similarity loss L CD , the overall formula is expressed as:

[0079] L=L int +αL grad +βL CD ,

[0080] Among them, α and β are the balance factors between the various loss terms. Specifically, the strength loss L int Calculate the fusion result image I fused The pixel-by-pixel intensity difference between the two input images I1 and I2 is defined as:

[0081]

[0082] Similarly, the gradient loss Lgra d It is used to measure the difference in the gradient of the input and output images and is defined as:

[0083]

[0084] In addition, the cosine divergence loss L is introduced CD , to measure the structural differences between branch features, its value range is between [-1,1], 1 means complete similarity, -1 means complete dissimilarity, defined as:

[0085]

[0086] where · represents the vector dot product and ∈ is a small constant to avoid the denominator being zero. Minimize L CD Able to enlarge F C and F D , thereby encouraging better exploration of the complementarity between branch features.

[0087] The present invention enhances the model's understanding of features at multiple levels through the design of diversified loss functions, thereby optimizing the quality of the final fusion result.

[0088] The following are two examples of multimodal image fusion methods based on dual-branch diffusion model-convolutional neural network, including infrared-visible light fusion and medical image fusion. In both sets of examples, the same evaluation indicators, environment configuration and comparison model are used:

[0089] (1) Six representative evaluation indicators are used to quantitatively evaluate the fusion performance of the method, including standard deviation (SD), entropy (EN), visual information fidelity (VIF), average gradient (AG), Q AB / F based on edge information, and spatial frequency (SF). The higher the score of these indicators, the better the fusion performance.

[0090] (2) The training process is divided into two stages: in the first stage, only the diffusion model is trained for noise prediction; in the second stage, intermediate features are extracted from the frozen diffusion model using the denoising time step t = [100, 200, 400], which are then used to train other modules. In the training stage, the Adam optimizer is used, the initial learning rate is set to 0.0001, and the batch size is 16; the balance factors α and β in the loss function are set to 1.00 and 0.05 respectively; all experiments are conducted on the NVIDIA GeForce RTX4090 GPU and implemented based on the PyTorch platform.

[0091] (3) The proposed method is compared with seven state-of-the-art image fusion methods, including U2Fusion, SDNet, SwinFusion, TarDAL, CDDFuse, DDFM and Dif-Fusion.

[0092] Example 1

[0093] This example is mainly aimed at infrared-visible light fusion scenarios. It uses the MSRS training set containing 1083 image pairs for training and the MSRS test set containing 361 image pairs for effect evaluation.

[0094] The visualization results are as follows Figure 5As shown in the figure, when using methods such as U2Fusion, SDNet and DDFM, the person in the red frame appears underexposed in the night environment, resulting in a blurred image. In contrast, TarDAL is overexposed. The method of the present invention clearly outlines the contours of the person and maximizes the contrast of the traffic sign in the green frame, improving its readability. It can be seen that the present invention effectively combines the thermal saliency information in the infrared image with the detailed texture in the visible light image to generate a fused image with the best visual effect.

[0095] Table 1 Comparison of infrared-visible light fusion indicators:

[0096] Model\Indicator SD EN VIF AG QAB / F SF U2F 27.71 5.561 0.545 2.899 0.421 9.242 SDN 17.32 5.255 0.489 2.707 0.370 8.691 SwinF 42.98 6.622 0.990 3.564 0.642 11.08 TarD 35.46 6.348 0.673 3.115 0.426 9.873 CDDF 43.38 6.699 1.045 3.748 0.689 11.55 DDFM 43.79 6.171 0.742 2.518 0.473 7.380 DIF 41.90 6.660 0.827 3.889 0.583 11.63 Ours 52.42 6.929 1.062 5.245 0.621 16.01

[0097] Table 1 shows the quantitative comparison results using six evaluation indicators on the MSRS test set. Compared with other advanced methods, the proposed method significantly outperforms other methods. Specifically, the best performance of SD and EN proves that the proposed method is able to integrate the richest original information; at the same time, Q AB / F is maintained at a high level, effectively preserving edge contours; the best VIF result proves that the proposed method provides the most attractive visual effect; in addition, in terms of AG and SF, the proposed results are improved by an average of 34.87% and 37.66%, respectively, verifying that the proposed method extracts the most detailed texture features. The quantitative results prove that the proposed method effectively integrates the saliency information in infrared images with the texture details in visible light images.

[0098] Example 2

[0099] This example is mainly aimed at medical image fusion scenarios. Based on the above pre-trained model, the Harvard Medical School open source dataset MRI-PET is used for testing and evaluation.

[0100] The visualization results are as follows Figure 6 As shown in the figure, U2Fusion, SDNet and DDFM are not good at preserving brightness information, resulting in distortion of important color information from PET. Meanwhile, SwinFusion, TarDAL, CDDFuse and Dif-Fusion lose texture details in MRI images, especially highlighted in the red box. In contrast, the method of the present invention effectively utilizes the rich color information in PET while preserving the clear texture details in MRI, thus achieving the best fusion effect.

[0101] Table 2 Comparison of medical image fusion indicators

[0102] Model\Indicator SD EN VIF AG QAB / F SF U2F 53.35 4.330 0.438 5.607 0.435 19.23 SDN 45.58 4.639 0.474 6.260 0.573 20.52 SwinF 74.34 4.547 <![CDATA[ 0.660 ]]> 6.747 <![CDATA[ 0.645 ]]> 22.19 TarD 57.63 4.695 0.568 5.248 0.481 18.82 CDDF <![CDATA[ 74.36 ]]> 4.196 0.649 <![CDATA[ 6.883 ]]> 0.644 <![CDATA[ 24.62 ]]> DDFM 61.22 3.917 0.652 5.325 0.552 18.87 DIF 70.70 5.115 0.565 6.473 0.589 20.71 Ours 74.86 <![CDATA[ 4.978 ]]> 0.668 7.825 0.699 28.57

[0103] Table 2 shows that the method of the present invention shows excellent results in all indicators when testing the MRI-PET dataset. In particular, the average improvements in AG, SF and Q AB / F were 13.69%, 16.04% and 8.37% respectively, proving that the brain nerves present good contrast with the surrounding environment. In addition, the excellent scores in SD and EN prove the full preservation of the original information, and the excellent performance in the VIF indicator proves the fidelity of the visual information of the fused image, thus providing effective support for medical diagnosis; and without additional fine-tuning, it surpasses other models, demonstrating its excellent generalization performance in medical image fusion tasks.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal image fusion method based on diffusion model-convolutional neural network, characterized in that: The following steps are involved: S1. Global Information Modeling: By introducing noise and using denoising U-Net to extract multi-scale features, the global information of the image is captured and modeled using a network branch based on a diffusion model; S2. Local detail feature extraction: extract multi-scale local detail features through multi-scale convolutional layers and hybrid attention modules, and use multi-scale feature aggregators to enhance hierarchical feature representation; S3. Design a complementary feature aggregation module: By calculating the intensity tensor in multiple one-dimensional spatial directions, a weight map based on the coordinate attention mechanism is obtained to capture the long-range dependency between global information and local detail features, and to perform feature aggregation by utilizing branch complementarity; S4. Design diverse loss functions: Explore the complementarity between features by combining intensity loss, gradient loss and cosine similarity loss.

2. According to claim 1, a multimodal image fusion method based on diffusion model-convolutional neural network is characterized in that: In step S1, a single-step denoising-denoising process is performed on the image at a number of time steps, and the intermediate features of the diffusion model are extracted to characterize the global information. The specific operations are: First, in the diffusion branch, Gaussian noise is introduced into the original input image X0 through a single-step process to obtain the noisy image X at a specific time step t. t ,Right now: Among them, ∈ is the noise that satisfies independent Gaussian distribution, is a set of predefined variance parameters; Then, the noisy image X t Input into the diffusion model for single-step denoising, the calculation formula is as follows: Among them, X t-1 is the noise image corresponding to time step t-1, z is a sample from the standard normal distribution, ∈ θ (X t ,t is the prediction of the real noise, σ t and α t is a predefined parameter, that is, when predicting the image with diffusion time step t-1 based on the image with diffusion time step t, the model captures the global feature information of the image.

3. The multimodal image fusion method based on diffusion model-convolutional neural network according to claim 1, characterized in that: In step S2, the features of each scale are gradually merged through a multi-scale feature aggregator, as follows: First, the features of the previous layer are upsampled to match the size of the next layer, and then the scaling factor γ and bias β are generated through the multi-layer perception mechanism to adjust the multi-scale feature ratio. The calculation formula is as follows: in, represents the feature corresponding to the k+1th convolution layer, ⊙ represents element-by-element multiplication, and γ k is the scaling parameter, β k is the offset parameter. Through this process, the multi-scale local detail features are effectively fused into the local detail features F C In the process, the multi-scale detail features of the original pure image are obtained, which complements the diffusion branch.

4. The multimodal image fusion method based on diffusion model-convolutional neural network according to claim 1, characterized in that: In step S3, the specific working process of the complementary feature aggregation module is as follows: First, extract the one-dimensional orthogonal direction attention tensor F of the branch feature F according to the following formula x and F y : F x ,F y =XAP(F),YAP(F), Among them, XAP and YAP represent average pooling of features along the horizontal and vertical directions, respectively, to obtain the feature saliency information in the corresponding directions; Then, the two one-dimensional intensity tensors of size h×1 and 1×w are multiplied to obtain the coordinate attention map matrix W of size h×w, which reflects the long-range dependencies in all directions. CFAM aggregates the global information F through the following formula D and local detail features F C : F R =W⊙F C +(1-W)⊙F D , Among them, F R Represents the aggregated features, and obtains the coordinate attention map through network training; Finally, the aggregated features are fed into the fusion head to generate multimodal image fusion results.

5. The multimodal image fusion method based on diffusion model-convolutional neural network according to claim 1, characterized in that: The loss function in step S4 includes the intensity loss L int , gradient loss L grad And cosine similarity loss L CD ,Right now: L=L int +αL grad +βL CD , Among them, α and β are the balance factors between the loss terms, and the intensity loss L int Calculate the fusion result image I fused The pixel-by-pixel intensity difference between the two input images I1 and I2 is defined as: Similarly, the gradient loss L grad It is used to measure the difference in the gradient of the input and output images and is defined as: In addition, the cosine divergence loss L is introduced CD , to measure the structural differences between branch features, its value range is between [-1,1], 1 means complete similarity, -1 means complete dissimilarity, defined as: where · represents the vector dot product and ∈ is a small constant to avoid the denominator being zero; minimizing L CD To enlarge F C and F D The structural differences between them.

Images