A multi-modal image fusion method based on diffusion model-convolutional neural network

Abstract

This invention discloses a multimodal image fusion method based on a diffusion model-convolutional neural network, including global information modeling, local detail feature extraction, design of a complementary feature aggregation module, and design of diverse loss functions. The network model consists of two branches: a diffusion-based branch and a CNN-based branch. In the diffusion branch, noise is introduced into the original input X0 according to the proposed time step selection strategy, and intermediate features are extracted during the denoising process to achieve global information modeling. In the CNN branch, multi-scale convolutional kernels and attention modules are used to extract and integrate local detail features. Subsequently, the proposed complementary feature aggregation module (CFAM) is used to generate an attention map based on coordinate attention to capture long-distance dependencies between features of each branch, thereby achieving effective aggregation. Finally, the aggregation result is input into the fusion head to obtain the fusion result.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning and image fusion technology, specifically relating to a multimodal image fusion method based on a diffusion model-convolutional neural network. Background Technology

[0002] Multimodal image fusion (MMIF) is a technique that integrates images from different sensors or imaging modalities to obtain a richer, more informative fused image. Among deep learning-based MMIF methods, convolutional neural networks (CNNs) and generative adversarial networks (GANs) are the most commonly used architectures. This technology is widely applied in various fields, including but not limited to infrared and visible light fusion (IVF) and medical image fusion (MIF). In IVF, infrared images provide the thermal features of the target, while visible light images capture rich texture and color information. This fusion technique is crucial in autonomous driving, drone nighttime surveillance, and video surveillance because it enhances scene understanding and improves the accuracy of target detection and recognition. For example, in autonomous driving, the fused image can provide a clearer view in low-light environments, helping the driving system make safer decisions. In MIF, different medical imaging techniques (such as MRI, CT, and ultrasound) can provide complementary information. By fusing these images, doctors can obtain a more comprehensive understanding of the patient, facilitating early diagnosis and personalized treatment. This fusion not only improves image clarity and contrast but also helps doctors identify minute lesions, thereby improving diagnostic accuracy and efficiency. However, existing technologies still have the following drawbacks: 1) CNN methods are limited by local receptive fields and cannot fully capture cross-modal information and long-range dependencies, resulting in a decrease in information fidelity; 2) Generative Adversarial Network (GAN) based technologies are difficult to train, leading to modal imbalance and affecting fusion quality.

[0003] To achieve higher quality fusion results and better application value, diffusion models have been proposed. Diffusion models have demonstrated excellent capabilities in various generative tasks, including text-to-image generation, image-to-image generation, and image inpainting. In recent years, these models have been increasingly applied to higher-level visual tasks, such as semantic segmentation and object detection, with representative models like DDPM achieving significant progress in this area. Simultaneously, the application of diffusion models in low-level visual tasks is also attracting increasing attention, involving techniques such as image restoration, super-resolution, and image fusion. A common approach to these tasks is to first introduce noise into the source image and then use a denoising U-Net structure to extract latent features to support subsequent processing. Although diffusion models can generate visually appealing fused images, their limited local perceptual capabilities and inherent noise-adding mechanism lead to a significant loss of detail and original information; furthermore, insufficient exploration of effective time step combinations fails to fully utilize the features of different denoising time steps.

[0004] Dif-Fusion improves color fidelity in infrared and visible light image fusion through a diffusion model. This primarily involves constructing a multi-channel data distribution, extracting multi-channel diffusion features, directly generating a three-channel fused image, and employing multi-channel gradient loss and intensity loss to preserve texture and intensity information. However, this method has limitations: firstly, it lacks compensation or recovery mechanisms for noisy images, leading to irreversible information loss; secondly, while the diffusion model has the ability to capture global macroscopic information, its extraction of local detail features is poor. Therefore, the fused image obtained by the Dif-Fusion method misses some subtle local features, thus affecting the overall quality of the fused image.

[0005] DDFM achieves multimodal image fusion by combining a denoising diffusion probability model (DDPM) with hierarchical Bayesian inference. This method first uses a pre-trained DDPM model for unconditional image generation, and then performs likelihood correction using the EM algorithm to preserve cross-modal information in the source image. However, it exhibits insufficient generalization ability when processing images with complex backgrounds and subtle features, and its high computational resource requirements due to the need to complete the entire back-diffusion process limit its application in resource-constrained environments. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a multimodal image fusion method based on a diffusion model-convolutional neural network, which aims to extract complementary features through diffusion and CNN branches and effectively aggregate these features.

[0007] The technical problem solved by this invention can be achieved through the following technical solutions:

[0008] The multimodal image fusion method based on diffusion model-convolutional neural network includes the following steps:

[0009] S1. Global Information Modeling: By introducing noise and extracting multi-scale features using a denoising U-Net, global information of the image is captured and modeled using a network branch based on a diffusion model.

[0010] S2. Local detail feature extraction: Multi-scale local detail features are extracted through multi-scale convolutional layers and hybrid attention modules, and the hierarchical feature representation is enhanced by a multi-scale feature aggregator;

[0011] S3. Design complementary feature aggregation module: By calculating the intensity tensor in multiple one-dimensional spatial directions, a weight map based on the coordinate attention mechanism is obtained, which captures the long-distance dependency between global information and local detailed features, and uses branch complementarity to perform feature aggregation;

[0012] S4. Design diverse loss functions: Explore the complementarity between features by combining intensity loss, gradient loss, and cosine similarity loss.

[0013] Furthermore, in step S1, a single-step noise addition-denoising process is performed on the image at several time steps, and intermediate features of the diffusion model are extracted to represent global information. Specifically, the operation is as follows:

[0014] First, in the diffusion branch, Gaussian noise is introduced into the original input X0 through a single-step process to obtain the noise image X at a specific time step t. t ,Right now:

[0015]

[0016] Where ∈ represents noise that follows an independent Gaussian distribution. It is a set of predefined variance parameters;

[0017] Then, X with noise t The input is fed into the diffusion model for single-step denoising processing, and the calculation formula is as follows:

[0018]

[0019] Among them, X t-1 This is the noise image at time step t-1, where z is a sample from a standard normal distribution, ∈ θ (X t ,t) is the prediction for the actual noise, σ t and α tThese are predefined parameters; that is, when predicting an image with a diffusion time step of t-1 based on an image with a diffusion time step of t, the model captures the global feature information of the image.

[0020] Furthermore, in step S2, features from various scales are gradually merged using a multi-scale feature aggregator, as detailed below:

[0021] First, the features of the previous layer are upsampled to match the size of the next layer. Then, a scaling factor γ and a bias β are generated through a multilayer perceptron mechanism to adjust the multi-scale feature ratio. The calculation formula is as follows:

[0022]

[0023] in, Represents the features corresponding to the (k+1)th convolutional layer, ⊙ denotes element-wise multiplication, and γ k It is the scaling parameter, β k It is the offset parameter. Through this process, multi-scale local detail features are effectively fused into the local detail feature FC to obtain the multi-scale detail features of the original pure image, which complements the diffusion branch.

[0024] Furthermore, in step S3, the specific workflow of the complementary feature aggregation module is as follows:

[0025] First, extract the one-dimensional orthogonal direction attention tensor F according to the following formula. x and F y :

[0026] F x ,F y =XAP(F),YAP(F),

[0027] Wherein, XAP and YAP represent average pooling of features along the horizontal and vertical directions, respectively, to obtain feature saliency information in the corresponding directions;

[0028] Then, the two one-dimensional intensity tensors of size h×1 and 1×w are multiplied to obtain a coordinate attention map matrix W of size h×w. This map reflects the long-range dependencies in all directions. CFAM aggregates global information F using the following formula. D and local detail features F C :

[0029] F R =W⊙F C +(1-W)☉F D ,

[0030] Among them, F R The aggregated features are represented by a coordinate attention map obtained through network training.

[0031] Finally, the aggregated features are fed into the fusion head to generate a multimodal image fusion result.

[0032] Furthermore, the loss function in step S4 includes intensity loss L. int Gradient loss L grad And cosine similarity loss L CD ,Right now:

[0033] L = L int +αL grad +βL CD ,

[0034] Where α and β are the balance factors among the various loss terms, and the intensity loss L int Calculate the fusion result image I fused The pixel-wise intensity difference between the two input images I1 and I2 is defined as:

[0035]

[0036] Similarly, gradient loss L grad Used to measure the difference in gradients between input and output images, defined as:

[0037]

[0038] In addition, a cosine divergence loss L is introduced. CD To measure the structural differences between branch features, its value ranges from [-1, 1], where 1 indicates complete similarity and -1 indicates complete dissimilarity, and is defined as:

[0039]

[0040] Where · represents the vector dot product, ∈ is a small constant used to avoid the denominator being zero; minimizing L CD To amplify F C and F D Structural differences between them.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] (1) The present invention provides a dual-branch framework for extracting diverse image features, which combines a diffusion model and a convolutional neural network. By utilizing the modeling ability of the diffusion model for global information and the sensitivity of CNN to local details, it achieves more refined and comprehensive image fusion, effectively solving the limitations of a single model in feature extraction and improving the fusion performance of multimodal images.

[0043] (2) The adaptive aggregation module of the present invention filters complementary features and dynamically adjusts the fusion weights of global and local features by generating coordinate attention maps; it captures long-range dependencies in different directions, thereby optimizing the combination of features and ensuring that the fusion result retains global information while taking into account details, thus improving the quality and visual effect of the fused image.

[0044] (3) This invention designs an unsupervised learning loss function, introducing three loss functions: intensity loss, gradient loss, and cosine similarity loss. It aims to comprehensively optimize the image fusion process. By evaluating the consistency of the fused image and the input image in terms of intensity and gradient, as well as the complementarity between branch features, it effectively optimizes the model performance without labeled data. This is a key technology for solving unsupervised image fusion tasks. Attached Figure Description

[0045] Figure 1 This is a flowchart of the multimodal image fusion method of the present invention;

[0046] Figure 2 This is a flowchart of the global information modeling process of the present invention;

[0047] Figure 3 This is a flowchart of the local detail feature extraction process of the present invention;

[0048] Figure 4 This is the complementary feature aggregation module of the present invention;

[0049] Figure 5 This is a comparison chart of the infrared-visible light fusion visualization results in Embodiment 1 of the present invention;

[0050] Figure 6 This is a comparison chart of the visualization results of medical image fusion in Embodiment 2 of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0052] like Figure 1 As shown, a multimodal image fusion method based on a diffusion model-convolutional neural network includes global information modeling, local detail feature extraction, design of a complementary feature aggregation module, and design of diverse loss functions. This embodiment employs a dual-branch diffusion-convolutional neural network (CNN) framework to achieve comprehensive multimodal image fusion, using infrared-visible image fusion (I2N) as an example. VF Taking the task as an example, the visible image V of the RGB channels IS (Dimensions are h×w×3) and infrared image I R(Size h×w×1) are combined to form the original input X0 (size h×w×4). The network model consists of two branches: a diffusion-based branch and a CNN-based branch. In the diffusion branch, noise is introduced into the original input X0 according to the proposed time step selection strategy, and intermediate features are extracted during the denoising process to achieve global information modeling. In the CNN branch, multi-scale convolutional kernels and attention modules are used to extract and integrate local detailed features. Subsequently, the proposed complementary feature aggregation module (CFAM) is used to generate an attention map based on coordinate attention to capture long-distance dependencies between features of each branch, thereby achieving effective aggregation. Finally, the aggregation result is input into the fusion head to obtain the fusion result. Details are as follows:

[0053] (i) Global information modeling: By introducing noise and extracting multi-scale features using a denoising U-Net, the global information of the image is captured and modeled using a network branch based on a diffusion model.

[0054] The denoising process of diffusion models in image fusion tasks is typically very time-consuming, making them less than ideal for applications requiring high efficiency. Furthermore, each denoising step involves complex computations and multiple iterations, significantly increasing the overall processing time. Therefore, traditional end-to-end diffusion model methods may not meet the requirements for fusion tasks demanding fast response and real-time performance. This high time consumption limits the application potential of diffusion models in real-time image fusion and dynamic scene processing.

[0055] Because during the denoising process, the diffusion model can model the global information of the image and encapsulate it into intermediate features. For example... Figure 2 As shown, this invention performs a single-step noise addition-denoising process on the image at several time steps, and extracts intermediate features of the diffusion model to characterize global information. First, in the diffusion branch, we introduce Gaussian noise into the original input X0 through a single-step process to obtain a noisy image X at a specific time step t. t ,Right now:

[0056]

[0057] Where ∈ represents noise that follows an independent Gaussian distribution. It is a set of predefined variance parameters. Next, the image X with noise will be... t The input is fed into the diffusion model for single-step denoising (reverse diffusion) processing. The calculation formula is as follows:

[0058]

[0059] Among them, X t-1 This is the noise image at time step t-1, where z is a sample from a standard normal distribution, ∈ θ (X t,t) is the prediction for the actual noise, σ t and α t These are predefined parameters. When predicting an image at diffusion time step t-1 based on an image at diffusion time step t, the model captures global feature information of the image. The following cross-time step aggregation module is used, employing multiple convolutional and attention modules to aggregate intermediate features from different time steps, forming the final output of the diffusion branch, represented as F. D This provides rich feature representations for global information modeling.

[0060] (ii) Local detail feature extraction: Multi-scale local detail features are extracted through multi-scale convolutional layers and hybrid attention modules, and the hierarchical feature representation is enhanced by multi-scale feature aggregator.

[0061] In existing diffusion-based fusion methods, images are damaged by added noise, resulting in the loss of detail information and affecting the fusion quality. Furthermore, diffusion models do not have a fine receptive field similar to convolutional kernels, so their ability to extract details is relatively weak, and they cannot fully capture and reconstruct the subtle features in the image.

[0062] This invention utilizes the excellent local perception capabilities of convolutional neural networks (CNNs) to capture detailed features, thus effectively supplementing the global information constructed by the diffusion model (DDPM). Figure 3 As shown, in the CNN branch, multi-scale convolutional kernels and hybrid attention modules (MABs) are used to extract multi-scale local detail features, and features at each scale are gradually merged through a multi-scale feature aggregator (MSFA).

[0063] First, the features of the previous layer are upsampled to match the size of the next layer. Then, a scaling factor γ and a bias β are generated by a multilayer perceptron (MLP) to adjust the multi-scale feature ratio. The calculation formula is as follows:

[0064]

[0065] in, Represents the features corresponding to the (k+1)th convolutional layer, ⊙ denotes element-wise multiplication, and γ k It is the scaling parameter, β k It is the offset parameter. Through this process, multi-scale local detail features are effectively aggregated into F. C This enhances the richness and accuracy of feature representation. In the CNN-based network branch, multi-scale detail features of the original clean image are obtained, complementing the diffusion branch.

[0066] (III) Propose a complementary feature aggregation module: By calculating the intensity tensor in multiple one-dimensional spatial directions, a weight map based on the coordinate attention mechanism is obtained, which captures the long-distance dependency between global information and local detailed features, and uses branch complementarity to perform feature aggregation.

[0067] The features extracted by the two branches mentioned above are complementary, including common and personalized features. Effectively mining and fully utilizing the relationship between them remains a challenge. Of the two common approaches, manually setting thresholds lacks generalization and is difficult to adapt to different application scenarios; while deep learning methods may cause significant changes in the distribution of the original feature values, resulting in distortion or overexposure. These problems severely affect the fusion effect. Therefore, it is necessary to design a method that can adaptively calculate the dynamic ratio of each branch aggregation based on the original attribute of the features, thereby balancing the quality and robustness of multimodal image fusion.

[0068] This invention designs a novel Complementary Feature Aggregation Module (CFAM) to effectively aggregate global information F. D and local detail features F C CFAM can capture long-distance dependencies in multiple directions by generating coordinate attention maps, thereby dynamically adjusting the aggregation weights of features from two branches.

[0069] like Figure 4 As shown, the specific workflow of the complementary feature aggregation module is as follows:

[0070] First, extract the one-dimensional orthogonal direction attention tensor F according to the following formula. x and F y :

[0071] F x F y =XAP(F),YAP(F),

[0072] Here, XAP and YAP represent average pooling of features along the horizontal and vertical directions, respectively, to obtain feature saliency information in the corresponding directions. Then, two one-dimensional intensity tensors of size h×1 and 1×w are multiplied to obtain a coordinate attention map matrix of size h×w, which reflects long-range dependencies in all directions. Therefore, CFAM aggregates global information F using the following formula. D and local detail features F C :

[0073] F R =W⊙F C +(1-W)☉F D ,

[0074] Among them, F R Representing aggregated features, a coordinate attention map is obtained through network training. CFAM fully utilizes the complementary properties of bi-branch features to effectively aggregate global information F. D and local detail features F C .

[0075] Finally, the aggregated features are fed into the fusion head to generate a multimodal image fusion result.

[0076] (iv) Designing diverse loss functions: By combining intensity loss, gradient loss and cosine similarity loss, we can explore the complementarity between features.

[0077] Multimodal image fusion is an unsupervised task, lacking explicit supervisory signals to guide model optimization. Therefore, common loss functions such as mean squared error and cross-entropy are unsuitable for this task, necessitating the design of novel loss functions that balance information fidelity and model operating mechanisms.

[0078] The diverse loss functions designed in this invention include: intensity loss L int Gradient loss L grad And cosine similarity loss L CD The overall formula is expressed as:

[0079] L = L int +αL grad +βL CD ,

[0080] Here, α and β are the balancing factors among the various loss terms. Specifically, the intensity loss L int Calculate the fusion result image I fused The pixel-wise intensity difference between the two input images I1 and I2 is defined as:

[0081]

[0082] Similarly, gradient loss Lgra d Used to measure the difference in gradients between input and output images, defined as:

[0083]

[0084] In addition, a cosine divergence loss L is introduced. CD To measure the structural differences between branch features, its value ranges from [-1, 1], where 1 indicates complete similarity and -1 indicates complete dissimilarity, and is defined as:

[0085]

[0086] Where · represents the vector dot product, and ∈ is a small constant used to avoid a denominator of zero. Minimize L CD Able to amplify F C and F D The structural differences between them encourage a better exploration of the complementarity between branch features.

[0087] This invention enhances the model's understanding of features at multiple levels through the design of diverse loss functions, thereby optimizing the quality of the final fusion result.

[0088] Below are two examples of multimodal image fusion methods based on a dual-branch diffusion model-convolutional neural network, including infrared-visible light fusion and medical image fusion. The same evaluation metrics, environment configurations, and comparison models were used in both sets of examples.

[0089] (1) Six representative evaluation metrics were used to quantitatively evaluate the fusion performance of the method, including standard deviation (SD), entropy (EN), visual information fidelity (VIF), average gradient (AG), edge-based Q AB / F and spatial frequency (SF). The higher the score of these metrics, the better the fusion performance.

[0090] (2) The training process is divided into two stages: In the first stage, only the diffusion model is trained for noise prediction; in the second stage, intermediate features are extracted from the frozen diffusion model using a denoising time step t = [100, 200, 400], and these features are then used to train other modules. During the training stage, the Adam optimizer is used with an initial learning rate of 0.0001 and a batch size of 16; the balance factors α and β in the loss function are set to 1.00 and 0.05, respectively; all experiments are performed on an NVIDIA GeForce RTX4090 GPU and implemented on the PyTorch platform.

[0091] (3) The method of the present invention is compared with seven state-of-the-art image fusion methods, including U2Fusion, SDNet, SwinFusion, TarDAL, CDDFuse, DDFM and Dif-Fusion.

[0092] Example 1

[0093] This example is mainly aimed at infrared-visible light fusion scenarios. It uses an MSRS training set containing 1083 image pairs for training and an MSRS test set containing 361 image pairs for performance evaluation.

[0094] Visualization results as follows Figure 5As shown, when using methods such as U2Fusion, SDNet, and DDFM, the person in the red box appears underexposed in nighttime environments, resulting in a blurry image. In contrast, TarDAL exhibits overexposure. The method of this invention clearly outlines the person's contours and maximizes the contrast of the traffic sign within the green box, improving its readability. Therefore, this invention effectively combines the thermal saliency information from infrared images with the detailed texture from visible light images, generating a fused image with optimal visual effects.

[0095] Table 1 Comparison of Infrared-Visible Light Fusion Indicators:

[0096] Models / Indicators SD EN VIF AG QAB / F SF U2F 27.71 5.561 0.545 2.899 0.421 9.242 SDN 17.32 5.255 0.489 2.707 0.370 8.691 SwinF 42.98 6.622 0.990 3.564 0.642 11.08 TarD 35.46 6.348 0.673 3.115 0.426 9.873 CDDF 43.38 6.699 1.045 3.748 0.689 11.55 DDFM 43.79 6.171 0.742 2.518 0.473 7.380 DIF 41.90 6.660 0.827 3.889 0.583 11.63 Ours 52.42 6.929 1.062 5.245 0.621 16.01

[0097] Table 1 presents the quantitative comparison results using six evaluation metrics on the MSRS test set. Compared with other state-of-the-art methods, our method significantly outperforms them. Specifically, the best performance in SD and EN demonstrates that our method can integrate the richest original information; simultaneously, QAB / F remains at a high level, effectively preserving edge contours; the best VIF result proves that our method provides the most appealing visual effect; furthermore, in AG and SF, our method achieves average improvements of 34.87% and 37.66%, respectively, validating that our method extracts the most detailed texture features. Quantitative results demonstrate that our invention effectively integrates salient information from infrared images with texture details from visible light images.

[0098] Example 2

[0099] This example is primarily geared towards medical image fusion scenarios. Based on the pre-trained model described above, it is tested and evaluated using the Harvard Medical School open-source dataset MRI-PET.

[0100] Visualization results as follows Figure 6 As shown, U2Fusion, SDNet, and DDFM are insufficient in preserving luminance information, leading to distortion of important color information from PET. Meanwhile, SwinFusion, TarDAL, CDDFuse, and Dif-Fusion lose texture details in MRI images, as particularly emphasized in the red box. In contrast, the method of this invention effectively utilizes the rich color information in PET while preserving clear texture details in MRI, thus achieving the best fusion effect.

[0101] Table 2 Comparison of Medical Image Fusion Indicators

[0102] Models / Indicators SD EN VIF AG QAB / F SF U2F 53.35 4.330 0.438 5.607 0.435 19.23 SDN 45.58 4.639 0.474 6.260 0.573 20.52 SwinF 74.34 4.547 <![CDATA[ 0.660 ]]> 6.747 <![CDATA[ 0.645 ]]> 22.19 TarD 57.63 4.695 0.568 5.248 0.481 18.82 CDDF <![CDATA[ 74.36 ]]> 4.196 0.649 <![CDATA[ 6.883 ]]> 0.644 <![CDATA[ 24.62 ]]> DDFM 61.22 3.917 0.652 5.325 0.552 18.87 DIF 70.70 5.115 0.565 6.473 0.589 20.71 Ours 74.86 <![CDATA[ 4.978 ]]> 0.668 7.825 0.699 28.57

[0103] Table 2 shows that the method of this invention exhibits superior performance across all metrics when tested on MRI-PET datasets. Specifically, it improves performance by an average of 13.69%, 16.04%, and 8.37% in AG, SF, and Q AB / F, respectively, demonstrating good contrast between brain neurons and their surrounding environment. Furthermore, the excellent scores in SD and EN demonstrate sufficient preservation of original information, and the superior performance on the VIF metric proves the fidelity of visual information in the fused image, thus providing effective support for medical diagnosis. Moreover, it outperforms other models without additional fine-tuning, showcasing its superior generalization performance in medical image fusion tasks.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal image fusion method based on a diffusion model-convolutional neural network, characterized in that, Includes the following steps: S1. Global Information Modeling: By introducing noise and extracting multi-scale features using a denoising U-Net, the global information of the image is captured and modeled using a network branch based on a diffusion model. At several time steps, a single-step noise addition-denoising process is performed on the image, and the intermediate features of the diffusion model are extracted to represent the global information. The specific operations are as follows: First, in the diffusion branch, Gaussian noise is introduced into the original input image X0 through a single-step process to obtain the noisy image X at a specific time step t. t ,Right now: Where ∈ represents noise that follows an independent Gaussian distribution. It is a set of predefined variance parameters; Then, X with noise t The input is fed into the diffusion model for single-step denoising processing, and the calculation formula is as follows: Among them, X t-1 This is the noise image at time step t-1, where z is a sample from a standard normal distribution, ∈ θ (X t ,t) is the prediction for the actual noise, σ t and α t These are predefined parameters, meaning that when predicting an image with a diffusion time step of t-1 based on an image with a diffusion time step of t, the model captures the global feature information of the image. S2. Local Detail Feature Extraction: Multi-scale local detail features are extracted through multi-scale convolutional layers and a hybrid attention module, and a multi-scale feature aggregator is used to enhance the hierarchical feature representation. Features from various scales are gradually merged through the multi-scale feature aggregator, as follows: First, the features of the previous layer are upsampled to match the size of the next layer. Then, a scaling factor γ and a bias β are generated through a multilayer perceptron mechanism to adjust the multi-scale feature ratio. The calculation formula is as follows: in, Represents the features corresponding to the (k+1)th convolutional layer, ⊙ denotes element-wise multiplication, and γ k It is the scaling parameter, β k It is the offset parameter. Through this process, multi-scale local detail features are effectively fused into the local detail feature F. C In this process, multi-scale detail features of the original clean image are obtained, which complement the diffusion branch; S3. Design of the complementary feature aggregation module: By calculating the intensity tensor in multiple one-dimensional spatial directions, a weight map based on the coordinate attention mechanism is obtained, capturing the long-distance dependency between global information and local detailed features, and using branch complementarity to perform feature aggregation; the specific workflow of the complementary feature aggregation module is as follows: First, extract the one-dimensional orthogonal direction attention tensor F of the branch features F according to the following formula. x and F y : F x ,Fy=XAP(F),YAP(F), Wherein, XAP and YAP represent average pooling of features along the horizontal and vertical directions, respectively, to obtain feature saliency information in the corresponding directions; Then, the two one-dimensional intensity tensors of size h×1 and 1×w are multiplied to obtain a coordinate attention map matrix W of size h×w. This map reflects the long-range dependencies in all directions. CFAM aggregates global information F using the following formula. D and local detail features F C : F R =W⊙F C +(1-W)⊙F D , Among them, F R The aggregated features are represented by a coordinate attention map obtained through network training. Finally, the aggregated features are fed into the fusion head to generate a multimodal image fusion result; S4. Design diverse loss functions: Explore the complementarity between features by combining intensity loss, gradient loss, and cosine similarity loss.

2. The multimodal image fusion method based on diffusion model-convolutional neural network according to claim 1, characterized in that, The loss function in step S4 includes intensity loss L. int Gradient loss L grad And cosine similarity loss L CD ,Right now: L=L int +αL grad +βL CD , Where α and β are the balance factors among the various loss terms, and the intensity loss L int Calculate the fusion result image I fused The pixel-wise intensity difference between the two input images I1 and I2 is defined as: Similarly, gradient loss L grad Used to measure the difference in gradients between input and output images, defined as: In addition, a cosine divergence loss L is introduced. CD To measure the structural differences between branch features, its value ranges from [-1, 1], where 1 indicates complete similarity and -1 indicates complete dissimilarity, and is defined as: Where · represents the vector dot product, ∈ is a small constant used to avoid the denominator being zero; minimizing L CD To amplify F C and F D Structural differences between them.

Images