Multi-modal image fusion method and system based on double-flow semantic alignment diffusion model
By proposing a multimodal image fusion method based on a dual-stream semantic alignment diffusion model, this method addresses the problems of unutilized modal characteristic differences and insufficient adaptability in existing technologies. It achieves high-quality, low-complexity multimodal image fusion and is suitable for real-time fusion in scenarios such as infrared-visible light images.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIAMEN UNIV OF TECH
- Filing Date
- 2026-04-08
- Publication Date
- 2026-05-29
Smart Images
Figure CN122115234A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision and image processing technology, specifically to a multimodal image fusion method and system based on a two-stream semantic alignment diffusion model. Background Technology
[0002] Multimodal image fusion aims to integrate complementary information from different modalities into a unified image, simultaneously preserving key information such as brightness, texture, structure, and semantics. Typical tasks include infrared and visible light fusion, MRI and PET fusion, and low-light enhancement and remote sensing data fusion. Its core objective is to improve visual readability while enhancing the accuracy of subsequent tasks (such as detection, recognition, and segmentation).
[0003] Existing fusion methods mainly include traditional transform domain methods, convolutional neural network (CNN) methods, generative model methods, and Transformer-based methods. These methods generally suffer from the following problems: 1. Modal characteristics differences are not fully utilized: Many existing methods use a "uniform encoder" to process different modes, ignoring the differences in statistical distribution between modes.
[0004] 2. Lack of adaptability in fusion mechanisms: Many deep fusion methods still rely on fixed rules, such as weighted averaging, maximum value selection, or simple concatenation. This leads to unstable performance of fusion based on fixed weights in different scenarios, making it difficult to dynamically adjust for the target and background regions.
[0005] Diffusion models have demonstrated advantages in image generation in recent years, including stability, high quality, and strong controllability. Therefore, their application in multimodal image fusion has attracted considerable attention. Current research on diffusion models in image fusion primarily involves combining two modalities of information in a multi-channel manner, using this as conditional input to guide the diffusion model in performing forward denoising and backward denoising to learn the generated distribution of the fused image; and introducing likelihood correction in the backward denoising stage. However, current research focuses on the combined implementation of "multi-channel diffusion modeling / correction and frequency domain detail enhancement," without considering the need for "adaptive fusion control while ensuring detail fidelity and energy saliency, and also considering computational efficiency." Therefore, there is still room for further improvement. Summary of the Invention
[0006] The purpose of this invention is to overcome the problems existing in current multimodal image fusion methods. To this end, this invention proposes a multimodal image fusion method based on a two-stream semantic alignment diffusion model, comprising the following steps: S1, acquire the multimodal image to be fused, and after concatenating the multimodal image in the channel dimension, input it into the semantic encoder to generate a fusion-guided semantic prior feature map S; S2, a fusion supervision signal is generated from the multimodal image using a pseudo-fusion function, and then the fusion supervision signal is input into the forward noise addition formula of the diffusion model to obtain a noisy fused image. ; S3, the noise-added fusion image After feature concatenation with the semantic feature map S, the input is given to the corresponding modality branch flow network to output the noise prediction result of the corresponding modality; wherein, the branch flow network is a conditional U-Net network with time step encoding; S4. Based on the noise predictions for each mode and their corresponding weighting coefficients, the final noise prediction results are generated. S5, based on the final noise prediction result and the noise fusion map The final fused image F is obtained by reconstructing the image based on the inverse process of the diffusion model.
[0007] Preferably, in step S1, the multimodal image includes an infrared image of the first modality. Visible light image with second mode The semantic encoder includes three consecutive 3×3 convolutional layers and a SiLU activation function. The three convolutional layers output feature maps with 32, 64, and 32 channels, respectively. The SiLU activation function is used to perform nonlinear modulation on the feature maps output by the convolutional layers to enhance the semantic encoder's ability to express complex structural features and improve the stability of the network training process.
[0008] Preferably, in step S2, the monitoring signal is fused. Represented as:
[0009] in, Indicates taking The maximum value in; The noise fusion map is obtained using the forward noise addition formula of the diffusion model. x t The process is represented as:
[0010] in, This represents the signal weighting coefficients that vary with the diffusion time step. Indicated by the The signal accumulation coefficient is calculated by the cumulative product. It is real random noise.
[0011] Preferably, in step S3, the branching flow network includes an infrared flow network and a visible light flow network. The infrared flow network is biased towards the energy saliency of the infrared mode; the visible light flow network is biased towards the texture structure of the visible light mode; the input of each branching flow network includes a noisy fusion map of dimension 1×H×W. The input tensor is (1+C)×H×W, formed by concatenating a semantic feature map S of dimension C×H×W through channels, along with time-step encoding; the output is the noise prediction result of the first modality. Noise prediction results for the second mode Where C represents the number of semantic feature channels, and H and W represent the height and width of the input multimodal image, respectively.
[0012] Preferably, in step S4: The noise prediction result of the first mode is obtained through a learnable Softmax noise fusion module. weight Noise prediction results for the second mode weight ;in:
[0013] and The learnable parameters corresponding to the noise of the first mode and the noise of the second mode are respectively, and the final noise prediction result is given. Represented as: .
[0014] Preferably, in step S5, the inverse process formula of the diffusion model is:
[0015] The final noise prediction result obtained using S4 As real random noise Substituting into the above equation and solving, we get Let F = The final fused image F is obtained through reconstruction.
[0016] Preferred options also include: A fusion loss is used to perform end-to-end optimization of the overall model, so that it simultaneously preserves the energy information of infrared light and the texture structure of visible light; wherein, the fusion loss includes: Noise prediction loss : ,in, This represents the actual noise introduced during the forward noise addition process, and the noise prediction loss is used to constrain the error between the predicted noise and the actual noise. This indicates that the L2 norm square operation is performed on the input difference; Strength consistency loss :
[0017] Gradient structure loss :
[0018] The fusion loss The following is obtained by weighted summation of the above losses:
[0019] in, and These are the strength consistency losses. and gradient structure loss The corresponding weighting coefficients are used to balance the impact of each loss term during the training process.
[0020] Preferably, the gradient magnitude of the gradient structure loss is calculated using the Sobel operator, including:
[0021] in, It represents the gradient magnitude of an image, used to characterize the image's edge and structural information; This represents the input image used in gradient calculation; and These represent the Sobel horizontal gradient operator and the vertical gradient operator, respectively. The gradient magnitudes calculated in the above manner are used to construct the gradient structure loss to enhance the structural consistency and edge sharpness of the fused image.
[0022] This invention also provides a multimodal image fusion system based on a two-stream semantic alignment diffusion model, comprising: The semantic encoding module is used to acquire the multimodal image to be fused, and to input the multimodal image into the semantic encoder after concatenating the multimodal image in the channel dimension to generate a fusion-guided semantic prior feature map S; The noise-adding fusion module is used to generate a fusion supervision signal from the multimodal image using a pseudo-fusion function, and then input the fusion supervision signal into the forward noise-adding formula of the diffusion model to obtain a noise-adding fused image. ; The noise prediction module is used to perform noise-added fusion on the image x. t After feature concatenation with the semantic feature map S, the input is given to the corresponding modality branch flow network to output the noise prediction result of the corresponding modality; wherein, the branch flow network is a conditional U-Net network with time step encoding; The weighting module is used to generate the final noise prediction result based on the noise prediction of each mode and its corresponding weighting coefficients; The image reconstruction module is used to reconstruct the image based on the final noise prediction result and the noise-added fusion image. The final fused image F is obtained by reconstructing the image based on the inverse process of the diffusion model.
[0023] By adopting the above solution, the present invention has the following advantages and beneficial effects: This invention first extracts fusion-guiding semantic prior features from multimodal images using a semantic encoder, and simultaneously constructs a fusion supervision signal using a pseudo-fusion function. A noisy fusion image is then generated through a forward diffusion process. Subsequently, the semantic prior features and the noisy fusion image are fused, and noise prediction is achieved via a multimodal branch network. A weighted strategy is then used to obtain the final noise estimate. Finally, a high-quality fusion image is reconstructed based on the inverse process of the diffusion model. This invention effectively combines semantic structural constraints with the generative capabilities of the diffusion model, fully preserving multimodal information, improving the detail integrity and semantic consistency of the fusion image, and achieving more robust and accurate multimodal image fusion. Attached Figure Description
[0024] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the specific embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some specific embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.
[0025] Figure 1 This is a flowchart illustrating the multimodal image fusion method based on a two-stream semantic alignment diffusion model provided in the first embodiment of the present invention. Figure 2 This is a qualitative result of an embodiment of the present invention on the RoadScene dataset; Figure 3 This is a qualitative result of the embodiments of the present invention on the MSRS dataset; Figure 4 This is a schematic diagram of a multimodal image fusion system based on a dual-stream semantic alignment diffusion model, provided in the second embodiment of the present invention. Detailed Implementation
[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0027] Please see Figure 1 The first embodiment of the present invention provides a multimodal image fusion method based on a two-stream semantic alignment diffusion model, comprising the following steps: S1, acquire the multimodal image to be fused, and after concatenating the multimodal image in the channel dimension, input it into the semantic encoder to generate a fusion-guided semantic prior feature map S; In this embodiment, specifically, the multimodal image refers to an image obtained through different imaging methods, such as a visible light image, an infrared image, an RGB image, a depth image, a CT image, an MRI image, etc. In this embodiment, specifically, the multimodal image includes a first-mode infrared image. Visible light image with second mode .
[0028] In this embodiment, the semantic encoder includes three consecutive 3×3 convolutional layers and a SiLU activation function; wherein the three convolutional layers output feature maps with 32, 64, and 32 channels respectively, and the SiLU activation function is expressed as follows: , For the Sigmoid function, The SiLU activation function is used to non-linearly modulate the feature map output by the convolutional layer, which is the input feature map, to enhance the semantic encoder's ability to express complex structural features and improve the stability of the network training process.
[0029] S2, generate a fusion supervision signal for the multimodal image using a pseudo-fusion function. Then the fused supervision signal The forward denoising formula of the input diffusion model is used to obtain the denoised fusion map. .
[0030] Among them, the fused supervision signal Represented as:
[0031] in, Indicates taking The maximum value in; The denoised fusion map is then obtained using the forward denoising formula of the diffusion model. x t The process is represented as:
[0032] in, This represents the signal weighting coefficients that vary with the diffusion time step. Indicated by the The signal accumulation coefficient is calculated by the cumulative product. It is real random noise.
[0033] S3, the noise-added fusion image After feature concatenation with the semantic feature map S, the input is given to the corresponding modality branch flow network to output the noise prediction result of the corresponding modality; wherein, the branch flow network is a conditional U-Net network with time step encoding.
[0034] In this embodiment, specifically, the branching flow network includes an infrared flow network and a visible light flow network. The infrared flow network is biased towards the energy saliency of the infrared mode; the visible light flow network is biased towards the texture structure of the visible light mode. The input of each branching flow network includes a noisy fused image of dimension 1×H×W. The input tensor is (1+C)×H×W, formed by concatenating a semantic feature map S of dimension C×H×W through channels, along with time-step encoding (using sine and cosine time-step encoding to map the diffusion time step t into a high-dimensional vector, which is then embedded into each layer of the U-Net network); the output is the noise prediction result of the first modality. Noise prediction results for the second mode Where C represents the number of semantic feature channels, and H and W represent the height and width of the input multimodal image, respectively.
[0035] S4 generates the final noise prediction result based on the noise prediction of each mode and its corresponding weighting coefficient.
[0036] In this embodiment, the noise prediction result of the first mode can be obtained through a learnable Softmax noise fusion module. weight Noise prediction results for the second mode weight ;in:
[0037] and The learnable parameters corresponding to the noise of the first mode and the noise of the second mode are respectively, and the final noise prediction result is given. Represented as: .
[0038] S5, based on the final noise prediction result and the noise fusion map The final fused image F is obtained by reconstructing the image based on the inverse process of the diffusion model.
[0039] In this embodiment, the inverse process formula of the diffusion model is:
[0040] The final noise prediction result obtained using S4 As real random noise Substituting into the above equation and solving, we get Let F = The final fused image F is obtained through reconstruction.
[0041] In this embodiment, specifically, after obtaining the final fused image F, a fusion loss can be used to perform end-to-end optimization of the overall model, so that it simultaneously retains the infrared energy information and the visible light texture structure; wherein, the fusion loss includes: Noise prediction loss : ,in, This represents the actual noise introduced during the forward noise addition process, and the noise prediction loss is used to constrain the error between the predicted noise and the actual noise. This indicates that the L2 norm square operation is performed on the input difference; Strength consistency loss :
[0042] Gradient structure loss :
[0043] The gradient magnitude of the gradient structure loss is calculated using the Sobel operator, including:
[0044] in, It represents the gradient magnitude of an image, used to characterize the image's edge and structural information; This represents the input image used in gradient calculation; and These represent the Sobel horizontal gradient operator and the vertical gradient operator, respectively. The gradient magnitudes calculated in the above manner are used to construct the gradient structure loss to enhance the structural consistency and edge sharpness of the fused image.
[0045] The fusion loss is obtained by weighted summation of the above losses:
[0046] in, and These are the strength consistency losses. and gradient structure loss The corresponding weighting coefficients are used to balance the impact of each loss term during the training process.
[0047] The following section will use infrared-visible image fusion as an example to illustrate the multimodal image fusion method (DSD) based on the dual-stream semantic alignment diffusion model proposed in this invention. 2 Fusion was experimentally validated on two public datasets, MSRS and RoadScene.
[0048] Experimental Setup: In this embodiment, the model employs a single-stage end-to-end diffusion training strategy. During training, the input image patch size is set to 128×128, the total number of training epochs is 120, and the batch size is set to 8. The Adam optimization algorithm is used as the optimizer, with an initial learning rate of 1×10⁻⁶. -4 A segmented decay strategy was employed, reducing the learning rate to 0.5 times its original value every 20 rounds. These parameters were kept consistent across all datasets to ensure the fairness and comparability of the experimental results.
[0049] See the instruction manual appendix Figure 2 and Figure 3 Each of these embodiments demonstrates an example of the present invention ( Figure 2 and Figure 3 DSD in 2 Fusion competes with other existing algorithms on the RoadScene and MSRS datasets (such as...). Figure 1 and Figure 2 Qualitative comparison results of DIDFuse, U2Fusion, SDNet, RFNet, and CDDFuse were presented. It can be seen that the embodiments of the present invention can more effectively fuse significant thermal radiation information from infrared images with detailed texture information from visible light images. In low-light or nighttime scenes, target regions contained in infrared images are clearly highlighted in the fusion result, making foreground targets more distinguishable from complex backgrounds. Simultaneously, background details that are difficult to discern in visible light images due to insufficient illumination still retain clear edges and complete structural contours in the fusion result, contributing to an accurate understanding of the overall scene.
[0050] As shown in Tables 1 and 2, further research demonstrates that, compared to other existing algorithms, this invention (i.e., DSD) is superior. 2 Fusion achieved the highest or near-highest scores on multiple evaluation metrics, including information entropy (EN), contrast (SD), spatial frequency (SF), mutual information (MI), difference correlation sum (SCD), visual information fidelity (VIF), structural similarity (SSIM), and edge preservation index (Qabf), outperforming most existing fusion methods in overall performance.
[0051] Table 1 MSRS Infrared-Visible Fusion Dataset
[0052] Table 2 RoadScene Infrared-Visible Fusion Dataset
[0053] It can be seen that some existing algorithms may optimize for a single metric (such as SSIM or Qabf), but often at the cost of sacrificing complementary properties such as information entropy or mutual information, resulting in a decrease in the overall information content of the fused image. In contrast, the embodiments of the present invention maintain a good balance among various evaluation metrics, ranking first in more than five metrics on multiple datasets, and maintaining strong competitiveness in the remaining metrics, fully demonstrating the stability and versatility of the method.
[0054] Furthermore, by employing a single-stage diffusion modeling and dual-stream noise prediction structure, this invention avoids multi-stage feature extraction and complex post-processing operations, thereby effectively reducing the number of model parameters and computational complexity. On a single high-performance GPU, the method of this invention can achieve high inference speed, meeting the requirements of real-time or near-real-time infrared-visible image fusion applications.
[0055] In contrast, some multi-stage or multi-branch fusion methods have significantly increased computational complexity and lower inference efficiency due to their complex structure; while some lightweight methods, although having fewer parameters, have obvious shortcomings in fusion quality.
[0056] In summary, this invention achieves a better balance between fusion accuracy and operational efficiency, demonstrating high engineering application value.
[0057] Please see Figure 4 The second embodiment of the present invention also provides a multimodal image fusion system based on a two-stream semantic alignment diffusion model, comprising: The semantic encoding module 210 is used to acquire the multimodal image to be fused, and to input the multimodal image into the semantic encoder after concatenating the multimodal image in the channel dimension to generate a fusion-guided semantic prior feature map S; The noise-adding fusion module 220 is used to generate a fusion supervision signal from the multimodal image using a pseudo-fusion function, and then input the fusion supervision signal into the forward noise-adding formula of the diffusion model to obtain a noise-adding fused image. ; Noise prediction module 230, used to analyze the noise-added fusion image After feature concatenation with the semantic feature map S, the input is given to the corresponding modality branch flow network to output the noise prediction result of the corresponding modality; wherein, the branch flow network is a conditional U-Net network with time step encoding; The weighting module 240 is used to generate the final noise prediction result based on the noise prediction of each mode and its corresponding weighting coefficients; Image reconstruction module 250 is used to reconstruct the image based on the final noise prediction result and the noise-added fusion image. The final fused image F is obtained by reconstructing the image based on the inverse process of the diffusion model.
[0058] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A multimodal image fusion method based on a two-stream semantic alignment diffusion model, characterized in that, Includes the following steps: S1, acquire the multimodal image to be fused, and after concatenating the multimodal image in the channel dimension, input it into the semantic encoder to generate a fusion-guided semantic prior feature map S; S2, a fusion supervision signal is generated from the multimodal image using a pseudo-fusion function, and then the fusion supervision signal is input into the forward noise addition formula of the diffusion model to obtain a noisy fused image. ; S3, the noise-added fusion image After feature concatenation with the semantic feature map S, the input is given to the corresponding modality branch flow network to output the noise prediction result of the corresponding modality; wherein, the branch flow network is a conditional U-Net network with time step encoding; S4. Based on the noise predictions for each mode and their corresponding weighting coefficients, the final noise prediction results are generated. S5, based on the final noise prediction result and the noise fusion map The final fused image F is obtained by reconstructing the image based on the inverse process of the diffusion model.
2. The multimodal image fusion method based on a two-stream semantic alignment diffusion model according to claim 1, characterized in that, In step S1, the multimodal image includes an infrared image of the first modality. Visible light image with second mode The semantic encoder includes three consecutive 3×3 convolutional layers and a SiLU activation function. The three convolutional layers output feature maps with 32, 64, and 32 channels, respectively. The SiLU activation function is used to non-linearly modulate the feature maps output by the convolutional layers to enhance the semantic encoder's ability to express complex structural features and improve the stability of the network training process.
3. The multimodal image fusion method based on a two-stream semantic alignment diffusion model according to claim 1, characterized in that, In step S2, the monitoring signal is fused. Represented as: in, Indicates taking The maximum value in; The noise fusion map is obtained using the forward noise addition formula of the diffusion model. x t The process is represented as: in, This represents the signal weighting coefficients that vary with the diffusion time step. Indicated by the The signal accumulation coefficient is calculated by the cumulative product. It is real random noise.
4. The multimodal image fusion method based on a two-stream semantic alignment diffusion model according to claim 2, characterized in that, In step S3, the branching flow network includes an infrared flow network and a visible light flow network. The infrared flow network is biased towards the energy saliency of the infrared mode; the visible light flow network is biased towards the texture structure of the visible light mode; the input of each branching flow network includes a noisy fusion map of dimension 1×H×W. The input tensor (1+C)×H×W is formed by concatenating the semantic feature map S of dimension C×H×W through channels, along with the time step encoding; The output is the noise prediction result of the first mode. Noise prediction results for the second mode Where C represents the number of semantic feature channels, and H and W represent the height and width of the input multimodal image, respectively.
5. The multimodal image fusion method based on a two-stream semantic alignment diffusion model according to claim 4, characterized in that, In step S4: The noise prediction result of the first mode is obtained through a learnable Softmax noise fusion module. weight Noise prediction results for the second mode weight ;in: and The learnable parameters corresponding to the noise of the first mode and the noise of the second mode are respectively, and the final noise prediction result is given. Represented as: .
6. The multimodal image fusion method based on a two-stream semantic alignment diffusion model according to claim 5, characterized in that, In step S5, the inverse process formula of the diffusion model is: The final noise prediction result obtained using S4 As real random noise Substituting into the above equation and solving, we get Let F = The final fused image F is obtained through reconstruction.
7. The multimodal image fusion method based on a two-stream semantic alignment diffusion model according to claim 5, characterized in that, Also includes: A fusion loss is used to perform end-to-end optimization of the overall model, so that it simultaneously preserves the energy information of infrared light and the texture structure of visible light; wherein, the fusion loss includes: Noise prediction loss : ,in, This represents the actual noise introduced during the forward noise addition process, and the noise prediction loss is used to constrain the error between the predicted noise and the actual noise. This indicates that the L2 norm square operation is performed on the input difference; Strength consistency loss : Gradient structure loss : The fusion loss The following is obtained by weighted summation of the above losses: in, and These are the strength consistency losses. and gradient structure loss The corresponding weighting coefficients are used to balance the impact of each loss term during the training process.
8. The multimodal image fusion method based on a two-stream semantic alignment diffusion model according to claim 7, characterized in that, The gradient magnitude of the gradient structure loss is calculated using the Sobel operator, including: in, It represents the gradient magnitude of an image, used to characterize the image's edge and structural information; This represents the input image used in gradient calculation; and These represent the Sobel horizontal gradient operator and the vertical gradient operator, respectively. The gradient magnitudes calculated in the above manner are used to construct the gradient structure loss to enhance the structural consistency and edge sharpness of the fused image.
9. A multimodal image fusion system based on a two-stream semantic alignment diffusion model, characterized in that, include: The semantic encoding module is used to acquire the multimodal image to be fused, and to input the multimodal image into the semantic encoder after concatenating the multimodal image in the channel dimension to generate a fusion-guided semantic prior feature map S; The noise-adding fusion module is used to generate a fusion supervision signal from the multimodal image using a pseudo-fusion function, and then input the fusion supervision signal into the forward noise-adding formula of the diffusion model to obtain a noise-adding fused image. ; The noise prediction module is used to analyze the noise-added fused image. After feature concatenation with the semantic feature map S, the input is given to the corresponding modality branch flow network to output the noise prediction result of the corresponding modality; wherein, the branch flow network is a conditional U-Net network with time step encoding; The weighting module is used to generate the final noise prediction result based on the noise prediction of each mode and its corresponding weighting coefficients; The image reconstruction module is used to reconstruct the image based on the final noise prediction result and the noise-added fusion image. The final fused image F is obtained by reconstructing the image based on the inverse process of the diffusion model.