Multi-modal image fusion method and system based on inverse diffusion model, device and readable storage medium

Abstract

The present application relates to a kind of multimodal image fusion methods of inverse diffusion model, comprising the following steps: step one, using inverse diffusion technology, visible light image is reversed to noise latent space, then using the feature of reversed visible light image, guide infrared image is reversed;Step two, guide by the inverse process in diffusion model, inject the appearance attribute of visible light into infrared feature, and the feature can generate infrared image with visible light style;Step three, design specific fusion rule, for the attention layer fusion of denoising process, reversed visible light and infrared feature are fused, the text interaction capability of model is retained, and language-driven fusion control is supported.The present application can directly generate high-quality fusion image without additional training or fine-tuning.The obtained fusion image is highly compatible with the base model, effectively solves the difference problem between data domains, and significantly improves the performance of downstream machine perception tasks.The present application significantly reduces the training cost, and provides an efficient and innovative solution for cross-domain tasks.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a multimodal image fusion method, system, device, and readable storage medium based on an inverse diffusion model. Background Technology

[0002] Image fusion technology is widely recognized for its ability to integrate complementary information from multiple source images into a single fused image. Due to the inherent physical limitations of various sensors, each modality has its limitations in capturing all the information in a scene. For example, visible light sensors, commonly used in everyday applications, are effective at capturing scene details but are highly sensitive to lighting conditions. Their performance degrades significantly in poor lighting conditions, such as at night or under overexposure. In contrast, infrared sensors are more robust to different lighting and weather conditions, capturing useful scene information both day and night. However, infrared images lack the detailed structural information provided by visible light sensors. By combining the complementary information of these two modalities, infrared and visible light image fusion technology can generate a fused image that retains as much of the effective information in the scene as possible.

[0003] With the development of generative learning techniques, numerous fusion methods have been developed for infrared and visible light image fusion tasks. These methods can generally be categorized into three types: autoencoders (AEs), generative adversarial networks (GANs), and diffusion models (DMs). AE-based methods typically employ complex network architectures to improve feature extraction. GAN-based methods utilize an adversarial framework, where the generator aims to produce a fused image that can fool the discriminator, which then strives to distinguish the generated image from the real image. Recently, DM-based methods have gained significant attention for their ability to produce high-quality results and are generally more stable than GANs. Despite these significant achievements, a key challenge remains unresolved: adapting the fused image to the downstream task. The spectrum captured by infrared sensors differs from that of visible light, resulting in significant differences in image appearance. During the fusion process, most existing methods force the fused image to maintain pixel-level similarity with its source image. Therefore, the fused image incorporates appearance features from both infrared and visible light modalities. From the perspective of appearance attributes, infrared, visible light, and the fused image belong to three distinct domains. In this context, while existing fusion methods perform well in traditional computer vision tasks, the emergence of underlying models introduces new challenges. Specifically, fused images, as independent domains, are difficult to integrate seamlessly with pre-trained base models. These models are typically trained on large-scale visible light image datasets and, to some extent, infrared images. However, few base models include fused images in their training data, resulting in inherent domain gaps when the fused images are applied directly to these models. When the same fused image is directly input into a pre-trained detection model, the results are less than ideal compared to fused images that are more similar to visible light. Summary of the Invention

[0004] To address the problem that fused images are not directly adaptable to high-level visual pre-trained models, this invention proposes a multimodal image fusion method based on an inverse diffusion model.

[0005] The technical solution adopted by this invention to solve the above problems is: a multimodal image fusion method based on an inverse diffusion model, comprising the following steps:

[0006] Step 1: Using reverse diffusion technology, the visible light image is reversed to the noise latent space, and then the features of the reversed visible light image are used to guide the infrared image to reverse.

[0007] Step 2: Guided by the inverse process in the diffusion model, the appearance attributes of visible light are injected into infrared features, which can generate infrared images with visible light style.

[0008] Step 3: Design specific fusion rules for the attention layer to fuse reversed visible light and infrared features during the denoising process, preserving the model's text interaction capabilities and supporting language-driven fusion control.

[0009] Furthermore, in step one, following the framework of the denoising diffusion probability model, the model includes a T-step diffusion process and a T-step reverse process; the forward process gradually transforms a clean image... Converted to white Gaussian noise; can be expressed as:

[0010] (1)

[0011] in, The image represents the time t. Refers to hyperparameters related to time t. It follows a standard normal distribution;

[0012] Its reverse process can be represented as:

[0013] (2)

[0014] In formula (2), It is about hyperparameters,

[0015] predict ,

[0016] Indicates pointing to direction

[0017] It is a diffusion model network model that generates visible features at step t. , and The expression is as follows:

[0018] (3)

[0019] In formula (3), and Predicting noise in the network Generated.

[0020] Furthermore, in step two, to incorporate visible light features into the infrared image, the update steps for the visible light style infrared image are defined as follows:

[0021]

[0022] (4)

[0023]

[0024] In formula (4) Indicates the weighting factor. This refers to visual cues; visual cues are crucial in guiding the noisy infrared image toward the visible direction at each time step; according to formulas (2), (3), and (4) It can also be expressed as:

[0025] (5)

[0026] (6)

[0027] Formula (4) can be rewritten as:

[0028] (7)

[0029] Furthermore, in step three, to maintain the visible appearance, visible features are used as the basic components of the fused image; the iterative generation process of the fused image can be expressed as:

[0030] (8)

[0031] A custom fusion rule is introduced to inject infrared information similar to visible light. Specifically, the fusion rule is applied to the self-attention layer in the denoising network. , expressed as:

[0032]

[0033] (9)

[0034] and This represents two hyperparameters that must satisfy the constraints. ;

[0035] To ensure that the fused image primarily retains visible light content, the query vector is maintained during the iteration process. Unchanged, utilizing the self-attention layer Preserve the visual characteristics of infrared images by injecting infrared information in the early stages of denoising.

[0036] Furthermore, a classifier-free approach is adopted to improve the overall quality of the generated fused image; in each denoising step t, the denoising network performs two forward propagations:

[0037] (10)

[0038] This represents the text embedding from a pre-trained text encoder; This represents the noise predicted according to formula (9), while This indicates that the result is generated unconditionally. It is the strength of the guiding vector.

[0039] The present invention also relates to a multimodal image fusion system based on an inverse diffusion model, comprising a computer module that applies the aforementioned multimodal image fusion method based on an inverse diffusion model.

[0040] The present invention also relates to a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described multimodal image fusion method based on an inverse diffusion model.

[0041] The present invention also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described multimodal image fusion method based on an inverse diffusion model.

[0042] Beneficial effects

[0043] This invention proposes a training-free fusion model that achieves efficient adaptation to downstream tasks through inversion technology, demonstrating several beneficial effects. First, this invention generates a fused image with a visible light style by integrating features from visible light images into the inversion process of infrared images. This process introduces weighting factors to flexibly control the intensity of visible light appearance attributes, ensuring adaptability to different task requirements. Second, this invention combines diffusion-based image editing technology to design customized fusion rules that not only preserve the model's text interaction capabilities but also support language-driven fusion control. By fully utilizing the capabilities of the pre-trained diffusion model, this invention can directly generate high-quality fused images without additional training or fine-tuning. The resulting fused image is highly compatible with the base model, effectively solving the problem of differences between data domains and significantly improving the performance of downstream machine perception tasks. This invention significantly reduces training costs and provides an efficient and innovative solution for cross-domain tasks. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the reverse diffusion multimodal fusion in this invention;

[0045] Figure 2 This is a visual comparison image of infrared-visible light fusion performed on the MSRS dataset of this invention. Detailed Implementation

[0046] The technical solution of the present invention will be described in detail below.

[0047] like Figure 1 As shown, the multimodal image fusion method based on the inverse diffusion model of the present invention specifically includes:

[0048] By using reverse diffusion technology, the visible light image is reversed to the noise latent space, and then the features of the reversed visible light image are used to guide the reversal of the infrared image.

[0049] The Denoising Diffusion Probabilistic Model (DDPM) framework is used, which includes a T-step diffusion process and a T-step reverse process. The forward process progressively cleans an image. Convert to white Gaussian noise. The t-th step is:

[0050] (1)

[0051] in, The image represents the time t. Refers to hyperparameters related to time t. It is a standard normal distribution;

[0052] Its reverse process is

[0053] (2)

[0054] In formula (2), It is about hyperparameters, predict ,and Indicates pointing to The direction. Indicates that the network is inputting... The predicted output is as follows. Represents variance, while This represents the standard normal vector that is independent and identically distributed.

[0055] At step t, visible features are generated. , and The expression is as follows:

[0056] (3)

[0057] In formula (3), and Predicting noise in the network Generated.

[0058] Specific implementation method two: such as Figure 1As shown, guided by the inverse diffusion model, the appearance attributes of visible light are injected into infrared features, which can generate infrared images with visible light style.

[0059] In order to incorporate visible light features into infrared images, this invention defines the following update steps for visible light-style infrared images:

[0060]

[0061] (4)

[0062]

[0063] In formula (4) This represents the weighting factor, while This refers to visual cues. Visual cues are crucial in guiding the noisy infrared image toward the visible direction at each time step. This is based on equations (2), (3), and (4). It can also be expressed as:

[0064] (5)

[0065] In formula (5), Indicates prediction Equation (5) can be simplified to:

[0066] (6)

[0067] Therefore, formula (4) can be rewritten as:

[0068] (7)

[0069] Specific implementation method three: such as Figure 1 As shown, a specific fusion rule is designed to fuse inverted visible and infrared features in the attention layer during the denoising process, preserving the model's textual interactivity and supporting language-driven fusion control. To maintain the visible appearance, visible features are used as the basic components of the fused image. The iterative generation process of the fused image can be expressed as:

[0070] (8)

[0071] Simply using the dependency formula (7) to obtain the fused image is insufficient because the update process is exactly the reverse of the visible image process, thus excluding infrared information. To address this issue, a custom fusion rule is introduced to inject infrared information similar to visible light. Specifically, the designed fusion rule is applicable to the self-attention layer in the denoising network. , can be expressed as:

[0072]

[0073] (9)

[0074] and This represents two hyperparameters that must satisfy the constraints. In the fusion rule strategy in formula (9), There is no explicit designation for exchanging information with infrared or visible images.

[0075] To ensure that the fused image primarily retains visible light content, the query vector is maintained during the iteration process. constant.

[0076] Utilizing the self-attention layer To preserve the visual characteristics of infrared images. In this process, infrared information is injected into the early steps of denoising, a crucial stage in the formation of the overall image layout.

[0077] As the iterations progress, the main structure and content of the fused image are largely determined. Subsequent steps focus on fine-tuning the image style. After initially injecting infrared information into the fused image, several steps are added to refine image details by incorporating visible light information. In the final stage, manual modification of the fused image's features is avoided to generate a more unified and visually consistent fused image.

[0078] Specific Implementation Method Four: Employing a classifier-free approach to improve the overall quality of the generated fused image. In each denoising step t, the denoising network performs two forward propagations:

[0079] (10)

[0080] This represents the text embedding from a pre-trained text encoder. This represents the noise predicted according to formula (9), while This indicates that the result is generated unconditionally. It is the strength of the guiding vector.

[0081] Specific implementation method five: such as Figure 1 As shown, the strategy for obtaining the fused image is as follows: First, an image encoder is used to embed the source infrared image and the visible image into a fused image. and Next, the diffusion process is reversed to construct a storage intermediate feature ( , , , ) and latent variables ( , The memory stores information for each time step t. In this stage, visible features are first generated. Then, these features are used to guide the corresponding infrared signatures. The generation of the fused image involves a forward diffusion process based on a memory bank and text prompts. During this process, a custom fusion rule called appearance feature injection is introduced to obtain the fused features. Subsequently, the fusion characteristics during the forward process... Transformed into a blended image with a visible style.

[0082] This invention uses three representative datasets: RoadScene, MSRS, and FMB. The RoadScene dataset contains 221 image pairs, the MSRS test dataset contains 361 image pairs, and the FMB test dataset consists of 280 image pairs.

[0083] Effect verification

[0084] The method proposed in this invention uses SD v1.5. The hyperparameters in equation (4) Set to 0.08, while in formula (10) The value is set to 2. In the experiment, the value in formula (9) is... and The values ​​were set to 0 and 70 respectively. All experiments were conducted on an NVIDIA GeForce RTX 3090 GPU equipped with 24GB of video memory.

[0085] Evaluation of the effect of fused datasets

[0086] Four evaluation metrics were used: cross-entropy (CE), entropy (EN), standard deviation (SD), and CLIPIQA (the perceptual score of CLIP). Higher values ​​for these metrics indicate better quality of the fused image. Specifically, CLIPIQA assesses both perceived image quality and perceived abstraction, aiming for a higher correlation with human visual perception.

[0087] Table 1 presents a performance comparison of fused images, summarizing the performance comparisons on the RoadScene test dataset, MSRS dataset, and FMB dataset. The best and second-best results for each metric are indicated by bold and underline, respectively, to facilitate quick identification of model performance.

[0088] Table 1

[0089]

[0090] Table 1 presents detailed quantitative comparisons of the present invention with current state-of-the-art (SOTA) methods on the RoadScene, MSRS, and FMB datasets.

[0091] This invention demonstrates superior performance across all metrics, showcasing its competitiveness. It achieved the highest score on the Standard Deviation (SD) metric, indicating excellent contrast in the generated fused image, primarily due to the transformed infrared image, which provides more detailed information than the original infrared image. Furthermore, this invention also excels on the CLIPIQA metric, reflecting its ability to effectively transfer visible appearances to the infrared image, resulting in a fused image with a visible style. In contrast, comparative methods typically preserve information from the original infrared image, leading to fused images with a distinctive appearance. The fused image of this invention achieves an excellent score in consistency with human visual perception, further highlighting its superiority.

[0092] exist Figure 2 This invention demonstrates two embodiments of MSRS and compares the performance of seven different methods with that of the present invention. Scene 1 is a depiction of a car at dusk, and Scene 2 is a traffic intersection with buildings in the background. Both scenes are affected by illumination degradation, resulting in low brightness of the visible images. In the fusion results, the fused image generated by U2Fusion is often dark, and details are difficult to discern. Other methods, such as MetaFusion, TIMFusion, CDDFuse, and DDFM, are able to maintain the brightness level of the visible images. More recent diffusion-based methods, such as Text-IF and Diff-IF, although slightly better than the original images in terms of visual effect, still suffer from the significant effects of illumination degradation. In contrast, the fused image generated by the method of this invention is more visually appealing, with sharper edges, producing a clear and visually pleasing fused image. These results highlight the stronger ability of using a pre-trained stable diffusion model to generate more realistic and high-quality images. The final generated fused image is characterized by high and natural contrast and vivid colors.

[0093] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments without departing from the scope of the present invention, based on the technical essence of the present invention and within the spirit and principles of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A multimodal image fusion method based on an inverse diffusion model, characterized in that, Includes the following steps: Step 1: Follow the framework of the diffusion model, which includes a T-step forward process and a T-step backward process. The forward process gradually transforms a clean image Converted to white Gaussian noise; can be expressed as: (1); where, The image represents the time t. Refers to the hyperparameters related to time step t. It follows a standard normal distribution; The reverse process in its diffusion model is represented as follows: (2); In formula (2), It is about hyperparameters, predict , Indicates pointing to direction, It is a diffusion model network model that generates visible features at step t. , and The expression is as follows: ， ， (3) ； In formula (3), the network model of the diffusion model In the input Noise can be predicted under certain conditions. ; and Noise is predicted Generated at that time; Step 2: To incorporate visible light features into the infrared image; according to formulas (2) and (3), the update steps for visible light style infrared can be defined as follows: ， In formula (4) Indicates the weighting factor. This refers to visual cues; visual cues are crucial in guiding the noisy infrared image toward the visible direction at each time step; according to formulas (2) and (3), in formula (4) It can also be expressed as: (5); (6); Formula (4) can be rewritten as: (7)； Step 3: To maintain a visible appearance, use visible features as the basic components of the fused image; fused image The iterative generation process can be expressed as: (8); A custom fusion rule is introduced to inject infrared information similar to visible light. Specifically, the fusion rule is applicable to the self-attention layer in the denoising network. , can be defined as: (9)； and This represents two hyperparameters that must satisfy the constraints. ; To ensure that the fused image primarily retains visible light content, the query vector is maintained during the iteration process. Unchanged, utilizing the self-attention layer Preserve the visual characteristics of infrared images by injecting infrared information in the early stages of denoising.

2. The multimodal image fusion method using an inverse diffusion model according to claim 1, characterized in that, A classifier-free approach is used to improve the overall quality of the generated fused image; a denoising network is used for two forward propagations. (10); This represents the text embedding from a pre-trained text encoder; This represents the noise predicted according to formula (9), while This indicates that the result is generated unconditionally. It is the strength of the guiding vector.

3. A multimodal image fusion system based on an inverse diffusion model, characterized in that, It includes a computer module that applies a multimodal image fusion method based on an inverse diffusion model as described in any one of claims 1 to 2.

4. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of any one of claims 1 to 2.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of any one of claims 1 to 2.

Images