Modal attribute-structure decoupled infrared and visible image fusion method

Abstract

The present application belongs to the technical field of image information processing, and relates to a modal attribute-structure decoupled infrared and visible image fusion method, which is constructed based on a pre-trained latent space diffusion model, and the overall network framework comprises an encoder of a variational autoencoder, a U-Net denoising network and a decoder of the variational autoencoder. The present application introduces structure LoRA parameters and attribute LoRA parameters into the pre-trained latent space diffusion model, and models the modal structure information and the modal attribute information as the differential change quantity of the model prediction noise before and after the introduction of LoRA. By applying a contrast constraint to the differential change quantity, the effective decoupling of the structure information and the attribute information in the diffusion noise space is realized, so that the important complementary features of different modalities are avoided to be lost in the fusion process. In the inference stage, the content information between different modalities can effectively perform cross-modal feature fusion, and finally generate a high-quality fusion image.

Description

Technical Field

[0001] This invention belongs to the field of image information processing technology and relates to a method for fusing infrared and visible images with modal attribute-structure decoupling. Background Technology

[0002] Currently, the technologies related to this invention include two aspects: the first is an image fusion method based on modal feature decomposition; the second is an image fusion method driven by diffusion model.

[0003] Modal feature decomposition-based fusion methods separate modal attribute information and modal structural information in images of different modalities by constructing shared feature branches with different structures and modality-specific feature branches, thereby achieving cross-modal feature fusion in a semantically consistent space. However, the modal decomposition paradigm of this type of method relies on complex modal decomposition network structures, and its decomposition results are limited by human prior assumptions, making it difficult to flexibly decouple modal features.

[0004] In recent years, diffusion denoising probabilistic models have emerged as a stable and controllable generative model. These models progressively add noise to construct a Markov chain during the forward process, and then approximate the inverse process of the Markov diffusion process by predicting the noise during the reverse process, thereby gradually generating the target image. Based on this principle, existing research has introduced diffusion models into infrared and visible light image fusion tasks. Existing diffusion model-driven image fusion methods use images from different modalities as conditional inputs to guide the diffusion model in generating the fusion result during denoising. However, these methods directly fuse different modal feature information, lacking effective decomposition of modal attribute and structural information. Due to domain differences between different modal features, the model easily loses important complementary information during denoising, leading to a decline in the quality of the fusion result.

[0005] In summary, the methods described above employ manually designed decomposition networks or direct fusion of cross-modal features, but they cannot effectively fuse complementary features between two modalities, resulting in limited fusion performance. To address this issue, a potential solution is to avoid constructing complex modal decomposition network structures and instead achieve adaptive decoupling and alignment of modal attributes and structures in the diffuse noise space through noise difference and contrastive learning mechanisms. This effectively alleviates domain differences in features across different modalities during denoising, thereby fully fusing complementary information between different modalities. Therefore, this invention proposes a modal attribute-structure decoupling method for fusing infrared and visible images. First, learnable low-rank adaptation (LoRA) parameters, namely attribute LoRA and structural LoRA, are introduced into the base model (pre-trained diffusion model). Then, during denoising, the model outputs the predicted noise after introducing attribute LoRA or structural LoRA, and performs difference calculation with the predicted noise of the base model, thus representing attribute and structural information as noise increments relative to the base model. Based on this, contrastive learning constraints are constructed between different noise increments, so that the structural noise increments of infrared and visible modes are close to each other in the latent space, while the attribute noise increments and structural noise increments under the same mode are far apart, thereby achieving cross-modal structural alignment and effective decoupling of attributes and structures in the diffuse noise space. Summary of the Invention

[0006] To address the challenge of effectively decomposing modal attributes and structural information in infrared and visible image fusion tasks, a modal attribute-structure decoupling method for infrared and visible image fusion is proposed. The core idea of ​​this method is to introduce attribute LoRA and structural LoRA into the base model, explicitly representing the learned content and style information as noise increments relative to the base model. Combined with a contrastive learning mechanism, this achieves alignment of infrared and visible modal content information and effective decoupling of content-style information in the noise space. Specifically, during the training phase, the denoising network of the base model consists of an encoder and a decoder. The encoder introduces structural LoRA to learn modal structural information, while the decoder introduces attribute LoRA to learn modal attribute information. Then, when calculating the structural noise increment, the attribute LoRA parameters are frozen. Similarly, when calculating the attribute noise increment, the structural LoRA parameters are frozen. Based on this approach, the model predicts that the incremental changes in noise originate from a single type of LoRA, thus obtaining the attribute and structural increments corresponding to the infrared and visible modalities. Finally, a contrastive loss is introduced to constrain different types of noise increments, strengthening the consistency of cross-modal content information by bringing the structural noise increments corresponding to the infrared and visible modalities closer together. Simultaneously, by widening the difference between attribute noise increments and structural noise increments within the same modality, the ambiguity between attribute and structural information is suppressed. Using the aforementioned optimization strategy, this invention effectively aligns infrared and visible modal content features in the latent space, while explicitly decoupling attribute and structural features within the same modality. Thanks to the attribute-structure decoupling learning during the training phase, the encoder of the denoising network can extract clean modal structural features free from attribute interference. Therefore, during the inference phase, the structural features of the infrared and visible modalities are directly added element-wise to obtain a complete fused representation. Finally, visible attribute LoRA parameters are injected into the decoder of the denoising network to perform attribute modulation, ultimately generating a fused image with complementary structures and visible appearance attributes.

[0007] The technical solution of this invention:

[0008] A modal attribute-structure decoupling method for infrared and visible image fusion is proposed, based on a pre-trained latent space diffusion model. The overall network framework includes a variational autoencoder encoder. Decoders using U-Net denoising network and variational autoencoder The steps are as follows:

[0009] First, infrared images and visible light images The encoders of the variational autoencoder are input respectively. This yields the corresponding latent representation:

[0010]

[0011]

[0012] in, Representing modes Low-dimensional feature representation in the latent space This indicates the infrared or visible light mode; subsequently, during the forward diffusion process, the... Perform a stepwise noise-adding operation for t time steps to obtain the corresponding noisy latent representation. The calculation process is as follows:

[0013]

[0014] in, Represents the noise scheduling coefficient The cumulative product is used to control the first... The proportional relationship between the latent representation and Gaussian noise at each time step. Indicates the first The noise scheduling coefficients corresponding to each time step. The Gaussian noise follows a standard normal distribution; then, the noisy latent representation is... Input the U-Net denoising network and output the noise prediction result. ;in, These are the basic parameters for the U-Net denoising network;

[0015] Two types of LoRA are introduced into the U-Net denoising network: learnable structural LoRA parameters are introduced on the encoder side. Used for modeling modes Structural information; LoRA parameter attribute introduced on the decoder side. Used for modeling modes The attribute information; when calculating the structural noise increment, the LoRA parameters on the decoder side are frozen. Only activate the structure LoRA parameters on the encoder side. And the potential representation of noise addition. Perform noise prediction and obtain the corresponding noise prediction results. Compare the noise prediction results with the noise prediction results. By subtraction, we obtain the modes. Corresponding structural noise increment:

[0016]

[0017] in, This represents the structural information learned from the structure LoRA parameters under the condition of frozen basis model parameters, used to describe the modes. The change in structural information relative to the pre-trained latent space diffusion model; when calculating the attribute noise increment, the structural LoRA parameters on the encoder side are frozen. Only activate the LoRA parameter attribute on the decoder side. The attribute noise prediction results were obtained in the same way. And further calculate the modes Corresponding attribute noise increment:

[0018]

[0019] in, This represents the attribute information learned from the LoRA parameters under the condition of freezing the pre-trained latent space diffusion model parameters, used to describe the modality. The amount of change in attribute information relative to the pre-trained latent space diffusion model;

[0020] Then, by minimizing the cosine distance between the structural noise increments of the infrared and visible modes, consistent alignment of cross-modal structural information is achieved, with cross-modal alignment loss... It is expressed as follows:

[0021]

[0022] in, This represents the cosine similarity function; simultaneously, by minimizing the cosine similarity between the structural noise increment and the attribute noise increment in the same mode, decoupling of structural and attribute information in the infrared and visible light modes is achieved, along with the same-mode decoupling loss. It is expressed as follows:

[0023]

[0024] Contrast constraint loss of noise increment It is expressed as follows:

[0025]

[0026] To ensure that the proposed network has stable image reconstruction capabilities, an image reconstruction loss is introduced. Constraints are imposed on the U-Net denoising network; during backdiffusion, the U-Net denoising network is based on the noisy latent representation at each time step. The noise components are predicted step by step, and the latent representation is iteratively denoised and updated using preset noise scheduling coefficients to obtain the final latent representation. Finally, the ultimate potential representation Decoder via variational autoencoder The corresponding reconstructed image is obtained through reconstruction. and with the input image The image reconstruction loss is calculated and defined as follows:

[0027]

[0028] Among them, the input image Including infrared images and visible light images , ; Indicates the input image With reconstructed image The absolute error per pixel between them;

[0029] Finally, the total loss of the proposed network Using contrastive constraint loss and image reconstruction loss Joint optimization is performed as follows:

[0030]

[0031] During the inference phase, the infrared image and the visible light image are respectively input into the encoder of the variational autoencoder. , to obtain in modality Low-dimensional feature representation in latent space Next, during the backdiffusion process, a U-Net denoising network is used to predict noise step by step; a complete fused structural feature representation is obtained by element-wise addition; and infrared structural features are obtained during the feature extraction process of each layer of the U-Net denoising network. With visible light structural features The fused structural features are obtained at the corresponding level through element-wise addition, and their form is as follows:

[0032]

[0033] in, Indicates the characteristics of the fusion structure. This represents an element-wise addition operation;

[0034] By injecting the learned visible light modal properties LoRA parameters The decoding process of the modulated U-Net denoising network is used to impart visible light modal attribute information to the fused structural features, thereby obtaining the noise prediction result after visible light modal attribute modulation. ;

[0035] During the stepwise process of the back diffusion, the modulated noise prediction result By progressively updating the latent representation, a fused latent representation is obtained. Finally, the latent representations are fused. Decoder for input variational autoencoder Decode the image to generate the final fused image. Its form is expressed as:

[0036]

[0037] The beneficial effects of this invention are as follows: This invention introduces structural LoRA and attribute LoRA into the diffusion model, modeling modal structural information and modal attribute information as differential changes in the model's prediction noise before and after the introduction of LoRA. By applying contrast constraints to these differential changes, effective decoupling of structural and attribute information is achieved in the diffusion noise space, thereby avoiding the loss of important complementary features of different modalities during the fusion process. In the inference stage, the content information between different modalities can effectively perform cross-modal feature fusion, ultimately generating a high-quality fused image. Attached Figure Description

[0038] Figure 1 A schematic diagram of the training structure of the LoRA modulation mode decoupling network is shown for comparison.

[0039] Figure 2 A schematic diagram of the inference structure of the LoRA modulation mode decoupling network is shown for comparison. Detailed Implementation

[0040] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0041] During the training phase, such as Figure 1 As shown, specifically, the infrared image and the visible light image are first input into the encoder of the variational autoencoder. Latent space encoding is performed on the source image to obtain the latent representation of the corresponding mode. Then, at a preset diffusion time step, noise is added to the latent representations of the two modes to construct the forward diffusion process of the pre-trained latent space diffusion model; subsequently, by alternately activating and freezing the structural LoRA and attribute LoRA, the structural noise increment of the corresponding mode m is calculated. and attribute noise increment Subsequently, by imposing similarity constraints on the structural noise increments of the infrared and visible light modes, consistent alignment of cross-modal structural information is achieved. Simultaneously, by imposing difference constraints on the structural noise increments and attribute noise increments within the same mode, structural and attribute information within the same mode is decoupled. Based on this, a joint cross-modal alignment loss is applied. and same-mode decoupling loss Constructing contrastive constraint loss This achieves the training objectives of cross-modal content alignment and attribute-structure decoupling; in addition, to ensure the image reconstruction capability of the pre-trained latent space diffusion model, an image reconstruction loss is introduced. This prompts the pre-trained latent space diffusion model to learn a noise representation; finally, the entire framework employs a contrastive constraint loss. and image reconstruction loss Total loss constituted by all optimization.

[0042] During the inference phase, the encoder of the U-Net denoising network directly extracts the clean structural features between two modalities, and then performs an element-wise addition operation to obtain a complete fused feature. This fused feature is then injected into the corresponding layer of the encoder, serving as an intermediate feature of the U-Net denoising network for subsequent denoising updates. Next, the visible light modal attribute LoRA parameters are injected into the decoder side of the U-Net denoising network. The corresponding noise prediction result is output. Based on the noise prediction result, backdiffusion update is performed step by step to obtain the final fused latent representation. Finally, the latent representations will be fused. Decoder for input variational autoencoder Decoding is performed to obtain the fused image. Through the above reasoning process, the network achieves complementary fusion of infrared and visible light modal content information and generates a fused image with visible light appearance characteristics.

Claims

1. A modal attribute-structure decoupling method for fusing infrared and visible images, characterized in that, This infrared and visible light image fusion method is based on a pre-trained latent space diffusion model. The overall network framework includes a variational autoencoder encoder. Decoders using U-Net denoising network and variational autoencoder ; The aforementioned variational autoencoder The specific implementation process is as follows: Infrared images and visible light images The encoders of the variational autoencoder are input respectively. This yields the corresponding latent representation: in, Representing modes Low-dimensional feature representation in the latent space This indicates the infrared or visible light mode; subsequently, during the forward diffusion process, the... Perform a stepwise noise-adding operation for t time steps to obtain the corresponding noisy latent representation. The calculation process is as follows: in, Represents the noise dispatch coefficient The cumulative product is used to control the first... The proportional relationship between the latent representation and Gaussian noise at each time step. Indicates the first The noise scheduling coefficients corresponding to each time step. This represents Gaussian noise that follows a standard normal distribution. The specific implementation process of the U-Net denoising network is as follows: Noise-adding latent representation Input the U-Net denoising network and output the noise prediction result. ;in, These are the basic parameters for the U-Net denoising network; Two types of LoRA are introduced into the U-Net denoising network: learnable structural LoRA parameters are introduced on the encoder side. Used for modeling modes Structural information; LoRA parameter attribute introduced on the decoder side. Used for modeling modes The attribute information; when calculating the structural noise increment, the LoRA parameters on the decoder side are frozen. Only activate the structure LoRA parameters on the encoder side. And the potential representation of noise addition. Perform noise prediction and obtain the corresponding noise prediction results. Compare the noise prediction results with the noise prediction results. By subtraction, we obtain the modes. Corresponding structural noise increment: in, This represents the structural information learned from the structure LoRA parameters under the condition of frozen basis model parameters, used to describe the modes. The change in structural information relative to the pre-trained latent space diffusion model; when calculating the attribute noise increment, the structural LoRA parameters on the encoder side are frozen. Only activate the LoRA parameter attribute on the decoder side. The attribute noise prediction results were obtained in the same way. And further calculate the modes Corresponding attribute noise increment: in, This represents the attribute information learned from the LoRA parameters under the condition of freezing the pre-trained latent space diffusion model parameters, used to describe the modality. The amount of change in attribute information relative to the pre-trained latent space diffusion model; Then, by minimizing the cosine distance between the structural noise increments of the infrared and visible modes, consistent alignment of cross-modal structural information is achieved, with cross-modal alignment loss... It is expressed as follows: in, This represents the cosine similarity function; simultaneously, by minimizing the cosine similarity between the structural noise increment and the attribute noise increment in the same mode, decoupling of structural and attribute information in the infrared and visible light modes is achieved, along with the same-mode decoupling loss. It is expressed as follows: Contrast constraint loss of noise increment It is expressed as follows: To ensure that the proposed network has stable image reconstruction capabilities, an image reconstruction loss is introduced. Constraints are imposed on the U-Net denoising network; during backdiffusion, the U-Net denoising network is based on the noisy latent representation at each time step. The noise components are predicted step by step, and the latent representation is iteratively denoised and updated using preset noise scheduling coefficients to obtain the final latent representation. .

2. The infrared and visible image fusion method with modal attribute-structure decoupling according to claim 1, characterized in that, The final potential representation Decoder via variational autoencoder The corresponding reconstructed image is obtained through reconstruction. and with the input image The image reconstruction loss is calculated and defined as follows: Among them, the input image Including infrared images and visible light images , ; Indicates the input image With reconstructed image The absolute error per pixel between them.

3. The infrared and visible image fusion method with modal attribute-structure decoupling according to claim 2, characterized in that, The total loss of the proposed network Using contrastive constraint loss and image reconstruction loss Joint optimization is performed as follows: During the inference phase, the infrared image and the visible light image are respectively input into the encoder of the variational autoencoder. , to obtain in modality Low-dimensional feature representation in latent space Next, during the backdiffusion process, a U-Net denoising network is used to predict noise step by step; a complete fused structural feature representation is obtained by adding elements one by one; and infrared structural features are obtained during the feature extraction process of each layer of the U-Net denoising network. With visible light structural features The fused structural features are obtained at the corresponding level through element-wise addition, and their form is as follows: in, Indicates the characteristics of the fusion structure. This represents an element-wise addition operation; By injecting the learned visible light modal properties LoRA parameters The decoding process of the modulated U-Net denoising network is used to impart visible light modal attribute information to the fused structural features, thereby obtaining the noise prediction result after visible light modal attribute modulation. ; During the stepwise process of back diffusion, the modulated noise prediction result By progressively updating the latent representation, a fused latent representation is obtained. Finally, the latent representations are fused. Decoder for input variational autoencoder Decode the image to generate the final fused image. Its form is expressed as: 。

Images