Multi-modal image synthesis method with contrast parameter decoupling

Abstract

The application belongs to the technical field of deep learning, and discloses a contrast parameter decoupling multimodal image synthesis method, which is constructed based on a diffusion model and mainly consists of a perception compression module, a latent denoising UNet model and a contrast parameter decoupling module.The application solves the problems of multimodal image scarcity and limited perception ability of single modal image in a complex environment based on the contrast parameter decoupling multimodal image synthesis method.The multimodal image generated by the application can fully exert the complementary advantages between modal semantics, effectively expand the multimodal data set, provide high-quality data support and driving for downstream tasks such as object classification, target detection and disaster monitoring, and improve the robustness and reliability of the model in a complex environment.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology and relates to a multimodal image synthesis method with decoupled contrast parameters. Background Technology

[0002] Currently, the technologies related to this invention include three aspects: first, text-conditional guided image generation technology; second, decoupled representation learning technology;

[0003] In recent years, the field of image generation has experienced explosive growth, with its applications deeply penetrating multiple areas such as image synthesis, sample augmentation, attribute transfer, and decoupling learning. With the continuous advancement of deep representation learning, generation paradigms are showing a diversified development trend. Currently, mainstream architectures include flow-based models, generative adversarial networks (GANs), autoregressive models, and diffusion models. These four types of models each have their own advantages in probability density modeling and sampling strategies. Although they differ, they all strive to find the optimal solution between high fidelity and diversity in generated samples, collectively forming the cornerstone of contemporary computer vision image generation tasks.

[0004] Among numerous generative paradigms, diffusion models stand out for their superior generation quality and training stability. Essentially, it's a probabilistic process based on Markov chains: the forward process progressively injects Gaussian noise to corrupt the data into pure noise, while the backward process trains the network to learn an inverse denoising distribution, thus reconstructing structured data from random noise. Compared to the common pattern collapse and training oscillation problems of GANs, diffusion architectures, represented by Denoising Diffusion Probabilistic Models (DDPM) and score-based generative models, exhibit superior distribution coverage and sample fidelity. To further overcome the bottleneck of high computational cost in high-dimensional space for pixel-level diffusion models, Latent Diffusion Models (LDMs), represented by Stable Diffusion, have emerged. This architecture innovatively introduces a perceptual compression mechanism, utilizing a variational autoencoder (VAE) to map images from a high-dimensional pixel space to a low-dimensional latent space. The core denoising process is then executed within the efficient latent space via a U-Net network, flexibly incorporating conditional control signals such as text using a cross-attention mechanism. Finally, the denoised latent features are mapped back to the pixel space via a VAE image decoder. By performing denoising in the latent space, this design significantly reduces computational complexity while substantially improving inference speed and generation stability.

[0005] Text-guided image generation, aiming to generate high-quality images semantically consistent with natural language descriptions, has become a research hotspot in the intersection of computer vision and natural language processing in recent years. Early generation methods were primarily based on Generative Adversarial Networks (GANs), with representative works including StackGAN, AttnGAN, and DF-GAN. These methods fuse textual and image features through multi-stage refinement or attention mechanisms. However, GAN-based methods often face problems such as training instability and pattern collapse, making it difficult to generate diverse high-resolution images. In recent years, generation methods based on Denoising Diffusion Probability Models (DDPM) have made groundbreaking progress. Text-to-image generation techniques, represented by OpenAI's DALL-E 2, Google's Imagen, and Stability AI's Stable Diffusion (SD) model, simulate the Markov chain process of gradually recovering a clear image from Gaussian noise. Combined with powerful pre-trained text encoders such as Contrastive Language-Image Pre-training (CLIP), they achieve fine-grained control over image content and style. This invention employs the SD diffusion model, leveraging its rich prior knowledge obtained through massive pre-training to significantly reduce the training difficulty of multimodal image generation. Simultaneously, by utilizing its efficient latent space and cross-attention mechanism, it achieves strict semantic consistency of the generated multimodal images through textual prompts as a unified control signal.

[0006] Decoupled representation learning eliminates the entanglement and dependency between features by separating the semantic content of an image from its specific attributes as independent latent variables. This mechanism enables generative models to independently perceive and control different visual factors, thereby achieving precise decoupled control of image semantics and attributes in complex generative tasks. Traditional methods mainly focus on constructing an explicit latent space to separate semantic content from specific attributes. A common strategy is to minimize the mutual information between latent factors to ensure their statistical independence. For example, DisPA decouples content and distortion features in point cloud evaluation by minimizing the mutual information between two-branch networks. Similarly, in the field of face swapping, FaceSwapper uses explicit semantic content and attribute encoders to distinguish between identity and attributes. Other recent related works, such as DRL, have built a new decoupling theoretical framework based on group theory, aiming to formally model the decoupling problem in self-supervised learning. On the other hand, to achieve more efficient and robust feature decoupling, parameter efficient fine-tuning (PEFT) methods, especially low-rank adaptive techniques (LoRA), have been widely used. Recent research has focused on using LoRA modules to learn different conceptual features separately, achieving structural feature decoupling. For example, ZipLoRA aims to reduce feature interference between independently trained content LoRA and attribute LoRA by optimizing orthogonality. Similarly, QR-LoRA utilizes the mathematical orthogonality of QR matrix factorization to directly decompose the internal parameter matrix of the diffusion model, thereby achieving endogenous decoupling of semantic content and specific attributes during optimization. In this invention, optimization is anchored in the weight parameter space, and the functional differences between matrices A and B in the LoRA adapter are used to constrain the low-rank matrix (A) to extract modality-invariant semantic bases, while the low-rank matrix (B) acts as a modulator to encode different modal attributes, thus achieving the transformation from single-text cues to multimodal image generation. Summary of the Invention

[0007] For multimodal image generation tasks, the goal of this invention is to generate paired images of multiple modalities. This allows the invention to serve as a data-driven approach, providing high-quality data support and impetus for downstream tasks such as object classification, target detection, and disaster monitoring, thereby improving the robustness and reliability of image generation in complex environments. Current research on multimodal image generation is mainly limited to the paradigm of "multimodal input to single-modal output," that is, using conditions such as text, semantic masks, or metadata to guide the generation of visible light images. However, due to the extreme scarcity of pixel-level paired images between different modalities such as visible light, infrared, and radar, research on multimodal image output is still in its early stages. This results in existing mainstream models generally lacking the ability to model non-visible light modalities, making it difficult to meet the generation requirements of infrared or radar images in complex scenes. To solve these problems, this invention proposes a multimodal image synthesis method with contrast parameter decoupling. The key to this method lies in deeply exploring the internal parameter manifold of the diffusion model and fully exploiting the inherent structural decoupling potential of the low-rank adapter matrix. By leveraging a contrastive optimization mechanism, this method employs a two-stage strategy to sequentially achieve precise decoupling from text to cross-modal invariant semantics, and adaptive reconstruction of specific modal attributes.

[0008] The technical solution of the present invention:

[0009] A multimodal image synthesis method with contrast parameter decoupling, based on a diffusion model, mainly consists of a perceptual compression module, a latent denoising UNet model, and a contrast parameter decoupling module (CPD); the steps are as follows:

[0010] Using a pre-trained variational autoencoder as a perceptual compression module, its image encoder Responsible for processing high-dimensional raw pixel images Projecting onto a low-dimensional latent feature space yields latent features x. Next, the latent features x undergo a forward diffusion noise reduction process, modeled as a Markov chain, where Gaussian noise is progressively injected into the latent features x according to time step t. Obtain noise characteristics x t Then, the noise feature x t The input is fed into a denoising UNet model that has integrated a contrast parameter decoupling module for inverse denoising. This denoising UNet model learns the denoising distribution by minimizing the difference between the predicted noise and the actual noise, thus obtaining the denoised feature x0. During the inverse denoising process, the text condition T is introduced through a cross-attention mechanism, which is synchronized with the time step. Embedded signals serve as guiding signals, controlling the generation of semantic content in the image; finally, the image decoder in the perceptual compression module... Responsible for reconstructing the denoised feature x0 back into the image pixel space, and outputting the final generated image. .

[0011] The contrast parameter decoupling module is inserted into each attention block of the potential denoising UNet model, establishing a new parameter space based on the parameter weights of each attention block. Its optimized structure consists of one semantic matrix A and three independent attribute matrices. Construct; where the dimension of the semantic matrix A is denoted as . Attribute matrix The dimension is denoted as m {1,2,3}, where 1 represents the attribute of modality 1, 2 represents the attribute of modality 2, and 3 represents the attribute of modality 3.

[0012] For the comparison parameter decoupling module inserted in the i-th attention block, use Let represent the pre-trained parameter matrix of the i-th attention block; first, in the pre-trained parameter matrix... Performing singular value decomposition on the above yields three matrices. , ,in, This represents the singular matrix at the output. This represents a diagonal matrix storing the intensity of singular values. The singular matrix at the input is represented as follows:

[0013]

[0014] Where K = min(m1,n), and n are the parameter matrices of the pre-training process. Height and width, Let k be the left singular vector of the i-th attention block. Let k be the singular value of the i-th attention block. Let k be the right singular vector of the i-th attention block;

[0015] Then, based on the singular value distribution of each attention block, from the pre-trained parameter matrix... Extracting the core parameter matrix for:

[0016]

[0017] in, Core parameter matrix Dimensions This indicates extracting the left singular matrix. diagonal matrix And right singular matrix The first r parameter dimensions;

[0018] Core parameter matrix It can be decomposed into two orthogonal bases in different directions, where the diagonal matrix obtained by the decomposition is... And right singular matrix The product of these terms serves as an orthogonal basis for the input, and the left singular matrix... The orthogonal basis serves as the output; two orthogonal bases in different directions serve as the geometric priors for the parameter update space.

[0019] Subsequently, a learnable semantic matrix A is introduced on the orthogonal basis at the input end, and an attribute matrix is ​​introduced on the orthogonal basis at the output end. And let the semantic matrix A be used as a semantic unraveler to capture the semantic information of the image, and the attribute matrix The module acts as an attribute adapter to store the modal attribute information of multimodal images; therefore, the overall optimization framework of the comparison parameter decoupling module is as follows:

[0020]

[0021] in, The diagonal matrix represents the structural paradigm of the i-th attention block during optimization. Right singular matrix and left singular matrix The parameter matrices are kept frozen, and only the semantic matrix A and the attribute matrix are optimized and updated. .

[0022] Training of the parameter decoupling module:

[0023] Using semantic loss L s and attribute loss L a To train the comparison parameter decoupling module;

[0024] Semantic loss L s : Employing a self-supervised contrastive learning method, any latent features If the input features of the i-th attention block are defined, then the semantic matrix A processes the features as follows:

[0025]

[0026] in, This represents the semantic features output by semantic matrix A after processing.

[0027] Based on the processing features of semantic matrix A, semantic features of each of the three modalities are obtained. Semantic features obtained from images of the same category but different modalities are designated as positive sample feature pairs, while semantic features obtained from images of different categories and different modalities are regarded as negative sample feature pairs.

[0028] Therefore, the contrast constraint loss between any pairwise modes u and v is defined as follows:

[0029]

[0030] in, (.) denotes an exponential function, and sim(.T) represents the cosine similarity calculation between any pairwise semantic features under the guidance of textual condition T. and This represents the semantic features obtained by processing images of category a in modes u and v using semantic matrix A. This represents the semantic features obtained by processing an image of category b in modality v with semantic matrix A. and These constitute positive sample feature pairs, while and These constitute negative sample feature pairs, where a≠b; τ represents the temperature hyperparameter, and B represents the total number of samples.

[0031] Since the contrast parameter decoupling module is injected into all attention blocks in the potential denoising UNet model, denoted as i∈{1,2,....,L}, where L represents the total number of attention blocks in the potential denoising UNet model, the contrast constraint loss term of all layers in the potential denoising UNet model is counted, and the final semantic loss L is calculated. s The expression is as follows:

[0032]

[0033] in, This indicates a summation calculation. This indicates that modalities 1 and 2 optimize the contrastive constraint loss of the i-th attention block under the guidance of text condition T. This indicates that modalities 1 and 3 optimize the contrast constraint loss of the i-th attention block under the guidance of text condition T. This indicates that modalities 2 and 3 optimize the contrast constraint loss of the i-th attention block under the guidance of text condition T;

[0034] Attribute loss L a Through contrastive optimization between multimodal models, semantic matrix A has learned to map invariant semantics under the guidance of textual conditions T. At this stage, semantic matrix A is frozen to preserve the learned semantics. Subsequently, the attribute matrix is ​​optimized... To adapt to different modal properties; the standard diffuse noise prediction loss is used to constrain the attribute matrix. To achieve attribute reconstruction, the attribute loss is calculated as follows:

[0035]

[0036] in, Represents the mathematical expectation. It is Gaussian noise added during the forward diffusion noise processing, and obeys... distributed, This indicates that the potential denoising UNet model is in text conditional Noise characteristics guided by time step t Predicted noise reduction; The method for calculating mean squared error;

[0037] The overall optimization loss of the proposed contrast parameter decoupling module is composed of semantic loss. and attribute loss The total loss is expressed as follows:

[0038] .

[0039] The beneficial effects of this invention are as follows: The multimodal image synthesis method based on contrast parameter decoupling solves the current problems of scarce multimodal images and limited perception capabilities of single-modal images in complex environments. The multimodal images generated by this invention can fully leverage the complementary advantages between modal semantics, effectively expand the multimodal dataset, provide high-quality data support and drive for downstream tasks such as object classification, target detection and disaster monitoring, and improve the robustness and reliability of models in complex environments. Attached Figure Description

[0040] Figure 1 It is the overall structure of a multimodal image synthesis method based on contrast parameter decoupling.

[0041] Figure 2 It is a multimodal generation inference architecture based on a multimodal image synthesis method with decoupling of contrast parameters. Detailed Implementation

[0042] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0043] Figure 1 This describes the overall structure of a multimodal image synthesis method with decoupled parameters. First, images of different modalities are sampled in batches from the multimodal dataset. m1 , m2 and m3 The sampled images of different modalities are first input into an image encoder for processing to obtain latent features x, where x is the latent feature obtained from the three modalities respectively. m1 , m2 and m3The term refers to a collective term for the latent features x, which are subjected to forward diffusion noise processing to output noise features x that are close to Gaussian noise. t Then the noise feature x t The input is fed into the latent denoising UNet model to predict noise and obtain the denoised features x0. A contrastive parameter decoupling module is embedded within the latent denoising UNet model, which operates on each attention block to achieve feature decoupling and adaptive attribute reconstruction. Taking the i-th attention block as an example, the latent features... The semantic features are obtained by processing the semantic matrix A. To achieve feature decoupling, and then semantic features Input into different attribute matrices The fused features obtained from the processing are then processed. As the output of the comparison parameter decoupling module, finally, the fused features output by the comparison parameter decoupling module. The features are injected into the attention blocks corresponding to the latent denoising UNet model to complete feature processing. Furthermore, the feature encoding obtained by the text prompt T through the text encoder is concatenated with the time step t and fed into each attention block of the latent denoising UNet model to ensure that image generation can be guided by the text prompt T. By performing multi-layer processing in this way, the denoising feature x0 output by the latent denoising UNet model can be obtained. Finally, the denoising feature x0 is input into the image decoder De for processing and output to obtain the generated multimodal image. m1 , m2 and m3 In the comparison parameter decoupling module, only the semantic matrix A and the attribute matrix are optimized. Wherein, the semantic matrix A is obtained by utilizing the semantic loss L s To optimize and ensure that it can produce robust and invariant semantics, different attribute matrices Then through attribute loss L s To optimize and adapt to the attribute information of different modal images.

[0044] Figure 2 This is the inference architecture for a contrast-parameter decoupling multimodal image synthesis method. The contrast-parameter decoupling module has already been trained; it only needs to be embedded into all attention blocks in the latent denoising UNet model. Finally, by simply inputting a relevant text prompt T specifying the content to be generated, the contrast-parameter decoupling multimodal image synthesis method can generate multiple pixel-level paired multimodal images from sampled Gaussian noise z. m1 , m2 and m3 These multimodal images can significantly expand the community's image data and provide high-quality image data support and drive for downstream image vision tasks (such as object classification, target detection, and disaster monitoring), thus promoting the development of related fields.

Claims

1. A multimodal image synthesis method with contrast parameter decoupling, characterized in that, This multimodal image synthesis method with contrast parameter decoupling is based on a diffusion model and mainly consists of a perceptual compression module, a latent denoising UNet model, and a contrast parameter decoupling module. The steps are as follows: Using a pre-trained variational autoencoder as a perceptual compression module, its image encoder Responsible for processing high-dimensional raw pixel images Project onto a low-dimensional latent feature space to obtain latent features x; Next, the latent feature x is subjected to forward diffusion noise addition. This forward diffusion noise addition process is modeled as a Markov chain, in which Gaussian noise is injected into the latent feature x step by step according to time step t. Obtain noise characteristics x t Then, the noise feature x t The input is fed into a denoising UNet model that has integrated a contrast parameter decoupling module for inverse denoising. This denoising UNet model learns the denoising distribution by minimizing the difference between the predicted noise and the actual noise, thus obtaining the denoised feature x0. During the inverse denoising process, the text condition T is introduced through a cross-attention mechanism, which is synchronized with the time step. Embedded signals serve as guiding signals, controlling the generation of semantic content in the image; finally, the image decoder in the perceptual compression module... Responsible for reconstructing the denoised feature x0 back into the image pixel space, and outputting the final generated image. .

2. The multimodal image synthesis method with decoupled contrast parameters according to claim 1, characterized in that, The contrast parameter decoupling module is inserted into each attention block of the potential denoising UNet model, establishing a new parameter space based on the parameter weights of each attention block. Its optimized structure consists of one semantic matrix A and three independent attribute matrices. Construct; where the dimension of the semantic matrix A is denoted as . Attribute matrix The dimension is denoted as m {1,2,3}, where 1 represents the attribute of modality 1, 2 represents the attribute of modality 2, and 3 represents the attribute of modality 3.

3. The multimodal image synthesis method with decoupled contrast parameters according to claim 2, characterized in that, For the comparison parameter decoupling module inserted in the i-th attention block, use Let represent the pre-trained parameter matrix of the i-th attention block; first, in the pre-trained parameter matrix... Performing singular value decomposition on the above yields three matrices. , ,in, This represents the singular matrix at the output. This represents a diagonal matrix storing the intensity of singular values. The singular matrix at the input is represented as follows: Where K = min( ,n), and n are the parameter matrices of the pre-training process. Height and width, Let k be the left singular vector of the i-th attention block. Let k be the singular value of the i-th attention block. Let k be the right singular vector of the i-th attention block; Then, based on the singular value distribution of each attention block, from the pre-trained parameter matrix... Extracting the core parameter matrix for: in, Core parameter matrix Dimensions This indicates extracting the left singular matrix. diagonal matrix And right singular matrix The first r parameter dimensions; Core parameter matrix It can be decomposed into two orthogonal bases in different directions, where the diagonal matrix obtained by the decomposition is... And right singular matrix The product of these terms serves as an orthogonal basis for the input, and the left singular matrix... The orthogonal basis serves as the output; two orthogonal bases in different directions serve as the geometric priors for the parameter update space. Subsequently, a learnable semantic matrix A is introduced on the orthogonal basis at the input end, and an attribute matrix is ​​introduced on the orthogonal basis at the output end. And let the semantic matrix A be used as a semantic unraveler to capture the semantic information of the image, and the attribute matrix The module acts as an attribute adapter to store the modal attribute information of multimodal images; therefore, the overall optimization framework of the comparison parameter decoupling module is as follows: in, The diagonal matrix represents the structural paradigm of the i-th attention block during optimization. Right singular matrix and left singular matrix The parameter matrices remain frozen; only the semantic matrix A and the attribute matrix are optimized and updated. .

4. The multimodal image synthesis method with decoupled contrast parameters according to claim 3, characterized in that, Training of the parameter decoupling module: Using semantic loss L s and attribute loss L a To train the comparison parameter decoupling module; Semantic loss L s : Employing a self-supervised contrastive learning method, any latent features If the input features of the i-th attention block are defined, then the semantic matrix A processes the features as follows: in, This represents the semantic features output by semantic matrix A after processing. Based on the processing features of semantic matrix A, semantic features of each of the three modalities are obtained. Semantic features obtained from images of the same category but different modalities are designated as positive sample feature pairs, while semantic features obtained from images of different categories and different modalities are regarded as negative sample feature pairs. Therefore, the contrast constraint loss between any pairwise modes u and v is defined as follows: in, (.) denotes an exponential function, and sim(.T) represents the cosine similarity calculation between any pairwise semantic features under the guidance of textual condition T. and This represents the semantic features obtained by processing images of category a in modes u and v using semantic matrix A. This represents the semantic features obtained by processing an image of category b in modality v with semantic matrix A. and These constitute positive sample feature pairs, while and These constitute negative sample feature pairs, where a≠b; τ represents the temperature hyperparameter, and B represents the total number of samples. Since the contrast parameter decoupling module is injected into all attention blocks in the potential denoising UNet model, denoted as i∈{1,2,....,L}, where L represents the total number of attention blocks in the potential denoising UNet model, the contrast constraint loss term of all layers in the potential denoising UNet model is counted, and the final semantic loss L is calculated. s The expression is as follows: in, This indicates a summation calculation. This indicates that modalities 1 and 2 optimize the contrastive constraint loss of the i-th attention block under the guidance of text condition T. This indicates that modalities 1 and 3 optimize the contrast constraint loss of the i-th attention block under the guidance of text condition T. This indicates that modalities 2 and 3 optimize the contrast constraint loss of the i-th attention block under the guidance of text condition T; Attribute loss L a Through contrastive optimization between multimodal models, semantic matrix A has learned to map invariant semantics under the guidance of textual conditions T. At this stage, semantic matrix A is frozen to preserve the learned semantics. Subsequently, the attribute matrix is ​​optimized... To adapt to different modal properties; the standard diffuse noise prediction loss is used to constrain the attribute matrix. To achieve attribute reconstruction, the attribute loss is calculated as follows: in, Represents the mathematical expectation. It is Gaussian noise added during the forward diffusion noise processing, and obeys... distributed, This indicates that the potential denoising UNet model is in text conditional Noise characteristics guided by time step t Predicted noise reduction; The method for calculating mean squared error; The overall optimization loss of the proposed contrast parameter decoupling module is composed of semantic loss. and attribute loss The total loss is expressed as follows: 。

Images