A Multi-Condition Guided Text-to-Image Generation Method Based on Decoupling and Multi-Domain Guidance Strategies
By employing decoupling and multi-domain guidance methods, and combining generation, appearance, and structure guidance branches, the structural alignment and semantic consistency issues in existing technologies are resolved, enabling flexible image generation and the completion of various downstream tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-13
- Publication Date
- 2026-03-13
AI Technical Summary
Existing image generation methods cannot simultaneously guarantee structural alignment and semantic consistency, and the trained models are inflexible and difficult to adapt to new conditions, resulting in poor generation performance.
By employing a decoupling and multi-domain guided strategy, we construct generation, appearance, and structure guided branches, and combine spatial and frequency domain supervision to achieve appearance consistency and structure alignment. We use principal component analysis and wavelet transform to extract features, calculate the loss, and update the latent features of the generation branches.
The generated images are highly aligned with structural conditions, have plug-and-play capabilities, can adapt to various structural controls, reduce computational overhead, and complete a variety of downstream tasks.
Smart Images
Figure CN120047565B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image generation and computer vision technology, and in particular relates to a text-to-image generation method based on decoupling and multi-domain guidance with multi-condition guidance, which aims to improve the semantic alignment and layout accuracy of combined text-to-image generation. Background Technology
[0002] In recent years, the emergence of diffusion models has driven significant progress in image generation tasks. Subsequently, large-scale diffusion models were derived after training on large amounts of paired text-image data that consumed substantial computational resources, achieving satisfactory results in text-to-image generation and image editing. However, text can only achieve basic content generation because it only provides coarse-grained information such as style and color, which is insufficient to accurately describe fine-grained spatial layout to achieve controllable text-to-image generation. Therefore, relying solely on previous large-scale diffusion models leads to poor controllability.
[0003] Some works have attempted to address the aforementioned controllability deficiencies. For example, ControlNet introduces a lightweight adapter outside its pre-trained text-to-image diffusion model to accept a series of user-input structural controls (such as edge maps, depth maps, and segmentation maps). However, ControlNet requires training different adapters for different control conditions, a time-consuming and resource-intensive process. Based on this, some fine-tuning-based methods attempt to design a unified architecture for multi-condition guided image generation. However, these methods overemphasize structural control at the expense of textual content expression, inevitably leading to semantic misalignment with the text prompts. This is because these methods simply utilize the hybrid features of textual conditions and structural control as guiding conditions for image generation, resulting in pixel-level structural control dominating the generation process while ignoring the visual expression in the text prompts. Furthermore, these methods typically focus only on spatial domain features, neglecting the crucial frequency domain representation, leading to poor detail texture and structural consistency.
[0004] Recently, some methods have attempted to address the resource consumption issues of training-based methods by employing training-free approaches. Among them, Plug-and-Play (PnP) proposes a training-free, controllable text-to-image generation method that modulates attention features by injecting intermediate features. However, PnP only designs a single structure-guided branch to achieve structural alignment with the control signal, which inevitably leads to appearance leakage—the appearance of the control signal permeates into the generated image. Another recently proposed training-free method, called FreeControl, is a two-stage network that first models a linear subspace of intermediate features and then generates images within that subspace under appearance and structure supervision. Because it is a non-end-to-end architecture, FreeControl suffers from the complexity of modeling the subspace in the first stage, resulting in inflexibility and poor practical application. This is mainly because a specific subject requires a corresponding subspace; however, this subspace needs to generate dozens of seed images for that object, which is laborious and costly.
[0005] In summary, existing methods cannot generate multi-condition guided images that simultaneously guarantee structural alignment and semantic consistency. Furthermore, these training-based models lack flexibility and cannot adapt to new conditions, leading to difficulties in deployment. Therefore, this paper proposes a multi-condition guided text-to-image generation method based on decoupling and multi-domain guidance. This method can adapt well to various structural conditions without training and simultaneously ensures high alignment with both structural conditions and text, achieving superior results compared to existing methods. In addition, this method can be plugged and played into various large-scale generative models, effectively completing common downstream tasks. Summary of the Invention
[0006] To address the aforementioned problems, this invention proposes a text-to-image generation method based on decoupling and multi-domain guidance with multiple conditions. Its key feature is the decoupling of appearance and structure generation, and the combined spatial and frequency domain guidance to achieve appearance consistency. The method includes the following steps:
[0007] 1. Construct a pre-trained stable diffusion model as the generation branch and initialize it with random noise. Input the user text and random noise into the generation model to perform diffusion generation. Extract the three components of the self-attention layer in the diffusion generation process: query Q, key K, and value V, and obtain the self-attention map in the generation process based on query Q and key K;
[0008] 2. Construct an appearance guidance branch, load a pre-trained stable diffusion model, and copy the initial random noise from the generation model as the initial noise for the appearance guidance branch. Input this noise, along with the user text, into the appearance guidance branch for the diffusion process. Extract the values of the self-attention layer during the appearance guidance process.
[0009] 3. Construct a structure guidance branch and load a pre-trained stable diffusion model; obtain condition-related noise from the structure control conditions using a denoising diffusion implicit model inversion method, and input it into the stable diffusion model. Extract the queries and keys of the self-attention layer during the structure guidance process to obtain the self-attention graph of the structure guidance branch;
[0010] 4. Principal component extraction (PCE) is used to extract the principal components of the self-attention map and values during the generation process, which serve as the structural and appearance representations of the generation branch, respectively. Simultaneously, PCE is used to extract the principal components of the appearance-guided branch values as the appearance-guided representation, and PCE is used to extract the principal components of the self-attention map of the structure-guided branch as the structure-guided representation. The structure-guided representations of the generation branch and the structure-guided branch are then processed using wavelet transform to extract features in four frequency bands. The high-frequency-related features in the four frequency bands are summed to obtain the frequency domain structure representation.
[0011] 5. Calculate and sum the appearance supervision loss, spatial structure supervision loss, and frequency structure supervision loss; the total loss is used to penalize and update the latent features of the generated branches through gradient propagation, and then used for subsequent denoising until the iteration step is 0. Finally, the image is restored to the pixel space using the image decoder to obtain the final generated image.
[0012] Step 1 includes the following steps:
[0013] Initialize a low-dimensional random noise that follows a standard Gaussian distribution, which is represented as follows;
[0014] (1)
[0015] The input text is encoded using a text encoder to obtain the mapped text embedding vector, which is represented as follows;
[0016] (2)
[0017] in For input text, This indicates text encoder encoding. This is a text embedding vector.
[0018] Generative branches are constructed using a pre-trained stable diffusion model, and the query, key, and value of the generative branches from the attention layer are extracted. This part is represented as follows:
[0019] (3)
[0020] in For the iteration time step, Let these represent the query, value, and key in the t-th iteration step, respectively. To generate branches, for Latent features of time steps.
[0021] The attention graph for generating branches is calculated based on the obtained query and key, and this part is represented as:
[0022] (4)
[0023] in For attention maps, express and The dimension is used to normalize the softmax value.
[0024] Step 2 includes the following steps:
[0025] The initial random noise of the copied branch is used as the initial noise for the appearance-guided branch. This part is represented as:
[0026] (5)
[0027] Construct an appearance-guided branch, extracting values from the attention layer. This part is represented as:
[0028] (6)
[0029] in For the iteration time step, These represent the values of the appearance-guided branch. For appearance guidance branches, For appearance guidance branches Hidden features of time steps.
[0030] Step 3 includes the following steps:
[0031] The initial noise of the structural guiding branch is obtained by using the DDIM inversion method to obtain the structural control conditions; this part is expressed as:
[0032] (7)
[0033] In the formula It is an inversion process. This indicates the structure control of the input. Initial noise for guiding branches in the structure.
[0034] Extract the query and key of the self-attention block in the structure-guided branch; this part is expressed as:
[0035] (8)
[0036] The attention graph of the structure-guided branch is calculated based on the obtained queries and keys, and this part is represented as follows;
[0037] (9)
[0038] Step 4 includes the following steps:
[0039] The appearance representation of generating branches and appearance-guided branches is extracted using principal component extraction technology. This part is expressed as:
[0040] (10)
[0041] In the formula Principal component analysis indicates that... and These represent the appearance representations of the generated branch and the appearance-guided branch, respectively.
[0042] The structural representations of generative branches and structure-guided branches are extracted using principal component extraction (PCE) techniques. This part is expressed as:
[0043] (11)
[0044] In the formula Principal component analysis indicates that... and These represent the structural representations of generating branches and structurally guided branches, respectively.
[0045] High-frequency structural representations are extracted from the structural representations of generating branches and structure-guided branches using wavelet transform, and this part is expressed as:
[0046] (12)
[0047] In the formula It is wavelet transform. and These represent the high-frequency appearance representations of generating branches and structural guiding branches, respectively.
[0048] Step 5 includes the following steps:
[0049] The calculation of the appearance supervision loss is expressed as follows:
[0050] (13)
[0051] In the formula This indicates the amount of principal component extracted.
[0052] The calculation of the spatial structure supervision loss is expressed as follows:
[0053] (14)
[0054] The frequency domain structure supervision loss is calculated, and this part is expressed as:
[0055] (15)
[0056] The sum of the losses from the three parts is calculated, and the latent features of the generated branch are updated. This part is expressed as:
[0057] (16)
[0058] (17)
[0059] In the formula , and These are hyperparameters used to balance the three boot branches. Hyperparameters represent the guiding strength.
[0060] Latent features are used for subsequent generation. When the iteration step is 0, an image decoder maps them from the low-dimensional latent space to the pixel space to obtain an image. The generation process is guided by both appearance-guided and structure-guided branches. The former uses principal component supervision to supervise the appearance representation, preserving the appearance of the input text and avoiding weakening the text content expression. The latter guides the generation process to align with the control signal structure through multi-domain structure supervision.
[0061] Beneficial Effects: Compared with existing technologies, this invention provides a multi-condition guided text-to-image generation method based on decoupling and multi-domain guided strategies. This method can generate images that conform to text descriptions and are structurally aligned based on arbitrary structural signals and text content, producing the following beneficial effects:
[0062] 1. No Training Required for Adapting to Arbitrary Structural Controls: Benefiting from the powerful multi-domain structural guidance strategy proposed in this method, which combines frequency and spatial domain supervision, it achieves strong guidance for generating branch structures. By using implicit structural representations, it enhances the structural consistency and high alignment between the generated image and the input structural control. This implicit guidance method enables the method to flexibly adapt to various types of structural control, generating images with diverse structures.
[0063] 2. Plug and Play: This method requires no training and can be directly applied to different types of generative models to achieve various generation effects. Training large models is expensive, and this method can significantly reduce computational overhead, keep up with the evolution of large models, and ensure excellent image generation results.
[0064] 3. Strong Generalization: This method can generalize to complete many common downstream tasks, such as image editing, image deblurring, image colorization, and image restoration. For single-object image editing tasks, we can achieve accurate editing. For multi-object image editing tasks, we can achieve partial editing. For image colorization and image deblurring, we support edge maps as structural control inputs and increase the penalty factors of the two guiding branches. In the image restoration task, we choose GLIGEN as our base generative model and provide bounding box information to obtain good generation results. Relying on decoupled appearance-guided branches and structure-guided branches and multi-domain structural supervision, this method has strong generalization ability and can complete many downstream tasks. Attached Figure Description
[0065] Figure 1 This is a flowchart illustrating the text-to-image generation process guided by the structure control of this invention.
[0066] Figure 2 This invention provides an analysis of appearance representation, structural representation, and frequency domain characteristics.
[0067] Figure 3 The images show the results of single-control single-object generation, single-control multiple-object generation, and multiple-control multiple-object generation completed by this invention.
[0068] Figure 4 The results of this invention are generated by eight different structural control inputs on three different versions of the stable diffusion model.
[0069] Figure 5 This is a quantitative result diagram of the generated images on the ImageNet-R-TI2I dataset with four different structural control inputs, based on the plug-and-play stable diffusion model of this invention with three different versions.
[0070] Figure 6 and Figure 7 This figure shows the qualitative and quantitative comparison results of the present invention with other methods on the ImageNet-R-TI2I dataset.
[0071] Figure 8 This figure shows the quantitative comparison results of the present invention with other methods on the ImageNet-R-TI2I dataset under a single condition and a single evaluation metric.
[0072] Figure 9 and Figure 10 These are the ablation experiment results for the number of principal components and the ablation experiment results for the three supervision functions, respectively.
[0073] Figure 11 and Figure 12 These are the qualitative and quantitative results of the ablation experiment for the appearance representation selection of this invention.
[0074] Figure 13 The results of this invention on downstream tasks are shown in the figure. Detailed Implementation
[0075] The multi-condition guided text generation image method based on decoupling and multi-domain guidance strategy of the present invention can accurately express the appearance information in the text while being highly aligned with the control conditions in terms of structure by utilizing decoupled appearance and structure guidance branches and multi-domain guidance strategy, thereby avoiding the problem of overemphasizing spatial structure and neglecting text expression.
[0076] The invention will be further described below with reference to specific embodiments.
[0077] Example 1:
[0078] like Figure 1 As shown, the overall architecture of this method consists of three parts: the generation process, the appearance-guided branch, and the structure-guided branch. The user input text is a sculpture of a castle, and the spatial guidance condition is an edge map of a castle. The generation process, appearance-guided branch, and structure-guided branch are initialized using three identical parallel stable diffusion models. The text information is input into the generation process and appearance-guided branch, and a Gaussian-distributed random noise is initialized as the initial noise for both. Subsequently, the spatial control condition is recovered into low-dimensional latent features using DDIM inversion and then input into the structure-guided branch for the diffusion process.
[0079] During the denoising process using three stable diffusion models, we extract the query Q, key K, and value V of the first self-attention block in the decoder during the generation process, the value V of the appearance guidance branch, and the Q and key K of the structure guidance branch. Then, we use principal component analysis to extract the first three principal components of these components as appearance and structure representations. Subsequently, we supervise the appearance and structure representations of the generation process and the guidance branch using appearance supervision, spatial structure supervision, and frequency structure supervision, respectively. After calculating the loss, we update the latent features of the generation process through gradient backpropagation. When the denoising iteration steps approach 0, denoising is complete, and the latent features are restored to pixel space using the image decoder to obtain the image.
[0080] Figure 2 This invention relates to the analysis of appearance representation, structural representation, and frequency domain features in its multi-condition guided text generation image method based on decoupling and multi-domain guided strategies. Figure 2As shown on the left, we extract Q, K, V, and attention map S from the self-attention layer of the U-Nets decoder every 200 iterations and obtain the first three principal components using Principal Component Analysis (PCA). Subsequently, we map these principal components to pseudo-color images using a pseudo-color technique. In the pseudo-color images, regions of the same color represent similar semantic information, while brighter regions correspond to regions with higher values in the original feature maps. We find that the principal component of V expresses richer appearance information (regions of the same color are roughly distributed across the castle surface), while the self-attention map S represents stronger structural information (the edges of the castle are brightest in the self-attention map). Inspired by these findings, we choose the principal component of V as the appearance representation for appearance supervision and the principal component of S as the structural representation to guide the generation process.
[0081] Furthermore, to investigate the relationship between frequency domain features and structural control, we employ Discrete Wavelet Transform (DWT) to extract high-frequency features from pseudo-color images. The visualization results are shown below. Figure 2 As shown on the right. We found that the high-frequency features are structurally roughly consistent with the control signal, which helps to achieve more comprehensive structural supervision in the structure-guided branch. This key finding motivates our method to utilize the basic frequency features as additional structural guidance to achieve better structural consistency.
[0082] Figure 3 The images show the results of single-control single-object generation, single-control multiple-object generation, and multiple-control multiple-object generation completed by the multi-condition guided text generation image method based on decoupling and multi-domain guided strategies of the present invention.
[0083] Figure 4 The present invention provides a multi-condition guided text-to-image method based on decoupling and multi-domain guidance strategies, which can be plugged and played onto three different versions of stable diffusion models, generating result images with eight different structural control inputs.
[0084] Figure 5 This image shows the quantitative results of the text-to-image generation method based on decoupling and multi-domain guidance strategy of the present invention, which is plug-and-play with three different versions of stable diffusion models, on the ImageNet-R-TI2I dataset with four different structural control inputs.
[0085] Figure 6 and Figure 7 The figure shows the qualitative and quantitative comparison results of the multi-condition guided text generation image method based on decoupling and multi-domain guided strategy of the present invention with other methods on the ImageNet-R-TI2I dataset.
[0086] Figure 8The figure shows the quantitative comparison results of the multi-condition guided text generation image method based on decoupling and multi-domain guided strategy of the present invention with other methods on the ImageNet-R-TI2I dataset with single-condition single evaluation index.
[0087] Figure 9 and Figure 10 These are, respectively, the ablation experiment results of the multi-condition guided text generation image method based on decoupling and multi-domain guided strategies of the present invention for the number of principal components, and the ablation experiment results of three supervised functions.
[0088] Figure 11 and Figure 12 These are the qualitative and quantitative results of ablation experiments on the appearance representation selection of the multi-condition guided text generation image method based on decoupling and multi-domain guidance strategies of the present invention.
[0089] Figure 13 The results of the text-to-image generation method based on decoupling and multi-domain guidance strategies of the present invention on downstream tasks are shown in the figure.
[0090] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
[0091] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.
Claims
1. A text-to-image generation method based on decoupling and multi-domain guidance strategies under multi-condition guidance, characterized in that... Decoupling the generation of appearance and structure, and achieving appearance consistency through multi-domain guidance in both spatial and frequency domains, includes the following steps: S1. Construct a pre-trained stable diffusion model as the generation branch and initialize a random noise; input the user text and random noise into the generation model to perform diffusion generation; extract the three components of the self-attention layer in the diffusion generation process: query Q, key K and value V, and obtain the self-attention map in the generation process based on query Q and key K. S2, construct the appearance guidance branch, load the pre-trained stable diffusion model, and copy the initial random noise in the generation model as the initial noise of the appearance guidance branch. Input it into the appearance guidance branch along with the user text to carry out the diffusion process; extract the value of the self-attention layer in the appearance guidance process. S3, construct the structure guidance branch and load the pre-trained stable diffusion model; obtain the condition-related noise from the structure control conditions through the denoising diffusion implicit model inversion method and input it into the stable diffusion model; extract the queries and keys of the self-attention layer in the structure guidance process to obtain the self-attention graph of the structure guidance branch; S4. Principal component extraction (PCE) is used to extract the principal components of the self-attention map and values during the generation process, which are used as the structural and appearance representations of the generation branch, respectively. At the same time, PCE is used to extract the principal components of the appearance guidance branch values as the appearance guidance representation, and PCE is used to extract the principal components of the self-attention map of the structure guidance branch as the structure guidance representation. The structural guidance representations of the generation branch and the structure guidance branch are used to extract four frequency band features through wavelet transform, and the high-frequency related features in the four frequency bands are added to obtain the frequency domain structural representation. S5, calculate and sum the appearance supervision loss, spatial structure supervision loss and frequency structure supervision loss; the total loss is used to penalize and update the latent features of the generated branch through gradient propagation, and then to perform subsequent denoising until the iteration step is 0. Finally, the image is restored to the pixel space using the image decoder to obtain the final generated image.
2. The text-to-image generation method based on decoupling and multi-domain guidance strategies as described in claim 1, characterized in that, The step S1 of building and generating branches includes the following steps: S11, initialize a low-dimensional random noise that follows a standard Gaussian distribution, this part is represented as (1) S12, the input text is encoded using a text encoder to obtain the mapped text embedding vector, which is represented as follows: (2) in For input text, This indicates text encoder encoding. For text embedding vectors; S13, using a pre-trained stable diffusion model to construct generative branches, and extracting the query, key, and value from the attention layer of the generative branches, this part is represented as (3) in For the iteration time step, Let these represent the query, value, and key in the t-th iteration step, respectively. To generate branches, for Hidden features of time steps; S14, Based on the query and key obtained in step S13, calculate the attention map for generating the branch, which is represented as: (4) in For attention maps, express and The dimension is used to normalize the softmax value.
3. The text-to-image generation method based on decoupling and multi-domain guidance strategies as described in claim 2, characterized in that, Step S2 specifically involves: S21, the initial random noise of the copied branch is used as the initial noise of the appearance-guided branch, this part is expressed as (5) S22, construct the appearance-guided branch, extracting values from the attention layer; this part is represented as... (6) in For the iteration time step, These represent the values of the appearance-guided branch. For appearance guidance branches, For appearance guidance branches Latent features of time steps.
4. The text-to-image generation method based on decoupling and multi-domain guidance strategies as described in claim 3, characterized in that, Step S3 specifically involves: S31, the initial noise of the structural guiding branch is obtained by using the DDIM inversion method to obtain the structural control conditions. This part is expressed as: (7) In the formula It is an inversion process. This indicates the structure control of the input. Initial noise for guiding branching in the structure; S32, extract the query and key of the self-attention block in the structure-guided branch, which is expressed as: (8) S33, Based on the queries and keys obtained in step S32, calculate the attention map of the structure-guided branch, which is represented as follows: (9)。 5. The text-to-image generation method based on decoupling and multi-domain guidance strategies as described in claim 4, characterized in that, Step S4 specifically involves: S41, Based on the values obtained in steps S13 and S22, principal component extraction (PCE) is used to extract the appearance representations of the generating branches and appearance-guided branches. This part is expressed as: (10) In the formula Principal component analysis indicates that... and These represent the appearance representations of the generated branch and the appearance-guided branch, respectively. S42, based on the values obtained in steps S13 and S32, principal component extraction (PCE) is used to extract the structural representations of the generating branches and structure-guided branches. This part is expressed as: (11) In the formula Principal component analysis indicates that... and These represent the structural representations of generating branches and structurally guided branches, respectively. S43, using wavelet transform, high-frequency structural representations are extracted from the structural representations of generating branches and structure-guided branches respectively. This part is expressed as: (12) In the formula It is wavelet transform. and These represent the high-frequency appearance representations of generating branches and structural guiding branches, respectively.
6. The text-to-image generation method based on decoupling and multi-domain guidance strategies as described in claim 5, characterized in that, Step S5 specifically involves: S51, Calculate the appearance supervision loss, which is expressed as: (13) In the formula Indicates the quantity of principal components extracted; S52, Calculate the spatial structure supervision loss, which is expressed as: (14) S53, Calculate the frequency domain structure supervision loss, which is expressed as: (15) S54, calculate the sum of the losses of the three parts, and update the latent features of the generated branch. This part is expressed as: (16) (17) In the formula , and The hyperparameters are used to balance the three boot branches. Hyperparameters represent guidance strength; S55, the latent features are used for subsequent generation. When the iteration step is 0, the image decoder is used to map them from the low-dimensional latent space to the pixel space to obtain an image. The generation process is guided by both appearance-guided branches and structure-guided branches. The former uses principal component supervision to supervise the appearance representation, preserving the appearance in the input text and avoiding weakening the expression of text content; the latter guides the generation process to align with the control signal structure through multi-domain structure supervision.