Layered guidance method for 2D image generation of trajectory-based target fine control

Abstract

The application discloses a layered guidance method for 2D image generation based on trajectory target fine control, and relates to the technical field of image generation. The method calculates the control loss, suppression loss and repair loss of the cross-attention diagram of the intermediate layer and the up-sampling layer of the Unet model part through the design of the layered guidance mechanism, and calculates the loss according to the energy function designed by the layered guidance to update the potential features through gradient back propagation, so that the effective layout control of the target is realized by combining the control loss and the suppression loss at low resolution, and the target shape control is ensured by combining the control loss and the repair loss at medium and high resolutions. The layered guidance method for 2D image generation based on trajectory target fine control provided by the application solves the defects that the image generation model based on trajectory cannot stably control the layout and cannot finely control the target, and does not need additional training.

Description

Technical Field

[0001] This invention relates to the field of image generation technology, and in particular to a hierarchical guided method for generating 2D images based on trajectory-based fine control of targets. Background Technology

[0002] In recent years, significant progress has been made in the field of image generation with the rapid development of text-to-image models trained on large-scale datasets. However, an inherent modal gap exists between text and images. This gap makes it difficult for generated images to perfectly match the input text prompts and to provide fine-grained guidance during image generation, such as specifying the exact location or shape of objects.

[0003] To address the limitations of controlling image generation with a single text, some models have made significant progress by introducing additional visual conditions to control image generation, building upon existing large-scale text-to-image generation models.

[0004] Among these, a trajectory-based image generation model (TraDiffusion) emerged, proposing a novel trajectory-based, training-free control method that achieves simpler and more natural image generation control. This model uses a distance matrix centered on the object's trajectory to simulate an object mask; however, this large control area cannot effectively stabilize the generation of object layouts. Furthermore, its approach lacks consideration for fine-grained target control. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a hierarchical guidance method for 2D image generation based on trajectory-based fine control of target. By designing a hierarchical guidance mechanism, the target layout is controlled at low resolution and the target shape generation is finely controlled at high resolution. This solves the defects of trajectory-based image generation models that cannot stably control the layout and cannot finely control the target, and at the same time, no additional training is required.

[0006] To address the aforementioned technical problems, this invention provides a hierarchical guidance method for 2D image generation based on trajectory-driven fine-grained control, comprising:

[0007] Step 1: Obtain the text prompts, trajectory, and corresponding control text labels for the trajectory;

[0008] Step 2: Use a visual language segmenter and a text encoder to segment the text prompt and extract text features. Use the visual language segmenter to segment the control text labels corresponding to the trajectory and calculate their indices in the cross-attention map list after the text prompt is segmented. Then initialize the latent features of the stable diffusion model.

[0009] Step 3: Convert the trajectory to the same resolution as the controlled cross-attention map as the control region; calculate the control loss for the cross-attention maps of the intermediate and upsampling layers of the Unet model part of the pre-trained stable diffusion model using a hierarchical guidance mechanism. Suppressing losses and repairing losses The loss is calculated based on the energy function of the hierarchical guided design, and the latent features are updated by gradient backpropagation. The control loss aligns the layout with the trajectory at each resolution, the suppression loss suppresses objects outside the trajectory at low resolution, and the repair loss refines the areas not fully controlled by the trajectory using attention feedback at medium and high resolution. The latest latent features are obtained step by step through a denoising process without classification guidance based on the stable diffusion model.

[0010] Step 4: Obtain the generated image under control through the decoder.

[0011] The technical solution provided by this invention has at least the following advantages:

[0012] By designing a hierarchical guidance mechanism, the control loss, suppression loss, and patching loss of the cross-attention map of the intermediate and upsampling layers of the Unet model are calculated. The loss is then calculated based on the energy function of the hierarchical guidance design, and the latent features are updated by gradient backpropagation. Thus, at low resolution, the control loss and suppression loss are combined to achieve effective layout control of the target. At medium and high resolution, the control loss and patching loss are combined to ensure the shape control of the target. This solves the defects of trajectory-based image generation models (TraDiffusion) in that they cannot stably control the layout and cannot finely control the target, and no additional training is required. Attached Figure Description

[0013] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0014] Figure 1 This is a schematic diagram illustrating the deep analysis results of the pre-trained stable diffusion model based on the present invention.

[0015] Figure 2 This is a general flowchart of the method according to an embodiment of the present invention;

[0016] Figure 3 This is an architecture diagram of an embodiment of the present invention;

[0017] Figure 4 This is an example diagram of target instability control based on trajectory morphological graph models in existing technologies;

[0018] Figure 5 This is an example diagram of the 8×8 cross-attention layout control module according to an embodiment of the present invention;

[0019] Figure 6This is a qualitative analysis diagram of the 16×16 cross-attention attention guidance design according to an embodiment of the present invention;

[0020] Figure 7 This is a qualitative analysis diagram of the shape guidance of 32×32 cross attention according to an embodiment of the present invention;

[0021] Figure 8 This is a qualitative comparison diagram of different methods in the embodiments of the present invention and the prior art;

[0022] Figure 9 This is a qualitative analysis diagram of repair losses according to an embodiment of the present invention;

[0023] Figure 10 Qualitative comparison diagram of different methods of the present invention and the prior art in the case of multiple target layout control problems. Detailed Implementation

[0024] This invention provides a hierarchical guidance method for 2D image generation with fine-grained target control based on trajectory. By designing a hierarchical guidance mechanism, it controls the target layout at low resolution and finely controls the target shape generation at high resolution, thus solving the defects of trajectory-based image generation models that cannot stably control the layout and cannot finely control the target, while requiring no additional training.

[0025] The overall concept of the technical solutions in the embodiments of the present invention is as follows:

[0026] To address the aforementioned issues, the inventors conducted a deep analysis of the pre-trained Stable Diffusion model, discovering that its cross-attention mechanism at low resolution focuses more on generating the target layout, while the target shape is refined at high resolution. As resolution increases, the ability to represent target shape becomes stronger, such as... Figure 1As shown. Based on this, the present invention redesigns the trajectory-based image generation model (TraDiffusion), providing a hierarchical guided method for trajectory-based fine-grained control of 2D image generation in the field of artificial intelligence technology. This method includes: obtaining text prompts for image generation provided by the user; obtaining the trajectory drawn by the user and the corresponding control text labels; segmenting the text prompts and extracting text features, and calculating the index position of the trajectory labels in the segmented text prompt list; initializing the initial latent features of the stable diffusion model; transforming the trajectory to the same resolution as the controlled cross-attention map as the control region; calculating the loss of the pre-trained stable diffusion model (Stable Diffusion) at different resolutions using the hierarchical guided mechanism designed according to the present invention; calculating the loss according to the energy function designed by the hierarchical guided method and then backpropagating the gradient to update the latent features; the loss includes control loss, suppression loss, and patching loss. The combination of control loss and suppression loss ensures effective layout control, while the interaction between control loss and patching loss improves shape generation; repeating until the denoising process is completed, decoding the latent features to obtain the generated image.

[0027] This embodiment provides, as Figure 2 As shown, it may include the following steps:

[0028] S1. Obtain text prompts, trajectories, and corresponding control text labels for the trajectories; control text labels are all contained within the text prompts, and one control text label corresponds to at least one trajectory. Users can input text prompts and control text labels for image generation on the front-end page and draw the trajectories corresponding to the control text labels.

[0029] S2. Use the visual language (CLIP) tokenizer and text encoder to segment the text prompt and extract text features. Use the visual language tokenizer to segment the control text labels corresponding to the trajectory and calculate their indices in the cross-attention map list after the text prompt is segmented. Then, initialize the latent features of the stable diffusion model based on the default or user-provided random seeds.

[0030] S3. Transform the trajectory to the same resolution as the controlled cross-attention map as the control region; calculate the control loss for the cross-attention maps of the intermediate and upsampling layers of the Unet model part of the pre-trained stable diffusion model using a hierarchical guidance mechanism. Suppressing losses and repairing losses The loss is calculated based on the energy function of the hierarchical guided design, and then the gradient backpropagation is used to update the latent features. The control loss aligns the layout with the trajectory at each resolution, the suppression loss suppresses objects outside the trajectory at low resolution, and the repair loss refines the areas not fully controlled by the trajectory using attention feedback at medium and high resolution. The latest latent features are obtained step by step through a denoising process without classification guidance based on the stable diffusion model.

[0031] In one possible implementation, the trajectory is transformed to the same resolution as the controlled cross-attention map as the control region, specifically including:

[0032] First, create a binary mask image with the same size as the controlled cross attention map. Calculate the ratio between the original size of the trajectory and the size of the cross attention map. Then, map the coordinates of the trajectory to the size of the binary mask image according to this ratio, and mark the trajectory position as 1 to form the control region.

[0033] The formula for controlling loss is as follows:

[0034]

[0035] in, This represents the value used to calculate control loss; Indicating the first generation of pre-trained U-Net List of cross-attention graphs for layers; This refers to the i-th trajectory in the trajectory list; The text label corresponding to the i-th trajectory; It is a calculation of trajectory labels In the list of cross attention maps Functions for indexing; Represents the i-th trajectory After coordinate transformation operation, and the first Layer Cross Attention Map Control areas with the same resolution.

[0036] The formula for suppressing loss is as follows:

[0037]

[0038] in, This represents the value calculated for the suppression loss.

[0039] The formula for repairing the loss is as follows:

[0040]

[0041] in, This represents the value calculated for repair losses; Before each boot process, by... Extracting the dynamically generated binary mask from the high-response region. Specifically, when the corresponding position The value exceeds the threshold hour, The value is set to 1; otherwise, it is set to 0. This embodiment uses empirical settings. It is 0.4.

[0042] The energy function formula for the hierarchical guided design is as follows:

[0043]

[0044] in, The value calculated by the energy function representing the hierarchical guidance mechanism; , , It is a scaling factor that adjusts the guidance intensity;

[0045] The energy function for controlling the layout of the 8×8 cross attention map in the intermediate layer of U-Net (suppressing objects outside the trajectory at low resolution by suppressing the suppression loss). The energy function for controlling the shape of the 16×16 cross-attention map in the U-Net upsampling layer. The energy function for shape control of the 32×32 cross-attention map in the U-Net upsampling layer (by refining the region not fully controlled by the trajectory at medium to high resolution using attention feedback through patching loss);

[0046] The formula for the energy function of layout control is as follows:

[0047] ,

[0048] in, L represents the value calculated by the layout control energy function. c and L s To control and mitigate losses;

[0049] The formula for the energy function of shape control is as follows:

[0050] ,

[0051] in, The value calculated by the shape control energy function; L f To repair the damage.

[0052] In step 3, the formula for the noise reduction process is:

[0053]

[0054] Among them, z tThis represents the potential representation at time step t; , where n represents the number of control targets; The calculation formula is as follows: , These are predefined coefficients in the pre-trained stable diffusion model, used to control the attenuation or scaling of noise.

[0055] S4. Obtain the controlled generated image through the decoder.

[0056] To verify the technical effectiveness of this invention, this embodiment uses FID to measure the quality of the generated image and DTL to measure the degree of fit between the generated target and the trajectory. However, when the target generation is abnormal, DTL cannot accurately assess whether the target is correctly generated following the trajectory. To solve this problem, this embodiment also proposes a new "IOT (Intersection OverTrajectory) metric," which checks the correctness of object generation by comparing the trajectory with the generated object mask and calculating their overlap ratio. A higher IOT indicates more accurate target generation.

[0057] This embodiment performs quantitative evaluation at COCO2014. Following previous work, this embodiment randomly samples 1000 images from the validation set to create simple and complex trajectory datasets, with each image containing 1 to 3 objects. In the simple trajectory dataset, the trajectory of each object is represented by a single curve, while in the complex trajectory dataset, the trajectory of each object contains 1 to 2 branches. Since the method of this invention emphasizes solving the problem of fine-grained object control, this embodiment constructs a unified dataset containing 500 simple trajectory examples and 1000 complex trajectory examples, totaling 1500 examples, named the “TRAT” dataset, for ablation studies.

[0058] Combination Figure 3 , Figure 4 , Figure 8 , Figure 10 The results of the ablation study experiments for each component in the model of this invention are as follows:

[0059]

[0060] As can be seen, compared with the original Stabel Diffusion model and TraDiffusion-based model, this invention exhibits the strongest level of control. This invention not only provides more stable control over target layout but also enables fine-grained control of the target.

[0061] Combination Figures 4-6 The results of the ablation experiments on the layered guidance mechanism of this invention are as follows:

[0062]

[0063] As can be seen, the best IOT and DTL performance is achieved when all modules of the hierarchical guidance mechanism of this invention are employed. While the FID score increases, this slight quality trade-off is worthwhile for achieving more precise object control. Furthermore, since the ablation dataset of this invention contains only 1,500 images, this gap will narrow as the dataset size increases.

[0064] Combination Figure 6 , Figure 9 The results of the ablation experiments on the layered guidance mechanism of this invention are as follows:

[0065]

[0066] As can be seen, by introducing the repair loss, this invention can effectively solve the problems of discontinuity in target generation and loss of details, while also improving IOT and DTL, thus enhancing the control capability of the method.

[0067] This invention designs a hierarchical guidance mechanism to calculate the control loss, suppression loss, and patching loss of the cross-attention map of the intermediate and upsampling layers of the Unet model. It then calculates the loss based on the energy function of the hierarchical guidance design and updates the latent features through gradient backpropagation. This achieves effective layout control of the target by combining control loss and suppression loss at low resolution, and ensures target shape control by combining control loss and patching loss at medium and high resolution. This solves the defects of trajectory-based image generation models (TraDiffusion) in that they cannot stably control the layout and cannot finely control the target, and does not require additional training.

[0068] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A hierarchical guided method for generating 2D images based on trajectory-based fine-grained control of targets, characterized in that, include: Step 1: Obtain the text prompts, trajectory, and corresponding control text labels for the trajectory; Step 2: Use a visual language segmenter and a text encoder to segment the text prompt and extract text features. Use the visual language segmenter to segment the control text labels corresponding to the trajectory and calculate their indices in the cross-attention map list after the text prompt is segmented. Then initialize the latent features of the stable diffusion model. Step 3: Convert the trajectory to the same resolution as the controlled cross-attention map as the control region; calculate the control loss for the cross-attention maps of the intermediate and upsampling layers of the Unet model part of the pre-trained stable diffusion model using a hierarchical guidance mechanism. Suppressing losses and repairing losses The loss is calculated based on the energy function of the hierarchical guided design, and the potential features are updated by gradient backpropagation. The control loss aligns the layout with the trajectory at each resolution, the suppression loss suppresses objects outside the trajectory at low resolution, and the repair loss refines the areas not fully controlled by the trajectory using attention feedback at medium and high resolution. A step-by-step denoising process based on a stable diffusion model, without classification guidance, is used to obtain the latest latent features. The repair loss L f The formula is as follows in, This represents the value calculated for repair losses; Before each boot process, by... Extract the dynamically generated binary mask from the high-response region. The energy function formula for the hierarchical guided design is as follows: in, The value calculated by the energy function representing the hierarchical guidance mechanism; , , It is a scaling factor that adjusts the guidance intensity; The energy function for controlling the layout of the 8×8 cross-attention map in the intermediate layers of U-Net. The energy function for controlling the shape of the 16×16 cross-attention map in the U-Net upsampling layer. The energy function for shape control of the 32×32 cross-attention map in the U-Net upsampling layer; The formula for the energy function of layout control is as follows: ， in, L represents the value calculated by the layout control energy function. c and L s To control and mitigate losses; The formula for the energy function of shape control is as follows: , in, The value calculated by the shape control energy function; L f To repair the damage; Step 4: Obtain the generated image under control through the decoder.

2. The method according to claim 1, characterized in that, The control loss L c The formula is as follows: in, This represents the value used to calculate control loss; Indicating the first generation of pre-trained U-Net List of cross-attention graphs for layers; This refers to the i-th trajectory in the trajectory list; The text label corresponding to the i-th trajectory; It is a calculation of trajectory labels In the list of cross attention maps Functions for indexing; Represents the i-th trajectory After coordinate transformation operation, and the first Layer Cross Attention Map Control areas with the same resolution.

3. The method according to claim 2, characterized in that, The suppression loss L s The formula is as follows: in, This represents the value calculated for the suppression loss.

4. The method according to claim 1, characterized in that: In step 3, the formula for the noise reduction process is: Among them, z t This represents the potential representation at time step t; , where n represents the number of control targets; The calculation formula is as follows: , These are predefined coefficients in the pre-trained stable diffusion model, used to control the attenuation or scaling of noise.

5. The method according to claim 1, characterized in that: In step 3, the trajectory is converted to the same resolution as the controlled cross-attention map as the control region, specifically including: Create a binary mask image with the same size as the controlled cross attention map, calculate the ratio between the original size of the trajectory and the size of the cross attention map, then map the coordinates of the trajectory to the size of the binary mask image according to the ratio, and mark the trajectory position as 1 to form the control region.

Images