Layered guidance method for 2D image generation of target fine control based on trajectory

Abstract

The invention discloses a hierarchical guidance method for 2D image generation based on track target fine control, and relates to the technical field of image generation. According to the method, control loss, suppression loss and repair loss of a cross attention map of a middle layer and an up-sampling layer of a Unet model part are calculated by designing a hierarchical guidance mechanism, loss is calculated according to an energy function of hierarchical guidance design, and potential features are updated through gradient return; therefore, effective layout control of the target is realized by combining control loss and suppression loss under low resolution, and shape control of the target is ensured by combining control loss and repair loss under middle and high resolution. According to the hierarchical guiding method for 2D image generation of target fine control based on the track, the defects that a track-based image generation model cannot stably control the layout and cannot finely control the target are overcome, and meanwhile extra training is not needed.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and in particular to a hierarchical guidance method for 2D image generation based on trajectory-based target fine control. Background Art

[0002] In recent years, with the rapid development of text-generated image models trained on large-scale datasets, the field of image generation has made significant progress. However, there is an inherent modality gap between text and images. This gap makes it difficult for the generated images to fully match the input text prompts, and it is also difficult to provide fine guidance in the image generation process, such as specifying the exact location or shape of the object.

[0003] To address the limitations of single text-controlled image generation, some models have made significant progress by introducing additional visual conditions to control image generation based on existing large-scale text-to-image models.

[0004] Among them, the trajectory-based image generation model (TraDiffusion) has emerged, which proposes a new trajectory-based training-free control method to achieve simpler and more natural image generation control. The model uses a distance matrix centered on the object trajectory to simulate the object mask, but this large control area cannot effectively stabilize the generation of the object layout. In addition, its method lacks consideration of fine control of the target. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a hierarchical guidance method for 2D image generation with fine control of trajectory-based targets. By designing a hierarchical guidance mechanism, the target layout is controlled at low resolution, and the target shape generation is finely controlled at high resolution, which solves the defects of the trajectory-based image generation model that cannot stably control the layout and cannot finely control the target, and no additional training is required.

[0006] In order to solve the above technical problems, the present invention provides a hierarchical guidance method for 2D image generation based on trajectory-based target fine control, comprising: Step 1: Get the text prompt, trajectory and control text label corresponding to the trajectory; Step 2: Use the visual language word segmenter and text encoder to segment the text prompt and extract text features. Use the visual language word segmenter to segment the control text label corresponding to the trajectory and calculate its subscript in the cross-attention map list after the text prompt is segmented; then initialize the potential features of the stable diffusion model; Step 3: Convert the trajectory to the same resolution as the controlled cross-attention map as the control area; calculate the control loss, suppression loss and repair loss for the cross-attention map of the intermediate layer and upsampling layer of the Unet model part of the pre-trained stable diffusion model through the hierarchical guidance mechanism, and calculate the loss according to the energy function designed by hierarchical guidance and then pass the gradient back to update the potential features; the control loss aligns the layout with the trajectory at each resolution, the suppression loss suppresses objects outside the trajectory at low resolution, and the repair loss uses attention feedback to refine the area not completely controlled by the trajectory at medium and high resolutions; the denoising process based on the stable diffusion model without classification guidance is denoised step by step to obtain the latest potential features; Step 4: Obtain the controlled generated image through the decoder.

[0007] The technical solution provided by the present invention has at least the following advantages: By designing a hierarchical guidance mechanism, the control loss, suppression loss and patching loss of the cross-attention map of the intermediate layer and upsampling layer of the Unet model are calculated, and the loss is calculated according to the energy function of the hierarchical guidance design and the gradient is passed back to update the potential features. In this way, the control loss and suppression loss are combined to achieve effective layout control at low resolution, and the control loss and patching loss are combined to ensure target shape control at medium and high resolutions. The defects of the trajectory-based image generation model (TraDiffusion) that it cannot stably control the layout and cannot finely control the target are solved, and no additional training is required. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0009] Figure 1 It is a schematic diagram of the in-depth analysis results of the pre-trained stable diffusion model of the present invention; Figure 2 It is an overall flow chart of the method of the embodiment of the present invention;.

[0010] Figure 3 is a structural diagram of an embodiment of the present invention; Figure 4 It is an example diagram of target instability control based on trajectory graph model in the prior art; Figure 5 is an example diagram of a layout control module of 8×8 cross attention according to an embodiment of the present invention; Figure 6 It is a qualitative analysis diagram of the attention guiding design of 16×16 cross attention in an embodiment of the present invention; Figure 7 It is a qualitative analysis diagram of the shape guidance of 32×32 cross attention according to an embodiment of the present invention; Figure 8is a qualitative comparison diagram of different methods of the embodiments of the present invention and the prior art; Fig. 9 is a qualitative analysis diagram of the repair loss according to the embodiment of the present invention; Fig.10 Qualitative comparison diagram of the embodiments of the present invention and different methods in the prior art under multiple target layout control problems. DETAILED DESCRIPTION

[0011] The embodiment of the present invention provides a hierarchical guidance method for 2D image generation with fine control of trajectory-based targets. By designing a hierarchical guidance mechanism, the target layout is controlled at low resolution and the target shape generation is finely controlled at high resolution. This solves the defects of the trajectory-based image generation model that cannot stably control the layout and cannot finely control the target, and no additional training is required.

[0012] The technical solution in the embodiment of the present invention has the following general idea: In order to solve the above problems, the inventors conducted an in-depth analysis of the pre-trained stable diffusion model and found that its cross-attention mechanism at low resolution focuses more on the generation of target layout, while the shape of the target is refined at high resolution. As the resolution increases, the target shape has stronger expressive power, such as Figure 1 As shown. Based on this, the present invention redesigns a trajectory-based image generation model (TraDiffusion), and provides a hierarchical guidance method for 2D image generation based on trajectory-based target fine control in the field of artificial intelligence technology, which includes: obtaining a text prompt for image generation provided by a user, obtaining a trajectory drawn by a user and a control text label corresponding to the trajectory; segmenting the text prompt and extracting text features, and calculating the subscript position of the trajectory label in the text prompt list after segmentation, and initializing the initial potential features of the stable diffusion model; transforming the trajectory to the same resolution as the controlled cross-attention map as the control area through coordinates, and calculating the loss of the cross-attention map at different resolutions of the pre-trained stable diffusion model (Stable Diffusion) according to the hierarchical guidance mechanism designed by the present invention, and calculating the loss according to the energy function designed by the hierarchical guidance and then gradient back-transferring to update the potential features, and the loss includes control loss, suppression loss and repair loss. The combination of control loss and suppression loss ensures effective layout control, and the interaction between control loss and repair loss improves shape generation; repeat until the denoising process is completed, and decode the potential features to obtain the generated image.

[0013] This embodiment provides, Figure 2 As shown, the following steps may be included: S1. Obtain text prompts, trajectories, and control text labels corresponding to the trajectories; control text labels are all included in the text prompts, and one control text label corresponds to at least one trajectory. Users can input text prompts and control text labels for image generation on the front-end page, and draw the trajectories corresponding to the control text labels.

[0014] S2. Use the visual language (CLIP) tokenizer and text encoder to tokenize the text prompt and extract text features. Use the visual language tokenizer to tokenize the control text label corresponding to the trajectory and calculate its subscript in the cross-attention map list after the text prompt is tokenized. Then initialize the latent features of the stable diffusion model based on the default or user-provided random seed.

[0015] S3. Convert the trajectory to the same resolution as the controlled cross-attention map as the control area; calculate the control loss, suppression loss and repair loss for the cross-attention maps of the intermediate layer and upsampling layer of the Unet model part of the pre-trained stable diffusion model through a hierarchical guidance mechanism, and calculate the loss according to the energy function designed by hierarchical guidance and then pass the gradient back to update the potential features; the control loss aligns the layout with the trajectory at each resolution, the suppression loss suppresses objects outside the trajectory at low resolution, and the repair loss uses attention feedback to refine the area not fully controlled by the trajectory at medium and high resolutions; the denoising process based on the stable diffusion model without classification guidance denoises step by step to obtain the latest potential features.

[0016] In one possible implementation, the trajectory is converted to the same resolution as the controlled cross-attention map as the control area, specifically including: First, a binary mask map of the same size as the controlled cross-attention map is created, and the ratio between the original size of the track and the size of the cross-attention map is calculated. Then, the coordinates of the track are mapped to the size of the binary mask map according to the ratio, and the track position is marked as 1 to form a control area.

[0017] The formula for the control loss is as follows:

[0018] in, represents the value of the control loss calculation; Indicates the first List of criss-cross attention maps for the layer; is the i-th trajectory in the trajectory list; is the text label corresponding to the i-th track; Is the calculated trajectory label In the cross attention map list Functions of the index in ; represents the i-th trajectory After the coordinate transformation operation Layer Cross Attention Map Control areas with the same resolution.

[0019] The formula for the suppression loss is as follows:

[0020] in, Represents the value of the suppression loss calculation.

[0021] The formula for the repair loss is as follows:

[0022] in, represents the value of the repair loss calculation; is before each boot step by Specifically, when the corresponding position The value exceeds the threshold hour, The value of is set to 1; otherwise, it is set to 0. This embodiment studies the empirical setting is 0.4.

[0023] The formula of the energy function of the hierarchical guided design is as follows:

[0024] in, represents the value calculated by the energy function of the hierarchical guidance mechanism; , , is a scaling factor that adjusts the guidance strength; The energy function for controlling the layout of the 8×8 cross-attention map in the middle layer of U-Net (suppressing objects outside the track at low resolution through suppression loss), The energy function for controlling the shape of the 16×16 cross-attention map of the U-Net upsampling layer, The energy function for the shape control of the 32×32 cross-attention map of the same U-Net upsampling layer (using attention feedback to refine areas not fully controlled by the trajectory at medium and high resolutions through inpainting loss); The formula of the energy function for layout control is as follows:

[0025] in, represents the value calculated by the layout control energy function; L c and L s To control losses and contain losses; The energy function for shape control is given by:

[0026] in, represents the value calculated by the shape control energy function; L f To repair the losses.

[0027] In step 3, the formula for the denoising process is:

[0028] Among them, z t represents the potential representation at time step t; , n represents the number of control targets; The calculation formula is, , It is a predefined coefficient of the pre-trained stable diffusion model, which is used to control the attenuation or scaling of the noise.

[0029] S4. Obtain the controlled generated image through the decoder.

[0030] To verify the technical effect of the present invention, this embodiment uses FID to measure the quality of the generated image, and DTL to measure the fit between the generated target and the trajectory. However, when the target generation is abnormal, DTL cannot accurately evaluate whether the target is correctly generated following the trajectory. To solve this problem, this embodiment also proposes a new "IOT (Intersection OverTrajectory) metric" to check the correctness of object generation by comparing the trajectory with the generated object mask and calculating their overlap ratio. The higher the IOT, the more accurate the target generation.

[0031] This embodiment is quantitatively evaluated on COCO2014. Following previous work, this embodiment randomly extracts 1,000 images from the validation set to create simple and complex trajectory datasets, each containing 1 to 3 objects. In the simple trajectory dataset, the trajectory of each object is represented by a curve, while in the complex trajectory dataset, the trajectory of each object contains 1 to 2 branches. Since the method of the present invention emphasizes solving the problem of fine-grained object control, this embodiment constructs a unified dataset containing 500 simple trajectory examples and 1,000 complex trajectory examples, a total of 1,500 examples, named the "TRAT" dataset, for ablation research.

[0032] Combination Figure 3 , Figure 4 , Figure 8 , Fig.10 , the results of the ablation study experiments on each component in the model of the present invention are as follows:

[0033] It can be seen that compared with the original stable diffusion model (Stabel Diffusion) and the trajectory-based model (TraDiffusion), the present invention shows the strongest control level. The present invention can not only control the target layout more stably, but also achieve fine control over the target.

[0034] Combination Figure 4-Figure 6 The results of the ablation experiment on the hierarchical guidance mechanism of the present invention are as follows:

[0035] It can be seen that the best IOT and DTL performance is achieved when all modules of the hierarchical guidance mechanism of the present invention are adopted. At the same time, the FID score has increased, however, in order to achieve more precise object control, this slight quality trade-off is worth it. In addition, since the ablation dataset of the present invention only contains 1,500 images, this gap will narrow as the dataset size increases.

[0036] Combination Figure 6 , Fig. 9 The results of the ablation experiment on the hierarchical guidance mechanism of the present invention are as follows:

[0037] It can be seen that after introducing the repair loss, the present invention can effectively solve the problems of discontinuous target generation and detail loss. At the same time, both IOT and DTL are improved, which enhances the control ability of the method.

[0038] The present invention designs a hierarchical guidance mechanism to calculate the control loss, suppression loss and repair loss of the cross-attention map of the intermediate layer and the upsampling layer of the Unet model, and calculates the loss according to the energy function of the hierarchical guidance design and then passes the gradient back to update the potential features, thereby combining the control loss and the suppression loss to achieve effective layout control at low resolution, and combining the control loss and the repair loss to ensure target shape control at medium and high resolutions, solving the defects of the trajectory-based image generation model (TraDiffusion) that it cannot stably control the layout and cannot finely control the target, and does not require additional training.

[0039] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A hierarchical guided method for 2D image generation based on trajectory-based target fine control, characterized in that: include: Step 1, obtain text prompts, trajectories and control text labels corresponding to the trajectories; Step 2: Use the visual language word segmenter and text encoder to segment the text prompt and extract text features. Use the visual language word segmenter to segment the control text label corresponding to the trajectory and calculate its subscript in the cross-attention map list after the text prompt is segmented; then initialize the potential features of the stable diffusion model; Step 3: Convert the trajectory to the same resolution as the controlled cross-attention map as the control area; calculate the control loss, suppression loss and repair loss for the cross-attention map of the intermediate layer and upsampling layer of the Unet model part of the pre-trained stable diffusion model through the hierarchical guidance mechanism, and calculate the loss according to the energy function designed by the hierarchical guidance and then pass the gradient back to update the potential features; the control loss aligns the layout with the trajectory at each resolution, the suppression loss suppresses objects outside the trajectory at low resolution, and the repair loss uses attention feedback to refine the area not fully controlled by the trajectory at medium and high resolutions; The denoising process without classification guidance based on the stable diffusion model denoises step by step to obtain the latest potential features; Step 4: Obtain the controlled generated image through the decoder.

2. The method according to claim 1, characterized in that The formula for the control loss is as follows: in, represents the value of the control loss calculation; Indicates the first List of criss-cross attention maps for the layer; is the i-th trajectory in the trajectory list; is the text label corresponding to the i-th track; Is the calculated trajectory label In the cross attention map list Functions of the index in ; represents the i-th trajectory After the coordinate transformation operation Layer Cross Attention Map Control areas with the same resolution.

3. The method according to claim 2, characterized in that The formula for the suppression loss is as follows: in, Represents the value of the suppression loss calculation.

4. The method according to claim 3, characterized in that The formula for the repair loss is as follows in, represents the value of the repair loss calculation; is before each boot step by A binary mask dynamically generated from the extractor for high response regions.

5. The method according to claim 1, characterized in that The formula of the energy function of the hierarchical guided design is as follows: in, represents the value calculated by the energy function of the hierarchical guidance mechanism; , , is a scaling factor that adjusts the guidance strength; The energy function for controlling the layout of the 8×8 cross-attention graph in the middle layer of U-Net, The energy function for controlling the shape of the 16×16 cross-attention map of the U-Net upsampling layer, The energy function for shape control of the 32×32 cross-attention map of the U-Net upsampling layer; The formula of the energy function for layout control is as follows: in, represents the value calculated by the layout control energy function; L c and L s To control losses and contain losses; The energy function for shape control is given by: in, represents the value calculated by the shape control energy function; L f To repair the losses.

6. The method according to claim 5, characterized in that: In step 3, the formula for the denoising process is: Among them, z t represents the potential representation at time step t; , n represents the number of control targets; The calculation formula is, , It is a predefined coefficient of the pre-trained stable diffusion model, which is used to control the attenuation or scaling of the noise.

7. The method according to claim 1, characterized in that: In step 3, the trajectory is converted to the same resolution as the controlled cross-attention map as the control area, specifically including: Create a binary mask map of the same size as the controlled cross-attention map, calculate the ratio between the original size of the track and the size of the cross-attention map, and then map the coordinates of the track to the size of the binary mask map according to the ratio, and mark the track position as 1 to form a control area.

Images