Trajectory-based training-free image generation method, electronic equipment and storage medium

Abstract

The invention discloses a trajectory-based training-free image generation method, an electronic device and a storage medium, which realize simple, user-friendly and controllable image generation, and comprises the following steps: obtaining a trajectory drawn by a user and a control text tag corresponding to a trajectory line segment, and obtaining a text prompt; calculating a distance matrix for each track line segment according to the control text label, performing word segmentation on the text prompt, extracting text features, performing word segmentation on the control text label of the track, and calculating a subscript of the control text label in a list after word segmentation of the text prompt; initializing potential features of the stable diffusion model; according to subscripts of the control text labels, control loss and movement loss of each control text label are calculated for cross attention of a specific layer of a Unet model part of a pre-trained stable diffusion model and are subjected to gradient return, and then potential features are updated based on a denoising process of unclassified guidance of the stable diffusion model; and repeating until denoising is completed, and finally decoding the updated potential features to obtain a generated image.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a trajectory-based training-free image generation method, electronic equipment and storage medium. Background Art

[0002] Image generation is a key task in computer vision and artificial intelligence, and has made remarkable progress in the past few years. In particular, some models in the existing technology can take text descriptions as input and generate high-quality images consistent with the text descriptions, showing impressive text-to-image performance.

[0003] In the task of text-generated images, text-based control is beneficial, but often lacks the precision and intuitive operation required for fine-grained adjustments in the generated images. Therefore, people are exploring the introduction of conditions such as edges, grayscale images, and semantic layouts in conditional processing methods to provide more subtle control over the generated images. These diverse conditional techniques broaden the application scope of generative models from design tasks to areas such as data generation. However, traditional conditional processing methods with conditions such as edges, grayscale images, and semantic layouts usually require the use of specialized tools, which makes them less friendly to users without expertise in this field; at the same time, the semantic masks in traditional conditional processing methods impose artificial priors on the structure of the image pairs, which will lead to oversaturated control.

[0004] Therefore, how to provide a simple and user-friendly controllable generation method has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to provide a trajectory-based training-free image generation method, electronic device and storage medium to solve the problems existing in the prior art and achieve simple, user-friendly and controllable image generation.

[0006] In order to achieve the above object, one of the solutions of the present invention is:

[0007] A trajectory-based training-free image generation method comprises the following steps:

[0008] Step 1: Obtain the control text labels corresponding to the trajectory drawn by the user and its trajectory segments, and obtain the text prompts;

[0009] Step 2: Calculate the distance matrix for each trajectory segment according to the control text label; use the visual language word segmenter and text encoder to segment the text prompt and extract text features; use the visual language word segmenter to segment the control text label of the trajectory and calculate its subscript in the list after the text prompt is segmented; initialize the potential features of the stable diffusion model;

[0010] Step 3: According to the subscript of the control text label calculated in step 2, the cross attention of the Mid and Decoder layers of the Unet model part of the pre-trained stable diffusion model is used to calculate the control loss and movement loss of each control text label and return the gradient, and then the potential features are updated based on the denoising process without classification guidance of the stable diffusion model;

[0011] Step 4: Repeat step 3 until the denoising process is completed, and finally decode the updated latent features to obtain the generated image.

[0012] In step 1, the front-end page is designed by using the python library Gradio, and an interface for users to draw trajectories and input text prompts is provided.

[0013] In the step 2, the distance matrix is ​​calculated by using the distanceTransform function of OpenCV.

[0014] In step 3, the calculation formula of the control loss is:

[0015]

[0016] Among them, E c (A (τ) , l i , w i ) represents loss control; A (τ) represents the attention map; l i represents the trajectory corresponding to the i-th control text label; w i Represents the i-th control text label; D μi Represents the i-th control text label w i The corresponding μth position of the distance matrix; ∈ represents an adjustable parameter used to avoid division by zero; Represents the association strength between the μth position and the i-th control text label in the τ-th layer attention map;

[0017] The calculation formula for moving loss is:

[0018]

[0019] Among them, E m (A (τ) , l i , w i ) represents the movement loss.

[0020] Preferably, in step 3, the calculation formula of the denoising process is:

[0021]

[0022] E(A (τ) , li , w i )=E c (A (τ) , l i , w i )+λ·E m (A (τ) , l i , w i );

[0023] Among them, z t+1 、z t Represent the potential features at time t+1 and time t respectively; σ t The calculation formula is α t represents a parameter that changes with time and is used to control the degree of noise addition; η represents a hyperparameter that controls the guidance strength and its value is > 0; represents a set of layers in Unet; n represents the number of control text labels; E(A (τ) , l i , w i ) represents the final distance-aware energy function; λ represents the hyperparameter controlling the motion loss strength coefficient.

[0024] The second solution of the present invention is:

[0025] An electronic device comprises a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the trajectory-based training-free image generation method.

[0026] The third solution of the present invention is:

[0027] A computer-readable storage medium stores a computer program; when the computer program is executed in the computer, the computer executes the trajectory-based training-free image generation method.

[0028] After adopting the above technical solution, the present invention has the following technical effects:

[0029] (1) It provides a promising solution for layout-controlled image generation, effectively addressing the limitations of mask-conditional and box-conditional techniques;

[0030] (2) Through qualitative and quantitative evaluations, it can be demonstrated that the present invention has better control ability than previous methods and has achieved significant improvements in the quality and accuracy of generated images;

[0031] (3) The present invention is adaptable to any type of trajectory input, allowing precise control over object attributes, relationships, and salient regions.

[0032] (4) The effectiveness of the proposed method is verified through qualitative and quantitative experiments, demonstrating enhanced control over target attributes, relationships, and visual input, thereby creating more natural and realistic images that are more consistent with trajectories of different shapes. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the present invention;

[0034] Figure 2 is a schematic diagram of the present invention;

[0035] Figure 3 It is an example diagram of the present invention generating an image based on the guidance of the prior structure;

[0036] Figure 4 It is a qualitative comparison diagram of different methods of the present invention and the prior art;

[0037] Figure 5 is a qualitative analysis diagram of the different components of the present invention;

[0038] Figure 6 It is a qualitative analysis diagram of the present invention when different intensity coefficients are set for movement loss;

[0039] Figure 7 is an example diagram of the present invention using trajectory control to control the target salient area;

[0040] Figure 8 is an example diagram generated by the present invention using an arbitrary trajectory control object;

[0041] Fig. 9 It is an example diagram of the control target attributes and relationships implemented by the present invention;

[0042] Fig.10 It is an example diagram of implementing control of visual input according to the present invention. DETAILED DESCRIPTION

[0043] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0044] The overall idea of ​​the present invention is as follows: a trajectory-based 2D controllable image generation method is proposed to control image generation in a simple and user-friendly way; firstly, a pre-trained text graph model (Stable Diffusion, stable diffusion model) is introduced, and in addition to the original required text conditions, a distance matrix list and a subscript list are obtained according to the trajectory provided by the user and its corresponding controlled text label; then, according to the control loss and movement loss designed by the present invention, the model loss is calculated and the gradient is back-transferred to update the potential features, and the denoising process based on the stable diffusion model without classification guidance is denoised step by step to obtain the latest potential features; finally, the controlled generated image is obtained through a decoder.

[0045] refer to Figure 1-2 As shown, the present invention discloses a trajectory-based training-free image generation method, comprising the following steps:

[0046] Step 1: Design the front-end page through the Python library Gradio; the user draws a trajectory on the front-end page and enters a text prompt for image generation; obtain the control text label corresponding to the trajectory drawn by the user and its trajectory segment, and obtain the text prompt, and return it to the back-end Python code; the trajectory drawn by the user usually includes different trajectory segments, and each trajectory segment has a corresponding control text label;

[0047] Step 2: Calculate the distance matrix for each trajectory segment according to the control text label; use the visual language (CLIP) word segmenter and text encoder to segment the text prompt and extract text features; use the visual language (CLIP) word segmenter to segment the control text label of the trajectory and calculate its subscript in the list after the text prompt is segmented; initialize the potential features of the stable diffusion model;

[0048] Step 3: According to the subscripts of the control text labels calculated in step 2, the cross attention (Cross Attention) of the Mid and Decoder layers of the Unet model part of the pre-trained stable diffusion model is used to calculate the control loss and movement loss of each control text label and return the gradient, and then the potential features are updated based on the denoising process without classification guidance of the stable diffusion model;

[0049] Step 4: Repeat step 3 until the denoising process is completed, and finally decode the updated latent features to obtain the generated image.

[0050] Specific embodiments of the present invention are shown below.

[0051] In step 2 above, the distance matrix is ​​calculated by the distanceTransform function of OpenCV.

[0052] In step 3 above, the calculation formula for control loss is:

[0053]

[0054] Among them, E c (A (τ) , l i , w i ) represents loss control; A (τ) represents the attention map; l i represents the trajectory corresponding to the i-th control text label; w i Represents the i-th control text label; D μi Represents the i-th control text label w i The μth position of the corresponding distance matrix; ∈ represents an adjustable parameter used to avoid division by zero, and its value is very small; Represents the association strength between the μth position and the i-th control text label in the τ-th layer attention map;

[0055] At the same time, the calculation formula for moving loss is:

[0056]

[0057] Among them, E m (A (τ) , l i , w i ) represents the movement loss.

[0058] Furthermore, in the above step 3, the calculation formula of the denoising process is:

[0059]

[0060] E(A (τ) , l i , w i )=E c (A (τ) , l i , w i )+λ·E m (A (τ) , l i , w i );

[0061] Among them, z t+1 、z t Represent the potential features at time t+1 and time t respectively; σ t The calculation formula is α t represents a parameter that changes with time and is used to control the degree of noise addition; η represents a hyperparameter that controls the guidance strength, and its value is > 0; represents a set of layers in Unet; n represents the number of control text labels; E(A(τ) , l i , w i ) represents the final distance-aware energy function; λ represents the hyperparameter that controls the motion loss strength coefficient. The distance-aware energy function proposed in this way and used to update the latent variables can guide the target to produce higher responses in areas closer to the trajectory drawn by the user without incurring additional training costs.

[0062] Through the above scheme, the present invention has the following advantages:

[0063] (1) It provides a promising solution for layout-controlled image generation, effectively addressing the limitations of mask-conditional and box-conditional techniques;

[0064] (2) Through qualitative and quantitative evaluations, it can be demonstrated that the present invention has better control ability than previous methods and has achieved significant improvements in the quality and accuracy of generated images;

[0065] (3) The present invention is adaptable to any type of trajectory input, allowing precise control over object attributes, relationships, and salient regions.

[0066] (4) The effectiveness of the present invention is underlined through qualitative and quantitative experiments, demonstrating the enhanced control over target attributes, relationships, and visual input, thereby creating more natural and realistic images that can better conform to trajectories of different shapes.

[0067] In addition, the present invention also discloses an electronic device, including a processor, a memory and an application program; the application program is stored in the memory and is configured to execute the above method by the processor.

[0068] The present invention also discloses a computer-readable storage medium, which stores a computer program; when the computer program is executed in a computer, the computer executes the above method. The storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0069] In order to verify the technical effect of the present invention, the quality of the generated image is measured using FID in this embodiment. Since the traditional metric is not suitable for evaluating the layout control of the trajectory-based image generation method, the present invention also proposes a new "distance to line (DTLZ) metric", which can be expressed as:

[0070]

[0071] Where, mask represents the mask, which is obtained by applying YOLOv8m-Seg to the generated image; e represents the base of the natural logarithm. The larger the DTL, the closer the generated image is to the trajectory drawn by the user. Therefore, DTL can not only verify the quality of the generated image, but also check the alignment of the layout.

[0072] We perform quantitative evaluation on COCO2014. Following previous work, we randomly sample 1000 images from the validation set, each with a caption containing at most 3 instances with a mask occupying more than 5%. We will sample instances that exist in the prompts in this paper instead of filling in the captions. See Figure 3 , 7 -10, experimental results show that: without the need for additional training or fine-tuning, the present invention can optimize the potential features of the stable diffusion model in combination with the proposed distance-aware energy function, and achieve user-friendly layout control. Moreover, the present invention can generate images more simply and naturally. In addition, the present invention is adaptable to arbitrary trajectory inputs, allowing precise control of object attributes, relationships, and salient areas.

[0073] The present invention conducts evaluation and a series of quantitative experiments on the benchmark dataset HICO-DET for human interaction detection and the open scene dataset Visual Genome.

[0074] Combination Figure 4-5 , the results of the ablation study experiments on each component in the model of the present invention are as follows:

[0075] method DTL(↑) FID(↓) Stabel Diffusion 0.0043 68.33 Prior scructure 0.0077 66.91 Expand trajectory 0.0080 64.87 The present invention (excluding movement loss) 0.0119 64.68 The present invention (complete) 0.0156 68.53

[0076] It can be seen that compared with the original stable diffusion model, the guidance method based on prior structure and the ExpandTrajectory method, the present invention shows the strongest control level, and the DTL score is twice that of the two baselines.

[0077] Combination Figure 6 ,The results of the ablation study experiment to test the effect of setting different intensity coefficients on the motion loss are as follows:

[0078] λ 0 1 5 10 20 100 DTL(↑) 0.0119 0.0124 0.0137 0.0156 0.0158 0.0096 FID(↓) 64.68 65.46 66.39 68.53 72.80 129.91

[0079] It can be seen that when the strength coefficient λ is 10, the best performance can be achieved.

[0080] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A trajectory-based training-free image generation method, characterized in that The following steps are involved: Step 1: Obtain the control text labels corresponding to the trajectory drawn by the user and its trajectory segments, and obtain the text prompts; Step 2: Calculate the distance matrix for each trajectory segment according to the control text label; use the visual language word segmenter and text encoder to segment the text prompt and extract text features; use the visual language word segmenter to segment the control text label of the trajectory and calculate its subscript in the list after the text prompt is segmented; initialize the potential features of the stable diffusion model; Step 3: According to the subscript of the control text label calculated in step 2, the cross attention of the Mid and Decoder layers of the Unet model part of the pre-trained stable diffusion model is used to calculate the control loss and movement loss of each control text label and return the gradient, and then the potential features are updated based on the denoising process without classification guidance of the stable diffusion model; Step 4: Repeat step 3 until the denoising process is completed, and finally decode the updated latent features to obtain the generated image.

2. The trajectory-based training-free image generation method according to claim 1, characterized in that: In step 1, the front-end page is designed by using the python library Gradio, and an interface for users to draw trajectories and input text prompts is provided.

3. The trajectory-based training-free image generation method according to claim 1, characterized in that: In the step 2, the distance matrix is ​​calculated by using the distanceTransform function of OpenCV.

4. The trajectory-based training-free image generation method according to claim 1, characterized in that: In step 3, the calculation formula of the control loss is: Among them, E c (A (τ) ,l i ,w i ) represents loss control; A (τ) represents the attention map; l i represents the trajectory corresponding to the i-th control text label; w i Represents the i-th control text label; D μi Represents the i-th control text label w i The corresponding μth position of the distance matrix; ∈ represents an adjustable parameter used to avoid division by zero; Represents the association strength between the μth position and the i-th control text label in the τ-th layer attention map; The calculation formula for moving loss is: Among them, E m (A (τ) ,l i ,w i ) represents the movement loss.

5. The trajectory-based training-free image generation method according to claim 4, characterized in that: In step 3, the calculation formula of the denoising process is: E(A (τ) ,l i ,w i )mE c (AM (τ) ,l i ,w i )+λ·E m (AM (τ) ,l i ,w i )4 Among them, z t+1 、z t Represent the potential features at time t+1 and time t respectively; σ t The calculation formula is α t represents a parameter that changes with time and is used to control the degree of noise addition; η represents a hyperparameter that controls the guidance strength, and its value is > 0; represents a set of layers in Unet; n represents the number of control text labels; E(A (τD ,l i ,w i ) represents the final distance-aware energy function; λ represents the hyperparameter controlling the motion loss strength coefficient.

6. An electronic device, characterized in that The invention comprises a processor, a memory and an application program; the application program is stored in the memory and is configured so that the processor executes the trajectory-based training-free image generation method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that The storage medium stores a computer program; when the computer program is executed in the computer, the computer is caused to execute the trajectory-based training-free image generation method according to any one of claims 1 to 5.

Images