Method for generating picture from special text based on step-by-step self-feedback

Abstract

The invention discloses a method for generating a picture by a special text based on step-by-step self-feedback, which comprises the following steps of: determining a spatial layout condition of each object in a scene based on a user text content; when a group of bounding boxes are generated in each round, the current result is fed back and output, and a final overall layout diagram is generated; generating a corresponding mask latent variable for each object by referring to the obtained final overall layout including the necessary information of each object; generating a potential image corresponding to each object, and combining all the potential images with the background image to generate an intermediate image; the content of each object bounding box in the intermediate image is evaluated one by one, multi-round iterative refinement is carried out on the content in the bounding box which does not meet the condition in combination with the output of the detail feedback module, and the content is fused into the original image until the content in each bounding box meets the condition; the method has important significance for improving the effect of an image generation task in an automatic system.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and layout-conditioned image generation, and particularly to a method for generating pictures from special texts based on step-by-step self-feedback. Background Art

[0002] The latest diffusion models have made remarkable progress in text-to-image generation and become powerful tools for creating diverse images and graphics from pure text prompts. The success of these models is attributed to the large-scale availability of multimodal datasets, the growth of computing resources, and the expansion of model parameters.

[0003] However, when the description length increases or the number of objects in the scene increases, modern diffusion models tend to ignore some of the prompt content, resulting in the omission of key details, misrepresentation, or the generated objects not matching the details in the prompt, which limits their ability to comprehensively process long prompts.

[0004] The reasons for the limitations of the above-mentioned diffusion models basically come from the following aspects:

[0005] 1) Insufficient ability to handle multiple objects and special scenes: Although diffusion models are good at generating high-quality images from short, single-object descriptions, as the description length increases and the number of objects in the scene increases, modern diffusion models tend to omit some of the prompt content, resulting in the loss of important information or inaccurate object representation. Especially when multiple spatially separated object instances need to be placed, the model may not be able to capture the specific positions and attributes of all objects.

[0006] 2) Challenges in cross-modal information fusion: To make the generated images consistent with the text prompts, diffusion models need to effectively map text information to visual features. However, this cross-modal information fusion is not always successful, especially when dealing with tasks involving special logics such as numbers and spatial relationships, the model may not be able to correctly understand and apply this information.

[0007] 3) Lack of an iterative optimization mechanism: In the process of generating the entire image at once, traditional diffusion models may not be able to precisely meet special text requirements. For example, in the initially generated global scene, the characteristics of some objects may not faithfully reflect the regulations in the prompt. Without an effective feedback and correction mechanism, it is difficult for the model to improve the initial result to ensure that the final output fully meets the expectations.

[0008] 4) Limitations of pure objective data as evaluation metrics: In an automated system, the existing text-to-image models usually use machine evaluation metrics such as IS and FID to evaluate the generated images. To some extent, these metrics ensure the quality of the generated images. However, the abstract layout information is more often a subjective imagination of humans, and the subjective imagination of humans about images cannot be simply represented by machine evaluation metrics. Therefore, in many cases, images with high machine evaluation metrics may not satisfy humans.

[0009] Therefore, the present invention attempts to specifically take into account the importance of human subjective evaluation. By simulating the way of human perceptual evaluation, the artistic quality and visual effect of the generated images are further improved. Since the end-users are humans, their preferences and feelings determine the success of the generated content. Considering human subjective evaluation can ensure that the generated works can better meet the expectations and needs of users. By simulating the way of human perceptual evaluation, the model can more accurately capture the elements and styles that people consider beautiful, thus generating more artistic and visually appealing images. Modern AI technology is not just a game of algorithms and data; it also needs to understand various humanistic factors such as human culture and aesthetics. Incorporating human subjective evaluation into consideration helps to promote the deep integration of technology and humanities. When image generation models can better understand and imitate human aesthetic standards, their application scope will also expand accordingly. Many fields such as digital art creation, film and television production, and advertising design can benefit from this. Therefore, it is necessary to specifically consider human subjective evaluation in text-to-image models, which is of great significance for improving the overall performance and applicability of the models. Summary of the Invention

[0010] In order to break through the limitations in the field of layout-conditioned image generation described above, the present invention provides a generation method that can split the text-image generation task into multiple steps. The technical solution of the present invention is as follows:

[0011] S1: Based on the text content containing special layout information input by the user, ask the LLM (Large Language Model) according to a preset template. Use the LLM to split and parse the text information with each single object as a unit, and determine the spatial layout of each object in the scene by combining the context and the abstract thinking ability of the LLM itself, and then generate multiple bounding boxes containing necessary information.

[0012] S2: Execute step S1 multiple times, and when generating a set of bounding boxes each time, call the Agent macro-layout feedback module to give feedback on the current result, and use linear interpolation to combine the feedback results of multiple iterations to generate the final overall layout diagram.

[0013] S3: Refer to the obtained final overall layout diagram containing the necessary information of each object, and generate the corresponding masked latent variable N for each object according to a preset algorithm.i ;

[0014] S4: Combine the mask latent variables N of each object i and text description i Generate a latent image corresponding to each object, and then combine all latent images with the background image to generate an intermediate image X excessive ;

[0015] S5: Evaluate the intermediate images X one by one according to the preset rules excessive The content of each object bounding box in the image is refined for multiple rounds, and the content after iterative refinement is fused into the original image until the content of each bounding box meets the conditions, thereby obtaining the final generated image;

[0016] Furthermore, the specific process of generating a bounding box in step S1 is as follows: input the text provided by the user into the LLM, ask questions according to the preset template, let the LLM split and parse the text information into single objects, and combine the context with LLM's own abstract thinking ability to determine the spatial layout of each object in the scene, and then generate multiple groups of bounding boxes with coordinate information such as (x, y, w, h) and brief text descriptions, where (x, y) refers to the coordinates of the upper left corner of the bounding box, and (w, h) refers to the width and height information of the bounding box.

[0017] Furthermore, the Agent macro layout feedback module in step S2 is an LLM that can evaluate the confidence of the layout diagram in combination with the text description. The LLM can be adapted to downstream tasks by fine-tuning, specific question templates, etc.

[0018] Furthermore, in step S2, a linear interpolation method is used to combine the feedback results of multiple iterations to generate the final overall layout diagram. The specific steps are as follows:

[0019] Assume that each set of layout frame information in the example contains 12 objects, the number of iterations is k, and the layout diagram obtained in each iteration is evaluated for confidence via the feedback module, and the confidence μ of the result of this round of iteration is output. j , the layout frame combination obtained in the jth iteration is Z j =[(x1,y1,w1,h1)(x2,y2,w2,h2)……(x i ,y i ,w i ,h i )……(x 12 ,y 12 ,w 12 ,h 12)], where j∈{1,2,3…k}, i∈{1,2,3…12}. The interpolation calculation process is recursive, and the specific calculation formula is as follows:

[0020]

[0021] in is the interpolation result calculated after the jth iteration. The number of iterations k is determined by the specific iteration result and the threshold ε (0<ε<1). When the current iteration result satisfies:

[0022]

[0023] When , the system stops iterating, k = j, and takes is the final interpolation result. From the above formula, we can see that when the value of ε is larger, the conditions for the system to end the iteration are easier to meet, the number of iterations is smaller, the interpolation result when the iteration is stopped is more discrete, the calculation process consumes fewer resources and the final effect is worse; when the value of ε is smaller, the conditions for the system to end the iteration are more difficult to meet, the number of iterations is larger, the interpolation result when the iteration is stopped is more convergent and tends to be stable, the calculation process consumes more resources and the final result is better. The specific value of ε can be determined according to the specific situation of the device. The above steps determine This is the final overall layout of the generated image.

[0024] Further, in step S3, the mask latent variable N i The definition is as follows: refers to the initial form of each object in the image, which is a low-resolution latent representation that contains the basic shape and position information of the object. In this way, the model can gradually refine the details of these objects in the subsequent generation process.

[0025] Further, the preset algorithm steps in step S3 are as follows:

[0026] 3-1) For each object i, first calculate its bounding box coordinates (x i ,y i ,w i ,h i ) Create a binary mask m i The mask is a matrix of the same size as the image, where:

[0027] In the area within the bounding box, the mask value is 1 (indicating that the area needs to be modified or paid attention to).

[0028] In areas outside the bounding box, the mask value is 0 (indicating that the area remains unchanged).

[0029] For example, suppose the image size is 512x512 and the bounding box (xi , y i , w i , h i If the coordinates of () are (100, 150, 200, 100), then the mask m i will be a 512x512 matrix with a value of 1 in the region from (100, 150) to (300, 250) and a value of 0 in the remaining regions.

[0030] The role of this binary mask is to ensure that the generated latent variables only affect the region where the object is located, without affecting the background or other objects.

[0031] 3 - 2) Generate a random initial noise latent variable N T , which is shared for all boxes during the subsequent denoising process to ensure a globally consistent perspective, style, and lighting.

[0032] During this process, a specific text prompt is used to generate the cross - attention map A corresponding to each box i , in the format of "[background prompt] with [box caption]", for example, "The gate of an ancient Chinese building". Use a pre - trained text encoder (such as CLIP) to encode the prompt, further map the text prompt into a key vector k and a value vector v, and for the query vector q at each spatial position u u and the key vector k of each text token v, calculate the similarity score between q u and k v . Usually, the dot product is used to measure the similarity, and the scores are normalized into a probability distribution through the Softmax function to obtain the cross - attention map A. The specific formula is as follows:

[0033]

[0034] Here, A uv represents the degree of correlation from the feature position u to the text token v, and dividing by is for stabilizing the gradient. Use the cross - attention map A uv to calculate the weighted combination and fuse the text feature v into the image feature. The specific formula is:

[0035] A = ∑ v∈V A uv ·v (4)

[0036] V is the set of token indices corresponding to the box caption. At this time, the cross - attention map A of the i - th box is obtained i .

[0037] 3 - 3) For each object, through the denoising process from to Generate latent variables containing a single instance, and use cross-attention maps to ensure that the object is placed within the specified bounding box. Specifically, strengthen the cross-attention from the pixels inside the box to the text tokens related to the box caption, while weakening the cross-attention from the pixels outside the box. Define a simple energy function E(A i , i, v) to continuously update the latent variables to minimize it, and the specific formula is as follows:

[0038]

[0039] Continuously execute this denoising process to obtain Finally obtain Finally, use the binary mask m of each object i to perform an element-wise multiplication on the latent variables to obtain the final masked latent variable N corresponding to the object i , that is:

[0040]

[0041] Since then, the masked latent variable N corresponding to each object is obtained i .

[0042] Furthermore, the S4 obtains the generated intermediate image X excessive The specific steps are as follows:

[0043] 4-1) To generate the latent image of each object, we need to use an image synthesis model M comp (such as LayoutDiffusion), and input the masked latent variable N of each object i , the mask m i , and the text description d i into the image synthesis model M comp to generate the latent image of each object The specific formula is as follows:

[0044]

[0045] 4-2) Use the same image synthesis model M comp , and generate the background image X at the corresponding position of each object according to the background description d of the text content b , and the specific generation formula is as follows: backgroud

[0046]

[0047] 4-3) Furthermore, for each object, use its corresponding masked latent variable to process its latent image Embed it into the corresponding background image to obtain the intermediate image X excessive , and the specific formula is as follows:

[0048]

[0049] This step ensures that the potential image of each object is correctly placed in the corresponding position of the background image, forming a complete image.

[0050] After the above steps, the generated intermediate image X excessive already contains the foreground parts of all objects and the background image. This image is generated based on the global scene layout, which has ensured macroscopically that each object is placed within its corresponding box. Although for the objects within each box, there may be some inaccuracies in certain details. Therefore, in the subsequent steps, each box needs to be iteratively optimized one by one to further ensure the alignment degree between the image and the text information at the detailed level.

[0051] Furthermore, the step-by-step iterative refinement in S5 is as follows:

[0052] 5-1) Evaluation and detection: For each bounding box o excessive in the generated intermediate image X i =(x i , y i , w i , h i ), and its corresponding text description d i , we first extract the image input of the bounding box o i =(x i , y i , w i , h i ) into the Agent detailed layout feedback module for details. In addition to directly calculating the similarity between the object and its text description using the CLIP model to obtain the mechanical similarity evaluation, we also choose to call the detailed layout feedback module of the external built-in LLM to first generate a new text description for the object to be evaluated, and then use CLIP to calculate the text similarity between the new text description and the original text description, aiming to imitate the perceptual evaluation of the alignment degree between the object and the text description by humans, and obtain the final score by synthesizing the two evaluations. According to the final score, we decide whether to re-perform the synthesis and evaluation process. Use the trained LLM in the feedback module to generate the text description d' i for this part of the image, and then obtain the similarity score between the content in the bounding box and the text description according to the following formula:

[0053]

[0054] where is the intermediate image X excessiveThe image content corresponding to the i-th box, where α is a hyperparameter used to measure the weight of the two losses, and CLIP(·,·) is a pre-trained multimodal model that takes an image and text as input and outputs a similarity score. If this score is below a set threshold, it indicates that the visual features of the current object do not match the text description and need to be adjusted again.

[0055] 5-2) Synthesis and guidance: For the bounding boxes whose similarity scores do not meet the requirements, in order to correct the objects therein, the text description d of the bounding box i is passed to a text-to-image model based on the Stable Diffusion model to generate a new object image as the reference object X ref_i =StableDiffusion(d i ). Subsequently, an external multimodal guidance loss L CLIP is used as the loss function for guidance, which consists of two parts:

[0056] Text consistency loss: Measures the cosine similarity loss between the content to be replaced and the text description.

[0057] Image consistency loss: Measures the cosine similarity loss between the synthesized object and the reference image. The specific formula is as follows:

[0058]

[0059] Furthermore, an image-to-image translation step can be performed on the initially generated image using the Stable Diffusion model for denoising while preserving semantic information. Starting from , noise is gradually added to generate a series of noisy latent variables X t , and the specific formula is as follows:

[0060]

[0061] Then, using the reverse diffusion process, the noise is gradually removed to restore the clean image. Specifically, at each time step t, the noise ∈ is updated and the new latent variable X t-1 is calculated:

[0062]

[0063] After multiple time steps of reverse diffusion, the final denoised image X0 is decoded, where is the gradient of the multimodal guidance loss L CLIP , and this gradient is used to update the latent variable X t , thereby gradually improving the generated image to obtain the image to be evaluated Restart the evaluation and detection steps in 4.1. If at this time the score(s) is satisfied to be greater than the preset threshold, then at this time For the final image generated in the i-th bounding box, start the "box-level iterative refinement" for the next bounding box. If the condition is still not satisfied, repeat the above steps until the condition is met.

[0064] 5-3) Gradually iterate and refine each bounding box step by step according to the preset rules in the above steps to obtain the global final image. It should be noted that the iterative optimization order of the bounding boxes should meet the following: when there is an overlapping or intersecting phenomenon in the bounding box area, the larger bounding box should be processed first, and the smaller bounding box should be processed later. At this time, for the small-area bounding box, there is a certain background image as a reference for optimization, combined with the context objects in the same area as a reference, considering as much information as possible, effectively avoiding the occlusion between objects, and the method of gradually generating the overall picture by object effectively combines the abstract spatial layout relationship between objects in the text information, so that the finally obtained picture is also aligned microscopically with the layout information in the text content to the greatest extent.

[0065] Advantages of the present invention compared with the prior art: The present invention starts from two parts of "macro layout generation" and "micro individual optimization" successively, avoiding the model from processing a large amount of information at the same time. At the same time, an intelligent Agent feedback module is connected in multiple links, which replaces humans to evaluate and feedback on the process to a certain extent, and combines multi-modal evaluation indexes to jointly guide the generation of high-quality pictures; at the same time, a scientific iterative optimization mechanism is designed, making it difficult for the model to improve the initial result to ensure that the final output fully meets the expectations. Brief Description of the Drawings

[0066] Figure 1 It is the overall framework diagram of the method for generating pictures from special texts based on step-by-step self-feedback;

[0067] Figure 2 It is an example diagram of the output image of the text-to-image model with the example text input of "Tongyi Wanxiang AI" for this method;

[0068] Figure 3 It is an example diagram of the obtained segmented text layout information using this method for this example text;

[0069] Figure 4 It is an example diagram of the layout block diagram drawn according to the text layout information;

[0070] Figure 5 It is the flow chart of the Agent macro layout feedback module participating in generating the final layout block diagram for this example;

[0071] Figure 6 It is for the Agent detailed layout feedback module to participate in the intermediate image Xexcessive Flowchart of iteratively optimizing to obtain the global final image. Detailed implementation manners

[0072] In order to make the objectives, technical solutions and advantages of this application clearer, the following further elaborates on this application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely used to explain this application and are not used to limit this application.

[0073] As Figure 1 shown, this method is mainly divided into the following steps:

[0074] S1: Based on the text content containing special layout information input by the user, ask the LLM according to a preset template, use the LLM to split and parse the text information in units of single objects, and determine the spatial layout of each object in the scene in combination with the context and the LLM's own abstract thinking ability, and then generate multiple bounding boxes containing necessary information;

[0075] S2: Execute S1 multiple times, and each time a set of bounding boxes is generated, call the Agent macro-layout feedback module to give feedback on the current result, and use linear interpolation to combine the feedback results of multiple iterations to generate the final overall layout diagram

[0076] S3: Refer to the obtained final overall layout diagram containing the necessary information of each object, and generate the corresponding masked latent variable N for each object according to a preset algorithm i ;

[0077] S4: Combine the masked latent variable N i of each object and the text description d i to generate the potential image corresponding to each object, and then combine all the potential images with the background image to generate the intermediate image X excessive ;

[0078] S5: Evaluate the content of each object bounding box in the intermediate image X excessive one by one according to preset rules, refine the content in the bounding boxes that do not meet the conditions through multiple rounds of iteration, and fuse the iteratively refined content into the original image until the content in each bounding box meets the conditions, and thus obtain the finally generated image.

[0079] The text content containing special layout information selected in this embodiment is as follows: "The picture shows an ancient Chinese building. In the center is the main gate, through which the trees in the courtyard can be seen. There are several dogs on the left side of the gate, and there are pedestrians coming and going on the right side. There are stone lions on both sides of the gate. On the head of the stone lion on the left side, there is an apple, while the stone lion on the right side is holding a firecracker in its mouth. On the second-floor terrace of the building, two people are playing chess. The one on the left is a man wearing a red top and green trousers, and the one on the right is a woman. And on the rooftop of the building, there is an assassin. Interestingly, there is a bird on his head."

[0080] Input the above text content into the Tongyi Wanxiang AI model, and the output result is as Figure 2 shown. It can be seen that only part of the text content is included in the picture. The "assassin" and "firecracker" mentioned in the text do not appear, and the "apple", "chess players", and "bird" do not generate according to the layout positions described in the text. The fundamental reason is that the existing text-to-image models still have limited semantic understanding of text content. Special text layouts are often closely related to specific content structures, rather than just simple visual descriptions. This requires the model to not only identify the key concepts in the text but also understand their meanings in the context and how to correctly map this information to the visual layout.

[0081] According to the steps of the present invention, input the text provided by the user into the LLM, ask questions according to the preset template, let the LLM split and analyze the text information with each object as a unit, determine the spatial layout of each object in the scene in combination with the context and the LLM's own abstract thinking ability, and then obtain multiple groups of bounding boxes with coordinate information such as (x, y, w, h) and brief text descriptions, where (x, y) refers to the coordinates of the upper left corner of the bounding box, and (w, h) refers to the width and height information of the bounding box. Obtain Figure 3 the example diagram of the text layout information after segmentation shown. Then, according to this layout information, draw Figure 4 the example diagram of the layout block diagram shown.

[0082] Subsequently, use the linear interpolation method to combine the feedback results of multiple iterations to generate the final overall layout diagram The specific steps are as follows:

[0083] Assume that each group of layout box information in the instance contains 12 objects, the number of iterations is k times, and the layout diagram obtained in each iteration is evaluated for confidence by the feedback module, and the confidence μ of the result of this round of iteration is output j , and the combination of layout boxes obtained in the j-th iteration is Z j = [(x1,y1,w1,h1)(x2,y2,w2,h2)……(x i ,y i ,w i ,hi )……(x 12 ,y 12 ,w 12 ,h 12 )] represents, where j ∈ {1, 2, 3... k} and i ∈ {1, 2, 3... 12}. The calculation process of interpolation is recursive, and its specific calculation formula is as follows:

[0084]

[0085] where is the interpolation result calculated after the j-th iteration. The number of iterations k is determined by the specific iteration result and the threshold ε (0 < ε < 1). When the current iteration result satisfies:

[0086]

[0087] the system stops iterating, k = j, and take as the final interpolation result. From the above formula, it can be seen that the larger the value of ε, the easier it is for the system to meet the end iteration condition, the fewer the number of iterations, the more discrete the interpolation result when stopping iteration, and the less resources are consumed in the calculation process but the final effect is worse; the smaller the value of ε, the more difficult it is for the system to meet the end iteration condition, the more iterations, the more convergent and stable the interpolation result when stopping iteration, and the more resources are consumed in the calculation process but the final result is better. The specific value of ε can be determined according to the specific situation of the device. The above steps determine as the final overall layout diagram of the generated picture, and the specific process is as Figure 5 shown.

[0088] According to the above steps and algorithms, the mask latent variable N i corresponding to each object and the intermediate image X excessive are gradually obtained, and the final "box-level iterative refinement" is started, and the overall steps are as Figure 6 shown:

[0089] 1) Evaluation and detection: For each bounding box o excessive in the generated intermediate image X i =(x i ,y i ,w i ,h i ), and its corresponding text description d i , we first extract the bounding box o i =(x i ,y i ,w i ,h iThe detailed layout feedback module of the image input Agent in (). In addition to directly calculating the similarity between the object and its text description using the CLIP model to obtain the mechanical similarity evaluation, the detailed layout feedback module of the external built-in LLM is also selected to generate a new text description for the object to be evaluated first, and then the CLIP is used to calculate the text similarity between the new text description and the original text description, aiming to imitate the human's perceptual evaluation of the alignment between the object and the text description, and the final score is obtained by synthesizing the two evaluations. Whether to re-perform the synthesis and evaluation process is determined according to the final score. Use the trained LLM in the feedback module to generate the text description d' of this part of the image i , and then the similarity score between the content in the bounding box and the text description is obtained according to the following formula:

[0090]

[0091] where is the intermediate image X excessive The image content corresponding to the i-th box in, α is a hyperparameter used to measure the weight of the two losses, and CLIP(·,·) is a pre-trained multimodal model that receives an image and text as inputs and outputs a similarity score. If this score is lower than the set threshold, it means that the visual features of the current object do not match the text description and need to be adjusted again.

[0092] 2) Synthesis and guidance: For the bounding boxes whose similarity scores do not meet the requirements, in order to correct the objects in them, the text description d of the bounding box i is passed to a text-to-image model based on the Stable Diffusion model to generate a new object image as the reference object X ref_i =StableDiffusion(d i ). Subsequently, an external multimodal guidance loss L CLIP is used as the loss function for guidance, which consists of two parts:

[0093] Text consistency loss: Measures the cosine similarity loss between the content to be replaced and the text description.

[0094] Image consistency loss: Measures the cosine similarity loss between the synthesized object and the reference image. The specific formula is as follows:

[0095]

[0096] Furthermore, an image-to-image translation step can be performed on the initially generated image , using the Stable Diffusion model for denoising while retaining semantic information. From Start by gradually adding noise to generate a series of noisy latent variables X t , and the specific formula is as follows:

[0097]

[0098] Then use the reverse diffusion process to gradually remove the noise and restore the clean image. Specifically, at each time step t, update the noise ∈ and calculate the new latent variable X t-1 :

[0099]

[0100] After reverse diffusion through multiple time steps, the final denoised image X0 is decoded, where is the gradient of the multimodal guidance loss L CLIP , and this gradient is used to update the latent variable X t , thereby gradually improving the generated image to obtain the image to be evaluated Restart the evaluation and detection steps in 4.1. If at this time satisfies that score(s) is greater than the preset threshold, then at this time is the final image generated in the i-th bounding box, and start the "box-level iterative refinement" for the next bounding box. If the condition is still not met, repeat the above steps until the condition is satisfied.

[0101] 3) Gradually iterate and refine each bounding box step by step according to the preset rules in the above steps to obtain the global final image. It should be noted that the iterative order of bounding box optimization should satisfy: when there is coverage or intersection in the bounding box area, first process the bounding box with a larger area, and then process the bounding box with a smaller area. At this time, for the small-area bounding box, there is a certain background image as a reference for optimization, combined with the context objects in the same area as a reference, considering as much information as possible, effectively avoiding occlusion between objects, and the method of generating the overall picture by objects step by step effectively combines the abstract spatial layout relationship between objects in the text information, so that the finally obtained picture is also aligned microscopically with the layout information in the text content to the greatest extent.

Claims

1. A method for generating pictures from special texts based on distributed self-feedback, characterized in that It includes the following steps: S1: Based on the text content containing implicit requirements for special layouts input by the user, ask the LLM according to a preset template. Use the LLM to split and parse the text information in units of a single object, and combine the context with the LLM's own abstract thinking ability to determine the spatial layout of each object in the scene, and then generate multiple bounding boxes containing necessary information; S2: Iteratively execute step S1. When generating a set of bounding boxes in each round, call the macro layout feedback module to give a feedback output on the current result. Based on the confidence evaluation, use linear interpolation to combine the iterative feedback results to generate the final overall layout map. S3: Refer to the obtained final overall layout diagram containing the necessary information of each object Generate a corresponding masked latent variable N for each object according to the cross-attention extraction algorithm i ; S4: Combine the masked latent variable N of the specific object i and the text description d i to generate a latent image corresponding to each object, and then combine all the latent images with the background image to generate an intermediate image X excessive ; S5: Evaluate the content of each object bounding box in the intermediate image X one by one according to the preset rules, combine the output of the detail feedback module, use reverse image-text description to refine the content in the bounding boxes that do not meet the conditions through multiple rounds of iteration, and fuse the iteratively refined content into the original image until the content in each bounding box meets the conditions, and thus obtain the final generated image. excessive Among them, for the content of each object bounding box, combine the output of the detail feedback module, use reverse image-text description to refine the content in the bounding boxes that do not meet the conditions through multiple rounds of iteration, and fuse the iteratively refined content into the original image until the content in each bounding box meets the conditions, and thus obtain the final generated image.

2. The method for generating a picture based on a special text with step-by-step self-feedback according to claim 1, wherein The specific process of generating bounding boxes in step S1 is as follows: Input the text provided by the user into the LLM, ask according to the preset template, let the LLM split and parse the text information in units of a single object, combine the context with the LLM's own abstract thinking ability to determine the spatial layout of each object in the scene, and then generate multiple groups of bounding boxes with coordinate information and brief text descriptions.

3. The method for generating pictures from special texts based on step-by-step self-feedback according to claim 1, wherein In step S2, the macro layout feedback module is an LLM that can evaluate the confidence of the layout diagram in combination with the text description.

4. The method for generating a picture based on a special text with step-by-step self-feedback according to claim 1, wherein In step S2, a linear interpolation method is used to combine the feedback results of multiple iterations to generate the final overall layout diagram The specific steps are as follows: Assume that each set of layout box information in the instance contains 12 objects, the number of iterations is k times, and the layout diagram obtained in each iteration is evaluated for confidence by the feedback module, and the confidence μ of the result of this round of iteration is output j , the layout box combination obtained in the j-th iteration is represented by Z j = [(x1, y1, w1, h1)(x2, y2, w2, h2)……(x i , y i , w i , h i )……(x 12 , y 12 , w 12 , h 12 ), where j ∈ {1, 2, 3……k}, i ∈ {1, 2, 3……12}, and the calculation process of interpolation is recursive. The specific calculation formula is as follows: wherein is the interpolation result calculated after the j-th iteration. The number of iterations k is determined by the specific iteration result and the threshold ε (0 < ε < 1). When the current iteration result satisfies: When The system stops iterating, k = j, and takes is the final interpolation result. From the above formula (2), it can be seen that when the value of ε is larger, the conditions for the system to end the iteration are easier to meet, the number of iterations is smaller, the interpolation result when the iteration is stopped is more discrete, the calculation process consumes less resources and the final effect is worse; when the value of ε is smaller, the conditions for the system to end the iteration are more difficult to meet, the number of iterations is larger, the interpolation result when the iteration is stopped is more convergent and tends to be stable, the calculation process consumes more resources and the final result effect is better. The specific value of ε depends on the specific situation of the device. The above steps determine This is the final overall layout of the generated image.

5. The method for generating a picture based on a special text with step-by-step self-feedback according to claim 1, wherein The masked latent variable N in step S3 i is defined as follows: It refers to the initial form of each object in the image, which is a low-resolution latent representation containing the basic shape and position information of the object.

6. The method for generating a picture based on a special text with step-by-step self-feedback according to claim 1, wherein The preset algorithm steps in step S3 are as follows: 3-1) For each object i, first create a binary mask m i according to its bounding box coordinates (x i , y i , w i , h i ). This mask is a matrix with the same size as the image, where: In the area within the bounding box, the mask value is 1, indicating that this area needs to be modified or concerned; In the area outside the bounding box, the mask value is 0, indicating that this area remains unchanged; 3-2) Generate a random initial noise latent variable N T , the variable N T is shared for all boxes during subsequent denoising processes to ensure a globally consistent perspective, style, and lighting; In this process, a specific text prompt is used to generate the cross-attention map A corresponding to each box i , the text prompt is encoded using a pre-trained text encoder, and the text prompt is further mapped into a key vector k and a text token v. For the query vector q at each spatial position u u and the key vector k of each text token v, calculate q u and k v The similarity score between them is measured using the dot product, and the score is normalized to a probability distribution through the Softmax function to obtain the cross-attention map A. The specific formula is as follows: Here A uv represents the degree of relevance from the spatial position u to the text token v, divided by is to stabilize the gradient. Using the cross-attention map A uv to calculate the weighted combination and fuse the text token v into the image features. The specific formula is: A = ∑ v∈V A uv ·v(4) V is a set of token indices corresponding to the box titles, and at this time, the cross-attention map A of the i-th box is obtained i ; 3-3) For each object, through a denoising process, from to generate a latent variable containing a single instance, utilize the cross-attention map to ensure that the object is placed within the specified bounding box, strengthen the cross-attention from the pixels within the box to the text tokens related to the box caption, while weakening the cross-attention from the pixels outside the box, define a simple energy function E(A i , i, v) to continuously update the latent variable to minimize it, and the specific formula is as follows: Continuously perform this denoising process Obtain Finally obtain Finally, use the binary mask m of each object i For the latent variable Perform an element-wise multiplication operation to obtain the final masked latent variable N corresponding to the object i , that is: Thereafter, the masked latent variable N corresponding to each object is obtained i .

7. The method for generating a picture based on a special text with step-by-step self-feedback according to claim 1, wherein Step S4 obtains the generated intermediate image X excessive The specific steps are as follows: 4-1) To generate the latent image of each object, an image synthesis model M is required comp , and the masked latent variable N of each object i , the mask m i and the text description d i are input into the image synthesis model M comp to generate the latent image of each object The specific formula is as follows: 4-2) Further, using the same image synthesis model M comp , according to the background description d of the text content b generate the background image X at the corresponding position of each object backgroud , and the specific generation formula is as follows: 4-3) Further, for each object, use its corresponding masked latent variable to embed its latent image into the corresponding background image to obtain an intermediate image X excessive , and the specific formula is as follows: This step ensures that the potential image of each object is correctly placed at the corresponding position in the background image, forming a complete image; After the above steps, the generated intermediate image X excessive already contains the foreground parts of all objects and the background image. In the subsequent steps, each box is iteratively optimized one by one to further ensure the alignment between the image and the text information at the detailed level.

8. The method for generating pictures based on step-by-step self-feedback for special texts according to claim 1, characterized in that The step-by-step iterative refinement in step S5 according to the preset rules is as follows: 5-1) Evaluation and Detection: For each bounding box o excessive in the generated intermediate image X i =(x i , y i , w i , h i ), and its corresponding text description d i , first extract the image input Agent detail layout feedback module in the bounding box o i =(x i , y i , w i , h i ). In addition to directly calculating the similarity between the object and its text description using the CLIP model to obtain the mechanical similarity evaluation, the detail layout feedback module of the external built-in LLM is also selected to generate a new text description for the object to be evaluated first, and then the CLIP is used to calculate the text similarity between the new text description and the original text description, aiming to imitate the human's perceptual evaluation of the alignment degree between the object and the text description. The final score is obtained by synthesizing the two evaluations. According to the final score, it is decided whether to re-perform the synthesis and evaluation process. The trained LLM in the feedback module is used to generate the text description d' i of this part of the image, and then the similarity score between the content in the bounding box and the text description is obtained according to the following formula: Among them is the intermediate image X excessive is the image content corresponding to the i-th box in excessive . α is a hyperparameter used to measure the weights of the two losses. CLIP(·,·) is a pre-trained multimodal model that takes an image and text as inputs and outputs a similarity score. If this score is lower than the set threshold, it means that the visual features of the current object do not match the text description and need to be readjusted; 5-2) Synthesis and Guidance: For bounding boxes whose similarity scores do not meet the requirements, in order to correct the objects therein, the text description d of the bounding box i is passed to a text-to-image model based on the StableDiffusion model to generate a new object image as the reference object X ref_i = StableDiffusion(d i ), and then an external multi-modal guidance loss L CLIP is used as the guidance with the loss function, and the multi-modal guidance loss L CLIP consists of two parts Composition: Text consistency loss: Measure the cosine similarity loss between the content to be replaced and the text description; Image consistency loss: Measure the cosine similarity loss between the synthesized object and the reference image; The specific formula is as follows: Furthermore, the initially generated image can undergo an image-to-image translation step, where denoising is performed using the Stable Diffusion model while preserving semantic information. Starting from , noise is gradually added to generate a series of noisy latent variables X t , and the specific formula is as follows: Then use the reverse diffusion process to gradually remove the noise and restore the clean image. At each time step t, update the noise ε and calculate the new latent variable X t-1 : After reverse diffusion over multiple time steps, the final denoised image X0 is decoded, where is the gradient of the multimodal guidance loss L CLIP , which is used to update the latent variable X t , thereby gradually improving the generated image to obtain the image to be evaluated Restart the evaluation and detection steps in step 5-1). If at this time satisfies that score(s) is greater than the preset threshold, then at this time is the final image generated in the i-th bounding box, and the "box-level iterative refinement" of the next bounding box starts. If the condition is still not met, repeat the above steps until the condition is satisfied; 5-3) Iteratively refine each bounding box step by step according to the above steps according to the preset rules. It should be noted that the optimization and iteration order of the bounding boxes should meet the following: When there is an overlapping or intersecting phenomenon in the bounding box area, first process the bounding box with a larger area, and then process the bounding box with a smaller area. At this time, for the small-area bounding box, there is a certain background image as a reference for optimization, and the context objects in the same area are combined as a reference, taking into account as much information as possible, effectively avoiding occlusion between objects, and the method of generating the overall picture object by object step by step effectively combines the abstract spatial layout relationship between objects in the text information, so that the finally obtained picture is microscopically aligned with the layout information in the text content to the greatest extent.

Images