Training-free text graph layout controllability enhancing method for intelligent data synthesis

Abstract

The invention discloses a training-free text graph layout controllability enhancing method for intelligent data synthesis, and is applied to the technical field of intelligent and computer vision. The method comprises the following steps: inputting a natural language description demand, and assigning an expected bounding box for a key object in a scene; in the early time step of reasoning, constructing a loss function to guide the cross attention information of each object to be concentrated into a specified bounding box, so as to realize the preliminary structure planning of the scene layout; in the later time step, the background and each object are separated and optimized, and richer detail description can be injected; and finally, performing weighted fusion on the background noise and the prediction noise according to the space mask through noise mixing to generate a final image. According to the method, the problems of object conflicts, semantic crosstalk and entity missing in complex scene synthesis can be solved without additional training, the synthesis training data with accurate pixel level and consistent global style is generated, and the data acquisition cost of intelligent perception recognition model training is greatly reduced.

Description

Technical Field

[0001] This invention relates to the fields of embodied intelligence and computer vision technology, and more specifically, to a training-free method for enhancing the controllability of text-based image layouts based on a diffusion model for efficiently synthesizing training data for robot perception models. Background Technology

[0002] The development of embodied AI heavily relies on large-scale training of robots in simulated environments, especially the training of their perception systems. To enable robots to accurately identify and locate objects in their environment, a large amount of image data with precise annotations (such as bounding boxes and masks) is required. Traditional data acquisition and manual annotation methods are costly, time-consuming, and unable to cover the endless variations in real-world scenes, severely limiting the iteration speed and generalization ability of embodied AI algorithms.

[0003] Text-to-image (TTO) generation technology offers a new approach to addressing this data bottleneck. This technology generates corresponding images based on user-provided natural language descriptions. However, achieving precise spatial layout control of multiple objects within the generated image remains a significant challenge. Existing layout control methods are mainly divided into two categories: trained methods and training-free methods.

[0004] Training-based methods rely on additional annotations (boundaries, masks, instance IDs) and expensive model retraining, making them difficult to directly transfer to open vocabularies or downstream production lines. GLIGEN, LayoutDiffusion, MIGC, and other techniques add gated self-attention, instance branching, or subtask controllers to the diffusion network to achieve bounding box-level spatial constraints. While these techniques can achieve layout control, they require expensive model retraining and are difficult to generalize to open vocabularies such as novel robots or specific industrial parts.

[0005] Most training-free methods (Boxdiff, Paint-with-Words, HFG, Backward-guidance, CSG, CollageDiffusion) assume single instances and non-overlapping backgrounds. When multiple user-specified bounding boxes overlap, occlude, or have semantic conflicts, cross-attention maps interfere with each other, leading to missing objects (e.g., apples not generated), attribute crosstalk (e.g., a robotic arm displaying the red of an apple), or edge artifacts. Existing multimodal data collaboration strategies (Blended, NoiseCollage) only perform spatial clipping and pasting, without dynamic coupling with cross-attention, and cannot adjust noise weights in real time according to instance importance, resulting in discontinuous textures in overlapping areas and inconsistent overall styles. Most methods lack explicit mitigation mechanisms for semantic competition between "environment background-foreground actuators" and "actuator-actuator" relationships, still resulting in missed generation, duplicate generation, or positional drift in complex scenes, falling short of "pixel-level, one-time, commercially deliverable" quality. Therefore, providing a training-free method capable of generating complex scene images with accurate layout, realistic details, and global consistency is a key technical challenge for providing high-quality synthetic data for the embodied intelligence field. Summary of the Invention

[0006] In view of this, the present invention provides a training-free method for enhancing the controllability of text-based image layout for embodied intelligence data synthesis. By dynamically coordinating attention guidance and noise mixing during the diffusion model inference process, the method efficiently solves the problem of accurate generation of complex multi-object scenes, thereby enabling the large-scale synthesis of high-quality training data for embodied intelligence systems.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A training-free method for enhancing the controllability of text-based image layouts in embodied intelligence data synthesis includes the following steps: S1. Obtain scene prior information: The user inputs the original natural language requirements for generating Chinese images of the simulation scene, selects the object instances that need to be controlled in the language, and defines the spatial location bounding box for them, while obtaining the detailed description of the corresponding objects. S2, Layout Guidance: The Wensheng Graph model is used to reason about the original natural language requirements. In some early time steps, the object bounding boxes defined in S1 are used to first Gaussian smooth the cross attention graph, and then a loss function is constructed to guide the attention information related to each object instance to concentrate on its corresponding bounding box position. The potential noise in the reasoning process is updated through backpropagation to form the initial spatial structure of the scene. S3. Multi-instance fusion and detail optimization: For the background region, calculate the cross-attention map; for the instance, use the corresponding detailed description text to optimize the instance detail features through the extended cross-attention mechanism; adopt a noise mixing strategy to weight and fuse the noise predicted in the background region with the noise predicted in each instance region to generate synthetic noise.

[0008] Optionally, the method for obtaining detailed descriptions in S1 is either a detailed description of the instance provided by the user or a detailed description generated using a language model.

[0009] Optionally, the loss function in S2 is: ; In the formula, This is the Gaussian-smoothed cross-attention map. B The bounding box of the object in the prior information. n This refers to the number of sequences obtained after the original natural language requirement text is processed by the CLIP encoder of the text-generated graph model. H The height of the cross-attention map, W The width of the cross-attention map. r Scaling factor i The position of the object instance in the sequence. i ∈[1,n].

[0010] Optionally, the formula for updating the potential noise in the inference process through backpropagation in S2 is: ; In the formula, For time step t Potential noise, For time step t The noise level, These are the control parameters for guiding intensity.

[0011] Optionally, the formula for calculating the background region cross-attention map in S3 is: ; In the formula, The calculated background cross-attention map, Queries for background features and It is a key and value provided by the original natural language requirements. d It is a dimension.

[0012] Optionally, the calculation formula for object instance detail features optimized through the extended cross-attention mechanism in S3 is as follows: ; In the formula, To expand the cross-attention graph obtained from cross-attention computation, Queries for instance features and The key and value provided for the detailed description For strength control parameters, This is a binarized mask image created using bounding boxes.

[0013] Optionally, the noise mixing strategy formula in S3 is: ; In the formula, For synthesized noise, For the first m Noise from an object instance For background noise, For the first m A binary mask generated from the bounding box of an object. To randomly discard the matrix, This represents the background weighting coefficient.

[0014] As can be seen from the above technical solution, compared with the prior art, the present invention provides a method for enhancing the controllability of text-based image layout without training for embodied intelligent data synthesis, which has the following beneficial effects: 1. Empowering Embodied Intelligent Data Generation: This invention solves the existing system-level defects of multi-instance layout conflict, semantic crosstalk, and instance missing; in terms of process flow, attention map refinement and noise dynamic mixing dual-channel collaboration replace traditional retraining or single-channel post-processing to achieve one-time pixel-level placement of multiple targets. These synthesized data are equivalent to having built-in pixel-level position annotations, which can be directly used to train and evaluate the robot's target detection, instance segmentation and other perception algorithms. 2. No new hardware is required. It can be seamlessly coupled with any pre-trained model by simply inserting attention and noise control modules. The process complexity is simplified from "labeling-retraining-iteration" to "inference is layout". The completeness and accuracy of actuators are significantly improved in complex multi-manipulator end effector scenarios, and the global consistency of the image is maintained at the commercial level. 3. Zero cost, high efficiency and strong compatibility: This invention is a "training-free" method that requires no model modification or additional training costs. Within 30 seconds, it can simplify the originally complex "modeling-annotation" data production process into a rapid inference process of "text + box → high-quality data". It provides a plug-and-play infrastructure for subsequent production line upgrades and model capability leaps, and is applicable to multiple industries such as industrial manufacturing and logistics warehousing robots. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 This is a flowchart of the method for enhancing the controllability of text-generated image layout without training based on a diffusion model according to the present invention. Figure 2 This is a flowchart illustrating the multimodal fusion and detail optimization process of the present invention. Figure 3 This is a schematic diagram illustrating the attention guidance of the present invention; Figure 4 This is a schematic diagram illustrating the generation effect of the present invention in an application scenario of embodied intelligent data synthesis. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] This invention discloses a training-free method for enhancing the controllability of text-based image layout in embodied intelligence data synthesis, such as... Figure 1 As shown (the final generated image in the figure, i.e., the dog and frisbee, is merely an example; in this embodiment of the invention, it could be a complex industrial scene containing a robotic arm and a target object), the steps include: S1. Obtain scene prior information: The user inputs the original natural language requirements for generating Chinese images of the simulation scene, selects the object instances that need to be controlled in the language, and defines the spatial location bounding box for them, while obtaining the detailed description of the corresponding objects. S2, Layout Guidance: The Wensheng Graph model is used to reason about the original natural language requirements. In some early time steps, the object bounding boxes defined in S1 are used to first Gaussian smooth the cross attention graph, and then a loss function is constructed to guide the attention information related to each object instance to concentrate on its corresponding bounding box position. The potential noise in the reasoning process is updated through backpropagation to form the initial spatial structure of the scene. S3, such as Figure 2As shown, multi-instance fusion and detail optimization are performed as follows: For the background region, the predicted noise of the background, robotic arm, cube, and ball is fused to calculate a cross-attention map; for the instances, the corresponding detailed description text is used to optimize the instance detail features through an extended cross-attention mechanism; a noise mixing strategy is adopted to weight and fuse the noise predicted in the background region with the noise predicted in each instance region to generate synthetic noise; the final simulation training image with accurate layout and rich details is obtained.

[0019] In the diagram, M represents the total number of instances that need to be controlled; : A collection of instance tags, each element Specify the first i A category or name of an object / entity; : A set of bounding boxes, each Provide the spatial location and dimensions of the corresponding instance in the diagram; : A collection of detailed description hints, each right exist The text describes semantic details such as the interior appearance and style.

[0020] In this embodiment of the invention, S2 uses the Stable diffusion-1.5 textural graph model.

[0021] Furthermore, the method for obtaining detailed descriptions in S1 is either for the user to provide detailed descriptions of the instance or for a detailed description to be generated using a language model.

[0022] Furthermore, the loss function in S2 is: ; In the formula, This is the Gaussian-smoothed cross-attention map. B The bounding box of the object in the prior information. n This refers to the number of sequences obtained after the original natural language requirement text is processed by the CLIP encoder of the text-generated graph model. H The height of the cross-attention map, W The width of the cross-attention map. r Scaling factor i The position of the object instance in the sequence. i ∈[1,n].

[0023] Furthermore, the formula for updating the potential noise in the inference process through backpropagation in S2 is as follows: ; In the formula, For time stept Potential noise, For time step t The noise level, These are the control parameters for guiding intensity.

[0024] Furthermore, the formula for calculating the background region cross-attention map in S3 is as follows: ; In the formula, The calculated background cross-attention map, Queries for background features and It is a key and value provided by the original natural language requirements. d It is a dimension.

[0025] Furthermore, in S3, the calculation formula for optimizing the detailed features of object instances through the extended cross-attention mechanism is as follows: ; In the formula, To expand the cross-attention graph obtained from cross-attention computation, Queries for instance features and The key and value provided for the detailed description For strength control parameters, This is a binarized mask image created using bounding boxes.

[0026] In an embodiment of the present invention, The calculation formula is: ; In the formula, To control the intensity of detail optimization.

[0027] Furthermore, the noise mixing strategy formula in S3 is as follows: ; In the formula, For synthesized noise, For the first m Noise from an object instance For background noise, For the first m A binary mask generated from the bounding box of an object, where inside the box = 1 and outside the box = 0; This is a random dropout matrix; in this embodiment, rate=1 by default, used to randomly drop instance noise outside the frame. This is the background weighting coefficient, which is set to 0.1 by default in this embodiment to control the proportion of background noise.

[0028] In one embodiment of the present invention, experiments were conducted on the method of this embodiment. The ablation experiment results are shown in Table 1, and the comparative experiment results are shown in Table 2. Table 1 Ablation Experiment Results

[0029] Table 2 Comparison of experimental results

[0030] In the table, LG refers to layout guidance, NB refers to noise mixing, ECA refers to extended cross-attention mechanism, mAP is used to measure layout success rate, and CLIP-I is used to measure the quality of generated images; experimental data are extracted from MSCOCO data, and the original image for CLIP-I metric calculation comes from MSCOCO. In this embodiment, the layout guidance effect is as follows: Figure 3 As shown, in embodied intelligence applications, "dog" and "boat" in the diagram can be replaced with "robot arm" and "target block," respectively, as the physical principle of attention guidance remains the same; the generated effect is as follows. Figure 4 As shown.

[0031] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0032] Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for enhancing the controllability of text-based image layout without training for embodied intelligence data synthesis, characterized in that, Includes the following steps: S1. Obtain scene prior information: The user inputs the original natural language requirements for generating Chinese images of the simulation scene, selects the object instances that need to be controlled in the language, and defines the spatial location bounding box for them, while obtaining the detailed description of the corresponding objects. S2, Layout Guidance: The Wensheng Graph model is used to reason about the original natural language requirements. In some early time steps, the object bounding boxes defined in S1 are used to first Gaussian smooth the cross attention graph, and then a loss function is constructed to guide the attention information related to each object instance to concentrate on its corresponding bounding box position. The potential noise in the reasoning process is updated through backpropagation to form the initial spatial structure of the scene. S3, Multi-instance fusion and detail optimization: For the background region, calculate the cross-attention map; for the instance, use the corresponding detailed description text to optimize the instance detail features through the extended cross-attention mechanism; A noise mixing strategy is adopted to weight and fuse the noise predicted in the background region with the noise predicted in each instance region to generate synthetic noise.

2. The method for enhancing the controllability of text-based image layout without training for embodied intelligent data synthesis according to claim 1, characterized in that, The method for obtaining detailed descriptions in S1 is either a detailed description of the instance provided by the user or a detailed description generated using a language model.

3. The method for enhancing the controllability of text-based image layout without training for embodied intelligence data synthesis according to claim 1, characterized in that, The loss function in S2 is: ； In the formula, This is the Gaussian-smoothed cross-attention map. B The bounding box of the object in the prior information. n This refers to the number of sequences obtained after the original natural language requirement text is processed by the CLIP encoder of the text-generated graph model. H The height of the cross-attention map, W The width of the cross-attention map. r Scaling factor i The position of the object instance in the sequence. i ∈[1,n].

4. The method for enhancing the controllability of text-based image layout without training for embodied intelligence data synthesis according to claim 1, characterized in that, The formula for updating the potential noise in the inference process through backpropagation in S2 is: ； In the formula, For time step t Potential noise, For time step t The noise level, These are the control parameters for guiding intensity.

5. The method for enhancing the controllability of text-based image layout without training for embodied intelligent data synthesis according to claim 1, characterized in that, The formula for calculating the background region cross-attention map in S3 is: ； In the formula, The calculated background cross-attention map, Queries for background features and It is a key and value provided by the original natural language requirements. d It is a dimension.

6. The method for enhancing the controllability of text-based image layout without training for embodied intelligence data synthesis according to claim 1, characterized in that, The formula for calculating the detailed features of object instances in S3, optimized through the extended cross-attention mechanism, is as follows: ； In the formula, To expand the cross-attention graph obtained from cross-attention computation, Queries for instance features and The key and value provided for the detailed description For strength control parameters, This is a binarized mask image created using bounding boxes.

7. The method for enhancing the controllability of text-based image layout without training for embodied intelligence data synthesis according to claim 1, characterized in that, The noise mixing strategy formula in S3 is: ； In the formula, For synthesized noise, For the first m Noise from an object instance For background noise, For the first m A binary mask generated from the bounding box of an object. To randomly discard the matrix, This represents the background weighting coefficient.

Images