Scene image generation method and apparatus with three-dimensional space and orientation control

Abstract

The present disclosure relates to the technical field of image processing and computer vision, and particularly relates to a scene image generation method and device with three-dimensional space and orientation control. The method comprises: generating an object token set and a scene token set based on a local object description of each object in a three-dimensional scene, a two-dimensional bounding box and a three-dimensional angle; inputting a global scene description, the object token set and the scene token set into a diffusion model; in an iterative denoising process, performing parallel cross-attention interaction between a visual hidden state and the object token set and the scene token set respectively, and injecting the interaction result into the visual hidden state; and generating a two-dimensional image corresponding to the scene description data according to a final denoising output hidden variable when the iterative denoising process is completed. The method overcomes the dependence on a dense three-dimensional mapping graph of related technologies through 2.5D light-weight conditional coding and double-flow differential cross-attention injection, and can realize multi-object accurate orientation and layout control.

Description

Technical Field

[0001] This disclosure relates to the fields of image processing and computer vision technology, and in particular to a method and apparatus for generating scene images with three-dimensional space and orientation control. Background Technology

[0002] In recent years, text-to-image diffusion models have made significant progress, enabling the generation of high-quality images based on natural language descriptions. However, plain text prompts struggle to precisely control the spatial position, scale, and 3D orientation (such as horizontal angle, pitch angle, and rotation angle) of each object in multi-object scenes, limiting their practicality in applications requiring fine composition, such as e-commerce displays, game scene design, and interior rendering.

[0003] Among related technologies, some rely on dense 3D mapping maps (such as depth maps or normalized object coordinate space maps) as input. While these provide rich geometric information, they require individual rendering of each object using a 3D rendering engine, resulting in high computational overhead, long preprocessing time, and a high barrier to entry for users who need to master 3D modeling software. Other methods only support control of a single angular dimension (such as the horizontal angle), cannot simultaneously control the pitch and rotation angles, and are difficult to extend to complex scenes containing multiple objects.

[0004] Therefore, how to achieve precise 3D orientation (all three angles) and spatial layout control for each object in a multi-object scene without relying on dense 3D mapping, while enabling the model to perceive global contextual information between objects to generate structurally reasonable images, has become an urgent technical problem to be solved. Summary of the Invention

[0005] In view of this, this disclosure proposes a method and apparatus for generating scene images with three-dimensional space and orientation control.

[0006] According to one aspect of this disclosure, a method for generating scene images with three-dimensional space and orientation control is provided, the method comprising:

[0007] Acquire scene description data of a 3D scene, the scene description data including a global scene description and multimodal information of each object in the 3D scene, the multimodal information including local object description, 2D bounding box and 3D angle;

[0008] The multimodal information of each object is feature-encoded and mapped, and fused to generate an object token set. The object token set includes the object token of each object, and the object token is used to represent object features containing local spatial and angular binding relationships.

[0009] The object token set is subjected to self-attention processing to generate a scene token set, which includes the scene token of each object. The scene token is used to characterize scene features that contain global physical context.

[0010] The global scene description, the object token set, and the scene token set are input into the pre-trained diffusion model;

[0011] In the iterative denoising process of the diffusion model, the visual latent state is subjected to parallel cross-attention interaction with the object token set and the scene token set, respectively, and the interaction result is injected into the visual latent state to obtain the updated visual latent state. The visual latent state is an intermediate latent variable representing image features in the visual flow of the diffusion model.

[0012] As the iterative denoising process is completed, a two-dimensional image corresponding to the scene description data is generated based on the latent variables of the final denoised output.

[0013] In one possible implementation, a dynamic layout mask is applied to the cross-attention between the visual latent state and the set of object tokens, while no mask is applied to the cross-attention between the visual latent state and the set of scene tokens. The dynamic layout mask is an attention mask matrix generated based on the two-dimensional bounding box of each object, used to constrain the receptive field of the cross-attention.

[0014] In another possible implementation, the feature encoding and mapping of the multimodal information of each object, and the fusion to generate an object token set, includes:

[0015] For the multimodal information of each object, the local object description is encoded as a text embedding vector, the two-dimensional bounding box is encoded as a bounding box embedding vector, and the three-dimensional angle is encoded as an angle embedding vector;

[0016] The text embedding vector, the bounding box embedding vector, and the angle embedding vector are concatenated to obtain a concatenated vector.

[0017] The spliced ​​vector is input into a multi-layer perceptron (MLP) for projection to generate the object token.

[0018] In another possible implementation, during the iterative denoising process of the diffusion model, the visual latent state is subjected to parallel cross-attention interaction with the object token set and the scene token set, respectively, and the interaction result is injected into the visual latent state to obtain an updated visual latent state, including:

[0019] In the iterative denoising process of the diffusion model, the object token set is projected onto the key space and value space to obtain the object key vector and object value vector; and the scene token set is projected onto the key space and value space to obtain the scene key vector and scene value vector.

[0020] Based on the query vector of the visual latent state, the object key vector, the object value vector, and the dynamic layout mask, calculate the local attention output;

[0021] Calculate the global attention output based on the query vector, the scene key vector, and the scene value vector;

[0022] The interaction result is obtained by weighted summing of the global attention output and the local attention output scaled by the gating parameters.

[0023] The interaction result is superimposed onto the visual latent state as a residual to obtain the updated visual latent state.

[0024] In another possible implementation, a positive offset time step sampling strategy is used when training the diffusion model. This positive offset time step sampling strategy is used to indicate that the sampling probability density of the time step is offset towards the beginning of the iterative denoising process, so that the diffusion model learns the global spatial layout of the object.

[0025] In another possible implementation, the diffusion model is trained using flow matching loss. When calculating the flow matching loss, a first weight is assigned to the spatial location within the union region of all two-dimensional bounding boxes, and a second weight is assigned to the spatial location within the background region. The first weight is greater than the second weight.

[0026] In another possible implementation, the method further includes:

[0027] Obtain the original image set, which includes multiple original images;

[0028] For each of the original images, if the original image only has 3D bounding box annotations, then the 3D angle of the object is calculated from the 3D bounding box of each object in the original image, and a 2D bounding box of each object is generated using a pre-trained object detection model.

[0029] If the original image only has two-dimensional bounding box annotations, then a pre-trained orientation prediction model is used to predict the three-dimensional angle of each object in the original image to obtain the three-dimensional angle of each object.

[0030] The pre-trained multimodal large language model is used to generate a global scene description of the original image and a local object description of each object.

[0031] Each original image, the global scene description corresponding to the original image, the local object description of each object in the original image, the two-dimensional bounding box, and the three-dimensional angle are organized into structured samples to form a dataset.

[0032] According to one aspect of this disclosure, a scene image generation apparatus with three-dimensional space and orientation control is provided, the apparatus comprising:

[0033] The acquisition module is used to acquire scene description data of a three-dimensional scene. The scene description data includes a global scene description and multimodal information of each object in the three-dimensional scene. The multimodal information includes local object descriptions, two-dimensional bounding boxes, and three-dimensional angles.

[0034] The conditional encoding module is used to perform feature encoding and mapping on the multimodal information of each object, and fuse them to generate an object token set. The object token set includes the object token of each object, and the object token is used to represent object features containing local spatial and angular binding relationships.

[0035] A context-aware dual-stream representation module is used to perform self-attention processing on the object token set to generate a scene token set, wherein the scene token set includes a scene token for each object, and the scene token is used to represent scene features containing global physical context.

[0036] An image generation module is used to input the global scene description, the object token set, and the scene token set into a pre-trained diffusion model. During the iterative denoising process of the diffusion model, the visual hidden state is subjected to parallel cross-attention interaction with the object token set and the scene token set, respectively, and the interaction result is injected into the visual hidden state to obtain an updated visual hidden state. The visual hidden state is an intermediate latent variable representing image features in the visual flow of the diffusion model. As the iterative denoising process is completed, a two-dimensional image corresponding to the scene description data is generated based on the latent variables of the final denoised output.

[0037] According to another aspect of this disclosure, a scene image generation apparatus with three-dimensional space and orientation control is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0038] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0039] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0040] This disclosure achieves precise control over the spatial position and 3D orientation (horizontal angle, pitch angle, rotation angle) of each object in a multi-object scene by explicitly providing a 2D bounding box and 3D angle for each object and encoding and fusing these multimodal information into object tokens. This is done without relying on dense 3D mapping maps such as depth maps or normalized object coordinate space maps, thus avoiding complex 3D rendering preprocessing and significantly reducing computational overhead and user operation barriers. Furthermore, the method performs self-attention processing on the object token set to generate scene tokens, enabling each object's token to perceive the geometric and semantic information of all other objects. This provides a global physical context for the diffusion model while enhancing the structural rationality and overall consistency between objects in the generated image. In addition, the method uses the global scene description, object tokens, and scene tokens as conditions to perform parallel cross-attention interactions with the intermediate hidden states in the visual flow of the diffusion model. The interaction results are injected only into the visual hidden states without affecting the text flow of the pre-trained model, thus seamlessly adapting to modern multimodal diffusion transformer architectures and protecting the integrity of the original text semantic representation. Users only need to provide local descriptions, bounding boxes, 3D angles, and global scene descriptions for each object to generate end-to-end 3D scene images that meet the requirements for accurate layout and orientation. This is suitable for practical applications such as e-commerce product display, game scene design, and interior decoration rendering.

[0041] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0042] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0043] Figure 1 A flowchart illustrating a scene image generation method with three-dimensional space and orientation control provided by an exemplary embodiment of this disclosure is shown.

[0044] Figure 2 A flowchart illustrating the overall system architecture provided by an exemplary embodiment of this disclosure is shown.

[0045] Figure 3 This illustrates the controllable generation effect provided by an exemplary embodiment of the present disclosure in different scenarios.

[0046] Figure 4The results show a qualitative comparison between the method provided in this disclosure and existing methods in the controllable generation task.

[0047] Figure 5 A typical labeled sample of the OrientLayout dataset provided in an exemplary embodiment of this disclosure is shown.

[0048] Figure 6 The qualitative comparison results of an ablation experiment provided by an exemplary embodiment of this disclosure are shown.

[0049] Figure 7 A schematic diagram of the structure of a scene image generation apparatus with three-dimensional space and orientation control provided in an exemplary embodiment of the present disclosure is shown.

[0050] Figure 8 This is a block diagram illustrating an apparatus according to an exemplary embodiment. Detailed Implementation

[0051] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0052] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0053] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0054] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are used only to distinguish one element / operation from another. Therefore, without departing from the teachings of the conception of embodiments of this disclosure, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0055] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0056] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0057] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.

[0058] First, some terms used in the embodiments of this disclosure will be introduced.

[0059] 1. Text-to-Image (T2I): A deep learning task that takes natural language text as input and generates corresponding images. In this embodiment, both the global scene description and the local object description are provided in text form, which are conditional inputs for the T2I task.

[0060] 2. Multi-Modal Diffusion Transformer (MM-DiT): A diffusion model architecture that explicitly decouples textual and visual latent states, such as Stable Diffusion 3 and later versions. The diffusion model in this disclosure can be designed based on the MM-DiT architecture, injecting layout and orientation conditions only into the visual flow while maintaining the independence of the text flow.

[0061] 3. 2.5-Dimensional (2.5D): A hybrid representation method between two-dimensional and three-dimensional. In this embodiment, a two-dimensional bounding box is used to represent the spatial position of an object, and three-dimensional angles (horizontal angle, pitch angle, rotation angle) are combined to represent the orientation of the object. The lightweight 2.5D conditions replace the dense three-dimensional mapping map to achieve precise spatial layout and three-dimensional orientation control.

[0062] 4. Binary Orientation Verification (BOV): The orientation evaluation protocol proposed in this disclosure. It transforms the traditional multi-directional classification task (such as dividing the horizontal angle into eight directions) into a simple "yes / no" judgment problem—only asking the evaluator "whether the object is facing the target direction," thereby eliminating subjective bias at the boundaries of the angle interval and obtaining reliable evaluation results through a multi-vote mechanism.

[0063] 5. Two-dimensional bounding box: A rectangular region representing the position and scale of an object in an image in the form of minimum-maximum coordinates. In this embodiment of the disclosure, the two-dimensional bounding box is used to constrain the generation region of each object and serves as the basis for generating the dynamic layout mask.

[0064] 6. Contrastive Language-Image Pre-Training (CLIP): A vision-language pre-training model proposed by OpenAI, capable of extracting semantically consistent image-text embedding vectors. This embodiment uses CLIP's text encoder to encode the local object description of each object into pooled embeddings, serving as the semantic components of the object token.

[0065] 7. Classifier-Free Guidance (CFG): A weighting coefficient balancing conditional and unconditional generation during diffusion model inference, used to control the degree to which generated images adhere to text prompts, layout conditions, etc. This embodiment employs a decoupled modality discarding strategy during training, allowing for flexible adjustment of the strength of layout and orientation conditions using CFG during the inference phase.

[0066] 8. MLP: A neural network consisting of multiple fully connected layers stacked together, typically containing non-linear activation functions. In this embodiment, MLP is used for encoding bounding box embeddings and angle embeddings, and for projecting multimodal concatenation vectors into a unified object token.

[0067] 9. Fréchet Inception Distance (FID): A standard evaluation metric for measuring the realism and diversity of generated images. It is obtained by calculating the Fréchet distance between the generated image and the real image in the Inception network feature space. The lower the value, the higher the generation quality. This disclosure uses FID as one of the evaluation metrics for overall image quality.

[0068] 10. Object-level CLIP Score (O-CLIP): A fine-grained semantic alignment evaluation metric proposed in this disclosure. The bounding box region of each object in the generated image is cropped out, and its CLIP similarity is calculated separately with the corresponding object's local description text. The average of these similarities is then applied to all objects, thereby more accurately evaluating the model's adherence to object attributes (color, material, motion, etc.).

[0069] 11. Large Language Model (LLM): such as GPT-4, Qwen and other pre-trained language models. In this embodiment of the disclosure, the LLM can be used as a layout planner to automatically parse the user's natural language instructions (such as "put a sofa and two side tables in the living room") into structured condition tuples (semantic description, bounding box, 3D angle).

[0070] 12. Multimodal Large Language Model (MLLM): A large language model capable of processing both images and text simultaneously. In this embodiment, MLLM is used to automatically generate a global scene description and a local object description for each object in the dataset, and employs a dual-image input strategy (complete scene image + cropped view) to eliminate background interference, achieving semantically decoupled object-level description.

[0071] 13. Root Mean Square Normalization (RMSNorm): An operation that uses only root mean square statistics for normalization and does not include learnable affine parameters. In this embodiment, RMSNorm is performed on the key vectors of the projected object tokens and scene tokens before the cross-attention calculation of the dual-stream injection module to stabilize the training process.

[0072] 14. Sigmoid Linear Unit (SiLU): A smooth, non-linear activation function, defined as... ,in For input values, The Sigmoid function is used. In this embodiment, the 2D bounding box encoding and 3D angle encoding can use SiLU as the activation function.

[0073] This disclosure relates to an image generation method for achieving precise layout and orientation control of multiple objects under lightweight 2.5D spatial-angular conditions in a T2I diffusion model, and a context-aware dual-stream representation and injection mechanism for eliminating the problem of multi-object attribute leakage. This disclosure proposes a systematic solution to address the shortcomings of related technologies, resolving four core technical issues.

[0074] To address a technical problem (how to achieve precise 3D orientation control and spatial layout control while overcoming the dependence of related technologies on dense 3D mapping maps), this disclosure addresses the issue by parameterizing each object in a 3D scene as a lightweight 2.5D triplet: a local object description, a 2D bounding box, and 3D angles (horizontal angle, pitch angle, and rotation angle). Semantic embeddings are extracted using a CLIP text encoder, bounding box embeddings are generated using Fourier position encoding and an MLP, angle embeddings are generated using trigonometric function mapping and an MLP, and finally, a unified object token is generated through MLP fusion projection. This solution completely avoids the use of dense 3D mapping maps such as depth maps and normalized object coordinate space maps, using only a small number of values ​​to simultaneously encode spatial position and 3D orientation, achieving precise control with zero latency and low overhead.

[0075] To address another technical challenge (how to simultaneously eliminate attribute leakage and maintain natural global scene harmony in multi-object generation scenarios), this disclosure constructs a differentiated masking strategy. First, by performing self-attention processing on object tokens, scene tokens containing global geometric and semantic relationships are generated, forming a dual-stream conditional space in parallel with the object tokens. Then, two cross-attention branches are executed in parallel within the visual flow of the diffusion model: a dynamic layout mask is applied to the object tokens (local flow), forcing each visual position to interact only with object tokens within its bounding box, completely blocking the penetration of attributes such as color and material between different objects; no mask is applied to the scene tokens (global flow), allowing the entire receptive field to be captured, thereby capturing occlusion relationships, lighting consistency, and physical interactions between objects, maintaining overall scene harmony. The two outputs are gated and aggregated before being injected into the visual latent state, thus simultaneously achieving "strict entity isolation" and "global context harmony."

[0076] To address another technical challenge (how to seamlessly integrate the aforementioned conditional control mechanism into modern MM-DiT architectures without polluting pre-trained text representations and maintaining high parameter efficiency), this disclosure protects the purity of the pre-trained text encoding path by injecting dual-stream conditional parameters only into the visual latent state, without modifying the text latent state at all. Simultaneously, the local and global streams share the same set of key-value projection matrices, significantly reducing the number of trainable parameters (approximately 15.3% of similar schemes). Furthermore, the use of zero-initialized trainable gating parameters ensures that the model behavior in the early stages of training is completely consistent with the original pre-trained MM-DiT, avoiding gradient instability and achieving plug-and-play adaptation.

[0077] To address another technical problem (how to construct a standardized dataset that can simultaneously serve layout control and orientation control, and design an objective and reliable orientation evaluation protocol), this disclosure constructs the OrientLayout dataset and proposes a BOV binary orientation verification protocol. In dataset construction, multiple data sources including COCO, Objectron, and Cityscapes are integrated, and two annotation pipelines are designed: for data sources providing 3D bounding boxes, 3D angles are calculated from the bounding boxes, and 2D bounding boxes are generated using an object detection model; for data sources providing only 2D bounding boxes, the OrientAnything orientation prediction model is used to predict 3D angles, and high-quality 2.5D samples, totaling 110,000 images, are generated through coordinate system standardization (using the OrientAnything output as a reference to correct category-level orientation ambiguity) and relation-aware semantic annotation (dual-image input from a multimodal large language model and interference exclusion instructions). In the evaluation protocol, the traditional multi-directional classification task is transformed into a simple binary judgment problem, only asking "whether the object is facing the target direction." A multi-voting mechanism eliminates subjective bias at angle boundaries, obtaining objective and reliable orientation control evaluation results.

[0078] The following describes a scene image generation method with three-dimensional space and orientation control provided by embodiments of this disclosure using several exemplary models.

[0079] Please refer to Figure 1 This document illustrates a flowchart of a scene image generation method with three-dimensional spatial and orientation control provided in an exemplary embodiment of the present disclosure. This embodiment uses the method in a computing device as an example for illustration. The method includes the following steps.

[0080] Step 101: Obtain scene description data for the 3D scene. The scene description data includes a global scene description and multimodal information for each object in the 3D scene. The multimodal information includes local object descriptions, 2D bounding boxes, and 3D angles.

[0081] Scene description data for a 3D scene refers to the complete set of conditions provided by the user to define the content of the image to be generated. This data includes a global scene description and multimodal information of each object in the 3D scene.

[0082] A global scene description is a piece of natural language text used to describe the macroscopic attributes of the entire scene, such as scene type, lighting conditions, overall style, and the interaction between objects.

[0083] Multimodal information refers to the conditional information provided for each object in a scene in three different modalities, including local object description, two-dimensional bounding box, and three-dimensional angle.

[0084] A local object description refers to a piece of natural language text used to describe the fine-grained visual attributes of a single object, such as color, material, shape, and motion. It should be noted that the local object description can be entered individually by the user or automatically extracted from the global scene description. This disclosure does not limit this aspect.

[0085] A two-dimensional bounding box is a rectangular region in an image that represents the position and scale of an object in the form of minimum-maximum coordinates. The coordinate values ​​are usually normalized to the interval [0,1].

[0086] A three-dimensional angle refers to an object's orientation parameter in three-dimensional space, comprising three components: horizontal angle (left and right rotation), pitch angle (up and down tilt), and rotation angle (rotation about its own axis), typically expressed in radians or degrees. In one optional implementation, dense 3D maps of each object can be received, and these maps can be converted into the aforementioned 2.5D conditional inputs (i.e., local object descriptions, two-dimensional bounding boxes, and three-dimensional angles) using projection techniques for subsequent conditional encoding.

[0087] In some embodiments, possible implementations for obtaining scene description data of a 3D scene include: One possible implementation is to receive user input through a graphical user interface: the user drags and draws the bounding box of each object on a canvas, sets the horizontal angle, pitch angle, and rotation angle using sliders or numerical input boxes, and fills in local object descriptions and global scene descriptions in text boxes. The front end collects all data and sends it to the back end in a structured format. Another possible implementation is to receive input through a command line or application programming interface: the user writes a configuration file in a predefined format, which contains a global scene description string and local descriptions, bounding box coordinates, and angle values ​​for each object. The back end parses the file to obtain the scene description data. Yet another possible implementation is to automatically parse using a large language model: the user inputs a natural language instruction (e.g., "Place a red cup on the left side of the living room, with the cup facing right"), and a pre-trained large language model is invoked to convert the instruction into structured scene description data, including a global description, local descriptions of each object, estimated bounding boxes, and 3D angles.

[0088] Step 102: Perform feature encoding and mapping on the multimodal information of each object, and fuse them to generate an object token set. The object token set includes the object token of each object. The object token is used to represent the object features containing local spatial and angular binding relationships.

[0089] Feature encoding refers to the process of converting raw input data into high-dimensional feature vectors that a model can process. Different types of data use different encoders.

[0090] Mapping refers to transforming encoded feature vectors into a unified feature space through one or more fully connected layers (MLPs), so that features from different modalities have the same dimension.

[0091] An object token set is a collection of object tokens for all objects in a scene, where each object token is a fixed-dimensional vector. An object token is a compact vector representation that integrates the semantics, spatial location, and 3D orientation information of a single object, preserving its local spatial and angular binding relationships.

[0092] The local spatial and angular binding relationship refers to the inherent correspondence between the spatial position (defined by the bounding box) of each object and its three-dimensional orientation (defined by the angle). For example, a cup located on the right side of an image and facing the left has a geometric constraint between its spatial position and orientation.

[0093] Object features refer to feature vectors that describe the comprehensive attributes of a single object, including texture, color, shape (from text), as well as position, scale (from bounding box), and orientation (from angle).

[0094] In some embodiments, for the multimodal information of each object, the local object description is encoded as a text embedding vector, the two-dimensional bounding box is encoded as a bounding box embedding vector, and the three-dimensional angle is encoded as an angle embedding vector; the text embedding vector, the bounding box embedding vector, and the angle embedding vector are concatenated to obtain a concatenated vector; the concatenated vector is input into the MLP for projection to generate an object token.

[0095] Text embedding vectors are high-dimensional feature vectors output by inputting local object descriptions into a pre-trained text encoder (such as the CLIP text encoder). These vectors provide a compact representation of the original text in the semantic space.

[0096] Bounding box embedding vectors are fixed-dimensional feature vectors (e.g., 256-dimensional) obtained by Fourier position encoding of two-dimensional bounding box coordinates and processing them through MLP, used to characterize the spatial position and scale information of an object.

[0097] Angle embedding vector refers to a fixed-dimensional feature vector (e.g., 256 dimensions) output after inputting the sine and cosine values ​​of the three-dimensional angle (a total of 6 dimensions) into a three-layer MLP, which is used to characterize the three-dimensional orientation information of an object.

[0098] A concatenated vector is a high-dimensional vector obtained by concatenating the text embedding vector, bounding box embedding vector, and angle embedding vector along the feature dimension. Its dimension is the sum of the dimensions of the three vectors.

[0099] MLP refers to a neural network consisting of multiple fully connected layers stacked together. Each layer typically contains a non-linear activation function (such as SiLU) and a normalization layer (such as LayerNorm) to map the input vector to the target feature space.

[0100] One possible implementation is to call a pre-trained CLIP text encoder (e.g., ViT-B / 32) for local object descriptions, inputting a string and outputting a pooled text embedding vector; for two-dimensional bounding boxes, the coordinates are first normalized to [0,1], then Fourier position encoding (using 16 frequency bands) is performed to map the 4-dimensional coordinates to a high-dimensional space, and then two layers of MLP (each followed by SiLU activation and LayerNorm) are used to output a 256-dimensional bounding box embedding; for three-dimensional angles, the sine and cosine of the three angles are calculated to obtain a 6-dimensional vector, and then three layers of MLP (SiLU+LayerNorm) are used to output a 256-dimensional angle embedding. The three embedding vectors described above are concatenated along the feature dimension into a high-dimensional vector (the dimension is the text embedding dimension plus 256 plus 256). This concatenated vector is then input into a fusion MLP (typically a two- or three-layer fully connected network; the intermediate dimensions can be designed to be 512 or 1024, and the output dimension is consistent with the object token dimension required by the subsequent model, such as 256 or 768). The fusion MLP performs non-linear projection, compressing the multimodal information into a unified representation space. The above encoding, concatenation, and mapping operations are performed independently for each object, and finally, all object tokens are collected to form an object token set.

[0101] Step 103: Perform self-attention processing on the object token set to generate a scene token set. The scene token set includes the scene token of each object. The scene token is used to represent scene features that contain the global physical context.

[0102] Self-attention processing is a sequence modeling operation that aggregates information from the entire sequence by calculating the attention weights of each element in the input sequence with all elements in the sequence (including itself), thereby capturing long-distance dependencies between elements.

[0103] The scene token set refers to the new token set output after the object token set is input into the scene self-attention network (Transformer). Each scene token has the same dimension as the object token.

[0104] Scene tokens are context-aware, relation-rich global representations. Specifically, scene tokens retain the features of an individual object while also encoding the global layout context. Through feature interactions, each object can perceive the attributes and spatial (position and orientation) information of all other objects in the scene. Scene tokens ultimately constitute the global flow in the network, primarily used to capture spatial interactions between multiple objects, natural physical blending, occlusion relationships, relative depth, and consistent lighting conditions. In other words, a scene token is a vector representation that not only contains information from the original object token but also incorporates the geometric and semantic information of all other objects in the scene. For example, a scene token for a cup might include information such as the presence of a chair to its right, the chair's orientation, and their relative positions.

[0105] The global physical context refers to the spatial layout relationships (front and back, left and right, size ratio) between objects in the entire scene, possible occlusion relationships, lighting consistency effects, and physical interactions between objects (such as one object being placed on top of another).

[0106] Scene features refer to the feature representations that describe the structured information of the entire scene. They are contrasted with individual object features and are used to guide the generative network to maintain global consistency.

[0107] In some embodiments, the possible implementation of self-attention processing on the object token set to generate the scene token set includes: inputting the object token set as a sequence into a scene self-attention network, which is a multi-layer (e.g., 4-layer) self-attention network, each layer containing multi-head self-attention (e.g., 8 heads) and a feedforward network, and employing residual connections and layer normalization. Each object token in the sequence interacts with all other object tokens through the self-attention layer, aggregating global information, and outputting a sequence of the same length after processing, i.e., the scene token set.

[0108] Step 104: Input the global scene description, object token set, and scene token set into the pre-trained diffusion model.

[0109] The pre-trained diffusion model refers to MM-DiT, which has been pre-trained on a large-scale image and text dataset. This model has two independent processing streams: a text stream (processing global scene descriptions) and a visual stream (processing image latent variables).

[0110] Step 105: In the iterative denoising process of the diffusion model, the visual latent state is subjected to parallel cross-attention interaction with the object token set and the scene token set, respectively, and the interaction result is injected into the visual latent state to obtain the updated visual latent state. The visual latent state is an intermediate latent variable representing image features in the visual flow of the diffusion model.

[0111] The iterative denoising process refers to the process by which a diffusion model generates an image, starting with a random noise latent variable that follows a standard Gaussian distribution, and gradually denoising according to a predefined time schedule. At each step, the noise in the current latent variable is predicted and a portion is subtracted, ultimately resulting in a clean image latent variable.

[0112] The visual latent state refers to the intermediate latent variables representing image features in the visual flow of the diffusion model at the current time step or in the current Transformer block. It is updated in each Transformer block. The updated visual latent state is the final visual latent state obtained after processing by all Transformer blocks. This state has incorporated the global description, object-level layout, and orientation conditions, and has undergone denoising updates at the current time step.

[0113] Parallel cross-attention interaction refers to executing two cross-attention branches simultaneously: the first branch performs cross-attention calculation with the query vector generated by the visual latent state and the key and value of the object token set, and applies a dynamic layout mask; the second branch performs cross-attention calculation with the same query vector and the key and value of the scene token set, without applying any mask; the two branches share the same set of key-value projection matrices.

[0114] The interaction result refers to the aggregated output of the two branches, which is the value of the global branch output and the local branch output combined by the gating parameter, and then multiplied by the conditional strength coefficient. The gating parameter is a trainable scalar or vector used to adjust the contribution of the local attention output to the final interaction result; it is initialized to 0 at the beginning of training and learned during training. The conditional strength coefficient is an adjustable hyperparameter that can be adjusted during inference to balance conditional control and generative diversity.

[0115] Visual flow refers to the feature path that processes latent variables of an image within the diffusion model. In MM-DiT, visual flow and text flow are explicitly decoupled.

[0116] Intermediate latent variables refer to the latent variables in the diffusion model before it reaches the final clean state during the denoising process, which are between the noise and the clean image.

[0117] In some embodiments, a dynamic layout mask is applied to the cross-attention between the visual latent state and the set of object tokens, while no mask is applied to the cross-attention between the visual latent state and the set of scene tokens. The dynamic layout mask is an attention mask matrix generated based on the two-dimensional bounding box of each object and used to constrain the receptive field of the cross-attention.

[0118] In some embodiments, during the iterative denoising process of the diffusion model, the object token set is projected onto the key space and value space to obtain object key vectors and object value vectors; the scene token set is also projected onto the key space and value space to obtain scene key vectors and scene value vectors; based on the query vector, object key vector, and object value vector of the visual latent state and the dynamic layout mask, the local attention output is calculated; based on the query vector, scene key vector, and scene value vector, the global attention output is calculated; the global attention output is weighted and summed with the local attention output scaled by the gating parameters to obtain the interaction result; the interaction result is superimposed on the visual latent state in the form of residuals to obtain the updated visual latent state.

[0119] The key space refers to the embedding space of key vectors in the attention mechanism. Input features are mapped to this space through a learnable projection matrix, which is used to calculate the attention score.

[0120] The value space refers to the embedding space of value vectors in the attention mechanism. Input features are mapped to this space through a learnable projection matrix, which is used for weighted summation to generate the output.

[0121] An object key vector is a sequence of vectors obtained by mapping a set of object tokens through a shared key projection matrix. It is used to calculate the attention score with the query vector.

[0122] An object value vector is a sequence of vectors obtained by mapping a set of object tokens through a shared value projection matrix. It is used to aggregate weighted values ​​in attention calculation.

[0123] Scene key vectors are vector sequences obtained by mapping a set of scene tokens through a shared key projection matrix.

[0124] Scene value vectors refer to the vector sequence obtained by mapping the scene token set through a shared value projection matrix.

[0125] The query vector is a sequence of vectors generated from the visual latent state through a linear transformation, used to calculate the similarity with the key vector in cross-attention.

[0126] Dynamic layout mask refers to the attention mask matrix dynamically generated based on the two-dimensional bounding box of each object. Its shape is the length of the visual sequence multiplied by the number of objects. The position corresponding to the visual position is 0 (attention is allowed) if it is inside the corresponding bounding box, and the other positions are negative infinity (attention is blocked).

[0127] Local attention output refers to the output features obtained by cross-attention calculation after taking the query vector, object key vector, and object value vector as inputs and applying a dynamic layout mask.

[0128] Global attention output refers to the output feature obtained by cross-attention calculation without applying any mask, using query vector, scene key vector, and scene value vector as input.

[0129] One possible implementation is to load the weights of the pre-trained diffusion model during inference and set the model to inference mode (without gradient calculation). Simultaneously, the global scene description is encoded into a textual latent state by the model's text encoder, which remains unchanged throughout the inference process. For each Transformer block in the diffusion model, the following operations are performed: generate a query vector from the current visual latent state; map the object token set using shared key and value projection matrices and perform RMSNorm; map the scene token set using the same key-value projection matrix and perform RMSNorm; compute local attention (applying a dynamic layout mask) to obtain the local branch output; compute global attention (without applying a mask) to obtain the global branch output; add the global branch output to the local branch output scaled by the gating parameters to obtain the aggregated result; and superimpose this aggregated result as a residual onto the original visual latent state to obtain the updated visual latent state. The dynamic layout mask is generated based on the 2D bounding box of each object. Its shape is the length of the visual sequence multiplied by the number of objects. In the mask, the position of the corresponding visual location within the corresponding bounding box is 0 (allowing attention), and other positions are negative infinity (blocking attention). This mask is added in the form of an additive bias before the Softmax of the cross-attention calculation, thereby forcing each visual location to interact only with object tokens located within its bounding box.

[0130] Step 106: As the iterative denoising process is completed, a two-dimensional image corresponding to the scene description data is generated based on the latent variables of the final denoised output.

[0131] As the iterative denoising process completes, a 2D image that conforms to the text description, 3D angle, and 2D bounding box control conditions is generated based on the latent variables of the final denoised output. The latent variables of the final denoised output refer to the clean latent variables obtained after all iterative denoising steps, which no longer contain noise. These latent variables are located in the latent space of the diffusion model, have the same dimension as the latent representation of the original image, and are the direct source for generating the final pixel image.

[0132] A two-dimensional image refers to the final output, i.e., an RGB image with pixel dimensions such as 512×512 or 1024×1024. This image is obtained by mapping the latent variables of the final denoised output to the pixel space through the decoder of a variational autoencoder. Its content should satisfy all control conditions provided by the user (including the two-dimensional bounding box of the object, the three-dimensional angle, and the local object description). The position, scale, and three-dimensional orientation of the objects in this image should be consistent with the condition tuple provided by the user in step 101.

[0133] One possible implementation is to use the updated visual latent state as the update result of the current denoising time step and continue to complete the remaining iterative denoising steps of the diffusion model: if the current step is not the last one, the updated visual latent state is used as the input of the next time step, and the conditional injection and state update in step 104 are repeated to obtain the latent variables of subsequent time steps until the final clean image latent variables are obtained; then the clean image latent variables are input into the decoder of the pre-trained variational autoencoder to map from the latent space to the pixel space to obtain the RGB image. If the updated visual latent state is already the final clean latent variable (e.g., using a one-step generative model or having reached the last time step), the final image is directly generated by decoding through the variational autoencoder without further iterations. After generating the final image, optional normalization, dequantization, or post-processing operations can be performed on the image before output.

[0134] In summary, this embodiment of the present disclosure achieves precise control over the spatial position and three-dimensional orientation (horizontal angle, pitch angle, rotation angle) of each object in a multi-object scene by explicitly providing a two-dimensional bounding box and three-dimensional angle for each object and encoding and fusing these multimodal information into object tokens. Simultaneously, it completely eliminates the need for dense 3D mapping maps such as depth maps or normalized object coordinate space maps, thus avoiding complex 3D rendering preprocessing and significantly reducing computational overhead and user operation barriers. Furthermore, this method performs self-attention processing on the object token set to generate scene tokens, enabling each object's token to perceive the geometric and semantic information of all other objects. This provides a global physical context for the diffusion model while enhancing the structural rationality and overall consistency between objects in the generated image. In addition, this method uses the global scene description, object tokens, and scene tokens as conditions to perform parallel cross-attention interactions with the intermediate hidden states in the visual flow of the diffusion model. The interaction results are injected only into the visual hidden states without affecting the text flow of the pre-trained model, thus seamlessly adapting to modern MM-DiT architectures and protecting the integrity of the original text semantic representation. Users only need to provide local descriptions, bounding boxes, 3D angles, and global scene descriptions for each object to generate end-to-end 3D scene images that meet the requirements for accurate layout and orientation. This is suitable for practical applications such as e-commerce product display, game scene design, and interior decoration rendering.

[0135] This disclosure proposes a controllable image generation framework for 2.5D multi-object scenes based on a two-stream architecture. The overall system architecture comprises five core components: (1) a conditional coding module; (2) a context-aware two-stream representation module; (3) a modal decoupling two-stream injection module; (4) a layout-aware training strategy; and (5) an OrientLayout dataset and BOV evaluation protocol. These are further elaborated below.

[0136] The basic idea of ​​the conditional coding module is to replace the dense 3D mapping map with a lightweight 2.5D hybrid conditional layout, parameterize each object in the scene into a triplet of local object description, 2D bounding box and 3D angle, and establish a precise space-angle anchor point for each object.

[0137] Let the target scenario be... It consists of 10 objects, denoted as 1. Each object is parameterized as a condition tuple. ,in For local object description of an object, Two-dimensional bounding box coordinates , These are three-dimensional angles (horizontal angle, pitch angle, and rotation angle).

[0138] For each object's local object description, object description embedding is performed first. This results in an independent semantic description for each object. The text is processed by a pre-trained CLIP text encoder, which extracts pooled text embeddings as text embedding vectors. Compared to traditional methods that use category labels, the embodiments of this disclosure encode detailed attribute descriptions (color, material, action, etc.), actively binding fine-grained visual attributes to specific entities, thus mitigating semantic entanglement in advance at the conditional level.

[0139] Next, bounding box encoding is performed. The normalized bounding box coordinates are... By extending to the high-frequency space through Fourier position coding (16 frequency bands), the sensitivity to subtle positional changes is enhanced, resulting in... Subsequently, the bounding box embedding vectors are generated through two layers of MLP, SiLU activation function, and LayerNorm normalization. .

[0140] When performing angle encoding, in order to strictly preserve the inherent periodicity of angles (angles have...) (Periodic cyclic characteristics), three-dimensional angle Mapping to the continuous trigonometric function space, i.e. .

[0141] The trigonometric function mapping is chosen here instead of directly using a 6D rotation representation because the latter's higher degrees of freedom increase the complexity of conditional distribution optimization in the generative framework. The resulting 6D vector is processed through a three-layer MLP, SiLU activation function, and LayerNorm normalization to generate an angle embedding vector. .

[0142] After completing the above three embeddings, the multimodal fusion stage begins. The three sets of embedding vectors are concatenated to obtain a concatenated vector, which is then projected onto a unified object token through the fusion network.

[0143]

[0144] The object tokens of all objects constitute a set. Semantic attributes, spatial location, and 3D orientation information are compressed into a compact, unified representation, which serves as the input for the subsequent two-stream architecture.

[0145] Compared to related technologies, the core advantages of this module are reflected in two aspects. On the one hand, it obtains precise control over space and three-dimensional angles simultaneously with lightweight 2.5D conditions (text + 4D bounding box + 3D angle), avoiding the high computational overhead caused by dense 3D mapping maps and achieving zero-latency conditional encoding. On the other hand, it binds text semantics with space-angle conditions, which alleviates the attribute leakage problem to a certain extent.

[0146] Analysis of problems involved in context-aware dual-flow representation modules: Current "local flow only" schemes do not apply to each object token. Apply bounding box-based spatial mask Its joint layout mask ( (where N is the length of the visual sequence and N is the total number of objects in the scene) is defined as:

[0147]

[0148] That is, when the visual position The bounding box of object i At that time, ,otherwise .

[0149] This mask is applied with an additive bias during cross-attention, forcing the attention weights of the region outside the bounding box to be reset to zero:

[0150]

[0151] in For querying the matrix, The key matrix, For value matrices, The joint layout mask matrix, d is the feature dimension of each attention head, Softmax is the normalized exponential function performed row by row (for all objects at each visual position), and Attn is the output of the cross-attention function.

[0152] This approach can eliminate attribute leakage, but because each entity is forced to be generated in a completely isolated receptive field, the network cannot perceive the geometric and semantic configuration of neighboring objects, resulting in a "cut-and-paste" effect.

[0153] Conversely, if the "global flow only" approach directly uses object tokens for maskless cross-attention, it can maintain scene harmony but inevitably reintroduces attribute leakage.

[0154] To resolve the aforementioned contradictions, this disclosure proposes a dual-stream parallel architecture, simultaneously establishing precise local anchor points and global semantic relationships at the conditional representation level. First, scene token generation (i.e., global stream) is performed. Then, object tokens are... The input is a multi-layer scene self-attention network. Through the self-attention mechanism, each object perceives the semantics and geometric configuration of all other entities, thereby transforming the localized object representation into scene tokens rich in relational information. :

[0155]

[0156] in For the first Scene tokens for individual objects. Scene tokens inherently contain global layout context information, providing a global sense field and promoting natural physical integration between objects.

[0157] Building upon this foundation, a dual-stream parallel condition space is constructed. On one hand, object tokens are retained as local flow representations, precisely recording the binding relationships between the spatial position, scale, and 3D orientation of each object. On the other hand, scene tokens serve as global flow representations, carrying the contextual relationships between objects. These two sets of representations coexist in parallel, forming a complete condition space that drives the local and global flows in the subsequent injection phase, respectively.

[0158] Compared to related technologies, the core advantage of this module lies in its ability to simultaneously establish precise local spatial anchors and global scene semantic relationships at the conditional representation level by transforming "isolated object tokens" into "relationship-aware scene tokens." This enables the subsequent generation process to both eliminate attribute leakage using masked local flows and capture interactions between objects using unmasked global flows. This alleviates attribute leakage while eliminating "cut-and-paste" artifacts and maintaining the overall harmony of the generated image.

[0159] The design concept of the modal decoupling dual-stream injection module draws on the core idea of ​​IP-Adapter, injecting dual-stream representations into the visual stream of the pre-trained MM-DiT backbone network in a modal decoupling manner, thereby strictly protecting the text semantic representation from conditional contamination.

[0160] In each network block of MM-DiT, the visual hidden state With text hidden state Explicit decoupling. Within any network block of the MM-DiT architecture, the visual hidden state... With text hidden state It is computed independently but intertwined through a joint attention mechanism, with the following steps: The visual sequence and text sequence input from the previous layer are respectively generated through linear projection to produce their respective queries (Q), keys (K), and values ​​(V), i.e., the visual side... And text-side Then, the updated visual hidden state of the current layer is output through a joint attention mechanism. With text hidden state This embodiment of the disclosure injects dual-stream condition information only. This avoids any interference with the pre-trained text encoding path.

[0161] Normalized visual query, i.e., the query vector of the visual latent state. The interaction occurs through two parallel, intersecting attention branches and the two-stream representation. The two branches share the key projection matrix. Sum projection matrix (Efficient parameter design) The projected key vectors are normalized by RMSNorm for stable training.

[0162] The first branch is a masked local flow. (For object tokens) Apply dynamic layout mask Strictly constrain the receptive field of each object token. Local flow, i.e., local attention output. The calculation is as follows:

[0163]

[0164] The goal of this branch is to eliminate attribute leakage: each visual location can only interact with object tokens located within its own bounding box, thereby blocking the permeation of visual attributes such as color and material between different objects.

[0165] The second branch is a global stream without a mask. (For scene tokens) Without imposing any spatial constraints, the entire receptive field is fully preserved. Global flow is equivalent to global attention output. The calculation is as follows:

[0166]

[0167] The Global Flow performs the "scene harmony preservation" task to capture the relationships between dense objects and maintain consistency in occlusion, lighting, and perspective.

[0168] Subsequently, the outputs of the two complementary pathways are aggregated through a trainable gating parameter and injected into the visual latent state as residuals. (Gating parameter) Zero initialization is used, and the aggregation method is as follows:

[0169]

[0170] in To control the adjustable parameters of conditional intensity, Zero initialization ensures that the model's behavior is consistent with the original pre-trained backbone in the early stages of training, avoiding gradient explosion in the early stages of training.

[0171] Compared to related technologies, the core advantages of this module are reflected in two aspects. On the one hand, the shared key-value projection matrix design significantly reduces parameter overhead (only 228M trainable parameters, approximately 15.3% of the similar solution SceneDesigner) while ensuring that the local flow and global flow interact in the same feature space. On the other hand, modal decoupling injection (injecting only the visual flow) preserves the semantics of the pre-trained text while achieving precise layout and 3D orientation control.

[0172] The embodiments disclosed herein employ a series of layout-aware strategies during the training phase to improve the controllable generation of multiple objects.

[0173] On one hand, the model employs rectified flow matching for end-to-end training. The forward probability path is hidden in the clean data. Compared with standard Gaussian noise Linear interpolation between:

[0174]

[0175] The network was optimized to predict flow rates. .

[0176] On the other hand, a forward offset time step sampling strategy is used when training the diffusion model. The forward offset time step sampling strategy is used to indicate that the sampling probability density of the time step is offset towards the beginning of the iterative denoising process, so that the diffusion model learns the global spatial layout of the object.

[0177] A diffusion model is a generative model that iteratively denoises noise latent variables into image latent variables. It typically includes a pre-trained network for predicting noise or velocity fields.

[0178] The forward-biased time-step sampling strategy refers to changing the sampling distribution of time steps during the training of the diffusion model, so that its probability density is shifted towards the end of the time step (i.e. the beginning of the denoising process, corresponding to the pure noise state), rather than uniform sampling or the traditional medium time step preference.

[0179] The time step refers to the discrete or continuous variable representing the noise level in the diffusion model. It usually takes values ​​in the range t∈[0,1], where t=0 corresponds to a clean image and t=1 corresponds to pure noise. In the iterative denoising process, the time step gradually decreases from t=1 to t=0.

[0180] The sampling probability density refers to the value of the probability distribution function of a random variable (in this case, time step t) around a certain value, which determines the relative frequency at which each time step is selected during the training process.

[0181] The starting point of the denoising process refers to the stage with the highest noise level in the diffusion model, that is, when the time step t is close to 1 (or t=T, depending on the specific parameterization). At this time, the latent variables of the image are almost pure Gaussian noise, and the macroscopic spatial layout has not yet been formed.

[0182] The global spatial layout of objects refers to the spatial position, size, relative positional relationship between all objects in the scene, and the overall scene structure, rather than local texture or color details.

[0183] Standard T2I generation uses Logit-Normal time step sampling (location parameter) The focus is on the intermediate denoising steps. However, the macroscopic spatial configuration generated by the layout conditions is mainly established in the early stage of high noise, requiring more training attention to the high-noise time steps. This embodiment introduces a positive position offset and sampling... ,set up , And mapped to time steps using the standard Logistic function. Where u is an intermediate random variable that follows a normal distribution, and its role is to serve as an intermediate quantity in the sampling process. After mapping, the time step t is obtained. is the mean parameter of the state distribution. In this embodiment, it is set to m=1.0 to control the center position of u, thereby affecting the overall offset direction of the final time step t. Let be the standard deviation parameter of the distribution. In this embodiment, it is set to . This is used to control the degree of dispersion of u, thereby adjusting the width of the time step sampling distribution. For the standard Logistic function, this function will The value is mapped from the real number field to the interval (0,1) as the value of time step t. The time step is the mapped value, where t→1 corresponds to the beginning of the denoising process (pure noise state), and t→0 corresponds to the clean image state. By setting... , The sampling density shifts towards higher noise levels (t approaches 1). That is, the strategy systematically shifts the time step density towards higher noise levels. The offset forces the model to prioritize global spatial localization during training and optimization, rather than focusing on high-frequency texture details too early.

[0184] In some embodiments, flow matching loss is used when training the diffusion model. When calculating the flow matching loss, a first weight is assigned to the spatial location within the union region of all two-dimensional bounding boxes, and a second weight is assigned to the spatial location within the background region. The first weight is greater than the second weight.

[0185] Flow matching loss is an objective function used to train a diffusion model, which optimizes the model by predicting the velocity field (i.e., the derivative of the linear path from noise to a clean image).

[0186] The union of two-dimensional bounding boxes refers to the spatial merging of the areas covered by the two-dimensional bounding boxes of all objects in the scene. That is, it is the set of pixel or visual token positions of all bounding boxes, and the overlapping part is only calculated once.

[0187] Spatial location refers to a unit on the image feature map, corresponding to a local region in the final image; in the latent space of the diffusion model, it is usually arranged in the form of a two-dimensional grid, with each grid point representing a spatial location.

[0188] The first weight refers to the loss multiplier assigned to each spatial location within the union of the bounding box. Its value is greater than 1 (e.g., 3.0), which is used to increase the model's attention to the prediction error within the object region.

[0189] The second weight refers to the loss multiplier assigned to each spatial location within the background region (non-bounding box union region), which is usually set to 1.0 to indicate normal treatment.

[0190] Unlike the traditional approach of uniformly calculating the mean square error across the entire spatial resolution, this embodiment dynamically increases the penalty weight within a specified bounding box area to guide the learning target to focus more on the foreground object region.

[0191] The objective function for spatially weighted flow matching is defined as:

[0192]

[0193] in, It is a spatially weighted flow matching loss function. The smaller its value, the closer the model's predicted velocity field is to the true velocity field. At the same time, the spatial weights guide the model to pay more attention to the foreground object region. The mathematical expectation symbol represents a latent variable on clean data. ,noise Time step Global scene description and object-level condition set The expectation is calculated from the joint distribution of the variables; during training, this expectation is approximated by randomly sampling these variables. The object-level condition set includes structured multimodal information such as the local description of each object, 2D bounding boxes, and 3D angles, which are processed by 2.5D encoding and a scene self-attention network to obtain object tokens and scene tokens. L is the visual sequence length, i.e., the total number of spatial locations in the image feature map. j is the index of the spatial location, used to traverse each spatial location in the image feature map. The loss weight for the j-th spatial location is obtained through dynamic calculation, and its value depends on whether the location falls within the union of all object bounding boxes. The velocity field is predicted by a neural network with parameter θ, and the input is a noisy latent variable. Time step and conditions and Output and A vector of the same dimension represents the instantaneous direction of the linear path from noise to clean data. For the true velocity field, that is, the derivative of the linear interpolation path from the clean image to the noise.

[0194] Spatial weight vector Calculated dynamically according to the following rules:

[0195]

[0196] in Let jj be the spatial weight vector, and let jj be its j-th component. Each spatial location is assigned an independent loss weight. A binary mask to represent the union region of all bounding boxes. Let be the value of the binary mask at the j-th spatial position. Foreground penalty intensity hyperparameter, satisfying (This embodiment takes the optimal value.) The background region maintains a standard uniform loss, while the loss weight for the foreground region is increased to [a higher percentage]. This multiplier effect prompts the network to impose stronger constraints on the object region.

[0197] During training, different conditional modalities (global text, local object description, all conditions) are randomly discarded to establish an unconditional baseline, supporting classifier-free guidance (CFG) in the inference phase

[18] . This explicit decoupling ensures that the model can independently focus on global semantics, local attributes, and layout geometry, avoiding harmful entanglement between different modalities.

[0198] The following hyperparameter configurations were used during model training. The scene self-attention network was set to 4 layers, with each layer containing 8 attention heads and a hidden dimension of 1280. The model was trained for a total of 60 epochs using distributed training on 7 NVIDIA RTX 5090 GPUs (32GB VRAM each). The input image resolution was uniformly set to 512×512 pixels, and the total batch size was 24. The optimizer used was AdamW. The learning rates for each module were as follows: the learning rate for the conditional encoder and the scene self-attention network was set to... The learning rate of the dual-stream adapter is set to The learning rate of the gating parameter is set to .

[0199] To support the training and evaluation of controllable 2.5D generation, this embodiment constructs the OrientLayout dataset, containing 110,000 high-quality samples. These samples can be sourced from MS-COCO (62,000 images), Objectron (42,000 images), and Cityscapes (6,000 images). Each sample has the following structure: The dataset contains the original image, a global scene description, and a triplet for each object (local object description, 2D bounding box, and 3D angle). Compared to existing datasets, OrientLayout's core advantage lies in providing standardized 2.5D spatial-angle annotations and semantically decoupled object-level descriptions, enabling the same dataset to serve both layout control and orientation control tasks for training and evaluation.

[0200] In some embodiments, a raw image set is first acquired. The raw image set is a collection of multiple raw images, which may originate from different public datasets such as MS-COCO, Objectron, and Cityscapes. Raw images refer to unprocessed natural images or images acquired by a 3D sensor, possibly with original annotations such as 2D bounding boxes or 3D bounding boxes.

[0201] For each original image, different processing flows are performed based on its annotation type. If the image has 3D bounding box annotations, the 3D angles (including horizontal, pitch, and rotation angles) of each object in the original image are calculated from its 3D bounding box. Simultaneously, a pre-trained object detection model (e.g., Grounding DINO) is used to generate corresponding 2D bounding boxes for each object. 3D bounding box annotations refer to the 3D rectangular box annotations provided for objects in an image, typically including the object's position, scale, and orientation information (such as rotation matrices or quaternions), commonly found in datasets like Objectron and Cityscapes. The object detection model is a pre-trained deep learning model capable of detecting objects in an image and outputting their 2D bounding boxes. If the image only has 2D bounding box annotations, a pre-trained orientation prediction model (e.g., OrientAnything) is used to predict the 3D angles of each object in the original image, obtaining the 3D angles of each object. The orientation prediction model is a pre-trained deep learning model capable of predicting the 3D orientation (horizontal, pitch, and rotation angles) of a cropped object image.

[0202] Next, a pre-trained multimodal large language model is used to generate a global scene description and local object descriptions for each object in the original image. A multimodal large language model refers to a large language model capable of processing both image and text inputs simultaneously, such as Qwen2.5-VL or GPT-4V. When generating the global scene description, the complete image is input into the multimodal large language model with the prompt "Please describe this scene in one sentence," resulting in a single-sentence description covering subject, lighting, style, and environmental information. When generating the local descriptions for each object, a dual-image input strategy is employed: two images are simultaneously provided to the multimodal large language model—the first is a complete scene image, where the target object is highlighted with a red bounding box as a visual cue; the second is a cropped view of the object. The prompt template explicitly requires the model to describe only the category attributes (such as color, material, and shape) of the target object within the red box, excluding background and overlapping object interference. To further improve annotation accuracy in dense scenes, automatic overlap analysis is performed before constructing the prompt for each object: the pairwise overlap rate between the target object and all other labeled objects in the image is calculated. When overlapping objects belong to different categories and their overlapping area exceeds 5% of the target area, an interference exclusion instruction is injected into the prompt (e.g., "ignore overlapping chairs and cups"). When objects belonging to the same category overlap, a disambiguation instruction is inserted (e.g., "There are multiple dogs in the scene; only describe the one within the red box"). All generated descriptions also undergo an automated post-processing pipeline to remove redundant prefixes common in multimodal large language models (e.g., "The object in the image is..."), red box illusions (e.g., "...is surrounded by a red box"), and redundant punctuation, ensuring that the final output is concise, attribute-centric descriptive text. This semantically decoupled annotation strategy ensures that local object descriptions can serve as accurate attribute anchors, effectively avoiding attribute entanglement during training caused by ambiguous expressions in the descriptive text at the data level.

[0203] Finally, each original image, its corresponding global scene description, and the local object descriptions, 2D bounding boxes, and 3D angles of each object in that original image are organized into a structured sample. Multiple such structured samples constitute the dataset. This dataset can be used for model training, validation, and testing.

[0204] During the construction of the OrientLayout dataset, two complementary data annotation pipelines are used to adapt to the annotation characteristics of data from different sources.

[0205] For the Objectron and Cityscapes pipelines: Both datasets provide native 3D bounding box annotations, from which our method directly calculates 3D angles (horizontal, pitch, and rotation angles). Since the 3D bounding box projections in these datasets do not accurately reflect the visual boundaries of objects in the image, Grounding DINO is used to re-detect 2D bounding boxes. Objects with a detection confidence score below 0.5 or whose bounding box area is less than 1% of the total image area are discarded.

[0206] For the COCO pipeline: COCO provides accurate 2D bounding boxes but lacks 3D orientation annotations. Each image is first center-cropped and scaled to 512×512 resolution. During preprocessing, object categories with ambiguous "standard frontal" definitions or rotational symmetry (such as isotropic objects like spheres and bottles) are excluded, while excessively dense scenes with more than 20 annotated objects are discarded. For each retained object, its bounding box area is first verified to be at least 10% of the total image area. Then, the object is extracted using a COCO segmentation mask, padded to a square shape, and input into a pre-trained orientation prediction model, OrientAnything, for 3D angle prediction. Prediction results with a confidence level below 0.8 are discarded. Finally, human annotators verify each false annotation result, filtering out incorrect orientation annotations and severely occluded objects to ensure the reliability of the final ground truth annotations.

[0207] Inherent conflicts in coordinate definitions exist between different data sources, requiring specialized standardization. A typical problem lies in the differing definitions of the "standard front" across object categories in the Objectron dataset. For example, Objectron's native pose defines the standard front of a cup as the sides of the handle; for a book, Objectron defines the front as pointing towards the spine and the cover, respectively. This category-specific coordinate convention means that the same horizontal angle value carries completely different visual semantics across different object categories, severely hindering the learning of a unified orientation representation. To address this issue, this method uses the output space of OrientAnything as a unified reference. By running OrientAnything on clipped object instances of each category in Objectron, the standard orientation definition of the category's coordinate system is determined. This allows for the correction of instances whose poses deviate from the standard, transforming objects of the same category into a consistent coordinate space. Objects that cannot be reliably aligned are excluded to ensure complete consistency in the horizontal angle semantics across all object categories in the final OrientLayout dataset.

[0208] In terms of relation-aware semantic annotation, both global scene descriptions and local object descriptions are automatically generated by the multimodal large language model Qwen2.5-VL. Global scene descriptions are obtained by showing the model a complete image and requesting it to generate a single-sentence description, focusing on subject, lighting, style, and environmental information. For object-level descriptions, the aforementioned dual-image input strategy, along with automatic overlap analysis and injection of exclusion instructions or disambiguation statements, is employed. All generated descriptions then undergo an automated post-processing pipeline to remove redundant prefixes, red-box illusions, and extraneous punctuation, ensuring concise core attribute descriptions in the output. This semantically decoupled annotation strategy ensures that local object descriptions can serve as accurate attribute anchors, effectively avoiding attribute entanglement during training caused by ambiguous expressions in the descriptive text at the data level.

[0209] In some embodiments, to address the shortcomings of related art in evaluation methods, this disclosure proposes a BOV (Browser-on-Video) approach. Existing automated evaluation metrics (e.g.) Applying an orientation predictor directly to the generated image often yields inconsistent results. Traditional manual evaluation requires judges to categorize object orientation into one of eight directions, which can lead to significant subjective bias at angular boundaries (e.g., an object at 20° is positioned between "front" and "right front"). Therefore, this protocol uses continuous true horizontal angles... Deterministically mapped to the nearest standard direction label The generated image is then shown to the evaluator, who is asked only a simple yes / no question: "Is this [object category] oriented towards the [target direction]?" To eliminate interference from individual anomalous responses and ensure the objectivity of the evaluation, this method introduces a majority voting mechanism. Optionally, for each test object, recruitment... Independent labelers (e.g.) ) to conduct an evaluation. Let the first The first object The answer from the annotator was The value is represented by 1, where 1 represents "Yes" and 0 represents "No". The object... Final verification results The decision is made by majority vote. The majority vote is taken when the number of annotators giving a positive answer is greater than or equal to a threshold (e.g., more than half). When the object's name is specified, the generation direction verification is passed.

[0210]

[0211] in, This is an indicator function; it is 1 if the condition is true, and 0 otherwise.

[0212] BOV accuracy Defined as the proportion of the number of objects that pass verification out of the total number of tested objects:

[0213]

[0214] in, This represents the total number of objects participating in the evaluation.

[0215] The technical solutions of the embodiments disclosed herein bring significant beneficial effects in the following aspects:

[0216] In terms of technical performance, orientation control accuracy is significantly improved. Illustratively, the PoseAdapter using this solution achieves an automated orientation accuracy of 87.02% and a manually verified accuracy of 95.44% on a single-object frontal subset facing the layout, respectively exceeding the closest baseline method, SceneDesigner, by 13.34 and 8.77 percentage points. In multi-object scenes, the BOV accuracy of this method reaches 93.22%, while the SceneDesigner only achieves 80.00%.

[0217] The spatial layout accuracy is comprehensively superior. In multi-object scenes, the spatial accuracy (spatial accuracy with an intersection-union ratio greater than 0.6) of this method reaches 91.67%, which is nearly twice as high as the scene designer's 47.48%; the average intersection-union ratio reaches 81.72%, while the scene designer's is only 46.38%.

[0218] The decoupling capability of multi-object attributes is significantly enhanced. The object-level CLIP score of this method reaches 0.310, which is better than the scene designer's 0.288. This quantitatively confirms that the two-stream architecture can effectively prevent attribute leakage in complex scenes.

[0219] The generated images have the best overall quality. The method achieves an FID score of 24.60, which is better than the Scene Designer's 25.10 and LooseControl's 36.78.

[0220] User research further validated the superiority of human perception. In a blind subjective evaluation experiment, the posture adapter achieved pass rates of 95.0%, 87.5%, and 95.5% in text consistency, orientation accuracy, and spatial layout in multi-object scenes, respectively, while the scene designer only achieved 60.2%, 45.5%, and 32.0%.

[0221] At the application level, this solution significantly lowers the barrier to entry. Users only need to provide text descriptions, bounding box coordinates, and angle values; no expertise in 3D modeling or rendering software is required, and conditional preprocessing takes zero time (compared to 13-14 seconds for rendering engine-based solutions). In terms of parameter efficiency, only 228M trainable parameters are available, approximately 15.3% of SceneDesigner's, allowing for the service of more instances with the same hardware resources. Furthermore, the modal decoupling injection strategy seamlessly integrates with modern multimodal diffusion transformers (MM-DiT) and other diffusion backbone architectures, exhibiting excellent forward compatibility. Through the decoupling modal discarding training strategy, users can independently adjust the trade-off between text fidelity and layout compliance during inference, flexibly adapting to the needs of different application scenarios.

[0222] In addition to the main implementation methods described above, this disclosure also envisions several feasible alternatives.

[0223] An alternative is to replace the currently used 2.5D conditions (i.e., 2D bounding box plus 3D angles) with a complete 3D bounding box (8 vertex coordinates) or a 9-DOF 3D pose (including 3D position, 3D dimensions, and rotation matrix) as conditional input. This approach provides more complete geometric information; however, its implementation relies on 3D labeled data, resulting in a higher barrier to user interaction. Furthermore, the significant increase in encoding dimensionality leads to increased complexity in the fusion network. Therefore, for applications that do not require precise 3D position accuracy and only need to control visual layout and orientation, the 2.5D approach proposed in this disclosure is more lightweight and practical.

[0224] Another alternative is to decompose the multi-object generation problem into N independent single-object generation subtasks, and use a cross-frame attention mechanism to share global context information among the subtasks, gradually synthesizing the multi-object scene during multi-frame diffusion. This approach can naturally avoid attribute leakage, but its inference computation increases linearly with the number of objects N, and it is difficult to effectively model the occlusion and depth relationships between objects, resulting in relatively weak guarantees of global harmony.

[0225] Another alternative is to replace the single fused object token with a hierarchical condition injection structure: injecting conditions of different granularities at different levels of the diffusion model, such as injecting global layout conditions at shallow levels and fine-grained attribute conditions at deeper levels. Theoretically, this approach can better match the semantic granularity of different levels of the diffusion model; however, the design of the hierarchical condition allocation strategy is more complex, and the difficulty of controlling training stability increases accordingly. This disclosure does not limit this approach.

[0226] Figure 2 A flowchart illustrating the overall system architecture provided by an exemplary embodiment of this disclosure is shown.

[0227] like Figure 2 As shown on the left, the conditional encoding module first performs 2.5D conditional encoding. For each object in the scene, its local object description is obtained. 2D bounding box and three-dimensional angle Local object description The text is fed into a text encoder (using a CLIP pre-trained model) to extract semantic embeddings; two-dimensional bounding boxes are also used. After Fourier position encoding, the data is fed into a bounding box encoder (implemented by an MLP) to obtain the bounding box embedding; 3D angle First, trigonometric function mapping is performed, then the results are fed into an angle encoder (implemented by an MLP) to obtain the angle embedding. The three embedding vectors are then concatenated and non-linearly projected into a fusion MLP to generate an object token for each object. The object tokens of all objects constitute a set. .

[0228] like Figure 2 As shown in the middle section, the context-aware dual-stream representation module receives a set of object tokens. .on the one hand, It is directly preserved as a local flow representation; on the other hand... The data is fed into a scene self-attention network, where self-attention processing enables each object token to perceive information from all other objects, outputting a set of scene tokens. This serves as a representation of the global flow. This leads to the formation of a parallel two-flow condition space: local flow... Global flow carries precise local space-angle bindings for each object. It contains the global physical context between objects.

[0229] like Figure 2 As shown on the right, the modal decoupling dual-stream injection module performs modal decoupling dual-stream injection in each block of the pre-trained MM-DiT module. The MM-DiT block receives the textual hidden state. (From global scene description) and visual hidden states (i.e., intermediate latent variables in the visual flow). From the visual latent state Generate a query vector Q and interact with the two-stream representations respectively: for the local stream, for the object token. Apply a dynamic layout mask (generated from the bounding box), perform masked cross-attention calculation, and obtain the local output; for the global stream, apply scene tokens. Without applying a mask, unmasked cross-attention computation is performed to obtain the global output. The two attention branches share the key-value projection matrix (K, V). Subsequently, the local and global outputs are aggregated using trainable gating parameters and injected into the visual hidden state as residuals. In the process, the updated visual latent state continues to participate in the subsequent denoising process. The figure clearly shows the complete information flow of conditional coding, two-stream representation generation, and conditional injection.

[0230] Figure 3 This illustration demonstrates the controllable generation effects provided by an exemplary embodiment of this disclosure in different scenarios. Example images of single-object scenarios include "a black horse walking on snow," "a wooden chair in a bright, minimalist room," "a motorcycle on a neon-lit rainy night street," and "a medieval knight in a misty ancient ruin." Each set of images demonstrates the method's precise control over the 3D orientation and spatial layout of a single object by changing the object's horizontal angle, pitch angle, rotation angle, and bounding box position. Example images of multi-object scenarios include "a bear wearing goggles and a jacket riding a red motorcycle," "a bear wearing goggles and a jacket standing next to a red motorcycle," and "a yellow bus, a green jeep, and a black sedan." These images reflect the strict attribute decoupling effect of this method in multi-object generation—the visual attributes of different objects, such as color, material, and identity, do not permeate each other, while the occlusion relationships and spatial layout between objects remain natural and harmonious. In summary, Figure 3 The advantages of this method in terms of fine control of a single object, isolation of attributes of multiple objects, and scene consistency are intuitively verified.

[0231] Figure 4 This paper presents a qualitative comparison of the method provided in this disclosure with existing methods (such as LooseControl and SceneDesigner) in controllable generation tasks. The generation results of LooseControl, SceneDesigner, and the proposed method are compared in single-object and multi-object scenes, along with visualizations of the conditional inputs for each method (2.5D annotation for this method, and CNOCS mapping maps or depth maps for the comparison methods). It can be seen that the proposed method outperforms the comparison methods in object orientation control, attribute decoupling, and scene harmony.

[0232] Figure 5This illustration shows a typical labeled sample of the OrientLayout dataset provided in an exemplary embodiment of this disclosure. The original image is on the left, and the complete labeled information of the sample is listed on the right in structured text form. Each sample contains a global scene description (full_caption) that summarizes the main content, lighting, style, and atmosphere of the entire image. In addition, each object in the scene is labeled with: a two-dimensional bounding box (visualized as a green box in the image), three-dimensional angles (including horizontal angle, pitch angle, and rotation angle), and a corresponding local object description. The example in the figure contains two objects: the first object is a dark blue car with silver rims, with bounding box coordinates of 3,0,507,375, and three-dimensional angles of 108.0° horizontal angle, 5.0° pitch angle, and 1.0° rotation angle; the second object is a woman wearing a floral dress and sunglasses sitting on a stone platform, with bounding box coordinates of 103,21,407,512, and three-dimensional angles of 354.0° horizontal angle, 5.0° pitch angle, and 2.0° rotation angle. The figure visually demonstrates that the OrientLayout dataset provides global scene semantics, fine-grained object-level descriptions, precise 2D spatial locations, and 3D orientation annotations, enabling it to provide standardized multimodal data support for the training and evaluation of 2.5D controllable generative models.

[0233] Figure 6 Qualitative comparison results of ablation experiments provided by an exemplary embodiment of this disclosure are shown to verify the respective roles of local and global flows in a two-stream architecture and the necessity of their synergy. The figure shows generated images under three configurations: using only global flow (no masked cross-attention), using only local flow (applying dynamic layout masks), and the complete two-stream method (local flow + global flow + gated aggregation). In the generated results using only global flow, the colors and materials of different objects intermingle (e.g., the colors of a pink flamingo and a white rooster are confused), reflecting an attribute leakage problem. In the generated results using only local flow, there is a lack of natural occlusion and lighting coordination between objects, presenting "cut-and-paste" artifacts. In the generated results of the complete two-stream method provided by this disclosure, object attributes remain independent, and the occlusion and lighting relationships are natural, indicating that the two-stream architecture can simultaneously eliminate attribute leakage and cut-and-paste artifacts. This demonstrates that by using masked local streams (eliminating attribute leakage) and unmasked global streams (maintaining global harmony) in parallel, and through gated aggregation, the method provided in this disclosure successfully resolves the fundamental contradiction in existing solutions where "strict isolation" and "global harmony" cannot be achieved simultaneously.

[0234] In summary, this disclosure provides a method and system for generating multi-object layout and orientation controllable images based on a dual-stream architecture. This method aims to address the challenge of simultaneously achieving precise layout and orientation control, as well as attribute leakage elimination, in multi-object scene generation. First, 2.5D hybrid condition acquisition and encoding are performed: multimodal conditional inputs for each object in the target scene are acquired, including at least semantic descriptions, 2D bounding boxes, and 3D angles; cross-modal feature encoding and mapping are performed on the multimodal conditions, fusing them to generate a unified object feature representation containing local spatial and angular binding relationships, i.e., a local object token. Second, a context-aware dual-stream representation is constructed: the unified object feature representation is input to a relationship-aware module (e.g., a scene self-attention network), enabling each object feature to perceive each other and extract global geometric and semantic relationships, generating a scene feature representation containing global physical context, i.e., a global scene token; local object features and global scene features together constitute a parallel dual-stream conditional space. Then, modal decoupling and parallel injection of the two streams are performed: during the iterative denoising process of the diffusion model, the text stream and the visual stream are explicitly decoupled, and the two streams are conditionally injected into the visual stream in parallel. Specifically, a dynamic spatial mask is applied to local object features to form a local stream, constraining the receptive field of features and eliminating multi-object attribute leakage; simultaneously, no mask is applied to global scene features to form a global stream, preserving the global receptive field and maintaining global scene harmony. Finally, image generation: the output features of the local and global streams are aggregated through a gating mechanism to guide the diffusion model in generating multi-object images that simultaneously possess accurate spatial layout, accurate 3D orientation, and visual attribute physical harmony.

[0235] This disclosure also provides a lightweight 2.5D hybrid conditional coding method. This coding method is based on the joint coding of object text description (CLIP pooling embedding), two-dimensional bounding boxes (Fourier position coding plus MLP), and three-dimensional angles (trigonometric function periodicity preservation plus MLP). It generates a unified object token by fusing MLPs to drive precise spatial and orientation control of the diffusion model, with a focus on protecting the multimodal conditional coding and fusion mechanism.

[0236] This disclosure also provides a context-aware dual-stream representation mechanism. This mechanism transforms isolated object tokens into relation-aware scene tokens using a scene self-attention network, and uses two sets of complementary tokens as a dual-stream representation architecture with local (masked) and global (unmasked) conditions, respectively. This mechanism, for the first time, simultaneously addresses the two conflicting goals of "attribute leakage elimination" (precise boundary constraints) and "scene harmony preservation" (inter-object relationship modeling) at the conditional representation level.

[0237] This disclosure also provides a modal decoupling parallel dual-stream injection strategy. This strategy implements parallel dual-path attention injection with control conditions in the MM-DiT visual stream, specifically including: a modal decoupling design that injects only the visual stream without interfering with the text stream; an efficient parameter design that shares the key-value projection matrix between the local and global streams; and a mechanism for aggregation using a trainable gating parameter γ.

[0238] This disclosure also provides a combination of layout-aware training strategies. This combination includes two training strategies specifically designed for controllable layout generation: first, positively offset Logit-Normal time-step sampling (mean 1.0, standard deviation 1.0), which prioritizes the establishment of global spatial structure in the early, noisy stages of the model; and second, dynamically spatially weighted bounding box mask loss (foreground region weight set to 3.0), which enhances the accurate localization learning of object regions. The two strategies work synergistically to produce a significant gain effect.

[0239] This disclosure also provides a method for constructing a standardized 2.5D dataset from multiple data sources. This method integrates data from three different sources and annotation systems: COCO, Objectron, and Cityscapes, to systematically construct 2.5D annotations. Specifically, it includes: a pseudo-annotation and manual verification pipeline based on OrientAnything; an Objectron cross-category coordinate system standardization alignment algorithm; a relation-aware semantic description generation method based on dual-image input prompts; and an automatic overlap analysis and interference elimination prompt mechanism for densely occluded scenes.

[0240] Compared with related technologies, the embodiments of this disclosure achieve an innovative combination of dual-stream architecture and 2.5D conditional coding. Related technologies typically optimize only one of the following: attribute decoupling (single-stream scheme) or orientation control (dense conditions based on rendering); the embodiments of this disclosure solve both problems simultaneously within a unified framework, and do not rely on a 3D rendering engine, thus possessing unique engineering value.

[0241] The methods provided in this disclosure cover a variety of application scenarios, such as e-commerce and advertising image generation (precisely specifying product orientation and position to generate multi-view display images in batches), interior design visualization (users specify furniture layout and orientation to automatically render interior renderings), game asset generation (automatically generating characters and props with specific orientations based on scene layout), robot simulation data generation (batch generating object images with specified orientations for robot perception training), and movie and animation pre-visualization (automatically generating scene concept maps based on object layouts in storyboards).

[0242] In terms of experimental verification, the embodiments of this disclosure have completed systematic experimental verification on the OrientLayout evaluation benchmark, including quantitative comparison experiments (compared with LooseControl and SceneDesigner), ablation experiments (verifying the effectiveness of each component of the dual-stream architecture and training strategy), hyperparameter sensitivity analysis (scanning the foreground weight α value), efficiency comparison analysis (parameter quantity and inference latency), and a 20-person blind test subjective evaluation experiment, which comprehensively verified the technical effectiveness and practicality of the embodiments of this disclosure.

[0243] The following are device embodiments of the present disclosure. For parts not described in detail in the device embodiments, please refer to the technical details disclosed in the above method embodiments.

[0244] Please refer to Figure 7 This illustration shows a schematic diagram of a scene image generation apparatus with three-dimensional spatial and orientation control, provided by an exemplary embodiment of the present disclosure. The apparatus can be implemented, in whole or in part, as a computing device through software, hardware, or a combination of both. The apparatus includes: an acquisition module 710, a conditional encoding module 720, a context-aware two-stream representation module 730, and an image generation module 740, wherein the image generation module 740 includes a modal decoupling two-stream injection module.

[0245] The acquisition module 710 is used to acquire scene description data of the three-dimensional scene. The scene description data includes a global scene description and multimodal information of each object in the three-dimensional scene. The multimodal information includes local object description, two-dimensional bounding box and three-dimensional angle.

[0246] The conditional encoding module 720 is used to perform feature encoding and mapping on the multimodal information of each object and fuse them to generate an object token set. The object token set includes the object token of each object. The object token is used to represent the object features containing local spatial and angular binding relationships.

[0247] The context-aware dual-stream representation module 730 is used to perform self-attention processing on the object token set to generate a scene token set. The scene token set includes the scene token of each object, and the scene token is used to represent scene features containing the global physical context.

[0248] The image generation module 740 is used to input the global scene description, object token set, and scene token set into the pre-trained diffusion model. During the iterative denoising process of the diffusion model, the modal decoupling dual-stream injection module performs parallel cross-attention interactions with the object token set and the scene token set, respectively, and injects the interaction results into the visual hidden state to obtain the updated visual hidden state. The visual hidden state is an intermediate hidden variable representing image features in the visual stream of the diffusion model. As the iterative denoising process is completed, a two-dimensional image corresponding to the scene description data is generated based on the hidden variables of the final denoised output.

[0249] In one possible implementation, a dynamic layout mask is applied to the cross-attention between the visual latent state and the set of object tokens, while no mask is applied to the cross-attention between the visual latent state and the set of scene tokens. The dynamic layout mask is an attention mask matrix generated based on the two-dimensional bounding box of each object, used to constrain the receptive field of the cross-attention.

[0250] In another possible implementation, the conditional coding module 720 is also used for:

[0251] For the multimodal information of each object, the local object description is encoded as a text embedding vector, the two-dimensional bounding box is encoded as a bounding box embedding vector, and the three-dimensional angle is encoded as an angle embedding vector.

[0252] The text embedding vector, bounding box embedding vector, and angle embedding vector are concatenated to obtain the concatenated vector.

[0253] The concatenated vector is input into the MLP for projection to generate object tokens.

[0254] In another possible implementation, the image generation module 740 is also used for:

[0255] In the iterative denoising process of the diffusion model, the object token set is projected onto the key space and value space to obtain the object key vector and object value vector; and the scene token set is projected onto the key space and value space to obtain the scene key vector and scene value vector.

[0256] The local attention output is calculated based on the query vector, object key vector, and object value vector of the visual latent state, as well as the dynamic layout mask.

[0257] Calculate the global attention output based on the query vector, scene key vector, and scene value vector;

[0258] The interaction result is obtained by weighted summing of the global attention output and the local attention output scaled by the gating parameters.

[0259] The interaction results are superimposed onto the visual latent state as residuals to obtain the updated visual latent state.

[0260] In another possible implementation, a forward-biased time-step sampling strategy is used when training the diffusion model. This strategy is used to indicate that the sampling probability density of the time step is shifted towards the beginning of the iterative denoising process, so that the diffusion model learns the global spatial layout of the object.

[0261] In another possible implementation, a flow matching loss is used when training the diffusion model. When calculating the flow matching loss, a first weight is assigned to the spatial location within the union region of all two-dimensional bounding boxes, and a second weight is assigned to the spatial location within the background region. The first weight is greater than the second weight.

[0262] In another possible implementation, the device also includes: a dataset building module, used for:

[0263] Obtain the original image set, which includes multiple original images;

[0264] For each original image, if the original image only has 3D bounding box annotations, the 3D angles of the objects are calculated from the 3D bounding boxes of each object in the original image, and a 2D bounding box of each object is generated using a pre-trained object detection model.

[0265] If the original image only has two-dimensional bounding box annotations, then the pre-trained orientation prediction model is used to predict the three-dimensional angle of each object in the original image to obtain the three-dimensional angle of each object.

[0266] Generate global scene descriptions and local object descriptions for each object in the original image using a pre-trained multimodal large language model.

[0267] Each original image, the corresponding global scene description, the local object description of each object in the original image, the 2D bounding box, and the 3D angle are organized into structured samples to form a dataset.

[0268] It should be noted that the above embodiments only illustrate the division of the above functional modules when implementing the device. In actual applications, the above functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0269] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0270] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0271] This disclosure also provides a scene image generation apparatus with three-dimensional space and orientation control, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0272] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0273] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0274] Figure 8 This is a block diagram illustrating an apparatus 1900 according to an exemplary embodiment. For example, apparatus 1900 may be provided as a server or terminal device. (Refer to...) Figure 8 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0275] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0276] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0277] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0278] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0279] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0280] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0281] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0282] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0283] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0284] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating scene images with three-dimensional spatial and orientation control, characterized in that, The method includes: Acquire scene description data of a 3D scene, the scene description data including a global scene description and multimodal information of each object in the 3D scene, the multimodal information including local object description, 2D bounding box and 3D angle; The multimodal information of each object is feature-encoded and mapped, and fused to generate an object token set. The object token set includes the object token of each object, and the object token is used to represent object features containing local spatial and angular binding relationships. The object token set is subjected to self-attention processing to generate a scene token set, which includes the scene token of each object. The scene token is used to characterize scene features that contain global physical context. The global scene description, the object token set, and the scene token set are input into the pre-trained diffusion model; In the iterative denoising process of the diffusion model, the visual latent state is subjected to parallel cross-attention interaction with the object token set and the scene token set, respectively, and the interaction result is injected into the visual latent state to obtain the updated visual latent state. The visual latent state is an intermediate latent variable representing image features in the visual flow of the diffusion model. As the iterative denoising process is completed, a two-dimensional image corresponding to the scene description data is generated based on the latent variables of the final denoised output.

2. The method according to claim 1, characterized in that, A dynamic layout mask is applied to the cross-attention between the visual hidden state and the set of object tokens, while no mask is applied to the cross-attention between the visual hidden state and the set of scene tokens. The dynamic layout mask is an attention mask matrix generated based on the two-dimensional bounding box of each object and used to constrain the receptive field of the cross-attention.

3. The method according to claim 1, characterized in that, The step of feature encoding and mapping of the multimodal information of each object, and fusing them to generate an object token set, includes: For the multimodal information of each object, the local object description is encoded as a text embedding vector, the two-dimensional bounding box is encoded as a bounding box embedding vector, and the three-dimensional angle is encoded as an angle embedding vector; The text embedding vector, the bounding box embedding vector, and the angle embedding vector are concatenated to obtain a concatenated vector. The spliced ​​vector is input into a multilayer perceptron (MLP) for projection to generate the object token.

4. The method according to claim 1, characterized in that, In the iterative denoising process of the diffusion model, the visual latent state is subjected to parallel cross-attention interaction with the object token set and the scene token set, respectively, and the interaction result is injected into the visual latent state to obtain the updated visual latent state, including: In the iterative denoising process of the diffusion model, the object token set is projected onto the key space and value space to obtain the object key vector and object value vector; and the scene token set is projected onto the key space and value space to obtain the scene key vector and scene value vector. Based on the query vector of the visual latent state, the object key vector, the object value vector, and the dynamic layout mask, calculate the local attention output; Calculate the global attention output based on the query vector, the scene key vector, and the scene value vector; The interaction result is obtained by weighted summing of the global attention output and the local attention output scaled by the gating parameters. The interaction result is superimposed onto the visual latent state as a residual to obtain the updated visual latent state.

5. The method according to any one of claims 1 to 4, characterized in that, The diffusion model is trained using a forward offset time step sampling strategy, which is used to indicate that the sampling probability density of the time step is offset towards the beginning of the iterative denoising process, so that the diffusion model learns the global spatial layout of the object.

6. The method according to any one of claims 1 to 4, characterized in that, The diffusion model is trained using flow matching loss. When calculating the flow matching loss, a first weight is assigned to the spatial position within the union region of all two-dimensional bounding boxes, and a second weight is assigned to the spatial position within the background region. The first weight is greater than the second weight.

7. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Obtain the original image set, which includes multiple original images; For each of the original images, if the original image only has 3D bounding box annotations, then the 3D angle of the object is calculated from the 3D bounding box of each object in the original image, and a 2D bounding box of each object is generated using a pre-trained object detection model. If the original image only has two-dimensional bounding box annotations, then a pre-trained orientation prediction model is used to predict the three-dimensional angle of each object in the original image to obtain the three-dimensional angle of each object. The pre-trained multimodal large language model is used to generate a global scene description of the original image and a local object description of each object. Each original image, the global scene description corresponding to the original image, the local object description of each object in the original image, the two-dimensional bounding box, and the three-dimensional angle are organized into structured samples to form a dataset.

8. A scene image generation device with three-dimensional spatial and orientation control, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images