A complex scene image generation system and method

Abstract

The application relates to a complex scene image generation system and method, which is a complex scene image generation system and method based on text representation scene graph decoupling and combination, so as to improve the modeling capability of object attributes and object relationships in a complex scene, and the controllability and generalization of a diffusion model in complex image generation are strengthened by decoupling and then combining the spatial relationship and interaction relationship of the scene graph representation.

Description

Technical Field

[0001] This invention relates to the field of image generation technology, and more specifically, to a system and method for generating images of complex scenes. Background Technology

[0002] Controlled image generation is a fundamental task in computer vision, aiming to generate images that conform to a description based on various conditions provided by the user (such as text, layout, semantic segmentation graphs, and scene graphs). Because scene graphs can more concretely represent the object attributes and relationships in complex scenes compared to other conditions, the task of generating complex scene images based on scene graphs shows broad application prospects in fields such as embodied intelligence, augmented / virtual reality, and artistic creation.

[0003] Because the relationships between objects in complex scenes involve both spatial and interactive relationships between multiple objects, linear text decoupling is insufficient to describe these relationships. Furthermore, existing text-guided diffusion models struggle to accurately generate the number of objects in a scene. While layout-guided diffusion models outperform text-guided methods in spatial and numerical modeling, they still fall short in object interaction modeling, tending to generate each object independently.

[0004] To fully model the relationships and number of objects in a scene, a scene graph-guided diffusion model is proposed for image generation in complex scenes. In this framework, nodes in the scene graph represent objects in the scene, and variables in the scene graph represent the spatial and interactive relationships between objects.

[0005] One of the main approaches of existing scene graph-guided diffusion models is to first convert the scene graph into a layout using a layout predictor, and then use a layout-guided diffusion model to generate scene images. However, this method has shortcomings in modeling object interaction relationships. Another approach based on semantic embedding is to encode the scene graph into node embeddings using a graph convolutional neural network as a guiding condition for the diffusion model. Although this approach models the interactions between objects in the scene, it is difficult to handle independent nodes in the scene graph, specifically exhibiting problems such as missing independent nodes and unreasonable positions. Summary of the Invention

[0006] The technical problem to be solved by the present invention is how to efficiently model object attributes and object relationships in complex scenes. In order to overcome the shortcomings of the prior art, the present invention provides a complex scene image generation system and a complex scene image generation method.

[0007] This invention provides a complex scene image generation system, comprising:

[0008] Visual-Language Module: Based on the objects and relationships between them in the scene graph, the module calls the visual-language model it contains to perform node embedding and edge embedding on the scene graph to obtain a text representation scene graph;

[0009] Layout-Semantic Variational Autoencoder Module: Decouples the spatial and interactive relationships of objects in the text representation scene graph into object-level layout embedding results and semantic embedding results through joint encoding and layout-semantic decoding;

[0010] Object-level fusion tagging module: Obtains object embedding results by combining the semantic embedding results with the layout embedding results through concatenation and empty embedding methods;

[0011] Combined masking attention mechanism module: Diffusion combination and volume attribute information integration of the object embedding result to obtain the volume attribute information corresponding to the scene graph;

[0012] Multi-layer sampling module: Extracts several embedded objects as samples from the layout embedding result and the semantic embedding result using object-level Gaussian noise sampling method, and uses the non-overlapping mask converted from the layout embedding result to perform noise estimation on the samples to obtain the hierarchical representation corresponding to the scene graph;

[0013] in,

[0014] The layout-semantic variational autoencoder module communicates with the vision-language module, the object-level fusion tagging module communicates with the layout-semantic variational autoencoder module, the combined masking attention mechanism module communicates with the object-level fusion tagging module, and the multi-layer sampling module communicates with the layout-semantic variational autoencoder module.

[0015] First, the complex scene image generation system disclosed in this invention generates a structured text-representation scene graph by setting a visual-language module, which helps to generate complex images by representing the scene through the text-representation scene graph. By setting a layout-semantic variational autoencoder module for decoupling scene space and interaction relationships, a reasonable one-to-many mapping relationship between scene graph and decoupled embedding is realized, overcoming the shortcomings of existing methods in modeling object interaction relationships.

[0016] Secondly, the complex scene image generation system disclosed in this invention, by introducing a combined masking attention mechanism module, can provide the decoupled object-level space and interaction relationship embeddings to the diffusion model for combined generation, thus avoiding the leakage of object-level attribute information. Simultaneously, the introduced multi-layer sampler mechanism module fully utilizes the diverse decoupled embeddings generated by the variational autoencoder, achieving consistent scene image generation under operations on scene graph nodes and attributes. Furthermore, it facilitates the handling of independent nodes in the scene graph, avoiding the problems of missing or improperly positioned independent nodes.

[0017] Finally, the complex scene image generation system disclosed in this invention can effectively achieve controllability in the generation of complex scene images, make full use of structured scene graphs to replace linear text structures to represent complex scenes, and has advantages in modeling the relationships and attributes of complex scenes. This is beneficial for the practical application of diffusion models in fields such as embodied intelligence, augmented / virtual reality, and artistic creation.

[0018] In one possible implementation, the visual-language model is described as follows:

[0019] The set of nodes formed by objects in the scene graph is:

[0020]

[0021] The set of edges formed by the relationships between objects in the scene graph is:

[0022]

[0023] node o i The semantic tags are Side e ij The semantic tags are

[0024] From node o i To node o j The triples of a relational join are:

[0025]

[0026] The structural expression of the visual-language model is:

[0027]

[0028] In the formula,

[0029] This represents a learnable embedding layer for the set of nodes;

[0030] This represents a learnable embedding layer for the edge set;

[0031] E CLIP This indicates a frozen pre-trained text encoder;

[0032] E box This indicates a bounding box coordinate space encoder using a multilayer perceptron;

[0033] b i =(x i ,y i ,w i ,h i) represents node o i The border;

[0034] Indicates a connection operation;

[0035] Indicates that the output node is embedded;

[0036] ε indicates output edge embedding;

[0037] The visual-language model used in this scheme can fully utilize the inherent separation semantics in language text, while promoting the alignment between images and scene graphs.

[0038] In one possible implementation, the layout-semantic variational autoencoder module includes:

[0039] Joint encoder: Obtains a layout-semantic joint image based on the text representation scene graph through convolution operations and KL divergence solving;

[0040] Layout decoder: Extracts the explicit spatial layout from the layout-semantic joint image to obtain the layout embedding result;

[0041] Semantic decoder: Extracts implicit interaction semantics from the layout-semantic joint image to obtain the semantic embedding result;

[0042] in,

[0043] The joint encoder communicates with the vision-language module, the layout decoder, and the semantic decoder respectively, and the layout decoder and the semantic decoder both communicate with the object-level fusion tagging module;

[0044] This scheme, through layout encoding and decoding modes, can decouple the spatial and interactive relationships in the text representation scene graph into object-level layout embedding and semantic embedding.

[0045] In one possible implementation, the joint encoder is configured to obtain the layout-semantic joint image by performing the following steps;

[0046] A1: Perform multiple graph convolution operations on the text representation scene graph to obtain the convolution operation image;

[0047] A2: A random vector is obtained by sampling from the parameterized Gaussian space of the nodes in the convolution operation image;

[0048] A3: Replace the bounding boxes in the convolution operation image with the random vector to obtain the bounding box embedding result;

[0049] A4: Perform a binomial cross operation between the bounding box embedding result and the standard Gaussian distribution corresponding to the parameterized Gaussian space to obtain the calculation result;

[0050] A5: The layout-semantic joint image is obtained by solving the KL divergence of the operation results.

[0051] In one possible implementation, the object-level fusion tagging module is configured to obtain the object embedding result by performing the following steps;

[0052] B1: Connect the Fourier transform of the semantic embedding result and the layout embedding result to obtain the single object embedding result;

[0053] B2: Perform learnable empty embedding on the single object embedding result to obtain an object empty embedding result that can adapt to different numbers of objects;

[0054] B3: Add attribute embedding to the empty embedding result of the object to obtain the object embedding result.

[0055] In one possible implementation, the combined masking attention mechanism module includes components forming a communication link structure:

[0056] Self-attention unit: Generates visual embedding based on the object embedding result using a diffusion model, and obtains the visual embedding result;

[0057] Combined masking attention unit: Based on the text characteristics of the visual embedding result, a masking attention map is obtained using a self-attention operation mechanism;

[0058] Cross-attention unit: Integrates volume attribute information of the mask attention map in a cross-attention manner to obtain volume attribute information corresponding to the scene map;

[0059] in,

[0060] The self-attention unit communicates with the object-level fusion marker module;

[0061] This approach effectively injects scene graph information into the diffusion process from the object level, and prevents semantic confusion and attribute leakage through attention masks.

[0062] In one possible implementation, the combined masking attention unit is configured to obtain the masking attention map by performing a combination of the following steps;

[0063] C1: An attention mask map is formed based on whether all pixels in the visual embedding result are visual embeddings or object embeddings of the same object;

[0064] C2: Perform a connection operation between the visual embedding result and the object embedding result to obtain a connection result;

[0065] C3: Obtain the attention map formed by the connection result and the attention mask map using a masked self-attention operation mechanism, and extract the first predetermined number of values ​​in the attention map, where the predetermined number is the number of labels in the visual embedding result, to obtain the masked attention map.

[0066] In one possible implementation, the multi-layer sampling module uses the following formula to estimate the noise of the samples:

[0067]

[0068] In the formula,

[0069] This represents the non-overlapping mask;

[0070] Represents the potential noise of the t-th embedded object;

[0071] This approach can accurately estimate noise, ensuring the consistency of the generated scene image content while manipulating the scene graph.

[0072] Another technical solution of the present invention is to provide a method for generating complex scene images, comprising the following steps:

[0073] S1: Based on the scene graph, a text representation scene graph is constructed through the visual-language module. Then, the text representation scene graph is used as the input of the layout-semantic variational autoencoder module to encode the spatial and object relationships in the text representation scene graph into object-level decoupled embeddings, thereby obtaining object-level layout embedding results and semantic embedding results.

[0074] S2: By combining the layout embedding result and the semantic embedding result through the object-level fusion tagging module and the combined masking attention mechanism module, and simultaneously integrating the volume attribute information, the volume attribute information corresponding to the scene graph is obtained.

[0075] S3: By using the layout embedding result and the semantic embedding result, the multi-layer sampling module performs sampling noise estimation on the scene to obtain the hierarchical representation corresponding to the scene graph.

[0076] This method constructs a text representation scene graph based on the scene graph, and obtains the object-level layout embedding results and semantic embedding results in step S2, and obtains the hierarchical representation corresponding to the scene graph in step S3. This ensures the controllability of complex scene image generation, and makes full use of the structured scene graph to replace the linear text structure to represent complex scenes. It has advantages in modeling the relationships and attributes of complex scenes, which is conducive to the practical application of diffusion models in fields such as embodied intelligence, augmented / virtual reality and artistic creation.

[0077] In one possible implementation, during the execution of step S2: the model parameters of the combined masking attention mechanism module are adjusted by the following loss function;

[0078]

[0079] This approach allows for the adjustment of model parameters, thereby optimizing the variational lower bound of evidence between sampling noise and prediction noise, conditioned on object-level information (i.e., the ground truth of spatial layout and the generated interactive semantics). Attached Figure Description

[0080] Figure 1 This is a schematic diagram of the structure of a complex scene image generation system disclosed in Embodiment 1 of the present invention;

[0081] Figure 2 This is a schematic diagram of the layout-semantic variational autoencoder module structure disclosed in Embodiment 1 of the present invention;

[0082] Figure 3 This is a flowchart of the operation of the combined encoder disclosed in Embodiment 1 of the present invention;

[0083] Figure 4 This is a flowchart of the object-level fusion marking module operation disclosed in Embodiment 1 of the present invention;

[0084] Figure 5 This is a schematic diagram of the combined masking attention mechanism module structure disclosed in Embodiment 1 of the present invention;

[0085] Figure 6 This is a flowchart of the combined masking attention unit operation disclosed in Embodiment 1 of the present invention;

[0086] Figure 7 This is a flowchart of a complex scene image generation method disclosed in Embodiment 2 of the present invention. Detailed Implementation

[0087] First, those skilled in the art should understand that these embodiments are merely used to explain the technical principles of the embodiments of this application and are not intended to limit the scope of protection of the embodiments of this application. Those skilled in the art can make adjustments as needed to adapt to specific application scenarios.

[0088] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the term "forming a communication link structure" means that the multiple communication elements or modules involved form a network structure or network link structure through communication connection. Communication or communication connection means that there is information transmission between the first feature and the second feature. This information transmission can be unidirectional or bidirectional. The communication connection can be realized by electrical connection of wires, radio connection, electrical connection of electromagnetic media (such as optical fiber, semiconductor), communication realized by channel, etc.

[0089] The present application will be further described in detail below using two embodiments, in conjunction with the accompanying drawings and specific embodiments.

[0090] Example 1

[0091] See Figures 1-5 As shown in the figure, this application discloses a complex scene image generation system, including a vision-language module, a layout-semantic variational autoencoder module, an object-level fusion tagging module, a combined masking attention mechanism module, and a multi-layer sampling module. The layout-semantic variational autoencoder module communicates with the vision-language module, the object-level fusion tagging module communicates with the layout-semantic variational autoencoder module, the combined masking attention mechanism module communicates with the object-level fusion tagging module, and the multi-layer sampling module communicates with the layout-semantic variational autoencoder module.

[0092] See Figure 1 As shown, in this embodiment, the visual-language module is configured to perform node embedding and edge embedding on the scene graph based on the objects and relationships between them, and to obtain a text-representation scene graph. The text-representation scene graph is a structured representation of the scene graph, and all nodes and edges in the scene graph have a semantic label.

[0093] The vision-language module is an important component of Graph Convolutional Networks (GCNs). This embodiment embeds a vision-language model into this module. During implementation, it uses the set of nodes formed by objects in the scene graph and the nodes from the scene graph as inputs. i To node o j The relational triples are used as input to the visual-language module. The visual-language model is described as follows:

[0094] The set of nodes formed by objects in the scene graph is:

[0095]

[0096] The set of edges formed by the relationships between objects in the scene graph is:

[0097]

[0098] node o i The semantic tags are Side e ij The semantic tags are

[0099] From node o i To node o j The triples of a relational join are:

[0100]

[0101] Therefore, the structural expression of the visual-language model is:

[0102]

[0103] In the formula,

[0104] A learnable embedding layer representing a set of nodes;

[0105] A learnable embedding layer representing a set of edges;

[0106] This indicates a frozen pre-trained text encoder;

[0107] E box This indicates a bounding box coordinate space encoder using Multi-Layer Perceptions (MLPs);

[0108] Indicates a connection operation;

[0109] b i =(x i ,y i ,w i ,h i ) represents node o i The border;

[0110] Indicates that the output node is embedded;

[0111] ε represents output edge embedding.

[0112] See Figure 2As shown, in this embodiment, the layout-semantic variational autoencoder module is configured to decouple the spatial and interactive relationships of objects in the text representation scene graph into object-level layout embedding results and semantic embedding results through joint encoding and layout-semantic decoding. The layout-semantic variational autoencoder module is a Conditional Variational Autoencoder (CVAE) module based on a convolutional neural network, including a joint encoder, a layout decoder, and a semantic decoder. The joint encoder communicates with the vision-language module, the layout decoder, and the semantic decoder, respectively. The layout decoder and the semantic decoder both communicate with the object-level fusion tagging module.

[0113] See Figure 3 As shown, in the layout-semantic variational autoencoder module, the joint encoder is configured to obtain the layout-semantic joint image by performing the following steps;

[0114] A1: Perform multiple graph convolution operations on the text representation scene graph to obtain the convolution operation image.

[0115] Specifically, the node embeddings and edge embeddings of the text representation scene graph are used as inputs to the joint encoder to jointly model layout and semantic information. The corresponding mathematical expression is:

[0116]

[0117]

[0118] In the formula,

[0119] l represents the layer index of the graphical co-encoder;

[0120] φ represents an intermediate feature;

[0121] GCN l This represents the graph convolution operation.

[0122] A2: Obtain random vectors by sampling from the parameterized Gaussian space of nodes in the convolution operation image.

[0123] Specifically, convolution operations embed image nodes. It integrates topological and interaction information by parameterizing it into a Gaussian space. To perform layout semantic modeling. In this case, the average value... and variance Estimated by two auxiliary perceptrons (MLPs), where D z This represents the spatial dimension of the node embedding. A random vector u is obtained by sampling in the parameterized Gaussian space. i .

[0124] A3: Replace the bounding boxes in the image from the convolution operation with random vectors to obtain the bounding box embedding result.

[0125] Specifically, the joint encoder uses text representations of the scene graph's edge embeddings and employs u i Node embedding after replacing bounding box embedding This is used to construct the potential space of the scene graph structure, thereby obtaining the bounding box embedding result.

[0126] A4: Perform a binomial cross operation between the bounding box embedding result and the standard Gaussian distribution corresponding to the parameterized Gaussian space to obtain the calculation result.

[0127] A5: Obtain the layout-semantic joint image by solving the KL divergence of the computation results.

[0128] The mathematical expressions for steps A4 to A5 are as follows:

[0129]

[0130] In the formula,

[0131] KL represents the KL divergence;

[0132] y represents the condition;

[0133] p(u|y) represents the prior distribution, which is the standard Gaussian distribution in the parameterized Gaussian space.

[0134] Please continue reading Figure 2 As shown, in the layout-semantic variational autoencoder module, the layout decoder is configured to separate the explicit spatial layout from the joint layout-semantic image to obtain the layout embedding result. The layout decoder is a graph convolutional neural network-based decoder capable of separating the explicit spatial layout from the latent space of the joint layout-semantic image. The layout decoder's parameters are optimized using the following objective function:

[0135]

[0136] In the formula,

[0137] This represents the predicted bounding box coordinates. In this embodiment, ground truth bounding box values ​​are used only during training, while diverse object layouts are generated during inference by sampling Gaussian noise.

[0138] Please continue reading Figure 2As shown, in the layout-semantic variational autoencoder module, the semantic decoder is configured to extract implicit interaction semantics from the layout-semantic joint image to obtain semantic embedding results. The semantic decoder is a graph convolutional neural network-based decoder capable of extracting implicit interaction semantics from the latent space of the layout-semantic joint image. The object-level semantic embeddings generated by the semantic decoder are used to guide the image generation of the diffusion model. These embeddings include visual-text aligned CLIP text embeddings, and their parameters are iteratively updated based on the diffusion loss of the reconstructed image.

[0139] See Figure 1 and Figure 4 As shown, in this embodiment, the object-level fusion tagging module is configured to obtain the object embedding result by performing the following steps;

[0140] B1: Concatenate the semantic embedding result with the Fourier transform of the layout embedding result to obtain the single object embedding result. Specifically, this is done by directly applying semantic embedding while using Fourier mapping. The layout information is encoded to obtain the embedding of a single object, and the corresponding formula is as follows:

[0141] Layout embedding is represented as:

[0142]

[0143] Semantic embedding is represented as:

[0144]

[0145] Furthermore, a single object is embedded as follows:

[0146]

[0147] B2: Perform learnable null embeddings on the single object embedding results to obtain object null embedding results that can adapt to different numbers of objects. Specifically, a learnable null embedding is defined, and the embedding length is padded to N. max This allows it to adapt to different numbers of objects.

[0148]

[0149] In the formula,

[0150] c null This represents a learnable empty embedding used for padding.

[0151] B3: Add attribute embedding to the empty object embedding result to obtain the object embedding result. Specifically, you can choose to add attribute embedding. To build an updated Among them, c i and ai Before merging, each process was handled by a separate MLP. Note that a i The method for obtaining this is similar to that of edge embedding in the text representation scene graph. Furthermore, this embodiment also defines a learnable empty embedding 'a' for cases where no attributes are specified. null .

[0152] See Figure 1 and Figure 5 As shown, in this embodiment, the combined masking attention mechanism module is configured to diffuse and combine the object embedding results and integrate volume attribute information to obtain the volume attribute information corresponding to the scene graph. See also Figure 5 As shown,

[0153] The combined masking attention mechanism module includes a self-attention unit, a combined masking attention unit, and a cross-attention unit forming a communication link structure. Specifically, the proposed combined masking attention unit is inserted between the self-attention unit and the cross-attention unit of the diffusion model. This technique can effectively inject scene graph information from the object level into the diffusion process, preventing semantic confusion and attribute leakage through attention masks. Specifically, the self-attention unit is configured to generate visual embeddings based on the object embedding results using a diffusion model, obtaining the visual embedding results. The cross-attention unit is configured to integrate volume attribute information from the masking attention graph using a cross-attention approach, obtaining the volume attribute information corresponding to the scene graph. The self-attention unit communicates with the object-level fusion tagging module.

[0154] See Figure 6 As shown, in this embodiment, the combined masking attention unit is configured to obtain the masking attention map by performing the following steps in combination;

[0155] C1: An attention mask map is formed based on whether all pixels in the visual embedding result belong to the same object visual embedding or object embedding. Specifically, a matrix is ​​constructed. This matrix represents the attention mask that depends on the layout embedding B, and it is constructed as follows:

[0156]

[0157] In the formula, "i and j belong to the same object" means that i and j point to the same visual embedding or object embedding.

[0158] C2: Performs a connection operation between the visual embedding result and the object embedding result to obtain the connection result.

[0159] C3: Obtain the attention map formed by the connection result and the attention mask map using a masked self-attention operation mechanism, and extract the first predetermined number of values ​​from the attention map. The number of predetermined values ​​is the number of labels in the visual embedding result, thus obtaining the masked attention map.

[0160] The mathematical expressions for the processes corresponding to steps C2 and C3 are as follows:

[0161]

[0162] In the formula,

[0163] The object embedding is represented by its dimensions, which are aligned with the visual embedding using MLP.

[0164] See Figure 1 As shown, in this embodiment, the multi-layer sampling module is configured to extract several embedded objects as samples from the layout embedding result and the semantic embedding result using object-level Gaussian noise sampling. The noise of the samples is then estimated using the non-overlapping mask converted from the layout embedding result to obtain the hierarchical representation corresponding to the scene graph. Specifically, the formula for noise estimation of the samples by the multi-layer sampling module is as follows:

[0165]

[0166] In the formula,

[0167] This represents the non-overlapping mask;

[0168] This represents the potential noise of the t-th embedded object.

[0169] Subsequently, the potential noise of each object is determined by N. l Update the weighted average of the cropped, denoised view.

[0170] The complex scene image generation system disclosed in this embodiment generates structured text-based scene graphs by setting up a visual-language module. The representation of the scene through these text-based scene graphs facilitates the generation of complex images. A layout-semantic variational autoencoder module is used to decouple scene space and interaction relationships, thereby achieving a reasonable one-to-many mapping relationship between the scene graph and the decoupled embeddings, overcoming the shortcomings of existing methods in modeling object interaction relationships. Simultaneously, by introducing a combined masking attention mechanism module, the decoupled object-level space and interaction relationship embeddings can be provided to a diffusion model for combined generation, avoiding the leakage of object-level attribute information in this process. The introduced multi-layer sampler mechanism module fully utilizes the diverse decoupled embeddings generated by the variational autoencoder, achieving consistent scene image generation under operations on scene graph nodes and attributes. Furthermore, it facilitates the handling of independent nodes in the scene graph, avoiding the problems of missing or improperly positioned independent nodes.

[0171] Example 2

[0172] This embodiment further discloses a method for generating complex scene images based on Embodiment 1. See [link to Embodiment 1]. Figure 7 As shown, the method includes the following steps:

[0173] S1: Based on the scene graph, a text representation scene graph is constructed through the vision-language module. Then, the text representation scene graph is used as the input of the layout-semantic variational autoencoder module to encode the spatial and object relationships in the text representation scene graph into object-level decoupled embeddings, thereby obtaining object-level layout embedding results and semantic embedding results.

[0174] S2: By combining the layout embedding results and semantic embedding results through the object-level fusion tagging module and the combined masking attention mechanism module, and integrating the volume attribute information at the same time, the volume attribute information corresponding to the scene graph is obtained.

[0175] During the execution of step S2: the model parameters of the combined masking attention mechanism module are adjusted using the following loss function;

[0176]

[0177] Furthermore, the overall framework proposed in Embodiment 1 of this invention adopts an end-to-end training loss, and the overall objective function is as follows:

[0178]

[0179] λ1, λ2, and λ3 are weighted hyperparameters, set to 1.0, 0.1, and 1.0, respectively.

[0180] S3: By using the multi-layer sampling module, the scene is sampled and noise is estimated with the help of layout embedding results and semantic embedding results to obtain the hierarchical representation of the scene graph.

[0181] The method in this embodiment constructs a text representation scene graph based on the scene graph, and obtains the object-level layout embedding result and semantic embedding result in step S2, and obtains the hierarchical representation corresponding to the scene graph in step S3. This ensures the controllability of complex scene image generation, and makes full use of the structured scene graph to replace the linear text structure to represent complex scenes. It has advantages in modeling the relationships and attributes of complex scenes, which is conducive to the practical application of diffusion models in fields such as embodied intelligence, augmented / virtual reality and artistic creation.

[0182] In the description of the embodiments of this application, it should be noted that the terms "inner" and "outer" and other terms indicating direction or positional relationship are based on the direction or positional relationship shown in the drawings. This is only for the convenience of description and does not indicate or imply that the device or component must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this application.

[0183] In the description of this application, the references to terms such as "an embodiment," "some embodiments," "in this embodiment," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0184] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A complex scene image generation system, characterized by, Comprise: A visual-linguistic module: according to the object and the relationship between objects in the scene graph, the visual-linguistic model contained therein is called to carry out node embedding and edge embedding on the scene graph, and a text representation scene graph is obtained; A layout-semantic variational auto-encoding module: the spatial relationship and the interaction relationship of the object in the text representation scene graph are decoupled into the layout embedding result and the semantic embedding result of the object level through the joint encoding and layout-semantic decoding mode; An object-level fusion marking module: the semantic embedding result and the layout embedding result are used to obtain the object embedding result through connection and empty embedding; A combined mask attention mechanism module: the object embedding result is diffused and combined, and the body attribute information is integrated to obtain the body attribute information corresponding to the scene graph; A multi-layer sampling module: a plurality of embedded objects are extracted as samples in the layout embedding result and the semantic embedding result in an object-level Gaussian noise sampling manner, and the samples are subjected to noise estimation by using a non-overlapping mask converted from the layout embedding result, and a hierarchical representation corresponding to the scene graph is obtained. Among them, The layout-semantic variational auto-encoding module communicates with the visual-linguistic module, the object-level fusion marking module communicates with the layout-semantic variational auto-encoding module, the combined mask attention mechanism module communicates with the object-level fusion marking module, and the multi-layer sampling module communicates with the layout-semantic variational auto-encoding module.

2. The complex scene image generation system of claim 1, wherein, The visual-linguistic model is expressed as follows: The node set formed by the objects in the scene graph is: The edge set formed by the relationship between the objects in the scene graph is: node o i The semantic tags are Side e ij The semantic tags are From node o i To node o j The triple of the relationship connection is: The structure expression of the visual-linguistic model is: In the formula, a learnable embedding layer representing the set of nodes; a learnable embedding layer representing the set of edges; E CLIP denotes a frozen pre-trained text encoder; E box represents a bounding box coordinate space encoder using a multi-layer perceptron; N o represents the number of objects in the scene graph; b i = (x i ,y i ,w i ,h i ) represents the bounding box of node o i ; represents a concatenation operation; represents an output node embedding; ε represents the output edge embedding.

3. The complex scene image generation system of claim 2, wherein, The layout-semantic variational auto-encoding module comprises: A joint encoder: a layout-semantic joint image is obtained based on the text representation scene graph through convolution operation and KL divergence solving; A layout decoder: an explicit spatial layout is separated from the layout-semantic joint image, and the layout embedding result is obtained; A semantic decoder: an implicit interaction semantics is separated from the layout-semantic joint image, and the semantic embedding result is obtained; Among them, The joint encoder respectively communicates with the visual-linguistic module, the layout decoder and the semantic decoder, and the layout decoder and the semantic decoder both communicate with the object-level fusion marking module.

4. The complex scene image generation system of claim 3, wherein, The joint encoder is configured to obtain the layout-semantic joint image by performing the following steps: A1: a plurality of graph convolution operations are performed on the text representation scene graph to obtain a convolution operation image; A2: a random vector is obtained by sampling from the parameterized Gaussian space of the nodes in the convolution operation image; A3: the random vector is used to replace the bounding box in the convolution operation image to obtain a bounding box embedding result; A4: a binomial cross operation is performed on the bounding box embedding result and the standard Gaussian distribution corresponding to the parameterized Gaussian space to obtain an operation result; A5: the layout-semantic joint image is obtained by solving the KL divergence of the operation result.

5. The complex scene image generation system according to claim 3 or 4, characterized in that, The object-level fusion marking module is configured to obtain the object embedding result by performing the following steps: B1: connecting the semantic embedding result with the Fourier transform of the layout embedding result to obtain a single object embedding result; B2: learning empty embedding of the single object embedding result to obtain an object empty embedding result for adapting different number of objects; B3: adding attribute embedding to the object empty embedding result to obtain the object embedding result.

6. The complex scene image generation system of claim 5, wherein, The combined mask attention mechanism module comprises a communication link structure formed by: a self-attention unit for generating a visual embedding according to the object embedding result in a diffusion model manner to obtain a visual embedding result; a combined mask attention unit for obtaining a mask attention map in a self-attention operation mechanism based on the text characteristics of the visual embedding result; a cross-attention unit for integrating body attribute information of the mask attention map in a cross-attention manner to obtain the body attribute information corresponding to the scene graph; wherein, the self-attention unit communicates with the object-level fusion mark module.

7. The complex scene image generation system according to claim 6, characterized by, The combined mask attention unit is configured to obtain the mask attention map by performing the following steps in a combined manner: C1: forming an attention mask map according to whether all pixels of the visual embedding result are the same object embedding or object embedding; C2: connecting the visual embedding result with the object embedding result to obtain a connection result; C3: obtaining an attention map formed by the connection result and the attention mask map in a mask self-attention operation mechanism, and extracting the first predetermined number of values in the attention map, the number of the predetermined number being the number of marks in the visual embedding result, to obtain the mask attention map.

8. The complex scene image generation system of claim 6, wherein, The formula for noise estimation of the sample by the multi-layer sampling module is as follows: wherein, This represents the non-overlapping mask; represents the latent noise of the tth embedded object.

9. A method of generating a complex scene image, characterized by, The complex scene image generation system according to any one of claims 1-8 comprises the following steps: S1: based on the scene graph, constructing a text representation scene graph through a vision-language module, and then inputting the text representation scene graph into a layout-semantic variational auto-encoding module to encode the space and object relationship in the text representation scene graph into object-level decoupled embedding to obtain an object-level layout embedding result and a semantic embedding result; S2: combining the layout embedding result and the semantic embedding result through an object-level fusion mark module and a combined mask attention mechanism module, and simultaneously integrating body attribute information to obtain body attribute information corresponding to the scene graph; S3: sampling noise estimation of the scene by means of the layout embedding result and the semantic embedding result through a multi-layer sampling module to obtain a hierarchical representation corresponding to the scene graph.

10. The complex scene image generation method of claim 9, wherein, In the process of performing the step S2: adjusting the model parameters of the combined mask attention mechanism module through the following loss function: wherein, ∈ represents sampling noise; Represents embedded layout; Represents semantic embedding; Represents attribute embedding.

Images