Complex scene image generation system and method

Abstract

The invention relates to a complex scene image generation system and method, in particular to a complex scene image generation system and method based on text representation scene graph decoupling and combination so as to improve the modeling capacity of object attributes and object relations in a complex scene, and the object attributes and object relations in the complex scene can be obtained by decoupling and then combining the space relations and interaction relations represented by the scene graphs. And the controllability and generalization of the diffusion model in the aspect of complex image generation are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and in particular to a complex scene image generation system and method. Background Art

[0002] Controllable image generation is a basic task in the field of computer vision, which aims to generate images that meet the description based on multiple conditions provided by the user (such as text, layout, semantic segmentation map, scene graph, etc.). Since scene graphs can express the object attributes and object relationships in complex scenes more specifically than other conditions, the complex scene image generation task based on scene graphs shows broad application prospects in the fields of embodied intelligence, augmented / virtual reality, and artistic creation.

[0003] Since the relationship between objects in complex scenes includes the spatial and interactive relationships between multiple objects, linear text decoupling is difficult to describe the relationship between objects in complex scenes. At the same time, the existing diffusion model based on text guidance is difficult to accurately generate the number of objects in the scene. Although the diffusion model based on layout guidance is better than the text guidance method in terms of spatial and quantitative modeling, it still has shortcomings in object interaction modeling, that is, each object tends to be generated independently.

[0004] In order to fully model the object relationships and number of objects in the scene, a scene graph-guided diffusion model is proposed for image generation of complex scenes. In this framework, the nodes in the scene graph represent the objects in the scene, and the changes in the scene graph represent the spatial and interactive relationships between objects.

[0005] One of the main approaches of existing scene graph-guided diffusion models is to first convert the scene graph into a layout through a layout predictor, and then use the layout-guided diffusion model to generate scene images. However, this method has shortcomings in modeling object interaction relationships. Another method based on semantic embedding is to encode the scene graph into node embeddings through a graph convolutional neural network as a guiding condition for the diffusion model. Although this method models the interaction between objects in the scene, it is difficult to handle independent nodes in the scene graph, which specifically manifests itself in the lack of independent nodes and unreasonable positions. Summary of the invention

[0006] The technical problem to be solved by the present invention is how to efficiently realize the modeling of object attributes and object relationships in complex scenes. In order to overcome the defects of the above-mentioned prior art, the present invention provides a complex scene image generation system and method, including a complex scene image generation system and a complex scene image generation method.

[0007] The present invention provides a complex scene image generation system, comprising:

[0008] Vision-language module: Based on the objects and the relationships between objects in the scene graph, the vision-language model contained therein is called to perform node embedding and edge embedding on the scene graph to obtain a text representation scene graph;

[0009] Layout-semantic variational autoencoder module: decouples the spatial relationship and interaction relationship of objects in the text representation scene graph into object-level layout embedding results and semantic embedding results through joint encoding and layout-semantic decoding;

[0010] Object-level fusion labeling module: obtaining an object embedding result by using the semantic embedding result and the layout embedding result through connection and empty embedding;

[0011] Combined mask attention mechanism module: diffusely combine the object embedding results and integrate the body attribute information to obtain the body attribute information corresponding to the scene graph;

[0012] Multi-layer sampling module: extracting a number of embedded objects as samples from the layout embedding result and the semantic embedding result by object-level Gaussian noise sampling, performing noise estimation on the samples by using non-overlapping masks converted from the layout embedding result, and obtaining a hierarchical representation corresponding to the scene graph;

[0013] in,

[0014] The layout-semantic variational autoencoder module communicates with the vision-language module, the object-level fusion labeling module communicates with the layout-semantic variational autoencoder module, the combined mask attention mechanism module communicates with the object-level fusion labeling module, and the multi-layer sampling module communicates with the layout-semantic variational autoencoder module.

[0015] Firstly, the complex scene image generation system disclosed in the present invention generates a structured text representation scene graph by setting a visual-language module, and the representation scene of the text representation scene graph is helpful for complex image generation; and a layout-semantic variational autoencoder module is set for decoupling the scene space and the interaction relationship, thereby realizing a one-to-many reasonable mapping relationship between the scene graph and the decoupled embedding, overcoming the shortcomings of the existing methods in modeling the interaction relationship between objects.

[0016] Secondly, the complex scene image generation system disclosed in the present invention can provide the decoupled object-level space and interaction relationship embedding to the diffusion model for combined generation by introducing a combined mask attention mechanism module, and in this process, the leakage of object-level attribute information can be avoided. At the same time, the multi-layer sampler mechanism module introduced makes full use of the diverse decoupled embeddings generated by the variational autoencoder to achieve consistent scene image generation when operating on scene graph nodes and attributes. It is also easy to process independent nodes in the scene graph, avoiding the problems of missing independent nodes and unreasonable positions.

[0017] Finally, the complex scene image generation system disclosed in the present invention can effectively achieve controllability in complex scene image generation, make full use of structured scene graphs instead of linear text structures to represent complex scenes, have advantages in modeling the relationships and attributes of complex scenes, and is conducive to the practical application of diffusion models in the fields of embodied intelligence, augmented / virtual reality, and artistic creation.

[0018] In a possible implementation, the visual-language model is expressed as follows:

[0019] The node set formed by objects in the scene graph is:

[0020]

[0021] The edge set formed by the relationship between objects in the scene graph is:

[0022]

[0023] Node o i The semantic label is Edge ij The semantic label is

[0024] From node o i To node o j The triples of relational connections are:

[0025]

[0026] The structural expression of the vision-language model is:

[0027]

[0028] In the formula,

[0029] a learnable embedding layer representing the set of nodes;

[0030] a learnable embedding layer representing the set of edges;

[0031] E CLIP Represents a frozen pre-trained text encoder;

[0032] E box represents a bounding box coordinate space encoder using a multi-layer perceptron;

[0033] b i =(x i ,y i ,w i ,h i) represents node o i The border of

[0034] Indicates a connection operation;

[0035] represents the output node embedding;

[0036] ε represents the output edge embedding;

[0037] The above-mentioned visual-language model adopted in this scheme can fully exploit the separation semantics inherent in language text while promoting the alignment between images and scene graphs.

[0038] In a possible implementation, the layout-semantic variational autoencoder module includes:

[0039] Joint encoder: obtaining a layout-semantic joint image based on the text representation scene graph by convolution operation and solving KL divergence;

[0040] Layout decoder: separating the explicit spatial layout from the layout-semantic joint image to obtain the layout embedding result;

[0041] Semantic decoder: separating implicit interaction semantics from the layout-semantic joint image to obtain the semantic embedding result;

[0042] in,

[0043] The joint encoder communicates with the vision-language module, the layout decoder, and the semantic decoder respectively, and the layout decoder and the semantic decoder both communicate with the object-level fusion labeling module;

[0044] This scheme can decouple the spatial and interaction relationships in the text representation scene graph into object-level layout embedding and semantic embedding through layout encoding and decoding modes.

[0045] In a possible implementation, the joint encoder is configured to obtain the layout-semantic joint image by performing the following steps:

[0046] A1: performing multiple graph convolution operations on the text representation scene graph to obtain a convolution operation image;

[0047] A2: obtaining a random vector by sampling from the parameterized Gaussian space of nodes in the convolution operation image;

[0048] A3: replacing the bounding box in the convolution operation image with the random vector to obtain a bounding box embedding result;

[0049] A4: performing a binomial crossover operation on the bounding box embedding result and a standard Gaussian distribution corresponding to the parameterized Gaussian space to obtain an operation result;

[0050] A5: Obtain the layout-semantic joint image by solving the KL divergence of the calculation result.

[0051] In a possible implementation, the object-level fusion labeling module is configured to obtain the object embedding result by performing the following steps:

[0052] B1: performing a concatenation operation on the semantic embedding result and the Fourier transform of the layout embedding result to obtain a single object embedding result;

[0053] B2: Perform a learnable empty embedding on the single object embedding result to obtain an object empty embedding result that can adapt to different numbers of objects;

[0054] B3: Add attribute embedding to the empty embedding result of the object to obtain the object embedding result.

[0055] In one possible implementation, the combined mask attention mechanism module includes:

[0056] Self-attention unit: generating visual embedding according to the object embedding result in a diffusion model manner to obtain a visual embedding result;

[0057] Combined mask attention unit: based on the text characteristics of the visual embedding result, a mask attention map is obtained by a self-attention operation mechanism;

[0058] Cross-attention unit: integrating the volume attribute information of the mask attention map in a cross-attention manner to obtain the volume attribute information corresponding to the scene map;

[0059] in,

[0060] The self-attention unit communicates with the object-level fusion labeling module;

[0061] This scheme can effectively inject scene graph information from the object level into the diffusion process and prevent semantic confusion and attribute leakage through attention masks.

[0062] In a possible implementation, the combined mask attention unit is configured to obtain the mask attention map by performing a combination of the following steps;

[0063] C1: forming an attention mask map according to whether all pixels of the visual embedding result are visual embeddings or object embeddings of the same object;

[0064] C2: performing a connection operation on the visual embedding result and the object embedding result to obtain a connection result;

[0065] C3: Obtain an attention map formed by the connection result and the attention mask map using a masked self-attention operation mechanism, and extract the first predetermined values ​​in the attention map, where the predetermined number is the number of markers in the visual embedding result, to obtain the masked attention map.

[0066] In a possible implementation manner, the multi-layer sampling module performs noise estimation on the sample using the following formula:

[0067]

[0068] In the formula,

[0069] represents the non-overlapping mask;

[0070] represents the potential noise of the t-th embedded object;

[0071] This scheme can accurately estimate the noise and ensure the consistency of the generated scene image content while manipulating the scene graph.

[0072] Another technical solution of the present invention is to provide a complex scene image generation method, comprising the following steps:

[0073] S1: Based on the scene graph, a text representation scene graph is constructed through the vision-language module, and then the text representation scene graph is used as the input of the layout-semantic variational autoencoder module to encode the spatial and object relationships in the text representation scene graph into object-level decoupled embeddings to obtain object-level layout embedding results and semantic embedding results;

[0074] S2: combining the layout embedding result and the semantic embedding result through the object-level fusion labeling module and the combined mask attention mechanism module, and integrating the volume attribute information at the same time, to obtain the volume attribute information corresponding to the scene graph;

[0075] S3: Using the multi-layer sampling module, a sampling noise estimation is performed on the scene with the help of the layout embedding result and the semantic embedding result to obtain a hierarchical representation corresponding to the scene graph.

[0076] This method constructs a text representation scene graph based on the scene graph, obtains the layout embedding result and semantic embedding result at the object level in step S2, and obtains the hierarchical representation corresponding to the scene graph in step S3, thereby ensuring the controllability of complex scene image generation and making full use of the structured scene graph to replace the linear text structure to represent the complex scene. It has advantages in modeling the relationships and attributes of complex scenes, and is conducive to the practical application of the diffusion model in the fields of embodied intelligence, augmented / virtual reality, and artistic creation.

[0077] In a possible implementation, during the execution of step S2: adjusting the model parameters of the combined mask attention mechanism module by the following loss function;

[0078]

[0079] This scheme adjusts the model parameters in such a way as to optimize the variational evidence lower bound between the sampled noise and the predicted noise conditioned on object-level information (i.e., the ground-truth spatial layout and the generated interaction semantics). BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a schematic diagram of the structure of a complex scene image generation system disclosed in Embodiment 1 of the present invention;

[0081] Figure 2 This is a schematic diagram of the structure of the layout-semantic variational autoencoder module disclosed in the first embodiment of the present invention;

[0082] Figure 3 This is a flow chart of the operation of the joint encoder disclosed in the first embodiment of the present invention;

[0083] Figure 4 This is a flowchart of the operation of the object-level fusion labeling module disclosed in the first embodiment of the present invention;

[0084] Figure 5 Schematic diagram of the structure of the combined mask attention mechanism module disclosed in the first embodiment of the present invention;

[0085] Figure 6 This is a flowchart of the operation of the combined mask attention unit disclosed in the first embodiment of the present invention;

[0086] Figure 7 This is a flow chart of a complex scene image generation method disclosed in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0087] First, those skilled in the art should understand that these implementations are only used to explain the technical principles of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. Those skilled in the art can make adjustments to them as needed to adapt to specific application scenarios.

[0088] In the description of the embodiments of the present application, it should be noted that, unless otherwise clearly specified and limited, the term "forming a communication link structure" refers to the multiple communication elements or modules involved forming a network structure or a network link structure through communication connections, and communication or communication connection refers to the transmission of information between the first feature and the second feature. This information transmission can be either unidirectional or bidirectional, and the communication connection can be achieved by electrical connection of wires, radio connection, electrical connection of electromagnetic media (such as optical fibers, semiconductors), communication achieved by channels, etc.

[0089] Two embodiments will be used below, and the present application will be further described in detail in conjunction with the accompanying drawings and specific embodiments.

[0090] Embodiment 1

[0091] See also Figure 1 to Figure 5 As shown, an embodiment of the present application discloses a complex scene image generation system, including a vision-language module, a layout-semantic variational autoencoder module, an object-level fusion labeling module, a combined mask attention mechanism module and a multi-layer sampling module, wherein the layout-semantic variational autoencoder module communicates with the vision-language module, the object-level fusion labeling module communicates with the layout-semantic variational autoencoder module, the combined mask attention mechanism module communicates with the object-level fusion labeling module, and the multi-layer sampling module communicates with the layout-semantic variational autoencoder module.

[0092] See also Figure 1 As shown, in this embodiment, the visual-language module is configured to call the visual-language model contained therein to perform node embedding and edge embedding on the scene graph according to the objects and the relationships between the objects in the scene graph, so as to obtain a text representation scene graph. The text representation scene graph is a structured scene expression of the scene graph, and all nodes and edges in the scene graph carry a semantic label.

[0093] The visual-language module is an important component of the graph convolutional neural network (GCNs). In this embodiment, the visual-language model is embedded in this module. In the implementation process, the node set formed by the objects in the scene graph and the node o in the scene graph are used as the input. i To node o j The relation-connected triplets are used as input to the visual-language module. The visual-language model is described as follows:

[0094] The node set formed by objects in the scene graph is:

[0095]

[0096] The edge set formed by the relationship between objects in the scene graph is:

[0097]

[0098] Node o i The semantic label is Edge ij The semantic label is

[0099] From node o i To node o j The triples of relational connections are:

[0100]

[0101] Based on this, the structural expression of the visual-language model is:

[0102]

[0103] In the formula,

[0104] A learnable embedding layer representing a collection of nodes;

[0105] A learnable embedding layer representing a collection of edges;

[0106] Represents a frozen pre-trained text encoder;

[0107] E box Represents a bounding box coordinate space encoder using Multi-Layer Perceptions (MLPs);

[0108] Indicates a connection operation;

[0109] b i =(x i ,y i ,w i ,h i ) represents node o i The border of

[0110] represents the output node embedding;

[0111] ε represents the output edge embedding.

[0112] See also Figure 2As shown, in this embodiment, the layout-semantic variational autoencoder module is configured to decouple the spatial relationship and interaction relationship of objects in the text representation scene graph into object-level layout embedding results and semantic embedding results through joint encoding and layout-semantic decoding. The layout-semantic variational autoencoder module is a module of a conditional variational autoencoder (CVAE) based on a convolutional neural network, including a joint encoder, a layout decoder and a semantic decoder, wherein the joint encoder communicates with the vision-language module, the layout decoder and the semantic decoder respectively, and the layout decoder and the semantic decoder both communicate with the object-level fusion labeling module.

[0113] See also Figure 3 As shown, in the layout-semantics variational self-encoder module, the joint encoder is set to obtain the layout-semantics joint image by performing the following steps;

[0114] A1: Perform multiple graph convolution operations on the text representation scene graph to obtain a convolution operation image.

[0115] Specifically, the node embedding and edge embedding of the textual representation scene graph are used as the input of the joint encoder to jointly model the layout and semantic information. The corresponding mathematical expression is:

[0116]

[0117]

[0118] In the formula,

[0119] l represents the layer index of the graphics joint encoder;

[0120] φ represents the intermediate feature;

[0121] GCN l Represents a graph convolution operation.

[0122] A2: Obtain random vectors by sampling from the parameterized Gaussian space of nodes in the convolution operation image.

[0123] Specifically, the convolution operation image node embedding It integrates topological and interaction information by parameterizing it into a Gaussian space to model the layout semantics. In this case, the average and variance They are estimated by two auxiliary perceptrons (MLP), where D z Represents the spatial dimension of node embedding. Sampling a random vector u in the parameterized Gaussian space i .

[0124] A3: Replace the bounding box in the convolution operation image with a random vector to obtain the bounding box embedding result.

[0125] Specifically, the joint encoder uses the edge embedding of the text representation scene graph and adopts u i Node embeddings instead of bounding box embeddings To form the latent space of the scene graph structure, we can obtain the bounding box embedding result.

[0126] A4: Perform a binomial crossover operation on the bounding box embedding result and the standard Gaussian distribution corresponding to the parameterized Gaussian space to obtain the operation result.

[0127] A5: The layout-semantic joint image is obtained by solving the KL divergence of the operation results.

[0128] The mathematical expressions for steps A4 to A5 are as follows:

[0129]

[0130] In the formula,

[0131] KL represents KL divergence;

[0132] y represents the condition;

[0133] p(u|y) represents the prior distribution, which is the standard Gaussian distribution in the parameterized Gaussian space

[0134] Please continue to see Figure 2 As shown in the figure, in the layout-semantic variational autoencoder module, the layout decoder is set to separate the explicit spatial layout from the layout-semantic joint image to obtain the layout embedding result. The layout decoder is a decoder based on a graph convolutional neural network, which can separate the explicit spatial layout from the latent space of the layout-semantic joint image. The layout decoder is optimized through the following objective function:

[0135]

[0136] In the formula,

[0137] Indicates the predicted bounding box coordinates. This embodiment only uses the true bounding box value during training, and generates a variety of object layouts by sampling Gaussian noise during inference.

[0138] Please continue to see Figure 2As shown in the figure, in the layout-semantics variational autoencoder module, the semantic decoder is set to separate the implicit interaction semantics from the layout-semantics joint image to obtain the semantic embedding result. The semantic decoder is a decoder based on a graph convolutional neural network, which can separate the implicit interaction semantics from the latent space of the layout-semantics joint image. The object-level semantic embedding generated by the semantic decoder is used to guide the image generation of the diffusion model. Its embedding contains the visual-text aligned CLIP text embedding, so its parameters will be iteratively updated as a whole according to the diffusion loss of the restored image.

[0139] See also Figure 1 and Figure 4 As shown, in this embodiment, the object-level fusion labeling module is configured to obtain the object embedding result by performing the following steps;

[0140] B1: Concatenate the semantic embedding result with the Fourier transform of the layout embedding result to obtain the single object embedding result. Specifically, the semantic embedding is directly applied while Fourier transform is used Encode the layout information to obtain a single object embedding. The corresponding formula is as follows:

[0141] The layout embedding is represented as:

[0142]

[0143] The semantic embedding is represented as:

[0144]

[0145] Then, a single object is embedded as:

[0146]

[0147] B2: Perform a learnable empty embedding on the single object embedding result to obtain an object empty embedding result that can adapt to different numbers of objects. Specifically, a learnable empty embedding is defined to fill the embedding length to N. max , thus adapting to different numbers of objects:

[0148]

[0149] In the formula,

[0150] c null represents a learnable empty embedding for padding.

[0151] B3: Add attribute embedding to the empty embedding result of the object to obtain the object embedding result. You can choose to add attribute embedding To build an updated Among them, c i and ai are processed by two MLPs before being merged. Note that a i The way to obtain is similar to the edge embedding in the text representation scene graph. In addition, this embodiment also defines a learnable empty embedding a for the case where no attribute is specified null .

[0152] See also Figure 1 and Figure 5 As shown, in this embodiment, the combined mask attention mechanism module is configured to diffusely combine the object embedding results and integrate the body attribute information to obtain the body attribute information corresponding to the scene graph. Figure 5 As shown,

[0153] The combined mask attention mechanism module includes a self-attention unit, a combined mask attention unit and a cross-attention unit that form a communication link structure. Specifically, the proposed combined mask attention unit is inserted between the self-attention unit and the cross-attention unit of the diffusion model. This technology can effectively inject scene graph information from the object level into the diffusion process, and prevent semantic confusion and attribute leakage through attention masks. Among them, the self-attention unit is configured to generate visual embedding according to the object embedding result in a diffusion model manner to obtain a visual embedding result. The cross-attention unit is configured to integrate the body attribute information of the mask attention graph in a cross-attention manner to obtain the body attribute information corresponding to the scene graph. The self-attention unit communicates with the object-level fusion labeling module.

[0154] See also Figure 6 As shown, in this embodiment, the combined mask attention unit is configured to obtain a mask attention map by performing a combination of the following steps;

[0155] C1: An attention mask is formed based on whether all pixels of the visual embedding result are visual embeddings or object embeddings of the same object. Specifically, the matrix is ​​constructed as This matrix represents the attention mask depending on the layout embedding B and is constructed as follows:

[0156]

[0157] Here, “i, j belong to the same object” means that i and j point to the visual embedding or object embedding of the same object.

[0158] C2: Concatenate the visual embedding result with the object embedding result to obtain the concatenated result.

[0159] C3: Use the self-attention operation mechanism with mask to obtain the attention map formed by the connection result and the attention mask map, and extract the first predetermined values ​​in the attention map. The number of predetermined values ​​is the number of labels in the visual embedding result to obtain the masked attention map.

[0160] The mathematical expressions of the processes corresponding to steps C2 and C3 are as follows:

[0161]

[0162] In the formula,

[0163] represents an object embedding whose dimensions are aligned with the visual embedding of using an MLP.

[0164] See also Figure 1 As shown, in this embodiment, the multi-layer sampling module is configured to extract a number of embedded objects as samples from the layout embedding result and the semantic embedding result in an object-level Gaussian noise sampling manner, and perform noise estimation on the samples using the non-overlapping masks converted from the layout embedding result to obtain a hierarchical representation corresponding to the scene graph. Specifically, the formula for noise estimation of the samples by the multi-layer sampling module is as follows:

[0165]

[0166] In the formula,

[0167] represents the non-overlapping mask;

[0168] represents the potential noise of the t-th embedded object.

[0169] Then, the potential noise of each object is given by N l The weighted average of the cropped denoised views is updated.

[0170] The complex scene image generation system disclosed in this embodiment generates a structured text representation scene graph by setting a visual-language module, and the representation scene of the text representation scene graph is helpful for complex image generation; by setting a layout-semantic variational autoencoder module for the decoupling of scene space and interaction relationship, a one-to-many reasonable mapping relationship between scene graph and decoupled embedding is achieved, overcoming the defects of the existing methods in modeling object interaction relationships. At the same time, by introducing a combined mask attention mechanism module, the decoupled object-level space and interaction relationship embedding can be provided to the diffusion model for combined generation, and the leakage of object-level attribute information can be avoided in this process. The multi-layer sampler mechanism module introduced at the same time makes full use of the diverse decoupled embedding generated by the variational autoencoder to achieve consistent scene image generation under the operation of scene graph nodes and attributes. It is also convenient to process independent nodes in the scene graph, avoiding the problems of missing and unreasonable positions of independent nodes.

[0171] Embodiment 2

[0172] This embodiment further discloses a complex scene image generation method based on the first embodiment. Figure 7 As shown, the method comprises the following steps:

[0173] S1: Based on the scene graph, a text representation scene graph is constructed through the vision-language module, and then the text representation scene graph is used as the input of the layout-semantic variational autoencoder module to encode the spatial and object relationships in the text representation scene graph into object-level decoupled embeddings, and obtain object-level layout embedding results and semantic embedding results.

[0174] S2: The layout embedding results and semantic embedding results are combined through the object-level fusion labeling module and the combined mask attention mechanism module, and the volume attribute information is integrated at the same time to obtain the volume attribute information corresponding to the scene graph.

[0175] In the process of executing step S2: adjusting the model parameters of the combined mask attention mechanism module by the following loss function;

[0176]

[0177] Furthermore, the overall framework proposed in the first embodiment of the present invention adopts an end-to-end training loss, and the overall objective function is as follows:

[0178]

[0179] where λ1, λ2, and λ3 are weighted hyperparameters, set to 1.0, 0.1, and 1.0, respectively.

[0180] S3: The multi-layer sampling module uses the layout embedding results and the semantic embedding results to estimate the sampling noise of the scene and obtain the hierarchical representation corresponding to the scene graph.

[0181] The method of this embodiment constructs a text representation scene graph based on the scene graph, obtains the layout embedding result and the semantic embedding result at the object level in step S2, and obtains the hierarchical representation corresponding to the scene graph in step S3, thereby ensuring the controllability of complex scene image generation and making full use of the structured scene graph to replace the linear text structure to represent the complex scene, which has advantages in the relationship and attribute modeling of complex scenes, and is conducive to the practical application of the diffusion model in the fields of embodied intelligence, augmented / virtual reality and artistic creation.

[0182] In the description of the embodiments of the present application, it should be noted that in the description of the present application, terms such as "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is only for the convenience of description, and does not indicate or imply that the device or component must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present application.

[0183] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "in the present embodiment", "specific example", or "some examples" etc. means that the specific features, mechanisms, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0184] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A complex scene image generation system, characterized in that: include: Vision-language module: Based on the objects and the relationships between objects in the scene graph, the vision-language model contained therein is called to perform node embedding and edge embedding on the scene graph to obtain a text representation scene graph; Layout-semantic variational autoencoder module: decouples the spatial relationship and interaction relationship of objects in the text representation scene graph into object-level layout embedding results and semantic embedding results through joint encoding and layout-semantic decoding; Object-level fusion labeling module: obtaining an object embedding result by using the semantic embedding result and the layout embedding result through connection and empty embedding; Combined mask attention mechanism module: diffusely combine the object embedding results and integrate the body attribute information to obtain the body attribute information corresponding to the scene graph; Multi-layer sampling module: extracting a number of embedded objects as samples from the layout embedding result and the semantic embedding result by object-level Gaussian noise sampling, performing noise estimation on the samples by using non-overlapping masks converted from the layout embedding result, and obtaining a hierarchical representation corresponding to the scene graph; in, The layout-semantic variational autoencoder module communicates with the vision-language module, the object-level fusion labeling module communicates with the layout-semantic variational autoencoder module, the combined mask attention mechanism module communicates with the object-level fusion labeling module, and the multi-layer sampling module communicates with the layout-semantic variational autoencoder module.

2. The complex scene image generation system according to claim 1, characterized in that: The vision-language model is expressed as follows: The node set formed by objects in the scene graph is: The edge set formed by the relationship between objects in the scene graph is: Node o i The semantic label is Edge ij The semantic label is From node o i To node o j The triples of relational connections are: The structural expression of the vision-language model is: In the formula, a learnable embedding layer representing the set of nodes; a learnable embedding layer representing the set of edges; E CLIP Represents a frozen pre-trained text encoder; E box represents a bounding box coordinate space encoder using a multi-layer perceptron; b i =(x i ,y i ,w i ,h i ) represents node o i The border of Indicates a connection operation; represents the output node embedding; ε represents the output edge embedding.

3. The complex scene image generation system according to claim 2, characterized in that: The layout-semantic variational autoencoder module includes: Joint encoder: obtaining a layout-semantic joint image based on the text representation scene graph by convolution operation and solving KL divergence; Layout decoder: separating the explicit spatial layout from the layout-semantic joint image to obtain the layout embedding result; Semantic decoder: separating implicit interaction semantics from the layout-semantic joint image to obtain the semantic embedding result; in, The joint encoder communicates with the vision-language module, the layout decoder, and the semantic decoder respectively, and the layout decoder and the semantic decoder both communicate with the object-level fusion labeling module.

4. The complex scene image generation system according to claim 3, characterized in that: The joint encoder is configured to obtain the layout-semantic joint image by performing the following steps; A1: performing multiple graph convolution operations on the text representation scene graph to obtain a convolution operation image; A2: obtaining a random vector by sampling from the parameterized Gaussian space of nodes in the convolution operation image; A3: replacing the bounding box in the convolution operation image with the random vector to obtain a bounding box embedding result; A4: performing a binomial crossover operation on the bounding box embedding result and a standard Gaussian distribution corresponding to the parameterized Gaussian space to obtain an operation result; A5: Obtain the layout-semantic joint image by solving the KL divergence of the calculation result.

5. The complex scene image generation system according to claim 3 or 4, characterized in that: The object-level fusion labeling module is configured to obtain the object embedding result by performing the following steps: B1: performing a concatenation operation on the semantic embedding result and the Fourier transform of the layout embedding result to obtain a single object embedding result; B2: Perform a learnable empty embedding on the single object embedding result to obtain an object empty embedding result that can adapt to different numbers of objects; B3: Add attribute embedding to the empty embedding result of the object to obtain the object embedding result.

6. The complex scene image generation system according to claim 5, characterized in that: The combined mask attention mechanism module includes: Self-attention unit: generating visual embedding according to the object embedding result in a diffusion model manner to obtain a visual embedding result; Combined mask attention unit: based on the text characteristics of the visual embedding result, a mask attention map is obtained by a self-attention operation mechanism; Cross-attention unit: integrating the volume attribute information of the mask attention map in a cross-attention manner to obtain the volume attribute information corresponding to the scene map; in, The self-attention unit communicates with the object-level fusion labeling module.

7. The complex scene image generation system according to claim 5, characterized in that: The combined mask attention unit is configured to obtain the mask attention map by performing a combination of the following steps; C1: forming an attention mask map according to whether all pixels of the visual embedding result are visual embeddings or object embeddings of the same object; C2: performing a connection operation on the visual embedding result and the object embedding result to obtain a connection result; C3: Obtain an attention map formed by the connection result and the attention mask map using a masked self-attention operation mechanism, and extract the first predetermined values ​​in the attention map, where the predetermined number is the number of markers in the visual embedding result, to obtain the masked attention map.

8. The complex scene image generation system according to claim 6, characterized in that: The multi-layer sampling module performs noise estimation on the sample as follows: In the formula, represents the non-overlapping mask; represents the potential noise of the t-th embedded object.

9. A complex scene image generation method, characterized in that: The complex scene image generation system according to any one of claims 1 to 8 comprises the following steps: S1: Based on the scene graph, a text representation scene graph is constructed through the vision-language module, and then the text representation scene graph is used as the input of the layout-semantic variational autoencoder module to encode the spatial and object relationships in the text representation scene graph into object-level decoupled embeddings to obtain object-level layout embedding results and semantic embedding results; S2: combining the layout embedding result and the semantic embedding result through an object-level fusion tagging module and a combined mask attention mechanism module, and integrating the volume attribute information at the same time to obtain the volume attribute information corresponding to the scene graph; S3: Using the multi-layer sampling module, a sampling noise estimation is performed on the scene with the help of the layout embedding result and the semantic embedding result to obtain a hierarchical representation corresponding to the scene graph.

10. The complex scene image generation method according to claim 9, characterized in that: In the process of executing step S2: the model parameters of the combined mask attention mechanism module are adjusted by the following loss function:

Images