Scene generation method and device, storage medium and electronic equipment

Abstract

The invention discloses a scene generation method and device, a storage medium and electronic equipment, and the method comprises the steps: obtaining a task description used for constructing a target scene, analyzing the task description based on a graph designer, obtaining a target scene graph corresponding to the task description, the target scene graph comprises object data in the task description, and the object data comprises object data in the target description; according to the object data and the spatial relationship between the object data, inputting the target scene graph into a trained scene generation model to obtain a target layout generated by the scene generation model based on the object data and the spatial relationship, and modeling an object corresponding to the object data obtained from a database, and generating the target scene according to the object modeling and the target layout. By adopting the specification, the efficiency, convenience and accuracy of scene generation are improved.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a scene generation method, device, storage medium, and electronic device. Background Art

[0002] Nowadays, with the development of the Internet and technology, indoor scenes can be constructed according to user input needs. However, when constructing scenes, there is insufficient data reserves, and the scenes required by users cannot be constructed, resulting in insufficient accuracy of the constructed scenes. Summary of the Invention

[0003] The embodiments of this specification provide a scene generation method, device, storage medium and electronic device, which can extract a target scene graph from a task description by combining a graph designer and a scene generation model, generate a corresponding target layout based on the target scene graph, and obtain a target scene corresponding to the target layout, so that the user only needs to input a task description to obtain the required target scene, and then realize the extraction of task description data according to the graph designer, and combine the layout generated by a single model to generate the required scene, thereby improving the efficiency, convenience and accuracy of scene generation.

[0004] In a first aspect, an embodiment of this specification provides a scene generation method, the method comprising:

[0005] Obtain a task description for constructing the target scenario;

[0006] Parsing the task description based on a graph designer to obtain a target scene graph corresponding to the task description, the target scene graph including object data in the task description and spatial relationships between the object data;

[0007] Inputting the target scene graph into a trained scene generation model, so that the scene generation model generates a target layout based on the object data and the spatial relationship;

[0008] The object data obtained in the database corresponds to object modeling, and the target scene is generated according to the object modeling and the target layout.

[0009] Through the above technical solution, the task description is parsed and inferred to obtain a target scene graph including object data and spatial relationships corresponding to the task description. The object parameters corresponding to the object data are inferred based on the target scene graph according to the scene generation model to generate a target layout. Then, the target scene corresponding to the task description is generated based on the object modeling and the target layout. Thus, by combining the graph designer and the scene generation model, the target scene graph is extracted from the task description, the corresponding target layout is generated according to the target scene graph, and the target scene corresponding to the target layout is obtained. This satisfies the user's need to input only the task description to obtain the required target scene, thereby realizing the extraction of task description data according to the graph designer and combining the layout generated by a single model to generate the required scene, thereby improving the efficiency, convenience and accuracy of scene generation.

[0010] In combination with the first aspect and the above implementation, in some possible implementations, parsing the task description based on the graph designer to obtain the target scene graph corresponding to the task description includes:

[0011] Parsing the task description based on a graph designer to obtain a root node in the task description;

[0012] Based on the root node, other nodes corresponding to the task description and the spatial relationship between each of the other nodes and the root node are determined to obtain a target scene graph corresponding to the task description.

[0013] Through the above technical solution, the graph designer obtains other nodes according to the determined root node in the task description, as well as the spatial relationship between the root node and other nodes, and other nodes, and outputs the target scene graph, thereby facilitating subsequent scene generation based on the target scene graph and improving the efficiency and accuracy of scene generation.

[0014] In combination with the first aspect and the above implementation, in some possible implementations, determining other nodes corresponding to the task description based on the root node, and the spatial relationship between each of the other nodes and the root node, to obtain a target scene graph corresponding to the task description, includes:

[0015] Gradually acquiring other nodes in the task description according to the root node, where the other nodes are nodes in the task description other than the root node;

[0016] sequentially determining the spatial relationship between the root node and each of the other nodes, and between each of the other nodes;

[0017] The text data representing the root node, the other nodes and the spatial relationship is format converted using a preset format conversion method to obtain a target scene graph corresponding to the task description.

[0018] Through the above technical solution, other nodes and corresponding spatial relationships are inferred based on the root node, and the corresponding text data is converted into a target scene graph, so that the scene generation model can recognize and process the target scene graph to realize scene generation.

[0019] In combination with the first aspect and the above implementations, in some possible implementations, inputting the target scene graph into a trained scene generation model to obtain a target layout generated by the scene generation model based on the object data and the spatial relationship includes:

[0020] Inputting the target scene graph into a trained scene generation model so that the scene generation model adds object data in the target scene graph to an initial layout, the object data including first object data and second object data, wherein the first object data is object data included in the target scene graph, and the second object data is inferred based on the target scene graph and is used to supplement the object data of the scene;

[0021] The scene generation model is used to adjust object parameters of the object data and positions of the object data in the initial layout based on the spatial relationship to generate a target layout corresponding to the target scene graph.

[0022] Through the above technical solution, the position and object parameters of the object data are adjusted according to the target scene graph based on the scene generation model, and the target layout corresponding to the target scene graph is generated, so that the user only needs to enter the task description to obtain the target scene, thereby realizing the layout generated according to a single model to generate the required scene, improving the efficiency and convenience of scene generation.

[0023] In combination with the first aspect and the above implementation manner, in some possible implementation manners, the object modeling corresponding to the object data obtained in the database, and generating the target scene based on the object modeling and the target layout include:

[0024] Obtaining object modeling corresponding to the object data in a database according to the object data and the object size of the object data;

[0025] The object modeling is added to the initial scene according to the target layout, the modeling parameters of the object modeling in the initial scene are adjusted based on the object parameters of the object data, and the position of the object modeling in the initial scene is adjusted based on the position of the object data to generate a target scene corresponding to the task description.

[0026] In combination with the first aspect and the above implementation manner, in some possible implementation manners, a prompt module is created in the graph designer, and an information identification prompt for the task description is generated based on the prompt module, wherein the information identification prompt is used to prompt the graph designer to obtain the object data and the spatial relationship corresponding to the object data based on the task description;

[0027] A preset format conversion mode of the graph designer is set, and the preset format conversion mode is used to convert the text data into a scene graph.

[0028] In combination with the first aspect and the above implementations, in some possible implementations, the sample scene graph is input into a scene generation model to determine sample object data and sample spatial relationships in the sample scene graph;

[0029] adding noise object data to the initial layout based on the sample object data, and adjusting the position, replacing objects, or deleting objects of the noise object data until a test layout is generated;

[0030] The test layout and the sample layout are compared to obtain a loss function of the scene generation model. If the loss function indicates that the scene generation model has converged, a trained scene generation model is obtained.

[0031] In combination with the first aspect and the above implementation, the scene generation model is trained while setting the sample object parameters of the sample object data according to the first probability to remain unchanged;

[0032] The scene generation model is trained while the spatial relationship corresponding to the sample object data remains unchanged according to the second probability setting.

[0033] In a second aspect, an embodiment of this specification provides a scene generation device, the device comprising:

[0034] A description acquisition unit, used to acquire a task description for constructing a target scenario;

[0035] A scene graph generating unit, configured to parse the task description based on a graph designer to obtain a target scene graph corresponding to the task description, wherein the target scene graph includes object data in the task description and spatial relationships between the object data;

[0036] a layout generation unit, configured to input the target scene graph into a trained scene generation model to obtain a target layout generated by the scene generation model based on the object data and the spatial relationship;

[0037] A scene generation unit is used to model the object corresponding to the object data obtained in the database, and generate the target scene according to the object modeling and the target layout.

[0038] In a third aspect, an embodiment of this specification provides a computer storage medium, which stores a plurality of instructions suitable for being loaded by a processor and executing the steps of the above method.

[0039] In a fourth aspect, an embodiment of this specification provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 A system architecture diagram of a scene generation method provided in an embodiment of this specification;

[0042] Figure 2 A schematic diagram of a flow chart of a scene generation method provided in an embodiment of this specification;

[0043] Figure 3 A schematic diagram of a flow chart of a scene generation method provided in an embodiment of this specification;

[0044] Figure 4 This is a schematic diagram of an example of generating a target layout provided in an embodiment of this specification;

[0045] Figure 5 This is a schematic diagram of an example of generating a target scene provided in an embodiment of this specification;

[0046] Figure 6 A schematic diagram of the structure of a scene generation device provided in an embodiment of this specification;

[0047] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0048] To make the features and advantages of this specification more obvious and easy to understand, the technical solutions in this specification are clearly and completely described below in conjunction with the drawings in this specification. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without making any creative efforts are within the scope of protection of this specification.

[0049] See Figure 1 , provides a system structure diagram for scene generation in the embodiment of this specification. Figure 1 As shown, the scene generation method provided in the embodiment of this specification can be applied to a terminal device to implement the process of generating a scene for a task description. The system structure provided in the embodiment of this specification mainly includes a terminal device 10, a task description 20 and a target scene 30. The terminal device 10 includes a graphic designer 11 and a scene generation model 12. The terminal device 10 can be a device for receiving and processing task descriptions, such as a personal computer, a server, etc. The graphic designer 11 can be a data processing module with data extraction and deduction functions, specifically a visual language model enhanced by thought chains. The scene generation model 12 can be a model with a layout generation function, and the graphic designer 11 and the scene generation model 12 can be stored in the terminal device 10. The task description 20 can be an instruction for triggering scene construction, specifically text data and / or image data. The target scene 30 can be a three-dimensional scene constructed according to the task description.

[0050] In the related art, in the process of constructing indoor scenes according to user needs, the method adopted is to construct the scene based on the text input by the user. However, due to the insufficient size of the data set, it is impossible to accurately construct a scene that meets the user's needs. Therefore, how to improve the accuracy of scene construction has become an urgent problem to be solved.

[0051] In the embodiment of the present specification, the terminal device 10 obtains a task description 20 for constructing a target scene 30. The terminal device 10 parses the task description 20 based on the graph designer 11 to obtain a target scene graph corresponding to the task description 20. The target scene graph is then input into the scene generation model 12 in the terminal device 10 to obtain a target layout generated by the scene generation model 12 based on the object data and spatial relationships in the target scene graph. The terminal device 10 obtains object modeling corresponding to the object data from the database and generates a target scene 30 based on the object modeling and the target layout. The database can be a data set including object modeling stored in the terminal device 10, or a server linked to the terminal device 10 and including a data set including object modeling, storing multiple object models. Thus, by combining the graph designer and the scene generation model, the target scene graph is extracted from the task description, a corresponding target layout is generated based on the target scene graph, and a target scene corresponding to the target layout is obtained. This satisfies the user's need to only input the task description to obtain the desired target scene, thereby achieving the extraction of task description data based on the graph designer and combining the layout generated by a single model to generate the desired scene, thereby improving the efficiency, convenience, and accuracy of scene generation.

[0052] based on Figure 1The system architecture shown below will be combined with Figure 2 , a detailed introduction to the scene generation method provided in the embodiments of this specification is given.

[0053] See Figure 2 , which is a flow chart of a scene generation method provided in the embodiment of this specification. Figure 2 As shown, the method may include the following steps S101 to S104.

[0054] S101, obtaining a task description for constructing a target scenario;

[0055] In one embodiment, a task description input by a user into a terminal device is obtained for constructing a target scene. The target scene may be the scene desired to be constructed according to the task description input by the user, such as a bedroom. The task description may be an instruction input by the user into the terminal device to trigger scene construction, and may specifically be text data or image data.

[0056] If the task description is text data, it can be text that includes scene content, such as "A bedroom, the nightstand in the bedroom is to the right of the bed." If the task description is image data, it can be an image that includes scene content, such as a photo, stick figure, or diagram. To improve the accuracy of the constructed target scene, the task description can also include both text data and image data.

[0057] S102, parsing the task description based on the graph designer to obtain a target scene graph corresponding to the task description;

[0058] In one embodiment, after receiving the input task description, the task description is parsed based on a graph designer to obtain a target scene graph corresponding to the task description. The graph designer may be a visual language model used to parse the task description to determine the scene content included in the task description. In order to improve the accuracy of the obtained target scene graph, the graph designer may be a visual language model enhanced based on thought chaining. The target scene graph may be data indicating the content of the scene corresponding to the task description obtained after the task description is parsed. Specifically, it may include object data in the task description, as well as spatial relationships between object data, etc.

[0059] Specifically, the graph designer may obtain the target scene graph by determining a root node in the task description, determining other nodes corresponding to the task description based on the root node, and the spatial relationship between the other nodes, to obtain the target scene graph corresponding to the task description.

[0060] The object data may be objects included in the task description, such as a bed, a bedside table, etc. The spatial relationship between the object data may be the distance, relative direction, etc. between the object data.

[0061] S103, inputting the target scene graph into the trained scene generation model, so that the scene generation model generates a target layout based on the object data and spatial relationships;

[0062] In one embodiment, the target scene graph output by the graph designer is input into a pre-trained scene generation model, so that the scene generation model makes inferences based on the object data in the target scene graph and the spatial relationships between the object data, adjusts the positions of the respective object data, and generates a target layout corresponding to the target scene graph. The scene generation model can be a model capable of generating a target layout based on the target scene graph, such as a hybrid diffusion model. The target scene graph can be a framework representing the object data and spatial relationships generated by the scene generation model based on the inference of the target scene graph.

[0063] The target layout output by the scene generation model includes, in addition to the object data in the target scene graph and the spatial relationship corresponding to the object data, the object parameters of the object data. Object parameters can include parameters such as object size and object orientation. Object size can be the size of the object data in the target layout determined by the scene generation model based on the object data. For example, the dimensions of the bed in the target layout are 1.8 meters in width, 2 meters in length, and 0.5 meters in height. Object orientation can be the placement direction of the object data in the target layout.

[0064] S104, modeling the object corresponding to the object data obtained in the database, and generating a target scene based on the object modeling and the target layout;

[0065] In one embodiment, based on the object data included in the target layout, object models matching the object data are retrieved from a database. The object models are placed according to the target layout, and object parameters such as the size and orientation of the object models are adjusted based on the object size of the object data in the target layout. A target scene is generated and output. The object models can be models corresponding to the object data stored in the database, and the scene is constructed by combining the object models. The target scene can be a scene constructed based on the object models and conforming to the task description entered by the user.

[0066] In an embodiment of the present specification, the task description is parsed and inferred to obtain a target scene graph including object data and spatial relationships corresponding to the task description, and the object parameters corresponding to the object data are inferred based on the target scene graph according to the scene generation model to generate a target layout. Then, the target scene corresponding to the task description is generated according to the object modeling and the target layout. Thus, the target scene graph is extracted from the task description by combining the graph designer and the scene generation model, the corresponding target layout is generated according to the target scene graph, and the target scene corresponding to the target layout is obtained. This satisfies the requirement that the user only needs to input the task description to obtain the required target scene, thereby realizing the extraction of the task description data according to the graph designer and combining the layout generated by a single model to generate the required scene, thereby improving the efficiency, convenience and accuracy of scene generation.

[0067] See Figure 3 , which is a flow chart of a scene generation method provided in the embodiment of this specification. Figure 3 As shown, the method may include the following steps S201 to S209.

[0068] S201, obtaining a task description for constructing a target scenario;

[0069] In one embodiment, a task description input by a user into a terminal device is obtained for constructing a target scene. The target scene may be the scene desired to be constructed according to the task description input by the user, such as a bedroom. The task description may be an instruction input by the user into the terminal device to trigger scene construction, and may specifically be text data or image data.

[0070] If the task description is text data, it can be text that includes scene content, such as "A bedroom, the nightstand in the bedroom is to the right of the bed." If the task description is image data, it can be an image that includes scene content, such as a photo, stick figure, or diagram. To improve the accuracy of the constructed target scene, the task description can also include both text data and image data.

[0071] It should be noted that to facilitate user input of the task description, a guiding statement can be displayed on the terminal device operated by the user. For example, it can be "Enter the scene content you want to create." If the task description only includes image data, the default is to identify the content included in the image data and generate the target scene corresponding to the image data.

[0072] S202, parsing the task description based on the graph designer to obtain the root node in the task description;

[0073] In one embodiment, after receiving the input task description, the terminal device parses the task description based on a graph designer to determine the root node in the task description. The graph designer may be a visual language model for parsing the task description to determine the scene content included in the task description.

[0074] To improve the accuracy of the generated target scene graph, the graph designer can be a visual language model enhanced by thought chaining. A thought chain is a process of deduction based on logical relationships. Using this visual language model enhanced by thought chaining as a graph designer to parse the task description accurately identifies the root node in the task description, allowing subsequent retrieval of the remaining content within the task description based on the root node.

[0075] The root node can be the core data in the task description. For example, if the task description is "A bedroom, with a nightstand to the right of the bed," the root node could be "Bed." It's understandable that the purpose of determining the root node is to derive other content in the task description from it. Therefore, to improve derivation efficiency, one feasible approach is to use the object at the center of the scene as the root node. Other methods for determining the root node are also possible, and the specific setting can be determined based on actual circumstances.

[0076] S203, gradually acquiring other nodes in the task description based on the root node;

[0077] In one embodiment, after determining the root node corresponding to the task description, the graph designer progressively retrieves other nodes in the task description based on the root node. It is understood that the root node and other nodes may be object data corresponding to the task description, where the object data represents objects that may be included in the task description. Other nodes may be nodes included in the task description other than the root node.

[0078] For example, the task description is "a bedroom, the bedside table in the bedroom is on the right side of the bed", the root node is "bed", and the other nodes extracted are "bedside table" and "bedroom".

[0079] S204, sequentially determining the spatial relationship between the root node and each other node, and between each other node;

[0080] In one embodiment, after obtaining the root node and other nodes, the spatial relationships between the root node and other nodes, and between other nodes and other nodes, are sequentially determined. The spatial relationships may be parameters indicating spatial concepts, such as relative directions and relative distances between the root node and other nodes, and between other nodes and other nodes.

[0081] It can be understood that in addition to the spatial relationship extracted according to the task description, the corresponding spatial relationship is also derived based on other nodes, so that the spatial relationship between the root node and other nodes conforms to the real logic while satisfying the task description, and there will be no situation that does not conform to the real logic, such as the bedside table being placed above the bed.

[0082] For example, according to the task description, the spatial relationship between the root node "Bed" and another node "Nightstand" is determined to be "Nightstand is to the right of the bed." Since bedstands and beds are typically placed right next to each other in real life, the spatial relationship between "Bed" and "Nightstand" includes "Nightstand is right next to the bed" in addition to "Nightstand is to the right of the bed." The specific content of the spatial relationship can be set during the graph designer training process based on actual situations.

[0083] S205, converting the text data representing the root node, other nodes, and spatial relationships using a preset format conversion method to obtain a target scene graph corresponding to the task description;

[0084] In one embodiment, after obtaining the object data including the root node and other nodes, and the spatial relationship corresponding to the object data, the text data of the object data and the spatial relationship is format converted based on the preset format conversion method pre-set in the graph designer to obtain the target scene graph corresponding to the task description. The target scene graph can be data that represents the object data corresponding to the task description and the spatial relationship corresponding to the object data, and can be recognized and processed by the scene generation model. The preset format conversion method can be a data processing method with data format conversion, for example, it can be a method of converting the object data and the spatial relationship using regular expressions. It can be understood that since the target scene graph is obtained by converting the format using regular expressions based on the text data, the data type of the target scene graph is still a text type.

[0085] Furthermore, in order to enable the graph designer to accurately obtain the target scene graph according to the task description, rules can be set in advance for the graph designer so that the visual language model based on the thinking chain can accurately deduce the corresponding target scene graph based on the text data and / or image data. Specifically, the set rules can be: creating a prompt module in the graph designer, generating information recognition prompts for the task description based on the prompt module, and the information recognition prompts are used to prompt the graph designer to obtain object data and the spatial relationship corresponding to the object data based on the task description. The prompt module can be a module for providing data extraction or data inference instructions to the visual language model. It is understandable that the visual language model can be an existing AI (Artificial Intelligence) question-and-answer tool, and after enhancing the visual language model with the thinking chain, it can achieve the efficient data processing capabilities of the visual language model and the data deduction capabilities of the thinking chain to provide guidance to the visual language model, so as to obtain a graph designer that can output object data and spatial relationships according to the task description.

[0086] Optionally, the rules set by the graph designer may also include: setting a preset format conversion method for the graph designer, the preset format conversion method is used to convert text data into a scene graph. It is understandable that in order to be able to process the data output by the graph designer according to the scene generation model, it is necessary to convert the format of the object data and spatial relationships. Therefore, a preset format conversion method is set in the graph designer in advance, such as using a regular expression to convert the object data and spatial relationships into a scene graph using a preset regular expression. It should be noted that the conversion format corresponding to the preset format conversion method and the conversion method can be set according to actual conditions.

[0087] S206, inputting the target scene graph into the trained scene generation model, so that the scene generation model adds object data in the target scene graph to the initial layout;

[0088] In one embodiment, the target scene graph output by the graph designer is input into a pre-trained scene generation model. After the scene generation model identifies object data in the target scene graph, it generates an initial layout and adds the object data to the initial layout. The scene generation model can be a model capable of generating a target layout based on the target scene graph, such as a hybrid diffusion model. The initial layout can be a blank layout to facilitate the addition and adjustment of added object data to the initial layout.

[0089] It is understandable that, in addition to being able to identify the first object data included in the target scene graph, the scene generation model can also infer the second object data based on the identified first object data. The object data may be the data of an object determined according to the target scene graph, the object data may include the first object data and the second object data, the first object data may be the object data included in the target scene graph, and the second object data may be the object data inferred by the scene generation model based on the first object data in the target scene graph, and is used to supplement the object data of the scene. The second object data may be the object data derived from the first object data, so that when a scene is subsequently generated based on the first object data and the second object data, the generated scene conforms to the logic in reality.

[0090] For example, if the task description is "a bedroom, where the nightstand is to the right of the bed," the first object data extracted is "bed," "nightstand," and "bedroom." Based on the first object data of "bed," the corresponding second object data of "mattress," "pillow," and "quilt" can be derived. Further deduction based on the first object data of "bedroom" yields second object data of "lamp" and "wardrobe."

[0091] S207, using the scene generation model to adjust object parameters of the object data and the position of the object data in the initial layout based on the spatial relationship, to generate a target layout corresponding to the target scene graph;

[0092] In one embodiment, a scene generation model is used to adjust the position and object parameters of the object data in the initial scene graph based on the spatial relationship corresponding to the object data in the target scene graph, so that the object size, object orientation, and position between the object data in the initial layout meet the target scene graph. The object parameters can be physical parameters that characterize the object data, such as the object size or object orientation of the object data. The object size can be data that characterizes the length, width, and height of the object data in the initial layout. The object orientation can be data that characterizes the placement direction of the object data in the initial layout, such as a bedside table against a wall with the bed and the bedside table facing the same direction.

[0093] Specifically, the object size of the object data may be adjusted by inferring the object size based on the object types in the object data and the differences in types between the object data. For example, if the object data includes a bed and a nightstand, then based on real-life experience, it can be determined that the length and width of the bed are greater than the length and width of the nightstand.

[0094] In order to ensure that the sizes of objects in the generated target scene are consistent with those in real life, a scene generation model is used to adjust the object sizes in the initial layout of the object data according to the object types between the object data, thereby generating a target layout corresponding to the target scene graph.

[0095] For example, according to the task description, the spatial relationship between the first object data "bed" and the first object data "bedside table" is determined to be "the bedside table is on the right side of the bed". If "wardrobe" is derived when deriving the second object data, then since the "wardrobe" is placed on the left side of the "bed", the spatial relationship between the "wardrobe" and the "bed" can be "the wardrobe is on the right side of the bed", and a certain distance is set between the "wardrobe" and the "bed".

[0096] For example, Figure 4 As shown in , if the spatial relationship between object A and object B in the target scene graph is "object A is next to object B, and object A is on the right side of object B", then Figure 4 The position between object A and object B in the initial layout changes from "object A is far away from object B and is on the left side of object B" before adjustment to "object A is close to object B and is on the right side of object B" after adjustment. The object parameters such as the orientation and size of "object A" and "object B" in the initial layout are adjusted to obtain the target layout corresponding to the target scene graph.

[0097] It is understandable that when adjusting the position and size of object data in the initial layout, it is necessary to ensure that the object data does not overlap, so as to avoid phenomena that are inconsistent with the concept of reality, such as clipping.

[0098] Furthermore, the scene generation model can be trained by inputting a sample scene graph into the scene generation model and determining sample object data and sample spatial relationships within the sample scene graph. The sample spatial relationships can be data representing the relative distances and relative orientations between the sample object data. Based on the sample object data, noise object data is added to the initial layout, and the noise object data is positionally adjusted, replaced, or deleted until a test layout is generated. The noise objects can be data added to the initial layout to test the scene generation model's object denoising capabilities. It should be noted that the noise objects can be data generated by sampling the object structures corresponding to real-world objects. To avoid data penetration of noise objects during model training, constrained sampling is employed during sampling, ensuring that the sample objects maintain their physical properties during model training. The object denoising capability can represent the scene generation model's ability to adjust the position and object parameters of object data. The test layout can be the layout generated by the scene generation model after denoising the noise objects during training. The test layout is compared with the sample layout to obtain a loss function for the scene generation model. If the loss function indicates convergence of the scene generation model, the trained scene generation model is obtained. The loss function can be a parameter that indicates the difference between the test layout and the sample layout. A larger loss function indicates a larger difference between the test layout and the sample layout. Therefore, if the loss function is less than a preset value, the difference between the test layout and the sample layout is considered small enough, the scenario generation model has converged, and the scenario generation model training is complete.

[0099] It should be noted that when the scene generation model denoises noise objects, different noise objects are added for different sample scene images. Noise objects can be added all at once or gradually, depending on the actual situation.

[0100] Optionally, during scene generation model training, to improve model training efficiency, one feasible approach is to train the scene generation model while maintaining the sample object parameters of the sample object data according to a first probability. Alternatively, the scene generation model can be trained while maintaining the spatial relationships corresponding to the sample object data according to a second probability. This reduces the amount of data required for model training and improves model training efficiency. For example, during scene generation model training, a 20% probability of not adjusting sample object parameters and a 20% probability of not adjusting the spatial relationships of sample objects can be set. Specific settings can be made based on actual circumstances.

[0101] S208, obtaining object modeling corresponding to the object data in a database according to the object data and the object size of the object data;

[0102] In one embodiment, after obtaining the target layout output by the scene generation model, a search is performed in the database based on the object type and object size of the object data to obtain an object model that matches the object type and has a similar object size ratio. The object type can be the object type that characterizes the object data. For example, if the object data is a bed, the object type corresponding to the object data is a bed. By querying the set corresponding to the object type in the database, the object type can be quickly located in the database, improving the efficiency of obtaining object modeling. The object model can be a model corresponding to the object data stored in the database, and the scene is constructed by combining the object modeling.

[0103] It is understood that the same object data can correspond to one or more object models. If the object data corresponds to multiple object models, the object modeling method for obtaining the object data can be to randomly select an object model or to select the object model with the highest evaluation among all object models. The object modeling evaluation can be based on the algorithm's score for the modeling output, or can be determined based on user usage frequency, or the user's score for the object modeling can be used as the modeling evaluation.

[0104] S209, adding the object modeling to the initial scene according to the target layout, adjusting the modeling parameters of the object modeling in the initial scene based on the object parameters of the object data, and adjusting the position of the object modeling in the initial scene based on the position of the object data, to generate a target scene corresponding to the task description;

[0105] In one embodiment, the acquired object modeling is added to the initial scene, and the modeling parameters of the object modeling are adjusted according to the object parameters in the target layout. Specifically, the modeling size of the object modeling is adjusted according to the object size in the target layout, the orientation of the object modeling in the initial scene is adjusted according to the object orientation in the target layout, and the position of the object modeling in the initial scene is adjusted based on the position of the object data in the target layout, so as to obtain the target scene corresponding to the task description. The initial scene can be a blank scene space. The modeling size can be the size of the object modeling in the initial scene. It is understandable that the initial size of the object modeling acquired from the database is fixed, so the modeling size of the object modeling needs to be adjusted so that the proportion of each object modeling in the initial scene conforms to the content displayed by the target layout. The target scene can be a scene generated in accordance with the task description, and the scene includes the objects included in the task description, as well as the spatial relationship between the objects.

[0106] It should be noted that, since the object modeling in the target scene is 3D modeling, the target scene generated based on the object modeling can be a 3D modeling in which the user can freely adjust the viewing angle, such as the scene layout in a 3D game.

[0107] For example, Figure 5 As shown, Figure 5 According to the target layout, the corresponding object data "bed", "bedside table", "wardrobe" and "lamp" are obtained and the corresponding object models are modeled. After adjusting the position and size of the object models, the target scene corresponding to the target layout is obtained.

[0108] In an embodiment of the present specification, by parsing and inferring the task description, a target scene graph including object data and spatial relationships corresponding to the task description is obtained, and the object parameters corresponding to the object data are inferred based on the target scene graph according to the scene generation model, a target layout is generated, and then a target scene corresponding to the task description is generated according to the object modeling and target layout. Thus, by combining the graph designer and the scene generation model, the target scene graph is extracted from the task description, the corresponding target layout is generated according to the target scene graph, and the target scene corresponding to the target layout is obtained, so that the user only needs to input the task description to obtain the required target scene, thereby realizing the extraction of the task description data according to the graph designer and combining the layout generated by a single model to generate the required scene, thereby improving the efficiency, convenience and accuracy of scene generation. In addition, by inferring other nodes in the task description through the graph designer, and inferring object data not included in the task description through the scene generation model, and the object parameters corresponding to each object data, the object data corresponding to the task description is improved, and the accuracy of the generated target scene is improved. Further, by training the graph designer and the scene generation model, the accuracy and reliability of the graph designer and the scene generation model in generating the target scene according to the task description are improved.

[0109] based on Figure 1 The system architecture shown below will be combined with Figure 6 , the board testing device provided in the embodiment of this specification is introduced in detail. It should be noted that, Figure 6 The scene generation device in the embodiment of this specification is used to execute Figures 2 to 5 For the convenience of explanation, only the part related to the embodiment of this specification is shown. For the specific technical details not disclosed, please refer to the embodiment of this specification. Figures 2 to 5 The embodiment shown.

[0110] See Figure 6 , is a schematic diagram of the structure of a scene generation device provided in the embodiment of this specification. Figure 6 As shown, the scene generation device 1 of the embodiment of this specification may include: a description acquisition unit 11, a scene graph generation unit 12, a layout generation unit 13 and a scene generation unit 14.

[0111] Description acquisition unit 11, used for description acquisition unit, for acquiring task description for constructing target scenario;

[0112] A scene graph generating unit 12 is configured to parse the task description based on a graph designer to obtain a target scene graph corresponding to the task description, wherein the target scene graph includes object data in the task description and spatial relationships between the object data;

[0113] a layout generation unit 13, configured to input the target scene graph into a trained scene generation model to obtain a target layout generated by the scene generation model based on the object data and the spatial relationship;

[0114] The scene generation unit 14 is configured to model the object corresponding to the object data acquired from the database, and generate the target scene according to the object modeling and the target layout.

[0115] Optionally, the scene graph generation unit 12 is further configured to:

[0116] Parsing the task description based on a graph designer to obtain a root node in the task description;

[0117] Based on the root node, other nodes corresponding to the task description and the spatial relationship between each of the other nodes and the root node are determined to obtain a target scene graph corresponding to the task description.

[0118] Optionally, the scene graph generation unit 12 is further configured to:

[0119] Gradually acquiring other nodes in the task description according to the root node, where the other nodes are nodes in the task description other than the root node;

[0120] sequentially determining the spatial relationship between the root node and each of the other nodes, and between each of the other nodes;

[0121] The text data representing the root node, the other nodes and the spatial relationship is format converted using a preset format conversion method to obtain a target scene graph corresponding to the task description.

[0122] Optionally, the layout generating unit 13 is further configured to:

[0123] Inputting the target scene graph into a trained scene generation model so that the scene generation model adds object data in the target scene graph to an initial layout, the object data including first object data and second object data, wherein the first object data is object data included in the target scene graph, and the second object data is inferred based on the target scene graph and is used to supplement the object data of the scene;

[0124] The scene generation model is used to adjust object parameters of the object data and positions of the object data in the initial layout based on the spatial relationship to generate a target layout corresponding to the target scene graph.

[0125] Optionally, the scene generating unit 14 is further configured to:

[0126] Obtaining object modeling corresponding to the object data in a database according to the object data and the object size of the object data;

[0127] The object modeling is added to the initial scene according to the target layout, the modeling size of the object modeling in the initial scene is adjusted based on the object size, and the position of the object modeling in the initial scene is adjusted based on the position of the object data to generate a target scene corresponding to the task description.

[0128] Optionally, the scene generation device 1 further includes a diagram designer setting unit 15, which is used to:

[0129] Creating a prompt module in the graph designer, and generating an information identification prompt for the task description based on the prompt module, wherein the information identification prompt is used to prompt the graph designer to obtain the object data and the spatial relationship corresponding to the object data based on the task description;

[0130] A preset format conversion mode of the graph designer is set, and the preset format conversion mode is used to convert the text data into a scene graph.

[0131] Optionally, the scene generation device 1 further includes a model training unit 16, which is used to:

[0132] Inputting the sample scene graph into the scene generation model to determine sample object data and sample spatial relationships in the sample scene graph;

[0133] adding noise object data to the initial layout based on the sample object data, and adjusting the position, replacing objects, or deleting objects of the noise object data until a test layout is generated;

[0134] The test layout and the sample layout are compared to obtain a loss function of the scene generation model. If the loss function indicates that the scene generation model has converged, a trained scene generation model is obtained.

[0135] Optionally, the model training unit 16 is further configured to:

[0136] Training the scene generation model while setting sample object parameters of the sample object data unchanged according to the first probability;

[0137] The scene generation model is trained while the spatial relationship corresponding to the sample object data remains unchanged according to the second probability setting.

[0138] In an embodiment of the present specification, by parsing and inferring the task description, a target scene graph including object data and spatial relationships corresponding to the task description is obtained, and the object parameters corresponding to the object data are inferred based on the target scene graph according to the scene generation model, a target layout is generated, and then a target scene corresponding to the task description is generated according to the object modeling and target layout. Thus, by combining the graph designer and the scene generation model, the target scene graph is extracted from the task description, the corresponding target layout is generated according to the target scene graph, and the target scene corresponding to the target layout is obtained, so that the user only needs to input the task description to obtain the required target scene, thereby realizing the extraction of the task description data according to the graph designer and combining the layout generated by a single model to generate the required scene, thereby improving the efficiency, convenience and accuracy of scene generation. In addition, by inferring other nodes in the task description through the graph designer, and inferring object data not included in the task description through the scene generation model, and the object parameters corresponding to each object data, the object data corresponding to the task description is improved, and the accuracy of the generated target scene is improved. Further, by training the graph designer and the scene generation model, the accuracy and reliability of the target scene graph generated by the graph designer and the scene generation model according to the task description are improved.

[0139] The embodiment of this specification also provides a computer storage medium that can store multiple program instructions, which are suitable for being loaded and executed by a processor as described above. Figure 1-Figure 5 The method steps of the embodiment shown, the specific execution process can be found in Figure 1-Figure 5 The detailed description of the illustrated embodiment will not be repeated here.

[0140] See Figure 7 , is a schematic diagram of the structure of an electronic device provided in the embodiment of this specification. Figure 7 As shown, the electronic device 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, an input / output interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. Figure 7 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, an input and output interface module, and a scene generation application.

[0141] exist Figure 7 In the electronic device 1000 shown, the input / output interface 1003 is mainly used to provide an input interface for the user and obtain data input by the user.

[0142] In one embodiment, the processor 1001 may be configured to call a scene generation application stored in the memory 1005 and specifically perform the following operations:

[0143] Obtain a task description for constructing the target scenario;

[0144] Parsing the task description based on a graph designer to obtain a target scene graph corresponding to the task description, the target scene graph including object data in the task description and spatial relationships between the object data;

[0145] Inputting the target scene graph into a trained scene generation model, so that the scene generation model generates a target layout based on the object data and the spatial relationship;

[0146] The object data obtained in the database corresponds to object modeling, and the target scene is generated according to the object modeling and the target layout.

[0147] Optionally, when the processor 1001 parses the task description based on the graph designer to obtain the target scene graph corresponding to the task description, the processor 1001 specifically performs the following operations:

[0148] Parsing the task description based on a graph designer to obtain a root node in the task description;

[0149] Based on the root node, other nodes corresponding to the task description and the spatial relationship between each of the other nodes and the root node are determined to obtain a target scene graph corresponding to the task description.

[0150] Optionally, when the processor 1001 determines other nodes corresponding to the task description based on the root node, and the spatial relationship between each of the other nodes and the root node, to obtain a target scene graph corresponding to the task description, the processor 1001 specifically performs the following operations:

[0151] Gradually acquiring other nodes in the task description according to the root node, where the other nodes are nodes in the task description other than the root node;

[0152] sequentially determining the spatial relationship between the root node and each of the other nodes, and between each of the other nodes;

[0153] The text data representing the root node, the other nodes and the spatial relationship is format converted using a preset format conversion method to obtain a target scene graph corresponding to the task description.

[0154] Optionally, when inputting the target scene graph into a trained scene generation model to obtain a target layout generated by the scene generation model based on the object data and the spatial relationship, the processor 1001 specifically performs the following operations:

[0155] Inputting the target scene graph into a trained scene generation model so that the scene generation model adds object data in the target scene graph to an initial layout, the object data including first object data and second object data, wherein the first object data is object data included in the target scene graph, and the second object data is inferred based on the target scene graph and is used to supplement the object data of the scene;

[0156] The scene generation model is used to adjust object parameters of the object data and positions of the object data in the initial layout based on the spatial relationship to generate a target layout corresponding to the target scene graph.

[0157] Optionally, when executing object modeling corresponding to the object data obtained from the database and generating the target scene according to the object modeling and the target layout, the processor 1001 specifically performs the following operations:

[0158] Obtaining object modeling corresponding to the object data in a database according to the object data and the object size of the object data;

[0159] The object modeling is added to the initial scene according to the target layout, the modeling parameters of the object modeling in the initial scene are adjusted based on the object parameters of the object data, and the position of the object modeling in the initial scene is adjusted based on the position of the object data to generate a target scene corresponding to the task description.

[0160] Optionally, the processor 1001 further performs the following operations:

[0161] Creating a prompt module in the graph designer, and generating an information identification prompt for the task description based on the prompt module, wherein the information identification prompt is used to prompt the graph designer to obtain the object data and the spatial relationship corresponding to the object data based on the task description;

[0162] A preset format conversion mode of the graph designer is set, and the preset format conversion mode is used to convert the text data into a scene graph.

[0163] Optionally, the processor 1001 further performs the following operations:

[0164] Inputting the sample scene graph into the scene generation model to determine sample object data and sample spatial relationships in the sample scene graph;

[0165] adding noise object data to the initial layout based on the sample object data, and adjusting the position, replacing objects, or deleting objects of the noise object data until a test layout is generated;

[0166] The test layout and the sample layout are compared to obtain a loss function of the scene generation model. If the loss function indicates that the scene generation model has converged, a trained scene generation model is obtained.

[0167] Optionally, the processor 1001 further performs the following operations:

[0168] Training the scene generation model while setting sample object parameters of the sample object data unchanged according to the first probability;

[0169] The scene generation model is trained while the spatial relationship corresponding to the sample object data remains unchanged according to the second probability setting.

[0170] In an embodiment of the present specification, by parsing and inferring the task description, a target scene graph including object data and spatial relationships corresponding to the task description is obtained, and the object parameters corresponding to the object data are inferred based on the target scene graph according to the scene generation model, a target layout is generated, and then a target scene corresponding to the task description is generated according to the object modeling and target layout. Thus, by combining the graph designer and the scene generation model, the target scene graph is extracted from the task description, the corresponding target layout is generated according to the target scene graph, and the target scene corresponding to the target layout is obtained, so that the user only needs to input the task description to obtain the required target scene, thereby realizing the extraction of the task description data according to the graph designer and combining the layout generated by a single model to generate the required scene, thereby improving the efficiency, convenience and accuracy of scene generation. In addition, by inferring other nodes in the task description through the graph designer, and inferring object data not included in the task description through the scene generation model, and the object parameters corresponding to each object data, the object data corresponding to the task description is improved, and the accuracy of the generated target scene is improved. Further, by training the graph designer and the scene generation model, the accuracy and reliability of the target scene graph generated by the graph designer and the scene generation model according to the task description are improved.

[0171] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0172] The above disclosure is only a preferred embodiment of this specification, and certainly cannot be used to limit the scope of rights of this specification. Therefore, equivalent changes made according to the claims of this specification are still within the scope covered by this specification.

Claims

1. A scene generation method, characterized in that: The method comprises: Obtain a task description for constructing the target scenario; Parsing the task description based on a graph designer to obtain a target scene graph corresponding to the task description, the target scene graph including object data in the task description and spatial relationships between the object data; Inputting the target scene graph into a trained scene generation model, so that the scene generation model generates a target layout based on the object data and the spatial relationship; The object data obtained in the database corresponds to object modeling, and the target scene is generated according to the object modeling and the target layout.

2. The method according to claim 1, characterized in that The graph designer is used to parse the task description to obtain a target scene graph corresponding to the task description, including: Parsing the task description based on a graph designer to obtain a root node in the task description; Based on the root node, other nodes corresponding to the task description and the spatial relationship between each of the other nodes and the root node are determined to obtain a target scene graph corresponding to the task description.

3. The method according to claim 2, characterized in that The determining, based on the root node, other nodes corresponding to the task description and the spatial relationship between each of the other nodes and the root node to obtain a target scene graph corresponding to the task description includes: Gradually acquiring other nodes in the task description according to the root node, where the other nodes are nodes in the task description other than the root node; sequentially determining the spatial relationship between the root node and each of the other nodes, and between each of the other nodes; The text data representing the root node, the other nodes and the spatial relationship is format converted using a preset format conversion method to obtain a target scene graph corresponding to the task description.

4. The method according to claim 1, wherein Inputting the target scene graph into a trained scene generation model to obtain a target layout generated by the scene generation model based on the object data and the spatial relationship includes: Inputting the target scene graph into a trained scene generation model so that the scene generation model adds object data in the target scene graph to an initial layout, the object data including first object data and second object data, wherein the first object data is object data included in the target scene graph, and the second object data is inferred based on the target scene graph and is used to supplement the object data of the scene; The scene generation model is used to adjust object parameters of the object data and positions of the object data in the initial layout based on the spatial relationship to generate a target layout corresponding to the target scene graph.

5. The method according to claim 1, wherein The object modeling corresponding to the object data obtained in the database, and generating the target scene according to the object modeling and the target layout, include: Obtaining object modeling corresponding to the object data in a database according to the object data and the object size of the object data; The object modeling is added to the initial scene according to the target layout, the modeling parameters of the object modeling in the initial scene are adjusted based on the object parameters of the object data, and the position of the object modeling in the initial scene is adjusted based on the position of the object data to generate a target scene corresponding to the task description.

6. The method according to claim 1, characterized in that The method further comprises: Creating a prompt module in the graph designer, and generating an information identification prompt for the task description based on the prompt module, wherein the information identification prompt is used to prompt the graph designer to obtain the object data and the spatial relationship corresponding to the object data based on the task description; A preset format conversion mode of the graph designer is set, and the preset format conversion mode is used to convert the text data into a scene graph.

7. The method according to claim 1, characterized in that The method further comprises: Inputting the sample scene graph into the scene generation model to determine sample object data and sample spatial relationships in the sample scene graph; adding noise object data to the initial layout based on the sample object data, and adjusting the position, replacing objects, or deleting objects of the noise object data until a test layout is generated; The test layout and the sample layout are compared to obtain a loss function of the scene generation model. If the loss function indicates that the scene generation model has converged, a trained scene generation model is obtained.

8. The method according to claim 7, characterized in that The method further comprises: Training the scene generation model while setting sample object parameters of the sample object data unchanged according to the first probability; The scene generation model is trained while the spatial relationship corresponding to the sample object data remains unchanged according to the second probability setting.

9. A scene generation device, characterized in that: The device comprises: A description acquisition unit, used to acquire a task description for constructing a target scenario; A scene graph generating unit, configured to parse the task description based on a graph designer to obtain a target scene graph corresponding to the task description, wherein the target scene graph includes object data in the task description and spatial relationships between the object data; a layout generation unit, configured to input the target scene graph into a trained scene generation model to obtain a target layout generated by the scene generation model based on the object data and the spatial relationship; A scene generation unit is used to model the object corresponding to the object data obtained in the database, and generate the target scene according to the object modeling and the target layout.

10. A computer storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the steps of the method according to any one of claims 1 to 8.

11. An electronic device comprising: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method according to any one of claims 1 to 8.

Images