Complex indoor scene generation method and system based on natural language

Abstract

The present invention proposes a complex indoor scene generation method and system based on natural language, which belongs to the field of indoor scene generation, including: S1: constructing a house frame vector database, and retrieving the most matching house frame structure through natural language input; S2: according to the most matching house frame structure, the layout of each room is generated and optimized in turn, and finally the house layout is obtained; S3: by rendering the object model from multiple angles, a rendering image is obtained, and an object picture vector database and an object text vector database are constructed based on the rendering image, and the similarity between the object in the house layout and the object in the two vector databases is calculated to select the optimal object; S4: by rendering the four angles of the optimal object, the orientation distribution is calculated, and the front orientation of the object is obtained; according to the position and orientation information of the object in the house layout, the combination is placed to obtain the final three-dimensional indoor scene. The method of the present invention makes the generated indoor scene closer to actual needs.

Description

Technical Field

[0001] The invention belongs to the field of indoor scene generation, and in particular relates to a natural language-based complex indoor scene generation method and system. Background Art

[0002] Existing indoor scene generation methods can be divided into two categories: end-to-end direct generation and generation guided by a large language model. The former mainly relies on a pre-trained model, which directly generates new scene layouts and furniture shapes by learning layout information and three-dimensional object features from a large number of scene datasets. Typical datasets include 3D-Front, Matterport3D, and Scannet. The latter uses large language model technology to define or adjust the scene layout (DSL), and then selects and combines from the object model library according to these instructions to build indoor scenes.

[0003] The end-to-end direct generation method first relies on high-quality and diverse scene datasets, but in general, existing scene datasets are scarce, expensive to obtain, and difficult to label. Secondly, the generative method cannot guarantee the rationality of the house layout. There are errors such as objects exceeding the boundaries of the house and objects intersecting, and the quality of the generated object model cannot be guaranteed; the generation method guided by the large language model only supports the generation of a single room in a regular area, and the room types are limited, the flexibility is insufficient, and the gap with the real scene is large. At the same time, when the objects are retrieved and placed, the front orientation of the objects is not considered, resulting in unnatural or illogical final scene results. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a complex indoor scene generation method based on natural language, comprising the following steps:

[0005] Step S1: extract basic information and description of house frame data using a large language model, build a house frame vector database, and retrieve the best matching house frame structure through natural language input;

[0006] Step S2: according to the best matching house frame structure, basic information and room connection relationship of each room are obtained, and layout generation and optimization are performed on each room in turn, so as to finally obtain the house layout;

[0007] Step S3: Rendering the object model from multiple angles to obtain a rendering image, constructing an object image vector database and an object text vector database based on the rendering image, calculating similarities between the objects in the house layout and the object image vector database and the object text vector database, and selecting the optimal object;

[0008] Step S4: Rendering the four angles of the optimal object, calculating the orientation distribution, and obtaining the front orientation of the object; combining and placing the objects according to the position and orientation information of the objects in the house layout to obtain the final three-dimensional indoor scene.

[0009] Beneficial effects:

[0010] The present invention provides a complex indoor scene generation method based on natural language, which aims to automatically generate the whole house indoor scene through natural language input, without relying on large-scale data sets for training, thereby significantly saving time and cost. The present invention utilizes the prompt engineering and algorithm optimization of the large language model to ensure the rationality of the layout and avoid problems such as objects exceeding the boundary or overlapping each other. In addition, the present invention retrieves high-quality objects from a rich and diverse object model library, and adds an orientation function before combination to ensure that each object can be placed in the correct direction, further improving the authenticity and aesthetics of the final generated scene. The present invention not only improves the quality of the generated results, but also provides detailed annotation information, so that the generated indoor scene is closer to actual needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of a flow chart of a complex indoor scene generation method based on natural language according to the present invention;

[0012] Figure 2 Retrieve the schematic diagram of the process flow for the house frame structure;

[0013] Figure 3 This is a schematic diagram of the object retrieval process;

[0014] Figure 4 This is a schematic diagram of the object orientation process;

[0015] Figure 5 This is a structural block diagram of a natural language-based complex indoor scene generation system of the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0017] Embodiment 1

[0018] like Figure 1 As shown, a natural language-based complex indoor scene generation method provided by an embodiment of the present invention includes the following steps:

[0019] Step S1: extract basic information and description of house frame data using a large language model, build a house frame vector database, and retrieve the best matching house frame structure through natural language input;

[0020] Step S2: according to the most matching house frame structure, the basic information and room connection relationship of each room are obtained, and the layout of each room is generated and optimized in turn, and finally the house layout is obtained;

[0021] Step S3: Rendering the object model from multiple angles to obtain a rendering image, constructing an object image vector database and an object text vector database based on the rendering image, calculating similarities between the objects in the house layout and the object image vector database and the object text vector database, and selecting the optimal object;

[0022] Step S4: Render the four angles of the optimal object, calculate the orientation distribution, and obtain the front orientation of the object; combine and place the objects according to the position and orientation information of the objects in the house layout to obtain the final three-dimensional indoor scene.

[0023] In one embodiment, the above step S1: using a large language model to extract basic information and description of house frame data, constructing a house frame vector database, and retrieving the most matching house frame structure through natural language input specifically includes:

[0024] Step S11: First, pre-process the house frame data in JSON format to obtain basic information, including: house frame grid information, house size information and the relationship between houses, including door and window information; then design prompt words including room size, house style, room type and quantity, and use the large language model to obtain the house description;

[0025] Step S12: vectorizing the house description using a Transformer encoder, adding size information, and constructing a house framework vector database, specifically comprising: encoding each word in the house description into a 512-dimensional house word embedding vector by word embedding, and generating a house position code for each word by using a sine or cosine function of different frequencies, and the house position code is also a 512-dimensional vector; adding the house word embedding vector and the house position code to obtain a final house description vector, and adding a house size information vector (1-dimensional vector) to the end of the vector, thereby constructing a house framework vector database;

[0026] House location code The formula is as follows:

[0027] ;

[0028] ;

[0029] in, For each word, the corresponding token's position number in the sequence, Dimensions encoding the location of the house;

[0030] Step S13: Split the input text of the house information to be retrieved into multiple words, generate a text word embedding vector and a text position code in the same way as step S12, and add the two together to obtain a description vector of the house to be retrieved;

[0031] Step S14: Calculate the cosine similarity between the house description vector to be retrieved and the house description vector after removing the size information vector, obtain the three results with the closest similarity, and calculate the house area ratio respectively , select The result closest to 0 is the best match for the house frame structure:

[0032] ;

[0033] in, is the housing area, is the area of ​​the house corresponding to the closest house description vector.

[0034] like Figure 2 As shown, a schematic diagram of the house frame structure retrieval process is shown.

[0035] In one embodiment, the above step S2: according to the most matching house frame structure, obtaining the basic information and room connection relationship of each room and sequentially generating and optimizing the layout of each room, and finally obtaining the house layout, specifically includes:

[0036] Step S21: According to the basic information of each room in the house frame structure, design indoor object selection prompt words, select appropriate object categories according to the basic requirements of the room and the special needs of the user, and refer to the room size and the existing knowledge of the large model to provide reference size information for each object to meet the needs and functions of different rooms;

[0037] Step S22: Design the indoor scene layout and generate prompt words. According to the fixed object information in the room, including walls, doors and windows, the irregular corner information of the room and the selected object information, a preliminary room layout is generated, which is represented as a directed graph ,in, Represents each individual object, middle is the name of the object, and A description of the style and details of the object, Represents the size information of the object. and Represents the position and orientation information of an object, using nodes Represents the room frame structure, including walls, floors and ceilings; Indicates the relative spatial relationship between objects;

[0038] Step S23: Designing an indoor scene layout check prompt word, checking the preliminary room layout to see if there are any objects beyond the space range and whether the positions of objects overlap, and if so, automatically modifying the relative relationship between objects;

[0039] Step S24: Adopt an adaptive backtracking algorithm to optimize the preliminary room layout to obtain the room layout. The specific steps are: topologically sort each object in the preliminary room layout, and then calculate the node depth according to the topological sorting result. And group; for objects with a depth of d, a predefined set of feasible position calculation functions are used ,in, is a set of feasible position calculation functions, which calculates the placeable area of ​​the object and places it based on the randomly sampled positions of the normal distribution. If an object with a depth of d cannot be placed because it always collides with other objects during the placement process, all objects with a depth greater than d are removed, and the objects with a depth of d-1 are sampled and placed again, otherwise the depth is +1.

[0040] In one embodiment, the above step S3: by rendering the object model from multiple angles to obtain a rendering image, constructing an object image vector database and an object text vector database based on the rendering image, calculating the similarity between the object in the house layout and the object image vector database and the object text vector database, and selecting the optimal object, specifically includes:

[0041] Step S31: Rendering the object model in the existing three-dimensional object library from multiple angles to obtain a rendering image;

[0042] Step S32: Use the CLIP encoder to extract the image embedding vector from the rendering, and map the embedding vectors of all non-frontal renderings to the front and add them up to get the final object corresponding image vector , build an object image vector database:

[0043] ;

[0044] in, is the embedding vector of the front rendering, is the embedding vector of all non-frontal renderings;

[0045] For the rendering image, the object description is generated by using the prompt word, and the text vector is generated by adopting the method of step S13, and the three-dimensional vector of the object size is added to the end thereof to obtain the object text vector database;

[0046] Step S33: Extract the description and size information of the objects in the room layout, and use the method of step S32 to generate the corresponding object image vector and object text vector; perform cosine similarity calculations with the vectors in the object image vector database and the object text vector database respectively, and finally weight them to obtain the three most similar objects, and then calculate the object size ratios respectively , select the result closest to 0 as the optimal object;

[0047] ;

[0048] in, are the length, width and height information of the objects in the room layout, The length, width and height information of the object corresponding to the closest object description vector are respectively described.

[0049] like Figure 3 As shown in Figure 1, a schematic diagram of the object retrieval process is shown.

[0050] In one embodiment, the above step S4: by rendering the four angles of the optimal object, calculating the orientation distribution, and obtaining the front orientation of the object; combining and placing the object according to the position and orientation information of the object in the house layout to obtain the final three-dimensional indoor scene, specifically includes:

[0051] Step S41: Rendering the optimal object from four angles to obtain a rendering image of the optimal object;

[0052] Step S42: Each optimal object rendering image is passed through ResNet to output image features And orthogonally project it onto the sphere, then the feature and Convolution is performed with the learned filters on to generate the signal on SO(3), ultimately producing a detailed distribution of 3D rotations;

[0053] Step S43: Select the orientation angle with the highest probability and calculate the orientation angle of the front of the object As the object faces forward:

[0054] ;

[0055] in, The camera angle corresponding to the object rendering, Predict the object angle corresponding to the rendering with the highest probability.

[0056] like Figure 4As shown, a schematic diagram of the object orientation process is shown.

[0057] The present invention designs a complex indoor scene generation method specifically for irregular areas. The method combines the real whole-house frame information to ensure that the generated whole-house results are closer to real life. In order to further improve the rationality of the scene generation results, the present invention adds a prompt project in the layout generation to check and iteratively modify the layout, and uses an algorithm to optimize the scene layout. Finally, a directional module is added before the objects are placed to optimize the orientation and position arrangement of each object. Through the above method, the quality of indoor scene generation can be significantly improved. Through specific experimental results (as shown in the following table), it is found that the method of the present invention shows obvious advantages in many aspects, the generated results are more reasonable, and most of the possible errors in the layout are eliminated. At the same time, it is better than other methods in multiple indicators (authenticity, functionality, layout and overall atmosphere), as shown in Table 1.

[0058] Table 1 Comparison of the results of the present invention with those of other methods

[0059]

[0060] Embodiment 2

[0061] like Figure 5 As shown, an embodiment of the present invention provides a complex indoor scene generation system based on natural language, including the following modules:

[0062] The module 51 for obtaining the house frame structure is used to extract the basic information and description of the house frame data by using the large language model, construct a house frame vector database, and retrieve the best matching house frame structure through natural language input;

[0063] The module 52 for obtaining the house layout is used to obtain the basic information and room connection relationship of each room according to the most matching house frame structure, and to generate and optimize the layout of each room in turn, and finally obtain the house layout;

[0064] The optimal object acquisition module 53 is used to render the object model from multiple angles to obtain a rendering image, construct an object image vector database and an object text vector database based on the rendering image, calculate the similarity between the object in the house layout and the object image vector database and the object text vector database, and select the optimal object;

[0065] The module 54 for constructing a three-dimensional indoor scene is used to render the four angles of the optimal object, calculate the orientation distribution, and obtain the front orientation of the object; according to the position and orientation information of the object in the house layout, the final three-dimensional indoor scene is obtained by combining and placing.

Claims

1. A complex indoor scene generation method based on natural language, characterized in that: include: Step S1: extract basic information and description of house frame data using a large language model, build a house frame vector database, and retrieve the best matching house frame structure through natural language input; Step S2: according to the best matching house frame structure, basic information and room connection relationship of each room are obtained, and layout generation and optimization are performed on each room in turn, so as to finally obtain the house layout; Step S3: Rendering the object model from multiple angles to obtain a rendering image, constructing an object image vector database and an object text vector database based on the rendering image, calculating similarities between the objects in the house layout and the object image vector database and the object text vector database, and selecting the optimal object; Step S4: Rendering the four angles of the optimal object, calculating the orientation distribution, and obtaining the front orientation of the object; combining and placing the objects according to the position and orientation information of the objects in the house layout to obtain the final three-dimensional indoor scene.

2. The natural language-based complex indoor scene generation method according to claim 1, characterized in that: The step S1: using a large language model to extract basic information and description of house frame data, constructing a house frame vector database, and retrieving the most matching house frame structure through natural language input, specifically includes: Step S11: First, pre-process the house frame data in JSON format to obtain basic information, including: house frame grid information, house size information and the relationship between houses, including door and window information; then design prompt words including room size, house style, room type and quantity, and use the large language model to obtain the house description; Step S12: vectorizing the house description by using a Transformer encoder, adding size information, and constructing a house framework vector database, specifically comprising: encoding each word in the house description into a house word embedding vector by word embedding, and generating a house position code for each word by using a sine or cosine function of different frequencies; adding the house word embedding vector and the house position code to obtain a final house description vector, and adding a house size information vector at the end of the vector; Step S13: Split the input text of the house information to be retrieved into multiple words, generate a text word embedding vector and a text position code in the same way as step S12, and add the two together to obtain a description vector of the house to be retrieved; Step S14: Calculate the cosine similarity between the house description vector to be retrieved and the house description vector after removing the size information vector, obtain the three results with the closest similarity, and calculate the house area ratio respectively. , select The result closest to 0 is the best match for the house frame structure: ； in, is the area of ​​the house, is the area of ​​the house corresponding to the closest house description vector.

3. The natural language-based complex indoor scene generation method according to claim 2, characterized in that: The step S2: according to the best matching house frame structure, the basic information and room connection relationship of each room are obtained and the layout of each room is generated and optimized in turn, and finally the house layout is obtained, which specifically includes: Step S21: according to the basic information of each room in the housing frame structure, design indoor object selection prompt words, select appropriate object categories according to the basic requirements of the room and the special needs of the user, and refer to the room size and the existing knowledge of the large model itself to provide reference size information of each object to meet the needs and functions of different rooms; Step S22: Design the indoor scene layout and generate prompt words. According to the fixed object information in the room, including walls, doors and windows, the irregular corner information of the room and the selected object information, a preliminary room layout is generated, which is represented as a directed graph ,in, Represents each individual object, middle is the name of the object, and A description of the style and details of the object, Represents the size information of the object. and Represents the position and orientation information of an object, using nodes Represents the room frame structure, including walls, floors and ceilings; Indicates the relative spatial relationship between objects; Step S23: Designing an indoor scene layout check prompt word, checking whether there are objects beyond the space range and whether the positions of objects overlap with each other for the preliminary room layout, and if so, automatically modifying the relative relationship between objects; Step S24: Adopting an adaptive backtracking algorithm to optimize the preliminary room layout to obtain the room layout, the specific steps are: topologically sorting each object in the preliminary room layout, and then calculating the node depth according to the topological sorting result. And group; for objects with a depth of d, a predefined set of feasible position calculation functions are used ,in, is a set of feasible position calculation functions, which calculates the placeable area of ​​the object and places it based on the randomly sampled positions of the normal distribution. If an object with a depth of d cannot be placed because it always collides with other objects during the placement process, all objects with a depth greater than d are removed, and the objects with a depth of d-1 are sampled and placed again, otherwise the depth is +1.

4. The natural language-based complex indoor scene generation method according to claim 3, characterized in that: The step S3: rendering the object model from multiple angles to obtain a rendering image, constructing an object image vector database and an object text vector database based on the rendering image, calculating similarities between the objects in the house layout and the object image vector database and the object text vector database, and selecting the optimal object, specifically includes: Step S31: performing multi-angle rendering on the object model to obtain a rendering image; Step S32: extract the image embedding vector of the rendering, and map the embedding vectors of all non-front renderings to the front and add them up to obtain the final object corresponding image vector , build an object image vector database: ； in, is the embedding vector of the front rendering image, are the embedding vectors of all the non-frontal renderings; For the rendering, the object description is generated by using the prompt word, and a text vector is generated by adopting the method of step S13, and a three-dimensional vector of the object size is added to the end thereof to obtain an object text vector database; Step S33: Extract the description and size information of the objects in the room layout, and use the method of step S32 to generate the corresponding object image vector and object text vector; perform cosine similarity calculations with the vectors in the object image vector database and the object text vector database, and finally weight the three closest objects, and then calculate the object size ratios respectively. , select the result closest to 0 as the optimal object; ； in, are the length, width and height information of the objects in the room layout, The length, width and height information of the object corresponding to the closest object description vector are respectively described.

5. The natural language-based complex indoor scene generation method according to claim 4, characterized in that: The step S4: by rendering the four angles of the optimal object, calculating the orientation distribution, and obtaining the front orientation of the object; According to the position and orientation information of the objects in the house layout, the final three-dimensional indoor scene is obtained by combining and placing them, which specifically includes: Step S41: Rendering the optimal object at four angles to obtain a rendering image of the optimal object; Step S42: Each of the optimal object rendering images is processed by ResNet to output image features And orthogonally project it onto the sphere, then the feature and Convolution is performed with the learned filters on to generate the signal on SO(3), ultimately producing a detailed distribution of 3D rotations; Step S43: Select the orientation angle with the highest probability and calculate the orientation angle of the front of the object As the object faces forward: ； in, The camera angle corresponding to the object rendering, The predicted angle of the object corresponding to the rendering with the highest probability.

6. A complex indoor scene generation system based on natural language, characterized in that: Includes the following modules: The module for obtaining the house frame structure is used to extract the basic information and description of the house frame data using the large language model, build a house frame vector database, and retrieve the best matching house frame structure through natural language input; A module for obtaining a house layout is used to obtain basic information and room connection relationships of each room according to the most matching house frame structure, and to generate and optimize the layout of each room in turn, so as to finally obtain the house layout; An optimal object acquisition module is used to obtain a rendering by rendering the object model from multiple angles, construct an object image vector database and an object text vector database based on the rendering, calculate similarities between the objects in the house layout and the object image vector database and the object text vector database, and select the optimal object; A three-dimensional indoor scene module is constructed, which is used to render the four angles of the optimal object, calculate the orientation distribution, and obtain the front orientation of the object; according to the object position and orientation information in the house layout, the final three-dimensional indoor scene is obtained by combining and placing.

Images