Open domain indoor scene hierarchy generation method and system based on large language model

Abstract

The invention belongs to the technical field of indoor scene synthesis, and particularly relates to an open domain indoor scene hierarchy generation method and system based on a large language model. Firstly, cue words are constructed according to user requirements and definition of a layered scene structure by utilizing a layered scene generation module, and a pre-training large language model is guided to output a structured text to describe layered scene representation, so that contradictory placement caused by a complex and dense spatial relationship is avoided. Then, in order to avoid the problem of inaccurate object placement caused by a rough spatial relationship, a fine-grained relative position inference module is adopted to further infer fine-grained relative positions, which are difficult to describe by text phrases, between objects with a text spatial relationship. The module may infer a reasonable relative position in open domain settings. And finally, optimizing each functional area by utilizing a divide-and-conquer layout optimization module, and then arranging the functional areas to form the whole scene, so that a physically feasible scene layout can be effectively generated.

Description

Technical Field

[0001] The present invention belongs to the technical field of indoor scene synthesis, and in particular relates to a method and system for generating hierarchical levels of open-domain indoor scenes based on a large language model. Background Art

[0002] Indoor scene design requires comprehensive consideration of spatial division, functional arrangement, and aesthetic creativity to determine the selection and placement of objects to form scene layouts, with the goal of automatically generating reasonable, realistic, and diverse 3D indoor scenes, especially considering arbitrary user requirements. However, due to the complexity of indoor scenes, most of them are limited to the scope of training data and cannot be generalized to arbitrary conditions. Recently, some pioneering works have leveraged the powerful generalization ability of pre-trained large language models (LLMs) to solve scene synthesis tasks on open domains, where LLMs are responsible for interpreting any textual requirements into detailed scene configurations. The challenge of this approach lies in obtaining reasonable and physically feasible scene layouts from LLM outputs.

[0003] The inventors found that the prior art has the following technical defects: On the one hand, LLM can use original knowledge and example sets to directly output numerical layouts. However, due to the lack of spatial reasoning ability of LLM, it is unable to understand the spatial relationship of the numerical layout, resulting in serious object overlap and out-of-bounds phenomena. On the other hand, compared with directly generating numerical layouts, letting LLM generate a text description of the spatial relationship of the scene can obtain more reliable answers. But this requires a method to convert text descriptions into digital layouts while maintaining the generalization of the entire pipeline. Existing methods predefine text phrases and numerical rules for several types of spatial relationships to obtain layouts. However, dense spatial relationships often cannot be satisfied at the same time, resulting in the configuration of the LLM output being inconsistent with the generation results, while rough relationships make it difficult to represent complex spatial positions, resulting in unreasonable placement of objects. Summary of the invention

[0004] In view of the above problems existing in the prior art, the present invention provides a method and system for generating a hierarchical structure of an open-domain indoor scene based on a large language model, which ensures the physical feasibility of the scene while significantly improving the rationality of the layout.

[0005] In order to solve the above technical problems, the technical solution of the present invention is: In a first aspect, the present invention provides a method for generating a hierarchy of open-domain indoor scenes based on a large language model, comprising: The scene structure is defined as a three-level hierarchy. The first level is the root node representing the entire scene. The second level is the internal nodes where each node represents a rectangular functional area. The third level is the leaf nodes representing objects in the corresponding area. Simple text phrases are used to represent the relationship between objects. Prompt words are constructed as input based on user needs and the definition of the scene structure to guide the pre-trained large language model to divide the functional areas. Structured text is output to describe the hierarchical scene representation, including the size of the object, text description, and the coarse-grained relative position attributes between objects. Train a fine-grained relative position reasoning network to infer fine-grained relative positions between objects with spatial relationships; it is based on a hierarchical scene structure and can infer reasonable relative positions in an open domain setting with the help of a pre-trained large visual semantic model; A divide-and-conquer layout optimization strategy is designed to optimize the scene layout from a hierarchical scene representation with fine-grained relative positions. It first optimizes locally within each functional region, and then globally optimizes to organize the regions into physically feasible scene layouts. Based on the text description of the object and the image of the object in the dataset, the cosine similarity is calculated using the pre-trained CLIP to retrieve the corresponding 3D object model, and then the object model is scaled and placed according to the scene layout to generate a complete scene.

[0006] As an optional implementation, in the three-level hierarchy, nodes are connected with two types of edges, namely, parent-child relationships representing the hierarchy and paired relationships between objects to represent their spatial relationships. Specifically, in order to reduce redundancy, an anchor object is set for each functional area, and only paired relationships between the anchor object and other objects belonging to the same functional area are allowed.

[0007] Furthermore, each node in the hierarchical scene structure contains the definition of properties. Assuming the scene is an axis-aligned rectangular plan, the root node Include size attributes and a text description of the scene ,Right now ,in Is a two-dimensional vector, representing the length and width of the scene; internal nodes Include size attributes , text description , Central location and direction attributes ,Right now ,in, is a two-dimensional vector, representing the length and width of the functional area, is a two-dimensional coordinate, Is a binary value indicating horizontal or vertical direction; leaf nodes Include text description , category label , the corresponding three-dimensional model and the size of the object's oriented bounding box , Central location ,direction ,Right now ,in, is a three-dimensional vector, representing the length, width and height of an object. is a three-dimensional coordinate; in addition, the pairwise spatial relationship Stores a rough text description And fine-grained relative position coordinates and relative direction ,Right now .

[0008] As an optional implementation, the construction prompt word is specifically: The first step is to assign a role and task description to the LLM, including a brief definition of the meaning of the nodes and the hierarchy they connect. The second step is to give a description of the preferred data format and predefined constraints, including the type of functional areas, possible anchor objects, and spatial relationships. Finally, a simple scenario example is presented to the LLM in the preferred format and with specific user requirements.

[0009] As an optional implementation, the process of constructing a fine-grained relative position reasoning network includes: constructing an input graph according to the hierarchical scene structure ,in, Represents a collection of object nodes, Indicates slave nodes in the same functional area arrive The set of directed edges of . From the input graph Take each vertex Description of the objects included , object size Information and each edge A rough textual description of the spatial relationships involved information.

[0010] Further, Using linear embedding coding, and Encode using the pre-trained CLIP text encoder. During training, use real data of relative position coordinates To enrich each edge information and encode it using linear embedding, where is a binary indicator of the alignment between two objects. Note that the relative position coordinate information is only used during training, i.e., not required during inference. After encoding, the node embedding and edge embedding for: .

[0011] Further, and Incorporate into input graph Each node and edge of the graph is conceptualized as a context graph , a variational graph neural network is used for encoding and decoding. The encoding process can be expressed as follows: , in, and represents the node embedding during the kth round of information transmission, represents edge embedding during the k-th round of information transmission; is the edge embedding update function, which refers to updating the edge embedding information with the node embedding of the kth round; is the node embedding update function, which means updating the node embedding information with neighbor nodes in the kth round; Representation and Node The set of connected neighbor nodes; represents the operation of taking the mean; after encoding, the edge embedding is parameterized as a Gaussian distribution; the decoder takes the updated context map as input and randomly samples from the Gaussian distribution to obtain the relative position features; finally, a separate MLP is used to decode the relative position information and output ,in is the relative position coordinate obtained by decoding, is the decoded relative direction, Binary indicator for the predicted alignment.

[0012] When training a fine-grained relative position reasoning network, the objective function is: , in, is the Kullback-Leibler divergence between the Gaussian distribution of the edge feature component and the posterior distribution, is the L1 loss in relative position, and is the cross entropy loss of the discretized relative orientation angle and the binary indicator of alignment.

[0013] As an optional implementation, the divide-and-conquer layout optimization includes two processes: local optimization and global optimization.

[0014] Furthermore, the local optimization is specifically as follows: for each functional area, local optimization is performed to minimize the relative position of the object and the relative position output by the fine-grained relative position reasoning module, with constraints to avoid object overlap and out-of-bounds, i.e. , in, and Indicates the position of the object and the corresponding anchor object in the functional area, including the center position coordinates and direction. Indicates the calculation of the relative position between two objects. It is the relative position between the object predicted by the network and the corresponding anchor object. Represents the collection of objects in the area. Constrain the overlap between the oriented bounding boxes of any two objects to be as small as possible, Aims to avoid the object bounding box being outside the region boundary. is the size of the region boundary, which is generated by the pre-trained LLM.

[0015] Furthermore, global optimization is specifically to organize the locally optimized areas to form a scene. Each functional area takes the direction of its anchor object as its own direction. Based on observations in daily life, functional areas are required to be placed against the wall, away from each other, and pointing to the inside of the scene, while avoiding object overlap and out-of-bounds. The optimization function is as follows: , in, Indicates the placement of the region, including the center position coordinates and direction. is the distance between the back face of the region and the border of the scene. is the distance between the bounding boxes of the two domains. is the set of regions in the scene. and Same as in local optimization.

[0016] Furthermore, the cosine similarity between the object image vector obtained by CLIP Image Encoder and the text description vector obtained by CLIP Text Encoder is calculated.

[0017] In a second aspect, the present invention provides an open domain indoor scene hierarchy generation system based on a large language model, comprising: The hierarchical scene generation module is configured to: define the scene structure as a three-level hierarchical structure, where the first level is the root node representing the entire scene, the second level is the internal node where each node represents a rectangular functional area, and the third level is the leaf node representing the objects belonging to the corresponding area; construct prompt words according to user needs and the definition of the scene structure, guide the pre-trained large language model to divide the functional areas, and output structured text to describe the hierarchical scene representation, including the size of the object, text description, and the coarse-grained relative position attributes between objects; The fine-grained relative position reasoning module is configured to infer fine-grained relative positions between objects with spatial relations; it is based on the hierarchical scene structure and uses a pre-trained large visual semantic model to infer reasonable relative positions in an open domain setting; The divide-and-conquer layout optimization module is configured to: optimize the scene layout from a hierarchical scene representation with fine-grained relative positions; it first performs local optimization inside each functional area, and then globally optimizes to organize the areas into physically feasible scene layouts; based on the text description of the object and the image of the object in the dataset, it uses the pre-trained CLIP to calculate the cospin similarity, retrieve the corresponding 3D object model, and then scale and place the object model according to the scene layout to generate a complete scene.

[0018] The present invention achieves the following technical effects: The present invention proposes a new method for hierarchical generation of open-domain indoor scenes based on a large language model, which uses a hierarchical scene structure as an intermediate representation to guide the refinement of spatial relations and the optimization of layout positions, ensuring the physical feasibility of the scene while improving the rationality of the layout. It mainly includes three modules: a hierarchical scene generation module, a fine-grained relative position reasoning module, and a divide-and-conquer layout optimization module. First, the hierarchical scene generation module generates a hierarchical scene structure with text descriptions by prompting a pre-trained large language. The structure has three levels, with the entire scene as the root node, the functional area as the internal node, and the object as the leaf node. Simple text phrases are used to represent the spatial relationship of the object, and the spatial relationship only exists inside the region. This avoids contradictory placement caused by complex and dense spatial relationships. Subsequently, in order to avoid the problem of inaccurate object placement caused by rough spatial relationships, a fine-grained relative position reasoning module is used to further infer the fine-grained relative positions between objects with textual spatial relationships that are difficult to describe with text phrases. Based on the hierarchical structure, with the help of a pre-trained visual semantic large model, this module can infer reasonable relative positions in an open-domain setting. Subsequently, the present invention designs a divide-and-conquer optimization, which optimizes each functional area separately and then arranges them to form the entire scene, which can effectively generate a physically feasible scene layout.

[0019] It is worth noting that the use of a hierarchical scene representation module has a dual advantage. First, the hierarchical scene structure provides a rough basis for object arrangement, alleviates contradictory placements with dense relations, and enhances the generalization ability of the network to infer fine-grained placements. Second, it naturally supports divide-and-conquer optimization to more efficiently generate scene layouts that match the descriptions generated by LLM. Extensive comparative experiments, ablation studies, and in-depth analysis are conducted both qualitatively and quantitatively. Experimental results show that the present invention can generate more reasonable and physically feasible scenes that are more in line with user needs and LLM arrangements. In addition, the present invention can be generalized for scene generation and interactive design in open domains. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of a method for generating hierarchical levels of open-domain indoor scenes based on a large language model in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of a hierarchical scene structure in Embodiment 1 of the present invention; Figure 3 Schematic diagram of eight possible alignment situations in Example 1 of the present invention; Figure 4 It is a qualitative comparison diagram of the scene generation results of different methods in Example 1 of the present invention; wherein, (a) is the scene generation result of ATISS, (b) is the scene generation result of DiffuScene, (c) is the scene generation result of LayoutGPT, (d) is the scene generation result of HOLODECK, and (e) is the scene generation result of the present invention; Figure 5 Schematic diagram of qualitative evaluation of ablation study in Example 1 of the present invention; wherein (a) is a hierarchical scene structure diagram of the input, (b) is the output result of the basic method, (c) is the output result after adding the fine-grained relative position reasoning network, and (d) is the output result after adding the fine-grained relative position reasoning network and the divide-and-conquer optimization strategy at the same time; the arrows in (a) represent the relationship between nodes, and the arrows in (b), (c), and (d) represent the direction of each piece of furniture; Figure 6 The following are different results of the open domain indoor scene generation task in Example 1 of the present invention; (a) generating scenes with different layouts under the same requirement; (b) generating different types of scenes according to the requirements of specific users; Figure 7 The following are different results of the open domain indoor scene editing task in Example 1 of the present invention; (a) is the artist's living room, (b) is the dining room of a family of four, (c) is the actress' bedroom, and (d) is the game room; Figure 8 This is a schematic diagram of a framework for hierarchical generation of open-domain indoor scenes based on a large language model in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0021] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0022] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0023] The present invention is further described below in conjunction with embodiments.

[0024] Example 1 like Figure 1 As shown, the present invention provides a method for hierarchical generation of open domain indoor scenes based on a large language model. The method includes three stages. The first is a hierarchical scene generation stage, which prompts the pre-trained LLM to generate a hierarchical scene structure with text descriptions. The hierarchical scene structure has three levels, with the entire scene as the root node, the functional area as the internal node, and the object as the leaf node. The nodes are connected with two types of edges, namely the parent-child relationship representing the hierarchy and the paired relationship between objects to represent their spatial relationship. Specifically, in order to reduce redundancy, an anchor object is set for each functional area, and only paired relationships between the anchor object and other objects belonging to the same functional area are allowed. As shown Figure 2 As shown. In this structure, simple text phrases are used to represent the spatial relationship of objects, and the spatial relationship only exists within the region. This avoids contradictory placement caused by complex and dense spatial relationships. The second is the fine-grained relative position reasoning stage. In order to avoid the problem of inaccurate object placement caused by rough spatial relationships, the present invention trains a fine-grained relative position reasoning network to further infer the fine-grained relative positions between objects with textual spatial relationships that are difficult to describe with text phrases. Based on the hierarchical structure, with the help of a pre-trained large visual semantic model, the network can infer reasonable relative positions in an open vocabulary setting. The third is the divide-and-conquer layout optimization stage. The present invention develops a divide-and-conquer optimization that optimizes each functional area separately and then arranges them to form the entire scene to effectively solve the physically feasible scene layout.

[0025] 1. Layered scene generation stage According to the definition of user needs and scene structure, prompt words are constructed to guide the pre-trained large language model to divide the functional areas. The pre-trained LLM takes the constructed prompts as input and outputs structured text to describe the hierarchical scene representation, including the size of objects, text descriptions, and coarse-grained relative position attributes between objects. The key challenge is to generate reasonable and informative spatial relationships to specify the scene layout.

[0026] Although existing works define dense relationships between objects to describe layouts, due to the insufficient spatial reasoning ability of LLMs, the more detailed the description, the more wrong or contradictory results they make. Therefore, the present invention requires LLMs to generate a hierarchical room structure to describe objects, only generate spatial relationships between objects belonging to the same area, and roughly specify their arrangement.

[0027] The present invention constructs input prompts with three components. The first is a description of the roles and tasks assigned to the LLM, including a brief definition of the hierarchy with the meaning of the nodes and their connections. The second is a description of the preferred data format and predefined constraints, including the type of functional areas, possible anchor objects, and spatial relationships. Finally, a simple scene example is presented to the LLM in the preferred format and with specific user requirements. At this stage, the LLM generates text descriptions and size attributes of the functional areas and objects, as well as a text description of the spatial relationships.

[0028] Each node in the hierarchical scene structure contains the definition of properties. Assuming the scene is an axis-aligned rectangular plan, the root node Include size attributes and a text description of the scene ,Right now ,in, Is a two-dimensional vector, representing the length and width of the scene; internal nodes Include size attributes , text description , Central location and direction attributes ,Right now ,in, is a two-dimensional vector, representing the length and width of the functional area, is a two-dimensional coordinate, Is a binary value indicating horizontal or vertical direction; leaf nodes Include text description , category label , the corresponding three-dimensional model and the size of the object's oriented bounding box , Central location ,direction ,Right now ,in, is a three-dimensional vector, representing the length, width and height of an object. is a three-dimensional coordinate; in addition, the pairwise spatial relationship Stores a rough text description And fine-grained relative position coordinates and relative direction ,Right now .

[0029] 2. Fine-grained relative position reasoning stage In order to avoid the problem of inaccurate object placement caused by rough spatial relationships, this paper proposes a hierarchical perception fine-grained relative position reasoning network to infer the fine-grained relative positions between related objects. The relative positions within each functional area show a more compact and generalizable prior, making it possible to train the network to infer positions in various scenarios. Figure 1 As shown, given the hierarchy generated by LLM, construct the input graph ,in, Represents a collection of object nodes, Indicates slave nodes in the same functional area arrive Although the input includes all objects in the scene, the functional areas are isolated from each other.

[0030] Furthermore, information is extracted from the hierarchical scene structure generated by LLM, where each vertex Contains the description of the object , the size of the object Information and each edge Rough text description including spatial relationships Further, Using linear embedding coding, the pre-trained CLIP text encoder is used and Encoding. During training, use the real data of relative position coordinates To enrich each edge information and encode it using linear embedding, where is a binary indicator of the alignment between two objects. Note that the relative position coordinate information is only used during training, i.e., not required during inference.

[0031] Incorporating this information into the input graph Each node and edge of the graph is conceptualized as a context graph , where the nodes are embedded in and edge embedding yes: .

[0032] Since there are only textual spatial relations between anchor objects and other objects, for edges without corresponding textual spatial relations, all-zero vectors are used as text embeddings.

[0033] For nodes with embedding and edge embedding Context diagram , a variational graph neural network is used for encoding and decoding. The encoding process can be expressed as follows: , in, and represents the node embedding during the kth round of information transmission, represents edge embedding during the k-th round of information transmission; is the edge embedding update function, which refers to updating the edge embedding information with the node embedding of the kth round; is the node embedding update function, which means updating the node embedding information with neighbor nodes in the kth round; Representation and Node The set of connected neighbor nodes; represents the operation of taking the mean; after encoding, the edge embedding is parameterized as a Gaussian distribution; the decoder takes the updated context map as input and randomly samples from the Gaussian distribution to obtain the relative position features; finally, a separate MLP is used to decode the relative position information and output ,in is the relative position coordinate obtained by decoding, is the decoded relative direction, Binary indicator for the predicted alignment.

[0034] In the post-processing stage, according to the predicted Correct the relative position of the prediction Specifically, if Figure 3 As shown, there are a total of eight possible alignments between any two objects. If the indicator classifies two objects as aligned, the alignment solution that is closest to the network's predicted position is selected from the eight possible cases.

[0035] During training, the CLIP text encoder is frozen and all other network layers are updated. The objective function is: , in, is the Kullback-Liebler divergence between the Gaussian distribution of the edge feature component and the posterior distribution. is the L1 loss in relative position. and is the cross entropy loss between the discrete relative orientation angle and the binary alignment metric.

[0036] 3. Divide and conquer layout optimization stage Given a hierarchical scenario of relative positions between related objects, the present invention designs a divide-and-conquer optimization to solve the final layout. Each functional area is first optimized locally, and then the global optimization organizes the areas into scenarios. This optimization is more effective than simple global optimization to produce reasonable and physically feasible layouts, or iteratively optimize the position of each object.

[0037] Local Optimization. For each functional region, local optimization is used to solve the object positions within the functional region. The local optimization is formulated to minimize the relative placement of objects and the positions inferred by the network, with constraints to avoid objects overlapping and outside the boundary, i.e. , in, and Indicates the position of the object and the corresponding anchor object in the functional area, including the center position coordinates and direction. Indicates the calculation of the relative position between two objects. is the relative position between the object predicted by the network and the corresponding anchor object. Represents the collection of objects in the area. Constrain the overlap between the oriented bounding boxes of any two objects to be as small as possible, Aims to avoid the object bounding box being outside the region boundary. is the size of the region boundary, which is generated by the pre-trained LLM.

[0038] Global optimization. Global optimization is used to organize these regions to form a scene. Each functional region takes the direction of its anchor object as its own direction. Based on observations in daily life, an optimization strategy is developed to place functional regions on the wall, away from each other, pointing to the inside of the scene, while avoiding object overlap and out-of-bounds. The optimization function is as follows: , in, Indicates the placement of the region, including the center position coordinates and direction. is the distance between the back face of the region and the border of the scene. is the distance between the bounding boxes of the two domains. is the set of regions in the scene. and Same as in local optimization.

[0039] After optimization, the coordinate system is transformed to obtain the position and direction of the object in the scene frame. Scoring retrieves 3D object models from the Objaverse and 3D-Front datasets, that is, the cosine similarity between the vector obtained by the object image through CLIP ImageEncoder and the vector obtained by the text embedding through CLIP Text Encoder is calculated. The object model is then scaled and placed according to the scene layout to generate a complete scene.

[0040] To achieve the task of indoor scene generation in open domains, existing research based on pre-trained large language models can be divided into two categories, including methods that directly output numerical layouts and methods that indirectly output spatial relationship descriptions.

[0041] It should be understood that the method of directly outputting numerical layout requires LLM to directly output the final result using original knowledge and example sets. At present, many studies focus on the two-dimensional layout of controllable scene image synthesis, such as UI Grammar introducing a syntax tree to guide LLM to generate UI layout. LayoutGPT first tried to use the retrieved scene layout and required LLM to output the numerical bounding box containing each object in the format of CSS. However, due to the lack of spatial reasoning ability, LLM cannot handle the complex relationships and changes of three-dimensional scenes, resulting in serious object overlap and out-of-bounds problems.

[0042] It should be understood that methods that indirectly output spatial relationship descriptions aim to generate textual scene descriptions using LLMs and convert them into 3D scenes. For example, Aladdin samples and generates a set of 3D texture assets from abstract descriptions and manually organizes them to build scenes. HOLODECK and AnyHome require LLMs to describe object relationships by selecting from a set of predefined atomic relationships, which are interpreted as fixed relative positions between objects and refined using rule-based optimization algorithms. However, it is very difficult to define a set of compact and informative atomic relationships. Dense and detailed object relationships are able to provide precise spatial arrangements but often lead to self-contradictions in the LLM output, while sparse, coarse-grained sets of relationships lead to coherent arrangements but cannot capture different spatial positions between objects.

[0043] To verify the effectiveness of the algorithm, the present invention conducted comparative experiments and ablation studies on the 3D-Front dataset. According to the settings of LayoutGPT, scenes with irregular floors are also filtered out from the dataset. Finally, the training set sizes of the bedroom and living room are 3397 and 690, while the corresponding test sets are 60 and 53.

[0044] The invention evaluates the generated scene from two perspectives. One is the physical feasibility of the scene, which is estimated by the ratio of the overlap of the oriented bounding boxes, i.e. the overlap rate, and the ratio outside the bounds, i.e. the out-of-bounds rate. The other is the rationality of the scene layout, for which the inventors selected some common object pairs, i.e. bed-nightstand, table-chair, coffee table-sofa, and measured the average KL-divergence between the relative position distribution of the real scene in the test set and the generated scene.

[0045] For all LLM-assisted methods, GPT-4 is used for evaluation. The Adam optimizer is set to train the fine-grained relative position reasoning network for 500 epochs with a batch size of 4 and a learning rate of 1e-4. Divide and conquer optimization is implemented using the GUROBI solver.

[0046] 1. Performance comparison We compare two types of state-of-the-art methods, including those that train deep neural networks from scratch, namely ATISS and DiffuScene, and LLM-assisted indoor scene synthesis, namely LayoutGPT and HOLODECK.

[0047] like Figure 4 As shown, the deep learning methods ATISS and DiffuScene generate results with reasonably placed objects. However, the network cannot guarantee the physical feasibility of the scene layout, sometimes resulting in overlapping and out-of-bounds objects. LayoutGPT uses contextual learning to infer digital layouts based on demonstrated examples, generating many incorrect directions and positions. HOLODECK produces relatively good results in terms of physical feasibility, but some objects are not placed in the optimal positions specified by LLM. In contrast, the present invention is able to produce more reasonable and feasible scene layouts.

[0048] Table 1 Quantitative comparison of scene generation results of different methods .

[0049] Table 1 verifies the observation of visual results (↑ in the table indicates that the higher the value of the evaluation index, the better the result generated; ↓ indicates that the lower the value of the evaluation index, the better the result generated). Among all the methods, the present invention achieves the best results in terms of physical feasibility (overlap rate and out-of-bounds rate) and reasonable relative position (KL Div.). It is worth noting that the data-driven method is good at the relative position of objects, the LLM-assisted optimization method, namely HOLODECK, obtains more feasible results, and the present invention takes advantage of the advantages of both and obtains the best results in all indicators.

[0050] Further, the present invention provides an additional quantitative comparison with HOLODECK, both of which use LLM to generate textual scene descriptions and then solve the scene layout. The difference is that HOLODECK requires dense and detailed spatial relations, while the present invention uses a hierarchical scene structure with sparse relations and a neural network to infer fine-grained relative positions. Table 2 reports the semantic alignment results between the description generated by LLM and the generated scene. Specifically, #Rel. calculates the percentage of relative positions that match the spatial relations generated by LLM; #Obj. calculates the percentage of objects in the generated scene that match the objects specified by LLM. Obviously, the results of the present invention are better, which means the advantage of using a hierarchical scene representation.

[0051] Table 2 Semantic alignment results between LLM-generated descriptions and generated scenes .

[0052] 2. User Survey The survey evaluated the quality of scenes generated by different methods. The survey showed 25 generated scenes in renderings and overhead views to 30 participants. Participants were asked to rate the scenes on a 5-point scale based on the three aspects of scene effectiveness, physical feasibility, and layout rationality.

[0053] Table 3 shows that the results of the present invention obtain the highest scores in terms of scene validity, physical feasibility, and layout rationality. This is because the present invention uses a hierarchical scene structure to prompt LLM generation, which can effectively generate feasible scenes aligned with the LLM arrangement. Otherwise, although HOLODECK can prevent objects from overlapping or going out of bounds, its results may destroy the LLM arrangement and may affect human activities, such as Figure 4 This is shown in the first row.

[0054] Table 3 Average scores of different model-generated scenarios in terms of scenario effectiveness, physical feasibility, and layout rationality in the user survey .

[0055] 3. Ablation study Table 4 Quantitative evaluation of ablation studies to verify the key designs of the present invention: fine-grained relative position reasoning network and divide-and-conquer layout optimization strategy .

[0056] The ablation study verifies the key design of the present invention, namely the fine-grained relative position reasoning network and the divide-and-conquer layout optimization strategy. Specifically, the corresponding stages are deleted based on the method of the present invention. When removing the fine-grained relative position reasoning network, the relative position coordinates of the text space relationship are predefined to replace the network's prediction; when removing the divide-and-conquer layout optimization strategy, the LLM is directly required to generate the coordinates of the anchor object and convert the relative positions of other objects into global coordinates without optimization and refinement. The results are shown in Table 4 and Figure 5 The results show that the fine-grained relative position reasoning network is of great significance in capturing the reasonable relative positions between objects, and the divide-and-conquer optimization ensures the physical feasibility of the generated scene layout.

[0057] 4. Open Vocabulary Scene Synthesis To demonstrate the generality of the present invention, Figure 6 Results for open domain scene generation are presented in . This section removes the constraints limiting the object categories in the LLM prompts. The results show that LLM is able to generate reasonable hierarchical scene structures for arbitrary requirements. In addition, the fine-grained relative position reasoning network and the divide-and-conquer layout optimization strategy can produce reasonable and realistic scenes with various hierarchical descriptions.

[0058] 5. Interactive scene editing The present invention also supports user-friendly language-guided interactive scenes. Specifically, the current state of the scene and the editing instructions are described as the input of the LLM, and an additional constraint is added in the divide-and-conquer optimization to keep the position of the invariant objects as much as possible. Figure 7 As shown, LLM can modify the current scene by adding and deleting objects. In addition, through the hierarchical scene structure and the method of the present invention, the edited scene can show minimal changes from the original scene while meeting the user's expectations and the LLM output results.

[0059] Example 2 like Figure 8 As shown, this embodiment provides an open domain indoor scene hierarchy generation system based on a large language model, including: The hierarchical scene generation module is configured to: define the scene structure as a three-level hierarchical structure, where the first level is the root node representing the entire scene, the second level is the internal node where each node represents a rectangular functional area, and the third level is the leaf node representing the objects belonging to the corresponding area; construct prompt words according to user needs and the definition of the scene structure, guide the pre-trained large language model to divide the functional areas, and output structured text to describe the hierarchical scene representation, including the size of the object, text description, and the coarse-grained relative position attributes between objects; The fine-grained relative position reasoning module is configured to infer fine-grained relative positions between objects with spatial relations; it is based on the hierarchical scene structure and uses a pre-trained large visual semantic model to infer reasonable relative positions in an open domain setting; The divide-and-conquer layout optimization module is configured to: optimize the scene layout from a hierarchical scene representation with fine-grained relative positions; it first performs local optimization inside each functional area, and then globally optimizes to organize the areas into physically feasible scene layouts; based on the text description of the object and the image of the object in the dataset, it uses the pre-trained CLIP to calculate the cospin similarity, retrieve the corresponding 3D object model, and then scale and place the object model according to the scene layout to generate a complete scene.

[0060] It should be noted that the above modules correspond to the steps described in Example 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above Example 1. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer executable instructions.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating hierarchical levels of open-domain indoor scenes based on a large language model, characterized in that: include: The scene structure is defined as a three-level hierarchy. The first level is the root node representing the entire scene. The second level is the internal nodes where each node represents a rectangular functional area. The third level is the leaf nodes representing objects in the corresponding area. Simple text phrases are used to represent the relationship between objects. Prompt words are constructed based on user needs and the definition of the scene structure to guide the pre-trained large language model to divide the functional areas. Structured text is output to describe the hierarchical scene representation, including the size of the object, text description, and the coarse-grained relative position attributes between objects. Train a fine-grained relative position reasoning network to infer the fine-grained relative positions between objects with spatial relationships; It is based on a hierarchical scene structure and can infer reasonable relative positions in an open domain setting with the help of a pre-trained large visual semantic model; Design a divide-and-conquer layout optimization strategy to optimize scene layout from a hierarchical scene representation with fine-grained relative positions; It first performs local optimization within each functional area and then global optimization to organize the areas into a physically feasible scene layout. It uses pre-trained CLIP to calculate cospin similarity based on the text description of the object and the image of the object in the dataset, retrieves the corresponding 3D object model, and then scales and places the object model according to the scene layout to generate a complete scene.

2. The method for generating hierarchical levels of open-domain indoor scenes based on a large language model according to claim 1, characterized in that: Each node in the hierarchical scene structure contains the definition of properties. Assuming the axis-aligned rectangular floor plan of the scene, the root node Include size attributes and a text description of the scene ,Right now ,in Is a two-dimensional vector, representing the length and width of the scene; internal nodes Include size attributes , text description , Central location and direction attributes ,Right now ,in, is a two-dimensional vector, representing the length and width of the functional area, is a two-dimensional coordinate, Is a binary value indicating horizontal or vertical direction; leaf nodes Include text description , category label , the corresponding three-dimensional model and the size of the object's oriented bounding box , Central location ,direction ,Right now ,in, is a three-dimensional vector, representing the length, width and height of an object. is a three-dimensional coordinate; in addition, the pairwise spatial relationship Stores a rough text description And fine-grained relative position coordinates and relative direction ,Right now .

3. The method for generating hierarchical levels of open-domain indoor scenes based on a large language model according to claim 1, characterized in that: The construction prompts are as follows: first, assign a role and task description to the LLM, including a brief definition of the meaning of nodes and the hierarchical structure they connect; second, give a description of the preferred data format and predefined constraints, including the type of functional area, possible anchor objects, and spatial relationships; finally, show the LLM a simple scenario example in the preferred format and with specific user requirements.

4. The method for generating hierarchical levels of open-domain indoor scenes based on a large language model according to claim 1, characterized in that: The process of building a fine-grained relative position reasoning network includes: constructing an input graph according to the hierarchical scene structure ,in, Represents a collection of object nodes, Indicates slave nodes in the same functional area arrive The set of directed edges of Take each vertex Description of the objects included , object size Information and each edge A rough textual description of the spatial relationships involved information.

5. The method for generating hierarchical levels of open domain indoor scenes based on a large language model according to claim 4, characterized in that: Also includes: Using linear embedding coding, and Encode using the pre-trained CLIP text encoder; during training, use real data of relative position coordinates To enrich each edge information and encode it using linear embedding, where is a binary indicator of the alignment between two objects; the relative position coordinate information is only used during training, i.e., not required during inference; after encoding, the node embedding and edge embedding for: 。 6. The method for generating hierarchical levels of open domain indoor scenes based on a large language model according to claim 5, characterized in that: Also includes: Will and Incorporate into input graph Each node and edge of the graph is conceptualized as a context graph , a variational graph neural network is used for encoding and decoding; the encoding process can be expressed in the following way: ， in, and represents the node embedding during the kth round of information transmission, represents edge embedding during the k-th round of information transmission; is the edge embedding update function, which refers to updating the edge embedding information with the node embedding of the kth round; is the node embedding update function, which means updating the node embedding information with neighbor nodes in the kth round; Representation and Node The set of connected neighbor nodes; represents the operation of taking the mean; after encoding, the edge embedding is parameterized as a Gaussian distribution; the decoder takes the updated context map as input and randomly samples from the Gaussian distribution to obtain the relative position features; finally, a separate MLP is used to decode the relative position information and output ,in is the relative position coordinate obtained by decoding, is the decoded relative direction, Binary indicator for the predicted alignment.

7. The method for generating hierarchical levels of open domain indoor scenes based on a large language model according to claim 1, characterized in that: When training a fine-grained relative position reasoning network, the objective function is: ， in, is the Kullback-Leibler divergence between the Gaussian distribution of the edge feature component and the posterior distribution, is the L1 loss in relative position, and is the cross entropy loss of the discretized relative orientation angle and the binary indicator of alignment.

8. The method for generating hierarchical levels of open-domain indoor scenes based on a large language model according to claim 1, characterized in that: Divide and conquer layout optimization includes two processes: local optimization and global optimization.

9. The method for generating hierarchical levels of open domain indoor scenes based on a large language model according to claim 8, characterized in that: The local optimization is as follows: for each functional area, local optimization is performed to minimize the relative position of the object and the relative position output by the fine-grained relative position reasoning module, and the constraints are used to avoid object overlap and out-of-bounds, that is, ， in, and Indicates the position of the object and the corresponding anchor point object in the functional area, including the center position coordinates and direction; Indicates the calculation of the relative position between two objects. is the relative position between the object predicted by the network and the corresponding anchor object; Represents the collection of objects in the area; Constrain the overlap between the oriented bounding boxes of any two objects to be as small as possible, Aims to avoid the object bounding box being outside the region boundary. is the size of the region boundary, which is generated by the pre-trained LLM; The global optimization is as follows: organize the locally optimized areas to form a scene, and each functional area takes the direction of its anchor object as its own direction; based on observations in daily life, the functional areas are required to be placed against the wall, away from each other, and pointed to the inside of the scene, while avoiding object overlap and out-of-bounds. The optimization function is as follows: ， in, Indicates the placement of the region, including the center position coordinates and direction; is the distance between the back of the region and the boundary of the scene; is the distance between the bounding boxes of the two domains; is the set of regions in the scene; and Same as in local optimization.

10. An open domain indoor scene hierarchical generation system based on a large language model, characterized in that: include: The hierarchical scene generation module is configured to: define the scene structure as a three-level hierarchical structure, where the first level is the root node representing the entire scene, the second level is the internal node where each node represents a rectangular functional area, and the third level is the leaf node representing the objects belonging to the corresponding area; construct prompt words according to user needs and the definition of the scene structure, guide the pre-trained large language model to divide the functional areas, and output structured text to describe the hierarchical scene representation, including the size of the object, text description, and the coarse-grained relative position attributes between objects; The fine-grained relative position reasoning module is configured to infer fine-grained relative positions between objects with spatial relations; it is based on the hierarchical scene structure and uses a pre-trained large visual semantic model to infer reasonable relative positions in an open domain setting; The divide-and-conquer layout optimization module is configured to: optimize the scene layout from a hierarchical scene representation with fine-grained relative positions; It first performs local optimization within each functional area and then global optimization to organize the areas into a physically feasible scene layout. It uses pre-trained CLIP to calculate cospin similarity based on the text description of the object and the image of the object in the dataset, retrieves the corresponding 3D object model, and then scales and places the object model according to the scene layout to generate a complete scene.

Images