Systems and methods for generating a controllable text transform 3D room mesh with layout

Abstract

A system for generating a 3D indoor scene from text input is provided. The system comprises a user interface, a text processing module, a scene code generator, a layout generation module, an appearance generation module, a NeRF module and a PeRF module. A user may enter text describing a room, and the text is processed into scene code. The generated scene code uses a directional bounding box to guide the generation of the 3D layout, thereby ensuring the spatial integrity. The appearance generation module then creates a visual representation object of the room using the panoramic image. The NeRF module can construct a basic 3D model, and the basic 3D model is further refined by the PeRF module so as to enhance the visual coherence.

Description

Technical Field

[0001] The present invention generally relates to artificial intelligence (AI) techniques for room image generation. More specifically, the present invention relates to a system and method for generating a controllable text-converted 3D room mesh with layout constraints. Background Art

[0002] High-quality 3D models are important for many applications, ranging from interior design to gaming to simulated AI avatars. Among all 3D content, indoor scenes have received particular attention. Typically, 3D indoor scenes are manually constructed by professional designers, a process that is time-consuming and expensive. Although the process of generating 3D models from text descriptions has become easier using generative modeling techniques such as Dreamfusion, Fantasia3D, Magic3D, and 3DFuse, it is still challenging to apply this technology to text-driven 3D indoor scene generation. Indoor scenes have strong semantic layout constraints; for example, adjacent walls must be vertical and the TV is usually facing the sofa, which is much more complex than object generation.

[0003] Existing text-driven 3D indoor scene generation methods, such as Text2Room and Text2NeRF, are based on an incremental framework. They generate 3D indoor scenes by generating views from different angles frame by frame, and then reconstructing the 3D mesh of the room based on these view images. However, this incremental method often fails to capture the overall layout of the room well, resulting in semantically unreasonable and unrealistic generated results.

[0004] For example, in one of the scenes generated by Text2Room, the living room may have multiple cabinets with duplicate objects, and the layout of the furniture does not follow a regular pattern. This problem is called the "Penrose Triangle problem", that is, although the local 3D structure looks reasonable, the overall lack of consistency. In addition, previous methods also cannot support user interoperability because the resulting 3D geometry and textures are not editable.

[0005] Therefore, in the field of indoor scene generation, a new method is urgently needed that can not only generate reasonable 3D structures with consistent interior layouts but also provide user-friendly editing functions. Summary of the invention

[0006] The purpose of the invention is to provide a system and method to solve the above problems existing in the prior art.

[0007] The present invention provides a flexible method for achieving editable and well-structured 3D (three-dimensional) indoor scene generation. The proposed method consists of two stages: a layout generation stage and an appearance generation stage. In the layout generation stage, the scene layout is parameterized using scene codes, and a text-driven diffusion model is learned to generate a text-driven layout. In the appearance generation stage, a ControlNet model is fine-tuned to generate a vivid panorama of the room guided by the scene layout. Subsequently, a high-quality 3D room model with a well-structured layout and realistic textures is generated. One of the innovative features of the proposed method is the support for interactive 3D scene editing. In addition, a mask-guided editing method is proposed, which allows users to adjust the size, position, and semantic category of furniture in the room.

[0008] According to a first aspect of the present invention, a system for computer-generated 3D indoor scene analysis is provided, the system comprising: a user interface, a text processing module, a scene code generator, a layout generation module, an appearance generation module, a neural radiation field (NeRF) module, and a panoramic enhanced radiation field (PeRF) module. The user interface is configured to receive user input about the room from a user in the form of text input, and convert it into a language or code recognizable by the system for processing. The text processing module communicates with the user interface and is configured to receive the user input and process it into a scene description. The scene code generator communicates with the text processing module and is configured to use a scene code diffusion model to convert the scene description from the text processing module into a set of scene codes. The layout generation module communicates with the scene code generator and is configured to generate a 3D layout of the room using a directional bounding box based on the scene code, wherein the 3D layout of the room retains the spatial integrity and the relationship between objects specified by the scene code. an appearance generation module, in communication with the layout generation module, configured to convert the 3D layout of the room from the layout generation module into a visual representation of the room, wherein the appearance generation module is further configured to convert the 3D layout of the room into a semantic layout using an equirectangular projection, and to generate a single panoramic image of the room based on the semantic layout. A NeRF module is in communication with the appearance generation module, configured to construct a basic 3D room model based on the panoramic image, and to generate representation objects of the room by capturing spatial depth. A PeRF module is in communication with the NeRF module, configured to refine the basic 3D room model by enhancing visual coherence, thereby generating a fully refined 3D room model.

[0009] According to a second aspect of the present invention, there is provided a method for using a system, wherein the system generates a 3D indoor scene analysis based on a computer, wherein the method comprises the following steps: receiving user input about a room from a user in the form of text input through a user interface; converting the user input into a language or code recognizable by the system for processing through the user interface; receiving the user input through a text processing module and processing it into a scene description; converting the scene description from the text processing module into a set of scene codes through a scene code generator using a scene code diffusion model; generating a 3D layout of the room through a layout generation module using a directional bounding box based on the scene code, wherein the The 3D layout of the room retains the spatial integrity and the relationship between objects specified by the scene code; through an appearance generation module, the 3D layout of the room from the layout generation module is converted into a visual representation of the room, wherein the appearance generation module is also configured to convert the 3D layout of the room into a semantic layout using an equirectangular projection, and generate a single panoramic image of the room based on the semantic layout; through a NeRF module, a basic 3D room model is constructed based on the panoramic image, and representative objects of the room are generated by capturing spatial depth; and, through a PeRF module, the basic 3D room model is refined using enhanced visual coherence to generate a fully refined 3D room model.

[0010] In some embodiments, a computer-implemented method is also provided for generating 3D materials of indoor scenes using a set of trained machine learning models. The method includes: providing a text description of the indoor scene as an input source to the trained machine learning model; obtaining a 3D bounding box of the room layout; based on the 3D layout of the room, using another fine-tuned machine learning model to generate a visually attractive panoramic image of the room; based on the generated panoramic image, recovering and texturing the 3D mesh of the room through depth estimation and surface reconstruction.

[0011] In some embodiments, a computer-implemented method is provided for generating editable 3D materials for indoor scenes. The method includes parameterizing the indoor scene into an overall scene code that is flexible and supports user interactive editing. The method also includes representing the indoor scene as a panoramic image while also considering a reasonable room layout from a physical perspective.

[0012] In some embodiments, a computer-implemented method for generating and editing panoramic images is also provided, wherein the method includes achieving high-quality results with cycle consistency through a pre-trained latent image diffusion model without the need for expensive editor-specific training.

[0013] Through the above configuration, the following positive effects are achieved:

[0014] (1): We design a two-stage approach to generate 3D rooms from textual input. Our approach involves separating the generation of geometric layout and visual appearance, allowing us to capture room layout constraints from real-world data while creating visually appealing results.

[0015] (2): The generation process of geometric layout and visual appearance is separated, which provides flexible control and editability of the generated room 3D model. Users can easily adjust the size, semantic category and position of furniture.

[0016] (3): We propose a novel approach for generating and editing panoramic images. Our approach uses a pre-trained latent image diffusion model and achieves high-quality results with cycle consistency without the need for expensive editing-specific training. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Hereinafter, embodiments of the present invention will be described in more detail with reference to the accompanying drawings, in which:

[0018] Figure 1 A schematic diagram of a model framework of a system for computer-generated 3D indoor scene analysis according to an embodiment of the present invention is presented;

[0019] Figure 2 A schematic diagram of a method for generating a 3D room model using the proposed system according to an embodiment of the present invention is presented;

[0020] Figure 3A , Figure 3B , Figure 4A , Figure 4B , Figure 5A and Figure 5B A qualitative demonstration diagram of a system using the method described in an embodiment of the present invention is provided. DETAILED DESCRIPTION

[0021] In the following description, as a preferred example, a system and method for generating a controllable text conversion 3D room grid with layout constraints are described. It will be apparent to those skilled in the art that modifications including additions and / or substitutions can be made without departing from the scope and spirit of the invention. Certain details may be omitted so as not to obscure the invention; however, this disclosure is written to enable those skilled in the art to practice the teachings herein without undue experimentation.

[0022] Text-driven 3D indoor scene generation has potential applications in gaming, film industry, and AR / VR. However, existing methods cannot accurately capture room layouts and lack the ability to flexibly edit individual objects in the room. To address these issues, in the present disclosure, a solution is proposed, called "Ctrl-Room", which is a system that can generate realistic 3D room models with designer-style layouts and high-quality textures based on simple text prompts. In addition, it also allows interactive editing operations, such as resizing or moving objects such as furniture. The proposed method enables layout modeling and appearance generation to be performed separately. It includes two stages: the first is a "layout generation stage", which trains a text-conditional diffusion model to learn layout distributions using overall scene code parameterization; and the second is an "appearance generation stage", which uses a fine-tuned ControlNet to generate vivid panoramic images of the room under the guidance of 3D scene layout and text prompts. In this way, high-quality 3D room models with realistic layouts and vivid textures can be created. Through scene code parameterization, the generated room model can be easily edited using a mask-guided editing module, and expensive editing tasks can be completed without specialized training. Overall, Ctrl-Room provides a powerful solution for text-driven 3D interior scene generation in various applications such as gaming, film industry, and AR / VR.

[0023] The following describes methods, systems, and computer-readable media for generating 3D (three-dimensional) editable assets of indoor scenes from text descriptions (eg, computer-based asset generation of 3D indoor scenes).

[0024] Figure 1 The model framework diagram of the system 100 based on computer-generated 3D indoor scene analysis according to an embodiment of the present invention is shown. The system 100 includes a user interface 102, a text processing module 110, a scene code generator 112, a layout generation module 114, a layout modification module 116, an appearance generation module 120, a NeRF (neural radiance field) module 122, a PeRF (panoptic-enhanced radiance field) module 124, and a panoramic update module 126.

[0025] These components are arranged for a two-stage approach to achieve the generation of editable and physically plausible 3D room meshes from textual cues, wherein the two stages include a layout generation stage and an appearance generation stage. The text processing module 110, the scene code generator 112, the layout generation module 114, and the layout modification module 116 collaborate in the layout generation stage. The appearance generation module 120, the NeRF module 122, the PeRF module 124, and the panoramic update module 126 collaborate in the appearance generation stage.

[0026] The layout generation phase includes providing an overall scene code, which is configured as a parameterized room layout, and the room layout can support a flexible user editing mode; the layout generation phase can also provide a generative model (for example, presented as a layout generation module 114) to learn the distribution arrangement of the room layout. The appearance generation phase includes a fine-tuned generative model (for example, a NeRF module 122 / PeRF module 124) to generate a vivid panoramic image with an indoor scene under the constraints of the room layout, and further generate / provide a high-quality 3D room with a well-structured layout and realistic textures. In addition, the user can adjust the size, position and / or semantic category of the furniture in the room by using a mask-guided editing option (which can be implemented at least through the user interface 102).

[0027] The user interface 102 is configured to receive input from the user about the room, which may be text or other types of input information. The user input may be related to the desired room type (e.g., living room, study, or bedroom). The user interface 102 may convert the input into a language or code that the system 100 can recognize and process. By translating the user input into a system-readable format, the user interface 102 ensures that the input can be effectively delivered to the corresponding components in the system 100. These components can then regenerate appropriate responses based on the user input, thereby enabling interaction and communication between the user and the system 100.

[0028] The text processing module 110 communicates with the user interface 102 and is configured to receive at least one text prompt provided by a user (which may be, for example, a user input) and process it into a scene description. Therefore, the input of the text processing module 110 is the text prompt, and the output of the text processing module 110 is the scene description.

[0029] The scene code generator 112 communicates with the text processing module 110 and is configured to convert the scene description of the text processing module 110 into a controllable scene code using a scene code diffusion model. Specifically, the scene code generator 112 includes a scene code diffusion model. During the conversion process, the scene code generator 112 receives the scene description from the text processing module 110 and processes it through the multi-layer structure of the QKV (query, key, value) mechanism in the scene code diffusion model. The scene code diffusion model is configured to gradually refine the scene description into a structured representation object. During this conversion process, scene code noise can be embedded to introduce changes and flexibility, so that the scene can be generated more robustly. As the diffusion progresses, the noise will gradually decrease, and then the scene code generator 112 will be able to output a set of scene codes representing the scene structure, which is conducive to preparing for further processing or rendering. Therefore, the input of the scene code generator 112 is the scene description, and the output of the scene code generator 112 is a set of scene codes.

[0030] The layout generation module 114 communicates with the scene code generator 112 and is configured to generate a 3D layout of the room using a directional bounding box based on the scene code. In this regard, the scene code generated by the scene code generator 112 can be used as a structured representation object that associates room elements and spatial relationships. The layout generation module 114 takes the scene code group as input and converts it into a 3D layout by using directional bounding boxes to locate and adjust the size of objects in the room. These bounding boxes represent key objects or key factors in the scene, such as walls, furniture, or fixed objects, and provide a modular approach to define the geometry and layout of the room. The output of the layout generation module 114 is a 3D layout of the room, which retains the spatial integrity and the relationship between objects specified by the scene code. Therefore, the input of the layout generation module 114 is a scene code group, and the output of the layout generation module 114 is a 3D layout of the room.

[0031] The layout modification module 116 communicates with the layout generation module 114 and is configured to allow the user to modify the layout in an interactive manner (e.g., change the position and size of the objects, and the modification request can be fed to the layout modification module 116 by the user interface 102). Specifically, the layout modification module 116 receives the 3D layout of the room and displays it to the user (e.g., the user who initially provides input content to the system 100). The layout modification module 116 provides an interface so that the user can adjust at least one scene code based on the displayed 3D layout of the room, such as changing the position or size of the objects in the scene (i.e., in the room), so that the system 100 can be flexible. In this regard, the user's modification action can directly change the applied scene code, which represents the object structure and spatial properties. For example, when the user changes the position or size of the object, the layout modification module 116 will update the relevant scene code to reflect these changes, thereby ensuring that the system 100 can remain flexible and also make the visual representation of the scene and the scene data consistent. By providing interactive and dynamic control over the room layout, the layout modification module 116 enhances user engagement and allows for personalized room design, which can be easily adapted to specific preferences or requirements. Thus, the input to the layout modification module 116 is the 3D layout of the room, and the output of the layout modification module 116 is updated data of the 3D layout of the room.

[0032] The appearance generation module 120 is in communication with the layout generation module 114 and is configured to convert a 3D room layout (e.g., a 3D layout of the room from the layout generation module 114) into a visual representation object of the room. The appearance generation module 120 converts the detailed layout provided by the layout generation module 114 into a semantic layout using an equirectangular projection. The semantic layout can capture the spatial relationships and object positions in the room, thereby responding to user input content and generating visual representation objects thereof.

[0033] After the conversion, the appearance generation module 120 can use cycle-consistent sampling techniques to ensure that the generated visual effect is consistent in terms of spatial representation and input content. In one embodiment, the conversion is implemented using a ControlNet model, which is a neural network that can be used to refine the visual output by maintaining consistency between different viewpoints. The output of the appearance generation module 120 is a single panoramic image that can fully represent the visual details of the room. Therefore, the input of the appearance generation module 120 is the 3D layout from the layout generation module 114, and the output of the appearance generation module 120 is a single panoramic image of the room.

[0034] The NeRF module 122 communicates with the appearance generation module 120 and is configured to construct / create a basic 3D room model based on the panoramic image generated by the appearance generation module 120. The NeRF module 122 generates representation objects of the room by capturing the spatial depth within the scene. In this regard, the NeRF module 122 utilizes neural radiation technology and analyzes the panoramic image so that the NeRF module 122 can capture and reconstruct the spatial depth and complexity of the scene. It can process the image to distinguish the surface and spatial relationship of objects in the room, thereby ensuring that the basic 3D room model can reflect the visual appearance of the room. Therefore, the input content of the NeRF module 122 is the panoramic image generated by the appearance generation module 120, and the output of the NeRF module 122 is a basic 3D room model (e.g., a NeRF model of the room) with realistic textures and consistency with the space. The results of the NeRF module 122 are provided to the PeRF module 124 for enhancement, and the PeRF module 124 is intended to focus on 3D rendering guided by the layout.

[0035] The PeRF module 124 communicates with the NeRF module 122 and is configured to refine the basic 3D room model generated by the NeRF module 122, incorporating layout constraints to ensure spatial consistency and semantic accuracy. The steps of the PeRF module 124 can enhance the visual coherence of the room by aligning the position, size, and texture of objects with the layout. Therefore, the output of the PeRF module 124 is a fully refined 3D room model (e.g., a layout-guided PeRF model of the room).

[0036] The panoramic update module 126 communicates with the PeRF module 124 and is configured to dynamically update the panoramic image and the 3D room model based on any modifications made by the user to the layout (e.g., modification requests input to the panoramic update module 126 from the user interface 102), such as changes in the position or size of objects. The panoramic update module 126 can ensure that the updated layout is reflected in the panoramic image and the 3D room model. For example, when the user views the fully detailed 3D room model presented by the PeRF module 124 and makes modifications, the panoramic update module 126 can dynamically update the 3D room model and request the appearance generation module 120, the NeRF module 122, and the PeRF module 124 to regenerate the 3D room model based on the user's changes. This allows the corresponding components to generate a revised 3D room model that better meets the user's latest requirements. The PeRF module 124 then provides the final output of the 3D room model.

[0037] Figure 2A schematic diagram of a method for generating a 3D room model using the proposed system 100 according to an embodiment of the present invention is shown. In the present invention, the proposed method is referred to as the CtrlRoom solution, which implements controllable text with layout constraints to 3D room mesh generation. In short, the proposed method involves a 3D room modeling process and is divided into two stages, including a layout generation stage and an appearance generation stage. In the layout generation stage, the purpose is to generate a 3D scene layout from the input text. In the appearance generation stage, the purpose is to generate a single panorama to represent the appearance guided by the scene layout. After the appearance generation stage, the expected result is to reconstruct a complete 3D room model. During the generation process, it is represented by a flexible scene layout defined by a set of oriented bounding boxes, and enables users to perform flexible editing operations. For example, after the user moves a chair, the method also includes generating a new panorama based on the updated scene layout, and then updating the 3D room model accordingly.

[0038] like Figure 2 As shown, the layout generation phase includes steps S10, S20, S30 and S40. In step S10, the user inputs a text prompt through the user interface 102. Then, the text processing module 110 processes the text prompt. In step S20, the scene code generator 112 starts the scene code diffusion based on the text prompt. In step S30, the scene code generator 112 generates a set of scene codes. This set of scene codes can be recorded in a user-friendly format. In one embodiment, the scene code is represented in the form of a structured table, where "rows" represent different entities in the scene (such as "walls" or "objects"), and "columns" correspond to different attributes or parameters associated with each entity / element (such as Ci, Li, Si, Ri). For example, the attributes or parameters include the category or form of the element, the position or layout information, the size or proportion of the element, and the direction or rotation positioning of the element. This table format facilitates the user to programmatically control, generate and modify the scene layout based on the scene code. In step S40, the layout generation module 114 can generate an initial 3D layout of the room using a directional bounding box based on the scene code. During the generation of the initial 3D layout, the system 110 allows the user to make interactive user modifications through the layout modification module 116, such as changing the position or size of objects in the scene, thereby allowing the user to adjust at least one item in the scene code, so that the system 100 can provide flexibility.

[0039] Next, the appearance generation phase includes steps S50, S60, S70 and S80, which work together to create a 3D room model. In step S50, the appearance generation module 120 uses the output of the layout generation phase and performs an equirectangular projection on it to produce a semantic layout. The semantic layout encodes the basic spatial information of the room and its objects, thereby achieving appearance generation. In step S60, the appearance generation module 120 combines the ControlNet model network and applies cycle-consistent sampling to generate a single panorama that captures the visual details of the entire room from a 360-degree perspective. Cycle-consistent sampling ensures that the visual appearance remains consistent in different areas of the panorama. In step S70, the NeRF module 122 uses the panorama to reconstruct the basic 3D room model and incorporate spatial depth into the scene. The NeRF model mainly focuses on capturing the structural details of the basic 3D room model. In step S80, the PeRF module 124 refines the basic 3D room model by combining layout constraints to ensure that the position, size and orientation of the objects are consistent with the expected design. Furthermore, the obtained layout-guided 3D room model can be further updated by the panoramic update module 126, which can provide dynamic revisions if the user changes the layout (e.g., moves or resizes objects). This function provides flexibility for iterative design and real-time adjustment. After step S80, the 3D room modeling process is considered to be completed, and a final 3D room model is generated for use or visualization.

[0040] The proposed method achieves the following effects: separation of the two-stage generation process; compact layout generation and layout-guided panoramic image generation; and flexible control and editing of the generated 3D room model by the user. Compared with Text2Room, the proposed method is able to generate reasonable layouts and vivid appearances. Here, "separating the two-stage generation process" means that the appearance generation module generates the 3D layout of the room and the NeRF module generates the basic 3D room model, which are different stages executed sequentially.

[0041] Figure 3A , Figure 3B , Figure 4A , Figure 4B , Figure 5A and Figure 5B A qualitative illustration of a system 100 using the method described in an embodiment of the present invention is provided. Figure 3A and Figure 3B Demonstrated a living room model; Figure 4A and Figure 4B Demonstrated a study model; Figure 5A and Figure 5BA bedroom model is demonstrated. In each figure, the original color rendering of the generated 3D room model is shown on the left, while its geometric rendering is presented on the right to highlight the geometric quality provided by the proposed method. Therefore, the proposed method can generate various common room types, and they can all follow the room layout of professional designer style.

[0042] As described above, the proposed system and method introduce a two-stage approach to generate 3D room meshes from text input, focusing on geometric layout generation and appearance generation. This separate-stage approach allows for better capture of real-world layout constraints while producing a vivid and detailed visual appearance. A key advantage of the proposed method is its flexibility, enabling users to easily adjust the size, semantic category, and position of furniture in the generated 3D room model. In addition, the method also includes a method for using a pre-trained latent image diffusion model to generate and edit panoramic images, ensuring high-quality results without expensive editor-specific training. Therefore, this flexible and efficient process makes it suitable for creating and manipulating 3D materials for indoor scenes based on text descriptions and user interactions.

[0043] The two-stage generation approach optimizes computational efficiency by leveraging a structured workflow. The separation of layout modeling and appearance generation ensures that each stage focuses on its specific task, thereby reducing the computational load of each stage. This separation results in reduced power consumption during operation as the system manages resources efficiently. In addition, the method speeds up computation by handling complex tasks at different stages, ultimately maximizing computational efficiency and improving overall performance.

[0044] The functional units and modules of the systems and methods according to the embodiments disclosed herein can be implemented in the form of hardware or software. That is, the system claimed for protection can be implemented completely as a machine instruction or as a combination of a machine instruction and a hardware element. Hardware elements include, but are not limited to, computing devices, computer processors, or electronic circuits, including but not limited to application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), microcontrollers, and other programmable logic devices configured or programmed according to the teachings of this disclosure. A person skilled in the art of software or electronics can easily prepare computer instructions or software codes running in a computing device, a computer processor, or a programmable logic device based on the teachings of this disclosure.

[0045] The system may include computer storage media, transient and non-transitory memory devices having computer instructions or software codes stored therein, which may be used to program or configure a computing device, a computer processor, or an electronic circuit to perform any of the processes of the present invention. The storage media, transient and non-transitory memory devices may include, but are not limited to, floppy disks, optical disks, Blu-ray disks, DVDs, CD-ROMs, magneto-optical disks, ROMs, RAMs, flash memory devices, or any type of medium or device suitable for storing instructions, codes, and / or data.

[0046] The system may also be configured as a distributed computing environment and / or cloud computing environment, where all or part of the machine instructions are executed in a distributed manner by one or more processing devices interconnected by a communication network, such as an intranet, a wide area network (WAN), a local area network (LAN), the Internet, and other forms of data transmission media.

[0047] The above description of the present invention is provided for the purpose of illustration and description. It is not intended to be exhaustive or to limit the present invention to the precise form disclosed. For those skilled in the art, many modifications and variations will be apparent.

[0048] The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, thereby enabling others skilled in the art to understand the invention for various embodiments and with various modifications as are suited to the particular use contemplated.

Claims

1. A system for analyzing 3D indoor scenes based on computer generation, comprising: a user interface configured to receive user input about the room from the user in the form of text input and convert it into a language or code recognizable by the system for processing; a text processing module, in communication with the user interface, configured to receive the user input and process it into a scene description; a scenario code generator, in communication with the text processing module, configured to convert the scenario description from the text processing module into a set of scenario codes using a scenario code diffusion model; a layout generation module, in communication with the scene code generator, configured to generate a 3D layout of a room using an oriented bounding box based on the scene code, wherein the 3D layout of the room preserves spatial integrity and relationships between objects specified by the scene code; an appearance generation module, in communication with the layout generation module, configured to convert the 3D layout of the room from the layout generation module into a visual representation of the room, wherein the appearance generation module is further configured to convert the 3D layout of the room into a semantic layout using an equirectangular projection and to generate a single panoramic image of the room based on the semantic layout; A Neural Radiance Field (NeRF) module, in communication with the appearance generation module, configured to construct a basic 3D room model based on the panoramic image and generate representation objects of the room by capturing spatial depth; as well as A panoramic enhanced radiated field (PeRF) module, in communication with the NeRF module, is configured to refine the basic 3D room model by enhancing visual coherence, thereby generating a fully refined 3D room model.

2. According to the system of claim 1, the generation phase of the 3D layout of the room by the appearance generation module and the generation phase of the basic 3D room model by the NeRF module are different phases executed in sequence.

3. According to the system according to claim 1, the scene code generator receives the scene description from the text processing module and processes it through the multi-layer structure of the QKV (Query, Key, Value) mechanism through the scene code diffusion model, and the scene code diffusion model is configured to gradually refine the scene description into objects of structured representation.

4. The system according to claim 3, wherein during the translation process of the scene code diffusion model, scene code noise is embedded by the scene code diffusion model to introduce variation and flexibility.

5. The system of claim 1, wherein the layout generation module uses the oriented bounding boxes to represent key objects or key elements in the room scene to provide a modular way to define the geometry and layout of the room.

6. The system according to claim 1, further comprising: A layout modification module communicates with the layout generation module and is configured to allow a user to interactively modify the 3D layout of the room, wherein the layout modification module is further configured to provide an interface so that the user is allowed to adjust at least one object in the scene code based on the displayed 3D layout of the room through the interface, thereby directly changing the applied scene code.

7. The system of claim 6, the layout modification module being further configured to update the applied scene code to reflect the user's modifications, thereby maintaining consistency between the visual representation of the room and scene data.

8. The system of claim 1, the semantic layout capturing spatial relationships and object locations within a room to generate a visual representation in response to user input.

9. The system according to claim 1, wherein the appearance generation module generates the single panoramic image of the room by adopting cycle-consistent sampling using a ControlNet model.

10. The system of claim 1, further comprising: A panoramic update module communicates with the PeRF module and is configured to dynamically update the panoramic image or the fully detailed 3D room model according to any modifications made by the user when the user views the fully detailed 3D room model presented by the PeRF module and makes modifications.

11. A method of using a system, the system being based on computer-generated 3D indoor scene analysis, wherein the method comprises: receiving, via the user interface, user input from a user regarding the room in the form of text input; Through the user interface, convert the user input into a language or code recognizable by the system for processing; Receiving the user input through a text processing module and processing it into a scene description; By means of a scenario code generator, the scenario description from the text processing module is converted into a set of scenario codes using a scenario code diffusion model; generating, by a layout generation module, a 3D layout of a room using an oriented bounding box based on the scene code, wherein the 3D layout of the room preserves spatial integrity and relationships between objects specified by the scene code; converting, by an appearance generation module, the 3D layout of the room from the layout generation module into a visual representation of the room, wherein the appearance generation module is further configured to convert the 3D layout of the room into a semantic layout using an equirectangular projection and to generate a single panoramic image of the room based on the semantic layout; Building a basic 3D room model based on the panoramic image through a Neural Radiance Field (NeRF) module, and generating representation objects of the room by capturing spatial depth; as well as The basic 3D room model is refined using enhanced visual coherence through a panoramic enhanced radiant field (PeRF) module, thereby generating a fully refined 3D room model. 12 . The method according to claim 11 , wherein the generation phase of the 3D layout of the room by the appearance generation module and the generation phase of the basic 3D room model by the NeRF module are different phases executed in sequence.

13. According to the method of claim 11, the scene code generator receives the scene description from the text processing module, and processes it through the multi-layer structure of the QKV (Query, Key, Value) mechanism through the scene code diffusion model, and the scene code diffusion model is configured to gradually refine the scene description into structured representation objects.

14. The method according to claim 13, wherein during the translation process of the scene code diffusion model, scene code noise is embedded by the scene code diffusion model to introduce variation and flexibility.

15. The method of claim 11, wherein the layout generation module uses the oriented bounding boxes to represent key objects or key elements in the room scene to provide a modular way to define the geometry and layout of the room.

16. The method according to claim 11, further comprising: Through the layout modification module, the user is allowed to interactively modify the 3D layout of the room, wherein the layout modification module is also configured to provide an interface so that the user is allowed to adjust at least one object in the scene code based on the displayed 3D layout of the room through the interface, thereby directly changing the applied scene code.

17. The method of claim 16, the layout modification module being further configured to update the applied scene code to reflect the user's modification, thereby maintaining consistency between the visual representation of the room and scene data.

18. The method of claim 11, wherein the semantic layout captures spatial relationships and object locations within a room to generate a visual representation in response to user input.

19. The method according to claim 11, wherein the appearance generation module generates the single panoramic image of the room by adopting cycle-consistent sampling using a ControlNet model.

20. The method according to claim 11, further comprising: Through the panoramic update module, when a user views the fully refined 3D room model presented by the PeRF module and makes modifications, the panoramic image or the fully refined 3D room model is dynamically updated according to any modifications made by the user.

Images