System and method for generating a controllable text-to-3D room mesh with layout constraints

Abstract

The present invention provides a system for generating a 3D indoor scene from text input. The system comprises a user interface, a text processing module, a scene code generator, a layout generation module, an appearance generation module, a NeRF module and a PeRF module. A user can input text describing a room, and the text is processed into scene code. The generated scene code is then used to guide the generation of a 3D layout using directional bounding boxes, ensuring spatial integrity. The appearance generation module then creates visual representation objects of the room using panoramic images. The NeRF module can construct a base 3D model, which is then refined by the PeRF module to enhance visual coherence.

Description

Technical Field

[0001] This invention generally relates to artificial intelligence (AI) technology for room image generation. More specifically, this invention relates to a system and method for generating adjustable text-to-text 3D room grids with layout constraints. Background Technology

[0002] High-quality 3D models are crucial for many applications, from interior design and games to simulated AI avatars. Among all 3D content, interior scenes receive particular attention. Typically, 3D interior scenes are manually constructed by professional designers, a process that is both time-consuming and expensive. While generative modeling techniques like Dreamfusion, Fantasia3D, Magic3D, and 3DFuse can simplify the process of generating 3D models from text descriptions, applying these techniques to text-driven 3D interior scene generation remains challenging. Interior scenes have strong semantic layout constraints; for example, adjacent walls must be perpendicular, and televisions typically face sofas, making this complexity far greater than object generation.

[0003] Existing text-driven 3D interior scene generation methods, such as Text2Room and Text2NeRF, are based on incremental frameworks. They generate 3D interior scenes by generating views from different angles frame by frame and then reconstructing the 3D mesh of the room based on these view images. However, this incremental approach often fails to capture the overall layout of the room well, resulting in semantically unreasonable and unrealistic generated results.

[0004] For example, in one of the scenes generated by Text2Room, the living room might contain multiple duplicate cabinet objects, and the furniture layout doesn't follow a conventional pattern. This problem is known as the "Penrose Triangle problem," where while the local 3D structures may appear plausible, the overall structure lacks consistency. Furthermore, previous methods couldn't support user interoperability because the generated 3D geometry and textures were not editable.

[0005] Therefore, in the field of indoor scene generation, there is an urgent need for a new method that can generate reasonable 3D structures with consistent internal layouts and provide user-friendly editing functions. Summary of the Invention

[0006] The purpose of this invention is to provide a system and method to solve the aforementioned problems existing in the prior art.

[0007] This invention provides a flexible method for generating editable and structurally sound 3D (three-dimensional) interior scenes. The proposed method comprises two stages: a layout generation stage and an appearance generation stage. In the layout generation stage, the scene layout is parameterized using scene code and a text-driven diffusion model is learned to generate a text-driven layout. In the appearance generation stage, a ControlNet model is fine-tuned to generate a vivid panoramic view of the room guided by the scene layout. Subsequently, a high-quality 3D room model with a structurally sound layout and realistic textures is generated. One of the innovative features of the proposed method is its support for interactive 3D scene editing. Furthermore, a mask-guided editing method is proposed, allowing users to adjust the size, position, and semantic categories of furniture within the room.

[0008] According to a first aspect of the present invention, a system for computer-generated 3D indoor scene analysis is provided. The system includes: a user interface, a text processing module, a scene code generator, a layout generation module, an appearance generation module, a neural radiation field (NeRF) module, and a panoramic enhanced radiation field (PeRF) module. The user interface is configured to receive user input about a room in the form of text input and convert it into a language or code recognizable by the system for processing. The text processing module communicates with the user interface and is configured to receive the user input and process it into a scene description. The scene code generator communicates with the text processing module and is configured to convert the scene description from the text processing module into a set of scene codes using a scene code diffusion model. The layout generation module communicates with the scene code generator and is configured to generate a 3D layout of the room using oriented bounding boxes based on the scene codes, wherein the 3D layout of the room preserves the spatial integrity and relationships between objects specified by the scene codes. An appearance generation module, communicating with the layout generation module, is configured to convert the 3D layout of the room from the layout generation module into a visual representation of the room. The appearance generation module is further configured to convert the 3D layout of the room into a semantic layout using an isometric rectangular projection and generate a single panoramic image of the room based on the semantic layout. A NeRF module, communicating with the appearance generation module, is configured to construct a basic 3D room model based on the panoramic image and generate representation objects of the room by capturing spatial depth. A PeRF module, communicating with the NeRF module, is configured to refine the basic 3D room model by enhancing visual coherence, thereby generating a fully refined 3D room model.

[0009] According to a second aspect of the present invention, a method for using a system is provided, the system being based on computer-generated 3D interior scene analysis, wherein the method includes the following steps: receiving user input about a room in the form of text input from a user through a user interface; converting the user input into a language or code recognizable by the system through the user interface for processing; receiving the user input and processing it into a scene description through a text processing module; converting the scene description from the text processing module into a set of scene codes using a scene code diffusion model through a scene code generator; and generating a 3D layout of the room using oriented bounding boxes based on the scene codes through a layout generation module, wherein... The 3D layout of the room preserves the spatial integrity and relationships between objects specified in the scene code; the appearance generation module converts the 3D layout of the room from the layout generation module into a visual representation of the room, wherein the appearance generation module is further configured to convert the 3D layout of the room into a semantic layout using equidistant rectangular projection, and generate a single panoramic image of the room based on the semantic layout; the NeRF module constructs a basic 3D room model based on the panoramic image and generates representation objects of the room by capturing spatial depth; and the PeRF module refines the basic 3D room model using enhanced visual coherence, thereby generating a fully refined 3D room model.

[0010] In some embodiments, a computer-implemented method is also provided for generating 3D material of an interior scene using a set of trained machine learning models. The method includes: providing a textual description of the interior scene as input to the trained machine learning models; obtaining a 3D bounding box of the room layout; generating a visually appealing panoramic image of the room based on the 3D room layout using another fine-tuned machine learning model; and reconstructing and texturing a 3D mesh of the room based on the generated panoramic image through depth estimation and surface reconstruction.

[0011] In some embodiments, a computer-implemented method is also provided for generating editable 3D assets of an interior scene. The method includes parameterizing the interior scene into a flexible, user-interactively editable overall scene code. The method also includes representing the interior scene as a panoramic image, while also taking into account a physically reasonable room layout.

[0012] In some embodiments, a computer-implemented method is also provided for generating and editing panoramic images. The method includes achieving high-quality results with cycle consistency through a pre-trained latent image diffusion model, without requiring expensive editing-specific training.

[0013] The above configuration achieves the following positive effects:

[0014] (1): A two-stage method for generating 3D rooms from text input is designed. The method separates the generation process of geometric layout and visual appearance, allowing room layout constraints to be captured from real-world data while creating visually appealing results.

[0015] (2): Separating the generation process of geometric layout and visual appearance provides flexible control and editability over the generated 3D room model. Users can easily adjust the size, semantic category, and position of furniture.

[0016] (3): A novel method for generating and editing panoramic images is proposed. The method uses a pre-trained latent image diffusion model to achieve high-quality results with cycle consistency without the need for expensive editing-specific training. Attached Figure Description

[0017] In the following, embodiments of the present invention will be described in more detail with reference to the accompanying drawings, wherein:

[0018] Figure 1 A schematic diagram of the model framework of a system for analyzing computer-generated 3D indoor scenes according to an embodiment of the present invention is shown;

[0019] Figure 2 A schematic diagram illustrating a method for generating a 3D room model using the proposed system according to an embodiment of the present invention is provided.

[0020] Figure 3A , Figure 3B , Figure 4A , Figure 4B , Figure 5A and Figure 5B A qualitative demonstration diagram of a system using the method described in the embodiments of the present invention is provided. Detailed Implementation

[0021] In the following description, as preferred examples, systems and methods for generating adjustable text-transformed 3D room grids with layout constraints are illustrated. It will be apparent to those skilled in the art that modifications, including additions and / or substitutions, can be made without departing from the scope and spirit of the invention. Specific details may be omitted to avoid obscuring the invention; however, this disclosure is prepared to enable those skilled in the art to practice the teachings herein without engaging in undue experimentation.

[0022] Text-driven 3D interior scene generation has potential applications in the gaming, film, and AR / VR industries. However, existing methods fail to accurately capture room layouts and lack the ability to flexibly edit individual objects within a room. To address these issues, this disclosure proposes a solution called "Ctrl-Room," a system capable of generating realistic 3D room models with designer-style layouts and high-quality textures based on simple text prompts. Furthermore, it allows for interactive editing operations, such as resizing or moving furniture. The proposed method separates layout modeling and appearance generation. It comprises two stages: the first, a "layout generation stage," trains a text-based conditional diffusion model to learn the layout distribution using global scene code parameterization; and the second, a "appearance generation stage," uses a finely tuned ControlNet, guided by the 3D scene layout and text prompts, to generate a vivid panoramic image of the room. This enables the creation of high-quality 3D room models with realistic layouts and vibrant textures. Through scene code parameterization, the generated room models can be easily edited using a mask-guided editing module, allowing for costly editing tasks without requiring specialized training. Overall, Ctrl-Room provides a powerful solution for text-driven 3D indoor scene generation in various applications such as gaming, film, and AR / VR.

[0023] The following describes methods, systems, and computer-readable media (e.g., computer-based 3D interior scene material generation) for generating corresponding 3D (three-dimensional) editable materials from text descriptions.

[0024] Figure 1 A schematic diagram of the model framework of a computer-generated 3D indoor scene analysis system 100 according to an embodiment of the present invention is shown. The system 100 includes a user interface 102, a text processing module 110, a scene code generator 112, a layout generation module 114, a layout modification module 116, an appearance generation module 120, a NeRF (neural radiance field) module 122, a PeRF (panoptic-enhanced radiance field) module 124, and a panoptic update module 126.

[0025] These components are arranged for a two-stage approach to generate editable and physically plausible 3D room meshes from text prompts, wherein the two stages include a layout generation stage and an appearance generation stage. Text processing module 110, scene code generator 112, layout generation module 114, and layout modification module 116 collaborate in the layout generation stage. Appearance generation module 120, NeRF module 122, PeRF module 124, and panorama update module 126 collaborate in the appearance generation stage.

[0026] The layout generation phase includes providing overall scene code configured for a parametric room layout that supports flexible user editing. This phase also provides a generative model (e.g., presented as layout generation module 114) to learn the distribution and arrangement of the room layout. The appearance generation phase includes fine-tuning the generative model (e.g., NeRF module 122 / PeRF module 124) to generate a vivid panoramic image of the interior scene under room layout constraints, and further generate / provide a high-quality 3D room with a well-structured layout and realistic textures. Furthermore, users can adjust the size, position, and / or semantic categories of furniture within the room using mask-guided editing options (at least accessible through user interface 102).

[0027] User interface 102 is configured to receive input from a user regarding a room, which can be text or other types of input information. The user input may relate to the desired room type (e.g., living room, study, or bedroom). User interface 102 can convert this input into a language or code that system 100 can recognize and process. By translating the user input into a system-readable format, user interface 102 ensures that this input is effectively delivered to the appropriate components within system 100. These components can then generate appropriate responses based on the user's input, thereby enabling interaction and communication between the user and system 100.

[0028] The text processing module 110 communicates with the user interface 102 and is configured to receive at least one text prompt provided by the user (which may be, for example, user input) and process it into a scene description. Therefore, the input to the text processing module 110 is the text prompt, and the output of the text processing module 110 is the scene description.

[0029] Scene code generator 112 communicates with text processing module 110 and is configured to convert the scene description from text processing module 110 into tunable scene code using a scene code diffusion model. Specifically, scene code generator 112 includes a scene code diffusion model. During the conversion process, scene code generator 112 receives and processes the scene description from text processing module 110 through a multi-layered structure of the QKV (Query, Key, Value) mechanism in the scene code diffusion model. The scene code diffusion model is configured to progressively refine the scene description into structured representations of objects. During this conversion process, scene code noise can be embedded to introduce variation and flexibility, thereby enabling more robust scene generation. As diffusion progresses, the noise gradually decreases, and then scene code generator 112 can output a set of scene code representing the scene structure, facilitating preparation for further processing or rendering. Therefore, the input of scene code generator 112 is the scene description, and the output of scene code generator 112 is a set of scene code.

[0030] The layout generation module 114 communicates with the scene code generator 112 and is configured to generate a 3D layout of the room using oriented bounding boxes based on scene codes. The scene codes generated by the scene code generator 112 can be used as structured representations of objects that associate room elements and spatial relationships. The layout generation module 114 takes the scene code set as input and converts it into a 3D layout by positioning and resizing objects within the room using oriented bounding boxes. These bounding boxes represent key objects or key elements in the scene, such as walls, furniture, or fixed objects, and provide a modular way to define the geometry and arrangement of the room. The output of the layout generation module 114 is a 3D layout of the room that preserves the spatial integrity and relationships between objects specified by the scene codes. Therefore, the input to the layout generation module 114 is a set of scene codes, and the output of the layout generation module 114 is a 3D layout of the room.

[0031] The layout modification module 116 communicates with the layout generation module 114 and is configured to allow users to interactively modify the layout (e.g., change the position or size of objects, and modification requests can be fed to the layout modification module 116 from the user interface 102). Specifically, the layout modification module 116 receives the 3D layout of the room and displays it to the user (e.g., the user who initially provides input to the system 100). The layout modification module 116 provides an interface that allows the user to adjust at least one scene code based on the displayed 3D layout of the room, such as changing the position or size of objects within the scene (i.e., within the room), enabling the system 100 to be flexible. The user's modification actions can directly change the applied scene code, which represents the object structure and spatial attributes. For example, when the user changes the position or size of an object, the layout modification module 116 updates the relevant scene code to reflect these changes, thereby ensuring that the system 100 remains flexible and that there is consistency between the visual representation of the scene and the scene data. By providing interactive and dynamic control over the room layout, the layout modification module 116 enhances user engagement and allows for personalized room designs, easily adaptable to specific preferences or requirements. Therefore, the input to the layout modification module 116 is the 3D layout of the room, and the output is updated data of the 3D room layout.

[0032] The appearance generation module 120 communicates with the layout generation module 114 and is configured to convert a 3D room layout (e.g., a 3D layout of a room from the layout generation module 114) into visual representations of the room objects. The appearance generation module 120 uses isometric rectangular projection to convert the detailed layout provided by the layout generation module 114 into a semantic layout. The semantic layout captures spatial relationships and object positions within the room, thereby responding to user input and generating visual representations of objects.

[0033] After the transformation, the appearance generation module 120 employs a cyclically consistent sampling technique to ensure that the generated visual effect maintains consistency in spatial representation and input content. In one embodiment, the transformation is implemented using a ControlNet model, a neural network used to refine the visual output by maintaining consistency across different viewpoints. The output of the appearance generation module 120 is a single panoramic image that comprehensively represents the visual details of the room. Therefore, the input to the appearance generation module 120 is a 3D layout from the layout generation module 114, while the output of the appearance generation module 120 is a single panoramic image of the room.

[0034] NeRF module 122 communicates with appearance generation module 120 and is configured to construct / create a basic 3D room model based on the panoramic image generated by appearance generation module 120. NeRF module 122 generates representations of objects within the room by capturing spatial depth within the scene. In this regard, NeRF module 122 utilizes neural radiation techniques and analyzes the panoramic image, enabling it to capture and reconstruct the spatial depth and complexity of the scene. It processes the image to discern object surfaces and spatial relationships within the room, ensuring that the basic 3D room model reflects the visual appearance of the room. Therefore, the input to NeRF module 122 is the panoramic image generated by appearance generation module 120, and the output of NeRF module 122 is a basic 3D room model with realistic textures and spatial consistency (e.g., a NeRF model of the room). The results from NeRF module 122 are provided to PeRF module 124 for enhancement, with PeRF module 124 designed to focus on layout-guided 3D rendering.

[0035] PeRF module 124 communicates with NeRF module 122 and is configured to refine the basic 3D room model generated by NeRF module 122, incorporating layout constraints to ensure spatial consistency and semantic accuracy. The steps of PeRF module 124 can enhance the visual coherence of the room by aligning the position, size, and texture of objects with the layout. Therefore, the output of PeRF module 124 is a fully refined 3D room model (e.g., a layout-guided PeRF model of the room).

[0036] The panorama update module 126 communicates with the PeRF module 124 and is configured to dynamically update the panorama and 3D room model based on any modifications made by the user to the layout (such as modification requests input from the user interface 102 to the panorama update module 126), such as changes in object position or size. The panorama update module 126 ensures that the updated layout is reflected in the panorama image and 3D room model. For example, when the user views and modifies the fully detailed 3D room model presented by the PeRF module 124, the panorama update module 126 can dynamically update the 3D room model and request the appearance generation module 120, NeRF module 122, and PeRF module 124 to regenerate the 3D room model based on the user's changes. This allows the corresponding components to generate a revised 3D room model that better meets the user's latest requirements. The PeRF module 124 then provides the final output of the 3D room model.

[0037] Figure 2A schematic diagram illustrating a method for generating a 3D room model using the proposed system 100 according to an embodiment of the present invention is provided. In this invention, the proposed method, referred to as the CtrlRoom solution, enables the generation of a 3D room mesh from controllable text with layout constraints. In short, the proposed method involves a 3D room modeling process divided into two phases: a layout generation phase and an appearance generation phase. In the layout generation phase, the objective is to generate a 3D scene layout from the input text. In the appearance generation phase, the objective is to generate a single panoramic view to represent the appearance guided by the scene layout. Following the appearance generation phase, the expected result is the reconstruction of a complete 3D room model. During the generation process, a flexible scene layout defined by a set of oriented bounding boxes is used, allowing for flexible editing by the user. For example, after the user moves a chair, the method further includes generating a new panoramic view based on the updated scene layout and then updating the 3D room model accordingly.

[0038] like Figure 2 As shown, the layout generation stage includes steps S10, S20, S30, and S40. In step S10, the user inputs a text prompt through the user interface 102. Then, the text processing module 110 processes the text prompt. In step S20, the scene code generator 112 initiates scene code diffusion based on the text prompt. In step S30, the scene code generator 112 generates a set of scene codes. This set of scene codes can be recorded in a user-friendly format. In one embodiment, the scene code is represented in the form of a structured table, where "rows" represent different entities in the scene (e.g., "walls" or "objects"), and "columns" correspond to different attributes or parameters (e.g., Ci, Li, Si, Ri) associated with each entity / element. For example, attributes or parameters include the element's category or form, position or layout information, element size or scale, and element orientation or rotation positioning. This table format facilitates the user's programmatic control, generation, and modification of the scene layout based on the scene code. In step S40, the layout generation module 114 generates an initial 3D layout of the room based on the scene code using oriented bounding boxes. During the generation of the initial 3D layout, system 110 allows users to make interactive user modifications through layout modification module 116, such as changing the position or size of objects in the scene, thereby allowing users to adjust at least one item in the scene code, thus providing flexibility to system 100.

[0039] Next, the appearance generation stage includes steps S50, S60, S70, and S80, which work together to create a 3D room model. In step S50, the appearance generation module 120 uses the output of the layout generation stage to perform isorectangular projection on it to generate a semantic layout. The semantic layout encodes the basic spatial information of the room and its objects, thereby achieving appearance generation. In step S60, the appearance generation module 120 combines a ControlNet model network and applies cyclic consistent sampling to generate a single panoramic view that captures the visual details of the entire room from a 360-degree perspective. Cyclic consistent sampling ensures that the visual appearance remains consistent across different regions of the panoramic view. In step S70, the NeRF module 122 uses the panoramic view to reconstruct the basic 3D room model and incorporates spatial depth into the scene. The NeRF model primarily focuses on capturing the structural details of the basic 3D room model. In step S80, the PeRF module 124 refines the basic 3D room model by incorporating layout constraints, ensuring that the position, size, and orientation of objects are consistent with the intended design. Furthermore, the obtained layout-guided 3D room model can be further updated via the panorama update module 126. If the user makes changes to the layout (e.g., moving or resizing objects), the panorama update module 126 can provide dynamic revisions. This feature provides flexibility for iterative design and real-time adjustments. After step S80, the 3D room modeling process is considered complete, and a usable or visualized final 3D room model is produced.

[0040] The proposed method achieves the following effects: separating the two-stage generation process; compact layout generation and layout-guided panoramic image generation; and flexible user control and editing of the generated 3D room model. Compared to Text2Room, the proposed method can generate reasonable layouts and vivid appearances. Here, "separating the two-stage generation process" means that the appearance generation module generates the 3D layout of the room, and the NeRF module generates the basic 3D room model, which are executed sequentially in different stages.

[0041] Figure 3A , Figure 3B , Figure 4A , Figure 4B , Figure 5A and Figure 5B A qualitative demonstration diagram of a system 100 using the method described in the embodiments of the present invention is provided. Figure 3A and Figure 3B A living room model was demonstrated; Figure 4A and Figure 4B A model of a study room was demonstrated; Figure 5A and Figure 5BA bedroom model is demonstrated. In each image, the left side shows the original color rendering of the generated 3D room model, while the right side presents its geometric rendering to highlight the geometric quality provided by the method of the present invention. Therefore, the proposed method can generate a variety of common room types, all of which can follow the room layout style of professional designers.

[0042] As described above, the proposed system and method introduce a two-stage approach to generate 3D room meshes from text input, focusing on geometric layout generation and appearance generation. This separate-stage approach allows for better capture of real-world layout constraints while producing a vivid and detailed visual appearance. A key advantage of the proposed method is its flexibility, enabling users to easily adjust the size, semantic category, and position of furniture in the generated 3D room model. Furthermore, the method includes a pre-trained latent image diffusion model for generating and editing panoramic images, ensuring high-quality results without the need for expensive, dedicated editing training. Therefore, this flexible and efficient process makes it suitable for creating and manipulating 3D assets for interior scenes based on text descriptions and user interactions.

[0043] The two-stage generation method optimizes computational efficiency by leveraging a structured workflow. Separating layout modeling from appearance generation ensures that each stage focuses on its specific task, reducing the computational load at each stage. This separation leads to reduced power consumption during runtime due to efficient resource management. Furthermore, the method accelerates computation by handling complex tasks at different stages, ultimately maximizing computational efficiency and improving overall performance.

[0044] The functional units and modules of the systems and methods disclosed herein can be implemented in hardware or software. That is, the claimed system can be implemented entirely as machine instructions or as a combination of machine instructions and hardware elements. Hardware elements include, but are not limited to, computing devices, computer processors, or electronic circuits, including, but not limited to, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, and other programmable logic devices configured or programmed according to the teachings of this disclosure. Those skilled in the art of software or electronics can readily prepare computer instructions or software code to run in computing devices, computer processors, or programmable logic devices based on the teachings of this disclosure.

[0045] The system may include a computer storage medium, a transient and non-transient memory device storing computer instructions or software code, which can be used to program or configure a computing device, computer processor, or electronic circuit to perform any of the processes of the present invention. The storage medium, transient and non-transient memory device may include, but is not limited to, floppy disks, optical disks, Blu-ray discs, DVDs, CD-ROMs, magneto-optical disks, ROMs, RAMs, flash memory devices, or any type of medium or device suitable for storing instructions, code, and / or data.

[0046] The system can also be configured as a distributed computing environment and / or a cloud computing environment, in which all or part of the machine instructions are executed in a distributed manner by one or more processing devices interconnected by a communication network, such as an intranet, a wide area network (WAN), a local area network (LAN), the Internet, and other forms of data transmission media.

[0047] The foregoing description of the present invention is provided for illustrative purposes. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations will be apparent to those skilled in the art.

[0048] These embodiments were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling those skilled in the art to understand the various embodiments of the invention and the various modifications suitable for particular purposes.

Claims

1. A system for analyzing computer-generated 3D indoor scenes, comprising: The user interface is configured to receive user input about the room from the user in the form of text input and convert it into a language or code that the system can recognize for processing; The text processing module communicates with the user interface and is configured to receive the user input and process it into a scene description. A scene code generator, communicating with the text processing module, is configured to use a scene code diffusion model to convert the scene description from the text processing module into a set of scene codes; The layout generation module communicates with the scene code generator and is configured to generate a 3D layout of a room using a directional bounding box based on the scene code, wherein the 3D layout of the room preserves the spatial integrity and relationships between objects specified by the scene code. An appearance generation module, communicating with the layout generation module, is configured to convert the 3D layout of the room from the layout generation module into a visual representation of the room. The appearance generation module is further configured to convert the 3D layout of the room into a semantic layout using an equidistant rectangular projection. The semantic layout captures spatial relationships and object positions within the room, thereby generating a visual representation in response to user input. The semantic layout is used as input to generate a single panoramic image of the room based on the semantic layout using a neural network. The Neural Radiation Field (NeRF) module, which communicates with the appearance generation module, is configured to construct a basic 3D room model based on the panoramic image and generate representation objects of the room by capturing spatial depth. as well as A panoramic enhanced radiation field (PeRF) module, communicating with the NeRF module, is configured to refine the basic 3D room model by enhancing visual coherence, thereby generating a fully refined 3D room model.

2. In the system according to claim 1, the generation stage of the 3D layout of the room by the appearance generation module and the generation stage of the basic 3D room model by the NeRF module are different stages executed sequentially.

3. In the system according to claim 1, the scene code generator receives the scene description from the text processing module and processes it through a multi-layer structure of QKV (Query, Key, Value) mechanism via the scene code diffusion model, wherein the scene code diffusion model is configured to gradually refine the scene description into structured representations of objects.

4. In the system according to claim 3, during the translation process of the scene code diffusion model, scene code noise is embedded by the scene code diffusion model to introduce variation and flexibility.

5. The system according to claim 1, wherein the layout generation module uses the directional bounding box to represent key objects or key elements in the room scene, so as to provide a modular way to define the geometry and layout of the room.

6. The system according to claim 1, further comprising: A layout modification module, which communicates with the layout generation module, is configured to allow a user to interactively modify the 3D layout of the room. The layout modification module is further configured to provide an interface that allows the user to adjust at least one object in the scene code based on the displayed 3D layout of the room, thereby directly changing the applied scene code.

7. The system according to claim 6, wherein the layout modification module is further configured to update the applied scene code to reflect the user's modification, thereby maintaining consistency between the visual representation of the room and the scene data.

8. The system according to claim 1, wherein the appearance generation module generates the single panoramic image of the room by using a ControlNet model and employing cyclic consistent sampling.

9. The system according to claim 1, further comprising: A panoramic update module, which communicates with the PeRF module, is configured to dynamically update the panoramic image or the fully detailed 3D room model based on any modifications made by the user when the user views and modifies the fully detailed 3D room model presented by the PeRF module.

10. A method using a system, the system being based on computer-generated 3D indoor scene analysis, wherein the method includes: The system receives user input about the room in the form of text input through the user interface. The user interface is used to convert the user input into a language or code that the system can recognize for processing. The text processing module receives the user input and processes it into a scene description. The scene description from the text processing module is converted into a set of scene codes using a scene code generator and a scene code diffusion model. The layout generation module generates a 3D layout of a room using a directional bounding box based on the scene code, wherein the 3D layout of the room preserves the spatial integrity and relationships between objects specified by the scene code. The appearance generation module converts the 3D layout of the room from the layout generation module into a visual representation of the room. The appearance generation module is further configured to convert the 3D layout of the room into a semantic layout using an equidistant rectangular projection. The semantic layout captures the spatial relationships and object positions within the room, thereby generating a visual representation in response to the user input. The semantic layout is used as input to generate a single panoramic image of the room based on the semantic layout using a neural network. A basic 3D room model is constructed based on the panoramic image using the Neural Radiation Field (NeRF) module, and the room's representation objects are generated by capturing spatial depth. as well as The basic 3D room model is refined using the Panoramic Enhanced Radiation Field (PeRF) module to enhance visual coherence, thereby generating a fully refined 3D room model.

11. The method according to claim 10, wherein the generation stage of the 3D layout of the room by the appearance generation module and the generation stage of the basic 3D room model by the NeRF module are different stages executed sequentially.

12. The method according to claim 10, wherein the scene code generator receives the scene description from the text processing module and processes it through a multi-layer structure of QKV (Query, Key, Value) mechanism via the scene code diffusion model, wherein the scene code diffusion model is configured to gradually refine the scene description into structured representations of objects.

13. The method of claim 12, wherein during the translation process of the scene code diffusion model, scene code noise is embedded by the scene code diffusion model to introduce variation and flexibility.

14. The method of claim 10, wherein the layout generation module uses the directional bounding box to represent key objects or key elements in the room scene, so as to provide a modular way to define the geometry and layout of the room.

15. The method of claim 10, further comprising: The layout modification module allows users to interactively modify the 3D layout of the room. The layout modification module is also configured to provide an interface that allows users to adjust at least one object in the scene code based on the displayed 3D layout of the room, thereby directly changing the applied scene code.

16. The method of claim 15, wherein the layout modification module is further configured to update the applied scene code to reflect user modifications, thereby maintaining consistency between the visual representation of the room and the scene data.

17. The method of claim 10, wherein the appearance generation module generates the single panoramic image of the room by using a ControlNet model and employing cyclic consistent sampling.

18. The method of claim 10, further comprising: Through the panoramic update module, when a user views and modifies the fully detailed 3D room model presented by the PeRF module, the panoramic image or the fully detailed 3D room model is dynamically updated according to any modifications made by the user.

Images