A method for constructing an indoor scene

Abstract

This application provides a method for constructing indoor scenes, applicable to the fields of 3D simulation and artificial intelligence. The method includes: parsing multimodal user input to obtain a structured room description information set; filtering candidate floor plan layouts to obtain a target floor plan layout; segmenting the target floor plan layout into room regions to obtain a set of binary masks for room walls; using the room wall binary mask set and the structured room description information to perform 3D room modeling to obtain a 3D empty room model; rendering the 3D empty room model using a furniture rendering model set obtained based on semantic retrieval to obtain a 3D room scene model; interactively iteratively editing the 3D room scene model based on multi-round user input of multimodal annotation information to obtain an initial 3D indoor full scene model; and performing optimization based on depth-semantic joint backprojection on the initial 3D indoor full scene model to obtain the target 3D indoor full scene model.

Description

Technical Field

[0001] This application relates to the fields of 3D simulation and artificial intelligence technology, specifically to a method for constructing indoor scenes. Background Technology

[0002] With the development of embodied intelligence and simulation training, the demand for highly realistic and controllable indoor virtual scene data is increasing. Existing procedural scene generation methods can generally generate visually realistic scenes, but they still have significant shortcomings in terms of controllability, such as "specifying the precise position of objects, relative relationships, and local style modifications": related solutions emphasize one-time generation realism but lack efficient and sustainable secondary editing mechanisms; related solutions with editing functions mainly rely on natural language, making it difficult for users to accurately express spatial position, direction, and local area constraints; when the generated results do not meet the requirements, related solutions often have to regenerate the entire result, without being able to make low-cost local corrections to the original result. At the same time, in related solutions, there is a lack of an executable and closed-loop technical link between 2D editing intentions and 3D asset placement, making it difficult to reliably map "visual modification intentions" into precise placement results in the 3D scene. Summary of the Invention

[0003] In view of the above problems, this application provides an indoor scene construction method to at least solve one of the existing problems.

[0004] The first aspect of this application provides a method for constructing an indoor scene, comprising: parsing multimodal requirements input by a user to obtain a set of structured room description information; filtering a set of candidate floor plan layouts generated based on the set of structured room description information to obtain a target floor plan layout, and segmenting the target floor plan layout into room regions to obtain a set of binary masks for room walls; performing 3D modeling of the room using the set of binary masks for room walls and the structured room description information corresponding to the target floor plan layout to obtain a 3D empty room model, and rendering the 3D empty room model using a set of furniture rendering models obtained based on semantic retrieval to obtain a 3D room scene model with furniture placement effects; interactively iteratively editing the 3D room scene model based on multimodal annotation information obtained from multiple rounds of interaction with the user to obtain an initial 3D indoor full scene model, and performing optimization based on depth-semantic joint backprojection on the initial 3D indoor full scene model to obtain a target 3D indoor full scene model.

[0005] According to an embodiment of this application, the above-mentioned parsing of user-inputted multimodal requirements to obtain a structured room description information set includes: calling a planning agent to evaluate the user-inputted multimodal requirements in the current global state, obtaining the evaluated multimodal requirements, wherein the current global state includes the three-dimensional scene indoor full scene model, room information set, and contextual history information within the current discrete time step; determining the parsing plugin used by the execution agent based on the evaluated multimodal requirements, and obtaining the structured room description information set by having the execution agent call the parsing plugin to parse the evaluated multimodal requirements.

[0006] According to an embodiment of this application, the above-mentioned filtering of candidate floor plan layout images generated based on a structured room description information set to obtain a target floor plan layout image includes: generating a candidate floor plan layout image set by calling a floor plan layout generation plugin through an execution agent based on the structured room description information set; calling a planning agent to score the candidate floor plan layout image set to obtain a score set, and filtering the candidate floor plan layout image set based on the score set to obtain an initial floor plan layout image; and calling an execution agent to redraw the initial floor plan layout image to obtain the target floor plan layout image.

[0007] According to an embodiment of this application, the above-mentioned method of using the room wall binary mask set and the structured room description information corresponding to the target floor plan to perform 3D room modeling to obtain a 3D empty room model includes: constructing an initial 3D empty room model representing multiple empty rooms in an indoor scene by executing an intelligent agent to call an empty room model generation plugin based on the room wall binary mask set; parsing the target structured description information corresponding to the target floor plan to obtain wall material description information and floor material description information; searching the material asset library based on the wall material description information and floor material description information to obtain a target wall asset set and a floor texture asset set; and rendering the initial 3D empty room model using the target wall asset set and floor texture asset set to obtain a 3D empty room model.

[0008] According to an embodiment of this application, the above-mentioned rendering of a 3D empty room model using a furniture rendering model set obtained based on semantic retrieval to obtain a 3D room scene model with furniture placement effect includes: generating furniture description information for each 3D empty room sub-model in the 3D empty room model by executing an intelligent agent to call a furniture selection plugin, thereby obtaining a furniture description information set; performing semantic retrieval on a furniture asset library using the furniture description information set to obtain a furniture rendering model matching each 3D empty room sub-model, thereby obtaining a furniture rendering model set; and performing image rendering on the 3D empty room model using the furniture rendering model set to obtain a 3D room scene model with furniture placement effect.

[0009] According to an embodiment of this application, the above-mentioned image rendering of a 3D empty room model using a furniture rendering model set to obtain a 3D room scene model with furniture placement effect includes: determining the camera parameters of the virtual camera in each 3D empty room sub-model and the camera position representing the spatial coordinates and orientation of the virtual camera, to obtain a camera parameter set and a camera position set; determining the absolute depth representing the real physical depth value of pixels in each 3D empty room sub-model according to the camera position set, to obtain an absolute depth set; performing initial rendering on each 3D empty room sub-model according to the camera position set, to obtain each initially rendered 3D empty room sub-model; and re-rendering each initially rendered 3D empty room sub-model using the furniture rendering model corresponding to each 3D empty room sub-model according to the camera parameter set and the absolute depth set, to obtain a 3D room scene model with furniture placement effect.

[0010] According to an embodiment of this application, the above-mentioned interactive iterative editing of the 3D room scene model based on the multimodal annotation information obtained from multiple rounds of interaction with the user to obtain the initial 3D indoor full scene model includes: calling an evaluation agent to perform multiple evaluations on the 3D room scene model, obtaining multiple evaluation results, and modifying the 3D room scene model based on the multiple evaluation results to obtain a 3D room scene model that has passed the evaluation; iteratively editing the 3D room scene model that has passed the evaluation based on the multi-round image annotation information, multi-round modification instruction information, and / or multi-round reference images provided by the user for the 3D room scene model that has passed the evaluation, until the preset editing conditions are met to obtain the initial 3D indoor full scene model.

[0011] According to embodiments of this application, the above-mentioned invocation of the evaluation agent to perform multiple evaluations on the 3D room scene model, obtaining multiple evaluation results, includes: invoking the planning agent to determine the parsing quality check plugin and the room structure check plugin used by the evaluation agent; invoking the parsing quality check plugin to perform data compliance checks on the structured room description information corresponding to the 3D room scene model, obtaining data compliance check results, wherein the data compliance check results include field integrity check results, data category legality check results, and description information quality check results; and invoking the room structure check plugin to perform spatial regularity checks on each 3D sub-room scene model in the 3D room scene model, obtaining spatial regularity check results, wherein the spatial regularity check results include room integrity check results, area overlap check results, and room geometric cleanliness check results.

[0012] According to an embodiment of this application, the above-mentioned optimization of the initial 3D indoor full-scene model based on depth-semantic joint backprojection to obtain the target 3D indoor full-scene model includes: calling a pre-trained monocular depth model to perform pixel-level depth prediction on the initial 3D indoor full-scene model to obtain a relative depth prediction set; calling a pre-trained semantic segmentation model to perform object segmentation on the initial 3D indoor full-scene model to obtain an object mask set; using the object mask set to remove objects from the initial 3D indoor full-scene model to obtain a 3D indoor blank scene model; and aligning the absolute depth set corresponding to the 3D indoor blank scene model with the relative depth prediction set to obtain a metric depth set. The object point cloud is recovered from the metric depth set and the camera parameters of the virtual camera corresponding to the 3D indoor blank scene model. The object point cloud is then de-outcrystallized and eroded to obtain the minimum 3D bounding box set for fitting the object. The pose of the object in the 3D indoor blank scene model is matched using the object mask set and the asset pre-rendered multi-view map corresponding to the object mask set to obtain the target orientation. The minimum 3D bounding box set is scaled to obtain the orientation of the minimum 3D bounding box set. The orientation of the object in the initial 3D indoor full scene model is determined by the orientation of the minimum 3D bounding box set and the target orientation, thus obtaining the target 3D indoor full scene model.

[0013] According to embodiments of this application, the aforementioned multimodal requirements include textual requirement information and / or image requirement information; wherein, the structured room description information set includes room category information, wall material description information, floor material description information, and room customization requirement information.

[0014] The second aspect of this application provides an indoor scene construction device, comprising: a requirement parsing module for parsing multimodal requirements input by a user to obtain a set of structured room description information; an image filtering and segmentation module for filtering a set of candidate floor plan layouts generated based on the set of structured room description information to obtain a target floor plan layout, and segmenting the target floor plan layout into room regions to obtain a set of binary masks for room walls; a model rendering module for performing 3D modeling of the room using the set of binary masks for room walls and the structured room description information corresponding to the target floor plan layout to obtain a 3D empty room model, and rendering the 3D empty room model using a set of furniture rendering models obtained based on semantic retrieval to obtain a 3D room scene model with furniture placement effects; and a model backprojection optimization module for interactively iteratively editing the 3D room scene model based on multimodal annotation information obtained from multiple rounds of interaction with the user to obtain an initial 3D indoor full scene model, and performing optimization based on depth-semantic joint backprojection on the initial 3D indoor full scene model to obtain a target 3D indoor full scene model.

[0015] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0016] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0017] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0018] The indoor scene construction method provided in this application, by employing multimodal joint requirement analysis, planar layout screening and segmentation combined with 3D modeling and rendering, at least partially solves the technical problems in traditional indoor scene construction, such as ambiguous expression of user editing intent, difficulty in mapping 2D modifications to 3D space, low efficiency of global regeneration, insufficient realism of scene layout, and weak process control. This achieves the technical effects of improving the accuracy and operability of interactive editing, realizing a closed-loop mapping from 2D to 3D editing, reducing iteration costs, optimizing the visual realism of the scene, strengthening the system's scalability and robustness, and ensuring that the results are verifiable and reproducible. At the same time, it specifically addresses the pain points of existing technologies and improves the practicality of the solution for engineering implementation. Attached Figure Description

[0019] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 An application scenario diagram of the indoor scene construction method according to an embodiment of this application is shown.

[0021] Figure 2 A flowchart of an indoor scene construction method according to an embodiment of this application is shown.

[0022] Figure 3 A diagram illustrating multi-agent cooperative operation according to an embodiment of this application is shown.

[0023] Figure 4 An interactive editing interface according to an embodiment of this application is shown.

[0024] Figure 5 The actual generated effect diagram according to the embodiment of this application is shown.

[0025] Figure 6 A block diagram of an electronic device suitable for implementing an indoor scene construction method according to an embodiment of this application is shown. Detailed Implementation

[0026] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0030] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0031] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0032] To at least address one of the problems in the prior art, this application provides an indoor scene construction method that can stably correlate intuitive annotations by users at the image level with object positioning, orientation, and scale adjustments in a 3D scene, thereby improving the controllability and interactive efficiency of indoor scene generation.

[0033] Figure 1 An application scenario diagram of the indoor scene construction method according to an embodiment of this application is shown.

[0034] like Figure 1 As shown, application scenario 100 according to this embodiment may include application scenarios such as 3D simulation, artificial intelligence, multimodal generation, 3D scene reconstruction, and intelligent agent collaborative control. Network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0035] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0036] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0037] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0038] It should be noted that the indoor scene construction method provided in this application embodiment can generally be executed by server 105. Correspondingly, the indoor scene construction device provided in this application embodiment can generally be located in server 105. The indoor scene construction method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the indoor scene construction device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0040] The following will be based on Figure 1 The described scene, through Figures 2-5 The indoor scene construction method of the disclosed embodiments is described in detail.

[0041] Figure 2 A flowchart of an indoor scene construction method according to an embodiment of this application is shown.

[0042] like Figure 2 As shown, the indoor scene construction method of this embodiment includes operations S210 to S240.

[0043] In operation S210, the multimodal requirements input by the user are parsed to obtain a set of structured room description information.

[0044] According to embodiments of this application, the aforementioned multimodal requirements include textual requirement information and / or image requirement information; wherein, the structured room description information set includes room category information, wall material description information, floor material description information, and room customization requirement information.

[0045] Multimodal requirements of user input ,in, This is a text requirement used to describe the user's target apartment type, style, and furniture preferences. For optional image requirements, it is used to provide segmentation references or layout sketches to the system performing the indoor scene construction method.

[0046] The target system executing the indoor scene construction method in discrete time steps Maintain global state: ,in, For the current 3D scene, For the collection of information from each room, For contextual historical memory. Initial memory. Press after each plugin execution renew: ,in, This serves as the plugin identifier for this call. For the execution results or evaluation summary.

[0047] The target system maintenance plugin collection for executing indoor scene construction methods: ,in, This indicates a user needs analysis plugin. This indicates a planar layout generation plugin. This indicates a plugin for generating empty room models. This indicates a furniture selection plugin. This indicates a parsing quality check plugin. This refers to a room structure check plugin.

[0048] The executing agent invokes the user requirement parsing plugin to parse multimodal requirements into a structured set of room description information. ,in ,in, Indicates the number of rooms. Indicates the first Room category information for each room Indicates the first Description of the wall materials in each room. Indicates the first Description of the floor material in each room. Indicates user regarding the first Customized requirements for each room (i.e., the user's specific needs for that room).

[0049] In operation S220, the candidate floor plan layout map set generated based on the structured room description information set is filtered to obtain the target floor plan layout map, and the target floor plan layout map is segmented into room regions to obtain the room wall binary mask set.

[0050] The above operation S220 includes operations such as generating candidate floor plan layouts, filtering candidate floor plan layouts, and redrawing the target floor plan layout, ultimately obtaining a binary mask set representing the walls of each room.

[0051] Based on a structured room description information set, a candidate floor plan set is generated by an intelligent agent calling a floor plan generation plugin. The candidate floor plans are then scored, and the optimal result is selected based on the scoring results. For the optimal result The target plan layout is obtained through processing. Then, the binary mask set of the room walls is obtained through segmentation. .

[0052] In operation S230, the room is modeled in 3D using the binary mask set of the room walls and the structured room description information corresponding to the target plan layout, resulting in a 3D empty room model. The 3D empty room model is then rendered using the furniture rendering model set obtained based on semantic retrieval, resulting in a 3D room scene model with furniture placement effects.

[0053] Before rendering the 3D empty room model, the target system executing the interior scene construction method calls the evaluation agent to perform a quality check on the parsing of the structured room description information and to check the room structure in the 3D empty room model.

[0054] By executing the intelligent agent and calling the empty room model generation plugin based on the binary mask set of the room walls... Constructing 3D Room Geometry and combined Match wall and floor texture assets.

[0055] By executing the intelligent agent and calling the furniture selection plugin, a furniture description set is formed for each room. Candidate furniture was retrieved from the asset database. and render the composite image. .

[0056] Render the initial image of the empty room for each room. Absolute depth and camera parameters , and then and Generate furniture placement renderings .

[0057] In operation S240, based on the multimodal annotation information obtained from multiple rounds of interaction with the user, the 3D room scene model is interactively iteratively edited to obtain the initial 3D indoor full scene model. Then, the initial 3D indoor full scene model is optimized based on depth-semantic joint back projection to obtain the target 3D indoor full scene model.

[0058] Multimodal annotation information includes image annotations made by the user on the 3D room scene model, as well as modification instructions and / or reference images input by the user.

[0059] Interactive editing: The user can edit the interface shown in the target system when executing the indoor scene construction method. After labeling, we can obtain Then enter the modified text (modification instruction information). And optional reference diagrams The system performs iterative editing until the termination condition is met (user satisfaction or reaching the preset maximum number of iterations).

[0060] Backprojection placement: Perform depth-semantic joint backprojection on the final rendering, and obtain the following results sequentially. , , , , Orientation towards the target This completes the scaling, pose calculation, and scene placement of the 3D assets.

[0061] Output: The target system that executes the indoor scene construction method outputs a renderable 3D scene and an object parameter table (object ID, location, scale, orientation, room to which it belongs), and can save intermediate products for backtracking and reproduction.

[0062] The optimization based on depth-semantic joint backprojection used in the above operation S240 refers to the technology chain that combines monocular depth estimation, semantic segmentation mask and camera parameters to restore objects in two-dimensional images to three-dimensional space and complete localization and pose determination.

[0063] The above embodiments of this application can first obtain high-quality placement effects through an image generation model (a pre-trained model built into the target system that performs the indoor scene construction method, which can exist in the form of a plugin), and then map the results back to the three-dimensional scene assets.

[0064] In embodiments of this application, user consent or authorization can be obtained before acquiring user information. For example, a request to acquire user information can be sent to the user before operation S240. If the user consents or authorizes the acquisition of user information, operation S220 is performed.

[0065] In the embodiments of this application, a corresponding operation entry point can be provided to the user, allowing the user to choose to agree to or reject the automated decision result. That is, before processing / making a decision on the user information, the user can provide an instruction to agree to or reject the processing / decision through the corresponding operation entry point. If the user agrees to the processing / decision, the user information is processed / decided, i.e., step S240 is executed. If the user rejects the processing / decision, the expert decision-making process is initiated.

[0066] The indoor scene construction method provided in this application, by employing multimodal joint requirement analysis, planar layout screening and segmentation combined with 3D modeling and rendering, at least partially solves the technical problems in traditional indoor scene construction, such as ambiguous expression of user editing intent, difficulty in mapping 2D modifications to 3D space, low efficiency of global regeneration, insufficient realism of scene layout, and weak process control. This achieves the technical effects of improving the accuracy and operability of interactive editing, realizing a closed-loop mapping from 2D to 3D editing, reducing iteration costs, optimizing the visual realism of the scene, strengthening the system's scalability and robustness, and ensuring that the results are verifiable and reproducible. At the same time, it specifically addresses the pain points of existing technologies and improves the practicality of the solution for engineering implementation.

[0067] The target system for implementing the indoor scene construction method in the above embodiments of this application includes multiple agents and multiple pre-trained data processing models, wherein the agents include a planning agent, an execution agent, and an evaluation agent.

[0068] The following describes specific implementation methods in conjunction with appendices. Figure 3 The intelligent agents used in the above embodiments of this application will be described in further detail.

[0069] Figure 3 A diagram illustrating multi-agent cooperative operation according to an embodiment of this application is shown.

[0070] like Figure 3 As shown, the planning agent, execution agent, and evaluation agent work collaboratively under a unified state and memory mechanism, supporting scene generation, quality checks, and iterative editing. Specifically, the planning agent is responsible for selecting the execution or evaluation plugin at each time step and maintaining the global state and memory; the execution agent is responsible for calling specific tool plugins to complete requirement analysis, layout generation, room modeling, furniture retrieval, and rendering generation; and the evaluation agent is responsible for performing compliance and quality checks on intermediate results at key nodes and returning interpretable evaluation conclusions.

[0071] According to an embodiment of this application, the above-mentioned parsing of user-inputted multimodal requirements to obtain a structured room description information set includes: calling a planning agent to evaluate the user-inputted multimodal requirements in the current global state, obtaining the evaluated multimodal requirements, wherein the current global state includes the three-dimensional scene indoor full scene model, room information set, and contextual history information within the current discrete time step; determining the parsing plugin used by the execution agent based on the evaluated multimodal requirements, and obtaining the structured room description information set by having the execution agent call the parsing plugin to parse the evaluated multimodal requirements.

[0072] The embodiments described above rely on a layered architecture of planning agent + execution agent + plug-in parsing to transform fragmented multimodal user input requirements into standardized, structured data that can be used for subsequent modeling. At each independent interaction point, the target system executing the indoor scene construction method uniformly retrieves complete environmental data, including the generated 3D indoor model, basic room information, historical dialogues, and operation records. This serves as the global basis for understanding the requirements, ensuring that the interpretation of requirements is combined with the scene context and does not deviate from the existing room design content. This achieves accurate conversion of unstructured multimodal user requirements into standardized structured room data, solving the problems of semantic chaos, misunderstanding, and inability to directly connect to the backend modeling process in multimodal commands.

[0073] According to an embodiment of this application, the above-mentioned filtering of candidate floor plan layout images generated based on a structured room description information set to obtain a target floor plan layout image includes: generating a candidate floor plan layout image set by calling a floor plan layout generation plugin through an execution agent based on the structured room description information set; calling a planning agent to score the candidate floor plan layout image set to obtain a score set, and filtering the candidate floor plan layout image set based on the score set to obtain an initial floor plan layout image; and calling an execution agent to redraw the initial floor plan layout image to obtain the target floor plan layout image.

[0074] The process of obtaining the target floor plan layout diagram in this application will be further described in detail below through specific embodiments.

[0075] Generate a candidate floor plan atlas based on the structured room requirement information set. Each of the floor plans is scored, and the one with the highest score is selected as the optimal result (i.e., the initial floor plan). .

[0076] Then, an image is generated by redrawing the image, retaining only the room structure and door and window information, which is the target floor plan: .

[0077] According to an embodiment of this application, the above-mentioned method of using the room wall binary mask set and the structured room description information corresponding to the target floor plan to perform 3D room modeling to obtain a 3D empty room model includes: constructing an initial 3D empty room model representing multiple empty rooms in an indoor scene by executing an intelligent agent to call an empty room model generation plugin based on the room wall binary mask set; parsing the target structured description information corresponding to the target floor plan to obtain wall material description information and floor material description information; searching the material asset library based on the wall material description information and floor material description information to obtain a target wall asset set and a floor texture asset set; and rendering the initial 3D empty room model using the target wall asset set and floor texture asset set to obtain a 3D empty room model.

[0078] The process of obtaining a three-dimensional empty room model provided in the above embodiments of this application will be further described in detail below through specific implementation methods.

[0079] The obtained target floor plan is divided into room regions to obtain the binary mask of each room's walls: .

[0080] By executing the intelligent agent to call the empty room model generation plugin ,according to Construct the 3D geometry of each room ;according to Retrieve matching wall and floor texture assets from the material asset library. And assign it to the room.

[0081] According to an embodiment of this application, the above-mentioned rendering of a 3D empty room model using a furniture rendering model set obtained based on semantic retrieval to obtain a 3D room scene model with furniture placement effect includes: generating furniture description information for each 3D empty room sub-model in the 3D empty room model by executing an intelligent agent to call a furniture selection plugin, thereby obtaining a furniture description information set; performing semantic retrieval on a furniture asset library using the furniture description information set to obtain a furniture rendering model matching each 3D empty room sub-model, thereby obtaining a furniture rendering model set; and performing image rendering on the 3D empty room model using the furniture rendering model set to obtain a 3D room scene model with furniture placement effect.

[0082] The process of obtaining a three-dimensional room scene model according to the above embodiments of this application will be further described in detail below through specific implementation methods.

[0083] By executing the intelligent agent to invoke the furniture selection plugin Generate a set of furniture descriptions for each room. and in the furniture asset library Perform semantic retrieval to obtain the most matching asset: This process generates a list of pairs containing "text description + candidate assets" and renders a furniture assembly diagram (i.e., a furniture rendering model set). Used for generating subsequent renderings.

[0084] According to an embodiment of this application, the above-mentioned image rendering of a 3D empty room model using a furniture rendering model set to obtain a 3D room scene model with furniture placement effect includes: determining the camera parameters of the virtual camera in each 3D empty room sub-model and the camera position representing the spatial coordinates and orientation of the virtual camera, to obtain a camera parameter set and a camera position set; determining the absolute depth representing the real physical depth value of pixels in each 3D empty room sub-model according to the camera position set, to obtain an absolute depth set; performing initial rendering on each 3D empty room sub-model according to the camera position set, to obtain each initially rendered 3D empty room sub-model; and re-rendering each initially rendered 3D empty room sub-model using the furniture rendering model corresponding to each 3D empty room sub-model according to the camera parameter set and the absolute depth set, to obtain a 3D room scene model with furniture placement effect.

[0085] The image rendering process of the three-dimensional room scene model provided in the above embodiments of this application will be further described in detail below through specific implementation methods.

[0086] For each room First, create a three-dimensional empty room model. Select camera position and render initial image Simultaneously record absolute depth and camera parameters Combined with furniture arrangement diagrams Generate furniture placement renderings .

[0087] According to an embodiment of this application, the above-mentioned interactive iterative editing of the 3D room scene model based on the multimodal annotation information obtained from multiple rounds of interaction with the user to obtain the initial 3D indoor full scene model includes: calling an evaluation agent to perform multiple evaluations on the 3D room scene model, obtaining multiple evaluation results, and modifying the 3D room scene model based on the multiple evaluation results to obtain a 3D room scene model that has passed the evaluation; iteratively editing the 3D room scene model that has passed the evaluation based on the multi-round image annotation information, multi-round modification instruction information, and / or multi-round reference images provided by the user for the 3D room scene model that has passed the evaluation, until the preset editing conditions are met to obtain the initial 3D indoor full scene model.

[0088] The following describes specific implementation methods in conjunction with appendices. Figure 4 The interactive editing process provided in the above embodiments of this application will be described in further detail.

[0089] Figure 4 An interactive editing interface according to an embodiment of this application is shown.

[0090] like Figure 4 As shown, the interactive editing interface includes a rendering area 1, annotation tools 2, a text input box 3, and a submission entry 4, which are used to enable multimodal interaction with users.

[0091] During the interactive editing phase, the user... Image annotation was performed to obtain And enter the modification command. And optional reference diagrams The target system executing the indoor scene construction method repeatedly calls the effect diagram generation process for iteration until the user is satisfied (or reaches the preset maximum number of iterations, which can be set according to actual needs).

[0092] According to embodiments of this application, the above-mentioned invocation of the evaluation agent to perform multiple evaluations on the 3D room scene model, obtaining multiple evaluation results, includes: invoking the planning agent to determine the parsing quality check plugin and the room structure check plugin used by the evaluation agent; invoking the parsing quality check plugin to perform data compliance checks on the structured room description information corresponding to the 3D room scene model, obtaining data compliance check results, wherein the data compliance check results include field integrity check results, data category legality check results, and description information quality check results; and invoking the room structure check plugin to perform spatial regularity checks on each 3D sub-room scene model in the 3D room scene model, obtaining spatial regularity check results, wherein the spatial regularity check results include room integrity check results, area overlap check results, and room geometric cleanliness check results.

[0093] The evaluation process provided in the above embodiments of this application will be further described in detail below through specific implementation methods.

[0094] By evaluating the agent call parsing quality check plugin ,right The system checks the completeness of fields, the validity of categories, and the quality of descriptions, and outputs pass / fail indicators and problem text.

[0095] The room structure inspection plugin was invoked by evaluating the agent. The floor plan and empty room model are checked, including room integrity, area overlap, and room geometric cleanliness (only the main structure is included).

[0096] According to an embodiment of this application, the above-mentioned optimization of the initial 3D indoor full-scene model based on depth-semantic joint backprojection to obtain the target 3D indoor full-scene model includes: calling a pre-trained monocular depth model to perform pixel-level depth prediction on the initial 3D indoor full-scene model to obtain a relative depth prediction set; calling a pre-trained semantic segmentation model to perform object segmentation on the initial 3D indoor full-scene model to obtain an object mask set; using the object mask set to remove objects from the initial 3D indoor full-scene model to obtain a 3D indoor blank scene model; and aligning the absolute depth set corresponding to the 3D indoor blank scene model with the relative depth prediction set to obtain a metric depth set. The object point cloud is recovered from the metric depth set and the camera parameters of the virtual camera corresponding to the 3D indoor blank scene model. The object point cloud is then de-outcrystallized and eroded to obtain the minimum 3D bounding box set for fitting the object. The pose of the object in the 3D indoor blank scene model is matched using the object mask set and the asset pre-rendered multi-view map corresponding to the object mask set to obtain the target orientation. The minimum 3D bounding box set is scaled to obtain the orientation of the minimum 3D bounding box set. The orientation of the object in the initial 3D indoor full scene model is determined by the orientation of the minimum 3D bounding box set and the target orientation, thus obtaining the target 3D indoor full scene model.

[0097] The optimization process based on depth-semantic joint backprojection provided in the above embodiments of this application will be further described in detail below through specific implementation methods.

[0098] Predicted depth is obtained from a pre-trained monocular depth model. The object mask set is obtained from the pre-trained semantic segmentation model. ; Using the depth information set after removing the object region With absolute depth set Alignment is performed to obtain the metric depth set. Combined with object mask set With the camera parameters of the virtual camera, from Restore the point cloud of the object Outlier removal and erosion denoising of point clouds of objects, and fitting of the minimum 3D bounding box set. As a basis for positioning; the object mask set Pose matching is performed between the pre-rendered multi-view map of the asset and the input multimodal model to select the optimal orientation. ;according to Perform scaling and combine direction and Calculate the final object orientation to achieve precise placement of the 3D asset.

[0099] Figure 5 The actual generated effect diagram according to the embodiment of this application is shown.

[0100] The following describes specific implementation methods in conjunction with appendices. Figure 5 The method for constructing indoor scenes provided in this application is described in further detail.

[0101] The indoor scene construction method provided in this application can be used when the user only inputs text requirements, i.e., the input does not provide... Only text requirements are provided. It can still execute according to the process of "requirements analysis - layout generation - room modeling - furniture selection - rendering generation - back projection placement". This application can run without relying on image input, which is only used as an enhancement constraint; under the condition of only text input, a structurally complete 3D interior scene can still be obtained.

[0102] The target system for implementing the indoor scene construction method in this application can integrate multiple data processing models, such as image generation models (e.g., Figure 5 As shown, the indoor scene realism that can be achieved using image generation models, depth estimation models, semantic segmentation models, and vector coding models are merely illustrative and not bound to any specific model. The purpose of this application is to realize "interactive annotation guidance + deep semantic back projection localization chain + multi-agent plug-in closed-loop process": (1) obtain the effect image generation operation. (2) Obtained through depth estimation and segmentation operations and (3) Recovery through back-projection optimization operation and fitting (4) Determined through pose matching operation And complete the placement.

[0103] The indoor scene construction method provided in this application can realize the batch generation and consistent editing of multi-room scenes: the input is a multi-room requirement such as "two bedrooms and one living room" or "three bedrooms and one living room". The target system executing the indoor scene construction method first generates a set of rooms. Subsequently, operations S220 to S240 are performed in parallel or serially for each room; the state is maintained at the global level by the planning agent. It also coordinates cross-room editing consistency. When a user performs a style redraw in the living room or moves furniture in the bedroom, the target system executing the interior scene construction method performs local iterative editing in the corresponding room, and checks room style consistency and object conflict relationships in the global phase, ultimately outputting a complete multi-room scene. The method provided in this application can be extended to complex multi-room scenes, possessing cross-room collaborative editing and unified scheduling capabilities.

[0104] It should be noted that the parameters and thresholds used in this application (such as scoring threshold, iteration rounds, and overlap determination threshold) can be preset according to task requirements; the asset library can use three-dimensional furniture models of different sources and sizes.

[0105] The indoor scene construction method provided in the above embodiments and specific implementations of this application has the following advantages:

[0106] (1) Improve the accuracy and operability of editing intent expression: The combined input method of "image annotation (closed curve / arrow / color block) + text command + optional reference image" is adopted to express the spatial relationship, target area and local modification needs that are difficult for users to describe with pure text. Compared with the text-only interaction method, it can reduce semantic ambiguity and make the system understand "where to change, how to change, and to what extent to change" more directly, thereby improving the success rate of interactive editing and the first-time modification hit rate.

[0107] (2) Achieve closed-loop mapping from 2D modification intention to 3D asset placement: Through the technical chain of “generating effect image → depth-semantic joint back projection → 3D asset positioning and orientation”, the editing results in the 2D image are converted into the position, scale and orientation parameters of the object in the 3D scene, which solves the problem of “the image layer is well modified but the 3D layer cannot be stably reproduced” in the existing solution, and significantly enhances the executability and consistency of the editing results in the 3D space.

[0108] (3) Reduce the cost of repeated regeneration and improve iteration efficiency: This application supports local iterative editing on existing results, rather than regenerating the entire scene each time. Its technical effect is that: when the same modification goal is met, unnecessary global recalculation can be reduced, the interaction path from "discovering the problem" to "getting a usable result" can be shortened, the efficiency of multi-round human-computer collaboration can be improved, and the threshold for use can be lowered.

[0109] (4) Improve the naturalness and visual realism of scene layout: This application adopts the construction path of "rendering the rendering first and then backfilling the 3D", giving priority to the prior ability of the image generation model in terms of indoor layout, object co-occurrence relationship and visual details, and then restoring the 3D placement parameters through back projection. Compared with the placement strategy that relies solely on the language model to directly output the coordinates, this method can obtain more stable results in terms of layout rationality and visual realism.

[0110] (5) Improve the scalability and robustness of the system engineering process: Through the division of labor and collaboration among planning agents, execution agents, and evaluation agents, as well as the encapsulation of plug-in capabilities and the phased inspection mechanism, this application can achieve traceable and reversible process control for "requirements analysis - layout generation - room modeling - furniture selection - interactive editing - back projection placement". This architecture facilitates the subsequent replacement of the basic model or the expansion of new plug-ins, and helps to maintain stable generation quality under complex input conditions.

[0111] (6) Enhanced verifiability and reproducibility of results: This application explicitly saves intermediate states (such as room structure masks, renderings, depth maps, object masks, point clouds, and 3D bounding boxes) during the process, which can be used to check the output quality and error sources at each stage step by step. Compared with black-box one-time generation, this mechanism makes it easier for technicians to reproduce experiments, locate problems, and perform parameter tuning.

[0112] (7) Corresponding solutions to the background technical problems: In response to the problems in the background technology such as "inconvenient editing, inaccurate intent, inability to make low-cost secondary modifications, and lack of two-dimensional to three-dimensional mapping", this application provides corresponding solutions through interactive annotation input, depth-semantic joint back projection chain and multi-agent plug-in closed-loop mechanism, which have clear technical relevance and engineering implementation value.

[0113] The second aspect of this application provides an indoor scene construction device, comprising: a requirement parsing module for parsing multimodal requirements input by a user to obtain a set of structured room description information; an image filtering and segmentation module for filtering a set of candidate floor plan layouts generated based on the set of structured room description information to obtain a target floor plan layout, and segmenting the target floor plan layout into room regions to obtain a set of binary masks for room walls; a model rendering module for performing 3D modeling of the room using the set of binary masks for room walls and the structured room description information corresponding to the target floor plan layout to obtain a 3D empty room model, and rendering the 3D empty room model using a set of furniture rendering models obtained based on semantic retrieval to obtain a 3D room scene model with furniture placement effects; and a model backprojection optimization module for interactively iteratively editing the 3D room scene model based on multimodal annotation information obtained from multiple rounds of interaction with the user to obtain an initial 3D indoor full scene model, and performing optimization based on depth-semantic joint backprojection on the initial 3D indoor full scene model to obtain a target 3D indoor full scene model.

[0114] It should be noted that the indoor scene construction device part in the embodiments of this application corresponds to the indoor scene construction method part in the embodiments of this application. The description of the indoor scene construction device part is specifically referred to in the indoor scene construction method part, and will not be repeated here.

[0115] Figure 6 A block diagram of an electronic device suitable for implementing an indoor scene construction method according to an embodiment of this application is shown.

[0116] like Figure 6As shown, an electronic device 600 according to an embodiment of this application includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0117] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0118] According to embodiments of this application, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0119] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0120] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0121] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.

[0122] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0123] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0124] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0125] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0127] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0128] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A method for constructing an indoor scene, characterized in that, include: The multimodal requirements input by the user are parsed to obtain a set of structured room description information; The candidate floor plan layout map generated based on the structured room description information set is filtered to obtain the target floor plan layout map, and the target floor plan layout map is segmented into room regions to obtain the room wall binary mask set. The room is modeled in 3D using the binary mask set of the room walls and the structured room description information corresponding to the target plan layout, resulting in a 3D empty room model. The 3D empty room model is then rendered using the furniture rendering model set obtained based on semantic retrieval, resulting in a 3D room scene model with furniture placement effects. Based on the multimodal annotation information obtained from multiple rounds of interaction with the user, the three-dimensional room scene model is interactively iteratively edited to obtain an initial three-dimensional indoor full scene model. Then, the initial three-dimensional indoor full scene model is optimized based on depth-semantic joint back projection to obtain the target three-dimensional indoor full scene model.

2. The method according to claim 1, characterized in that, The multimodal requirements input by the user are parsed to obtain a structured room description information set, including: The planning agent is invoked to evaluate the multimodal requirements input by the user in the current global state, and the evaluated multimodal requirements are obtained. The current global state includes the three-dimensional indoor full-scene model, room information set and context history information in the current discrete time step. Based on the evaluated multimodal requirements, the parsing plugin used by the executing agent is determined. The executing agent calls the parsing plugin to parse the evaluated multimodal requirements, thereby obtaining the structured room description information set.

3. The method according to claim 1, characterized in that, The candidate floor plan layouts generated based on the structured room description information set are filtered to obtain the target floor plan layouts, including: Based on the structured room description information set, the candidate floor plan layout atlas is generated by executing the intelligent agent to call the floor plan layout generation plugin; The planning agent is invoked to score the candidate planar layout map set to obtain a score set, and the candidate planar layout map set is filtered based on the score set to obtain an initial planar layout map. The execution agent is invoked to redraw the initial planar layout to obtain the target planar layout.

4. The method according to claim 1, characterized in that, Using the binary mask set of the room walls and the structured room description information corresponding to the target floor plan, a 3D model of the room is obtained, including: Based on the binary mask set of the room walls, an initial three-dimensional empty room model representing multiple empty rooms in the indoor scene is constructed by executing the intelligent agent to call the empty room model generation plugin. The target structured description information corresponding to the target floor plan is parsed to obtain wall material description information and floor material description information; The material asset library is searched based on the wall material description information and the floor material description information to obtain the target wall asset set and floor texture asset set; The initial 3D empty room model is rendered using the target wall asset set and the ground texture asset set to obtain the 3D empty room model.

5. The method according to claim 1, characterized in that, The 3D empty room model is rendered using a furniture rendering model set obtained based on semantic retrieval, resulting in a 3D room scene model with furniture placement effects, including: By executing the intelligent agent to call the furniture selection plugin, furniture description information is generated for each 3D empty room sub-model in the 3D empty room model, thus obtaining a furniture description information set; Using the furniture description information set, semantic retrieval is performed on the furniture asset library to obtain a furniture rendering model that matches each of the three-dimensional empty room sub-models, thus obtaining a furniture rendering model set. The furniture rendering model set is used to render the three-dimensional empty room model to obtain the three-dimensional room scene model with furniture placement effect.

6. The method according to claim 5, characterized in that, The three-dimensional empty room model is rendered using the furniture rendering model set to obtain the three-dimensional room scene model with furniture placement effects, including: Determine the camera parameters of the virtual camera in each of the three-dimensional empty room sub-models and the camera position representing the spatial coordinates and orientation of the virtual camera to obtain the camera parameter set and camera position set; Based on the camera bit set, determine the absolute depth of each of the three-dimensional empty room sub-models that represents the true physical depth value of a pixel, and obtain the absolute depth set; Based on the camera bit set, perform initial rendering on each of the three-dimensional empty room sub-models to obtain each initially rendered three-dimensional empty room sub-model. Based on the camera parameter set and the absolute depth set, each of the initially rendered 3D empty room sub-models is re-rendered using the furniture rendering model corresponding to each of the 3D empty room sub-models, to obtain the 3D room scene model with furniture placement effect.

7. The method according to claim 1, characterized in that, Based on the multimodal annotation information obtained from multiple rounds of interaction with the user, the 3D room scene model is interactively and iteratively edited to obtain an initial 3D indoor full scene model, including: The evaluation agent is invoked to perform multiple evaluations on the 3D room scene model, and the multiple evaluation results are obtained. Based on the multiple evaluation results, the 3D room scene model is modified to obtain a 3D room scene model that passes the evaluation. Based on the multi-round image annotation information, multi-round modification instruction information and / or multi-round reference images provided by the user for the evaluated 3D room scene model, the evaluated 3D room scene model is iteratively edited until the preset editing conditions are met, and the initial 3D indoor full scene model is obtained.

8. The method according to claim 7, characterized in that, The evaluation agent is invoked to perform multiple evaluations on the 3D room scene model, resulting in multiple evaluation results, including: The planning agent is invoked to determine the parsing quality check plugin and room structure check plugin used by the evaluation agent; The parsing quality check plugin is invoked to perform a data compliance check on the structured room description information corresponding to the 3D room scene model, and the data compliance check results are obtained. The data compliance check results include field integrity check results, data category legality check results, and description information quality check results. The room structure inspection plugin is invoked to perform a spatial regularity check on each 3D sub-room scene model in the 3D room scene model, and the spatial regularity check results are obtained. The spatial regularity check results include room integrity check results, area overlap check results, and room geometric cleanliness check results.

9. The method according to claim 1, characterized in that, Optimization based on depth-semantic joint backprojection is performed on the initial 3D indoor full-scene model to obtain the target 3D indoor full-scene model, which includes: A pre-trained monocular depth model is invoked to perform pixel-level depth prediction on the initial 3D indoor full-scene model to obtain a relative depth prediction set. A pre-trained semantic segmentation model is then invoked to perform object segmentation on the initial 3D indoor full-scene model to obtain an object mask set. The objects in the initial 3D indoor full scene model are removed using the object mask set to obtain a 3D indoor blank scene model. The absolute depth set corresponding to the 3D indoor blank scene model is then aligned with the relative depth prediction set to obtain a metric depth set. Using the metric depth set and the camera parameters of the virtual camera corresponding to the three-dimensional indoor blank scene model, the object point cloud is recovered from the metric depth set, and the object point cloud is de-outcrystallized and eroded to obtain the minimum three-dimensional bounding box set for fitting the object. The pose of objects in the 3D indoor blank scene model is matched using the object mask set and the pre-rendered multi-view map of the assets corresponding to the object mask set to obtain the target orientation. The minimum 3D bounding box set is scaled to obtain its orientation, and the orientation of objects in the initial 3D indoor full-scene model is determined using the orientation of the minimum 3D bounding box set and the target orientation, thus obtaining the target 3D indoor full-scene model.

10. The method according to any one of claims 1 to 9, characterized in that, The multimodal requirements include textual requirement information and / or image requirement information; The structured room description information set includes room category information, wall material description information, floor material description information, and room customization requirements information.

Images