Method, device and equipment for generating three-dimensional scene based on spatial reasoning, and medium

Abstract

The disclosure provides a three-dimensional scene generation method and device based on spatial reasoning, equipment and a storage medium. The method comprises: generating a two-dimensional house structure image based on a user instruction; generating a two-dimensional scene image comprising a furnishing object based on the house structure image; performing spatial understanding based on the scene image to obtain a two-dimensional bounding box and a position height range of the object; obtaining three-dimensional layout information of a three-dimensional model comprising the object based on the two-dimensional bounding box and the position height range of the object; correcting a region corresponding to the object based on the three-dimensional model of the object in the scene image and the three-dimensional layout information; rendering to obtain a scene visualization image based on the three-dimensional layout information; and iteratively modifying parameters of the three-dimensional layout information based on the scene visualization image. After exiting the iterative loop according to the modification result, a three-dimensional layout scene corresponding to the scene image is obtained.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the field of artificial intelligence technologies such as large models, spatial intelligence, and spatial understanding. Background Technology

[0002] As spatial intelligence gradually becomes a focus of artificial intelligence research, the application scenarios for the automated synthesis of high-quality 3D scenes are increasing. For example, it can be used in immersive augmented reality or virtual reality, robot simulation, and the construction of training data for spatial understanding and generative tasks. 3D scene generation technology can use probabilistic models to estimate the distribution of layout parameters for indoor scenes; however, limitations in the scale and quality of training data may lead to homogenized or overly simplified 3D scenes generated by the model. Scene synthesis schemes based on Large Language Models (LLMs), however, are limited by the output modalities of LLMs and still have shortcomings in terms of spatial rationality, realism, and diversity. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, and storage medium for generating three-dimensional scenes based on spatial reasoning, in order to solve or alleviate one or more technical problems in the prior art.

[0004] In a first aspect, this disclosure provides a method for generating 3D scenes based on spatial reasoning, including: Generate a two-dimensional apartment layout image based on user instructions; A two-dimensional scene image including furnishings is generated based on the apartment layout image; Based on the scene image, spatial understanding is performed to obtain the two-dimensional bounding box and position height range of the object; Based on the object's two-dimensional bounding box and position height range, three-dimensional layout information including the object's three-dimensional model is obtained; The region corresponding to the object is corrected based on the 3D model of the object in the scene image and the 3D layout information; A scene visualization image is rendered based on the aforementioned 3D layout information; Based on the scene visualization image, the parameters of the 3D layout information are iteratively corrected. After exiting the iteration loop according to the correction result, the 3D layout scene corresponding to the scene image is obtained.

[0005] Secondly, this disclosure provides a three-dimensional scene generation device based on spatial reasoning, comprising: The structural diagram generation module generates two-dimensional floor plan images based on user commands. The scene image generation module is used to generate a two-dimensional scene image including furnishing objects based on the apartment layout image; The spatial understanding module is used to perform spatial understanding based on the scene image to obtain the two-dimensional bounding box and position height range of the object; The layout acquisition module is used to obtain three-dimensional layout information including the three-dimensional model of the object based on the two-dimensional bounding box and the position and height range of the object. A 3D correction module is used to correct the region corresponding to the object based on the 3D model of the object in the scene image and the 3D layout information; The rendering module is used to render a scene visualization image based on the three-dimensional layout information; The correction module is used to iteratively correct the parameters of the 3D layout information based on the scene visualization image, and after exiting the iteration loop according to the correction result, obtain the 3D layout scene corresponding to the scene image.

[0006] Thirdly, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0007] Fourthly, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of the present disclosure.

[0008] Fifthly, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of the present disclosure.

[0009] The beneficial effects of the technical solution provided in this disclosure include at least the following: after generating a two-dimensional scene image, three-dimensional layout information is obtained based on spatial understanding, and further correction and modification are performed through spatial reasoning, which can improve the spatial rationality and realism of the final three-dimensional layout scene. Furthermore, using an intelligent agent system built with a multimodal understanding and generation model to execute the method of the embodiments of this disclosure can generate richer three-dimensional layout scenes with more modalities, improving scene diversity. It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments provided according to this disclosure and should not be construed as limiting the scope of this disclosure.

[0011] Figure 1 This is a flowchart illustrating a three-dimensional scene generation method based on spatial reasoning according to an embodiment of the present disclosure; Figure 2 This is a flowchart illustrating a three-dimensional scene generation method based on spatial reasoning according to an embodiment of the present disclosure; Figure 3 This is a flowchart illustrating a three-dimensional scene generation method based on spatial reasoning according to an embodiment of the present disclosure; Figure 4 This is a flowchart illustrating a three-dimensional scene generation method based on spatial reasoning according to an embodiment of the present disclosure; Figure 5 This is a schematic diagram of a smart agent framework based on a multimodal large model according to an embodiment of the present disclosure; Figure 6 This is a schematic diagram of the structure of a three-dimensional scene generation device based on spatial reasoning according to an embodiment of the present disclosure; Figure 7 This is a schematic diagram of the structure of a three-dimensional scene generation device based on spatial reasoning according to an embodiment of the present disclosure; Figure 8 This is a block diagram of an electronic device used to implement embodiments of the present disclosure. Detailed Implementation

[0012] The present disclosure will now be described in further detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0013] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0014] Figure 1 This is a flowchart illustrating a three-dimensional scene generation method based on spatial reasoning according to an embodiment of the present disclosure. In one embodiment, the method may include: S101. Generate a two-dimensional apartment layout image based on user instructions; S102. Generate a two-dimensional scene image including furnishing objects based on the apartment layout image; S103. Based on the scene image, perform spatial understanding to obtain the two-dimensional bounding box and position height range of the object; S104. Based on the two-dimensional bounding box and position height range of the object, obtain three-dimensional layout information including the three-dimensional model of the object; S105. Based on the scene image and the three-dimensional model of the object in the three-dimensional layout information, the region corresponding to the object is corrected; S106. Render a scene visualization image based on the three-dimensional layout information; S107. Based on the scene visualization image, the parameters of the three-dimensional layout information are iteratively corrected. After exiting the iteration loop according to the correction result, the three-dimensional layout scene corresponding to the scene image is obtained.

[0015] In this embodiment, the 3D scene generation method may include a 2D design stage (S101 to S102) and a 3D generation stage (S103 to S107). In the 2D design stage, the input user instructions may include multimodal design instructions (which may include text descriptions and reference images, etc.). Parsing the user's multimodal design instructions can generate a 2D floor plan image that meets the instruction requirements. Based on this 2D floor plan image, a multimodal understanding and generation model can be used to generate a 2D scene image including furnished objects, such as a top-view scene image including furniture. The multimodal understanding and generation model may employ a large language model base, a diffusion model base, or a hybrid base, and can be trained using floor plan datasets, home decoration floor plan datasets, and software-rendered top-view 2D floor plans as training data. The multimodal understanding and generation model may include the functionality of a multimodal understanding model and / or a multimodal generation model. For example, multimodal understanding models can be used to perform feature fusion and semantic parsing on multimodal input information such as images and text, extracting key information such as wall structure, door and window positions, furniture categories, and spatial relationships in interior scenes, and completing cross-modal information alignment and structured semantic output. As another example, multimodal generation models take one or more of the following as input: text instructions, structured scene parameters, and apartment layout constraints, and integrate semantic logic and visual rendering capabilities to automatically generate two-dimensional floor plans, interior scene images with furniture layouts, or three-dimensional scene representations.

[0016] This disclosure does not limit the specific model structure and training data types used in the multimodal understanding and generation model, as long as the corresponding functions can be achieved. In the 3D generation stage, spatial understanding can be performed on the scene image: First, object detection can be performed on the scene image to detect the 2D bounding boxes of the objects, and corresponding semantic labels and appearance descriptions with texture descriptions can be generated for each 2D bounding box. Then, the spatial relationships of each object are recorded, such as the support and connection relationships between objects, and the relationship between objects and building structures, such as the walls they lean against, hanging objects, or wall hangings, resulting in a scene map with the 2D bounding boxes of the objects and their spatial relationships. The angle and / or orientation of each object are estimated based on the scene map and the 2D scene image. The height range of each object is estimated based on the scene map and common sense priors. Next, dimensionality upscaling is performed. Based on camera parameters and the 2D bounding box of each object, projection is performed in the metric world space, and the projection is corrected according to the camera height and the height range of the objects to obtain the true 2D bounding box. Finally, the true size information of the object is determined based on the true 2D bounding box and the object's angle and / or orientation, resulting in the basic 3D bounding box of each object. Next, the 2D scene image is segmented based on the basic 3D bounding box of each object, resulting in segmented image regions for each object's basic 3D bounding box. Based on the segmented image regions, semantic labels, and appearance descriptions, the top view is converted into a three-quarter view. Then, a textured 3D model, also known as a 3D asset or 3D mesh, is generated from the three-quarter view using a 3D asset generation model. Then, spatial reasoning is performed on the preliminary 3D layout information. Spatial reasoning can include instance-level reasoning and layout-level reasoning. In instance-level reasoning, the regions corresponding to objects in the scene image are compared with the corresponding regions in the object's 3D model, thereby correcting the object instance yaw angle and size. In layout-level reasoning, the scene visualization image rendered from the 3D layout information is evaluated, and the 3D layout information is corrected based on the evaluation results. After one or more iterations, the final 3D layout scene is obtained.

[0017] In this embodiment, an intelligent agent system can be used to execute a 3D scene generation method based on spatial reasoning. For example, a top-level intelligent agent orchestrates specific tasks and corresponding underlying tools for agents at different stages. For instance, the intelligent agent in the 2D design stage and the intelligent agent in the 3D generation stage further include several sub-intelligent agents, each responsible for a specific sub-task and capable of calling its corresponding underlying tools or models. The intelligent agents (including sub-intelligent agents) can be built based on multimodal understanding and generative models, and the tools that can be called include fast renderers, geometric calculation tools, etc. The intelligent agent system can be fully autonomously orchestrated and executed end-to-end, or it can support interactive user intervention.

[0018] According to embodiments of this disclosure, after generating a two-dimensional scene image, three-dimensional layout information is obtained based on spatial understanding, and further corrected and refined through spatial reasoning, which can improve the spatial rationality and realism of the final three-dimensional layout scene. Furthermore, using an intelligent agent system built with a multimodal understanding and generation model to execute the methods of embodiments of this disclosure can generate richer three-dimensional layout scenes with more modalities, improving scene diversity.

[0019] Figure 2 This is a flowchart illustrating a spatial reasoning-based 3D scene generation method according to an embodiment of the present disclosure. This method can be used to implement one or more features of the aforementioned spatial reasoning-based 3D scene generation method. In one embodiment, generating a 2D apartment layout image based on user instructions includes: S201. Enable the user-defined model design agent to generate scene description information and apartment structure information that conform to the user's instructions; S202. Call the rendering tool to render a top-down view of the apartment structure based on the apartment structure information.

[0020] In this embodiment, the intelligent agent in the two-dimensional design stage may include the following sub-intelligent agents, such as: a floor plan design intelligent agent, an image generation intelligent agent, and a design reasoning intelligent agent. Driven by user instructions, the floor plan design intelligent agent can first generate a set of detailed scene description information conforming to the user instructions, as well as floor plan structure information in JSON format. The scene description information may include specific descriptions such as what furniture to place or what design to make, as specified in the user instructions. In interactive mode, the intelligent agent can guide the user to gradually supplement, refine, and adjust the floor plan structure information through dialogue interaction. In automatic mode, the intelligent agent can autonomously complete the floor plan structure information. Subsequently, the intelligent agent can call a rendering tool to render a top-down view of the floor plan structure image, align the image space with the metric world space, and record the corresponding camera parameters. The image space, also known as the two-dimensional pixel space, is measured in pixels and describes the row and column position of each point in the image. The metric world space may include a three-dimensional world coordinate system with real physical units of measurement (meters, centimeters), and may include the real dimensions of objects, walls, doors, windows, etc., which can be understood as the truth space of the three-dimensional scene. Aligning the image space with the metric world space allows for the solution and storage of camera intrinsic and extrinsic parameters, constructing a mapping from 2D pixel coordinates to 3D world coordinates with physical measurements. This enables the conversion of image observation information into a 3D spatial representation with real physical dimensions. Using floor plan design agents, floor plan structures can be generated and flexibly adjusted. These floor plan images do not yet include furnishings such as furniture and appliances, and can therefore be considered as empty floor plans.

[0021] In one implementation, such as Figure 2As shown, generating a two-dimensional scene image including furnished objects based on the apartment layout image includes: S203. Using an image generation agent, based on the apartment structure image, the scene description information, and camera parameters, a multimodal generation model is invoked to synthesize a two-dimensional scene image with furnishing objects.

[0022] In this embodiment, an image generation agent uses the aforementioned vacant apartment layout image, scene description information, and camera parameters to invoke a multimodal generation model to synthesize a two-dimensional scene image with a top-down view of furnishing objects, such as furniture. The multimodal generation model can extract geometric features such as walls, doors, windows, room outlines, and room boundaries from the apartment layout image to obtain spatial geometric constraint features; it can extract semantic features such as the category, size, placement constraints, and style requirements of objects like furniture from the scene description information; and it can parse camera parameters to obtain camera constraints such as the framing area, zoom level, and viewport. Then, based on the spatial geometric constraint features and object semantic features, it infers the legal layout of objects within the apartment room. Finally, it fuses the spatial geometric constraint features, object semantic features, and camera constraints using multimodal features to generate a two-dimensional scene image with furnishing objects within a framing range limited by camera parameters.

[0023] Figure 3 This is a flowchart illustrating a spatial reasoning-based 3D scene generation method according to an embodiment of the present disclosure. This method can be used to implement one or more features of the aforementioned spatial reasoning-based 3D scene generation method. In one embodiment, the method may further include: S301. Use a design reasoning agent to repeatedly verify whether the house structure image and the scene image are aligned; S302. If the apartment layout image is aligned with the scene image, the verification ends.

[0024] In this embodiment, the design reasoning agent can be built based on a visual understanding model. In the 2D design phase, it acts as a verifier, forming a closed loop of reasoning and action framework with the two generative agents (the floor plan design agent and the image generation agent) in this phase. If the design reasoning agent repeatedly verifies that the floor plan image and the scene image are not aligned, it can regenerate the floor plan image through the floor plan design agent or regenerate the scene image through the image generation agent, and then verify again, until the floor plan image and the scene image are aligned or the required number of verifications is reached. The visual understanding model belongs to a multimodal understanding model. In one example, the visual understanding model performs instance parsing on both a 2D floor plan image and a 2D scene image containing objects, identifying building components such as walls, doors, and windows, as well as furniture instances. Object instances in the 2D scene image are masked to eliminate interference from object graphics, extracting building component information from the scene image. Combining pre-stored camera parameters, the pixel coordinates of the floor plan image are transformed to a metric world space, resulting in a set of building components with real physical scale (e.g., walls, doors, windows, room outlines, etc.). Similarly, the pixel coordinates of the scene image are transformed to the same metric world space, resulting in a set of all elements with real physical scale (e.g., walls, doors, windows, room outlines, furniture, interior decorations, etc.). Using walls, doors, windows, and room outlines as comparison benchmarks, instance matching of the two sets of building components in different images is performed. Consistency checks are conducted across multiple dimensions, including component geometry, spatial location, and room topology, outputting information such as whether the floor plans of the two images are aligned and any deviations in abnormal components. This visual understanding model can be built upon architectures such as object detection and / or segmentation models.

[0025] In one implementation, cyclically verifying whether the floor plan image and the scene image are aligned includes cyclically executing one or more of the following steps until the floor plan image and the scene image are aligned and the loop exits: The image filter is called to compare the floor plan image and the scene image. If the floor plan of the floor plan image and the scene image are inconsistent, the scene image is regenerated and then compared again. The floor plan repair tool is invoked to compare the floor plan image and the scene image. If the floor plan images and the scene image are inconsistent, the floor plan information is adjusted incrementally, and the floor plan image is re-rendered and then compared again.

[0026] In this embodiment, the design inference agent can invoke an image filter to compare the floor plan image with the scene image. If the floor plan structures of the two images are inconsistent, the scene image is regenerated and the comparison continues until a set of basically consistent images is generated, at which point the verification direction is switched. The design inference agent can also invoke a floor plan repairer to compare the scene image with the floor plan image. If the floor plan structures of the two images are inconsistent, the floor plan information is incrementally adjusted and the floor plan image is re-rendered, and then the comparison is repeated.

[0027] The two comparison methods described above can be performed individually or both. For example, the image filter can be called first to perform the comparison step. If the floor plan structures of the two images are inconsistent, the scene image can be regenerated. Then, the floor plan repairer can be called to perform the comparison step. If they are still inconsistent, the floor plan structure image can be regenerated. This process is repeated until they are consistent or the required number of iterations is reached. Alternatively, the floor plan repairer can be called first, followed by the image filter. When the comparison results of the images are aligned, the generation-verification loop can be exited, and the next stage can begin. By designing the inference agent to align the floor plan structures in the floor plan structure image and the scene image, the consistency of the physical scale of the 2D images can be improved, thereby enhancing the realism and accuracy of the subsequently generated 3D scene.

[0028] In one implementation, such as Figure 2 As shown, spatial understanding is performed based on the scene image to obtain the two-dimensional bounding box and position height range of the object, including: S204. Using the scene parsing agent to call the multimodal understanding model, perform spatial understanding based on the scene image to obtain the two-dimensional bounding box, semantic label and appearance description of each object in the scene image; S205. Record the spatial relationships of the objects to obtain a spatial relationship diagram; S206. Based on the spatial relationship diagram and the scene image, predict the orientation angle of the object; S207. Based on the spatial relationship diagram, predict the position and height range of the object.

[0029] In this embodiment of the disclosure, during the 3D generation stage, a structured 3D scene can be extracted from the 2D image based on the apartment structure information, apartment structure image, and scene image generated during the 2D design stage. The 3D generation stage can only include one or more of the following sub-intelligent agents: scene parsing agent, dimensionality-upgrading agent, asset generation agent, and spatial reasoning agent. Among them, the scene parsing agent can call a multimodal understanding model to perform spatial understanding reasoning based on the 2D scene image: (1) Target detection: detect the 2D bounding boxes of each object in the scene image from the 2D scene image, and generate a set of open semantic labels and appearance descriptions (which may include texture descriptions). (2) Spatial relationship graph extraction: record the spatial relationship of each object, including the support and connection relationship between objects, the relationship between objects and building structures, etc., to obtain a spatial relationship graph. For example, a table lamp is placed on a table, cabinet 1 and cabinet 2 are spliced ​​together, and a sofa is against a wall, etc. Based on the spatial relationship graph and the 2D scene image, the orientation angle of the object is estimated. For example, in the scene image, the initial candidate angle of the object orientation is obtained by using the two-dimensional bounding box of the object; the relationship between the object and the building structure, the support, adjacency, and attachment of the adjacent objects in the spatial relationship diagram is read, and the candidate angle is corrected by using the wall orientation and the orientation of the adjacent objects as constraints; the optimal angle that meets the spatial relationship constraints is selected as the estimated orientation of the object. (3) Height range estimation: The agent infers the position height range of each object based on the spatial relationship diagram and common sense prior, including the highest height and the lowest height. For example, the large language model is used to read the topological relationship of each object in the spatial relationship diagram, such as support, attachment, and landing, and the common sense prior knowledge base of indoor objects is retrieved; the initial height range of each object is derived by combining the object category and support constraints; the physical rationality of the height range is then verified to eliminate illegal solutions such as objects penetrating the support surface and exceeding the building floor height; finally, the position height range corresponding to each object is obtained.

[0030] According to embodiments of this disclosure, by using topological constraints between objects and buildings, and between objects in a spatial relationship diagram, the accuracy of the predicted object orientation and height can be improved, the three-dimensional attributes of the objects can be enhanced, and an accurate data foundation can be provided for the subsequent generation of three-dimensional scenes.

[0031] In one implementation, such as Figure 2 As shown, based on the object's two-dimensional bounding box and position height range, three-dimensional layout information including the object's three-dimensional model is obtained, including: S208. Using an upgraded intelligent agent, a basic three-dimensional layout is obtained based on the two-dimensional bounding box, the camera parameters, and the position height range; S209. Use 3D assets to generate intelligent agents, and generate corresponding 3D meshes based on all objects in the basic 3D layout.

[0032] In this embodiment, the dimensionality-enhancing agent generates a 3D bounding box for each object based on its 2D bounding box, camera parameters, and position / height range, through processes such as perspective projection. A basic 3D layout can be obtained from the 3D bounding boxes generated for each object in the 2D scene image. The 3D asset generation agent can then generate 3D meshes corresponding to each object based on its 3D bounding boxes in the basic 3D layout and its texture in the 2D scene image, thus obtaining the 3D assets corresponding to the 2D scene image.

[0033] In one implementation, a basic three-dimensional layout is obtained based on the two-dimensional bounding box, the camera parameters, and the position height range, including: The projection geometry calculation tool is invoked to project the two-dimensional bounding box of the object in the image space onto a two-dimensional projection region in the metric world space based on the camera parameters; Based on the camera height, the projected two-dimensional region, and the position height range in the camera parameters, the offset of the projected two-dimensional region is calculated; Based on the projected two-dimensional region and its offset, the true two-dimensional bounding box of the object in the metric world space is calculated. The center position of the object is determined based on the real two-dimensional bounding box; Based on the real two-dimensional bounding box, the center position of the object, and the orientation angle of the object, the real size information of the object is determined; The basic 3D layout is generated based on the actual size information of the object.

[0034] In this embodiment, the upgraded agent invokes a projection geometry calculation tool to project the object's two-dimensional bounding box from image space to a projected two-dimensional region in the metric world space based on previously recorded camera parameters. Then, due to the height difference, the two-dimensional region obtained by perspective projection (i.e., the projected two-dimensional region) is corrected to obtain the object's actual occupied area (i.e., the actual two-dimensional bounding box). Specifically, based on the camera height, projected two-dimensional region, and height range in the camera parameters, the correction amount (i.e., the offset) of the projected two-dimensional region can be calculated. By merging the projected two-dimensional region and its offset, the actual two-dimensional bounding box of the object in the metric world space can be calculated, thereby determining the object's center position. Combined with the previously estimated object orientation angle, the object's actual size information, such as the corrected length and width, can be determined. Combined with the height range, a basic three-dimensional layout (including the three-dimensional bounding boxes of each object) can be generated. The three-dimensional bounding boxes of each object can represent the approximate position, size, and location of the three-dimensional mesh. The shape of the 3D mesh may not be a regular shape such as a cuboid or cube, but an irregular shape, such as a table or a chair. 3D bounding boxes cannot describe the precise positional relationship when two objects overlap. 3D meshes, as a more precise method of scene description, can describe the precise positional relationship between models. For example, suppose a chair is located in the space under a table. Using only a 3D bounding box might only indicate whether the table and chair overlap, while a 3D mesh can precisely describe this relationship of the chair being in the space under the table.

[0035] According to embodiments of this disclosure, by correcting the offset of the two-dimensional bounding box of an object through projection, camera parameters, and predicted height, the true size of the object can be made more accurate, thereby improving the realism and accuracy of the subsequently generated three-dimensional scene.

[0036] In one implementation, generating a corresponding 3D mesh based on all objects in the basic 3D layout includes: The image block region of each object is segmented using a two-dimensional bounding box for each object in the scene image; The multimodal generation model is invoked to convert the image patch region into a three-quarter view based on the image patch region, semantic labels, and appearance description; The image is called to generate a 3D model. Based on the three-quarter view, a 3D mesh and texture map of the object are generated to obtain a textured 3D model of the object. The texture map is generated based on the texture in the image block region. The 3D layout information includes the textured 3D model of the object.

[0037] In this embodiment, the 3D asset generation agent can first segment the image block region within the 2D bounding box of each object from the scene image. Here, the 2D bounding box can be the uncorrected 2D projected region or the corrected real 2D bounding box. The 3D asset generation agent calls a multimodal generation model to convert the image block region into a three-quarter view based on the object's image block region, semantic tags, and appearance description. Then, it calls the image to the 3D generation model to generate a 3D mesh of the object based on the three-quarter view. For example, based on the structured spatial relationship diagram, the object's 2D outline, estimated orientation, and height range, it recovers the 3D bounding box of each object; it calls the prior mesh template of the corresponding object based on the object category, and performs affine transformation on the template according to parameters such as size, orientation, and height; it performs spatial placement and collision verification on the transformed object meshes and the building structure mesh, outputting a 3D mesh model. Then, based on the texture in the image block region of each object, it generates a texture map on the 3D mesh of the object, obtaining a textured 3D model of the object. The 3D model of each object and the 3D model of the background image can constitute the overall 3D asset. By utilizing the oblique observation information provided by the three-quarter view and combining it with texture to complete the 3D mesh reconstruction, the geometry of the object's side appearance can be filled in, the model details can be enriched, and a complete 3D model can be obtained, providing high-quality 3D assets for simulation rendering.

[0038] In one implementation, the region corresponding to the object is corrected based on the 3D model of the object in the scene image and the 3D layout information, including one or more of the following steps: Using a yaw angle correction agent, the rendering tool is invoked to obtain the layout image corresponding to the 3D model of each object; the layout image is compared with the area corresponding to the object in the scene image once or multiple times; the yaw angle is adjusted according to the comparison results until the orientation of the layout image and the area corresponding to the object in the scene image are consistent; Using a size correction agent, the rendering tool is invoked to obtain the layout image corresponding to the 3D model of each object; the layout image is compared with the size of the object in the scene image once or multiple times; the size is adjusted according to the comparison result until the size of the layout image is consistent with the size of the object in the scene image.

[0039] In this embodiment, the spatial reasoning agent can perform instance-level reasoning and / or layout-level reasoning. In instance-level reasoning, the spatial reasoning agent may include a yaw angle correction agent and / or a size correction agent, which invokes a rendering tool to obtain a layout image corresponding to the 3D model of each object; then compares the region corresponding to the object in the scene image with the region corresponding to the object in the layout image to perform object instance yaw angle and size correction. If the yaw angle and / or size of the object in the layout image and the scene image are inconsistent, the yaw angle and / or size of the object in the 3D model can be adjusted, and then the layout image is regenerated for further comparison. After one or more comparisons and corrections, the yaw angle and / or size of the object in the layout image and the scene image can reach a basic consistency (the difference does not exceed a threshold), or the process stops after reaching a certain number of iterations.

[0040] According to embodiments of this disclosure, by using a spatial reasoning agent to perform yaw angle and / or size correction on objects, it is possible to correct angular deviations and / or scale errors, eliminate spatial conflicts such as objects passing through walls or being misaligned, ensure that each object is placed in compliance with regulations in the measurement space, improve the geometric realism of the three-dimensional mesh, and ensure the reliability of subsequent simulation and rendering results.

[0041] Figure 4 This is a flowchart illustrating a spatial reasoning-based 3D scene generation method according to an embodiment of the present disclosure. This method can be used to implement one or more features of the aforementioned spatial reasoning-based 3D scene generation method. In one embodiment, the parameters of the 3D layout information are iteratively corrected based on the scene visualization image. After exiting the iteration loop according to the correction result, a 3D layout scene corresponding to the scene image is obtained, including: S401. Evaluate whether there are any unreasonable items in the scene visualization image; S402. If there are unreasonable items in the scene visualization image, the parameters of the three-dimensional layout information are corrected based on the unreasonable items to obtain the corrected three-dimensional layout information. S403. Render the corrected 3D layout information to obtain an updated scene visualization image; S404. Continue to evaluate whether there are any unreasonable items in the scene visualization image until there are no unreasonable items or the maximum number of iterations is reached, then exit the iteration loop and obtain the three-dimensional layout scene corresponding to the scene image.

[0042] In this embodiment of the disclosure, during layout layer reasoning, the spatial reasoning agent may include a verifier agent, used to evaluate the scene visualization image rendered from the 3D layout information and determine whether there are any unreasonable items in the scene visualization image. Unreasonable items may be of various types, such as unreasonable spatial layout items, unreasonable geometric dimensions, unreasonable rendering visual items, and unreasonable semantic functions. For example, unreasonable spatial layout items include furniture crossing walls or boundaries, furniture objects intersecting and overlapping, objects floating in mid-air, and insufficient passageways; unreasonable geometric dimensions include distorted furniture scale ratios, abnormal object orientations and postures, and inconsistencies between the rendered floor plan and the original floor plan; unreasonable rendering visual items include disordered lighting and shadows, distorted material textures, and perspective distortion; unreasonable semantic functions include mismatches between the type and quantity of rendered furniture and the description of home furnishings. Based on the unreasonable items in the evaluation results, the relevant parameter values ​​in the 3D layout information can be corrected. After correction, a new scene visualization image can be re-rendered, and the scene visualization image can be further evaluated for the presence of unreasonable items. After one or more iterations, if no unreasonable items are found in the scene visualization image, or if the maximum number of iterations has been reached, the iteration loop can be exited, resulting in the final 3D layout scene. Based on the final 3D layout scene, the final scene visualization image can be rendered. By evaluating unreasonable items in the scene visualization image rendered from the 3D layout information, generation defects such as wall penetrations, abnormal dimensions, and missing components can be identified, anomalies can be filtered out, and timely iterative optimization can be performed to improve the accuracy and realism of the output scene.

[0043] Figure 5 This is a schematic diagram of an intelligent agent framework based on a Large Multimodal Model (LMM) according to an embodiment of the present disclosure, which utilizes its multimodal generation and understanding capabilities to enhance the diversity and realism of scene synthesis.

[0044] The overall architecture of the technical solution in the embodiments of this disclosure is as follows: Figure 5As shown, this architecture can include two collaborative phases: a top-view scene design phase and a 3D generation phase. Each phase can further organize the agents into several sub-agents, each responsible for a specific sub-task, capable of calling its corresponding underlying tools, and uniformly orchestrated by a top-level agent called the PipelineComposer. For example, the top-view scene design phase could include agents performing floor plan design sub-tasks, agents performing planar design reasoning sub-tasks, etc., while the 3D generation phase could include agents performing scene parsing sub-tasks, agents performing spatial reasoning sub-tasks, etc. Each phase exposes interfaces to the outside world in the form of stage-agents or tools, receiving input or outputting data. The agents can be built based on multimodal understanding and multimodal generation models, and the tools can include fast renderers, geometric calculation tools, etc. Multiple intelligent agents and their corresponding tools collectively constitute a hierarchical intelligent agent system, cascading from high-level orchestration to low-level execution. This system can perform fully autonomous end-to-end orchestration and execution, and also supports interactive user intervention.

[0045] In this embodiment of the disclosure, after the user inputs a multimodal design instruction, the system enters the top-view design stage. The multimodal design instruction may include text descriptions, reference images, etc. For example, the text description may include the user's 3D scene generation requirements, such as "The supermarket has an L-shaped layout. The entrance is located on the north side. The entrance area has two automatic checkout machines and a neatly arranged shopping cart storage area. The main retail area is located on the south side. A set of wall-mounted shelves is arranged on the west wall, with four rows of double-sided shelves in the middle, mainly displaying snacks, beverages, packaged foods, and daily necessities. The southeast side is used as a fresh produce area...". Reference images may include floor plan images, scene photos, etc. The top-view design stage can parse the user's input multimodal design instruction and use the user-defined floor plan design agent to generate a structured floor plan that meets the requirements. Then, a multimodal generation model can be used to generate a top-view scene image including furniture furnishings based on the structured floor plan, entering the 3D generation stage. In the 3D generation stage, a multimodal understanding model can be used to perform analytical reasoning based on the top-view scene image to obtain a structured 3D scene. Further, 3D assets associated with the structured 3D scene can be generated. Finally, rendering tools can be invoked to complete the final scene rendering.

[0046] I. Top-view scene design stage In this embodiment of the disclosure, this stage may consist of three sub-agents: a floor plan designer agent, an image generator agent, and a design reasoner agent.

[0047] 1. Smart body for floor plan design In this embodiment, the floor plan design agent can first generate a set of detailed scene description information and JSON-formatted floor plan structure information F, driven by user commands. Specifically, in interactive mode, when the user only provides a rough scene type description, the floor plan design agent will gradually supplement, refine, and adjust the floor plan structure information and scene description information through dialogue interaction. In automatic mode, the floor plan design agent will autonomously complete the information completion. Subsequently, the floor plan design agent can call a rendering tool to render a top-down view of the floor plan structure Image_1, align the image space with the metric world space, and record the corresponding camera parameters C for subsequent analysis.

[0048] 2. Image generation intelligent agent In this embodiment of the disclosure, the image generation agent can synthesize a top-down view scene image Image_2 with furniture furnishings based on the above-mentioned vacant apartment structure image Image_1, scene description information (such as other descriptions in the user instructions such as what furniture to place, what design to do), and camera parameters C, by calling a multi-modal generation model.

[0049] 3. Design a reasoning agent In this embodiment, the design reasoning agent is mainly built from a visual understanding model, acting as a verifier throughout the top-view scene design phase. Together with the aforementioned floor plan design agent and image generation agent, it forms a closed loop of reasoning-and-acting (ReAct) framework. Specifically, a two-way verification strategy can be used. An image filter compares the floor plan structure image Image_1 with the generated scene image Image_2. If the floor plan structures of the two images are inconsistent, Image_2 is regenerated until a set of substantially consistent images is generated, at which point the verification direction is switched. Then, a floor plan fixer compares the generated scene images Image_2 and Image_1. If inconsistencies in floor plan structures are found, the floor plan structure information F is incrementally adjusted, and an empty floor plan structure image Image_1 is re-rendered. When the two sets of images are fully aligned, the generate-and-verify loop exits, and the data flows to the next stage.

[0050] II. 3D Generation Stage In this embodiment of the disclosure, a structured 3D scene can be extracted based on the floor plan information F, floor plan image Image_1, and complete floor plan scene image Image_2 generated during the top-view scene design stage. This can include the following agents or tools: a scene parser agent, a 2D-to-3D lifting tool, an asset generation agent, and a spatial reasoning agent.

[0051] 1. Scene Analysis Intelligent Agent In this embodiment of the disclosure, the scene parsing agent can use a multimodal understanding model to perform understanding and reasoning based on the scene image Image_2, including object detection, scene graph extraction, and height range estimation, as follows: (1) Object detection can detect the 2D bounding boxes of all newly added objects in the image and simultaneously generate a set of open semantic labels and appearance descriptions, such as appearance descriptions with texture descriptions.

[0052] (2) Scene graph extraction can record the spatial relationships of each object, including the support and connection relationships between objects, and the relationship between objects and building structures, such as the walls they lean against, hanging or wall-mounted structures, etc. The orientation angle of objects can be further estimated based on the scene graph (e.g., spatial relationship scene graph, topology, etc.) and the visual image Image_2.

[0053] (3) Height range estimation can be based on scene graph information and common sense priors (such as the prediction results of the model) to infer the height range [h_low, h_high] of each object.

[0054] 2.2D to 3D Upgrading Tool In this embodiment of the disclosure, a projection geometry calculation tool, such as a 2D-to-3D dimensionality upgrade tool, can be called at this stage to perform 2D to 3D extraction calculations to generate a basic 3D layout, which will be further optimized and adjusted in subsequent stages.

[0055] For example, based on camera parameters C, a 2D bounding box in image space can be projected onto a 2D region in metric world space. Due to height differences, the 2D region obtained from perspective projection needs correction to obtain the actual area occupied by the object. Based on the camera height, the projected 2D region, and the height range [h_low, h_high], the correction amount (e.g., offset) for the projected 2D region can be accurately calculated. Combining these two factors allows for the calculation of the object's true 2D bounding box in metric world space, thus determining the object's center position. Combining this with the object's orientation angle further determines the object's true size information. This generates a basic 3D layout (e.g., including a 3D bounding box).

[0056] 3.3D Asset Generation Intelligent Agent In this embodiment of the disclosure, the 3D asset generation agent can generate a corresponding 3D mesh based on all instances in a basic 3D layout. Specifically, this may include: (1) Segment each instance (e.g., furniture, floor, wall, ceiling, etc.) into the corresponding region (image block region) in Image_2.

[0057] (2) Call the multimodal generation model to convert the top view into a three-quarter view based on the segmented image block regions, semantic labels and appearance descriptions.

[0058] (3) Call the image-to-3D generation model to generate a textured 3D model based on the 3 / 4 view; in this step, texture maps of the house structure (furniture, floor, walls, ceiling) can be generated synchronously based on Image_2 (each image block area).

[0059] 4. Spatial Reasoning Agent In this embodiment of the disclosure, after the processing described in the above embodiments, a preliminary 3D layout can be obtained. This preliminary 3D layout (e.g., denoted as S_0) may include the apartment layout information F and the structure and asset information of all object instances. However, object parameter estimation may contain errors. Furthermore, issues such as object out-of-bounds collisions, wall penetration, etc., may still remain in the overall apartment layout dimension. S_0 can be fine-tuned based on a spatial reasoning agent from the object parameter dimension of the instance layer and the overall layout layer dimension. Specifically, this may include: (1) Instance-level reasoning can use two reasoning action (ReAct) sub-agents to perform object instance yaw angle and size correction: (a) Yaw Angle Correction Agent: Because objects may have symmetrical or complex shapes such as "L" or "U" shapes, the orientation angle estimated by the scene parsing agent may be ambiguous; at the same time, the 3D assets generated by the 3D asset generation agent may also be non-regular. These factors may cause the object placement orientation to be abnormal. In the yaw angle correction step, for each instance, the yaw angle correction agent can call the rendering tool to obtain the placement image of the instance's assets, repeatedly compare it with the corresponding area of ​​the instance in Image_2, and adjust the asset yaw angle until the image areas of both are strictly consistent with the positive direction.

[0060] (b) Size Correction Agent: Since the scene parsing agent calculates object sizes based on a cube assumption, the actual generated 3D assets may not strictly match the calculated sizes. The size correction agent can call the rendering tool to obtain the layout image of the instance asset, repeatedly compare it with the corresponding area of ​​the instance in Image_2, and adjust the size. By repeatedly adjusting the size information, the rendered image of the instance layout is kept consistent with the corresponding area of ​​the instance in Image_2.

[0061] (2) Layout layer reasoning can use the ReAct paradigm for layout adjustment. Specifically, the input can be a base reference image Image_2, 3D layout information S_0 obtained from Image_2 through a series of understanding and reasoning actions, and a scene visualization image rendered from S_0. The model needs to evaluate the most significant unreasonable items in the scene, adjust the parameter values ​​of S_0 to obtain S_1, and update the real-time scene visualization image before proceeding to the next iteration. The loop exits when the validator agent no longer reports any remaining issues or the maximum number of iterations of the system is reached, resulting in the final 3D layout scene S.

[0062] According to the embodiments of this disclosure, the following beneficial effects can be achieved: 1. It eliminates the need to collect large amounts of labeled 3D indoor scene data, reducing the need for data annotation, decreasing training and usage costs, and improving overall efficiency.

[0063] 2. The use of multimodal large models provides more direct spatial intelligence capabilities such as scene image generation, 3D parameter estimation, and layout reasoning, which improves the diversity, realism and generalization of scene synthesis.

[0064] 3. By adopting a generate-and-verify strategy, the quality, fidelity, and spatial rationality of the final generated scene are improved through iterative self-correction and optimization.

[0065] 4. The overall framework is based on the native capabilities of multimodal models, avoiding dependence on external model / database resources. It only introduces basic general-purpose 3D visual geometry calculations for data transformation and organization. This makes the framework highly scalable, allowing it to improve its performance in sync with the upgrades of cutting-edge multimodal models without requiring complex replacement of dependent components, thus enhancing the framework's universality and flexibility.

[0066] Figure 6 This is a schematic diagram of a three-dimensional scene generation apparatus based on spatial reasoning according to an embodiment of the present disclosure. In one embodiment, the apparatus may include: The structural diagram generation module 601 generates a two-dimensional apartment structure image based on user instructions. Scene image generation module 602 is used to generate a two-dimensional scene image including furnishing objects based on the apartment structure image; The spatial understanding module 603 is used to perform spatial understanding based on the scene image to obtain the two-dimensional bounding box and position height range of the object; The layout acquisition module 604 is used to obtain three-dimensional layout information including the three-dimensional model of the object based on the two-dimensional bounding box and the position and height range of the object. The three-dimensional correction module 605 is used to correct the region corresponding to the object based on the three-dimensional model of the object in the scene image and the three-dimensional layout information; Rendering module 606 is used to render a scene visualization image based on the three-dimensional layout information; The correction module 607 is used to iteratively correct the parameters of the three-dimensional layout information based on the scene visualization image, and after exiting the iteration loop according to the correction result, obtain the three-dimensional layout scene corresponding to the scene image.

[0067] Figure 7This is a schematic diagram of the structure of a 3D scene generation device based on spatial reasoning according to an embodiment of the present disclosure. The device may include: a structure diagram generation module 701, a scene diagram generation module 702, a spatial understanding module 703, a layout acquisition module 704, a 3D correction module 705, a rendering module 706, and a correction module 707. The functions and roles of each module can be found in the description of the modules of the 3D scene generation device based on spatial reasoning described above, and will not be repeated here. In one embodiment, the structure diagram generation module 701 may include: Design submodule 7011 is used to enable the user-defined model design agent to generate scene description information and floor plan structure information that conform to user instructions; The rendering submodule 7012 is used to call the rendering tool to render a top-down view of the apartment structure based on the apartment structure information.

[0068] In one embodiment, the scene image generation module 702 is further configured to use an image generation agent to synthesize a two-dimensional scene image with furnishing objects by calling a multimodal generation model based on the apartment structure image, the scene description information, and camera parameters.

[0069] In one implementation, such as Figure 7 As shown, the device may further include: Verification module 708 is used to use a design reasoning agent to cyclically verify whether the house structure image and the scene image are aligned; The verification module 708 is also used to end the verification when the apartment structure image is aligned with the scene image.

[0070] In one implementation, the verification module 708 can also be used to repeatedly execute one or more of the following steps until the floor plan image and the scene image are aligned and the loop is exited: The image filter is called to compare the floor plan image and the scene image. If the floor plan of the floor plan image and the scene image are inconsistent, the scene image is regenerated and then compared again. The floor plan repair tool is invoked to compare the floor plan image and the scene image. If the floor plan images and the scene image are inconsistent, the floor plan information is adjusted incrementally, and the floor plan image is re-rendered and then compared again.

[0071] In one implementation, such as Figure 7 As shown, the spatial understanding module 703 may include: Submodule 7031 is invoked to use the scene parsing agent to invoke the multimodal understanding model, perform spatial understanding based on the scene image, and obtain the two-dimensional bounding box, semantic label and appearance description of each object in the scene image; The recording submodule 7032 is used to record the spatial relationships of the objects to obtain a spatial relationship diagram; Angle prediction submodule 7033 is used to predict the orientation angle of the object based on the spatial relationship diagram and the scene image; The height prediction submodule 7034 is used to predict the position height range of the object based on the spatial relationship diagram.

[0072] In one implementation, such as Figure 7 As shown, the layout acquisition module 704 includes: The dimensionality-upgrading submodule 7041 is used to obtain a basic three-dimensional layout based on the two-dimensional bounding box, the camera parameters, and the position height range using the dimensionality-upgrading intelligent agent; The 3D production submodule 7042 is used to generate intelligent agents using 3D assets, generating corresponding 3D meshes based on all objects in the basic 3D layout.

[0073] In one implementation, the dimension-upgrading submodule 7041 can also be used to invoke a projection geometry calculation tool to project the two-dimensional bounding box of the object in the image space onto a projected two-dimensional region in the metric world space based on the camera parameters; calculate the offset of the projected two-dimensional region based on the camera height, the projected two-dimensional region, and the position height range in the camera parameters; calculate the true two-dimensional bounding box of the object in the metric world space based on the projected two-dimensional region and its offset; determine the center position of the object based on the true two-dimensional bounding box; determine the true size information of the object based on the true two-dimensional bounding box, the center position of the object, and the orientation angle of the object; and generate the basic three-dimensional layout based on the true size information of the object.

[0074] In one implementation, the 3D production submodule 7042 can also be used to segment the image block region of each object using the 2D bounding box of each object in the scene image; call a multimodal generation model to convert the image block region into a three-quarter view based on the image block region, semantic tags, and appearance description; call an image-to-3D generation model to generate a 3D mesh and texture map of the object based on the three-quarter view to obtain a textured 3D model of the object, wherein the texture map is generated based on the texture in the image block region, and the 3D layout information includes the textured 3D model of the object.

[0075] In one embodiment, the three-dimensional correction module 705 is further used in one or more of the following steps: Using a yaw angle correction agent, the rendering tool is invoked to obtain the layout image corresponding to the 3D model of each object; the layout image is compared with the area corresponding to the object in the scene image once or multiple times; the yaw angle is adjusted according to the comparison results until the orientation of the layout image and the area corresponding to the object in the scene image are consistent; Using a size correction agent, the rendering tool is invoked to obtain the layout image corresponding to the 3D model of each object; the layout image is compared with the size of the object in the scene image once or multiple times; the size is adjusted according to the comparison result until the size of the layout image is consistent with the size of the object in the scene image.

[0076] In one implementation, such as Figure 7 As shown, the correction module 707 may include: The evaluation submodule 7071 is used to evaluate whether there are any unreasonable items in the scene visualization image; The correction submodule 7072 is used to correct the parameters of the three-dimensional layout information based on the unreasonable items in the scene visualization image to obtain the corrected three-dimensional layout information. The rendering submodule 7073 is used to render the corrected 3D layout information to obtain an updated scene visualization image; The evaluation submodule 7071 can also be used to continue evaluating whether there are any unreasonable items in the scene visualization image until there are no unreasonable items or the maximum number of iterations is reached, then exit the iteration loop and obtain the three-dimensional layout scene corresponding to the scene image.

[0077] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0078] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0079] Figure 8 This is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Figure 8As shown, the electronic device includes a memory 810 and a processor 820. The memory 810 stores a computer program that can run on the processor 820. The number of memories 810 and processors 820 can be one or more. The memory 810 can store one or more computer programs, which, when executed by the electronic device, cause the electronic device to perform the methods provided in the above-described method embodiments. The electronic device may also include a communication interface 830 for communicating with external devices and performing data exchange and transmission.

[0080] If the memory 810, processor 820, and communication interface 830 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0081] Optionally, in a specific implementation, if the memory 810, processor 820, and communication interface 830 are integrated on a single chip, then the memory 810, processor 820, and communication interface 830 can communicate with each other through an internal interface.

[0082] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0083] Further, optionally, the aforementioned memory may include read-only memory and random access memory, and may also include non-volatile random access memory. The memory may be volatile or non-volatile, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct RAMBUS RAM (DR RAM).

[0084] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line, DSL) or wireless (e.g., infrared, Bluetooth, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer, or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)). It is worth noting that the computer-readable storage media mentioned in this disclosure can be non-volatile storage media; in other words, it can be non-transient storage media.

[0085] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0086] In the description of the embodiments of this disclosure, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0087] In the description of the embodiments disclosed herein, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0088] In the description of embodiments of this disclosure, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0089] The above description is merely an exemplary embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A method for generating a 3D scene based on spatial reasoning, comprising: Generate a two-dimensional apartment layout image based on user instructions; A two-dimensional scene image including furnishings is generated based on the apartment layout image; Based on the scene image, spatial understanding is performed to obtain the two-dimensional bounding box and position height range of the object; Based on the object's two-dimensional bounding box and position height range, three-dimensional layout information including the object's three-dimensional model is obtained; The region corresponding to the object is corrected based on the 3D model of the object in the scene image and the 3D layout information; A scene visualization image is rendered based on the aforementioned 3D layout information; Based on the scene visualization image, the parameters of the 3D layout information are iteratively corrected. After exiting the iteration loop according to the correction result, the 3D layout scene corresponding to the scene image is obtained.

2. The method according to claim 1, wherein, Generate a two-dimensional floor plan image based on user instructions, including: The floor plan is used to design an intelligent agent that generates scene description information and floor plan structure information that conform to user instructions; The rendering tool is invoked to render a top-down view image of the apartment layout based on the apartment layout information.

3. The method according to claim 2, wherein, Generate a two-dimensional scene image including furnishings based on the apartment layout image, including: An image-generating agent uses the floor plan image, scene description information, and camera parameters to call a multimodal generation model to synthesize a two-dimensional scene image with furnishings.

4. The method according to claim 3, further comprising: The design reasoning agent is used to repeatedly verify whether the house layout image and the scene image are aligned; The verification ends when the floor plan image and the scene image are aligned.

5. The method according to claim 4, wherein, The system iteratively verifies whether the floor plan image and the scene image are aligned, including repeatedly executing one or more of the following steps until the floor plan image and the scene image are aligned and the loop exits: The image filter is called to compare the floor plan image and the scene image. If the floor plan of the floor plan image and the scene image are inconsistent, the scene image is regenerated and then compared again. The floor plan repair tool is invoked to compare the floor plan image and the scene image. If the floor plan images and the scene image are inconsistent, the floor plan information is adjusted incrementally, and the floor plan image is re-rendered and then compared again.

6. The method according to claim 1, wherein, Based on the scene image, spatial understanding is performed to obtain the two-dimensional bounding box and position height range of the object, including: The scene parsing agent calls the multimodal understanding model to perform spatial understanding based on the scene image, and obtains the two-dimensional bounding box, semantic label and appearance description of each object in the scene image; A spatial relationship diagram is obtained by recording the spatial relationships of the objects. Based on the spatial relationship diagram and the scene image, predict the orientation angle of the object; Based on the spatial relationship diagram, the position and height range of the object are predicted.

7. The method according to claim 6, wherein, Based on the object's two-dimensional bounding box and position / height range, three-dimensional layout information including the object's three-dimensional model is obtained, including: Using an upgraded intelligent agent, a basic three-dimensional layout is obtained based on the two-dimensional bounding box, camera parameters, and the position height range; Intelligent agents are generated using 3D assets, and corresponding 3D meshes are generated based on all objects in the basic 3D layout.

8. The method according to claim 7, wherein, Based on the two-dimensional bounding box, the camera parameters, and the position and height range, a basic three-dimensional layout is obtained, including: The projection geometry calculation tool is invoked to project the two-dimensional bounding box of the object in the image space onto a two-dimensional projection region in the metric world space based on the camera parameters; Based on the camera height, the projected two-dimensional region, and the position height range in the camera parameters, the offset of the projected two-dimensional region is calculated; Based on the projected two-dimensional region and its offset, the true two-dimensional bounding box of the object in the metric world space is calculated. The center position of the object is determined based on the real two-dimensional bounding box; Based on the real two-dimensional bounding box, the center position of the object, and the orientation angle of the object, the real size information of the object is determined; The basic 3D layout is generated based on the actual size information of the object.

9. The method according to claim 7, wherein, Generate a corresponding 3D mesh based on all objects in the basic 3D layout, including: The image block region of each object is segmented using a two-dimensional bounding box for each object in the scene image; The multimodal generation model is invoked to convert the image patch region into a three-quarter view based on the image patch region, semantic labels, and appearance description; The image is called to generate a 3D model. Based on the three-quarter view, a 3D mesh and texture map of the object are generated to obtain a textured 3D model of the object. The texture map is generated based on the texture in the image block region. The 3D layout information includes the textured 3D model of the object.

10. The method according to any one of claims 7 to 9, wherein the region corresponding to the object is corrected based on the scene image and the three-dimensional model of the object in the three-dimensional layout information, comprising one or more of the following steps: Using a yaw angle correction agent, the rendering tool is invoked to obtain the layout image corresponding to the 3D model of each object; the layout image is compared with the area corresponding to the object in the scene image once or multiple times; the yaw angle is adjusted according to the comparison results until the orientation of the layout image and the area corresponding to the object in the scene image are consistent; Using a size correction agent, the rendering tool is invoked to obtain the layout image corresponding to the 3D model of each object; the layout image is compared with the size of the object in the scene image once or multiple times; the size is adjusted according to the comparison result until the size of the layout image is consistent with the size of the object in the scene image.

11. The method according to claim 9, wherein, Based on the scene visualization image, the parameters of the 3D layout information are iteratively corrected. After exiting the iteration loop according to the correction result, the 3D layout scene corresponding to the scene image is obtained, including: Assess whether there are any unreasonable items in the scene visualization image; If there are unreasonable items in the scene visualization image, the parameters of the three-dimensional layout information are corrected based on the unreasonable items to obtain the corrected three-dimensional layout information; The corrected 3D layout information is rendered to obtain an updated scene visualization image; Continue to evaluate whether there are any unreasonable items in the scene visualization image until there are no unreasonable items or the maximum number of iterations is reached, then exit the iteration loop and obtain the 3D layout scene corresponding to the scene image.

12. A three-dimensional scene generation device based on spatial reasoning, comprising: The structural diagram generation module generates two-dimensional floor plan images based on user commands. The scene image generation module is used to generate a two-dimensional scene image including furnishing objects based on the apartment layout image; The spatial understanding module is used to perform spatial understanding based on the scene image to obtain the two-dimensional bounding box and position height range of the object; The layout acquisition module is used to obtain three-dimensional layout information including the three-dimensional model of the object based on the two-dimensional bounding box and the position and height range of the object. A 3D correction module is used to correct the region corresponding to the object based on the 3D model of the object in the scene image and the 3D layout information; The rendering module is used to render a scene visualization image based on the three-dimensional layout information; The correction module is used to iteratively correct the parameters of the 3D layout information based on the scene visualization image, and after exiting the iteration loop according to the correction result, obtain the 3D layout scene corresponding to the scene image.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images