Multi-agent-based single-view simulation scene generation method and device and medium

Abstract

The invention discloses a multi-agent-based single-view simulation scene generation method and device and a medium, and belongs to the technical field of three-dimensional modeling simulation, and the method comprises the steps: recognizing all objects in a single indoor RGB image through a sensing agent, carrying out the positioning and scale calculation of each object, and outputting sensing data; reasoning the spatial relationship of each object in the RGB image through a relationship reasoning agent, and constructing a scene graph; retrieving a three-dimensional model corresponding to each object from the three-dimensional model data set according to each sensing data through a retrieval agent, and performing scale constraint to obtain a three-dimensional model set; placing each three-dimensional model in the three-dimensional model set through a layout intelligent agent according to the scene graph and the pose information of each object; and performing collision detection and visual comparison on the first simulation scene through the optimization intelligent agent, and performing layout optimization according to detection and comparison results. Therefore, by implementing the method, the simulation scene which is accurate in geometry and consistent in semantics can be reconstructed according to the single view.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional modeling simulation, and in particular to a single-view simulation scene generation method and device based on multiple agents and a medium. BACKGROUND

[0002] Existing three-dimensional scene generation techniques mainly include the following: first, single-view indoor scene generation techniques such as CAST3D and Diorama, which extract visual semantic information and spatial position relationships of related objects from a single RGB image through a pre-trained visual base model, and then reconstruct an indoor scene. However, this method only extracts semantic information and rough spatial position relationships of related objects in the scene, lacks size constraints of objects in the picture and spatial constraints in the scene, and is prone to proportion distortion. At the same time, it lacks collision, support and stability detection, and it is difficult to ensure that the generated scene is used for simulation. This method cannot automatically transform object position relationships under related physical space constraints to achieve automatic construction of simulation data.

[0003] Second, language / layout-guided scene synthesis techniques, which accept language or indoor scene layout as input, extract and retrieve reasonable semantic information of related objects from the input information, and infer reasonable spatial position relationships and placement, and finally output a 3D scene. For example, Holodeck and I-Design generate diverse three-dimensional environments and capture semantic / functional attributes from text prompts, and LayoutVLM optimizes three-dimensional layouts with visual-language models to ensure consistency with images and instructions. However, this method mainly relies on text or layout priors and does not involve picture content analysis and feedback, and cannot be compatible with the task scenario of simulation environment data construction and achieve ideal results. At the same time, the optimization process focuses on feasibility and reasonableness, and lacks explicit supervision information to constrain physical space consistency.

[0004] Finally, virtual scene generation techniques for robots, such as VR-Robot, which need to collect multiple frames of indoor scene data and generate corresponding virtual scenes based on 3D Gaussian reconstruction, and export 3D Gaussian to Mesh format. However, this method needs to collect continuous indoor scene video frames as input, and needs to collect additional data and reconstruct when the scene changes, and the effect of the virtual scene depends on the effect of the Mesh exported from the 3D Gaussian. At the same time, this method can only reconstruct static scenes, and when the position relationship of objects in the scene changes, the entire scene needs to be reconstructed, which has high iterative update cost. Therefore, this method will cause a substantial increase in construction cost when constructing large-scale scene data, and it is difficult to efficiently generate diverse and physically feasible variants. SUMMARY

[0005] The application provides a multi-agent-based single-view simulation scene generation method, device and medium.

[0006] The application provides a multi-agent-based single-view simulation scene generation method, device and medium. The indoor single RGB image is input into a pre-trained perception agent, so that the perception agent identifies all objects in the RGB image, and performs positioning and scale calculation on each object, and outputs perception data of each object; wherein the perception data includes description text, a bounding box, a mask, relative scale and pose information of each object; The RGB image, the bounding box and the mask are input into a pre-trained relationship reasoning agent, so that the relationship reasoning agent reasons the spatial relationship of each object in the RGB image, and constructs a scene graph; Each perception data is input into a pre-trained retrieval agent, so that the retrieval agent retrieves a three-dimensional model corresponding to each object from a three-dimensional model data set according to each perception data and performs scale constraint, to obtain a three-dimensional model set; The scene graph, the three-dimensional model set and the pose information of each object are input into a pre-trained layout agent, so that the layout agent places each three-dimensional model in the three-dimensional model set according to the scene graph and the pose information of each object, to obtain a first simulation scene; The first simulation scene is input into a pre-trained optimization agent, so that the optimization agent performs collision detection and visual comparison on the first simulation scene, and performs layout optimization according to the detection and comparison results, to obtain a simulation scene.

[0007] The application can provide multi-modal representation of objects for subsequent three-dimensional reconstruction by identifying objects from a single RGB image and outputting perception data containing geometric scale and pose by a perception agent; can analyze and structure the spatial position relationship in the scene to provide semantic constraints for scene layout by reasoning the spatial relationship of objects based on visual information to construct a scene graph by a relationship reasoning agent; can match object models that are consistent with the target appearance and size from the database to ensure the geometric accuracy of the reconstructed objects by retrieving three-dimensional models according to perception data and performing scale constraint by a retrieval agent; can convert relationship semantics into geometric constraints to complete the preliminary construction of the scene by placing three-dimensional models according to the scene graph and pose information by a layout agent; and can improve the physical rationality and layout consistency of the scene by performing collision detection and visual comparison optimization on the preliminary scene by an optimization agent. Compared with the prior art, the application can reconstruct a simulation scene that is geometrically accurate and semantically consistent according to a single view.

[0008] Further, the RGB image in the single room is input into a pre-trained perception agent, so that the perception agent identifies all objects in the RGB image, and each object is positioned and scaled, and the perception data of each object is output, including: all objects in the RGB image are identified by a GPT-4o model, and the corresponding description text of each object is output; the bounding box and mask of each object are positioned by a GroundedSAM-v2 model; the depth map of the RGB image is estimated by a MoGe-2 model, and based on the camera internal parameter and the depth map, the scene point cloud of the RGB image is calculated; wherein the camera internal parameter is the internal parameter of the camera shooting the RGB image; based on the bounding box and mask of each object, the point cloud corresponding to each object is separated from the scene point cloud, and the relative scale of each object is determined based on the point cloud of each object; the pose information of each object is determined by a FoundationPose model, and the description text, bounding box, mask, relative scale and pose information corresponding to the same object are merged into a perception data.

[0009] The embodiments of the present application can accurately obtain the description text, measurement scale and spatial pose of the objects in the scene by using a series of pre-trained models such as GPT-4o, GroundedSAM-v2, MoGe-2 and FoundationPose to cooperatively complete identification, segmentation, depth estimation and pose determination, and form complete structured perception data.

[0010] Further, the RGB image, bounding box and mask are input into a pre-trained relationship reasoning agent, so that the relationship reasoning agent reasons the spatial relationship of each object in the RGB image, and constructs a scene graph, including: based on the bounding box and mask of each object, the spatial relationship of each object in the RGB image is reasoned by a GPT-4o model in multiple rounds, and the spatial relationship appearing in the multiple reasoning results with a frequency exceeding a preset threshold is retained to obtain a spatial relationship set; a scene graph is constructed based on the spatial relationship set.

[0011] The embodiments of the present application can ensure the accuracy and reliability of the spatial relationship in the constructed scene graph by performing multiple rounds of relationship reasoning based on the bounding box and mask and retaining high-frequency relationships, and avoid errors that may occur in a single reasoning.

[0012] Further, the perception data is input into a pre-trained retrieval agent, so that the retrieval agent retrieves a three-dimensional model corresponding to each object from a three-dimensional model data set according to the perception data, and performs scale constraint to obtain a three-dimensional model set, including: The description text of each object is feature-encoded by a CLIP model to obtain a text feature of each object; Based on the text feature of each object, a three-dimensional model corresponding to each object is retrieved from an Objavers data set and a 3D-FRONT data set by an OpenShape model to obtain a first three-dimensional model set; According to the mask of each object, each three-dimensional model in the first three-dimensional model set is appearance-matched to obtain a second three-dimensional model set; Based on the relative scale and pose information of each object, each three-dimensional model in the second three-dimensional model set is scale-constrained to obtain a three-dimensional model set.

[0013] The embodiment of the application can gradually filter out a three-dimensional model that is highly matched with the target object in semantics, appearance and size by sequentially performing a three-stage retrieval process of text feature retrieval, visual appearance matching and geometric scale constraint, and ensure the quality of the retrieval result.

[0014] Further, the scene graph, the three-dimensional model set and the pose information of each object are input into a pre-trained layout agent, so that the layout agent places each three-dimensional model in the three-dimensional model set according to the scene graph and the pose information of each object to obtain a first simulation scene, including: All semantic relationships in the scene graph are analyzed and converted into geometric constraints; Based on the geometric constraints and the pose information of each object, each three-dimensional model in the three-dimensional model set is placed in Blender to obtain a first simulation scene.

[0015] The embodiment of the application can realize semantic-to-geometric mapping and complete preliminary scene layout following spatial relationship constraints by analyzing and converting the semantic relationships in the scene graph into specific geometric constraints and then placing them in Blender.

[0016] Further, the first simulation scene is input into a pre-trained optimization agent, so that the optimization agent performs collision detection and visual comparison on the first simulation scene, and performs layout optimization according to the detection and comparison results to obtain a simulation scene, including: All bounding boxes with intersections in the first simulation scene are obtained, and a first layout optimization is performed on the three-dimensional model corresponding to each bounding box to obtain a second simulation scene. render the second simulation scene from the same perspective based on the camera pose parameters of the RGB image, and visually compare the rendering result with the RGB image, and perform second layout optimization on each three-dimensional model in the second simulation scene according to the comparison result, to obtain a simulation scene.

[0017] The embodiment of the present application can solve the object collision problem and improve the visual consistency of the reconstructed scene and the input image by sequentially performing physical optimization of collision detection and visual optimization based on rendering visual comparison, to obtain a physically reasonable and visually realistic simulation scene.

[0018] Further, after the first simulation scene is input to the pre-trained optimization agent to enable the optimization agent to perform collision detection and visual comparison on the first simulation scene, and to perform layout optimization according to the detection and comparison results to obtain a simulation scene, the method further comprises: inputting the simulation scene to a pre-trained automated scene editing agent to enable the automated scene editing agent to extract object information of the simulation scene, and to generate an adjustment strategy based on the object information through an LLM model; iteratively adjusting and rendering the simulation scene based on the adjustment strategy, and evaluating the simulation scene generated in each iteration through the LLM model, and outputting the simulation scene generated in the current iteration when the evaluation result meets the layout target corresponding to the adjustment strategy.

[0019] The embodiment of the present application can automatically generate diversified scene variants by introducing an LLM-driven automated scene editing agent for iterative adjustment and evaluation, to support efficient construction of large-scale simulation data.

[0020] Another embodiment of the present application also provides a single-view simulation scene generation device based on multiple agents, comprising a perception agent module, a relationship inference agent module, a retrieval agent module, a layout agent module and an optimization agent module. The perception agent module is configured to input a single indoor RGB image to a pre-trained perception agent to enable the perception agent to identify all objects in the RGB image, and to position and calculate the scale of each object, and to output perception data of each object; wherein the perception data comprises a description text, a bounding box, a mask, relative scale and pose information of each object. The relationship inference agent module is configured to input the RGB image, the bounding box and the mask to a pre-trained relationship inference agent to enable the relationship inference agent to infer the spatial relationship of each object in the RGB image, and to construct a scene graph. The retrieval agent module is configured to input each piece of the perception data into a pre-trained retrieval agent, so that the retrieval agent retrieves a three-dimensional model corresponding to each object from a three-dimensional model data set according to each piece of the perception data and performs scale constraint to obtain a three-dimensional model set. The layout agent module is configured to input the scene graph, the three-dimensional model set and the pose information of each object into a pre-trained layout agent, so that the layout agent places each three-dimensional model in the three-dimensional model set according to the scene graph and the pose information of each object to obtain a first simulation scene. The optimization agent module is configured to input the first simulation scene into a pre-trained optimization agent, so that the optimization agent performs collision detection and visual comparison on the first simulation scene, and performs layout optimization according to the detection and comparison results to obtain a simulation scene.

[0021] Another embodiment of the present application also provides a terminal device, comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, when the processor executes the computer program, the steps of the method for generating a single-view simulation scene based on multiple agents are implemented.

[0022] Another embodiment of the present application also provides a computer-readable storage medium item, comprising a stored computer program, when the computer program runs, the device where the computer-readable storage medium is located is controlled to execute the steps of the method for generating a single-view simulation scene based on multiple agents. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 A flowchart of an embodiment of the method for generating a single-view simulation scene based on multiple agents provided by the present application; Figure 2 A structural diagram of an embodiment of the device for generating a single-view simulation scene based on multiple agents provided by the present application. DETAILED DESCRIPTION

[0024] To make the purposes, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this application; the use of the terms "including," "comprising," or "having" and variations thereof herein is intended to be broad and encompass the terms "consisting of" and "consisting essentially of" and variations thereof. Unless otherwise required by context, singular terms shall include pluralities and vice versa.

[0026] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise explicitly and specifically limited.

[0027] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily independent or alternative embodiments to each other. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0028] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0029] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two), and similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0030] Reference Figure 1 To solve the problems of high data cost and lack of physical consistency constraints in the prior art, an embodiment of the present application provides a single-view simulation scene generation method based on multi-agent, comprising steps S101 to S105: Step S101, inputting a single indoor RGB image to a pre-trained perception agent, so that the perception agent identifies all objects in the RGB image, and performs positioning and scale calculation on each object, and outputs perception data of each object; wherein the perception data includes description text, bounding box, mask, relative scale and pose information of each object.

[0031] It should be noted that inputting a single indoor RGB image into a pre-trained perceptual agent, enabling the agent to identify all objects in the RGB image, locate and scale each object, and output perceptual data for each object, refers to providing a real indoor RGB image. and room space constraints For example, given an image with a resolution of Meeting room scene images The dimensional constraints of the meeting room are: ,image It can cover object information related to the entire conference room. It identifies images through a sensing agent. The system collects all objects, such as chairs and tables, and obtains text prompts that describe the category, appearance, and shape of each object, as well as the position and orientation information of each object.

[0032] Preferably, the step of inputting a single indoor RGB image into a pre-trained perceptual agent, enabling the perceptual agent to identify all objects in the RGB image, locate and scale each object, and output perceptual data for each object, includes: First, the GPT-4o model is used to identify all objects in the RGB image and outputs descriptive text corresponding to each object.

[0033] Specifically, the GPT-4o model is a visual language model capable of recognizing images. Find all objects in the list and output the description text for each object. , description text It covers information such as the category, appearance, and shape of the corresponding object.

[0034] Second, the bounding boxes and masks of each object are located using the GroundedSAM-v2 model.

[0035] Specifically, GroundedSAM-v2 is a computer vision model that combines object detection and instance segmentation, enabling it to locate images. bounding box of each object instance and mask .

[0036] Third, the depth map of the RGB image is estimated using the MoGe-2 model, and the scene point cloud of the RGB image is calculated based on the camera intrinsics and the depth map; wherein, the camera intrinsics are the internal parameters of the camera that captured the RGB image.

[0037] Specifically, the MoGe-2 model is a monocular depth estimation model capable of estimating image depth. Depth map And based on depth map Calculated scene point cloud The specific process is as follows: ; ; in, The model is MoGe-2. For camera internal parameters; For scene points.

[0038] Fourth, based on the bounding boxes and masks of each object, the point clouds corresponding to each object are separated from the scene point cloud, and the relative scale of each object is determined based on the point clouds of each object.

[0039] Specifically, by using the object's bounding box and mask, the point cloud belonging to the object is extracted from the entire scene point cloud, thereby determining its relative size. The point cloud extraction process for each object is shown below: ; in, For the first Point cloud of an object; and The first The mask and bounding box of an object; This is a projection function used to project points in a 3D scene. Projected back onto the two-dimensional image plane.

[0040] Fifth, the pose information of each object is determined by the FoundationPose model, and the description text, bounding box, mask, relative scale and pose information corresponding to the same object are merged into a single perceptual data.

[0041] Specifically, the FoundationPose model is a 6D object pose estimation model that can estimate pose from images. The pose information of each object in three-dimensional space is identified and segmented.

[0042] Step S102: Input the RGB image, bounding box, and mask into the pre-trained relational reasoning agent so that the relational reasoning agent can infer the spatial relationships of the objects in the RGB image and construct a scene graph.

[0043] It should be noted that inputting the RGB image, bounding boxes, and mask into a pre-trained relational reasoning agent, so that the agent can infer the spatial relationships between the objects in the RGB image and construct a scene graph, refers to using the bounding boxes obtained during the perception phase. and a mask , the agent infers the image by relational reasoning The structured spatial relationship between each object in the image, such as a table leaning against a wall, etc.

[0044] Preferably, the input of the RGB image, the bounding box and the mask to the pre-trained relational reasoning agent enables the relational reasoning agent to infer the spatial relationship of each object in the RGB image and construct a scene graph, which comprises: First, based on the bounding box and the mask of each object, the spatial relationship of each object in the RGB image is inferred by a GPT-4o model through multiple rounds of reasoning, and the spatial relationship appearing in the multiple rounds of reasoning results with a frequency exceeding a preset threshold is retained to obtain a spatial relationship set.

[0045] Specifically, to ensure the accuracy of the relationship, a statistical integration method is used to aggregate the results of multiple reasoning cycles, and the spatial relationship appearing in the multiple rounds of reasoning results with a frequency exceeding a preset threshold is retained as the final effective relationship.

[0046] Second, based on the spatial relationship set, a scene graph is constructed.

[0047] Specifically, the scene graph Can be saved in JSON format.

[0048] Step S103, input each of the perception data to the pre-trained retrieval agent to enable the retrieval agent to retrieve the three-dimensional model corresponding to each of the objects from the three-dimensional model data set according to each of the perception data and perform scale constraint to obtain a three-dimensional model set.

[0049] It should be noted that inputting each of the perception data to the pre-trained retrieval agent to enable the retrieval agent to retrieve the three-dimensional model corresponding to each of the objects from the three-dimensional model data set according to each of the perception data and perform scale constraint to obtain a three-dimensional model set means that based on the description text related to the object , the agent filters out a candidate set for each object from the three-dimensional model data set, and uses the mask The object three-dimensional model most consistent with the target features is filtered out from the candidate set .

[0050] Preferably, the input of each of the perception data to the pre-trained retrieval agent to enable the retrieval agent to retrieve the three-dimensional model corresponding to each of the objects from the three-dimensional model data set according to each of the perception data and perform scale constraint to obtain a three-dimensional model set comprises: First, the description text of each object is feature-encoded by a CLIP model to obtain each text feature.

[0051] Specifically, the CLIP model is a multi-modal model capable of encoding each description text into a feature space.

[0052] Secondly, based on each text feature, a three-dimensional model corresponding to each object is retrieved from the Objavers dataset and the 3D-FRONT dataset through an OpenShape model to obtain a first three-dimensional model set.

[0053] Specifically, the OpenShape model is a 3D shape retrieval model capable of retrieving a candidate three-dimensional model of a corresponding category from the two 3D model datasets of Objavers and 3D-FRONT based on each text feature.

[0054] Thirdly, each three-dimensional model in the first three-dimensional model set is appearance-matched according to the mask of each object to obtain a second three-dimensional model set.

[0055] Specifically, the mask of each object can be used to filter the candidate three-dimensional model and identify the three-dimensional model that best matches the target object.

[0056] Fourthly, each three-dimensional model in the second three-dimensional model set is scale-constrained based on the relative scale and pose information of each object to obtain a three-dimensional model set.

[0057] Specifically, each three-dimensional model can be size-matched according to the relative scale, and the size of the three-dimensional model that does not match can be scaled; each three-dimensional model can be pose-matched according to the pose information, and the pose of the three-dimensional model that does not match can be moved.

[0058] In step S104, the scene graph, the three-dimensional model set, and the pose information of each object are input to a pre-trained layout agent, so that the layout agent places each three-dimensional model in the three-dimensional model set according to the scene graph and the pose information of each object to obtain a first simulation scene.

[0059] It should be noted that inputting the scene graph, the three-dimensional model set, and the pose information of each object to the pre-trained layout agent so that the layout agent places each three-dimensional model in the three-dimensional model set according to the scene graph and the pose information of each object to obtain a first simulation scene means that based on the pose information obtained in the perception stage and the scene graph describing the relationship between each object, the three-dimensional model of each object is filtered through the layout agent and placed in the corresponding reconstructed scene.

[0060] ​​Preferably, the scene graph, the set of three-dimensional models, and the pose information of each object are input into a pre-trained layout agent, so that the layout agent places each three-dimensional model in the set of three-dimensional models according to the scene graph and the pose information of each object to obtain a first simulation scene, including: First, all semantic relationships in the scene graph are analyzed, and each semantic relationship is converted into a geometric constraint.

[0061] Specifically, the geometric constraint can be adjacency, containment, or alignment with a wall, etc.

[0062] Second, based on each geometric constraint and the pose information of each object, each three-dimensional model in the set of three-dimensional models is placed in Blender to obtain a first simulation scene.

[0063] Specifically, Blender is a three-dimensional computer graphics software that can build an interactive 3D scene, and each three-dimensional model can be placed in the 3D scene built by Blender based on geometric constraints and pose information.

[0064] Step S105, input the first simulation scene into a pre-trained optimization agent, so that the optimization agent performs collision detection and visual comparison on the first simulation scene, and performs layout optimization according to the detection and comparison results to obtain a simulation scene.

[0065] It should be noted that inputting the first simulation scene into a pre-trained optimization agent, so that the optimization agent performs collision detection and visual comparison on the first simulation scene, and performs layout optimization according to the detection and comparison results to obtain a simulation scene means that the optimization agent performs collision detection on each three-dimensional model in the simulation scene, and fine-tunes the three-dimensional models that collide; at the same time, the simulation scene is rendered from the same perspective as the image , the rendering result is visually compared with the image , and the relationship layout in the simulation scene is optimized according to the comparison result.

[0066] Preferably, the first simulation scene is input into a pre-trained optimization agent, so that the optimization agent performs collision detection and visual comparison on the first simulation scene, and performs layout optimization according to the detection and comparison results to obtain a simulation scene, including: First, obtain all the bounding boxes that exist in the intersection in the first simulation scene, and perform first layout optimization on the three-dimensional models corresponding to each bounding box to obtain a second simulation scene; Specifically, collision detection is performed by checking the intersection between object bounding boxes, and then iterative adjustment is performed to solve the overlapping problem while not violating the relationship constraints.

[0067] Second, based on the camera pose parameters of the RGB image, the second simulation scene is rendered from the same perspective, and the rendering result is visually compared with the RGB image. According to the comparison result, the second layout optimization is performed on each three-dimensional model in the second simulation scene, and a simulation scene is obtained.

[0068] Specifically, the reconstructed Blender scene is rendered from the same perspective using the camera pose parameters estimated from the RGB image, and then the rendered image is visually compared with the RGB image to generate correction feedback about the appearance and spatial arrangement of the objects, which drives the iterative optimization process.

[0069] Preferably, after the first simulation scene is input into the pre-trained optimization agent to enable the optimization agent to perform collision detection and visual comparison on the first simulation scene, and to perform layout optimization according to the detection and comparison results to obtain a simulation scene, the method further comprises: inputting the simulation scene into a pre-trained automated scene editing agent to enable the automated scene editing agent to extract object information of the simulation scene, and to generate an adjustment strategy based on the object information through an LLM model; adjusting and rendering the simulation scene based on the adjustment strategy, and evaluating the simulation scene generated in each iteration through the LLM model, and outputting the simulation scene generated in the current iteration when the evaluation result meets the layout target corresponding to the adjustment strategy.

[0070] It should be noted that the automated scene editing agent can adjust the spatial relationship of the three-dimensional models under physical reasonable constraints. Within a specified area, the automated scene editing agent can move, rotate or rearrange the three-dimensional models while maintaining collision-free placement and relationship semantics. By iteratively applying these adjustments, multiple physically valid scene variants can be generated from a single RGB image, thereby efficiently creating a diverse training dataset without the need for repeated reconstruction processes or manual annotation.

[0071] Specifically, using the bpy library of Python, object information such as category, pose, size and text attributes is extracted from the current simulation scene, and the object information is encoded into JSON format for communication with a large language model (LLM). Under the guidance of spatial constraints such as collision avoidance and adjacency rules, the LLM model generates an adjustment strategy for the position and direction of the three-dimensional models. Based on the adjustment strategy, the simulation scene is iteratively adjusted and rendered, and the simulation scene generated in each iteration is evaluated through the LLM model until all layout targets are met.

[0072] It should be noted that an interactive scene editing interface can be constructed based on a model context protocol (MCP) of Blender, which allows a user to manually adjust object positions, directions and attributes while receiving real-time visual feedback.

[0073] Embodiments of the present application can provide multi-modal representations of objects for subsequent three-dimensional reconstruction by identifying objects from a single RGB image and outputting perception data containing geometric scales and poses by a perception agent; can parse and structure the spatial positional relationships within a scene to provide semantic constraints for scene layout by constructing a scene graph based on visual information to infer spatial relationships of objects by a relationship reasoning agent; can match object models from a database that match the target appearance and size to ensure the geometric accuracy of reconstructed objects by retrieving three-dimensional models and performing scale constraints according to the perception data by a retrieval agent; can convert relationship semantics into geometric constraints to complete the preliminary construction of the scene by placing three-dimensional models according to the scene graph and pose information by a layout agent; can improve the physical reasonableness and layout consistency of the scene by performing collision detection and visual comparison optimization on the preliminary scene by an optimization agent.

[0074] Optionally, in embodiments of the present application, the single indoor RGB image is input into a pre-trained perception agent, so that the perception agent identifies all objects in the RGB image, and locates and calculates the scales of each object, and outputs perception data of each object, including: identifying all objects in the RGB image by a GPT-4o model and outputting corresponding description texts of each object; locating the bounding boxes and masks of each object by a GroundedSAM-v2 model; estimating the depth map of the RGB image by a MoGe-2 model, and calculating the scene point cloud of the RGB image based on the camera intrinsic parameters and the depth map; wherein the camera intrinsic parameters are the internal parameters of the camera that captures the RGB image; separating the point cloud corresponding to each object from the scene point cloud based on the bounding boxes and masks of each object, and determining the relative scales of each object based on the point cloud of each object; determining the pose information of each object by a FoundationPose model, and merging the description text, bounding box, mask, relative scale and pose information corresponding to the same object into a perception data.

[0075] The embodiment of the application can accurately obtain the description text, metric scale and spatial pose of the objects in the scene by adopting a series of pre-trained models such as GPT-4o, GroundedSAM-v2, MoGe-2 and FoundationPose to cooperatively complete recognition, segmentation, depth estimation and pose determination, and form complete structured perception data.

[0076] Optionally, in the embodiment of the application, the inputting the RGB image, the bounding box and the mask into the pre-trained relationship inference agent to make the relationship inference agent infer the spatial relationship of each object in the RGB image and construct a scene graph comprises: Based on the bounding box and the mask of each object, the spatial relationship of each object in the RGB image is inferred through the GPT-4o model in multiple rounds, and the spatial relationship appearing in the multiple-round inference result with a frequency exceeding a preset threshold is retained to obtain a spatial relationship set; The scene graph is constructed based on the spatial relationship set.

[0077] The embodiment of the application can ensure the accuracy and reliability of the spatial relationship in the constructed scene graph by performing multiple rounds of relationship inference based on the bounding box and the mask and retaining high-frequency relationships, and avoid errors that may occur in single inference.

[0078] Optionally, in the embodiment of the application, the inputting each perception data into the pre-trained retrieval agent to make the retrieval agent retrieve the three-dimensional model corresponding to each object from the three-dimensional model data set according to each perception data and perform scale constraint to obtain a three-dimensional model set comprises: The description text of each object is feature-encoded through the CLIP model to obtain each text feature; Based on each text feature, the three-dimensional model corresponding to each object is retrieved from the Objavers data set and the 3D-FRONT data set through the OpenShape model to obtain a first three-dimensional model set; According to the mask of each object, each three-dimensional model in the first three-dimensional model set is appearance-matched to obtain a second three-dimensional model set; Based on the relative scale and pose information of each object, each three-dimensional model in the second three-dimensional model set is scale-constrained to obtain a three-dimensional model set.

[0079] The embodiment of the application can gradually filter out the three-dimensional model highly matched with the target object in semantics, appearance and size by sequentially performing the three-stage retrieval process of text feature retrieval, visual appearance matching and geometric scale constraint, and ensure the quality of the retrieval result.

[0080] Optionally, in the embodiment of the present application, the inputting the scene graph, the set of three-dimensional models and the pose information of each object into a pre-trained layout agent, so that the layout agent places each three-dimensional model in the set of three-dimensional models according to the scene graph and the pose information of each object, to obtain a first simulation scene, comprises: analyzing all semantic relationships in the scene graph, and converting each semantic relationship into a geometric constraint; placing each three-dimensional model in the set of three-dimensional models in Blender based on each geometric constraint and the pose information of each object, to obtain a first simulation scene.

[0081] The embodiment of the present application can realize semantic-to-geometric mapping and complete preliminary scene layout following spatial relationship constraints by analyzing and converting semantic relationships in the scene graph into specific geometric constraints and placing them in Blender.

[0082] Optionally, in the embodiment of the present application, the inputting the first simulation scene into a pre-trained optimization agent, so that the optimization agent performs collision detection and visual comparison on the first simulation scene, and performs layout optimization according to the detection and comparison results, to obtain a simulation scene, comprises: obtaining all intersection boundary boxes in the first simulation scene, and performing first layout optimization on the three-dimensional models corresponding to each boundary box, to obtain a second simulation scene; rendering the second simulation scene from the same perspective based on the camera pose parameters of the RGB image, and performing visual comparison between the rendering result and the RGB image, and performing second layout optimization on each three-dimensional model in the second simulation scene according to the comparison result, to obtain a simulation scene.

[0083] The embodiment of the present application can solve the object collision problem and improve the visual consistency of the reconstructed scene and the input image by sequentially performing physical optimization of collision detection and visual optimization based on rendering visual comparison, to obtain a physically reasonable and visually realistic simulation scene.

[0084] Optionally, in the embodiment of the present application, after the inputting the first simulation scene into a pre-trained optimization agent, so that the optimization agent performs collision detection and visual comparison on the first simulation scene, and performs layout optimization according to the detection and comparison results, to obtain a simulation scene, further comprises: inputting the simulation scene into a pre-trained automated scene editing agent, so that the automated scene editing agent extracts object information of the simulation scene, and generates an adjustment strategy based on the object information through an LLM model; The simulation scene is iteratively adjusted and rendered based on the adjustment strategy, and the simulation scene generated in each iteration is evaluated by the LLM model. When the evaluation result satisfies the layout target corresponding to the adjustment strategy, the simulation scene generated in the current iteration is output.

[0085] The embodiments of the present invention introduce an automated scene editing agent driven by LLM for iterative adjustment and evaluation, which can automatically generate diverse scene variants and support the efficient construction of large-scale simulation data.

[0086] like Figure 2 As shown, based on the above method embodiments, corresponding apparatus embodiments are provided; One embodiment of the present invention provides a single-view simulation scene generation device based on multiple agents, including: a perception agent module 201, a relation reasoning agent module 202, a retrieval agent module 203, a layout agent module 204, and an optimization agent module 205; The perception agent module 201 is used to input a single indoor RGB image into a pre-trained perception agent, so that the perception agent can identify all objects in the RGB image, locate and calculate the scale of each object, and output perception data of each object; wherein, the perception data includes descriptive text, bounding box, mask, relative scale and pose information of each object. The relation reasoning agent module 202 is used to input the RGB image, bounding box and mask into the pre-trained relation reasoning agent, so that the relation reasoning agent can infer the spatial relationship of each object in the RGB image and construct a scene graph; The retrieval agent module 203 is used to input the perception data into the pre-trained retrieval agent, so that the retrieval agent can retrieve the three-dimensional model corresponding to each object from the three-dimensional model dataset based on the perception data and perform scale constraints to obtain a set of three-dimensional models. The layout agent module 204 is used to input the scene map, the three-dimensional model set and the pose information of each object into the pre-trained layout agent, so that the layout agent places each three-dimensional model in the three-dimensional model set according to the scene map and the pose information of each object to obtain the first simulation scene. The optimized agent module 205 is used to input the first simulation scene into the pre-trained optimized agent, so that the optimized agent performs collision detection and visual comparison on the first simulation scene, and performs layout optimization based on the detection and comparison results to obtain the simulation scene.

[0087] Optionally, in the embodiment of the present application, the perception intelligent agent module 201 comprises an object recognition submodule, an object positioning submodule, a scene point cloud submodule, a relative scale submodule and an object position submodule. The object recognition submodule is configured to identify all objects in the RGB image by a GPT-4o model and output a description text corresponding to each object. The object positioning submodule is configured to locate a bounding box and a mask of each object by a GroundedSAM-v2 model. The scene point cloud submodule is configured to estimate a depth map of the RGB image by a MoGe-2 model, and calculate a scene point cloud of the RGB image based on a camera intrinsic parameter and the depth map, wherein the camera intrinsic parameter is an internal parameter of a camera for shooting the RGB image. The relative scale submodule is configured to separate a point cloud corresponding to each object from the scene point cloud based on the bounding box and the mask of each object, and determine a relative scale of each object based on the point cloud of each object. The object position submodule is configured to determine pose information of each object by a FoundationPose model, and combine the description text, the bounding box, the mask, the relative scale and the pose information of the same object into one perception data.

[0088] In the embodiment of the present application, a series of pre-trained models such as GPT-4o, GroundedSAM-v2, MoGe-2 and FoundationPose are used to cooperatively complete identification, segmentation, depth estimation and pose determination, so that the description text, the metric scale and the spatial pose of the object in the scene can be accurately obtained, and complete structured perception data is formed.

[0089] Optionally, in the embodiment of the present application, the relationship reasoning intelligent agent module 202 comprises a relationship reasoning submodule and a scene graph construction submodule. The relationship reasoning submodule is configured to perform multi-round reasoning on the spatial relationship of each object in the RGB image by a GPT-4o model based on the bounding box and the mask of each object, and retain a spatial relationship appearing with a frequency higher than a preset threshold in the multi-round reasoning result to obtain a spatial relationship set. The scene graph construction submodule is configured to construct a scene graph based on the spatial relationship set.

[0090] In the embodiment of the present application, multi-round relationship reasoning based on the bounding box and the mask is performed, and high-frequency relationships are retained, so that the accuracy and reliability of the spatial relationship in the constructed scene graph can be ensured, and errors possibly occurring in single reasoning can be avoided.

[0091] Optionally, in the embodiment of the application, the retrieval agent module 203 comprises a feature coding sub-module, a model retrieval sub-module, an appearance matching sub-module and a scale constraint sub-module. The feature coding sub-module is configured to code the description text of each object by a CLIP model to obtain a text feature of each object. The model retrieval sub-module is configured to retrieve a three-dimensional model corresponding to each object from an Objavers dataset and a 3D-FRONT dataset based on the text feature of each object by an OpenShape model to obtain a first three-dimensional model set. The appearance matching sub-module is configured to perform appearance matching on each three-dimensional model in the first three-dimensional model set according to the mask of each object to obtain a second three-dimensional model set. The scale constraint sub-module is configured to perform scale constraint on each three-dimensional model in the second three-dimensional model set based on the relative scale and pose information of each object to obtain a three-dimensional model set.

[0092] The embodiment of the application can gradually filter out three-dimensional models that are highly matched with the target object in terms of semantics, appearance and size by sequentially performing the three-stage retrieval process of text feature retrieval, visual appearance matching and geometric scale constraint, thereby ensuring the quality of the retrieval result.

[0093] Optionally, in the embodiment of the application, the layout agent module 204 comprises a geometric constraint sub-module and a model placement sub-module. The geometric constraint sub-module is configured to analyze all semantic relationships in the scene graph and convert each semantic relationship into a geometric constraint. The model placement sub-module is configured to place each three-dimensional model in the three-dimensional model set in Blender based on each geometric constraint and the pose information of each object to obtain a first simulation scene.

[0094] The embodiment of the application can realize the mapping from semantics to geometry and complete the preliminary scene layout following the spatial relationship constraint by analyzing and converting the semantic relationship in the scene graph into specific geometric constraints and then placing them in Blender.

[0095] Optionally, in the embodiment of the application, the optimization agent module 205 comprises a collision detection sub-module and a visual contrast sub-module. The collision detection sub-module is configured to obtain all bounding boxes with intersections in the first simulation scene and perform first layout optimization on the three-dimensional model corresponding to each bounding box to obtain a second simulation scene. The visual comparison submodule is configured to render the second simulation scene from the same perspective based on camera pose parameters of the RGB image, compare the rendering result with the RGB image, and perform second layout optimization on each three-dimensional model in the second simulation scene according to a comparison result, to obtain a simulation scene.

[0096] The embodiment of the present application can solve the object collision problem and improve the visual consistency of the reconstructed scene and the input image by sequentially performing physical optimization of collision detection and visual optimization based on rendering visual comparison, to obtain a physically reasonable and visually realistic simulation scene.

[0097] Optionally, in the embodiment of the present application, the optimization intelligent agent module 205 is followed by an adjustment strategy submodule and an adjustment evaluation submodule. The adjustment strategy submodule is configured to input the simulation scene into a pre-trained automated scene editing intelligent agent, to enable the automated scene editing intelligent agent to extract object information of the simulation scene, and to generate an adjustment strategy based on the object information through an LLM model. The adjustment evaluation submodule is configured to perform iterative adjustment and rendering on the simulation scene based on the adjustment strategy, and to evaluate the simulation scene generated in each iteration through the LLM model, and when the evaluation result meets the layout target corresponding to the adjustment strategy, output the simulation scene generated in the current iteration.

[0098] The embodiment of the present application can automatically generate diversified scene variants by introducing an LLM-driven automated scene editing intelligent agent for iterative adjustment and evaluation, to support efficient construction of large-scale simulation data.

[0099] It can be understood that the above-mentioned device item embodiment corresponds to the method item embodiment of the present application, and can realize the method of generating a single-view simulation scene based on multiple intelligent agents provided by any one of the above-mentioned method item embodiments of the present application.

[0100] The embodiment of the present application can recognize objects from a single RGB image and output perception data containing geometric scales and poses through the perception agent module 201, can provide multi-modal representations of objects for subsequent three-dimensional reconstruction; can parse and structure the spatial position relationship in the scene to provide semantic constraints for scene layout through the relationship reasoning agent module 202 based on visual information to infer the spatial relationship of objects to construct a scene graph; can match object models that match the target appearance and size from the database to ensure the geometric accuracy of the reconstructed objects through the retrieval agent module 203 according to the perception data to retrieve three-dimensional models and perform scale constraints; can convert relationship semantics into geometric constraints to complete the preliminary construction of the scene through the layout agent module 204 according to the scene graph and pose information to preliminarily place three-dimensional models; and can improve the physical rationality and layout consistency of the scene through the optimization agent module 205 to perform collision detection and visual comparison optimization on the preliminary scene.

[0101] It should be noted that the apparatus embodiments described above are only schematic, and some or all of the modules thereof can be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the apparatus embodiment provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement it without creative labor.

[0102] On the basis of the above-mentioned embodiment of the single-view simulation scene generation method based on multiple agents, another embodiment of the present application provides a terminal device, which comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, a single-view simulation scene generation method based on multiple agents according to any one of the embodiments of the present application is realized.

[0103] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present application. The one or more modules can be a series of computer program instructions that can complete a specific function, which are used to describe the execution process of the computer program in the terminal device.

[0104] The terminal device can be a desktop computer, a notebook computer, a palm computer and a cloud server, etc. The terminal device can include, but is not limited to, a processor and a memory.

[0105] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like, which is the control center of the terminal device and connects all parts of the terminal device through various interfaces and lines.

[0106] On the basis of the above-mentioned method embodiment, another embodiment of the present application provides a computer readable storage medium, including a stored computer program, wherein when the computer program runs, the device where the computer readable storage medium is located executes the method of any one of the above-mentioned method embodiments of the present application.

[0107] The modules / units integrated in the device / terminal equipment, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of each method embodiment can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0108] The above-mentioned is the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principle of the present application, a number of improvements and refinements can be made, and these improvements and refinements are also considered to be within the protection scope of the present application.

Claims

1. A multi-agent based single-view simulation scene generation method, characterized in that, The method comprises the following steps: inputting an indoor single RGB image into a pre-trained perception agent to enable the perception agent to identify all objects in the RGB image and perform positioning and scale calculation on each object, and output perception data of each object; wherein the perception data includes description text, bounding box, mask, relative scale and pose information of each object; inputting the RGB image, bounding box and mask into a pre-trained relationship reasoning agent to enable the relationship reasoning agent to infer the spatial relationship of each object in the RGB image and construct a scene graph; inputting each perception data into a pre-trained retrieval agent to enable the retrieval agent to retrieve the corresponding three-dimensional model of each object from a three-dimensional model dataset according to each perception data and perform scale constraint to obtain a three-dimensional model set; inputting the scene graph, the three-dimensional model set and the pose information of each object into a pre-trained layout agent to enable the layout agent to place each three-dimensional model in the three-dimensional model set according to the scene graph and the pose information of each object to obtain a first simulation scene; inputting the first simulation scene into a pre-trained optimization agent to enable the optimization agent to perform collision detection and visual comparison on the first simulation scene and perform layout optimization according to the detection and comparison results to obtain a simulation scene.

2. The multi-agent based single-view simulated scene generation method of claim 1, wherein, The method of inputting an indoor single RGB image into a pre-trained perception agent to enable the perception agent to identify all objects in the RGB image and perform positioning and scale calculation on each object, and output perception data of each object, comprises: identifying all objects in the RGB image by a GPT-4o model and outputting the corresponding description text of each object; positioning the bounding box and mask of each object by a GroundedSAM-v2 model; estimating the depth map of the RGB image by a MoGe-2 model, and calculating the scene point cloud of the RGB image based on the camera intrinsic parameter and the depth map; wherein the camera intrinsic parameter is the internal parameter of the camera shooting the RGB image; separating the point cloud corresponding to each object from the scene point cloud based on the bounding box and mask of each object, and determining the relative scale of each object based on the point cloud of each object; determining the pose information of each object by a FoundationPose model, and merging the description text, bounding box, mask, relative scale and pose information of the same object into one perception data.

3. The multi-agent based single-view simulated scene generation method of claim 1, wherein, The method of inputting the RGB image, bounding box and mask into a pre-trained relationship reasoning agent to enable the relationship reasoning agent to infer the spatial relationship of each object in the RGB image and construct a scene graph, comprises: performing multi-round reasoning on the spatial relationship of each object in the RGB image based on the bounding box and mask of each object by a GPT-4o model, and retaining the spatial relationship appearing in the multi-round reasoning result more than a preset threshold to obtain a spatial relationship set; constructing a scene graph based on the spatial relationship set.

4. The multi-agent based single-view simulated scene generation method of claim 1, wherein, The perception data is input into a pre-trained retrieval agent, so that the retrieval agent retrieves a three-dimensional model corresponding to each object from a three-dimensional model data set according to the perception data, and performs scale constraint to obtain a three-dimensional model set, including: The description text of each object is encoded by a CLIP model to obtain a text feature of each object; Based on the text feature of each object, a three-dimensional model corresponding to each object is retrieved from an Objavers data set and a 3D-FRONT data set by an OpenShape model to obtain a first three-dimensional model set; According to the mask of each object, each three-dimensional model in the first three-dimensional model set is matched in appearance to obtain a second three-dimensional model set; Based on the relative scale and pose information of each object, each three-dimensional model in the second three-dimensional model set is subjected to scale constraint to obtain a three-dimensional model set.

5. The multi-agent based single-view simulated scene generation method of claim 1, wherein, The scene graph, the three-dimensional model set, and the pose information of each object are input into a pre-trained layout agent, so that the layout agent places each three-dimensional model in the three-dimensional model set according to the scene graph and the pose information of each object to obtain a first simulation scene, including: All semantic relationships in the scene graph are analyzed, and each semantic relationship is converted into a geometric constraint; Based on each geometric constraint and the pose information of each object, each three-dimensional model in the three-dimensional model set is placed in Blender to obtain a first simulation scene.

6. The multi-agent based single-view simulated scene generation method of claim 1, wherein, The first simulation scene is input into a pre-trained optimization agent, so that the optimization agent performs collision detection and visual comparison on the first simulation scene, and performs layout optimization according to the detection and comparison results to obtain a simulation scene, including: All intersection bounding boxes in the first simulation scene are obtained, and a first layout optimization is performed on the three-dimensional model corresponding to each bounding box to obtain a second simulation scene; Based on the camera pose parameters of the RGB image, the second simulation scene is rendered from the same perspective, and the rendering result is compared with the RGB image, and a second layout optimization is performed on each three-dimensional model in the second simulation scene according to the comparison result to obtain a simulation scene.

7. The multi-agent based single-view simulated scene generation method of claim 1, wherein, After the first simulation scene is input into a pre-trained optimization agent, so that the optimization agent performs collision detection and visual comparison on the first simulation scene, and performs layout optimization according to the detection and comparison results to obtain a simulation scene, it further includes: The simulation scene is input into a pre-trained automated scene editing agent, so that the automated scene editing agent extracts object information of the simulation scene, and generates an adjustment strategy based on the object information through an LLM model; Based on the adjustment strategy, the simulation scene is iteratively adjusted and rendered, and the simulation scene generated in each iteration is evaluated by the LLM model, and when the evaluation result meets the layout target corresponding to the adjustment strategy, the simulation scene generated in the current iteration is output.

8. A multi-agent based single-view simulation scene generation apparatus, comprising: including: The perception agent module, the relationship reasoning agent module, the retrieval agent module, the layout agent module, and the optimization agent module; The perception agent module is configured to input a single indoor RGB image into a pre-trained perception agent, so that the perception agent identifies all objects in the RGB image, locates and calculates the scale of each object, and outputs perception data of each object; wherein the perception data includes a description text, a bounding box, a mask, relative scale and pose information of each object; The relationship reasoning agent module is configured to input the RGB image, the bounding box and the mask into a pre-trained relationship reasoning agent, so that the relationship reasoning agent infers the spatial relationship of each object in the RGB image and constructs a scene graph; The retrieval agent module is configured to input each perception data into a pre-trained retrieval agent, so that the retrieval agent retrieves a three-dimensional model corresponding to each object from a three-dimensional model data set according to each perception data and performs scale constraint to obtain a three-dimensional model set; The layout agent module is configured to input the scene graph, the three-dimensional model set and the pose information of each object into a pre-trained layout agent, so that the layout agent places each three-dimensional model in the three-dimensional model set according to the scene graph and the pose information of each object to obtain a first simulation scene; The optimization agent module is configured to input the first simulation scene into a pre-trained optimization agent, so that the optimization agent performs collision detection and visual comparison on the first simulation scene, and performs layout optimization according to the detection and comparison results to obtain a simulation scene.

9. A terminal device, comprising: A computer program product including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the multi-agent based single-view simulation scene generation method according to any one of claims 1-7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer program product includes: A stored computer program, wherein when the computer program is running, the device where the computer readable storage medium is located is controlled to execute the multi-agent based single-view simulation scene generation method according to any one of claims 1-7.

Images