A method and system for scene generation based on a physics engine and a large model

By combining a physics engine and a large model-based scene generation method, the semantic and physical feasibility issues of complex 3D scene generation in existing technologies are solved. Stability assessment and feedback correction are achieved, generating 3D scenes that are both semantically consistent and physically feasible, suitable for data acquisition for robot learning.

CN123116845APending Publication Date: 2026-11-17FEIJIE COSI INTELLIGENT TECHNOLOGY (SHANGHAI) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611399357.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-09
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies struggle to simultaneously satisfy semantic consistency and physical feasibility when generating complex 3D scenes, leading to issues such as overlapping, suspension, and insufficient support in the generated scenes. Furthermore, they lack the ability to control scene stability and style parameters, making them unsuitable for data acquisition in robot learning.

Method used

A scene generation method based on a physics engine and a large model is adopted. By structured parsing of the natural language description input by the user, combined with the 3D asset library and physical attribute parameters, spatial and physical predicates are generated. Two-dimensional layout optimization and 3D simulation verification are carried out, and iterative correction is performed until the quality conditions are met. Stability assessment and feedback correction are output.

Benefits of technology

The generated scenes can maintain an effective state under gravity, friction and contact constraints, improving the physical realism and executability of complex scenes, enhancing the controllable generation capability of scenes under open vocabulary conditions, improving generation efficiency and result stability, and providing the data interface required for robot simulation tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123116845A_ABST
    Figure CN123116845A_ABST
Patent Text Reader

Abstract

This invention discloses a scene generation method and system based on a physics engine and a large model, belonging to the field of 3D scene generation technology. The method includes the following steps: acquiring a natural language scene description input by the user, performing structured parsing, and forming a standardized semantic representation; retrieving corresponding object models from a 3D asset library based on the standardized semantic representation; using a large model to parse the standardized semantic representation, generating spatial and physical predicates, assigning parameters to the predicates, and arranging them into a structured constraint set; iteratively optimizing the initial 2D layout based on the object's 2D convex hull and constraint set; performing physical simulation and stability evaluation on the initial 2D layout to obtain a physically verified final 3D placement state; and performing quality evaluation and feedback correction on the final 3D placement state until the quality conditions are met, and outputting the result. This invention uses the above method to generate 3D scene results that combine semantic consistency, structural complexity, and physical feasibility.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the technical field of 3D scene generation, and in particular to a scene generation method and system based on a physics engine and a large model. Background Technology

[0002] In recent years, with the rapid development of embodied intelligence, robot operation learning, digital twins, and virtual training environments, the technology for automatically generating 3D scenes for complex tasks has gradually shifted from static display-type modeling to interactive and executable modeling. Traditional 3D scene construction mainly serves graphics rendering, game art, and digital content production, focusing on the scene's appearance realism, geometric integrity, and asset richness. However, for robot operation, scene reasoning, and task execution, mere visual appeal is far from sufficient. More importantly, the scene must satisfy physical laws such as contact, support, stacking, containment, balance, and boundary constraints; maintain stability in the simulation environment; and be directly invoked by downstream planning, control, and learning algorithms. Therefore, how to automatically generate complex 3D scenes that conform to semantic intent and meet physical feasibility requirements, starting from natural language descriptions, has become a crucial fundamental problem in the fields of intelligent simulation and robot data generation.

[0003] From a technological evolution perspective, this field has roughly gone through three stages: The first stage was mainly rule-driven and procedural modeling, relying on manual layout templates and collision rules, which limited scalability; the second stage shifted to data-driven methods, using large-scale indoor datasets or diffusion models to learn common layout patterns, but struggled to handle long-tail objects and complex stacking relationships; the third stage introduced large language models and visual language models, combining natural language understanding with scene planning, significantly enhancing the controllable generation capability under open vocabularies. However, application requirements have upgraded from "visualized scenes" to "operable / trainable scenes," and simple language models are insufficient to meet physical constraints such as stable support, reasonable contact, and container space. Therefore, scene generation technology is developing towards a comprehensive direction that integrates language understanding, asset retrieval, constraint solving, physical simulation, and feedback correction.

[0004] The existing technologies related to this invention mainly fall into three categories: The first category is scene generation based on program rules, which achieves clear results but relies on human experience and is difficult to cover complex stacking and containment relationships; the second category is scene layout technology based on data learning, which learns layout distribution through transformers or diffusion models, but the training data lacks fine-grained interaction relationships, making it prone to problems such as interleaving and instability; the third category is open-vocabulary scene generation based on large language models or visual language models, which uses language models to output object lists, spatial predicates, or placement suggestions, and then combines retrieval and solvers to recover the 3D scene, enhancing semantic expressive power. In addition, some technologies have begun to introduce physical priors, basic simulations, or probabilistic simulations to verify stability.

[0005] Existing technologies still suffer from the following shortcomings: First, most methods focus on "where to place" objects, but fail to systematically address "whether they can be stably placed there," lacking a three-dimensional constraint solution mechanism based on real physical processes, which easily leads to problems such as interleaving, suspension, and insufficient support. Second, there is a disconnect between the expressive power of open lexicalities and high-precision physical feasibility: large language models excel at semantic understanding but are weak in three-dimensional reasoning, while traditional solvers struggle to absorb fine-grained semantic requirements, and the semantic planning layer and physical execution layer lack unified predicate expression and closed-loop collaboration. Third, the ability to scale to complex scenes is insufficient; static collision detection easily fails as the number of objects increases, and there is a lack of refined geometric detection, failure attribution, and iterative correction mechanisms, resulting in low generation efficiency and unstable results. Fourth, it is difficult to achieve explicit control over scene stability and style parameters (such as crowding level, support ratio, etc.), and there is a lack of a process that incorporates factors such as mass, friction, and support area into a unified parametric modeling process. Fifth, there is a general lack of data closed loops for robot learning; the generated scenes have not undergone physical feasibility verification and are difficult to directly use for data collection for tasks such as grasping and handling. Therefore, developing a scene generation system that integrates large-scale model semantic planning, 3D asset retrieval, spatial predicate solving, physics engine simulation, and feedback correction has become an urgent problem to be solved in this field. Summary of the Invention

[0006] The purpose of this invention is to provide a scene generation method and system based on a physics engine and a large model, enabling the system to automatically complete scene element extraction, 3D asset matching, spatial and physical predicate generation, 2D spatial solution, 3D physical placement verification, stability assessment, and feedback iterative correction starting from the natural language description input by the user, thereby generating a 3D scene result that combines semantic consistency, structural complexity, and physical feasibility.

[0007] To achieve the above objectives, this invention provides a scene generation method based on a physics engine and a large model, comprising the following steps: S1. Obtain the natural language scene description input by the user, perform structured parsing, extract object category, quantity information, scene type, style requirements and control parameters, and form a standardized semantic representation that can be directly called in subsequent generation; S2. Based on the standardized semantic representation, retrieve the corresponding object model in the 3D asset library and supplement the physical attribute parameters for each target object model. The physical attribute parameters include model orientation, support probability, mass range, friction coefficient range and centroid offset range. S3. Utilize the large model to parse the standardized semantic representation, generate spatial predicates and physical predicates, assign parameters to the predicates and arrange them into a set of structured constraints; S4. Based on the two-dimensional convex hull and constraint set of the object, iterative optimization is performed to obtain a two-dimensional initial layout with no out-of-bounds and low overlap. S5. Perform physical simulation and stability evaluation on the initial two-dimensional layout to obtain the final three-dimensional placement state that has passed physical verification. S6. Perform quality assessment and feedback correction on the final 3D placement state. If the preset quality conditions are not met, generate natural language feedback information and input it into the large model for iterative correction until the quality conditions are met, obtain the final 3D scene, and output the scene geometry results, object parameters, evaluation indicators, and data interfaces for robot simulation tasks.

[0008] Preferably, step S1 specifically includes the following: receiving the instruction text, task supplementary information and control parameters given by the user on the interface, identifying the object category, quantifiers, functional requirements, relative position indicators and style constraint words in the text, further determining the scene type and support base type corresponding to the current instruction, and determining the support surface boundary, capacity constraint and scene template to be used first when solving the problem; The semantic information described above is mapped into structured fields, including object numbering rules, support surface size range, target stability level, compactness level, clutter level, and whether stacking, containment, and partial support are allowed. Finally, the results are organized into a unified scene semantic template, which is used as shared input for subsequent operations.

[0009] Preferably, step S2 specifically includes the following: based on the semantic information output in step S1, generating descriptive text for retrieval, establishing a mapping relationship between object number and descriptive text, retrieving the corresponding model in the local or cloud-based 3D asset library, and obtaining a candidate asset set; Candidate assets are reordered by combining text features, rendered view features, and geometric metadata. Further, attribute estimations are performed on the candidate assets, including supportability, forward orientation, mass range, friction range, and centroid offset range, to support subsequent 2D layout and 3D physics calculations. The final result is a collection of objects that meet the semantic requirements and have complete attribute fields.

[0010] Preferably, step S3 specifically includes the following: based on the scene semantic template and asset attributes, identify the geometric and physical relationships between objects; generate spatial predicates for two-dimensional position and orientation constraints for objects located on the support surface, including left, right, front, back, center alignment, boundary alignment, symmetrical arrangement, and relative orientation; generate physical predicates for objects with three-dimensional interaction requirements, including containment, stacking, support, and arbitrary feasible placement. Based on user input or prior experience, add numerical or hierarchical parameters to spatial and physical predicates, including object spacing, relative offset, support area ratio, stability target, compactness, and clutter. All predicates are uniformly arranged to establish object solving order, dependency relationships, and conflict handling rules, forming a structured set of constraints.

[0011] Preferably, step S4 specifically includes the following: receiving the spatial relationships and object attributes output in step S3, including object geometrical circumscribed information, support surface range, relative position, alignment relationship, rotation relationship, and grouping relationship; Generate a two-dimensional convex hull representation for each object as a collision approximation; Based on the object's solution state and predicate dependency, the distance, offset, anchor point position, and orientation parameters are iteratively optimized using the overlapping area of ​​the convex hull between objects and the distance from the object's center to the support boundary as penalty terms. The iteration results are checked for out-of-bounds and the congestion is evaluated. When the penalty value is lower than the preset threshold, the initial two-dimensional layout is output. When the preset number of iterations is reached and convergence is still not achieved, the failed objects, congested areas and conflict reasons are given as feedback information.

[0012] Preferably, step S5 specifically includes the following: performing occupancy meshing processing on the current scene and candidate objects to transform continuous geometry into a discrete occupancy representation that can be quickly traversed; A mesh search is performed on the surfaces that the candidate object may come into contact with to filter out candidate poses that are non-intersecting, have a sufficient contact area, and whose centroid projection falls inside the supporting convex hull. The physics engine is invoked to dynamically verify the candidate poses after screening. It detects whether the objects will slip, tip over, pop out of the container, or squeeze and intersect each other under the action of gravity, friction and contact. Candidate poses whose displacement changes are kept within the allowable range are determined as candidate poses that pass the dynamic verification. For candidate poses that pass dynamic verification, perturbation sampling is performed around position, attitude, mass, friction, and centroid offset. The proportion of poses that remain in balance after perturbation is counted to obtain a stability score. The verified candidate poses, contact states, and stability scores will be output for use by subsequent modules.

[0013] Preferably, step S6 specifically includes the following: performing a syntax and solution status check on the output of step S5 to determine whether there are problems such as predicate format errors, incomplete object solving, object interpenetration, exceeding the support boundary, infeasible stacking posture, or obvious instability after physical simulation, and recording the reasons for failure according to object granularity. Analyze the remaining available space inside the current support surface or container to identify vacant areas, overcrowded areas, and potential areas that can be filled further, providing environmental-level hints for subsequent large model corrections; The current scene is comprehensively scored based on indicators including scene stability, number of objects, surface coverage, compactness, neatness, clutter, and visual question-and-answer evaluation results. The above diagnostic results, empty area descriptions and comprehensive scores are transformed into natural language feedback that is friendly to large models, enabling large models to supplement objects, modify parameters or rewrite some predicates in a targeted manner. The final output of the scene results is in the form of a graphical interface or data interface, including a list of objects, pose parameters, physical properties, stability scores and solution logs, and provides standardized input for downstream simulation platforms, robot learning platforms or evaluation systems.

[0014] Preferably, in step S5, when the user's goal is to generate extreme placement cases, after perturbing and sampling the candidate poses, low-steady-state effective solutions are selected according to the principle of not collapsing but being closer to the critical state.

[0015] A scene generation system based on a physics engine and a large model includes a scene instruction acquisition unit, an asset retrieval and attribute modeling unit, a predicate generation and constraint arrangement unit, a spatial constraint solving unit, a physical simulation and stability evaluation unit, a feedback correction unit, and a scene display module. The scene instruction acquisition unit is used to acquire the natural language scene description input by the user, perform structured parsing on it, and generate scene semantic templates; The asset retrieval and attribute modeling unit is used to retrieve the 3D asset library based on the scene semantic template, obtain the corresponding 3D model and supplement the physical attributes to obtain a set of 3D objects with physical attributes. The predicate generation and constraint orchestration unit is used to generate spatial and physical predicates based on scene semantic templates and a set of 3D objects using a large model, and to assign constraint parameters to them to obtain a set of structured constraints. The spatial constraint solving unit is used to solve spatial constraints on the support surface based on the spatial predicates in the structured constraint set and the two-dimensional convex hull of the three-dimensional object, and generate a two-dimensional initial layout. The physical simulation and stability evaluation unit is used to perform physical simulation and stability evaluation on the two-dimensional initial layout based on the physical predicates in the structured constraint set and the physical properties of the three-dimensional objects, so as to obtain the three-dimensional scene placement state that has passed physical verification. The feedback correction unit is used to perform quality assessment and feedback correction on the placement state of the 3D scene. If the preset quality conditions are not met, natural language feedback information is generated and input into the large model for iterative correction until the quality conditions are met, and the final 3D scene is obtained. The scene display module is used to output scene geometry results, object parameters, evaluation metrics, and data interfaces for robot simulation tasks.

[0016] Therefore, the above-mentioned scene generation method and system based on a physics engine and a large model, adopted in this invention, has the following advantages: (1) In this invention, by introducing a 3D verification process driven by a physics engine after the 2D spatial layout, and combining support screening, contact checking and stability evaluation, the final pose of the object must simultaneously satisfy semantic relationships and physical laws. As a result, the generated scene no longer stays at the level of visual rationality, but can maintain an effective state under gravity, friction and contact constraints, which significantly improves the physical realism and executability of the complex scene generation results.

[0017] (2) In this invention, large model execution scene instruction parsing and predicate generation are introduced, enabling the system to understand the high-level task intent and fine-grained style requirements given by the user, and further map them into executable spatial and physical predicates. Users can not only specify "which objects to place", but also specify the relative position, support strength, stability, neatness, compactness and disorder between objects, which significantly enhances the scene controllable generation capability under open vocabulary conditions.

[0018] (3) In this invention, two-dimensional convex hull detection is used to replace simple axis-aligned bounding boxes, making collision judgment closer to the real outer contour of the object; at the same time, the parameters of each spatial predicate are gradually optimized through parameter iteration, and with the help of empty area recognition and crowding feedback, the failed areas are located and corrected. This not only takes into account geometric accuracy, but also avoids the high cost of directly performing full-scale three-dimensional mesh collision detection, which significantly improves the solution success rate and layout accuracy in high-density local scenes.

[0019] (4) In this invention, a feedback module covering syntax checking, failure attribution, empty area identification, quality evaluation and natural language writing is constructed. It can feed back the specific reasons for the failure to the large model, so that subsequent iterations have direction and purpose. This closed-loop structure significantly enhances the robustness of the system in complex scenarios and can gradually fill in the empty areas, optimize the stacking structure and reduce the risk of instability with each iteration, so that the final scenario is more complete, compact and natural.

[0020] (5) In this invention, the final pose of the object is given in the output stage, and asset identification, physical parameters, stability indicators and solution log are also output simultaneously, so as to facilitate direct reading by the capture planning, trajectory execution, reinforcement learning and imitation learning modules.

[0021] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a scene generation method based on a physics engine and a large model according to the present invention. Figure 2 This is a block diagram of the scene instruction acquisition unit of a scene generation system based on a physics engine and a large model according to the present invention. Figure 3 This is a block diagram illustrating the principle of the asset retrieval and attribute modeling unit in a scene generation system based on a physics engine and a large model, as described in this invention. Figure 4 This is a block diagram illustrating the predicate generation and constraint arrangement unit principle of a scene generation system based on a physics engine and a large model according to the present invention. Figure 5 This is a block diagram illustrating the principle of a spatial constraint solving unit in a scene generation system based on a physics engine and a large model, according to the present invention. Figure 6 This is a block diagram of the physical simulation and stability evaluation unit of a scene generation system based on a physics engine and a large model according to the present invention. Figure 7 This is a block diagram of the feedback correction unit and scene display principle of a scene generation system based on a physics engine and a large model according to the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Specific model specifications need to be selected and determined according to the actual specifications of the device, etc. The specific selection calculation method adopts existing technology in the art, and therefore will not be described in detail.

[0024] Example like Figure 1 As shown, this invention provides a scene generation method based on a physics engine and a large model, comprising the following steps: S1. Obtain the natural language scene description input by the user, perform structured parsing, extract object category, quantity information, scene type, style requirements and control parameters, and form a standardized semantic representation that can be directly called in subsequent generation; Specifically, the system receives user-generated text commands, supplementary task information, and control parameters from the user interface. It then identifies object categories, quantifiers, functional requirements, relative position indicators, and style constraints within the text. For example, it extracts elements such as the number of people, plates, cups, the central empty area of ​​the table, and the degree of neatness from the phrase "preparing a dinner table for four people, with plates neatly arranged, cups on the left, and a central area for taking and placing items." To enhance engineering adaptability, it can also standardize ambiguous expressions, such as converting descriptions like "near," "a small stack," or "relatively messy" into range-type or level-type parameters within the system. The process further determines the scenario type and support base type corresponding to the current instruction, such as a desktop placement scenario, shelf organization scenario, box containment scenario, or ground stacking scenario. It then determines the priority support surface boundaries, capacity constraints, and scenario template to be used during the solution process. This semantic information is mapped into structured fields, including object numbering rules, support surface size range, target stability level, compactness level, disorder level, and whether stacking, containment, and partial support are allowed. Finally, the results are organized into a unified scenario semantic template, which serves as shared input for subsequent operations. Through this step, the large model does not directly produce difficult-to-verify 3D coordinates, but first forms a clear and semantically complete intermediate expression. This not only improves the reusability of the language understanding results but also provides a standardized interface for subsequent predicate-level solutions, avoiding problems such as semantic gaps, missed object identification, and parameter ambiguity from the source.

[0025] S2. Based on the standardized semantic representation, retrieve the corresponding object model in the 3D asset library and supplement the physical attribute parameters for each target object model. The physical attribute parameters include model orientation, support probability, mass range, friction coefficient range and centroid offset range. Specifically, based on the semantic information (object category, functional characteristics, and style requirements) output in step S1, descriptive text for retrieval is generated, such as "white ceramic plate," "shallow basket for holding books," and "slender metal wrench." A mapping relationship between object numbers and descriptive text is established, and corresponding models are retrieved from the local or cloud-based 3D asset library. Candidate sets can be obtained by using keyword recall, vector retrieval, or a combination of both. When suitable objects are lacking in the asset library, external pathways such as text-to-image and image-to-3D mesh conversion can be invoked to automatically supplement new assets, thereby supporting open vocabulary scene generation. Candidate assets are reordered by integrating textual features, rendered view features, and geometric metadata to improve the consistency between search results and user semantics. Further attribute estimations are performed on candidate assets, including supportability, forward orientation, mass range, friction range, and centroid offset range. If necessary, size scaling rules, capacity estimation, and typical placement surface judgment can be added to support subsequent 2D layout and 3D physics calculations. For example, closed books and flat trays have a higher probability of support, while flexible bags and tilted decorative pieces have a lower probability. Round-bottomed containers and slender-footed tall cups require careful consideration of centroid position and friction variations during stability assessment.

[0026] The final result is a collection of objects that meet the semantic requirements and have complete attribute fields.

[0027] S3. Utilize the large model to parse the standardized semantic representation, generate spatial predicates and physical predicates, assign parameters to the predicates and arrange them into a set of structured constraints; Specifically, based on scene semantic templates and asset attributes, the geometric and physical relationships between objects are identified. For objects located on the support surface, spatial predicates with two-dimensional position and orientation constraints are generated, including left, right, front, back, center alignment, boundary alignment, symmetrical arrangement, and relative orientation. For objects with three-dimensional interaction requirements, physical predicates are generated, including containment, stacking, support, and arbitrary feasible placement. The system adds numerical or hierarchical parameters to the spatial and physical predicates based on user input or prior experience, including object spacing, relative offset, support area ratio, stability target, compactness, and clutter. If the user does not explicitly provide parameters, the system can automatically provide initial values ​​based on scene type, object size, and prior experience.

[0028] All predicates are uniformly arranged to establish object solving order, dependency relationships, and conflict handling rules, forming a structured set of constraints.

[0029] Compared to directly outputting coordinates and angles from a large model, this step can significantly reduce geometric inference errors. Furthermore, through the hierarchical design of spatial and physical predicates, it improves the interpretability, adjustability, and scalability of complex scenes, laying the foundation for the stable generation of high-density, multi-object, and highly interactive scenes.

[0030] S4. Based on the two-dimensional convex hull and constraint set of the object, iterative optimization is performed to obtain a two-dimensional initial layout with no out-of-bounds and low overlap. Specifically, it involves receiving the spatial relationships and object attributes output in step S3, including object geometrical bounding information, support surface range, relative position, alignment relationship, rotation relationship, and grouping relationship; A two-dimensional convex hull representation is generated for each object to replace the traditional axis-aligned bounding box as a collision approximation. Since objects in desktop and shelf-like scenarios often have rotational postures and irregular outer contours, the two-dimensional convex hull has higher geometric accuracy than the simple bounding box, while avoiding the high computational overhead of directly performing a full 3D mesh intersection test.

[0031] Based on the object's solution status and predicate dependencies, the distance, offset, anchor point position, and orientation parameters are iteratively optimized using the overlapping area of ​​the convex hull between objects and the distance from the object's center to the support boundary as penalty terms; the current candidate layout is continuously corrected; when the horizontal coordinate, vertical coordinate, and yaw angle of an object have been explicitly determined by the predicate or can be derived from other solved objects, the object is determined to be a solvable object and can enter the current round of layout update; The iteration results are checked for out-of-bounds and the congestion is evaluated. When the penalty value is lower than the preset threshold, the initial two-dimensional layout is output. When the preset number of iterations is reached and convergence is still not achieved, the failed objects, congested areas and conflict reasons are given as feedback information.

[0032] S5. Perform physical simulation and stability evaluation on the initial two-dimensional layout to obtain the final three-dimensional placement state that has passed physical verification. Specifically, this involves performing occupancy meshing on the current scene and candidate objects, transforming continuous geometry into a discrete occupancy representation that can be quickly traversed; A mesh search is performed on the surfaces that the candidate object may come into contact with to filter out candidate poses that are non-intersecting, have a sufficient contact area, and whose centroid projection falls inside the supporting convex hull; for predicates such as "put into container", candidate poses can be obtained by releasing the object from above the container and allowing it to sink under the action of gravity.

[0033] The physics engine is invoked to dynamically verify the candidate poses after screening. It detects whether the objects will slip, tip over, pop out of the container, or squeeze and intersect each other under the action of gravity, friction and contact. Candidate poses whose displacement changes are kept within the allowable range are determined as candidate poses that pass the dynamic verification. To further distinguish between "accidental stability" and "robust stability," perturbation sampling is performed on candidate poses that have passed dynamic verification, focusing on position, attitude, mass, friction, and centroid offset. The proportion of poses that remain in equilibrium after perturbation is statistically analyzed to obtain a stability score (stability probability or stability level evaluation). When the user's goal is to generate extreme placement cases, after perturbation sampling of candidate poses, low-steady-state effective solutions are selected based on the principle of not collapsing but being closer to the critical state. The verified candidate poses, contact states, and stability scores are output for use by subsequent modules.

[0034] S6. Perform quality assessment and feedback correction on the final 3D placement state. If the preset quality conditions are not met, generate natural language feedback information and input it into the large model for iterative correction until the quality conditions are met, obtain the final 3D scene, and output the scene geometry results, object parameters, evaluation indicators and data interface for robot simulation tasks. Specifically, the output of step S5 is checked for syntax and solution status to determine whether there are problems such as predicate format errors, incomplete object solving, object interpenetration, exceeding support boundaries, infeasible stacking posture, or obvious instability after physical simulation, and the reasons for failure are recorded according to object granularity. Analyze the remaining available space inside the current support surface or container to identify vacant areas, overcrowded areas, and potential areas that can be filled further, providing environmental-level hints for subsequent large model corrections; The current scene is comprehensively scored based on indicators including scene stability, number of objects, surface coverage, compactness, neatness, clutter, and visual question-and-answer evaluation results. The diagnostic results, empty area descriptions, and comprehensive scores are transformed into natural language feedback that is friendly to the large model. For example, it can point out that "there is still an empty space in the left rear area", "the cup and plate are currently partially overlapping", and "the books are stacked unstablely and the support ratio needs to be increased", so that the large model can supplement objects, modify parameters, or rewrite some predicates in a targeted manner. The final output of the scene results is in the form of a graphical interface or data interface, including a list of objects, pose parameters, physical properties, stability scores and solution logs, and provides standardized input for downstream simulation platforms, robot learning platforms or evaluation systems.

[0035] like Figures 2-7 As shown, a scene generation system based on a physics engine and a large model includes a scene instruction acquisition unit, an asset retrieval and attribute modeling unit, a predicate generation and constraint arrangement unit, a spatial constraint solving unit, a physical simulation and stability evaluation unit, a feedback correction unit, and a scene display module. The scene instruction acquisition unit includes a text input module, an element extraction module, a scene recognition module, a parameter specification module, and a template output module. It is used to acquire the natural language scene description input by the user, perform structured parsing, and generate a scene semantic template. Specifically, the text input module receives the instruction text, supplementary task information, and control parameters provided by the user on the interface; the element extraction module identifies object categories, quantifiers, functional requirements, relative position indicators, and style constraint words in the text; the scene recognition module further determines the scene type and supporting base type corresponding to the current instruction; the parameter specification module maps the aforementioned semantic information into structured fields; and the template output module organizes the results into a unified scene semantic template.

[0036] The asset retrieval and attribute modeling unit includes an object description module, an asset retrieval module, a multi-model matching module, an attribute annotation module, and a candidate asset output module. It is used to retrieve the 3D asset library based on scene semantic templates, obtain the corresponding 3D models, and supplement their physical attributes to obtain a set of 3D objects with physical attributes. Specifically, the object description module generates suitable descriptive text for retrieval based on the object category, functional characteristics, and style requirements output by the scene instruction acquisition unit, and establishes a mapping relationship between object numbers and descriptions. The asset retrieval module retrieves corresponding models from the local or cloud-based 3D asset library. The multi-model matching module re-ranks candidate assets by integrating text features, rendered view features, and geometric metadata. The attribute annotation module further estimates the attributes of the selected assets. The candidate asset output module provides a set of objects that meet semantic requirements and have complete attribute fields.

[0037] The predicate generation and constraint orchestration unit includes a semantic relation parsing module, a spatial predicate generation module, a physical predicate generation module, a parameter assignment module, and a constraint set output module. It is used to generate spatial and physical predicates based on a scene semantic template and a set of 3D objects using a large model, and assign constraint parameters to them to obtain a structured constraint set. Specifically, the semantic relation parsing module identifies target relationships between objects based on the scene template and asset attributes; the spatial predicate generation module generates 2D position and orientation constraints for objects located on support surfaces; the physical predicate generation module generates constraints such as containment, stacking, support, and arbitrary feasible placement for objects with 3D interaction requirements; the parameter assignment module adds numerical or hierarchical parameters to the predicates according to the user's fine-grained requirements; and the constraint set output module uniformly orchestrates all predicates, establishes the object solution order, dependencies, and conflict handling rules, and forms a structured constraint set that can be directly called by subsequent spatial and physical solvers.

[0038] The spatial constraint solving unit includes a constraint input module, a convex hull detection module, a parameter iteration module, a boundary verification module, and an initial layout output module. It is used to solve spatial constraints on the support surface based on spatial predicates in the structured constraint set and the two-dimensional convex hulls of the 3D objects, generating a two-dimensional initial layout. Specifically, the constraint input module receives spatial relationships and object attributes output by the predicate generation and constraint orchestration unit; the convex hull detection module generates a two-dimensional convex hull representation for each object; the parameter iteration module corrects the current candidate layout according to the object's solution state and predicate dependencies; the boundary verification module performs boundary checks and crowding assessments on the iterated results; and the initial layout output module outputs the two-dimensional initial scene.

[0039] The physics simulation and stability evaluation unit includes a candidate layout input module, a voxel sampling module, a support screening module, a physics simulation module, a stability scoring module, and a result confirmation module. It performs physics simulation and stability evaluation on the initial 2D layout based on the physical predicates in the structured constraint set and the physical properties of the 3D objects, obtaining a physically verified 3D scene placement state. Specifically, the candidate layout input module receives the 2D basic layout from the spatial solution unit, along with the geometric mesh and physical property range of the object to be placed; the voxel sampling module performs occupancy meshing on the current scene and candidate objects; the support screening module performs meshing search on surfaces that the candidate objects may contact; the physics simulation module calls the physics engine to dynamically verify the selected candidate poses; the stability scoring module performs perturbation sampling; and the result confirmation module outputs the verified 3D poses, contact states, and stability scores to subsequent modules.

[0040] The feedback correction unit includes a result diagnosis module, a void area identification module, a quality assessment module, and an instruction write-back module. It is used to perform quality assessment and feedback correction on the placement state of the 3D scene. If the preset quality conditions are not met, natural language feedback information is generated and input into the large model for iterative correction until the quality conditions are met, resulting in the final 3D scene. Specifically, the result diagnosis module checks the syntax and solution status of the current solution output; the void area identification module analyzes the remaining available space inside the current support surface or container; the quality assessment module provides a comprehensive score for the current scene; and the instruction write-back module converts the diagnostic results, void area descriptions, and quality evaluations into natural language feedback that is friendly to the large model.

[0041] The scene display module is used to output scene geometry results, object parameters, evaluation metrics, and data interfaces for robot simulation tasks.

[0042] Therefore, this invention employs a scene generation method and system based on a physics engine and a large model to generate 3D scene results that combine semantic consistency, structural complexity, and physical feasibility.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A scene generation method based on a physics engine and a large model, characterized in that: Includes the following steps: S1. Obtain the natural language scene description input by the user, perform structured parsing, extract object category, quantity information, scene type, style requirements and control parameters, and form a standardized semantic representation that can be directly called in subsequent generation; S2. Based on the standardized semantic representation, retrieve the corresponding object model in the 3D asset library and supplement the physical attribute parameters for each target object model. The physical attribute parameters include model orientation, support probability, mass range, friction coefficient range and centroid offset range. S3. Utilize the large model to parse the standardized semantic representation, generate spatial predicates and physical predicates, assign parameters to the predicates and arrange them into a set of structured constraints; S4. Based on the two-dimensional convex hull and constraint set of the object, iterative optimization is performed to obtain a two-dimensional initial layout with no out-of-bounds and low overlap. S5. Perform physical simulation and stability evaluation on the initial two-dimensional layout to obtain the final three-dimensional placement state that has passed physical verification. S6. Perform quality assessment and feedback correction on the final 3D placement state. If the preset quality conditions are not met, generate natural language feedback information and input it into the large model for iterative correction until the quality conditions are met, obtain the final 3D scene, and output the scene geometry results, object parameters, evaluation indicators, and data interfaces for robot simulation tasks.

2. The scene generation method based on a physics engine and a large model according to claim 1, characterized in that: Step S1 specifically includes the following: The system receives the instruction text, task supplement information, and control parameters given by the user on the interface. It identifies the object category, quantifiers, functional requirements, relative position indicators, and style constraint words in the text. It further determines the scene type and support base type corresponding to the current instruction and determines the support surface boundary, capacity constraint, and scene template to be used first when solving the problem. The semantic information described above is mapped into structured fields, including object numbering rules, support surface size range, target stability level, compactness level, clutter level, and whether stacking, containment, and partial support are allowed. Finally, the results are organized into a unified scene semantic template, which is used as shared input for subsequent operations.

3. The scene generation method based on a physics engine and a large model according to claim 2, characterized in that: Step S2 specifically includes the following: Based on the semantic information output in step S1, generate descriptive text for retrieval and establish a mapping relationship between object number and descriptive text. Retrieve the corresponding model in the local or cloud-based 3D asset library to obtain a candidate asset set. Candidate assets are reordered by combining text features, rendered view features, and geometric metadata. Further, attribute estimations are performed on the candidate assets, including supportability, forward orientation, mass range, friction range, and centroid offset range, to support subsequent 2D layout and 3D physics calculations. The final result is a collection of objects that meet the semantic requirements and have complete attribute fields.

4. The scene generation method based on a physics engine and a large model according to claim 3, characterized in that: Step S3 specifically includes the following: Based on scene semantic templates and asset attributes, identify the geometric and physical relationships between objects, and generate spatial predicates for two-dimensional position and orientation constraints for objects located on the support surface, including left, right, front, back, center alignment, boundary alignment, symmetrical arrangement and relative orientation; Generate physical predicates for objects with 3D interaction requirements, including containment, stacking, support, and arbitrary feasible placement; Based on user input or prior experience, add numerical or hierarchical parameters to spatial and physical predicates, including object spacing, relative offset, support area ratio, stability target, compactness, and clutter. All predicates are uniformly arranged to establish object solving order, dependency relationships, and conflict handling rules, forming a structured set of constraints.

5. The scene generation method based on a physics engine and a large model according to claim 4, characterized in that: Step S4 specifically includes the following: Receive the spatial relationships and object attributes output in step S3, including object geometrical bounding information, support surface range, relative position, alignment relationship, rotation relationship, and grouping relationship; Generate a two-dimensional convex hull representation for each object as a collision approximation; Based on the object's solution state and predicate dependency, the distance, offset, anchor point position, and orientation parameters are iteratively optimized using the overlapping area of ​​the convex hull between objects and the distance from the object's center to the support boundary as penalty terms. The iteration results are checked for out-of-bounds and the crowding is evaluated. When the penalty value is lower than the preset threshold, the two-dimensional initial layout is output. If convergence is not achieved after the preset number of iterations, the failed objects, congested regions, and reasons for conflicts will be provided as feedback information.

6. The scene generation method based on a physics engine and a large model according to claim 5, characterized in that: Step S5 specifically includes the following: performing occupancy meshing processing on the current scene and candidate objects to transform continuous geometry into a discrete occupancy representation that can be quickly traversed; A mesh search is performed on the surfaces that the candidate object may come into contact with to filter out candidate poses that are non-intersecting, have a sufficient contact area, and whose centroid projection falls inside the supporting convex hull. The physics engine is invoked to dynamically verify the candidate poses after screening. It detects whether the objects will slip, tip over, pop out of the container, or squeeze and intersect each other under the action of gravity, friction and contact. Candidate poses whose displacement changes are kept within the allowable range are determined as candidate poses that pass the dynamic verification. For candidate poses that pass dynamic verification, perturbation sampling is performed around position, attitude, mass, friction, and centroid offset. The proportion of poses that remain in balance after perturbation is counted to obtain a stability score. The verified candidate poses, contact states, and stability scores will be output for use by subsequent modules.

7. The scene generation method based on a physics engine and a large model according to claim 6, characterized in that: Step S6 specifically includes the following: Perform syntax and solution status checks on the output of step S5 to determine if there are any issues such as predicate format errors, incomplete object solving, object interpenetration, exceeding support boundaries, infeasible stacking posture, or significant instability after physical simulation, and record the reasons for failure according to object granularity. Analyze the remaining available space inside the current support surface or container to identify vacant areas, overcrowded areas, and potential areas that can be filled further, providing environmental-level hints for subsequent large model corrections; The current scene is comprehensively scored based on indicators including scene stability, number of objects, surface coverage, compactness, neatness, clutter, and visual question-and-answer evaluation results. The above diagnostic results, empty area descriptions and comprehensive scores are transformed into natural language feedback that is friendly to large models, enabling large models to supplement objects, modify parameters or rewrite some predicates in a targeted manner. The final output of the scene results is in the form of a graphical interface or data interface, including a list of objects, pose parameters, physical properties, stability scores and solution logs, and provides standardized input for downstream simulation platforms, robot learning platforms or evaluation systems.

8. The scene generation method based on a physics engine and a large model according to claim 7, characterized in that: In step S5, when the user's goal is to generate extreme placement cases, after perturbing and sampling the candidate poses, low-steady-state effective solutions are selected according to the principle of not collapsing but being closer to the critical state.

9. A scene generation system based on a physics engine and a large model, used to implement the scene generation method based on a physics engine and a large model as described in any one of claims 1-8, characterized in that: It includes a scene command acquisition unit, an asset retrieval and attribute modeling unit, a predicate generation and constraint arrangement unit, a spatial constraint solving unit, a physical simulation and stability evaluation unit, a feedback correction unit, and a scene display module; The scene instruction acquisition unit is used to acquire the natural language scene description input by the user, perform structured parsing on it, and generate scene semantic templates; The asset retrieval and attribute modeling unit is used to retrieve the 3D asset library based on the scene semantic template, obtain the corresponding 3D model and supplement the physical attributes to obtain a set of 3D objects with physical attributes. The predicate generation and constraint orchestration unit is used to generate spatial and physical predicates based on scene semantic templates and a set of 3D objects using a large model, and to assign constraint parameters to them to obtain a set of structured constraints. The spatial constraint solving unit is used to solve spatial constraints on the support surface based on the spatial predicates in the structured constraint set and the two-dimensional convex hull of the three-dimensional object, and generate a two-dimensional initial layout. The physical simulation and stability evaluation unit is used to perform physical simulation and stability evaluation on the two-dimensional initial layout based on the physical predicates in the structured constraint set and the physical properties of the three-dimensional objects, so as to obtain the three-dimensional scene placement state that has passed physical verification. The feedback correction unit is used to perform quality assessment and feedback correction on the placement state of the 3D scene. If the preset quality conditions are not met, natural language feedback information is generated and input into the large model for iterative correction until the quality conditions are met, and the final 3D scene is obtained. The scene display module is used to output scene geometry results, object parameters, evaluation metrics, and data interfaces for robot simulation tasks.