Simulation scene and trajectory automatic generation method and system for robot operation

Abstract

The application discloses a kind of robot simulation scene and trajectory automatic generation method and system, receive natural language operation instruction, and through semantic analysis, visual asset generation, three-dimensional reconstruction and physical credible layout generation three-dimensional simulation scene that can be used for robot operation training;Utilize video generation model to synthesize the continuous image sequence of target operation task, and extract two-dimensional visual motion trajectory in target object area;By depth estimation, three-dimensional motion fusion, grasping posture generation and inverse kinematics mapping, two-dimensional visual motion trajectory is converted into robot end effector trajectory and corresponding robot joint trajectory, the simulation scene and expert trajectory generated are paired, form training data set for robot imitation learning strategy training, and support the strategy obtained by training is deployed to simulation environment or real robot hardware.The application can reduce the cost of robot training data acquisition and artificial trajectory design, improve the degree of automation of simulation scene and operation trajectory generation.

Description

Technical Field

[0001] This invention belongs to the field of robotics technology, and relates to the interdisciplinary fields of robot learning, generative artificial intelligence, computer vision and physical simulation. Specifically, it relates to a method and system for automatically generating simulation scenarios and trajectories for robot operations. Background Technology

[0002] The operation of general-purpose robots in unstructured environments requires strategies that can generalize across different objects, scenes, and semantic targets. Although imitation learning has made significant progress in recent years in acquiring complex behaviors from expert demonstrations, its performance is fundamentally limited by the scale and diversity of training data.

[0003] Collecting large-scale real-world datasets via teleoperation is extremely costly, time-consuming, and difficult to scale to long-tail scenarios. For example, collecting a dataset containing 100,000 operation demonstrations would require thousands of hours of dedicated operators and cost millions of dollars. While simulation offers a scalable alternative, existing simulators typically rely on fixed asset libraries and manually designed tasks, limiting the semantic and visual diversity required for robust simulation-to-real-world transfer.

[0004] The emergence of generative base models offers a promising approach to addressing this data scarcity problem. Large Language Models (LLMs) and diffusion models encapsulate extensive world knowledge about object semantics, spatial relationships, and visual availability. However, most existing work explores using base models to generate static 3D scenes or high-level task plans, rarely producing fully executable robotic tasks. Video generation can produce visually plausible motion, but due to limited physical foundations, it often violates geometric consistency or kinematic constraints, making it unsuitable for direct use in robot control.

[0005] The existing technology has the following core defects:

[0006] 1) Reliance on fixed asset libraries: Some existing automated simulation scene generation methods are still limited by predefined 3D asset libraries, making it difficult to generate truly novel objects and scenes, thus limiting the semantic and visual diversity of training data.

[0007] 2) Trajectory generation relies on manual design: Traditional motion planners require task-specific engineering design and lack the semantic understanding capabilities needed to handle open-vocabulary manipulation tasks.

[0008] 3) Lack of physical basis in video generation: Existing video generation models mainly operate in 2D pixel space, making it difficult to extract accurate and physically reliable 3D trajectories for robot control.

[0009] 4) Difficulty in transferring from simulation to reality: Existing methods require a large amount of domain randomization or real-world fine-tuning, making it difficult to stably achieve zero-sample transfer without real demonstration data and target domain fine-tuning.

[0010] Therefore, there is an urgent need in this field for a technical solution that can automatically synthesize dynamic and physically feasible operation trajectories without human supervision, in order to solve the bottleneck problem of training data for general robots. Summary of the Invention

[0011] Purpose of the invention: The purpose of this invention is to overcome the shortcomings of existing technologies, such as high cost of robot training data collection, limited diversity of simulation environments, and reliance on manual design for operation trajectory generation. It proposes a method and system for automatically generating simulation scenarios and trajectories for robot operation, which can directly generate open vocabulary, simulation-ready operating environments, and executable expert trajectories from natural language instructions, thereby achieving scalable and general-purpose robot learning.

[0012] Technical solution: The present invention provides a method for automatically generating simulation scenes and trajectories for robot operations, comprising the following steps:

[0013] S1. Command Reception and Initialization: Receive natural language commands and initialize the simulation environment and generated model parameters;

[0014] S2, Semantic to Simulation Scene Synthesis: Converting natural language instructions into physically believable interactive 3D simulation scenes, including four stages: semantic parsing, visual synthesis, 3D reconstruction and physical layout;

[0015] S3. Trajectory generation based on video prior: Using the stabilized simulation scene as the initial frame, the continuous time series of the specified operation task is synthesized using the video generation model, and the two-dimensional motion trajectory of the target object is extracted.

[0016] S4. Alignment of visual motion to robot trajectory: Extracting operable 3D robot end effector trajector trajector trajectories from 2D visual prediction, including depth estimation, 3D motion fusion and inverse kinematics mapping;

[0017] S5. Dataset Construction and Policy Training: The generated simulation scene and expert trajectory are paired to form a training dataset for robot imitation learning policy training.

[0018] S6. Zero-shot transfer and deployment: The trained policy is generalized to objects not seen in the simulation with zero shots, and zero-shot transfer to real robot hardware is achieved through photo-conditional scene alignment.

[0019] Furthermore, step S2 is implemented as follows:

[0020] S21. Semantic parsing and asset inventory generation: A large language model is used as an advanced semantic planner to decompose the abstract scene description into a structured JSON-formatted asset inventory, explicitly specifying object categories and style elements, and injecting specific negative constraints to ensure the quality of downstream reconstruction.

[0021] S22, Generative Visual Synthesis: A diffusion model is used to generate high-fidelity 2D image assets for each asset inventory item, and a segmentation model is applied for accurate background removal. A hybrid segmentation scheme of semantic priority and visual backup is adopted to improve robustness.

[0022] S23, Memory-efficient 3D reconstruction: Dynamically offloading encoder and decoder subnetworks between CPU and GPU, and strategically trading computation time for VRAM optimization, to achieve the reconstruction of complex high-polygon geometry on standard consumer hardware;

[0023] S24, Physically Reliable Layout: Combining large language model spatial reasoning and physics engine verification, it performs constraint-based layout through collision detection and gravity alignment, and performs procedural spatial jitter to systematically solve bounding box overlap.

[0024] Furthermore, the negative constraint in step S21 is to prohibit the generation of highly reflective or transparent surfaces.

[0025] Furthermore, modifiers are added to the diffusion model prompts described in step S22 to reduce environmental bias and avoid built-in shadows.

[0026] Furthermore, step S3 is implemented as follows:

[0027] S31, Scene Stabilization and Style Optimization: A brief dynamic stabilization phase is performed in the physics engine to resolve micro-penetration between objects, the stabilized scene is captured as the initial frame, and optionally an image editing model is used to style the initial frame to increase visual diversity.

[0028] S32, Video Sequence Synthesis: Input the processed initial frame into the video generation model to synthesize a continuous time sequence of the specified operation task, and apply targeted negative cues to force the diffusion model to focus on the rigid body kinematics of the manipulator and the target object, and suppress artifacts that violate physical laws.

[0029] S33. Visual motion extraction: Run a segmentation model on the initial frame to obtain a high-quality mask of the target object, uniformly sample two-dimensional points within the object mask, and use a tracking model to track these points across frames to generate a dense two-dimensional motion trajectory consistent with the object.

[0030] Furthermore, the negative prompts mentioned in step S32 include "no camera motion", "no object deformation", "no background flickering", and "rigid body motion".

[0031] Furthermore, step S33 employs a mask-restricted tracking method, tracking only points within the target object's region.

[0032] Furthermore, step S4 is implemented as follows:

[0033] S41. Depth Estimation: Using a visual metric depth estimation model based on the Transformer architecture, the metric depth of each frame in the video frame sequence is estimated, providing metric cues for 3D enhancement.

[0034] S42, 3D motion fusion: The tracked 2D points are fused with depth information to generate a coherent 3D motion trajectory, and lightweight post-processing is applied to remove abnormal trajectories and suppress time jitter.

[0035] S43. Grasping posture generation: Generate a feasible 6-DOF robot grasping posture for the target object using a diffusion-based grasping generation model.

[0036] S44. Inverse Kinematics Mapping: Based on the grasping posture, the three-dimensional motion of the target object is converted into the desired motion of the robot's end effector. The robot joint angles at each time step are calculated by the inverse kinematics solver to generate a physically executable trajectory.

[0037] Furthermore, step S6 is implemented as follows:

[0038] S61, Real Scene Capture: Take a snapshot of the real target scene as additional alignment input;

[0039] S62, Simulation Scene Matching: Instantiate an initial frame in the simulation that matches and optimizes the style of the real scene, and calibrate the simulator to match the real camera parameters and robot workspace boundaries.

[0040] S63, Zero-shot deployment: Deploy policies trained only on synthetic data to physical hardware with zero samples, without any real-world demonstrations or target domain fine-tuning.

[0041] The present invention discloses an automatic simulation scene and trajectory generation system for robot operation, comprising:

[0042] The semantic-to-simulation scene synthesis module is used to convert natural language instructions into an interactive 3D simulation environment. It includes: a semantic parsing and asset inventory generation unit, which uses a large language model to decompose abstract scene descriptions into a structured JSON-formatted asset inventory; a generative visual synthesis unit, which uses an image diffusion model to generate high-fidelity 2D image assets and applies an image segmentation model for background removal; a memory-efficient 3D reconstruction unit, which uses a 2D-to-3D generation model to elevate 2D assets into 3D meshes, supporting dynamic CPU and GPU offloading; and a physically reliable layout unit, which combines large language model spatial reasoning and physical simulation engine verification to construct a physically reliable 3D scene layout.

[0043] The trajectory generation module based on video priors is used to simulate expert operation behavior using a video-based model. It includes: a scene stabilization and style optimization unit, which performs a dynamic stabilization phase in the physics simulation engine and performs style optimization using an image editing model; a video generation unit, which uses a video generation model to synthesize a continuous time series of the operation task; and a visual motion extraction unit, which uses a point tracking model to track the two-dimensional motion trajectory of the target object.

[0044] The alignment module from visual motion to robot trajectory is used to extract operable 3D robot trajectories from 2D visual predictions. It includes: a depth estimation unit that estimates the metric depth for each frame using a depth estimation model; a 3D motion fusion unit that fuses the 2D trajectory with the depth information into a coherent 3D motion trajectory using a 3D motion fusion module; and an inverse kinematics mapping unit that generates the grasping posture using a grasping posture generation model and calculates the robot joint angles using an inverse kinematics solver.

[0045] The dataset construction and policy training module is used to generate training datasets and train the robot to imitate and learn policies.

[0046] The zero-shot migration deployment module is used to achieve zero-shot migration from simulation to real robot hardware.

[0047] Beneficial effects: Compared with the prior art, the present invention has at least the following beneficial effects:

[0048] 1. This invention can automatically convert natural language operation instructions into three-dimensional simulation scenes that can be used for robot operation training, reducing the reliance on fixed three-dimensional asset libraries and artificial scene construction, and improving the semantic and visual diversity of simulation scenes;

[0049] 2. This invention utilizes a video generation model to provide prior knowledge of the operation motion, and obtains a two-dimensional visual trajectory related to the motion of the target object through point tracking methods such as target object extraction, negative cue constraints, and mask restrictions, thereby reducing the need for manual design of operation trajectories or task-specific motion planning rules.

[0050] 3. This invention converts the two-dimensional visual motion in the generated video into the trajectory of the robot's end effector and joint trajectory through depth estimation, three-dimensional motion fusion, grasping posture generation and inverse kinematics mapping, so that the operation behavior generated by the generative visual model can be converted into physically executable robot training data.

[0051] 4. This invention can organize the generated simulation scene, visual observation and expert trajectory into a training dataset for robot imitation learning strategy training, thereby improving the efficiency of robot training data generation.

[0052] 5. This invention supports the construction of matching simulation scenarios based on real scene images and the deployment of strategies trained on synthetic data to real robot hardware, which helps reduce the need for real robot demonstration data acquisition and target domain fine-tuning. Attached Figure Description

[0053] Figure 1 This is a flowchart of a simulation scenario and trajectory automatic generation method for robot operations;

[0054] Figure 2 This is a schematic diagram of a simulation scenario and trajectory automatic generation system architecture for robot operations;

[0055] Figure 3 These are qualitative example diagrams of tasks of varying difficulty synthesized using this invention;

[0056] Figure 4 These are example images of scenes synthesized using this invention, featuring different object instances, textures, and layouts.

[0057] Figure 5 The simulation evaluation results are shown in the figure.

[0058] Figure 6 This is a flowchart of an end-to-end real-world migration implemented using this invention. Detailed Implementation

[0059] The present invention will now be described in further detail with reference to the accompanying drawings.

[0060] like Figure 1 As shown, this invention proposes an automatic generation method for simulation scenes and trajectories for robot operation. Based on natural language commands input by humans, it automatically generates a 3D simulation environment with open vocabulary and physically executable expert demonstration trajectories for training robot operation strategies with strong generalization capabilities. The method mainly includes semantic-to-physical scene synthesis, video-prior-based trajectory generation, and simulation-to-generation alignment. The specific implementation process is as follows:

[0061] S1. Command Reception and Initialization: Receives natural language commands input by the user, such as "In a simple living room, put the bowl on the plate". Optionally, it can receive real-world scene photos as additional input. It initializes simulation environment parameters, generates model parameters, and robot model parameters.

[0062] S2. Semantic-to-Physical Scene Synthesis: This involves converting natural language instructions into physically believable interactive 3D simulation scenes, specifically including four stages:

[0063] Semantic parsing and asset inventory generation: The Qwen-Max large language model is used as a high-level semantic planner to decompose the abstract scene description into a structured JSON-formatted asset inventory, explicitly specifying environment categories, object categories, and style elements. Specific negative constraints are injected, such as prohibiting highly reflective or transparent surfaces, to ensure the quality of downstream reconstruction.

[0064] Generative visual synthesis: High-fidelity 2D image assets are generated for each asset inventory entry using a Flux diffusion model. Modifiers such as "studio lighting" and "pure white background" are added in the prompts to reduce environmental bias and avoid built-in shadows. The SAM3 segmentation model is applied for accurate background removal, and a hybrid segmentation scheme combining semantic priority and visual fallback is employed to improve robustness.

[0065] Memory-efficient 3D reconstruction: Upgrade 2D assets to 3D meshes using SAM3D. During inference, encoder and decoder subnetworks are dynamically offloaded between the CPU and GPU, strategically trading computation time for VRAM optimization to achieve reconstruction of complex high-polygon geometry on standard consumer hardware.

[0066] Physically Reliable Layout: Combining large language model spatial reasoning and Genesis physics engine verification, a physically reliable 3D scene layout is constructed. LLM infers approximate real-world metric dimensions for each object, and simple heuristics are applied to correct distortions introduced during 3D upscaling. Constraint-based layout is performed through AABB collision detection and gravity alignment, anchor points are snapped to the ground plane, attachments are placed on supporting surfaces, and procedural spatial jitter is used to systematically resolve bounding box overlaps.

[0067] S3. Trajectory Generation Based on Video Priors: Using the stabilized simulation scene as the initial frame, a continuous time series of the specified operation task is synthesized using the video basic model, and the 2D motion trajectory of the target object is extracted. This includes three stages:

[0068] Scene stabilization and style optimization: A brief dynamic stabilization phase (approximately 0.5 seconds) is performed in the Genesis physics engine to resolve residual micro-penetration between objects, capturing the stabilized scene as the initial frame. Qwen-Image-Edit is used to style the initial frame, automatically applying predefined style perturbations to increase visual diversity, while also providing a user-friendly interface for user-specified stylization.

[0069] Video Generation: The style-optimized initial frames are input into the Wan2.2-I2V-Flash video generation model to synthesize a continuous time series (typically 16-32 frames) of the specified operation task. Targeted negative cues are applied, including "no camera motion," "no object deformation," "no background flicker," and "rigid body motion," etc. The forced diffusion model focuses on the rigid body kinematics of the manipulator and the target object, effectively suppressing dreamlike deformation artifacts that often violate physical feasibility.

[0070] Visual motion extraction: SAM3 is rerun on the initial frame after visual processing to obtain a high-quality mask of the target object. Uniform sampling (typically 100-200 points) is performed within the object mask, and these points are tracked across frames using CoTracker3 to produce dense but consistent 2D trajectories for the object.

[0071] It should be noted that the video generation model outputs continuous image frames in a two-dimensional pixel space, which do not contain joint angles, end effector poses, or operational trajectories that can be directly used for robot control, or satisfy robot kinematic constraints. Directly controlling the robot based on the generated video is easily affected by factors such as target object occlusion, background changes, object deformation, inconsistent depth scales, and temporal jitter, making it difficult to obtain physically executable robot actions. Therefore, this invention further proposes an alignment step from visual motion to robot trajectory, converting the visual motion of the target object in the generated video into an executable robot trajectory.

[0072] S4. Alignment of visual motion to robot trajectory: Extracting operable 3D robot end effector trajector trajector trajectories from 2D visual predictions, including:

[0073] Depth estimation: Perform depth estimation on each frame of the generated video to obtain the depth information of the target object in each frame;

[0074] 3D motion fusion: The two-dimensional point trajectories tracked within the target object area are fused with the corresponding depth information to obtain a continuous three-dimensional object motion trajectory, and abnormal trajectories and time jitter are suppressed.

[0075] Grasping posture generation: Generate a feasible robot grasping posture based on the three-dimensional shape and pose of the target object;

[0076] Inverse kinematics mapping: Based on the grasping posture and the motion trajectory of the 3D object, the desired pose sequence of the robot's end effector is calculated, and the corresponding robot joint trajectory is obtained by solving inverse kinematics, thereby generating a physically executable robot operation trajectory.

[0077] S5. Dataset Construction and Policy Training: The generated simulation scene and expert trajectory are paired to form a training dataset for robot imitation learning policy training.

[0078] S6. Zero-shot transfer and deployment: The trained policy is generalized to objects not seen in the simulation with zero shots, and zero-shot transfer to real robot hardware is achieved through photo-conditional scene alignment.

[0079] Zero-sample transfer refers to deploying a strategy trained on synthetic data to real robot hardware without collecting demonstration data from artificial robots in the target real-world scenario or making additional policy fine-tuning for the target real-world scenario.

[0080] Real-world scene capture: Take snapshots of the real target scene as additional alignment input.

[0081] Simulation scene matching: Instantiate initial frames in the simulation to match the real scene and optimize the style, and calibrate the simulator to match the real camera parameters and robot workspace boundaries.

[0082] Zero-shot deployment: Deploy policies trained solely on synthetic data to physical hardware with zero samples, without any real-world demonstrations or target domain fine-tuning.

[0083] like Figure 2 As shown, the present invention also provides a simulation scene and trajectory automatic generation system for robot operations, comprising:

[0084] The semantic-to-simulation scene synthesis module is used to convert natural language instructions into an interactive 3D simulation environment. It includes: a semantic parsing and asset inventory generation unit, which uses a large language model to decompose abstract scene descriptions into a structured JSON-formatted asset inventory; a generative visual synthesis unit, which uses an image diffusion model to generate high-fidelity 2D image assets and applies an image segmentation model for background removal; a memory-efficient 3D reconstruction unit, which uses a 2D-to-3D generation model to elevate 2D assets into 3D meshes, supporting dynamic CPU and GPU offloading; and a physically reliable layout unit, which combines large language model spatial reasoning and physical simulation engine verification to construct a physically reliable 3D scene layout.

[0085] The trajectory generation module based on video priors is used to simulate expert operation behavior using a video-based model. It includes: a scene stabilization and style optimization unit, which performs a dynamic stabilization phase in the physics simulation engine and performs style optimization using an image editing model; a video generation unit, which uses a video generation model to synthesize a continuous time series of the operation task; and a visual motion extraction unit, which uses a point tracking model to track the two-dimensional motion trajectory of the target object.

[0086] The alignment module from visual motion to robot trajectory is used to extract operable 3D robot trajectories from 2D visual predictions. It includes: a depth estimation unit, which estimates the metric depth for each frame using a depth estimation model; a 3D motion fusion unit, which fuses the 2D trajectory with the depth information into a coherent 3D motion trajectory using a 3D motion fusion module; and an inverse kinematics mapping unit, which generates the grasping posture using a grasping posture generation model and calculates the robot joint angles using an inverse kinematics solver.

[0087] The dataset construction and policy training module is used to generate training datasets and train the robot to imitate and learn policies.

[0088] The zero-shot transfer deployment module is used to achieve zero-shot transfer from simulation to real robot hardware. For example... Figure 3 As shown, the present invention can synthesize operational tasks across different difficulty levels: 1) Simple: such as "putting a cup on a plate"; 2) Medium: such as "stacking three blocks in a bedroom"; 3) Complex: involving long-distance, cross-regional movement and operation tasks.

[0089] like Figure 3 As shown on the right, this invention can automatically extract precise end-effector trajectories for these tasks. Furthermore, as... Figure 4 As shown, the scenes generated by this invention possess high visual and semantic diversity. Scene types cover "futuristic workstations," "space-themed rooms," "cozy and simple rooms," and "luxury bedrooms," demonstrating that this system has powerful open vocabulary generation capabilities and is not limited by a fixed asset library.

[0090] like Figure 5 As shown, the generated trajectory was validated on a large scale in a simulation environment: the Transformer-based Action Block Strategy (ACT) was trained using diverse synthetic scenes. Figure 5 The curve on the right shows how the policy success rate changes as the number of synthetic trajectories increases. Experiments show that when the synthetic data size reaches 2500, the policy achieves a peak zero-sample success rate of 36.96% on completely unseen geometric objects (such as mugs).

[0091] like Figure 6As shown, this invention supports an end-to-end real-world migration process. First, images of the real target scene are acquired, and key information such as target objects, containers, desktops, and the robot's workspace are identified. Based on this, a simulation layout matching the real scene is constructed in a simulation environment. Then, in the simulation environment, the video-driven trajectory synthesis method of this invention generates corresponding expert operation trajectories, which are used as training data for robot policy training. Finally, the policy trained based on the synthesized data is directly deployed to real robot hardware, enabling the robot to complete corresponding operation tasks without requiring real human demonstration data of the target scene. Specifically, in Figure 6 In the illustrated embodiment, the present invention constructs a matching simulation scene based on the yellow cube, red target bowl, tabletop, and robotic arm in a real-world scenario, and generates an operation trajectory in the simulation environment for grasping the yellow cube and placing it into the red target bowl. After strategy training, the real robot can perform the same operation in a physical environment. This embodiment demonstrates that the trajectory generated by the present invention not only has visual plausibility but also meets the kinematic execution requirements of the robot, thereby facilitating the transfer from simulation environment to real robot execution.

[0092] The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all fall within the protection scope of the present invention.

Claims

1. A method for automatically generating simulation scenarios and trajectories for robot operations, characterized in that, Includes the following steps: S1. Command Reception and Initialization: Receive natural language commands and initialize the simulation environment and generated model parameters; S2, Semantic to Simulation Scene Synthesis: Converting natural language instructions into physically believable interactive 3D simulation scenes, including four stages: semantic parsing, visual synthesis, 3D reconstruction and physical layout; S3. Trajectory generation based on video prior: Using the stabilized simulation scene as the initial frame, the continuous time series of the specified operation task is synthesized using the video generation model, and the two-dimensional motion trajectory of the target object is extracted. S4. Alignment of visual motion to robot trajectory: Extracting operable 3D robot end effector trajector trajector trajectories from 2D visual prediction, including depth estimation, 3D motion fusion and inverse kinematics mapping; S5. Dataset Construction and Policy Training: The generated simulation scene and expert trajectory are paired to form a training dataset for robot imitation learning policy training. S6. Zero-shot transfer and deployment: The trained policy is generalized to objects not seen in the simulation with zero shots, and zero-shot transfer to real robot hardware is achieved through photo-conditional scene alignment.

2. The method for automatically generating simulation scenarios and trajectories for robot operations according to claim 1, characterized in that, The implementation process of step S2 is as follows: S21. Semantic parsing and asset inventory generation: A large language model is used as an advanced semantic planner to decompose the abstract scene description into a structured JSON-formatted asset inventory, explicitly specifying object categories and style elements, and injecting specific negative constraints to ensure the quality of downstream reconstruction. S22, Generative Visual Synthesis: A diffusion model is used to generate high-fidelity 2D image assets for each asset inventory item, and a segmentation model is applied for accurate background removal. A hybrid segmentation scheme of semantic priority and visual backup is adopted to improve robustness. S23, Memory-efficient 3D reconstruction: Dynamically offloading encoder and decoder subnetworks between CPU and GPU, and strategically trading computation time for VRAM optimization, to achieve the reconstruction of complex high-polygon geometry on standard consumer hardware; S24, Physically Reliable Layout: Combining large language model spatial reasoning and physics engine verification, it performs constraint-based layout through collision detection and gravity alignment, and performs procedural spatial jitter to systematically solve bounding box overlap.

3. The method for automatically generating simulation scenarios and trajectories for robot operations according to claim 2, characterized in that, The negative constraint in step S21 is to prohibit the generation of highly reflective or transparent surfaces.

4. The method for automatically generating simulation scenarios and trajectories for robot operations according to claim 2, characterized in that, Add modifiers to the diffusion model prompts described in step S22 to reduce environmental bias and avoid built-in shadows.

5. The method for automatically generating simulation scenarios and trajectories for robot operations according to claim 1, characterized in that, The implementation process of step S3 is as follows: S31, Scene Stabilization and Style Optimization: A brief dynamic stabilization phase is performed in the physics engine to resolve micro-penetration between objects, the stabilized scene is captured as the initial frame, and optionally an image editing model is used to style the initial frame to increase visual diversity. S32, Video Sequence Synthesis: Input the processed initial frame into the video generation model to synthesize a continuous time sequence of the specified operation task, and apply targeted negative cues to force the diffusion model to focus on the rigid body kinematics of the manipulator and the target object, and suppress artifacts that violate physical laws. S33. Visual motion extraction: Run a segmentation model on the initial frame to obtain a high-quality mask of the target object, uniformly sample two-dimensional points within the object mask, and use a tracking model to track these points across frames to generate a dense two-dimensional motion trajectory consistent with the object.

6. The method for automatically generating simulation scenarios and trajectories for robot operations according to claim 5, characterized in that, The negative prompts mentioned in step S32 include "no camera motion", "no object deformation", "no background flickering" and "rigid body motion".

7. The method for automatically generating simulation scenarios and trajectories for robot operations according to claim 5, characterized in that, Step S33 employs a mask-restricted tracking method, tracking only points within the target object's region.

8. The method for automatically generating simulation scenarios and trajectories for robot operations according to claim 1, characterized in that, The implementation process of step S4 is as follows: S41. Depth Estimation: Using a visual metric depth estimation model based on the Transformer architecture, the metric depth of each frame in the video frame sequence is estimated, providing metric cues for 3D enhancement. S42, 3D motion fusion: The tracked 2D points are fused with depth information to generate a coherent 3D motion trajectory, and lightweight post-processing is applied to remove abnormal trajectories and suppress time jitter. S43. Grasping posture generation: Generate a feasible 6-DOF robot grasping posture for the target object using a diffusion-based grasping generation model. S44. Inverse Kinematics Mapping: Based on the grasping posture, the three-dimensional motion of the target object is converted into the desired motion of the robot's end effector. The robot joint angles at each time step are calculated by the inverse kinematics solver to generate a physically executable trajectory.

9. The method for automatically generating simulation scenarios and trajectories for robot operations according to claim 1, characterized in that, The implementation process of step S6 is as follows: S61, Real Scene Capture: Take a snapshot of the real target scene as additional alignment input; S62, Simulation Scene Matching: Instantiate an initial frame in the simulation that matches and optimizes the style of the real scene, and calibrate the simulator to match the real camera parameters and robot workspace boundaries. S63, Zero-shot deployment: Deploy policies trained only on synthetic data to physical hardware with zero samples, without any real-world demonstrations or target domain fine-tuning.

10. A simulation scene and trajectory automatic generation system for robot operations using the method described in any one of claims 1 to 9, characterized in that, include: The semantic-to-simulation scene synthesis module is used to convert natural language instructions into an interactive 3D simulation environment. It includes: a semantic parsing and asset inventory generation unit, which uses a large language model to decompose abstract scene descriptions into a structured JSON-formatted asset inventory; a generative visual synthesis unit, which uses an image diffusion model to generate high-fidelity 2D image assets and applies an image segmentation model for background removal; a memory-efficient 3D reconstruction unit, which uses a 2D-to-3D generation model to elevate 2D assets into 3D meshes, supporting dynamic CPU and GPU offloading; and a physically reliable layout unit, which combines large language model spatial reasoning and physical simulation engine verification to construct a physically reliable 3D scene layout. The trajectory generation module based on video priors is used to simulate expert operation behavior using a video-based model. It includes: a scene stabilization and style optimization unit, which performs a dynamic stabilization phase in the physics simulation engine and performs style optimization using an image editing model; a video generation unit, which uses a video generation model to synthesize a continuous time series of the operation task; and a visual motion extraction unit, which uses a point tracking model to track the two-dimensional motion trajectory of the target object. The alignment module from visual motion to robot trajectory is used to extract operable 3D robot trajectories from 2D visual predictions. It includes: a depth estimation unit that estimates the metric depth for each frame using a depth estimation model; a 3D motion fusion unit that fuses the 2D trajectory with the depth information into a coherent 3D motion trajectory using a 3D motion fusion module; and an inverse kinematics mapping unit that generates the grasping posture using a grasping posture generation model and calculates the robot joint angles using an inverse kinematics solver. The dataset construction and policy training module is used to generate training datasets and train the robot to imitate and learn policies. The zero-shot migration deployment module is used to achieve zero-shot migration from simulation to real robot hardware.

Images