A robot simulation data generation method based on three-dimensional Gaussian splashing

Abstract

The application belongs to the technical field of embodied intelligent robots, and particularly relates to a robot simulation data generation method based on three-dimensional Gaussian splashing. The method comprises the following steps: based on multi-view images, reconstructing a Gaussian representation of a static scene through a 3D Gaussian splashing algorithm, and decomposing the Gaussian representation into multiple independent semantic components by using a semantic segmentation algorithm; realizing accurate alignment of a robot body Gaussian model and a URDF model through a two-stage coordinate system alignment procedure; on the basis of completing the coordinate system alignment, constructing a graphics-physical hybrid simulation engine; generating a high-fidelity image sequence from a virtual camera perspective rendering, and pairing the image sequence with corresponding robot action data to form a robot training dataset; and modifying scene parameters in multiple dimensions to expand the diversity of the training dataset. The application can automatically generate large-scale, diversified and highly consistent with the real world in vision training data.

Description

Technical Field

[0001] This application belongs to the field of embodied intelligent robot technology, specifically relating to a method for generating robot simulation data based on three-dimensional Gaussian splashing. Background Technology

[0002] In the field of robotics, the generation of high-fidelity simulation data is crucial for training robust visual motion strategies. Currently, the mainstream technical approach, which combines physical simulators (such as PyBullet or Isaac Sim) with neural rendering methods (such as NeRF), faces inherent limitations such as the visual gap and insufficient dynamic synchronization capabilities.

[0003] In recent years, 3D Gaussian Splatting (3DGS) technology has provided a new approach for building a simulation closed loop that "comes from reality and goes back to reality" (Real-to-Sim-to-Real, or Sim2Real for short) due to its ability to quickly reconstruct high-quality 3D scenes.

[0004] However, most related research focuses on scene reconstruction based on 3DGS. When attempting to combine static 3DGS scenes with dynamic physical simulation to build an automated data generation system, three core challenges arise: Lack of dynamic driving mechanism: Due to the lack of a mechanism to map the physical simulation state to the 3DGS scene in real time and accurately, it is difficult to achieve high-fidelity dynamic interactive rendering.

[0005] Insufficient data augmentation capabilities: Due to the lack of effective decomposition and componentization methods for 3DGS scenes, as well as procedural augmentation methods, it is impossible to generate large-scale and diverse training data from a single scene.

[0006] Insufficient coordinate system alignment accuracy: Existing alignment methods are insufficient to meet the accuracy requirements of high-fidelity driving, resulting in systematic errors between the simulation and 3DGS rendering coordinate systems, making effective driving difficult. Summary of the Invention

[0007] This application proposes a high-fidelity robot simulation data generation scheme based on 3D Gaussian splashing, which solves the problems of lack of dynamic driving mechanism, insufficient data augmentation capability and insufficient coordinate system alignment accuracy in the prior art.

[0008] This application provides a method for generating high-fidelity robot simulation data based on 3D Gaussian splashing, including: Based on multi-view images, a Gaussian representation of a static scene is reconstructed using a 3D Gaussian splashing algorithm, and the Gaussian representation is decomposed into multiple independent semantic components using a semantic segmentation algorithm, including a robot body Gaussian model, an interactive object Gaussian model, and a background Gaussian model. The precise alignment of the robot body Gaussian model and URDF model is achieved through a two-stage coordinate system alignment process, including: generating an initial transformation matrix through coarse alignment based on point cloud matching, and optimizing the transformation matrix through fine alignment based on the 3D Gaussian splash differentiable rendering characteristics. Based on the completion of the coordinate system alignment, a graphics-physics hybrid simulation engine is constructed, in which 3D Gaussian splash is used as the rendering front end and the physics simulator is used as the dynamics back end. Through the state synchronization mechanism, the joint angles and object poses output by the physics simulator are mapped in real time to the pose transformations of the robot body Gaussian model and the interactive object Gaussian model. Based on the background Gaussian model, the transformed robot body Gaussian model, and the interactive object Gaussian model, a high-fidelity image sequence is rendered from the perspective of a virtual camera, and the image sequence is paired with the corresponding robot motion data to form a robot training dataset. By employing procedural enhancement steps based on the independent semantic components, scene parameters are modified in multiple dimensions to expand the diversity of the training dataset.

[0009] In some embodiments of this disclosure, the two-stage coordinate system alignment step includes: The initial transformation matrix is ​​obtained by extracting the center point cloud of the robot body Gaussian model and matching it with the point cloud of the robot URDF model obtained from the physical simulator using the ICP algorithm. Based on the differentiable rendering characteristics of 3D Gaussian splashing, the binarized contour maps of the robot body Gaussian model and the URDF model are rendered from multiple standard perspectives. The L2 loss between the contour maps is used as the optimization objective to iteratively optimize the transformation matrix, thereby achieving fine alignment between the robot body Gaussian model and the URDF model.

[0010] In some embodiments of this disclosure, the programmatic enhancement steps include: Based on the aforementioned graphics-physics hybrid simulation engine, new motion trajectories are generated by programmatically modifying scene parameters, and image sequences are rendered along these new motion trajectories to form diverse new demonstration data.

[0011] In some embodiments of this disclosure, the procedural enhancement steps include object pose and trajectory enhancement: At the start of the simulation, the initial position and orientation of the Gaussian model of the interactive object in the workspace are randomized. Call the motion planner or inverse kinematics solver in the physics simulator to generate a physically feasible motion trajectory from the current robot posture to the grasped object, and further to the target placement area.

[0012] In some embodiments of this disclosure, the procedural enhancement step further includes object type enhancement: A 3D Gaussian object library is pre-built, and an independent Gaussian model is created for each object by scanning real objects or using 3D content generation models. During data generation, a Gaussian model of a new object is randomly selected from the object library with a preset probability to replace the original object's Gaussian model; Generate the corresponding grasping pose and motion trajectory based on the geometric features of the new object.

[0013] In some embodiments of this disclosure, the procedural enhancement step further includes illumination condition enhancement: Random scaling and translation are applied to the spherical harmonic coefficients of all Gaussian elements in the scene to simulate different light intensities, color temperatures, and ambient light. Add noise to the diffuse color channel to increase the realism of the materials.

[0014] In some embodiments of this disclosure, the procedural enhancement steps further include camera view enhancement: Randomly sample the virtual camera positions within a preset radius centered on the workbench; Keep the camera's optical axis pointing toward the center of the workspace, and randomly perturb the camera's orientation within a preset angle range to increase the rendering perspective; Render the image sequence from each new perspective to generate multi-view training data.

[0015] In some embodiments of this disclosure, the procedural enhancement step further includes robot body enhancement: Replace the Gaussian model of the robot body with a 3D Gaussian model of a robot with a different configuration; Load the URDF kinematic model of the new robot and recalculate the transformation relationships of all joint links; Generate a suitable motion trajectory using the inverse kinematics solver of the new robot.

[0016] In some embodiments of this disclosure, the real-time mapping of joint angles and object poses output by the physical simulator to the pose transformation of the robot body Gaussian model and the interactive object Gaussian model includes: The robot body Gaussian model includes multiple linked Gaussian models corresponding to joints. For each link, the transformation matrix relative to the base is calculated using forward kinematics based on the joint angle, and applied to the corresponding linked Gaussian model to update the pose of its Gaussian elements. For an interactive object, the transformation matrix relative to its initial position is calculated based on the object's pose and applied to the corresponding Gaussian model of the interactive object to update the pose of its Gaussian elements.

[0017] In some embodiments of this disclosure, the training dataset is used to train a visual motion policy model, wherein the visual motion policy model, after being deployed on a robot, directly uses images captured by a real camera as input for policy execution to achieve zero-shot Sim2Real transfer.

[0018] In summary, the high-fidelity robot simulation data generation method based on 3D Gaussian splashing provided in the embodiments of this application achieves real-time and accurate mapping from physical simulation state to 3D Gaussian model pose by constructing a graphics-physics hybrid simulation engine and introducing a state synchronization mechanism. This enables the rendering front-end to respond instantly to the output of the dynamics back-end, ensuring a high degree of consistency between the generated image sequence and the robot's motion trajectory. This fundamentally solves the dynamic synchronization problem and achieves high-fidelity dynamic rendering consistency. By programmatically modifying scene parameters (such as object pose, lighting conditions, and camera viewpoint), automated, multi-dimensional data augmentation is achieved, generating large-scale, diverse new demonstration data. This effectively overcomes the limitations of real-world data acquisition and significantly improves the generalization ability and robustness of the strategy. By performing a multi-coordinate system alignment step, combined with coarse ICP alignment and fine optimization based on differentiable rendering, the transformation matrix is ​​iteratively optimized to achieve high-precision alignment between the robot's Gaussian model and URDF model. This effectively avoids state transfer errors, laying a solid foundation for zero-shot Sim2Real transfer and significantly bridging the gap between simulation and reality. Finally, by using the generated dataset to train the visual motion policy model and directly using real camera images as input after deployment, zero-shot transfer was successfully achieved, significantly reducing data acquisition costs and deployment complexity, and constructing an efficient and low-cost data generation pipeline. (See attached figures.) The features and advantages of this application will be more clearly understood by referring to the accompanying drawings, which are illustrative and should not be construed as limiting the application in any way. In the drawings: Figure 1 This is the front-end processing flow of the high-fidelity robot simulation data generation method based on 3DGS proposed in this application; Figure 2 This is the back-end generation process of the high-fidelity robot simulation data generation method based on 3DGS proposed in this application; Figure 3 This is a schematic diagram of a computer system to which this application applies; Figure 4 This is a flowchart illustrating a method for generating high-fidelity robot simulation data based on 3D Gaussian splashing, according to some embodiments of this application. Detailed Implementation

[0019] In the following detailed description, numerous specific details of this application are illustrated by example to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that this application can be practiced without these details. It should be understood that the terms “system,” “apparatus,” “unit,” and / or “module” used in this application are one way of distinguishing different parts, elements, sections, or components at different levels in a sequential arrangement. However, these terms may be replaced with other expressions if other expressions can achieve the same purpose.

[0020] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly connected to or coupled to or communicate with other devices, units, or modules, or there may be intermediate devices, units, or modules present, unless the context explicitly indicates otherwise. For example, the term "and / or" as used herein includes any one and all combinations of one or more of the relevant listed items.

[0021] The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate that explicitly identified features, integrals, steps, operations, elements, and / or components are included, and such expressions do not constitute an exclusive list, and other features, integrals, steps, operations, elements, and / or components may also be included.

[0022] Referring to the following description and accompanying drawings, these and other features and characteristics, operating methods, functions of related structural elements, combinations of parts, and economics of manufacture of this application can be better understood, wherein the description and drawings form part of the specification. However, it is clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. It is understood that the drawings are not drawn to scale.

[0023] Various structural diagrams are used in this application to illustrate various variations of the embodiments according to this application. It should be understood that the preceding or following structures are not intended to limit this application. The scope of protection of this application is determined by the claims.

[0024] The learning of robot visual motion strategies, especially imitation learning and reinforcement learning, heavily relies on large-scale, high-quality training data. Currently, there are two main approaches to acquiring such data: real-world data collection and physical simulation generation. Among these, physical simulation generation is increasingly becoming the mainstream method.

[0025] Physical simulation data generation refers to generating data using simulators such as Mujoco, PyBullet, and Isaac Sim. This method is low-cost, highly efficient, can be parallelized on a large scale, and is absolutely safe. However, its core bottleneck lies in the huge gap between simulation and reality. This is mainly due to the following factors: The visual gap: Traditional simulators mostly render based on polygonal meshes, and the images they generate differ significantly from the real world in terms of texture, lighting, shadows, reflections, and materials. Visual strategies trained on these "fake" images often become "blind" to images captured by real cameras, causing the strategies to fail.

[0026] The Physics Gap: Physical parameters (such as friction, stiffness, and damping) in simulators are difficult to model accurately, especially in complex contact dynamic tasks, and deviate from real physical behavior.

[0027] Asset creation is difficult: Creating a high-fidelity 3D model (mesh) for a simulation environment is a time-consuming and labor-intensive process that requires professional 3D modeling skills.

[0028] To bridge the Sim2Real gap, existing technologies have explored various approaches. Domain randomization enhances the robustness of strategies by randomizing visual and physical parameters (such as lighting, texture, color, and friction) in simulations, but this is a brute-force approach that does not fundamentally improve the realism of the simulation. Two-dimensional image augmentation methods, such as style transfer on real images or modification of scene elements (such as background and robot appearance) through generative models (GANs, Diffusion Models), lack consistency in three-dimensional space, easily generate "fake" data that does not conform to physical laws, and have limited dimensional augmentation capabilities.

[0029] In recent years, the rise of new view synthesis technologies such as Neural Radiation Field (NeRF) and 3D Gaussian Splatting has provided new possibilities for creating high-fidelity digital replicas of the real world. 3DGS technology can quickly reconstruct photorealistic 3D scenes from multiple viewpoints with fast rendering speed. Applying 3DGS technology to robot simulation, constructing a closed loop of "from reality to reality" (Real2Sim2Real), holds the promise of fundamentally solving the visual gap problem.

[0030] However, effectively combining static 3DGS scene representation with dynamic physical simulation processes, and building upon this foundation a system capable of automatically, scalably, and diversely generating robot interaction data, remains a pressing technical challenge. Existing sporadic attempts mostly focus on scene reconstruction itself, lacking a systematic approach to decompose the scene, align coordinate systems, drive dynamic interactions, and on this basis, perform large-scale, multi-dimensional data augmentation and synthesis.

[0031] Therefore, there is an urgent need for a new technical solution that can fully utilize the rendering advantages of 3DGS and the dynamic simulation advantages of traditional physics engines to build an efficient, low-cost, and high-fidelity robot data generation pipeline, thereby significantly improving the Sim2Real transfer performance of visual strategies.

[0032] In view of this, this application proposes a method for generating high-fidelity robot simulation data based on 3DGS. The method first reconstructs a high-fidelity 3DGS scene from multi-view images and decomposes it into independent semantic components using semantic segmentation. Then, through a two-stage coordinate system alignment strategy, it achieves high-precision matching between the 3DGS scene and the physical simulation coordinate system, laying the foundation for dynamic driving.

[0033] The complete implementation process of the method is as follows: Figure 1 and Figure 2 As shown, both constitute the entire process from static scene construction to dynamic data generation. Specifically: Figure 1 The core steps of high-fidelity scene reconstruction, decomposition, and alignment are shown, covering the front-end processing flow from data acquisition to achieving precise coordinate system alignment.

[0034] Figure 2 This demonstrates the core steps of hybrid rendering and data augmentation compositing, namely... Figure 1 Based on the output, a backend generation process is implemented that dynamically drives, renders, and enhances data.

[0035] By organically combining the above processes, this application systematically solves the problem of combining 3DGS static scenes with dynamic physical simulation, and achieves the goal of automatically and on a large scale generating high-quality simulation data.

[0036] Figure 3 This is a schematic diagram of a computer system to which this application applies. Figure 3The computer system shown includes a simulation data generation server connected to a policy training unit and multiple image acquisition devices. Based on multi-view image data collected by the image acquisition devices and relying on its integrated physical simulation environment, the simulation data generation server constructs a large-scale robot simulation training dataset through programmatic multi-dimensional data augmentation. This dataset is used by the policy training unit to train visual motion strategies. Wherein: The image acquisition device is used to collect synchronous, multi-view visual data of the robot's working scene in a real-world environment. The acquired RGB-D image sequence is the core data source for high-fidelity 3D Gaussian splash scene reconstruction, directly determining the visual realism of subsequent simulation data. Physically, it manifests as a group of consumer-grade or industrial-grade cameras (such as smartphones or RGB-D depth cameras) deployed around the real robot's workspace. Their deployment location and number need to be flexibly configured according to the field-of-view coverage requirements of the specific task scenario, typically distributed around the workbench to ensure visual coverage of key operating areas.

[0037] The simulation data generation server serves as the core computing and processing unit of the system. Its core function is to execute a complete "reconstruction-alignment-drive-enhancement" pipeline: taking multi-view images as input, it reconstructs and decomposes static scenes through a 3D Gaussian splashing algorithm; it integrates a physical simulation environment to perform dynamic state calculations and drives the scene to perform high-fidelity dynamic rendering through a state synchronization mechanism; and finally, it generates large-scale, highly diverse robot simulation training datasets in batches through programmatic multi-dimensional data enhancement.

[0038] The simulation data generation server can be a standalone, clustered, or distributed server. Its integrated physical simulation environment (such as the PyBullet engine) runs as a software module within the server.

[0039] The policy training unit is responsible for training the robot's visual motion policy model using machine learning algorithms (such as imitation learning and reinforcement learning) based on a large-scale training dataset provided by the simulation data generation server. Its goal is to produce control policies that can be directly deployed to real robots and achieve zero-shot Sim2Real transfer.

[0040] The strategy training unit can be flexibly deployed according to the computing load and project stage. It can be deployed as a software service and simulation data generation server to share hardware resources, or it can be deployed as an independent unit to obtain training datasets online or offline.

[0041] Figure 4This is a flowchart illustrating a method for generating high-fidelity robot simulation data based on 3D Gaussian splashing, according to some embodiments of this application. In some embodiments, the method for generating high-fidelity robot simulation data based on 3D Gaussian splashing is... Figure 3 The simulation data generation server shown executes the method, which includes the following steps: S410, based on multi-view images, reconstructs the Gaussian representation of the static scene using the 3D Gaussian splashing algorithm, and uses a semantic segmentation algorithm to decompose the Gaussian representation into multiple independent semantic components, including the robot body Gaussian model, the interactive object Gaussian model, and the background Gaussian model.

[0042] like Figure 1 As shown: The first step is the acquisition of static scene data. In some embodiments of this application, a consumer-grade camera (such as a smartphone) is used to capture a video or a set of static images of a real-world robot work scene (including the robot body, workbench, background, and several initial objects). Then, using the acquired image data, a Gaussian representation of the entire static scene is reconstructed using a 3DGS algorithm. Finally, scene decomposition is performed, and the reconstructed... It is decomposed into multiple independent, manipulable semantic spaces. Specifically, this includes: background Gaussian... Gaussian robot body And Gaussian of each interactive object in the scene. .

[0043] The decomposition operation is implemented using existing segmentation algorithms (such as SAM), and is completed by obtaining the masks of the robot and the object. Semantic segmentation.

[0044] S420, the precise alignment of the robot body Gaussian model and URDF model is achieved through a two-stage coordinate system alignment step, including: generating an initial transformation matrix based on coarse alignment of point cloud matching, and optimizing the transformation matrix based on the 3D Gaussian splash differentiable rendering characteristics.

[0045] To drive scene reconstruction, the arbitrary coordinate system of 3DGS must be... With physics simulator / real-world robot base coordinate system Precise alignment. For example... Figure 1 As shown, this process consists of two steps. The first step is coarse alignment, which is achieved by extracting the robot's Gaussian... The center point cloud is matched with the robot's URDF model point cloud (obtained from the simulator) using the Iterative Closest Point (ICP) algorithm to obtain an initial transformation matrix. The second step is fine alignment. Utilizing the differentiable rendering characteristics of 3DGS, the alignment transformation matrix is ​​optimized to minimize the differences (such as L2 loss) between the robot's 3DGS model contour (mask) rendered from multiple standard perspectives and the contour rendered from URDF. This step significantly improves alignment accuracy and is crucial for achieving precise motion control. After alignment, the... Further decomposed into each individual joint link This prepares for subsequent kinematic actuation.

[0046] S430, based on the completion of the coordinate system alignment, a graphics-physics hybrid simulation engine is constructed, in which 3D Gaussian splash is used as the rendering front end and the physics simulator is used as the dynamics back end. Through the state synchronization mechanism, the joint angles and object poses output by the physics simulator are mapped in real time to the pose transformations of the robot body Gaussian model and the interactive object Gaussian model.

[0047] After aligning the coordinate system, this step builds a graphics-physics hybrid simulation engine to achieve real-time mapping from physical states to graphical representations. like Figure 2 As shown, the physics simulator, acting as the dynamics backend, is responsible for simulating the task flow. At each time step t, it outputs the state of all dynamic objects in the scene, including the angles of each joint of the robot. And the pose (position and orientation) of an object. .

[0048] Then, through a state synchronization mechanism, the joint angles and object poses output by the physical simulator are mapped in real time to the pose transformation of the Gaussian model. Specifically: for the robot, based on the joint angles... Using forward kinematics, the transformation matrix of each link j relative to the base is calculated. .Will Applied to the corresponding Gaussian model Update the pose of all its Gaussian elements. For an object, based on the object pose provided by the simulator. Calculate its transformation matrix relative to the initial position. And apply it to the corresponding Gaussian model of the object. Update the pose of its Gaussian elements.

[0049] S440, based on the background Gaussian model, the transformed robot body Gaussian model, and the interactive object Gaussian model, a high-fidelity image sequence is rendered from the perspective of a virtual camera, and the image sequence is paired with the corresponding robot motion data to form a robot training dataset.

[0050] Specifically, the first step is to perform scene compositing, which involves combining all the transformed dynamic components ( , ) and static background Reassemble; then render from the perspective of one or more virtual cameras to generate a high-fidelity image of the current time step t. .

[0051] Finally, the rendered image sequence Robot motion data corresponding to the physics simulator At (e.g., end effector pose or joint angle) are paired to form a training data pair. , At The final output consists of a large number of such data pairs, forming a training dataset for robot vision motion strategies.

[0052] S450, through procedural enhancement steps based on the independent semantic components, the scene parameters are modified in multiple dimensions to expand the diversity of the training dataset.

[0053] Based on the 3DGS scenes generated by S440, this application can perform multi-dimensional data augmentation on single or small amounts of real-world demo data, generating more new demo data. Augmentation dimensions include: 1. Object pose and trajectory enhancement: Randomize the initial pose of the object at the start of the simulation, and then use a motion planner or inverse kinematics solver to generate new, physically feasible robot motion trajectories.

[0054] 2. Object Type Enhancement: Establish a 3DGS object library (obtainable by scanning real objects or using 3D content generation models). When generating data, randomly select a new object from the library. Replace the original object It generates the corresponding grabbing pose and motion trajectory.

[0055] 3. Scene Appearance Enhancement: Gaussian rendering of the background Replace it with another pre-reconstructed 3DGS scene.

[0056] 4. Enhanced lighting conditions: Simulate different lighting intensities, color temperatures, and ambient light by randomly scaling, translating, and adding noise to the spherical harmonics coefficients or diffuse colors of all Gaussian elements in the scene.

[0057] 5. Enhanced Camera Viewpoint: Leveraging the ability of 3DGS to render from any new viewpoint, the position and orientation (towards the workspace) of the virtual camera are randomized during simulation to generate multi-viewpoint data, thereby improving the robustness of the strategy to viewpoint changes.

[0058] 6. Robot body enhancement: Replace the 3DGS model with that of another robot and use the kinematic model of the new robot to generate trajectories, thereby enabling policy learning across robot platforms.

[0059] One embodiment of this application demonstrates the practical deployment and verification of the methods described in S410-S450. Specifically: System hardware environment: This system can be deployed on standard high-performance computing workstations, with a typical configuration as follows: CPU: Intel Core i9 / AMD Ryzen 9 Memory: 64GB DDR4 or higher GPU: NVIDIA RTX 3090 / 4090 or equivalent graphics card (for 3DGS reconstruction and rendering) Storage: High-speed NVMe SSD System software environment: Operating System: Ubuntu 20.04 LTS Core algorithm libraries: PyTorch, CUDA 3DGS Implementation: Based on the open-source 3DGaussianSplatting project Physics simulator: PyBullet or NVIDIA Isaac Sim can be selected. Robot Control: ROS (Robot Operating System) Detailed implementation process: Step 1: Scene Reconstruction and Asset Preparation First, real-world scene capture is performed. An operator uses a smartphone to walk around a workbench equipped with a Franka Emika Panda robotic arm, recording approximately 60 seconds of 4K video. A toy cube to be grasped and a target placement area are placed on the workbench. The video is then input into the 3DGS reconstruction pipeline, where approximately 200 still images are extracted from the video. COLMAP software is used to calculate the camera pose, which is then input into the 3DGS reconstruction pipeline to generate an initial Gaussian cloud file (.ply format) for the entire scene.

[0060] After reconstruction, scene decomposition is performed. For the robot, the Franka robotic arm's URDF file is loaded, and its mesh model is generated in zero pose. By aligning this mesh with the scene's Gaussian cloud, the Gaussian point cloud belonging to the robot can be initially segmented. For the object, the toy cubes are manually or using SAM (SegmentAnythingModel) to annotate the image. These 2D masks are then used to isolate the Gaussian model of the toy cubes during 3DGS reconstruction. The remaining Gaussian point clouds were classified as background. .

[0061] In addition, it can scan multiple different objects (such as apples, screwdrivers, and beverage cans) separately, create an independent 3DGS model for each object, and store it in the object library for later enhancement.

[0062] Step 2: Coordinate System Alignment and Dynamic Model Construction The core of this step is coordinate system alignment. First, the ICP algorithm is used to align the segmented... The point cloud is coarsely aligned with the URDF model point cloud. A fine alignment process based on differentiable rendering is then initiated. This process renders simultaneously from 10 fixed camera positions. The binarized contour maps of the URDF model are used, and the pixel difference loss between the contour maps is backpropagated. The SE(3) transformation matrix between the two coordinate systems is iteratively optimized until the loss converges. Finally, a high-precision alignment relationship is obtained. After alignment, the aligned contour maps are... Based on the URDF link information, it is further segmented into 7 joint links and Gaussian models corresponding to the end effectors. ,..., , }

[0063] Step 3: Data Synthesis and Augmentation Once the preparations are complete, the graphics-physics hybrid simulation engine can be launched to generate training data. This example starts with a simple expert trajectory of "grabbing a block and placing it in the target area" (generated using a motion planner in the simulator), generating 10,000 augmented data points.

[0064] For each new data point generated, the system randomly combines multiple enhancement types. For example, for viewpoint enhancement, a new camera pose is randomly sampled on a sphere with the center of the worktable as its center and a radius between 0.8m and 1.2m; for object enhancement, a new object (such as an apple) is randomly selected from the object library with a 30% probability. Replace the block Alternatively, illumination enhancement can be performed by applying a random multiplication factor (0.7-1.3) and an additive bias (-0.1-0.1) to the spherical harmonic coefficients of all Gaussian points to simulate different brightness and contrast.

[0065] After determining the enhancement type, the system generates a new motion trajectory. First, the initial position and pose of the object (block or apple) on the table are randomized. Then, the inverse kinematics solver in the physics simulator (PyBullet) is invoked to plan a new motion trajectory from the current robot pose to grasping the object and placing it in the target area. The trajectory consists of a series of joint angles { , ,..., } and object pose { , ,..., }composition.

[0066] Finally, the system renders a sequence of images along the new trajectory, for each time step t in the trajectory: The system calculates the transformation matrix for each joint link using forward kinematics, and applies the transformation matrix to { ,..., Applying object pose transformation to Combining the Gaussian model after all transformations with a static background. Render a 224x224 RGB image from the camera viewpoint selected in step 1. .

[0067] The rendered image Corresponding robot actions (For example, the relative pose change of the end effector) is saved as a data pair.

[0068] Step Four: Strategy Training and Real-World Deployment The generated dataset will be used for policy training. The generated 10,000 trajectory data points (approximately 100 data pairs per point) will be fed into a Transformer-based imitation learning model (such as ACT). Training will take approximately 10 hours on a single NVIDIA RTX 4090 GPU.

[0069] After training, the trained model weights are loaded onto the control computer connected to the real Franka robotic arm. The robot is then started, allowing it to observe the scene through a real camera.

[0070] In the testing phase, when toy blocks were randomly placed on a real workbench, the strategy was activated, and the robot was able to reliably identify the blocks, plan its path, and successfully complete the grasping and placement tasks. Further testing showed that even when the blocks were replaced with apples not seen during training, or under different lighting conditions, the strategy maintained a high success rate, verifying the effectiveness of the data generated by this invention and its Sim2Real transfer capabilities.

[0071] Through the above embodiments, this application fully demonstrates how to start from real-world scenarios, utilize 3DGS and physics engines to build a powerful data generation factory, provide high-quality, large-scale, and diversified "nutrients" for robot policy learning, and ultimately achieve robust real-world deployment.

[0072] In summary, the high-fidelity robot simulation data generation method based on 3D Gaussian splashing provided in the embodiments of this application achieves real-time and accurate mapping from physical simulation state to 3D Gaussian model pose by constructing a graphics-physics hybrid simulation engine and introducing a state synchronization mechanism. This enables the rendering front-end to respond instantly to the output of the dynamics back-end, ensuring a high degree of consistency between the generated image sequence and the robot's motion trajectory. This fundamentally solves the dynamic synchronization problem and achieves high-fidelity dynamic rendering consistency. By programmatically modifying scene parameters (such as object pose, lighting conditions, and camera viewpoint), automated and multi-dimensional data augmentation is achieved, generating large-scale and diverse new demonstration data. This effectively overcomes the limitations of real data acquisition and significantly improves the generalization ability and robustness of the strategy. By performing multi-coordinate system alignment steps, combined with ICP coarse alignment and fine optimization based on differentiable rendering, the transformation matrix is ​​iteratively optimized to achieve high-precision alignment between the robot's Gaussian model and URDF model. This effectively avoids state transfer errors and lays a solid foundation for zero-shot Sim2Real transfer, significantly bridging the gap between simulation and reality. Finally, by using the generated dataset to train the visual motion policy model and directly using real camera images as input after deployment, zero-shot transfer was successfully achieved. This significantly reduces data acquisition costs and deployment complexity, and constructs an efficient and low-cost data generation pipeline. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding descriptions in the foregoing device embodiments, and will not be repeated here.

[0073] Although the subject matter described herein is provided in the general context of execution on a computer system in conjunction with an operating system and applications, those skilled in the art will recognize that other implementations can also be executed in conjunction with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will understand that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframes, etc., and can also be used in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may reside on both local and remote memory storage devices.

[0074] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0075] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of this application and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of this application should be included within the protection scope of this application. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A method for generating high-fidelity robot simulation data based on 3D Gaussian splashing, characterized in that, include: Based on multi-view images, a Gaussian representation of a static scene is reconstructed using a 3D Gaussian splashing algorithm, and the Gaussian representation is decomposed into multiple independent semantic components using a semantic segmentation algorithm, including a robot body Gaussian model, an interactive object Gaussian model, and a background Gaussian model. The precise alignment of the robot body Gaussian model and URDF model is achieved through a two-stage coordinate system alignment process, including: generating an initial transformation matrix through coarse alignment based on point cloud matching, and optimizing the transformation matrix through fine alignment based on the 3D Gaussian splash differentiable rendering characteristics. Based on the completed coordinate system alignment, a graphics-physics hybrid simulation engine is constructed, in which 3D Gaussian splashing serves as the rendering front-end and the physics simulator serves as the dynamics back-end. Through a state synchronization mechanism, the joint angles and object poses output by the physics simulator are mapped in real time to the pose transformations of the robot body Gaussian model and the interactive object Gaussian model. The robot body Gaussian model contains multiple linked Gaussian models corresponding to joint links. The state synchronization mechanism includes: for each link, calculating the transformation matrix relative to the base through positive kinematics based on the joint angle, and applying it to the corresponding link Gaussian model to update the pose of its Gaussian elements; for the interactive object, calculating its transformation matrix relative to the initial position based on the object pose, and applying it to the corresponding interactive object Gaussian model to update the pose of its Gaussian elements. Based on the background Gaussian model, the transformed robot body Gaussian model, and the interactive object Gaussian model, a high-fidelity image sequence is rendered from the perspective of a virtual camera, and the image sequence is paired with the corresponding robot motion data to form a robot training dataset. The scene parameters are modified in multiple dimensions through procedural enhancement steps based on the independent semantic components to expand the diversity of the training dataset. The procedural enhancement steps are based on the graphics-physics hybrid simulation engine. By procedurally modifying the scene parameters, new motion trajectories are generated, and image sequences are rendered along the new motion trajectories. The procedural enhancement steps include object pose and trajectory enhancement: at the start of the simulation, the initial position and posture of the Gaussian model of the interactive object in the workspace are randomized, and the motion planner or inverse kinematics solver in the physics simulator is called to generate a physically feasible motion trajectory from the current robot posture to the grasped object, and further to the target placement area.

2. The method according to claim 1, characterized in that, The two-stage coordinate system alignment step includes: The initial transformation matrix is ​​obtained by extracting the center point cloud of the robot body Gaussian model and matching it with the point cloud of the robot URDF model obtained from the physical simulator using the ICP algorithm. Based on the differentiable rendering characteristics of 3D Gaussian splashing, the binarized contour maps of the robot body Gaussian model and the URDF model are rendered from multiple standard perspectives. The L2 loss between the contour maps is used as the optimization objective to iteratively optimize the transformation matrix, thereby achieving fine alignment between the robot body Gaussian model and the URDF model.

3. The method according to claim 1, characterized in that, The procedural enhancement steps also include object type enhancement: A 3D Gaussian object library is pre-built, and an independent Gaussian model is created for each object by scanning real objects or using 3D content generation models. During data generation, a Gaussian model of a new object is randomly selected from the object library with a preset probability to replace the original object's Gaussian model; Generate the corresponding grasping pose and motion trajectory based on the geometric features of the new object.

4. The method according to claim 1, characterized in that, The procedural enhancement steps also include illumination condition enhancement: Random scaling and translation are applied to the spherical harmonic coefficients of all Gaussian elements in the scene to simulate different light intensities, color temperatures, and ambient light. Add noise to the diffuse color channel to increase the realism of the materials.

5. The method according to claim 1, characterized in that, The procedural enhancement steps also include camera view enhancement: Randomly sample the virtual camera positions within a preset radius centered on the workbench; Keep the camera's optical axis pointing toward the center of the workspace, and randomly perturb the camera's orientation within a preset angle range to increase the rendering perspective; Render the image sequence from each new perspective to generate multi-view training data.

6. The method according to claim 1, characterized in that, The programmed enhancement steps also include robot body enhancement: Replace the Gaussian model of the robot body with a 3D Gaussian model of a robot with a different configuration; Load the URDF kinematic model of the new robot and recalculate the transformation relationships of all joint links; Generate a suitable motion trajectory using the inverse kinematics solver of the new robot.

7. The method according to claim 1, characterized in that: The training dataset is used to train a visual motion policy model, which, after being deployed on the robot, directly uses images captured by a real camera as input for policy execution to achieve zero-shot Sim2Real transfer.

Images