Editable robot operation simulation system and method

Abstract

The invention provides an editable robot operation simulation system and method, and relates to the technical field of robot simulation systems.The system comprises a dynamic semantic enhancement modeling module which reconstructs a scene from an input video through a Gaussian representation method, expands the reconstructed scene through time and semantic information and outputs a scene space-time dynamic Gaussian model; the space-time dynamic Gaussian model executes an extensible editing operation according to a user command, and outputs edited scene simulation data; the automatic simulation module converts a user command into structured configuration through multi-agent cooperation, and calls the editable video simulation module to generate robot simulation operation required by the structured configuration; and the robot operation learning module analyzes key frames of fault cases of robot simulation operation by using a visual language model, provides explanation for potential fault reasons, and outputs a simulation solution. According to the invention, the difference between simulation of the robot operation simulation platform and reality can be reduced, and the expansion capability of robot simulation is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot simulation systems, and in particular to an editable robot operation simulation system and method. Background Art

[0002] With the rapid development of industrial automation and intelligent manufacturing, the role of robot operation simulation systems in production line planning, process verification and personnel training has become increasingly prominent. However, training in an actual robot operation environment requires not only a large amount of money to purchase and maintain real robot equipment, but also the provision of special venues and supporting facilities for training, which undoubtedly increases the training cost. Therefore, studying general robot operation strategies is of great significance to promoting automated intelligent manufacturing.

[0003] The development of general robotic manipulation strategies has made significant progress, driven by large-scale demonstration data across diverse environments. However, the high cost and low efficiency of collecting real-world demonstrations hinder the scalability of data acquisition. While existing robotic manipulation simulation platforms provide a controlled and versatile environment for developing advanced robotic learning methods, the gap between simulation and reality remains a significant obstacle. While existing technologies have also explored the reconstruction and editing capabilities of 3D Gaussian layers (3DGS) and demonstrated some capabilities, developing systematic pipelines to integrate these technologies into the robotics field, particularly expanding the capabilities of robotic simulation, remains an unexplored area. Therefore, narrowing the gap between simulation and reality in robotic manipulation simulation platforms is a crucial technical challenge to be addressed. Summary of the Invention

[0004] In order to solve the problems that the above-mentioned existing technologies are difficult to narrow the gap between the simulation of the robot operation simulation platform and reality and the expansion capability of the robot simulation is poor, the present invention proposes an editable robot operation simulation system and method, which can effectively narrow the gap between the simulation of the robot operation simulation platform and reality and improve the expansion capability of the robot simulation.

[0005] In order to achieve the above technical effects, the technical solutions of the present invention are as follows: An editable robot operation simulation system, comprising: The dynamic semantic enhancement modeling module is used to reconstruct the scene of the input video using Gaussian representation, expand the reconstructed scene with temporal and semantic information, and output a spatiotemporal dynamic Gaussian model of the scene; An editable video simulation module, configured to execute scalable editing operations on the spatiotemporal dynamic Gaussian model according to user commands and output edited scene simulation data; an automated simulation module for converting user commands into structured configurations through multi-agent collaboration and mobilizing the editable video simulation module to generate robot simulation operations required for the structured configurations; The robot operation learning module is used to use a preset visual language model to analyze the key frames of the fault cases of the robot simulation operation, provide explanations for potential fault causes, and output simulation solutions.

[0006] Preferably, the reconstructed scene is expanded using temporal and semantic information, the object positions in the reconstructed scene are expanded into a covariance matrix, and the spatiotemporal dynamic Gaussian model of the scene is constructed by expanding the covariance matrix into a 4D ellipsoid as follows:

[0007] in, Represents the view rendering result output by the scene spatiotemporal dynamic Gaussian model, where N is the number of Gaussians. is the opacity of the i-th Gaussian, is the opacity of the j-th Gaussian, is the color of the i-th Gaussian, t represents the time, represents the visibility of the i-th Gaussian at time t, represents the probability distribution of the i-th Gaussian at pixel position x′, represents the visibility of the i-th Gaussian at time t, represents the probability distribution of the i-th Gaussian at pixel position x′; The parameters of the scene spatiotemporal dynamic Gaussian model are encoded as identity Further expansion of , we can get 2D identity features in dynamic scenes as follows:

[0008] in, is the identity label of the i-th Gaussian.

[0009] Preferably, in the process of reconstructing the scene, the scene spatiotemporal dynamic Gaussian model is trained using a total loss function L, and the calculation expression of the total loss function L is as follows:

[0010] in, represents the mean square error loss of 2D rendering, Represents the weight coefficient of the mean square error loss term for 2D rendering, represents the standard cross entropy loss, represents the weight coefficient of the standard cross entropy loss term, represents the KL divergence loss, Represents the weight coefficient of the KL divergence loss term.

[0011] Preferably, the extensible editing operation includes retrieval of target objects, object deletion, object insertion and object modification.

[0012] Preferably, the search target objects include: According to user commands, the open world detection model is used to identify the mask ID corresponding to the target 2D object and obtain the corresponding mask ID; The mask ID is matched with the segmentation mask rendered by the scene spatiotemporal dynamic Gaussian model. When the mask matching result cannot cover the fine-grained part of the target object, the incremental semantic distillation method is used, specifically: Render the 2D object mask corresponding to the retrieved Gaussian representation, and use the open-world detection model to verify whether the 2D object mask corresponds to the target object required by the user command. If the verification fails, use the bounding box as a hint to perform finer-grained segmentation of the input video through the semantic segmentation model to obtain new fine-grained labels. Only the new fine-grained labels are used to fine-tune the identity encoding to gradually extract new object semantics.

[0013] Preferably, the object deletion includes: determining a 3D object to be removed, and deleting a 3D object Gaussian function corresponding to the 3D object to remove the 3D object in a simulated environment, inspecting the background area after removing the 3D object in the simulated environment, identifying fuzzy holes caused by insufficient observation, detecting and locating the fuzzy holes using an open world detection model, and for each view containing a fuzzy hole, applying a LAMA repair algorithm to repair the fuzzy hole to generate a repaired view, generating a new Gaussian function near the deleted area to simulate the space originally occupied by the removed 3D object, and fine-tuning the new Gaussian function using the repaired view.

[0014] Preferably, the object insertion includes: obtaining a large-scale Gaussian data set as an initial database, retrieving an insertion object from the initial database, inserting the insertion object into the original surrounding scene, and adjusting the position and size of the insertion object to adapt to the insertion position, comparing the color of the insertion object with that of the original surrounding scene, and when the color of the insertion object is inconsistent with that of the original surrounding scene, using the libcom algorithm to render the insertion object and the original surrounding scene to obtain a rendered image, and using the rendered image to fine-tune the spherical harmonic function of the Gaussian distribution of the insertion object.

[0015] Preferably, the object modification includes 3D object size modification, 3D object position modification, 3D object color modification, 3D object style / texture modification and physical simulation.

[0016] Preferably, the multi-agent collaboration includes a video simulation master agent, a retrieval agent, a scene editing agent, a 3D asset agent, a scene optimization agent, and a scene rendering agent; The video simulates a general agent that decomposes user commands into simplified, specific natural language instructions and assigns tasks to other agents. The retrieval agent is configured to receive the retrieval agent task assigned by the video simulation master agent, process the prompt in the form of a preset location code, and use an object retrieval operator; The scene editing agent is used to receive the scene editing agent task assigned by the video simulation general agent, process the operation prompt of the preset color code, and apply the simulation operator to realize the required simulation operation; The 3D asset agent is used to receive 3D asset agent tasks assigned by the video simulation general agent and organize and retrieve 3D assets according to user specifications; The scene refining agent is used to receive the scene refining agent task assigned by the video simulation general agent and refine the scene; The scene rendering agent is used to receive the scene rendering agent task assigned by the video simulation general agent and generate external camera parameters for scene rendering.

[0017] The present invention also proposes an editable robot operation simulation method, comprising the following steps: S1. Reconstruct the scene from the input video using Gaussian representation, expand the reconstructed scene with temporal and semantic information, and output a spatiotemporal dynamic Gaussian model of the scene; S2 for the spatiotemporal dynamic Gaussian model according to user commands, perform scalable editing operations, output edited scene simulation data; S3 is used to convert user commands into a structured configuration through multi-agent collaboration, and mobilize the editable video simulation module to generate the robot simulation operation required for the structured configuration; S4. is used to analyze key frames of the fault case of the robot simulation operation using a preset visual language model, provide explanations for potential fault causes, and output simulation solutions.

[0018] Compared with the prior art, the technical solution of the present invention has the following beneficial effects: The present invention proposes an editable robot operation simulation system and method. First, the dynamic semantic enhancement modeling module reconstructs the scene of the input video using Gaussian representation, expands the reconstructed scene with time and semantic information, and outputs a spatiotemporal dynamic Gaussian model of the scene, which significantly improves the accuracy and scalability of scene reconstruction and effectively narrows the gap between simulation and reality; secondly, the editable video simulation module supports users to flexibly edit the scene, thereby enhancing the adaptability and scalability of robot operation simulation; then, the automated simulation module is used to realize the automatic configuration and execution of user commands through multi-agent collaboration, which greatly reduces the operation threshold and improves the simulation efficiency; further, the robot operation learning module is combined with the visual language model to perform intelligent analysis and interpretation of fault cases, providing data support for the optimization of simulation solutions, which can effectively analyze learning problems and generate corresponding simulation requirements, thereby improving the expansion capability of robot simulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A block diagram of an editable robot operation simulation system proposed in an embodiment of the present invention is shown; Figure 2 A diagram showing technical details of implementing editable robot operation video simulation using RoboPearls, as proposed in an embodiment of the present invention; Figure 3 A diagram showing the technical details of some submodules of RoboPearls proposed in an embodiment of the present invention; Figure 4 The figure shows the overall framework of RoboPearls proposed in the embodiment of the present invention; Figure 5 A qualitative example diagram showing the generation of action sequences in a simulation environment and the real world proposed in an embodiment of the present invention; Figure 6 A diagram showing a qualitative simulation of the RLbench simulation dataset proposed in an embodiment of the present invention; Figure 7 A diagram showing a qualitative simulation of a real-world dataset proposed in an embodiment of the present invention; Figure 8 A flowchart of an editable robot operation simulation method proposed in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0020] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent; It is understandable to those skilled in the art that descriptions of certain well-known contents may be omitted in the drawings.

[0021] To facilitate understanding of this embodiment, first, the prior art information of this embodiment is introduced as follows: For simulation platforms for robotic manipulation, to advance general-purpose robotic manipulation strategies, various physics-based simulation platforms have been developed to provide efficient and scalable methods to scale specific data. Based on physics engines such as Isaac Sim, PyBullet, and MuJoCo, many robotic manipulation environments and various skills have been developed, covering rigid body manipulation and soft body manipulation for simulating deformable robots, objects, and fluids. In addition, some work has also established standardized simulation benchmarks. These works greatly reduce the dependence on expensive real-world data collection. However, despite the tremendous efforts, the gap between simulation and reality remains a major challenge. Recently, there has been significant progress in the commercialization of robots. For example, Hi Robot and Helix employ visual language action (VLA) models based on visual language models (VLMs). However, collecting visual language data or scaling existing datasets for these models remains a challenge that traditional simulation methods cannot solve.

[0022] For 3DGS-based scene simulation, scene reconstruction and simulation has been a long-standing research problem. Currently, 3D Gaussian Layers (3DGS) has revolutionized this field by providing explicit representations and high-fidelity real-time rendering, greatly expanding the capabilities of NeRF. In addition to fast rendering, the explicit representation of 3DGS also facilitates a series of downstream tasks, including dynamic reconstruction, geometric editing, physical simulation, and scene understanding. Recently, several studies have explored using 3DGS for robotic manipulation tasks. For example, GaussianGrasper and GraspSplats utilize feature layers to support grasp queries through language, while Robo-GS integrates 3D Gaussian kernels to enhance digital asset representations for robotic arms. Although previous research has explored the reconstruction and editing capabilities of 3DGS and demonstrated or simple applications in the robotics field, developing system pipelines to expand robotic simulation systems and improve robotic performance remains an unexplored area.

[0023] For LLM Agents, i.e., Large Language Model (LLM) Agents, significant progress has been made, such as GPT-4 and DeepSeek-R1, which have demonstrated outstanding capabilities in various fields. By integrating LLMs as agents, many works have enhanced problem-solving capabilities in interactive and autonomous applications. For example, AutoGen utilizes organized LLM agents to form manipulation procedures and code programming. ChatSim employs LLM-agent collaborative workflows to edit 3D driving scenarios, while RoboGen uses generative models and LLMs to generate robotic tasks. In this work, we utilize LLM agents to decompose user simulation requirements into specific commands for preset editing functions, thereby automating and simplifying the simulation process.

[0024] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0025] The positional relationships described in the drawings are for illustrative purposes only and should not be construed as limiting this patent; Example 1 like Figure 1 and Figure 2 This embodiment provides an editable robot operation simulation system, including: Dynamic semantic enhancement modeling module, which is used to reconstruct the scene using Gaussian representation of the input video to build a photo-realistic simulation, extend the reconstructed scene with temporal and semantic information to achieve a wide range of simulation operations, and output a spatiotemporal dynamic Gaussian model of the scene; In the dynamic semantics enhanced modeling module, 3DGS explicitly represents 3D scenes with multiple Gaussian primitives. However, the original 3DGS representation lacks temporal modeling in dynamic scenes and is insufficient to meet the needs of real-world robotic environments. To address this issue, we enable Gaussian primitives to propagate over time, thereby capturing the spatiotemporal dynamics of the scene. Specifically, we treat the temporal and spatial dimensions equally, by expanding the position into The covariance matrix is ​​expanded into a 4D ellipsoid to formulate the scene spatiotemporal dynamic Gaussian model, where and is a 4D rotation matrix that can be decomposed into a pair of isotropic rotations. Each frame in a dynamic scene can be represented as a view in 3D static space, with a timestamp of As a condition, the reconstructed scene is expanded with time and semantic information, the object positions in the reconstructed scene are expanded into a covariance matrix, and the scene spatiotemporal dynamic Gaussian model is constructed by expanding the covariance matrix into a 4D ellipsoid as follows:

[0026] in, Represents the view rendering result output by the scene spatiotemporal dynamic Gaussian model, where N is the number of Gaussians. is the opacity of the i-th Gaussian, is the opacity of the j-th Gaussian, is the color of the i-th Gaussian, t represents the time, represents the visibility of the i-th Gaussian at time t, represents the probability distribution of the i-th Gaussian at pixel position x′, represents the visibility of the i-th Gaussian at time t, represents the probability distribution of the i-th Gaussian at pixel position x′; To realize scene simulation, we need to decompose the observed scene into different components for further manipulation. Based on previous work of 3DGS scene understanding, we extend a new parameter, identity encoding , which is a low-dimensional learnable embedding that allows grouping Gaussians according to object instances. Identity encoding is supervised by leveraging the 2D mask prediction of SAM, which has demonstrated impressive open-world segmentation capabilities. Similar to rendering RGB colors, we further extend the spatio-temporal dynamic Gaussian model of the scene with identity encoding as a parameter, obtaining 2D identity features in dynamic scenes as follows:

[0027] where, is the identity label of the i-th Gaussian; then we apply an additional linear layer and function for identity classification.

[0028] During the reconstruction of the scene, we use the two-dimensional rendering mean square error loss , for semantic learning, we use the standard cross-entropy loss Referring to Gaussian Grouping, we also use the KL divergence loss to enforce 3D spatial consistency, thereby limiting the identity encodings of the top K nearest Gaussians to be close in the feature space, thereby mitigating the occlusion problem within objects.

[0029] The total loss function L is used to train the spatio-temporal dynamic Gaussian model of the scene, and the calculation expression of the total loss function L is as follows:

[0030] where, represents the two-dimensional rendering mean square error loss, represents the weight coefficient of the two-dimensional rendering mean square error loss term, represents the standard cross-entropy loss, represents the weight coefficient of the standard cross-entropy loss term, represents the KL divergence loss, represents the weight coefficient of the KL divergence loss term.

[0031] An editable video simulation module is used for the spatio-temporal dynamic Gaussian model to perform scalability editing operations according to user commands, and output edited scene simulation data; ​​In the editable video simulation module, the extensible editing operations include retrieval of target objects, object deletion, object insertion, and object modification.

[0032] After training and grouping the dynamic semantic augmented Gaussians, we improved and refined various simulation operators to cover different scenarios. To perform simulation based on user commands, the first step is to retrieve the target object, which includes: According to user commands, the open world detection model is used to identify the mask ID corresponding to the target 2D object and obtain the corresponding mask ID; The mask ID is matched with the segmentation mask rendered by the scene spatiotemporal dynamic Gaussian model. When the mask matching result cannot cover the fine-grained part of the target object, the incremental semantic distillation method is used, specifically: Render the 2D object mask corresponding to the retrieved Gaussian representation, and use the open-world detection model to verify whether the 2D object mask corresponds to the target object required by the user command. If the verification fails, use the bounding box as a hint to perform finer-grained segmentation of the input video through the semantic segmentation model to obtain new fine-grained labels. Only the new fine-grained labels are used to fine-tune the identity encoding to gradually extract new object semantics.

[0033] Here, since SAM does not directly support language cues, we adopt the open-world detection model G-DINO to identify the desired 2D object and obtain its corresponding mask ID. This mask ID is then matched with the segmentation mask we rendered. However, due to the infinite granularity of objects based on user needs, such as retrieving a small button on the stove, the 2D mask used for training may not cover these fine-grained object parts, resulting in retrieval failure. To address this issue, we propose incremental semantic distillation (ISD) to gradually distill object semantics into the scene. After retrieving the desired object Gaussian, we render a 2D object mask and use the open-world detection model G-DINO to verify whether it corresponds to the desired object (such as a small button) or incorrectly retrieves the entire object (such as a stove). If the target object is not recognized, we further use the bounding box as a hint for SAM to perform finer-grained segmentation. We only fine-tune the identity encoding using the new fine-grained labels , thereby gradually extracting new object semantics.

[0034] The object removal scheme involves identifying the 3D object to be removed and deleting the 3D object Gaussian function corresponding to the 3D object to remove the 3D object in a simulated environment. The background area after the 3D object is removed is then inspected in the simulated environment to identify blur holes caused by insufficient observation. These blur holes are detected and located using an open-world detection model. For each view containing a blur hole, the LAMA inpainting algorithm is applied to inpaint the blur hole to generate an inpainted view. A new Gaussian function is generated near the removed area to simulate the space originally occupied by the removed 3D object. The inpainted view is used to fine-tune the new Gaussian function to ensure seamless reconstruction of the missing background. In this scheme, 3D object removal is achieved by simply deleting the 3D object Gaussian function. However, in real videos captured from a small number of viewpoints, removing an object may leave blur holes in the background due to insufficient observation. To address this issue, we first detect blur holes using the open-world detection model G-DINO and apply LAMA inpainting to each view. We then generate new Gaussians near the removed area and fine-tune only these newly introduced Gaussians using the inpainted view to ensure seamless reconstruction of the missing background.

[0035] The object insertion process involves obtaining a large-scale Gaussian dataset as an initial database, retrieving the insertion object from the initial database, inserting the inserted object into the original surrounding scene, adjusting the inserted object's position and size to accommodate the insertion location, and comparing the color of the inserted object with the original surrounding scene. If the inserted object's color is inconsistent with the original surrounding scene, the inserted object and the original surrounding scene are rendered using the libcom algorithm to generate a rendered image. This rendered image is then used to fine-tune the spherical harmonics (SH) of the inserted object's Gaussian distribution. In this scheme, we use the large-scale Gaussian datasets ShapeSplat and uCO3D as our initial database, enabling direct retrieval and use. When inserting objects from external sources, even after appropriate position and size adjustments, the inserted object may still exhibit significant color contrast with the original scene. To address this issue, we employ libcom to refine the rendered image to ensure color consistency between the surrounding scene and the newly inserted object. We then use the refined rendered image to fine-tune the spherical harmonics (SH) of the inserted object's Gaussian distribution, which typically takes only a few minutes.

[0036] The object modification includes 3D object size modification, 3D object position modification, 3D object color modification, 3D object style / texture modification and physical simulation.

[0037] For 3D object size modification, we adjust the properties of the target Gaussian function by scaling it accordingly.

[0038] For 3D object position modification, it is a combination of object removal and object insertion. We first remove the target object from its original position, and then insert it into the desired position.

[0039] For 3D object color modification, we adjust the spherical harmonics (SH) of the object's Gaussian function to preserve the learned 3D scene geometry. Simply changing the color can cause severe distortion, so we carefully adopt the CIELAB color space, which can perform color modification while preserving the original lighting effect.

[0040] For 3D object style / texture modification, we use 3D regularized nearest neighbor feature matching (NNFM) loss, which extracts features from rendered images and reference images using VGG16, and then minimizes the cosine distance between each pixel in the rendered image and its nearest neighbor in the reference image. To further prevent artifacts caused by SH refinement at the object boundary, we 1) only optimize the SH parameters of the target 3D object Gaussian to preserve the spatial details of the background, and 2) regularize the optimization using the original reconstruction loss.

[0041] For physics simulation, we achieve physics simulation by integrating physical properties into Gaussian primitives. Physical parameters include material density, Young's modulus, and Poisson's ratio. According to PhysGaussian, we can manually set the physical parameters of the target object and use the Material Point Method (MPM) to predict their motion. In addition, to reduce the dependence on manual parameter assignment, we combine GPT4V with a material library to automatically assign corresponding physical properties to enhance physics simulation.

[0042] An automated simulation module for converting user commands into structured configurations through multi-agent collaboration, and mobilizing the editable video simulation module to generate the required robot simulation operations for the structured configurations; In the automated simulation module, to connect the multi-agent collaboration components, we use LLM agents as "strings" to automate and simplify the simulation production process. However, directly applying a single LLM agent will encounter problems of multi-step reasoning and cross-referencing multiple operators. To solve this problem, we deploy multi-LLM agent collaboration, each agent equipped with a unique prompt and a custom toolset. Each agent first uses its specialized LLM prompt to interpret and convert user simulation commands into structured configurations, and then calls the corresponding toolset to generate the required simulation. The multi-agent collaboration includes a video simulation master agent, a retrieval agent, a scene editing agent, a 3D asset agent, a scene optimization agent, and a scene rendering agent; The video depicts a simulation manager agent, which acts as a team leader, decomposing user commands into simplified, specific natural language instructions and dispatching tasks to other agents. To achieve command decomposition, we designed a series of prompts for its LLM. The core idea of ​​these prompts is to describe the set of simulation operators, specify the overall goal, and define the output form using examples.

[0043] The retrieval agent is used to receive the retrieval agent task assigned by the video simulation general agent, with a preset location code Locate <object>The agent processes prompts in the form of and uses object retrieval operators; the returned results include the mask ID and the Gaussian distribution location of the target object. For simple queries such as "red cup," the agent directly retrieves the corresponding object. For more complex queries such as "cup closest to pressure cooker," the LLM performs a multi-step process: it first identifies the pressure cooker and cup, then determines the spatial relationship between them to retrieve the desired object. The scene editing agent is used to receive the scene editing agent task assigned by the video simulation general agent, process the preset color code Change color of <object>and apply the simulation operator to achieve the desired simulation operation; The 3D Asset Agent receives 3D asset agent tasks assigned by the Video Simulation Master Agent and organizes and retrieves 3D assets according to user specifications. First, it utilizes the LLM to interpret user commands and retrieves pre-set 3D objects from our Gaussian database by matching object attributes (such as color and type). If a match is not possible, the agent uses the object Gaussian generative model to synthesize the desired object and then incorporates it into the database to promote scalability and versatility.

[0044] The scene refinement agent receives scene refinement agent tasks assigned by the video simulation master agent and refines the scene; this refinement agent improves the overall simulation quality. Because each agent operates independently, cumulative edits can degrade the quality of the scene reconstruction. Therefore, a final refinement step is applied to all modified Gaussians to ensure realism and coherence across the entire simulation. The scene rendering agent receives scene rendering agent tasks assigned by the video simulation master agent and generates external camera parameters for scene rendering. First, it uses the LLM to interpret user viewpoint adjustment instructions into relative camera parameters based on the position and orientation of the original viewpoint. It then returns a simulated image with the desired viewing angle.

[0045] The robot operation learning module is used to use a preset visual language model to analyze the key frames of the fault cases of the robot simulation operation, provide explanations for potential fault causes, and output simulation solutions.

[0046] In the robotic manipulation learning module, we leverage the visual language model (VLM) to analyze keyframes of failure cases and provide detailed explanations of potential failure causes (such as object location, color, and background texture). After identifying the specific failure cause, we further prompt the VLM to generate corresponding simulation solutions in natural language. These text instructions are then input into our proposed automatic simulation framework, which generates targeted simulations to enhance model training. Automatically identifying problems and generating simulation commands closes the simulation loop, like the "clasp" of a pearl necklace, forming our complete approach framework, RoboPrearls, ultimately driving more efficient and powerful robotic learning.

[0047] In this embodiment, we propose an editable robot operation simulation system, see Figures 2 to 4 The robot operation simulation system is denoted as the RoboPearls robot operation video simulation framework. This is an editable video simulation framework built on the 3DGS. By developing and customizing improved functional modules, the reconstruction editing technology is integrated into the system pipeline in the field of robotics. In particular, the capabilities of robot simulation are expanded to solve the problems of low scalability of real-world demonstration data collection and high cost and low efficiency of diversified real data collection, thus bridging the gap between simulation and reality. The gap between simulation and reality is a major obstacle. We introduce the RoboPearls robot operation video simulation framework, which can perform scalable editing and simulation of simulated videos and real-world video data, bridging the challenge of the gap between simulation and reality. As the simulation platform only generates diversified data and cannot provide improvement solutions for policy improvement, we use the visual language model (VLM) to analyze the key frames of fault cases and provide detailed explanations of potential fault causes (such as object position, color, and background texture). Through experimental verification, our closed-loop strategy can effectively analyze learning problems and generate corresponding simulation requirements, thereby improving the performance of robots. Specifically, first, the dynamic semantic enhancement modeling module reconstructs the scene using Gaussian representation of the input video, expands the reconstructed scene with time and semantic information, and outputs a spatiotemporal dynamic Gaussian model of the scene, which significantly improves the accuracy and scalability of scene reconstruction and effectively narrows the gap between simulation and reality; secondly, the editable video simulation module supports users to flexibly edit the scene, enhancing the adaptability and scalability of robot operation simulation; then the automated simulation module is used to realize the automatic configuration and execution of user commands through multi-agent collaboration, greatly reducing the operation threshold and improving simulation efficiency; further, the robot operation learning module is combined with the visual language model to perform intelligent analysis and interpretation of fault cases, providing data support for the optimization of simulation solutions, which can effectively analyze learning problems and generate corresponding simulation requirements, thereby improving the scalability of robot simulation.

[0048] RoboPearls is a fully assembled "pearl necklace" for robot manipulation. The framework is built on top of 3DGS, ensuring that photo-realistic and visually stunning images can be constructed from demonstration videos. Figure 1 RoboPearls enhances the Gaussian representation (like "oysters producing pearls") to accommodate various simulation operators ("pearls"), extending it to incorporate time propagation to capture the spatiotemporal dynamics of the scene and embedding semantic features extracted from SAM to achieve scene understanding capabilities. Second, RoboPearls refines and improves various simulation operators ("pearls") to cover a wide range of everyday scenarios. For example, it can use external digital assets to change object color or texture, remove objects, and insert objects. It also supports physics simulation. These capabilities are powered by carefully designed submodules, including the Incremental Semantic Distillation (ISD) module and the 3D Regularized NNFM loss (3D-NNFM). Third, compared to traditional simulations that rely heavily on manual intervention, RoboPearls is designed to be highly user-friendly and capable of handling complex requirements. Users can interact with RoboPearls using simple natural language commands to generate desired simulations. Specifically, to string the individual pearls together, we utilize multiple tailored Large Language Model (LLM) agents as "strings" to automate and streamline the simulation production process. The LLM agent decomposes user simulation requests into simplified and specific commands to implement the preset editing functions. Finally, to complete the pearl necklace, we integrated a visual language model (VLM) as a "clasp" to close the simulation loop. It can identify and analyze problems in robot learning and generate corresponding simulation requests to improve robot performance.

[0049] Example 2 This embodiment further illustrates the effects of an editable robot operation simulation system proposed in the above embodiment as shown in Table 1.

[0050] Table 1 Task completion success rate of different methods

[0051] We systematically evaluate robustness across various conditions by presenting task completion success rates for 13 perturbations on COLOSSEUM in Table 1. Our method, RoboPearls, shows significant performance improvements across all perturbations, with average success rates improving by 17.5% and 10.8% over RVT and RVT2, respectively, demonstrating its strong robustness to environmental variations (e.g., illumination changes) and object-level perturbations (e.g., color changes).

[0052] Table 2 Evaluation task results of different methods

[0053] See Table 2 for results on several challenging tasks in RLBench to assess general operational performance. Overall, RoboPearls achieved an average success rate of 68.0% and 78.0%, respectively, improving upon the baseline model by 5.6% and 7.9%. These benchmark improvements highlight the efficiency of our editable simulation framework, which leverages a wide range of operators to effectively handle a variety of scenarios.

[0054] Table 3 RoboPearls performance evaluation table

[0055] Table 4 RoboPearls performance comparison evaluation table

[0056] As shown in Tables 3 and 4, we evaluated the performance of RoboPearls on a real robot system built on a Kinova Gen3 manipulator. As shown in the table, RoboPearls successfully performed real tasks with excellent generalization ability, while the baseline struggled, verifying our effectiveness in real environments. In addition, we first verified the visual Figure 1 With consistent 3D simulation capabilities, IP2P can be viewed as a simulator in 2D image space. While IP2P offers limited improvements in some cases, our significantly superior results demonstrate the effectiveness of our simulation in terms of spatiotemporal consistency. We also validate the impact of incorporating VLM, which can effectively analyze learning problems and generate corresponding simulation requirements, thereby improving robot performance.

[0057] Figure 5 For qualitative examples of generating action sequences in simulation and the real world, see Figure 5 We present two qualitative examples of generated action sequences, both in simulation and in the real world. In the correct case, the agent is instructed to "stack the other cups on top of the red cup." Results show that while previous agents struggled with the task, our RoboPearls accurately identified each cup and successfully stacked them on top of the red one. This improvement is attributed to our method's efficient simulation, which enables more accurate understanding of various scenarios.

[0058] See also Figure 6 and Figure 7 RoboPearls consistently achieves photorealistic, visually immersive, and realistic rendering on RLbench simulation datasets and real-world datasets, including Ego4D and Open X-Embodiment datasets, using various operators. Figure 1 Precise simulation.

[0059] Example 3 See also Figure 8 ,This implementation proposes an editable robot operation simulation method, which includes the following steps: S1. Reconstruct the scene from the input video using Gaussian representation, expand the reconstructed scene with temporal and semantic information, and output a spatiotemporal dynamic Gaussian model of the scene; S2 for the spatiotemporal dynamic Gaussian model according to user commands, perform scalable editing operations, output edited scene simulation data; S3 is used to convert user commands into a structured configuration through multi-agent collaboration, and mobilize the editable video simulation module to generate the robot simulation operation required for the structured configuration; S4. is used to analyze key frames of the fault case of the robot simulation operation using a preset visual language model, provide explanations for potential fault causes, and output simulation solutions.

[0060] In this embodiment, the input video is first reconstructed using Gaussian representation, the reconstructed scene is expanded using time and semantic information, and a spatiotemporal dynamic Gaussian model of the scene is output, which significantly improves the accuracy and scalability of scene reconstruction and effectively narrows the gap between simulation and reality; secondly, by supporting users to flexibly edit the scene, the adaptability and scalability of robot operation simulation are enhanced; then, through multi-agent collaboration, the automatic configuration and execution of user commands are realized, which greatly reduces the operation threshold and improves simulation efficiency; further, the visual language model is combined to perform intelligent analysis and interpretation of fault cases, providing data support for the optimization of simulation solutions, which can effectively analyze learning problems and generate corresponding simulation requirements, thereby improving the scalability of robot simulation.

[0061] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. A person skilled in the art would be able to make other variations or modifications based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.< / object> < / object>

Claims

1. An editable robot operation simulation system, characterized in that: include: The dynamic semantic enhancement modeling module is used to reconstruct the scene of the input video using Gaussian representation, expand the reconstructed scene with temporal and semantic information, and output a spatiotemporal dynamic Gaussian model of the scene; An editable video simulation module, configured to execute scalable editing operations on the spatiotemporal dynamic Gaussian model according to user commands and output edited scene simulation data; an automated simulation module for converting user commands into structured configurations through multi-agent collaboration and mobilizing the editable video simulation module to generate robot simulation operations required for the structured configurations; The robot operation learning module is used to use a preset visual language model to analyze the key frames of the fault cases of the robot simulation operation, provide explanations for potential fault causes, and output simulation solutions.

2. The editable robot operation simulation system according to claim 1, characterized in that: The reconstructed scene is expanded using temporal and semantic information, the object positions in the reconstructed scene are expanded into a covariance matrix, and the spatiotemporal dynamic Gaussian model of the scene is constructed by expanding the covariance matrix into a 4D ellipsoid as follows: in, Represents the view rendering result output by the scene spatiotemporal dynamic Gaussian model, where N is the number of Gaussians. is the opacity of the i-th Gaussian, is the opacity of the j-th Gaussian, is the color of the i-th Gaussian, t represents the time, represents the visibility of the i-th Gaussian at time t, represents the probability distribution of the i-th Gaussian at pixel position x′, represents the visibility of the i-th Gaussian at time t, represents the probability distribution of the i-th Gaussian at pixel position x′; The parameters of the scene spatiotemporal dynamic Gaussian model are encoded as identity Further expansion of , we can get 2D identity features in dynamic scenes as follows: in, is the identity label of the i-th Gaussian.

3. The editable robot operation simulation system according to claim 1, characterized in that: During the scene reconstruction process, the scene spatiotemporal dynamic Gaussian model is trained using the total loss function L. The calculation expression of the total loss function L is as follows: in, represents the mean square error loss of 2D rendering, Represents the weight coefficient of the mean square error loss term for 2D rendering, represents the standard cross entropy loss, represents the weight coefficient of the standard cross entropy loss term, represents the KL divergence loss, Represents the weight coefficient of the KL divergence loss term.

4. The editable robot operation simulation system according to claim 2, characterized in that: The extensible editing operations include retrieval of target objects, object deletion, object insertion, and object modification.

5. The editable robot operation simulation system according to claim 4, characterized in that: The search target objects include: According to user commands, the open world detection model is used to identify the mask ID corresponding to the target 2D object and obtain the corresponding mask ID; The mask ID is matched with the segmentation mask rendered by the scene spatiotemporal dynamic Gaussian model. When the mask matching result cannot cover the fine-grained part of the target object, the incremental semantic distillation method is used, specifically: Render the 2D object mask corresponding to the retrieved Gaussian representation, and use the open-world detection model to verify whether the 2D object mask corresponds to the target object required by the user command. If the verification fails, use the bounding box as a hint to perform finer-grained segmentation of the input video through the semantic segmentation model to obtain new fine-grained labels. Only the new fine-grained labels are used to fine-tune the identity encoding to gradually extract new object semantics.

6. The editable robot operation simulation system according to claim 4, characterized in that: The object deletion includes: determining a 3D object to be removed and deleting a 3D object Gaussian function corresponding to the 3D object to remove the 3D object in a simulated environment; inspecting a background area after the 3D object is removed in the simulated environment; identifying fuzzy holes caused by insufficient observation; detecting and locating the fuzzy holes using an open world detection model; applying a LAMA repair algorithm to each view containing the fuzzy hole to repair the fuzzy hole to generate a repaired view; generating a new Gaussian function near the deleted area to simulate the space originally occupied by the removed 3D object; and fine-tuning the new Gaussian function using the repaired view.

7. The editable robot operation simulation system according to claim 4, characterized in that: The object insertion includes: obtaining a large-scale Gaussian data set as an initial database, retrieving an insertion object from the initial database, inserting the insertion object into the original surrounding scene, adjusting the position and size of the insertion object to adapt to the insertion position, comparing the color of the insertion object with that of the original surrounding scene, and when the colors of the insertion object and the original surrounding scene are inconsistent, using the libcom algorithm to render the insertion object and the original surrounding scene to obtain a rendered image, and using the rendered image to fine-tune the spherical harmonic function of the Gaussian distribution of the insertion object.

8. The editable robot operation simulation system according to claim 4, characterized in that: The object modification includes 3D object size modification, 3D object position modification, 3D object color modification, 3D object style / texture modification and physical simulation.

9. The editable robot operation simulation system according to claim 4, characterized in that: The multi-agent collaboration includes a video simulation general agent, a retrieval agent, a scene editing agent, a 3D asset agent, a scene optimization agent, and a scene rendering agent; The video simulates a general agent that decomposes user commands into simplified, specific natural language instructions and assigns tasks to other agents. The retrieval agent is configured to receive the retrieval agent task assigned by the video simulation master agent, process the prompt in the form of a preset location code, and use an object retrieval operator; The scene editing agent is used to receive the scene editing agent task assigned by the video simulation general agent, process the operation prompt of the preset color code, and apply the simulation operator to realize the required simulation operation; The 3D asset agent is used to receive 3D asset agent tasks assigned by the video simulation general agent and organize and retrieve 3D assets according to user specifications; The scene refining agent is used to receive the scene refining agent task assigned by the video simulation general agent and refine the scene; The scene rendering agent is used to receive the scene rendering agent task assigned by the video simulation general agent and generate external camera parameters for scene rendering.

10. An editable robot operation simulation method, characterized in that: The following steps are involved: S1. Reconstruct the scene from the input video using Gaussian representation, expand the reconstructed scene with temporal and semantic information, and output a spatiotemporal dynamic Gaussian model of the scene; S2 for the spatiotemporal dynamic Gaussian model according to user commands, perform scalable editing operations, output edited scene simulation data; S3 is used to convert user commands into a structured configuration through multi-agent collaboration, and mobilize the editable video simulation module to generate the robot simulation operation required for the structured configuration; S4. is used to analyze key frames of the fault case of the robot simulation operation using a preset visual language model, provide explanations for potential fault causes, and output simulation solutions.

Images