Simulation verification method and device of body data, electronic equipment and storage medium
By combining video generation models and simulation platforms, embodied operation trajectories are generated and verified, solving the problem of applying robot embodied data in simulation environments, achieving accurate mapping and system evaluation, and improving robot operation accuracy and data quality.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-12
- Publication Date
- 2026-07-14
AI Technical Summary
In existing technologies, robot-embodied data is difficult to apply effectively in simulation environments, making it difficult to improve operational accuracy. Furthermore, simulation data has poor matching with real data, resulting in insufficient safety and repeatability.
Embody operation trajectories are generated using video generation models, and these trajectories are replicated and verified in a simulation scenario using the playback method of a simulation platform. The verification and evaluation are combined with simulation physical parameters to establish the correlation between data quality and strategy performance.
It achieves accurate mapping and verification of embodied data in a physical simulation environment, ensuring the physical feasibility of the generated trajectory. Furthermore, it establishes a mapping relationship between simulation verification indicators and actual strategy performance through system evaluation, thereby improving the robot's operational accuracy and data quality.
Smart Images
Figure CN122389392A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and simulation technology, and specifically provides simulation verification methods, apparatus, electronic devices, and storage media for embodied data. Background Technology
[0002] With the development of artificial intelligence, more and more robots are being used in industrial applications. In order to improve the operational accuracy of robots, it is even more important to improve their level of intelligence.
[0003] Currently, robots are deployed in real-world physical environments, using sensors (RGB cameras, depth cameras, LiDAR, etc.) to collect real-world operational data, which is then used for model training. However, this approach requires building physical robots, sensor systems, and operating environments; each data collection session is time-consuming and labor-intensive, making it difficult to scale. Furthermore, because environmental factors in the real world (lighting, object position, friction, etc.) are difficult to control precisely, it's hard to repeatedly verify the same task, and scenario reproduction is challenging. Additionally, certain dangerous operations or extreme scenarios cannot be safely collected in real-world environments, posing safety limitations. Moreover, real-world data annotation relies on manual methods, making it difficult to obtain accurate state and action labels.
[0004] In related technologies, simulation models are used to construct virtual environments, generating large amounts of simulation data for robots through randomization strategies. However, this approach suffers from systematic deviations between simulated physical parameters, visual rendering, and the real world, leading to a sharp decline in the performance of strategies that perform well in simulations when transferred to real robots. Furthermore, simulation environments typically lack the complexity and diversity of the real world, making it difficult to capture subtle visual features and physical properties in real-world scenarios. Additionally, the action sequences generated by randomization or reinforcement learning may not conform to human operating habits, and the data distribution may not match the real-world task.
[0005] In related technologies, generative artificial intelligence models (such as diffusion models and video generation models) are used for data augmentation or scene synthesis. For example, image generation tools are used to generate scene images, or video generation models are used to synthesize embodied operation sequences. However, these generative models typically optimize visual realism and lack explicit constraints on physical laws, potentially leading to physical inconsistencies in the generated video sequences (such as object penetration or violations of gravity). Furthermore, the visual output of generative models makes it difficult to verify the feasibility of generated operation sequences on real robots, hindering the assessment of task success rates. Additionally, generative AI models are typically generated in an open-loop manner, lacking a closed-loop mechanism to feed the generated data back to the physical verification environment, thus preventing iterative optimization of data quality. Summary of the Invention
[0006] The present invention aims to solve the above-mentioned technical problems, namely, the problem that existing robot data obtained through simulation is difficult to apply to robot control to improve robot operation accuracy.
[0007] In a first aspect, the present invention provides a simulation verification method for embodied data, comprising:
[0008] Using a video generation model, multiple embodied operation trajectories are generated based on scene information and agent information. Each embodied operation trajectory includes an image sequence, a state sequence, and an action sequence.
[0009] Using the playback method of the simulation platform, the scene is replicated based on the scene information, the agent information and the multiple embodied operation trajectories, to obtain multiple actual execution trajectories generated by the agent in the simulation scene in response to the multiple embodied operation trajectories.
[0010] The multiple actual execution trajectories are verified in the simulation scenario to obtain multiple verification results corresponding to the multiple actual execution trajectories;
[0011] The data validity of the multiple verification results is evaluated, and the correlation between data quality and strategy performance is obtained based on the data validity evaluation results.
[0012] In some embodiments of the present invention, the step of generating multiple embodied operation trajectories based on scene information and agent information using a video generation model includes:
[0013] Extract the agent model, environment layout information, initial pose of the agent, and initial pose of the interactive object from the scene information and agent information.
[0014] Using a preset world model, based on the agent model, environmental layout information, the initial pose of the agent, and the initial pose of the interactive objects of the agent, embodied operation trajectory generation processing is performed to obtain the multiple embodied operation trajectories.
[0015] In some embodiments of the present invention, the playback mechanism of the simulation platform is used to perform scene replication processing based on the scene information, the agent information, and the multiple embodied operation trajectories, to obtain multiple actual execution trajectories generated by the agent in the simulation scene corresponding one-to-one with the multiple embodied operation trajectories, including:
[0016] Using the simulation platform, the simulation scene is constructed based on the agent model, the environmental layout information, the initial pose of the agent, and the initial pose of the objects that can interact with the agent.
[0017] By employing a playback method on a simulation platform, the intelligent agent is controlled to sequentially execute corresponding actions according to the multiple embodied operation trajectories in the simulation scenario, and actual action parameters are collected to obtain the multiple actual execution trajectories.
[0018] In some embodiments of the present invention, the construction of the simulation scene using the simulation platform, based on simulation physical parameters, the intelligent agent model, the environmental layout information, the initial pose of the intelligent agent, and the initial pose of objects interactive with the intelligent agent, includes:
[0019] Determine the transformation matrix between the first coordinate system of the preset world model and the second coordinate system of the simulation scene;
[0020] Obtain the correspondence between the joint information output by the intelligent agent in the preset world model and the joint information of the intelligent agent in the virtual environment of the simulation platform;
[0021] Using the virtual environment of the simulation platform, the simulation scene is constructed based on the simulation physical parameters, the intelligent agent model, the environment layout information, the initial pose of the intelligent agent, the initial pose of the interactive objects of the intelligent agent, the transformation matrix and the correspondence. The simulation physical parameters include at least one of gravity parameters, friction coefficient and collision deformation parameters.
[0022] In some embodiments of the present invention, the process of using the playback mechanism of the simulation platform to control the intelligent agent to sequentially execute corresponding actions according to the multiple embodied operation trajectories and collect actual action parameters to obtain the multiple actual execution trajectories further includes:
[0023] Disable random noise and run the playback method with a fixed time step and a fixed random seed.
[0024] In some embodiments of the present invention, the step of verifying the plurality of actual execution trajectories in the simulation scenario to obtain a plurality of verification results corresponding one-to-one with the plurality of actual execution trajectories includes:
[0025] Based on the multiple task objectives corresponding to the multiple embodied operation trajectories, the execution result information of the multiple tasks corresponding to the multiple actual execution trajectories in the simulation scenario is verified.
[0026] The task execution results for the same task are statistically analyzed to obtain task statistics.
[0027] In some embodiments of the present invention, the task statistics include execution success rate and trajectory smoothness;
[0028] The step of evaluating the data validity of the multiple verification results and obtaining the correlation between data quality and strategy performance based on the data validity evaluation results includes:
[0029] Select multiple embodied operation trajectories from the task statistics results that have an execution success rate greater than the first preset threshold and a trajectory smoothness greater than the second preset threshold, and construct a dataset;
[0030] The visual language action model was trained and its parameters were tuned based on the dataset.
[0031] The performance of the trained and hyperparameter-tuned visual language action model was evaluated on a standard embodied benchmark test set.
[0032] Based on the performance evaluation results, a comparative analysis was conducted on data filtering strategies under different simulation verification thresholds to obtain the correlation between data quality and strategy performance.
[0033] In a second aspect, the present invention provides a simulation verification device for embodied data, comprising:
[0034] The embodied operation trajectory generation module is used to generate multiple embodied operation trajectories based on scene information and agent information using a video generation model. Each embodied operation trajectory includes an image sequence, a state sequence, and an action sequence.
[0035] The scene replication module is used to perform scene replication processing based on the scene information, the agent information and the multiple embodied operation trajectories using the playback method of the simulation platform, so as to obtain multiple actual execution trajectories generated by the agent in the simulation scene in a one-to-one correspondence with the multiple embodied operation trajectories;
[0036] The trajectory execution verification module is used to verify the multiple actual execution trajectories in the simulation scenario and obtain multiple verification results corresponding to the multiple actual execution trajectories.
[0037] The data validity assessment module is used to assess the validity of the multiple verification results and obtain the correlation between data quality and strategy performance based on the data validity assessment results.
[0038] In some embodiments of the present invention, the embodied operation trajectory generation module is used to extract the agent model, environment layout information, initial pose of the agent, and initial pose of the interactive object with the agent from the scene information and agent information; the embodied operation trajectory generation module is also used to perform embodied operation trajectory generation processing based on the agent model, environment layout information, initial pose of the agent, and initial pose of the interactive object with the agent using a preset world model to obtain the multiple embodied operation trajectories.
[0039] In some embodiments of the present invention, the scene replication module is used to construct the simulation scene using the simulation platform based on the agent model, the environmental layout information, the initial pose of the agent, and the initial pose of the interactive objects of the agent; the scene replication module is also used to use the playback method of the simulation platform to control the agent to perform corresponding actions in the simulation scene according to the multiple embodied operation trajectories and collect actual action parameters to obtain the multiple actual execution trajectories.
[0040] In some embodiments of the present invention, the scene replication module is used to determine the transformation matrix between the first coordinate system of the preset world model and the second coordinate system of the simulation scene; the scene replication module is also used to obtain the correspondence between the joint information output by the agent in the preset world model and the joint information of the agent in the virtual environment of the simulation platform; the scene replication module is also used to construct the simulation scene using the virtual environment of the simulation platform based on simulation physical parameters, the agent model, the environment layout information, the initial pose of the agent, the initial pose of the interactive object of the agent, the transformation matrix and the correspondence, wherein the simulation physical parameters include at least one of gravity parameters, friction coefficient and collision deformation parameters.
[0041] In some embodiments of the present invention, the scene replication module is used to disable random noise and run the playback method with a fixed time step and a fixed random seed in the process of controlling the agent to perform corresponding actions in sequence according to the multiple embodied operation trajectories and collecting actual action parameters in the playback mechanism of the simulation platform to obtain the multiple actual execution trajectories.
[0042] In some embodiments of the present invention, the trajectory execution verification module is used to verify the multiple task execution result information corresponding to the multiple actual execution trajectories in the simulation scenario based on the multiple task targets corresponding to the multiple embodied operation trajectories; the trajectory execution verification module is also used to perform statistics on the task execution results corresponding to the same task to obtain task statistics results.
[0043] In some embodiments of the present invention, the task statistics results include execution success rate and trajectory smoothness; the data validity evaluation module is used to select multiple embodied operation trajectories corresponding to execution success rates greater than a first preset threshold and trajectory smoothness greater than a second preset threshold from the task statistics results to construct a dataset; the data validity evaluation module is also used to train and tune the visual language action model based on the dataset; the data validity evaluation module is also used to use the trained and tuned visual language action model to perform performance evaluation on a standard embodied benchmark test set; the data validity evaluation module is also used to compare and analyze data filtering strategies under different simulation verification thresholds based on the performance evaluation results to obtain the correlation between data quality and strategy performance.
[0044] In a third aspect, the present invention provides an electronic device comprising:
[0045] Memory, used to store computer program products;
[0046] A processor is configured to execute a computer program product stored in the memory, and when the computer program product is executed, to implement the method described in the first aspect above.
[0047] In a fourth aspect, the present invention provides a computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method described in the first aspect above.
[0048] In a fifth aspect, the present invention provides a computer program product including computer program instructions, which, when executed by a processor, cause the processor to perform the method described in the first aspect.
[0049] By employing the above technical solution, this invention can accurately map the generated embodied trajectory operation data to a physical simulation environment and verify its physical feasibility. Furthermore, this invention can accurately replicate the simulation scenario, enabling fair comparison and system evaluation of different generated trajectories. Moreover, this invention can evaluate the effectiveness of simulation verification data and establish a mapping relationship between simulation verification indicators and actual strategy performance. Attached Figure Description
[0050] The preferred embodiments of the present invention are described below with reference to the accompanying drawings, in which:
[0051] Figure 1 This is a flowchart illustrating the simulation verification method for embodied data in some embodiments of the present invention;
[0052] Figure 2 This is a structural block diagram of a simulation verification device for embodied data in some embodiments of the present invention;
[0053] Figure 3 These are structural block diagrams of electronic devices in some embodiments of the present invention. Detailed Implementation
[0054] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.
[0055] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.
[0056] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.
[0057] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.
[0058] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.
[0059] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.
[0060] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.
[0061] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.
[0062] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0063] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.
[0064] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.
[0065] Figure 1 This is a flowchart illustrating the simulation verification method for embodied data in some embodiments of the present invention. For example... Figure 1 As shown, the simulation verification method for embodied data includes the following steps:
[0066] S1: Using a video generation model, multiple embodied operation trajectories are generated based on scene information and agent information. Each embodied operation trajectory includes an image sequence, a state sequence, and an action sequence.
[0067] In some embodiments of this disclosure, S1 may include the following steps:
[0068] S1-1: Extract the agent model, environment layout information, initial pose of the agent, and initial pose of objects that can interact with the agent from scene information and agent information.
[0069] The process involves acquiring scene information and agent information for the scene to be replicated. Scene information may include the environmental layout of the scene. Agent information may include the agent model and its initial pose within the scene. Additionally, it may include the initial poses of objects that can interact with the agent in the scene. For example, if the agent's task in the scene includes grasping a target object, the initial pose of the target object can be acquired. This allows for improved accuracy in capturing agent actions by analyzing changes in both the agent's and the target object's poses when the agent grasps the object.
[0070] In this embodiment, the intelligent agent may include robots and drones, as well as other mobile intelligent devices that can be used to complete a specified task. For example, when the intelligent agent is used for ground industrial operations, the intelligent agent mainly uses robots (robot dogs); when the intelligent agent is used for high-altitude operations, the intelligent agent mainly uses drones.
[0071] It should be noted that the following embodiments use a robot as an example for illustration, but those skilled in the art will understand that other intelligent devices capable of fully specifying tasks are also applicable to this invention.
[0072] S1-2: Using a preset world model, based on the agent model, environmental layout information, the agent's initial pose, and the initial pose of the interactive objects, embodied operation trajectories are generated to obtain multiple embodied operation trajectories.
[0073] When the agent information includes the robot's URDF / USD model and the robot's initial pose, scene information and agent information are used as inputs to generate a variety of embodied operation trajectories using a video generation model.
[0074] Among them, the video generation model can adopt the world model of Cosmos Predict 2.5, which brings powerful world simulation capabilities to the field of embodied intelligence of robots. It can predict and generate what will happen next in a virtual world based on text, pictures or videos, and is an important cornerstone for building physical AI.
[0075] Each embodied manipulation trajectory includes an image sequence, a state sequence, and a motion sequence. The image sequence can include RGB / depth image sequences. The state sequence can include robot joint angles and end effector (e.g., gripper / dexterous hand) poses. The motion sequence can include joint control commands or end effector velocity commands.
[0076] S2: Using the playback method of the simulation platform, the scene is replicated based on scene information, agent information and multiple embodied operation trajectories, resulting in multiple actual execution trajectories generated by the agent in the simulation scene in response to multiple embodied operation trajectories.
[0077] In some embodiments of this disclosure, S2 includes the following steps:
[0078] S2-1: Using a simulation platform, construct a simulation scene based on the agent model, environmental layout information, the agent's initial pose, and the initial poses of objects that can interact with the agent. The simulation platform can be the NVIDIA Isaac Sim platform, built on the Omniverse platform, used for testing and training AI robots in a virtual environment.
[0079] By taking the agent model, environmental layout information, the agent's initial pose, and the initial pose of objects that can interact with the agent as inputs, a simulation scene can be constructed using a simulation platform.
[0080] In some embodiments of the present invention, S2-1 includes the following steps:
[0081] S2-1-1: Determine the transformation matrix between the first coordinate system of the preset world model and the second coordinate system of the simulation scene.
[0082] To solve the problem of accurate alignment between the preset world model generation state and the Isaac Sim simulation state, a transformation matrix is first established between the first coordinate system (e.g., image coordinate system) of the preset world model and the second coordinate system (e.g., world coordinate system) of the simulation scene.
[0083] S2-1-2: Obtain the correspondence between the joint information output by the agent in the preset world model and the joint information of the agent in the virtual environment of the simulation platform.
[0084] Next, we define the naming convention for robot joints to ensure that the joint angles output by the world model correspond one-to-one with the joint names in Isaac Sim.
[0085] Then, an initial state encoding protocol is constructed to encode the scene information, robot initial pose, and environment layout in the preset world model input into a configuration file that can be parsed by Isaac Sim.
[0086] S2-1-3: Utilizing the virtual environment of the simulation platform, a simulation scene is constructed based on simulation physical parameters, the agent model, environmental layout information, the initial pose of the agent, the initial pose of objects that can interact with the agent, and transformation matrices and corresponding relationships. The simulation physical parameters include at least one of gravity parameters, friction coefficient, and collision deformation parameters.
[0087] Based on the initial poses of the robot and interactive objects provided by the preset world model, the robot's joint angles, base position, and the posture of environmental objects are precisely set, and the simulation physical parameters (gravity parameters / friction coefficient / collision deformation parameters) are configured to ensure consistency with the physical conditions during the training of the preset world model.
[0088] S2-2: Using the playback method of the simulation platform, the intelligent agent is controlled to execute corresponding actions in sequence according to multiple embodied operation trajectories in the simulation scene and the actual action parameters are collected to obtain multiple actual execution trajectories.
[0089] Based on the action sequence generated by the preset world model, control commands are executed frame by frame in the simulation, and the actual execution state is recorded. For example, if the preset world model generates a set of action sequences for a robotic arm to grab an orange, the robotic arm in the simulation can be controlled to perform the corresponding actions based on this set of action sequences.
[0090] The execution of S2-1-3 also includes: disabling random noise and running the playback method with a fixed time step and a fixed random seed.
[0091] Preload all necessary USD assets (bots / objects / environments) before replay to reduce runtime loading latency.
[0092] Disable random noise and run the replay method with a fixed time step and a fixed random seed to ensure the repeatability of the replay process. Failure to perform the above steps may result in different physical calculation results, affecting the success rate of execution. For example, items that could have been picked up may slip away.
[0093] In keyframes, such as those in the middle stages of execution (e.g., each stage of a long task, like picking up an orange and putting it in a bowl), the moment the gripper catches the orange can be used as a keyframe to save the simulation state checkpoint. This allows for restarting the replay from any intermediate state, facilitating debugging and parallel verification.
[0094] Design a parallel replay scheduler to support the simultaneous execution of replay verification for multiple scenarios, thereby improving verification efficiency.
[0095] S3: Verify multiple actual execution trajectories in the simulation scenario to obtain multiple verification results corresponding to the multiple actual execution trajectories.
[0096] In some embodiments of the present invention, S3 includes the following steps:
[0097] S3-1: Based on multiple task objectives corresponding to multiple embodied operation trajectories, verify the execution result information of multiple tasks corresponding to multiple actual execution trajectories in the simulation scenario.
[0098] The motion sequence generated by the world model is input into the simulation robot controller. The actual joint angles, end-effector poses, and object interaction states achieved during the simulation are recorded. Based on the defined task objectives (such as grasping success rate, placement accuracy, trajectory tracking error, etc.), the success of trajectory execution is determined.
[0099] Define the criteria for judging task completion (such as the object reaching the target area, the target pose error being less than a threshold, etc.), and count the proportion of trajectories that successfully complete the task.
[0100] 1. Joint velocity continuity index:
[0101] This value is used to measure the smoothness of changes in robot joint speed; a smaller value indicates fewer sudden speed changes.
[0102]
[0103] in, A quantity representing the continuity of joint velocity. Indicates the number of points on the trajectory. Indicates the first Joint velocity vectors of trajectory points Indicates the time step.
[0104] 2. Acceleration smoothness index: evaluates the severity of changes in joint acceleration and more sensitively reflects impact and vibration.
[0105]
[0106] in, Indicates the smoothing amount of acceleration. Indicates the first The joint acceleration vector of each trajectory point Represents the Euclidean norm (multi-joint summation).
[0107] 3. End effector path smoothness: Evaluate the smoothness of the end effector path in Cartesian space to avoid sharp turns and jitter.
[0108]
[0109] in, This indicates the path smoothing amount of the end effector. Represents the end position vector. Indicates the path length between adjacent points ( ), This indicates the total path length.
[0110] 4. Based on the overall smoothness score, a threshold for eliminating trajectories with severe jitter is determined.
[0111]
[0112] in, This represents the overall smoothness score. , and All are constants. This represents the normalized metric (e.g., min-max normalization). If any metric exceeds the threshold... (like If the value is 0, it is considered to be shaking violently and will be deleted.
[0113] In addition, it can also verify physical feasibility and evaluate time efficiency.
[0114] Among them, physical feasibility verification: by simulating the collision of the collision ball and monitoring the joint parameters of the robot, physical violations (collision, penetration, joint over-limit, torque over-limit) are detected in the trajectory execution process, and physically infeasible trajectories are marked.
[0115] Time efficiency assessment: Statistical analysis of trajectory execution time and task completion time to determine the time efficiency of different trajectory strategies.
[0116] S3-2: Statistical analysis of the task execution results for the same task to obtain task statistics results.
[0117] Batch verification of different generated trajectories for the same task is performed, and indicators such as execution success rate, average completion time, failure mode distribution, and trajectory smoothness are statistically analyzed.
[0118] S4: Perform data validity assessment on multiple validation results, and obtain the correlation between data quality and strategy performance based on the data validity assessment results.
[0119] In some embodiments of the present invention, S4 includes the following steps:
[0120] S4-1: Select multiple embodied operation trajectories from the task statistics results that have an execution success rate greater than the first preset threshold and a trajectory smoothness greater than the second preset threshold, and construct a dataset.
[0121] S4-2: Training and parameter tuning of the visual language action model based on the dataset.
[0122] S4-3: Utilize the trained and tuned visual language action model to evaluate its performance on standard embodied benchmark sets (such as ManiSkill, RLBench, CALVIN, etc.) to assess the task success rate of the fine-tuned strategy.
[0123] S4-4: Based on the performance evaluation results, a comparative analysis of data filtering strategies under different simulation verification thresholds is conducted to obtain the correlation between data quality and strategy performance.
[0124] By comparing data filtering strategies under different simulation verification thresholds, the correlation between data quality indicators and the performance of the final strategy is analyzed. Data generated by the preset world model is simulated and replayed. Data is filtered according to the success rate and execution time. Data under different success rate thresholds and time thresholds are grouped and trained to obtain the model's test results. A data validity verification report is generated, proving that the data that has been rigorously verified by simulation is more valid.
[0125] Data quality scoring: Based on comprehensive simulation verification indicators (success rate, smoothness, physical feasibility), a data quality scoring function is designed to assign a quality score to each trajectory. The data quality scoring function (for each trajectory) is as follows:
[0126]
[0127] in, This indicates whether the trajectory successfully completed the target task. A score of 0 is given upon failure. This represents the smoothness score (the higher the score, the smoother the surface). This represents the physical feasibility score (the higher the score, the better the physical constraints are met). Represents the weighting coefficients, satisfying ,recommend .
[0128] For smoothness scores First, calculate the original smoothness cost. (The smaller the value, the smoother the surface):
[0129]
[0130] in, , and All are constants; the components are defined as follows (assuming the trajectory contains...). There are points, and the time step is [number]. ):
[0131] Joint velocity change index: .
[0132] Joint jerk energy: .in, .
[0133] End-path bend degree (second difference): Recommended coefficient: .
[0134] Will Mapped to Smoothness fraction (exponential decay form, avoiding the influence of extreme values):
[0135]
[0136] in: Represents the successful trajectories in the dataset. Median or mean (can be set according to prior knowledge, for example) ). Representing the sensitivity coefficient, recommended This form guarantees ,and The larger the value, the lower the score.
[0137] Physical feasibility score Check whether the trajectory violates the following physical constraints, using a penalty product approach: .in Indicates the first The degree of normalization violation of class constraints Indicates the penalty coefficient (recommended) Common constraints and their degree of violation definitions:
[0138] Joint position limit violation: .in , .
[0139] Joint velocity limit violation: .
[0140] Joint acceleration limit violation (if available): .
[0141] Collision detection: In the event of a collision, if it occurs, it is usually directly caused to... (i.e., setting) (The product is 0). If there are no violations, then all , .
[0142] The final score is determined; substitute the above results into the initial formula: The default values for the parameters are as follows: ; ; : Statistical analysis from successful trajectory samples (or set to 1.0); (Regarding collisions) ).
[0143] Stratified sampling strategy: Stratify the data according to the quality score, design comparative experiments (high-quality data group, medium-quality data group, low-quality data group, random sampling group) and control the consistency of data volume.
[0144] Policy fine-tuning protocol: Define a fine-tuning process based on GR00T N1.6 (learning rate, number of training rounds, batch size, data augmentation strategy) to ensure that these variables are consistent for each experiment for the same task. For example, set the learning rate to 0.00001 for the orange catching task to ensure the fairness of comparative experiments.
[0145] Benchmark Suite: Evaluate the task success rate, average completion time, and generalization ability of fine-tuned policies on the standard embodied intelligence benchmark suite.
[0146] Validation report generation: Automatically generates a data validity verification report, including data quality distribution, comparative experimental results, and statistical test conclusions, demonstrating the role of simulation verification in improving data quality and strategy performance.
[0147] By employing the aforementioned technical solutions, this invention, based on a world model, can generate visually realistic embodied operation video sequences, accurately mapping the generated embodied trajectory operation data to a physical simulation environment and verifying its physical feasibility. Furthermore, the world model of this invention can generate diverse operation trajectories based on the same initial conditions, accurately replicating the simulation scene through the Replay method, enabling fair comparison and system evaluation of different generated trajectories. Moreover, since traditional simulation verification only focuses on the success rate of trajectory execution in simulation but lacks correlation analysis with the performance of the actual policy model, this invention evaluates the effectiveness of simulation verification data and establishes a mapping relationship between simulation verification indicators and actual policy performance.
[0148] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.
[0149] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0150] Figure 2 This is a structural block diagram of a simulation verification device with specific data in some embodiments of the present invention. For example... Figure 2 As shown, the simulation verification device for embodied data includes:
[0151] The embodied operation trajectory generation module 100 is used to generate multiple embodied operation trajectories based on scene information and agent information using a video generation model. Each embodied operation trajectory includes an image sequence, a state sequence, and an action sequence.
[0152] The scene replication module 200 is used to replicate the scene based on scene information, agent information and multiple embodied operation trajectories using the playback method of the simulation platform, so as to obtain multiple actual execution trajectories generated by the agent in the simulation scene in a one-to-one correspondence with multiple embodied operation trajectories.
[0153] The trajectory execution verification module 300 is used to verify multiple actual execution trajectories in a simulation scenario and obtain multiple verification results corresponding to the multiple actual execution trajectories.
[0154] The data validity assessment module 400 is used to assess the data validity of multiple validation results and obtain the correlation between data quality and strategy performance based on the data validity assessment results.
[0155] In some embodiments of the present invention, the embodied operation trajectory generation module 100 is used to extract the agent model, environment layout information, initial pose of the agent, and initial pose of the interactive object of the agent from scene information and agent information; the embodied operation trajectory generation module 100 is also used to perform embodied operation trajectory generation processing based on the agent model, environment layout information, initial pose of the agent, and initial pose of the interactive object of the agent using a preset world model, to obtain multiple embodied operation trajectories.
[0156] In some embodiments of the present invention, the scene replication module 200 is used to construct a simulation scene using a simulation platform based on the agent model, environmental layout information, the initial pose of the agent, and the initial pose of the interactive objects of the agent; the scene replication module 200 is also used to use the playback method of the simulation platform to control the agent to perform corresponding actions in the simulation scene according to multiple embodied operation trajectories and collect actual action parameters to obtain multiple actual execution trajectories.
[0157] In some embodiments of the present invention, the scene replication module 200 is used to determine the transformation matrix between the first coordinate system of the preset world model and the second coordinate system of the simulation scene; the scene replication module 200 is also used to obtain the correspondence between the joint information output by the intelligent agent in the preset world model and the joint information of the intelligent agent in the virtual environment of the simulation platform; the scene replication module 200 is also used to construct a simulation scene using the virtual environment of the simulation platform based on simulation physical parameters, intelligent agent model, environmental layout information, initial pose of the intelligent agent, initial pose of interactive objects with the intelligent agent, and transformation matrix and correspondence, wherein the simulation physical parameters include at least one of gravity parameters, friction coefficient and collision deformation parameters.
[0158] In some embodiments of the present invention, the scene replication module 200 is used to disable random noise and run the playback method with a fixed time step and a fixed random seed when the simulation scene control agent executes corresponding actions in sequence according to multiple embodied operation trajectories and collects actual action parameters to obtain multiple actual execution trajectories using the playback mechanism of the simulation platform.
[0159] In some embodiments of the present invention, the trajectory execution verification module 300 is used to verify the execution result information of multiple tasks corresponding to multiple actual execution trajectories in the simulation scenario based on multiple task targets corresponding to multiple embodied operation trajectories; the trajectory execution verification module 300 is also used to perform statistics on the task execution results corresponding to the same task to obtain task statistics results.
[0160] In some embodiments of the present invention, the task statistics results include execution success rate and trajectory smoothness; the data validity evaluation module 400 is used to select multiple embodied operation trajectories corresponding to execution success rates greater than a first preset threshold and trajectory smoothness greater than a second preset threshold from the task statistics results to construct a dataset; the data validity evaluation module is also used to train and tune the visual language action model based on the dataset; the data validity evaluation module 400 is also used to use the trained and tuned visual language action model to perform performance evaluation on a standard embodied benchmark test set; the data validity evaluation module 400 is also used to compare and analyze data screening strategies under different simulation verification thresholds based on the performance evaluation results to obtain the correlation between data quality and strategy performance.
[0161] It should be noted that the specific implementation of the embodied data simulation verification device in this disclosure is similar to the specific implementation of the embodied data simulation verification method in this disclosure, and the technical effects of the embodied data simulation verification device in this disclosure are similar to the technical effects of the embodied data simulation verification method in this disclosure. For details, please refer to the description of the embodied data simulation verification method section. In order to reduce redundancy, it will not be described in detail.
[0162] In addition, this disclosure also provides an electronic device, including:
[0163] Memory, used to store computer programs;
[0164] A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the simulation verification method for embodied data as described in any of the above embodiments of this disclosure.
[0165] Below, for reference Figure 3 To describe an electronic device according to embodiments of this disclosure. For example... Figure 3 As shown, the electronic device includes one or more processors and memory.
[0166] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.
[0167] The memory can store one or more computer program products, and the memory can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products can be stored on the computer-readable storage medium, and the processor can run the computer program products to implement the embody data simulation verification methods of the various embodiments of this disclosure described above, and / or other desired functions.
[0168] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).
[0169] In addition, the input device may also include, for example, a keyboard, a mouse, etc.
[0170] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.
[0171] Of course, for the sake of simplicity, Figure 3 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.
[0172] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the simulation verification methods for embodied data according to various embodiments of this disclosure as described in the foregoing sections of this specification.
[0173] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of this disclosure. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0174] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the simulation verification method for embodied data according to various embodiments of this disclosure as described in the foregoing portion of this specification.
[0175] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.
[0176] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0177] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0178] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.
[0179] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.
[0180] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.
[0181] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.
[0182] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.
[0183] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.
Claims
1. A simulation verification method for embodied data, characterized in that, include: Using a video generation model, multiple embodied operation trajectories are generated based on scene information and agent information. Each embodied operation trajectory includes an image sequence, a state sequence, and an action sequence. Using the playback method of the simulation platform, the scene is replicated based on the scene information, the agent information and the multiple embodied operation trajectories, to obtain multiple actual execution trajectories generated by the agent in the simulation scene in response to the multiple embodied operation trajectories. The multiple actual execution trajectories are verified in the simulation scenario to obtain multiple verification results corresponding to the multiple actual execution trajectories; The data validity of the multiple verification results is evaluated, and the correlation between data quality and strategy performance is obtained based on the data validity evaluation results.
2. The method according to claim 1, characterized in that, The method utilizes a video generation model to generate multiple embodied operation trajectories based on scene information and agent information, including: Extract the agent model, environment layout information, initial pose of the agent, and initial pose of the interactive object from the scene information and agent information. Using a preset world model, based on the agent model, environmental layout information, the initial pose of the agent, and the initial pose of the interactive objects of the agent, embodied operation trajectory generation processing is performed to obtain the multiple embodied operation trajectories.
3. The method according to claim 2, characterized in that, The replay mechanism of the simulation platform performs scene replication processing based on the scene information, the agent information, and the multiple embodied operation trajectories, resulting in multiple actual execution trajectories generated by the agent in the simulation scene, corresponding one-to-one with the multiple embodied operation trajectories, including: Using the simulation platform, the simulation scene is constructed based on the agent model, the environmental layout information, the initial pose of the agent, and the initial pose of the objects that can interact with the agent. By employing a playback method on a simulation platform, the intelligent agent is controlled to sequentially execute corresponding actions according to the multiple embodied operation trajectories in the simulation scenario, and actual action parameters are collected to obtain the multiple actual execution trajectories.
4. The method according to claim 3, characterized in that, The process of constructing the simulation scene using the simulation platform, based on simulation physical parameters, the intelligent agent model, the environmental layout information, the initial pose of the intelligent agent, and the initial poses of objects that can interact with the intelligent agent, includes: Determine the transformation matrix between the first coordinate system of the preset world model and the second coordinate system of the simulation scene; Obtain the correspondence between the joint information output by the agent in the preset world model and the joint information of the agent in the virtual environment of the simulation platform; Using the virtual environment of the simulation platform, the simulation scene is constructed based on the simulation physical parameters, the intelligent agent model, the environment layout information, the initial pose of the intelligent agent, the initial pose of the interactive objects of the intelligent agent, the transformation matrix and the correspondence. The simulation physical parameters include at least one of gravity parameters, friction coefficient and collision deformation parameters.
5. The method according to claim 3, characterized in that, The replay mechanism using the simulation platform, in the process of controlling the agent in the simulation scene to sequentially execute corresponding actions according to the multiple embodied operation trajectories and collecting actual action parameters to obtain the multiple actual execution trajectories, further includes: Disable random noise and run the playback method with a fixed time step and a fixed random seed.
6. The method according to any one of claims 1-5, characterized in that, The process involves verifying the multiple actual execution trajectories in the simulation scenario to obtain multiple verification results corresponding one-to-one with the multiple actual execution trajectories, including: Based on the multiple task objectives corresponding to the multiple embodied operation trajectories, the execution result information of the multiple tasks corresponding to the multiple actual execution trajectories in the simulation scenario is verified. The task execution results for the same task are statistically analyzed to obtain task statistics.
7. The method according to claim 6, characterized in that, The task statistics include execution success rate and trajectory smoothness; The step of evaluating the data validity of the multiple verification results and obtaining the correlation between data quality and strategy performance based on the data validity evaluation results includes: Select multiple embodied operation trajectories from the task statistics results that have an execution success rate greater than the first preset threshold and a trajectory smoothness greater than the second preset threshold, and construct a dataset; The visual language action model was trained and its parameters were tuned based on the dataset. The performance of the trained and hyperparameter-tuned visual language action model was evaluated on a standard embodied benchmark test set. Based on the performance evaluation results, a comparative analysis was conducted on data filtering strategies under different simulation verification thresholds to obtain the correlation between data quality and strategy performance.
8. A simulation verification device for embodied data, characterized in that, include: The embodied operation trajectory generation module is used to generate multiple embodied operation trajectories based on scene information and agent information using a video generation model. Each embodied operation trajectory includes an image sequence, a state sequence, and an action sequence. The scene replication module is used to perform scene replication processing based on the scene information, the agent information and the multiple embodied operation trajectories using the playback method of the simulation platform, so as to obtain multiple actual execution trajectories generated by the agent in the simulation scene in a one-to-one correspondence with the multiple embodied operation trajectories; The trajectory execution verification module is used to verify the multiple actual execution trajectories in the simulation scenario and obtain multiple verification results corresponding to the multiple actual execution trajectories. The data validity assessment module is used to assess the validity of the multiple verification results and obtain the correlation between data quality and strategy performance based on the data validity assessment results.
9. An electronic device, characterized in that, include: Memory, used to store computer program products; A processor for executing a computer program product stored in the memory, wherein when the computer program product is executed, it implements the method described in any one of claims 1-7.
10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-6.