Robot simulation data generalization method and device and storage medium

Abstract

The embodiment of the invention provides a robot simulation data generalization method and device, a storage medium and a computer program product. Comprising the steps of obtaining scene parameters of a target task in a robot simulation environment; decomposing the target task into a plurality of sub-tasks, wherein each sub-task is at least associated with one specific operation object; selecting a matched source demonstration fragment from a pre-stored robot demonstration data fragment library according to the task type of the subtask and the associated specific operation object; on the basis of the difference between the actual pose of a target operation object associated with the current subtask and the demonstration pose of the target operation object in the selected source demonstration fragment, space coordinate transformation with the target operation object as the center is carried out on a robot motion track in the source demonstration fragment, noise is injected, and a preliminary generalization track is generated; and performing physical verification and optimization on all the preliminary generalization trajectories to obtain a simulation trajectory data set for robot simulation learning training.

Description

Technical Field

[0001] This application relates to the field of robotics technology, specifically to a method, apparatus, storage medium, and computer program product for generalizing robot simulation data. Background Technology

[0002] Robotics technology is rapidly developing towards complex physical interactions and embodied intelligence, which has created an unprecedented demand for large-scale, high-fidelity simulation data. Under the embodied intelligence paradigm, robots need to learn physical laws and perform complex tasks through active and continuous multimodal interactions with the environment. Therefore, simulation platforms must not only provide realistic vision, but also accurately simulate multiple complex factors such as physical interactions, body dynamics, sensor noise, and environmental behavior.

[0003] However, current widely adopted simulation data generation methods heavily rely on manual parameter settings and numerous repetitive experiments. When faced with highly heterogeneous, dynamically changing real-world tasks and long-tail scenarios, these methods exhibit bottlenecks such as low data generation efficiency, high costs, and difficulty in systematically covering massive environmental variables. These methods are typically tightly coupled with specific simulation environments or task settings. Changing tasks or adjusting models often requires redesigning and generating new data, resulting in low reusability, long update cycles, and poor flexibility. The generated data often lacks sufficient generalization ability, making it difficult to support cross-scenario transfer learning. This can easily lead to perceptual biases, inaccurate actions, and abnormal interactions in actual deployments, severely impacting the model's adaptability, safety, and task success rate. Summary of the Invention

[0004] The purpose of this application is to provide a robot simulation data generalization method, device, storage medium, and computer program product to solve the technical problems of existing simulation data generation methods being inefficient, costly, lacking diversity in the generated data and being strongly coupled with specific tasks or scenarios, making it difficult to effectively generalize to new environments or different tasks, and resulting in a decline in the performance of robot models in the real world and a significant gap between simulation and reality.

[0005] To achieve the above objectives, the first aspect of this application provides a robot simulation data generalization method, the method comprising: In a robot simulation environment, the scene parameters of the target task are obtained. The robot simulation data generalization scene parameters include at least the object identifier and initial pose of the operation object. The robot simulation data generalization target task is decomposed into multiple sub-tasks that the robot executes sequentially, and each sub-task is associated with at least one specific operation object; For each subtask, the task type and associated specific operation object of the subtask are generalized based on the robot simulation data, and a matching source demonstration segment is selected from the pre-stored robot demonstration data segment library. Based on the difference between the actual pose of the target operation object associated with the current subtask and the demonstration pose of the target operation object generalized from the robot simulation data in the selected source demonstration segment, the robot motion trajectory in the robot simulation data generalization source demonstration segment is transformed into spatial coordinates centered on the target operation object of the robot simulation data generalization, and noise is injected to generate a preliminary generalized trajectory adapted to the current subtask. Physical verification and trajectory optimization are performed on the preliminary generalized trajectories of all sub-tasks to obtain simulation trajectory data for robot imitation learning training.

[0006] In this embodiment of the application, the method further includes a step of constructing a robot demonstration data fragment library. The robot simulation data generalization construction step includes: collecting the original demonstration data when the user operates the robot to complete the task; identifying the task boundary signals triggered by changes in the state of the operating object or preset instructions in the robot simulation data generalization original demonstration data; dividing the robot simulation data generalization original demonstration data into multiple independent demonstration data fragments according to the robot simulation data generalization task boundary signals, and extracting the operating object identifier, demonstration pose and robot motion trajectory associated with each fragment, and storing them in the robot demonstration data fragment library.

[0007] In this embodiment, based on the robot simulation data generalization task boundary signal, the original demonstration data of robot simulation data generalization is divided into multiple independent demonstration data segments, and the operation object identifier, demonstration pose, and robot motion trajectory associated with each segment are extracted. This includes: for each demonstration data segment, establishing a semantic binding relationship between the robot simulation data generalization demonstration data segment and the operation object associated with the robot simulation data generalization demonstration data segment, and performing coordinate transformation on the robot motion trajectory of the robot simulation data generalization demonstration data segment based on the object center coordinate system of the operation object associated with the robot simulation data generalization, so as to establish a geometric binding relationship between the robot simulation data generalization demonstration data segment and the operation object associated with the robot simulation data generalization.

[0008] In this embodiment of the application, selecting a matching source demonstration segment from a pre-stored robot demonstration data segment library includes: generalizing the task type, associated object attributes, and current scene constraints of the sub-task based on the robot simulation data, and executing a selection strategy; the robot simulation data generalization selection strategy includes at least one of the nearest neighbor object matching strategy, robot distance optimization strategy, and success rate weighting strategy; based on the calculation result of the selection strategy, determining at least one source demonstration segment with the highest matching degree for each sub-task as the selection result.

[0009] In this embodiment, the spatial coordinate transformation centered on the target object of the robot simulation data generalization includes: determining the rigid body transformation matrix between the demonstration pose of the target object of the robot simulation data generalization in the robot simulation data generalization source demonstration segment and the actual pose of the target object of the robot simulation data generalization in the current subtask; and transforming the robot motion trajectory in the robot simulation data generalization source demonstration segment from the coordinate system centered on the demonstration object to a new coordinate system centered on the current target object through the robot simulation data generalization rigid body transformation matrix to obtain the redirected trajectory.

[0010] In this embodiment of the application, physical verification includes: performing execution verification in a simulation environment on the generated robot simulation data to generalize a preliminary generalized trajectory; and selecting one or more trajectories that meet preset quality standards based on the verification results; wherein the preset quality standards for robot simulation data generalization include at least task completion, collision-free motion trajectory, and motion smoothness.

[0011] In this embodiment of the application, the generalization method further includes: standardizing and encapsulating the simulation trajectory data that has been physically verified and optimized, adding metadata indexes, and constructing a standardized robot imitation learning simulation training dataset.

[0012] A second aspect of this application provides a robot simulation data generalization apparatus, comprising: a memory configured to store instructions; and a processor configured to retrieve instructions from the memory and, when executing the instructions, to implement any of the aforementioned robot simulation data generalization methods.

[0013] A third aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform any of the robot simulation data generalization methods described above.

[0014] The fourth aspect of this application provides a computer program product, including a computer program, a robot simulation data generalization computer program that, when executed by a processor, implements any of the above-mentioned robot simulation data generalization methods.

[0015] This application proposes a method for generalizing robot simulation data. By decomposing human demonstration data into sub-task fragments bound to the operating object and constructing a skill library, and utilizing rigid body coordinate transformation and controllable noise, these fragments are adapted to new scenarios to generate diverse trajectory data. This significantly improves the efficiency and quality of simulation data generation, producing a large amount of high-fidelity training data that conforms to physical laws using only a small number of demonstrations. This effectively narrows the gap between simulation and real-world environments, providing a more reliable training foundation for robot learning.

[0016] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 The schematic diagram illustrates a flowchart of a robot simulation data generalization method according to an embodiment of this application; Figure 2 A schematic diagram illustrating the operation flowchart of a robot simulation data generalization system according to an embodiment of this application is shown. Figure 3 This illustration schematically shows the operation flowchart of the multimodal teleoperation data acquisition module according to an embodiment of the present application; Figure 4 This illustration schematically shows a flowchart of the operation of the object-centered subtask decomposition module according to an embodiment of this application; Figure 5 This illustration schematically shows the operation flowchart of the multidimensional spatial transformation generalization and parallel trajectory generation module according to an embodiment of this application; Figure 6 The diagram illustrates the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0019] Figure 1 The illustration shows a flowchart of a robot simulation data generalization method according to an embodiment of this application. Figure 1 As shown in one embodiment of this application, a robot simulation data generalization method is provided, including the following steps: Step 101: In the robot simulation environment, obtain the scene parameters of the target task. The robot simulation data generalization scene parameters include at least the object identifier and initial pose of the operation object.

[0020] In the robot simulation platform, the system obtains the scene parameters for the current task by parsing the simulation environment configuration interface or scene description file. Scene parameters include at least the object identifier and the initial pose of the manipulated object. The object identifier is a unique identifier assigned to each manipulable object in the scene, used for semantic matching with objects in the demonstration data fragment library in subsequent steps. This identifier is typically stored as a string or an enumeration value. The initial pose of the manipulated object refers to the pose of each object in the simulation world coordinate system, including its three-dimensional position coordinates (X, Y, Z) and three-dimensional rotational attitude. This pose data is obtained in real-time through the simulation engine's physics state query interface.

[0021] In one embodiment, the system first loads the simulation scene file corresponding to the target task, initializes the physics engine, calls the environment interface to traverse the list of predefined operation objects in the scene, extracts their identifiers and current poses one by one, and forms a structured scene parameter dataset for subsequent subtask decomposition and fragment matching.

[0022] Step 102: Decompose the robot simulation data generalization target task into multiple sub-tasks that the robot executes in sequence, with each sub-task associated with at least one specific operation object.

[0023] The system receives a target task, which is a configuration file for a target simulation scenario. This file specifies the objects existing in the environment and the target state description to be achieved. The system extracts one or more core operational intentions from the target state description, such as grasping, placing, and stacking. Then, it identifies the target object associated with each operational intention. The extracted operational intentions and associated target object information are combined to form one or more sub-task-level matching requirement descriptions. For example, for the target of placing A onto B, two matching requirements may be generated: Requirement 1: {Operation Intention: "Place", Reference Object: "Object B"}; Requirement 2: {Operation Intention: "Grasp", Operation Object: "Object A"}. In this embodiment, the grasping and placing requirements are not for planning robot actions, but rather for retrieving source fragments with similar semantics and object relationships from the demonstration pool. The above sub-task matching requirement descriptions, along with the geometric information of the objects in the scenario, are input into the strategy selection module.

[0024] Step 103: For each subtask, based on the robot simulation data, generalize the task type of the subtask and the specific operation object associated with it, and select a matching source demonstration segment from the pre-stored robot demonstration data segment library.

[0025] In one embodiment, the method further includes a step of constructing a robot demonstration data fragment library. The robot simulation data generalization construction step includes: collecting raw demonstration data when a user operates the robot to complete a task; identifying task boundary signals triggered by changes in the state of the manipulated object or preset instructions in the raw demonstration data; dividing the raw demonstration data into multiple independent demonstration data fragments based on the task boundary signals, and extracting the associated manipulated object identifier, demonstration pose, and robot motion trajectory for each fragment, storing them in the robot demonstration data fragment library. The operator controls the virtual robot through a multimodal teleoperation interface integrated into the Isaac Lab simulation environment. The system synchronously records data streams from the VR headset, hand tracking device, keyboard SE3 controller, and motion capture system, and ensures time alignment of data from each channel through a real-time synchronization manager. The recorded raw data is stored in HDF5 format and includes at least a timestamp, robot end effector pose, manipulated object pose, and gripper state. The system monitors or analyzes subtask completion signals in the original demonstration data in real time or offline, detecting transition points from 0 to 1 in the signal's time series. Using these transition points as reference points, the system reverse-locates the corresponding subtask start point, thus determining the start and end time boundaries of each independent subtask. Based on the obtained start and end times, the system extracts data within the corresponding time window from the original data, forming discrete subtask data blocks. For each data block, the system determines the identifier of at least one operation object associated with the subtask segment by querying the object reference mapping in the environment configuration or performing heuristic object association analysis. Simultaneously, it extracts the robot motion trajectory sequence corresponding to the segment and the demonstration pose of the associated object at the start of the segment. The storage structure of each stored segment includes the associated object identifier, object demonstration pose, robot motion trajectory, and subtask semantic tags, forming a structured skill template that can be queried and invoked in subsequent steps.

[0026] In one embodiment, based on the robot simulation data generalization task boundary signal, the original demonstration data of the robot simulation data generalization is divided into multiple independent demonstration data segments, and the operation object identifier, demonstration pose, and robot motion trajectory associated with each segment are extracted. This includes: for each demonstration data segment, establishing a semantic binding relationship between the robot simulation data generalization demonstration data segment and the operation object associated with the robot simulation data generalization demonstration data segment, and performing coordinate transformation on the robot motion trajectory of the robot simulation data generalization demonstration data segment based on the object center coordinate system of the operation object associated with the robot simulation data generalization, to establish a geometric binding relationship between the robot simulation data generalization demonstration data segment and the operation object associated with the robot simulation data generalization. By reading a predefined object reference mapping table from the simulation environment configuration, the specific object identifier corresponding to the operation intention of the segment is determined; if the mapping information is missing, the most likely target object to be operated and its confidence level are inferred and recorded by analyzing the interaction distance and posture change patterns between the robot and various objects in the scene in the segment data. Based on the object center coordinate system of the associated operation object, the coordinate transformation is performed on the robot motion trajectory recorded in the segment to establish a geometric binding relationship. Specifically, the pose of the object in the world coordinate system at the start of the segment is obtained, and a central coordinate system for the object is constructed based on this pose. Then, the pose sequence of the robot's end effector recorded within the segment in the world coordinate system is uniformly converted into a pose sequence relative to the central coordinate system of the object. After this conversion, the robot's motion trajectory is represented as a series of relative pose changes relative to the manipulated object itself. Through the above process, each segment stored in the demonstration data segment library becomes a structured skill template that simultaneously contains semantic and geometric binding relationships.

[0027] In one embodiment, selecting matching source demonstration segments from a pre-stored robot demonstration data segment library includes: generalizing the task type, associated object attributes, and current scene constraints of the subtask based on robot simulation data, and executing a selection strategy; the robot simulation data generalization selection strategy includes at least one of the following: nearest neighbor object matching strategy, robot distance optimization strategy, and success rate weighted strategy; based on the calculation results of the selection strategy, determining at least one source demonstration segment with the highest matching degree for each subtask as the selection result. The system executes a pre-set intelligent selection strategy based on the semantic label of the task type of the current subtask to be processed, the attributes of its associated operation objects, and the constraints of the current scene. The selection strategy is maintained and executed by the strategy selection module, and specifically includes at least one of the following: nearest neighbor object matching strategy: calculating the similarity of the current target object with the objects bound to each segment in the segment library in terms of key attributes, and prioritizing the segment with the closest object features. Robot distance optimization strategy: evaluating the approximate path length or joint change required to execute the motion trajectory recorded by different segments in the segment library from the current state of the robot, and prioritizing the segment with the lowest motion cost. Success rate weighted strategy: Based on the statistical weight of each segment's historical generalization usage and successful generation of valid trajectories, segments with high historical success rates are prioritized. Selection strategies can be used individually or in combination. Based on the strategy calculation results, the system calculates a matching score for one or more candidate segments for the current subtask, and finally determines the at least one source demonstration segment with the highest matching score as the selection result, outputting it to the subsequent transformation module.

[0028] Step 104: Based on the difference between the actual pose of the target operation object associated with the current subtask and the demonstration pose of the target operation object generalized from the robot simulation data in the selected source demonstration segment, perform a spatial coordinate transformation centered on the target operation object in the robot simulation data generalization source demonstration segment on the robot motion trajectory, and inject noise to generate a preliminary generalized trajectory adapted to the current subtask.

[0029] In one embodiment, the spatial coordinate transformation centered on the target object of the robot simulation data generalization includes: determining the rigid body transformation matrix between the demonstration pose of the target object in the robot simulation data generalization source demonstration segment and the actual pose of the target object in the current subtask; transforming the robot motion trajectory in the robot simulation data generalization source demonstration segment from a coordinate system centered on the demonstration object to a new coordinate system centered on the current target object through the robot simulation data generalization rigid body transformation matrix to obtain the redirected trajectory. The system reads the initial pose of the target object recorded in the source demonstration segment at the time of recording, and simultaneously obtains the actual initial pose of the target object in the current subtask scene. The system calculates the complete spatial transformation parameters required to move and rotate the object from the demonstration pose to the actual pose, which are represented by a rigid body transformation matrix. The system reads the robot motion trajectory stored in the source demonstration segment, which is recorded with the demonstration pose as the reference coordinate system. The system applies the rigid body transformation matrix calculated in the previous step to each pose point of the motion trajectory. This operation transforms the entire motion trajectory from the original coordinate system centered on the demonstration object to a new coordinate system centered on the current target object, thus obtaining a redirected trajectory described in the new object-relative coordinate system.

[0030] In one embodiment, after generating the redirection trajectory, the system adds random perturbations according to preset rules to simulate operational differences in a real environment. Specifically, this includes adding small offsets to the three-dimensional coordinates of the movement path, introducing minor angle changes in the grasping posture, and randomly adjusting the speed of the entire action. These controllable noises ensure that each generated trajectory exhibits reasonable and natural differences while maintaining correct operational logic, thereby significantly increasing the diversity and realism of the training data.

[0031] Step 105: Perform physical verification and trajectory optimization on the preliminary generalized trajectories of all sub-tasks to obtain simulation trajectory data for robot imitation learning training.

[0032] The system executes the initial generalized trajectories corresponding to all subtasks synchronously in a parallel simulation environment. Each simulation instance runs independently, with the physics engine calculating the complete interaction process between the robot and the environment as it moves along the trajectory in real time, and recording the final task completion status, collision detection logs, and motion state data. Based on the verification results, the system evaluates each trajectory according to preset quantification standards. Trajectories that fail verification are discarded, while those that pass verification are included in the valid set. The system then aligns and logically concatenates all valid trajectories that pass verification according to the execution order of their respective subtasks, combining them into complete simulation trajectory data.

[0033] In one embodiment, physical verification includes: generating a preliminary generalized trajectory from the generated robot simulation data and performing execution verification in a simulation environment; based on the verification results, selecting one or more trajectories that meet preset quality standards; wherein, the preset quality standards for robot simulation data generalization include at least task completion, collision-free motion trajectory, and motion smoothness. The system replays and executes the generated preliminary generalized trajectory in the simulation environment. The simulation environment calculates and simulates all interactions between the robot and other objects in the scene when the robot moves strictly according to the trajectory, based on its physics engine, and records the state data of the entire execution process. Based on the simulation execution results, the system performs multi-dimensional quantitative evaluation of each trajectory. The evaluation criteria include at least: determining whether the preset target state of the sub-task is achieved after the trajectory is executed; checking whether the end effector of the robot has unexpected collisions with non-operational objects or the environment during the entire movement process; analyzing the continuity of joint angle changes or end effector acceleration, evaluating the smoothness of the trajectory movement, and avoiding abrupt changes or unnatural jitters. Based on preset quality thresholds, the system selects trajectories that meet the requirements in all the above dimensions as valid simulation data output. Tracks that fail verification will be discarded or marked for analysis.

[0034] In one embodiment, the generalization method further includes: standardizing and encapsulating the simulation trajectory data that has passed physical verification and trajectory optimization, and adding metadata indexes to construct a standardized robot imitation learning simulation training dataset. The system performs uniform formatting on all simulation trajectory data that has passed physical verification and optimization. Each trajectory, along with its associated scene configuration, object information, and verification process records, is encapsulated into an independent data file conforming to the HDF5 standard format. During encapsulation, the system generates and writes a structured metadata index for each data file. This index includes at least the task type identifier, the identifier and category of the operated object, the trajectory success rate score, the motion smoothness index, the generation timestamp, and the corresponding source demonstration segment reference information. Through the above processing, a large amount of scattered trajectory data is integrated into a standardized dataset with a unified interface and complete descriptive information, which can be directly used as input to train various robot imitation learning models.

[0035] In one embodiment, a robot simulation data generalization system is provided. It includes: a multimodal teleoperation data acquisition module for acquiring raw demonstration data from human experts operating the robot to complete tasks; this module integrates multiple teleoperation device interfaces and a synchronization management unit to achieve real-time fusion and structured recording of multi-source operation data; an object-centered subtask decomposition module for dividing the acquired raw demonstration data into multiple independent subtask demonstration segments based on object state changes or preset command signals, and establishing semantic and geometric binding relationships between each segment and a specific operation object to construct a structured robot demonstration data segment library; and a multidimensional spatial transformation generalization and parallel trajectory generation module for intelligently selecting matching source segments from the demonstration data segment library according to the target task scenario, generating a preliminary generalized trajectory through spatial coordinate transformation centered on the target object and noise injection, and outputting a standardized simulation trajectory dataset after parallel physical verification and quality screening.

[0036] Figure 1 This is a flowchart illustrating a robot simulation data generalization method in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0037] In one embodiment, Figure 2The schematic diagram illustrates the operation flowchart of a robot simulation data generalization system according to an embodiment of this application. First, multimodal teleoperation data acquisition is performed. Operators control a virtual robot to complete tasks in the simulation environment through integrated VR headsets, hand tracking devices, and keyboards as teleoperation interfaces. The system synchronously records pose, gesture, and control command data from all devices, ensuring time alignment through a real-time synchronization manager, and stores it in a structured format as raw demonstration data. Next, object-centric subtask decomposition (automatic annotation) is performed. Based on the subtask completion signals in the raw demonstration data, the system automatically identifies task boundaries and segments independent subtask fragments. For each fragment, its operational object is determined through object association analysis, establishing semantic binding; simultaneously, the robot's motion trajectory in the fragment is transformed to a coordinate system centered on that object, establishing geometric binding. This ultimately forms a structured subtask fragment library with complete annotation information. Finally, multidimensional space transformation generalization and parallel trajectory generation are performed. For new target task scenarios, the system intelligently selects matching source segments from the subtask segment library, adapts the segment trajectories to the target objects in the new scenario through rigid body transformation, and injects position, rotation, and timing noise to increase diversity. Parallel simulation is used to physically verify and screen the quality of the generated preliminary trajectories, outputting a standardized simulation trajectory dataset for robot imitation learning training.

[0038] In one embodiment, Figure 3 The schematic diagram illustrates the operation flowchart of the multimodal teleoperation data acquisition module according to an embodiment of this application. The system starts the Isaac Lab-based environment framework, initializes the VR headset, full-body motion capture device, and keyboard SE3 control interface in parallel, and completes the loading of drivers for each device and the establishment of communication links. It receives sensor data streams from each device in real time through asynchronous I / O channels and uses a timestamp synchronization mechanism to align the data streams in time. Through the device adapter, the spatial positioning data from the VR headset, the posture data from hand tracking, and the control commands from the keyboard are uniformly converted into SE3 operation commands recognizable by the robot in the simulation environment. Based on the fused operation commands, the system generates corresponding robot joint control signals, specifically including position, velocity, or torque control commands for each joint of the arm and the right-hand end effector. The system drives the robot in the simulation environment to perform actions. The aforementioned control commands are sent to the simulation engine, and the physics engine calculates and updates the robot's joint states and end-effector poses based on the commands, thereby reproducing the operator's teleoperation actions in the simulation environment. While the robot performs actions, the HDF5 data logger synchronously records the robot end effector pose, object pose, gripper state, and operation intention label at each time step according to the DatagenInfo data structure, forming a complete and structured multimodal demonstration data file.

[0039] In one embodiment, Figure 4 The flowchart illustrating the operation of the object-centric subtask decomposition module according to an embodiment of this application is shown. The module acquires raw acquisition data generated by the multimodal teleoperation data acquisition module, which has been structured and stored using an HDF5 data logger. The system scans the subtask completion signal time series recorded in the raw data, identifies the transition point from 0 to 1 as the subtask end time, and reverses to locate the corresponding start boundary, thereby determining the start and end time index of each subtask. Based on the predefined object reference mapping relationship in the environment configuration, the system determines the specific operation object associated with each subtask segment and establishes semantic binding; if the configuration is missing, heuristic inference is performed by analyzing the interaction patterns between the robot and scene objects within the segment. Based on the identified start and end time indices, the system accurately extracts the end effector posture sequence, object posture change sequence, and gripper action sequence within the corresponding time period from the raw data, and adds semantic tags such as grasping, placing, and stacking to each subtask segment. The extracted data is integrated into a SubTaskSegment object, and its integrity is verified. For each validated subtask segment, an object-centered coordinate transformation is performed to convert the robot's motion trajectory from the world coordinate system to the object-centered coordinate system of its associated object, achieving geometric binding and extracting kinematic features. Based on the semantic and geometric binding relationship, binding constraints are established between the skill segment and its manipulated object, and its transferability score is calculated. Finally, all processed skill segments are integrated and stored in a subtask model library. This library establishes object indexes, semantic indexes, and metadata indexes for each segment, forming a reusable and queryable set of structured skill templates.

[0040] In one embodiment, Figure 5The schematic diagram illustrates the operation flowchart of the multidimensional spatial transformation generalization and parallel trajectory generation module according to an embodiment of this application. The generalization scheduler receives the target task description and accesses a pre-built subtask fragment library. Based on the current subtask type, associated object attributes, and scene constraints, it executes one or more strategies among nearest neighbor object matching, robot distance optimization, and success rate weighting to intelligently select the most matching source demonstration fragment from the subtask fragment library. The selected source demonstration fragment is then sent to the multidimensional spatial transformation engine. This engine calculates the rigid body transformation matrix between the actual pose of the target object in the current scene and the object demonstration pose recorded in the source fragment, and uses this matrix to perform a spatial coordinate transformation centered on the target object on the robot motion trajectory in the source fragment, achieving geometric redirection of the trajectory. The transformed trajectory enters the noise injection generalization module. This module injects controllable noise conforming to a uniform distribution, Gaussian distribution, or a specific structure into the position, rotation, and time dimensions of the trajectory, respectively, generating multiple trajectory variants with reasonable variations in the parameter space. The parallel generation coordinator asynchronously schedules batch trajectory generation tasks to multiple parallel simulation environments for execution, achieving large-scale parallel generation. During the generation process, a quality assessment system checks each generated trajectory in real time, evaluating criteria including physical plausibility, task completion, motion smoothness, and collision-free safety. Only trajectories that pass the evaluation are retained. Simultaneously, an adaptive parameter optimizer dynamically adjusts parameters such as noise intensity and transformation strategies based on the performance metrics of the generated results to continuously optimize the quality of subsequently generated data. Finally, trajectory data that passes all checks are standardized to HDF5 format and appended with a complete metadata index, serving as the final generalized data output, forming a high-quality simulation dataset that can be directly used for imitation learning training.

[0041] In one embodiment, a robot simulation data generalization device (not shown in the figure) is provided, comprising: The memory is configured to store instructions; The processor is configured to retrieve instructions from memory and, when executing instructions, to implement any of the aforementioned robot simulation data generalization methods.

[0042] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and the robot simulation data generalization method can be implemented by adjusting the kernel parameters.

[0043] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0044] This application provides a storage medium on which a program is stored, which, when executed by a processor, implements the above-described robot simulation data generalization method.

[0045] This application provides a processor for running a program, wherein the program executes the above-described robot simulation data generalization method during runtime.

[0046] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor A01, a network interface A02, memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The network interface A02 is used for communication with external terminals via a network connection. When executed by the processor A01, the computer program B02 implements a robot simulation data generalization method.

[0047] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0048] This application provides a computer (electronic) device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements any of the robot simulation data generalization methods described above.

[0049] This application also provides a computer program product that, when executed on a data processing device, is suitable for performing steps of an initialization method for robot simulation data generalization.

[0050] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0051] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0052] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0053] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0054] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0055] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0056] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0057] It should also be noted that the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0058] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for generalizing robot simulation data, characterized in that, The generalization method includes: In a robot simulation environment, scene parameters for the target task are acquired, wherein the scene parameters include at least the object identifier and initial pose of the object being manipulated; The target task is decomposed into multiple sub-tasks that the robot executes sequentially, and each sub-task is associated with at least one specific operation object; For each subtask, a matching source demonstration segment is selected from a pre-stored robot demonstration data segment library based on the task type of the subtask and the specific operation object associated with it; Based on the difference between the actual pose of the target object associated with the current subtask and the demonstration pose of the target object in the selected source demonstration segment, the robot's motion trajectory in the source demonstration segment is transformed into spatial coordinates centered on the target object, and noise is injected to generate a preliminary generalized trajectory adapted to the current subtask; Physical verification and trajectory optimization are performed on the initial generalized trajectories of all sub-tasks to obtain a simulation trajectory dataset for robot imitation learning training.

2. The generalization method according to claim 1, characterized in that, The method also includes a step of constructing a robot demonstration data fragment library, the construction step comprising: Collect raw demonstration data when users operate the robot to complete tasks; Identify task boundary signals in the original demonstration data that are triggered by changes in the state of the operating object or preset instructions. Based on the task boundary signal, the original demonstration data is divided into multiple independent demonstration data segments, and the operation object identifier, demonstration pose and robot motion trajectory associated with each segment are extracted and stored in the robot demonstration data segment library.

3. The generalization method according to claim 2, characterized in that, The step of dividing the original demonstration data into multiple independent demonstration data segments based on the task boundary signal, and extracting the operation object identifier, demonstration pose, and robot motion trajectory associated with each segment, includes: For each demonstration data segment, a semantic binding relationship is established between the demonstration data segment and the associated operation object. Based on the object center coordinate system of the associated operation object, the robot motion trajectory of the demonstration data segment is transformed to establish a geometric binding relationship between the demonstration data segment and the associated operation object.

4. The method according to claim 1, characterized in that, The step of selecting a matching source demonstration segment from a pre-stored library of robot demonstration data segments includes: Based on the task type, associated object attributes, and current scene constraints of the subtask, a selection strategy is executed; the selection strategy includes at least one of the nearest neighbor object matching strategy, robot distance optimization strategy, and success rate weighted strategy. Based on the calculation results of the selection strategy, at least one source demonstration segment with the highest matching degree is determined as the selection result for each subtask.

5. The method according to claim 1, characterized in that, Spatial coordinate transformation centered on the target object includes: Determine the rigid body transformation matrix between the demonstration pose of the target operation object in the source demonstration segment and the actual pose of the target operation object in the current subtask; The robot's motion trajectory in the source demonstration clip is transformed from a coordinate system centered on the demonstration object to a new coordinate system centered on the current target object through the rigid body transformation matrix, thus obtaining the redirected trajectory.

6. The generalization method according to claim 1, characterized in that, The physical verification includes: The generated preliminary generalized trajectory was then validated in a simulation environment. Based on the verification results, one or more trajectories that meet the preset quality standards are selected; wherein, the preset quality standards include at least task completion, collision-free motion trajectory, and motion smoothness.

7. The generalization method according to claim 1, characterized in that, The generalization method further includes: The simulated trajectory data that has passed physical verification and trajectory optimization is standardized, encapsulated, and metadata indexes are added to construct a standardized robot imitation learning simulation training dataset.

8. A robot simulation data generalization device, characterized in that, include: The memory is configured to store instructions; The processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the robot simulation data generalization method according to any one of claims 1 to 7.

9. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the robot simulation data generalization method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the robot simulation data generalization method according to any one of claims 1 to 7.

Images