Body-aware data generation method and system, storage medium, program product

Abstract

This application provides an embodied intelligence data generation method and system, storage medium, and program product. The embodied intelligence data generation method includes: receiving source reference data; determining target scene representation data; determining target action representation data corresponding to the target scene based on the source reference data and the target scene representation data; enabling the target embodied intelligence agent to execute the target task in the target scene within a simulation environment based on the target action representation data, and acquiring process data corresponding to the task execution process and / or result data corresponding to the task execution result; and generating embodied intelligence data based on at least one of the target action representation data, the process data, and the result data. This embodiment improves the efficiency of embodied intelligence data generation and enhances the scene adaptability, task association capability, and training application value of embodied intelligence data.

Description

Technical Field

[0001] This application relates to the field of embodied intelligence technology, and in particular to embodied intelligence data generation methods and systems, storage media, and program products. Background Technology

[0002] Against the backdrop of continuous development in robotics, artificial intelligence, and automatic control technologies, embodied intelligence has gradually become an important research direction in the field of intelligent robotics. Embodied intelligence emphasizes that intelligent agents interact with the external environment through their ontological structure, perception capabilities, and execution capabilities, completing processes such as perception, decision-making, control, and task execution during the interaction. To enable embodied intelligent agents to perform complex tasks such as grasping, transporting, placing, assembling, tool use, and scene interaction, it is usually necessary to build a data foundation related to the task process to support subsequent model training, policy optimization, execution verification, and performance evaluation. However, obtaining high-quality reference execution data during embodied intelligence training is costly, data generalization ability under scene changes is insufficient, and the generated data has a weak correlation with the task execution process.

[0003] Based on this, embodiments of this application provide a method and system for generating embodied intelligent data, a storage medium, and a program product to improve related technologies. Summary of the Invention

[0004] The purpose of this application is to provide a method and system for generating embodied intelligence data, a storage medium, and a program product, to improve the problems of high cost of obtaining high-quality reference execution data, insufficient data generalization ability under scene changes, and weak correlation between generated data and task execution process in the related embodied intelligence training process.

[0005] In a first aspect, embodiments of this application provide a method for generating embodied intelligence data. The method includes: receiving source reference data, which characterizes reference execution information of a source execution entity for a target task; determining target scene characterization data, which characterizes a target scene corresponding to the target task, and includes at least one of object pose information, environmental parameter information, scene observation information, and scene semantic information; determining target action characterization data corresponding to the target scene based on the source reference data and the target scene characterization data, which characterizes the action process of a target embodied intelligence agent performing the target task in the target scene, and includes at least one of target action information, target trajectory information, target control information, and strategy output information; based on the target action characterization data, causing the target embodied intelligence agent to perform the target task in the target scene within a simulation environment, and obtaining process data corresponding to the task execution process and / or result data corresponding to the task execution result; and generating embodied intelligence data based on at least one of the target action characterization data, the process data, and the result data.

[0006] In some embodiments, the source reference data includes at least one of source demonstration information, source action information, source trajectory information, source control information, source state information, and source observation information; the source execution entity includes at least one of physical embodied intelligent agent, virtual embodied intelligent agent, and human-machine teleoperation system.

[0007] In some embodiments, determining the target scene representation data includes: randomizing at least one of the object pose, background environment, lighting conditions, and material style in the target scene to obtain the target scene representation data.

[0008] In some embodiments, randomizing the poses of objects in the target scene includes: resetting the target embodied agent to a reference pose and opening the gripper of the target embodied agent; traversing at least some objects in the target scene to obtain the current poses of the corresponding objects, and randomly offsetting at least some objects within a preset offset range to set a new pose for the corresponding objects based on the random offset results.

[0009] In some embodiments, determining the target action representation data corresponding to the target scene based on the source reference data and the target scene representation data includes: determining the end-effector pose in the target scene based on the relative pose relationship between the target object and the end-effector in the source scene and the pose of the target object in the target scene, so that the source scene matches the relative pose relationship between the target object and the end-effector in the target scene; wherein the relative pose relationship between the target object and the end-effector in the source scene is determined based on the source reference data, the target object pose in the target scene is determined based on the target scene representation data, and the target action representation data is determined based on the end-effector pose in the target scene.

[0010] In some embodiments, determining the target action representation data corresponding to the target scene based on the source reference data and the target scene representation data further includes: adding a starting point corresponding to the current position of the target embodied agent; and performing trajectory interpolation processing based on the starting point and the end effector pose to generate a path point sequence.

[0011] In some embodiments, the source reference data includes source action information and / or source trajectory information in the source scene. Determining the target action representation data corresponding to the target scene based on the source reference data and the target scene representation data includes: dividing the reference action process corresponding to the target task into action stages based on the source action information and / or the source trajectory information; converting at least a portion of the source action information and / or source trajectory information into relative action representation data relative to the target object in the source scene; and mapping the relative action representation data to the target action representation data corresponding to the target scene based on the target object pose corresponding to the target scene representation data.

[0012] In some embodiments, mapping the relative motion representation data to target motion representation data corresponding to the target scene based on the target object pose corresponding to the target scene representation data includes: adaptively adjusting the operation points corresponding to the relative motion representation data based on the target object pose and contact position change information, and generating corresponding target motion representation data.

[0013] In some embodiments, after mapping the relative motion representation data to the target motion representation data corresponding to the target scene, the embodied intelligence data generation method further includes: optimizing the target motion representation data based on motion optimization processing so that the optimized target motion representation data satisfies at least one of kinematic constraints, dynamic constraints, smoothness constraints, and obstacle avoidance constraints.

[0014] In some embodiments, the target action representation data includes target action information and / or target trajectory information, and the target embodied intelligent agent uses the control parameter configuration corresponding to the source reference data when performing the target task; wherein, the controller generates target control information based on the target action information and / or the target trajectory information, and in combination with the control parameter configuration.

[0015] In some embodiments, the step of enabling the target embodied intelligent agent to perform the target task in the target scene of the simulation environment based on the target action representation data includes: sending the target scene representation data and the target action representation data to the simulation execution terminal through a communication interface, so that the simulation execution terminal constructs the target scene and drives the target embodied intelligent agent to perform the target task.

[0016] In some embodiments, acquiring process data corresponding to the task execution process and / or result data corresponding to the task execution result includes: receiving the process data and / or result data from the simulation execution terminal; wherein, the process by which the simulation execution terminal acquires the process data and / or result data includes: parsing the target trajectory information and gripper position information in the target motion representation data; performing inverse kinematics solution on the target pose corresponding to the target trajectory information to obtain joint control information; and, based on the joint control information and the gripper position information, driving the target embodied intelligent agent to execute the target task frame by frame, and recording the process data and / or result data during the execution process.

[0017] In some embodiments, driving the target embodied agent to perform the target task frame by frame includes: in the grasping phase, establishing a connection between the target object and the gripper through an attachment process; and in the placement phase, detaching the connection between the target object and the gripper through a separation process.

[0018] In some embodiments, the process data includes at least one of image information, depth information, joint state information, end-effector pose information, and contact state information; the result data includes at least one of task completion markers, quality scores, and result status information.

[0019] In some embodiments, generating embodied intelligence data based on at least one of the target action representation data, the process data, and the result data includes: repeatedly performing target scene representation data determination, target action representation data determination, task execution, and data acquisition processing until the generated embodied intelligence data reaches the target data volume.

[0020] In some embodiments, the repeated execution of target scene representation data determination, target action representation data determination, task execution, and data acquisition processing includes: dividing the data generation task corresponding to the target data volume into multiple sub-tasks; allocating the multiple sub-tasks to multiple simulation execution terminals for parallel execution; receiving embodied intelligence data returned by multiple simulation execution terminals, and summarizing the returned embodied intelligence data to obtain a target dataset corresponding to the target data volume.

[0021] Secondly, embodiments of this application provide an embodied intelligent data generation system, comprising: a data receiving module for receiving source reference data, the source reference data being used to characterize reference execution information of a source execution subject for a target task; a scene determination module for determining target scene characterization data, the target scene characterization data being used to characterize a target scene corresponding to the target task, the target scene characterization data including at least one of object pose information, environmental parameter information, scene observation information, and scene semantic information; and an action determination module for determining a target action characterization corresponding to the target scene based on the source reference data and the target scene characterization data. The data includes target action representation data, which is used to represent the action process of a target embodied intelligent agent performing the target task in the target scene, and the target action representation data includes at least one of target action information, target trajectory information, target control information, and policy output information; a task execution module, which is used to enable the target embodied intelligent agent to perform the target task in the target scene in the simulation environment based on the target action representation data, and to obtain process data corresponding to the task execution process and / or result data corresponding to the task execution result; and a data generation module, which is used to generate embodied intelligent data based on at least one of the target action representation data, the process data, and the result data.

[0022] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described embodied intelligent data generation methods.

[0023] Fourthly, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps of any of the above-described embodied intelligent data generation methods.

[0024] This application provides a method and system for generating embodied intelligence data, a storage medium, and a program product. By receiving source reference data and using it to characterize the reference execution information of the source execution entity for the target task, it can extract the task action patterns, trajectory patterns, and control patterns contained in a small number of reference execution results. This provides a data foundation for generating embodied intelligence data in different scenarios, thereby improving the reliance on large-scale manual collection in the acquisition process of related embodied intelligence data. By determining the target scene representation data and using at least one of object pose information, environmental parameter information, scene observation information, and scene semantic information to characterize the target scene corresponding to the target task, it is possible to establish a correspondence between the data generation process and new scene conditions, thereby improving the adaptability of the generated data to changes in different scenes. By determining the target action representation data based on the source reference data and the target scene representation data, it is possible to transfer the task execution relationship in the reference execution information to the target scene, enabling the target embodied intelligence agent to form an adapted action process in the target scene, thereby improving the rationality of the generated data at the task semantic level and the action execution level. By enabling a target embodied intelligent agent to perform a target task in a target scene within a simulated environment based on target action representation data, and acquiring process data corresponding to the task execution process and result data corresponding to the task execution result, it is possible to obtain process feedback and result feedback corresponding to the action execution while generating action representations, thereby improving the completeness and usability of embodied intelligent data. By generating embodied intelligent data based on at least one of the target action representation data, process data, and result data, training data samples corresponding to action representations, execution processes, and execution results can be formed, thereby improving the data support capability for subsequent embodied intelligent model training, testing, and optimization. This embodiment addresses the problems of high cost of obtaining high-quality reference execution data, insufficient data generalization ability under scene changes, and weak correlation between generated data and task execution process in related embodied intelligent training processes. It adopts the concept of collaboratively generating target action representation data based on source reference data and target scene representation data, and combining it with the execution results of the simulation environment to generate embodied intelligent data, thereby improving the efficiency of embodied intelligent data generation and enhancing the scene adaptability, task correlation ability, and training application value of embodied intelligent data. Attached Figure Description

[0025] The embodiments of this application are further described below with reference to the accompanying drawings and specific implementation details.

[0026] Figure 1 This is a flowchart illustrating an embodied intelligence data generation method provided in an embodiment of this application.

[0027] Figure 2 This is a flowchart of an embodiment of intelligent data generation provided in this application.

[0028] Figure 3This is a task orchestration flowchart provided in an embodiment of this application.

[0029] Figure 4 This is a scenario generalization flowchart provided in an embodiment of this application.

[0030] Figure 5 This is a simulation execution flowchart provided in an embodiment of this application.

[0031] Figure 6 This is a communication flowchart between a client and a server provided in an embodiment of this application.

[0032] Figure 7 This is a structural block diagram of an embodied intelligent data generation system provided in an embodiment of this application.

[0033] Figure 8 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the embodiments of this application.

[0035] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0036] In recent years, robot manipulation technology based on imitation learning has made significant progress. However, the performance of related technical solutions is often highly dependent on large-scale, high-quality demonstration datasets. Currently, there are two main ways to obtain such demonstration data: one is to perform task demonstrations and collect corresponding data in a real physical environment through manual teleoperation; the other is to execute tasks and collect data using a simulation environment. The former method can obtain data that is relatively close to real execution conditions, but it often requires expensive hardware, experienced technicians, and a long data acquisition cycle. While the latter method can reduce the physical costs and safety risks in the real data acquisition process to some extent, there is still room for improvement in the scene coverage, motion transfer capabilities, and execution fidelity of the generated data.

[0037] Furthermore, the relevant data collection and augmentation methods also have the following problems: First, the data collection cost is high, especially when collecting high-quality trajectories through manual remote operation in real physical environments. This often requires significant investment of manpower, equipment, and time; collecting tens of thousands of high-quality trajectories may take months or even longer, thus limiting the efficiency of algorithm iteration and model updates. Second, the scene generalization ability is weak. The data collected by these methods is often strongly correlated with specific scene conditions, such as being tightly bound to fixed object poses, background environments, and lighting conditions. Therefore, the robot strategies trained using these methods often lack adaptability when facing constantly changing object poses, material styles, and lighting conditions in real-world environments. Third, The decoupling ability between the trajectory and the object is weak. The relevant data amplification methods usually only make simple perturbations to the global trajectory, such as superimposing random noise on the global trajectory. It is difficult to establish a stable correspondence between the operation trajectory and the operation object. When the object pose changes, the original global trajectory is difficult to adaptively migrate to the new object position, which can easily cause the generated trajectory to be unreasonable at the geometric and semantic level, affecting the task execution effect. Fourth, the data fidelity is insufficient. Especially when using simple perturbation methods to amplify the trajectory, it is easy to destroy the dynamic feasibility of the original trajectory, which can generate action data that violates the robot's kinematic constraints, has an uneven trajectory change, or is difficult to execute directly. As a result, the relevant generated data is difficult to use directly for subsequent policy training.

[0038] See Figure 1 , Figure 1 This is a flowchart illustrating an embodied intelligence data generation method provided in an embodiment of this application.

[0039] To address the issues of high cost of acquiring high-quality reference execution data, insufficient data generalization ability under changing scenarios, and weak correlation between generated data and task execution process during embodied intelligence training, this application provides an embodied intelligence data generation method, which includes steps S101 to S105.

[0040] Step S101: Receive source reference data, which is used to characterize the reference execution information of the source execution entity for the target task.

[0041] Step S102: Determine target scene representation data, wherein the target scene representation data is used to represent the target scene corresponding to the target task, and the target scene representation data includes at least one of object pose information, environmental parameter information, scene observation information, and scene semantic information.

[0042] Step S103: Based on the source reference data and the target scene representation data, determine the target action representation data corresponding to the target scene. The target action representation data is used to represent the action process of the target embodied intelligent agent performing the target task in the target scene, and the target action representation data includes at least one of target action information, target trajectory information, target control information and policy output information.

[0043] Step S104: Based on the target action representation data, enable the target embodied intelligent agent to execute the target task in the target scene of the simulation environment, and obtain the process data corresponding to the task execution process and / or the result data corresponding to the task execution result.

[0044] Step S105: Generate embodied intelligence data based on at least one of the target action representation data, the process data, and the result data.

[0045] In some embodiments, the embodied intelligence data generation method can be executed by an embodied intelligence data generation system. The execution of the embodied intelligence data generation system can be implemented in software, hardware, or a combination of both. The embodied intelligence data generation system may include a client and a server. For example, the client can be used to execute steps S101-S105, and the server (as an example of a simulation execution terminal) can be used to receive target action representation data from the client, control the target embodied intelligence agent to execute the target task in the target scene within the simulation environment, and send process data and / or result data to the client.

[0046] Embodied intelligence data generation methods can be understood as a process centered around an embodied intelligent agent performing tasks. This involves organizing, transforming, simulating, and recording reference execution information, scene information, action information, execution process information, and execution result information to form embodied intelligence data that can be used for subsequent training, evaluation, verification, or optimization. The focus of embodied intelligence data generation methods is not simply replicating the original demonstration, but rather combining it with new scene conditions to transfer reference execution information into new action representations, and then forming new task data through simulation execution to create embodied intelligence data. Embodied intelligence data generation methods can be used for tasks such as grasping, moving, placing, and tool use, as well as various embodied task scenarios such as opening doors, pressing, plugging, and assembling.

[0047] Source reference data refers to reference execution information used to characterize the execution process of a source executor around a target task. The role of source reference data is to provide the foundational basis for task execution, enabling subsequent data generation processes to extract action relationships, trajectory relationships, control relationships, and state change patterns from existing execution experience. In some embodiments, the source reference data may include at least one of source demonstration information, source action information, source trajectory information, source control information, source state information, and source observation information. For this embodiment, the value of source reference data lies in carrying the reference execution patterns of "how to complete the target task."

[0048] The source execution entity can refer to the execution source that generates the source reference data. The introduction of the source execution entity is primarily used to distinguish the front-end reference execution process from the back-end data generation process, thereby making the source of the source reference data more flexible. In some embodiments, the source execution entity may include at least one of a physical embodied intelligent agent, a virtual embodied intelligent agent, and a human-machine teleoperation system. The source execution entity can be a physical embodied intelligent agent, such as a real robotic arm, mobile robot, dual-arm robot, or wheeled robot; it can also be a virtual embodied intelligent agent, such as a robot model in a simulation platform; or it can be a human-machine teleoperation system, such as a teleoperation execution link composed of a human operating end, a master hand device, a motion capture device, and a slave robot.

[0049] In some embodiments, source reference data can be acquired by a human-machine teleoperation system controlling a virtual avatar to perform a target task. For example, an operator can control a virtual avatar in a simulation environment through a human-machine interface, a master hand device, a motion capture device, or other teleoperation input devices, enabling the virtual avatar to perform tasks such as approaching, grasping, moving, and placing a target object. During this process, at least one of the following can be recorded simultaneously: end effector pose information, gripper motion information, joint control information, target object pose change information, and task process image information, serving as source reference data. This approach allows for the acquisition of reference execution information corresponding to the target task without relying on repeated real-world data collection with a real avatar, thereby reducing hardware costs and acquisition cycle pressure during source reference data acquisition.

[0050] It should be noted that the controlled object of the human-machine teleoperation system is not limited to a virtual embodied intelligent agent. In some embodiments, the human-machine teleoperation system can control a real embodied intelligent agent to perform a target task to collect source reference data; in other embodiments, the human-machine teleoperation system can also control a virtual embodied intelligent agent to perform a target task to collect source reference data. In other words, the human-machine teleoperation system can form a teleoperation execution link with either a real embodied intelligent agent or a virtual embodied intelligent agent, thereby giving greater flexibility to the object of source reference data collection.

[0051] A target task can refer to the task content that needs to be executed by a target embodied intelligent agent in a target scenario, and that needs to generate embodied intelligent data around the execution process. A target task can be a single-step task or a multi-stage task; it can be any partial task among approaching, grasping, lifting, and placing, or it can be a complete composite task consisting of approaching, grasping, moving, and placing. Target tasks can also be extended to tasks such as opening and closing doors, pushing and pulling drawers, pressing buttons, sorting objects, operating tools, and assembling. In terms of implementation principles, the target task provides unified semantic constraints for source reference data selection, target scenario construction, target action representation data generation, and simulation execution, enabling various types of data to be organized around the same task objective.

[0052] Target scene representation data refers to a set of information used to represent the scene state corresponding to a target task. The role of target scene representation data is to provide scene conditions for the data generation process, enabling the source reference data to establish a correspondence with the new task environment. Target scene representation data can be represented in parametric form, such as object position parameters, object pose parameters, lighting parameters, background parameters, and material parameters; in observational form, such as images, depth information, point clouds, and segmentation results; or in semantic form, such as object category, object functional attributes, contact area labels, and operable area labels. Target scene representation data can be generated randomly, acquired through a perception module, or imported through pre-configuration. By introducing target scene representation data, new execution samples can be generated for the same task under different object positions, backgrounds, lighting conditions, or material conditions.

[0053] Object pose information refers to the position and orientation information of a target object or related objects in a target scene. Object pose information can be used to describe the spatial state of an object in three-dimensional space, such as the object's translational position, rotational posture, orientation angle, gripping surface direction, and contact surface direction. Object pose information can be represented in the form of coordinates, matrices, quaternions, Euler angles, or a pose representation method suitable for the specific implementation platform. In this embodiment, the important role of object pose information is to establish the spatial relationship between the target object and the end effector, enabling the relative relationship of "target object-end effector" in the source scene to be transferred to the target scene, thereby supporting trajectory redirection, gripping point adjustment, and task action mapping.

[0054] Environmental parameter information refers to information used to characterize the environmental state or configuration of a target scene. Environmental parameter information can include various aspects such as background environment information, lighting conditions, material style information, desktop parameters, ground parameters, interference object parameters, and physical properties. For example, environmental parameter information may include background color, background texture, light source direction, light source brightness, target object surface material, friction coefficient of the supporting plane, and obstacle distribution. The role of environmental parameter information is to enable the target scene to reflect not only changes in the object's own position but also changes in visual and physical conditions, thereby enhancing the value of the generated data in visual generalization and environmental adaptation.

[0055] Scene observation information refers to information obtained through perception methods and used to characterize the state of a target scene. Scene observation information can include various types such as color images, depth images, point cloud data, semantic segmentation images, instance segmentation images, normal information, and voxel information. The characteristic of scene observation information is that it is not directly equivalent to parameter configuration, but rather is the observation result formed by the camera, depth sensor, simulation rendering module, or other perception channels. In this embodiment, scene observation information plays two roles: firstly, it participates in the characterization of the target scene; secondly, it can serve as the basic input for subsequent strategy generation, action determination, or process data recording, thereby improving the consistency between generated data and real-world perception input.

[0056] Scene semantic information refers to information used to describe the categories, functions, relationships, and task semantic constraints of objects in a target scene. Scene semantic information can include target object category information, object identification information, semantic labels for operable areas, semantic labels for contact areas, supporting relationship information, adjacency relationship information, target placement area information, and task stage semantic information. The role of scene semantic information is to ensure that the data generation process not only relies on geometric position changes but also incorporates high-level semantics such as "what the object is," "what the object can do," and "what the relationships are between objects" for action mapping and task transfer, thereby improving the rationality of generated actions at the task semantic level.

[0057] Target action representation data refers to a set of information used to represent the action process of a target embodied intelligent agent when performing a target task in a target scene. Target action representation data is a key intermediate carrier connecting "reference execution information" and "simulation execution process." Its essence lies in combining the reference execution rules in the source reference data with the scene conditions in the target scene representation data to form an action representation adapted to the current target scene. Target action representation data can be represented as discrete action sequences, continuous trajectory sequences, control command sequences, policy output results, relative action representations in a local coordinate system, or optimized executable paths. Through target action representation data, the target embodied intelligent agent can perform task actions in a new scene that are semantically consistent with the source reference data, but whose spatial position and execution details have been adapted to the new scene conditions.

[0058] A target embodied intelligent agent can be an execution object that actually performs a target task in a target scenario and generates corresponding process data and result data. A target embodied intelligent agent can be a robotic arm, a dual-arm robot, a wheeled robot, a legged robot, a wheel-legged robot, a mobile manipulation robot, a humanoid robot, or a combination of a gripping device and a mobile platform, etc. The target embodied intelligent agent can be the same as or different from the source execution entity; for example, a physical embodied intelligent agent can generate source reference data, and then a virtual embodied intelligent agent in the simulation environment can execute the target action representation data.

[0059] Target action information refers to information describing the specific actions of a target embodied intelligent agent when performing a target task in a target scenario. Target action information can be discrete action categories, such as approaching, grasping, lifting, placing, and releasing; or it can be continuous action parameters, such as the direction of movement, speed, angular velocity, gripper opening / closing value, and grasping force of the end effector. Target action information emphasizes "what action is performed" and "how the action changes," and can be used to describe action semantics and action parameters. Target action information can be directly mapped from source reference data or indirectly determined by a policy model, controller, or trajectory generation module.

[0060] Target trajectory information refers to information describing the spatial motion path of a target embodied intelligent agent performing a target task within a target scene. Target trajectory information can include end-effector pose sequences, pathpoint sequences, joint space trajectories, interpolated dense pathpoint sequences, and transition trajectories between the starting point and the target point. The process of acquiring target trajectory information may include, for example, determining the relative relationship between the target object and the end-effector in the source scene based on source reference data, then determining new end-effector poses or pathpoints by combining the object poses in the target scene, and adding starting points and interpolation points as needed to generate a continuous trajectory. Target trajectory information primarily reflects "along which path the task is performed."

[0061] Target control information refers to the control-layer information used to drive the target embodied intelligent agent to perform target tasks. Target control information may include joint position control information, joint velocity control information, joint torque control information, gripper position control information, end effector control information, or controller output command information. Target control information can be obtained by processing target motion information and target trajectory information through the controller, or it can be further transformed from policy output results.

[0062] Policy output information refers to information output by task policies, action planning logic, or decision models to guide the target embodied agent in performing target tasks. Policy output information can manifest as action probability distributions, action selection results, target pose prediction results, local path suggestions, control mode switching results, or task phase decision results. The relationship between policy output information and target action information, target trajectory information, and target control information can be parallel or upstream / downstream. For example, policy output information can first provide grasping and placement policies, and then further convert them into target trajectory information and target control information. By writing policy output information, various implementation methods such as end-to-end policy generation, model inference generation, and rule-based decision generation can be protected.

[0063] A simulation environment refers to a virtual execution environment used to host the target scene, the target embodied intelligent agent, and the task execution process. A simulation environment can include robot models, object models, background environment models, lighting models, physics engines, collision detection mechanisms, attachment and separation mechanisms, sensor simulation modules, and data recording modules. The role of the simulation environment is to reproduce the task execution process at a lower cost without relying on large-scale real-world data acquisition, and to generate multimodal data corresponding to the task process. The simulation environment can not only perform trajectory execution but also record various types of data such as images, depth, joint states, contact states, and task completion markers, thus providing a stable execution platform for the generation of embodied intelligent data.

[0064] Process data refers to data generated by a target embodied intelligent agent during the execution of a target task in a simulated environment. Process data emphasizes "process" and "temporal" characteristics, corresponding to continuously changing information during execution. In some embodiments, process data may include at least one of the following: image information, depth information, joint state information, end-effector pose information, contact state information, force feedback information, gripper state information, and object state change information. The role of process data is to reflect what happens during task execution, enabling subsequent training or evaluation to improve the model's understanding of the dynamic process of the task by utilizing complete execution process samples, rather than relying solely on a single result.

[0065] Result data can refer to information generated after a target embodied intelligent agent completes a target task, used to characterize the task execution result. In some embodiments, result data may include at least one of the following: task completion markers (e.g., task success markers, task failure markers), quality scores, result status information, final pose information of the target object, placement deviation results, grasping stability results, collision results, and trajectory smoothness evaluation results. Result data is characterized by a greater emphasis on the final state, completion status, and quality evaluation. The role of result data includes adding result labels and quality labels to embodied intelligent data samples, ensuring that the generated data not only includes the action process but also a result representation of the effectiveness of the action process.

[0066] Embodied intelligence data refers to data samples formed by integrating at least one of target action representation data, process data, and result data, which can be used for embodied intelligence training, validation, evaluation, or optimization. Embodied intelligence data can be a single sample or a batch dataset; it can be structured data containing only trajectory and state, or multimodal samples containing images, depth, trajectory, control, and result labels simultaneously. Embodied intelligence data can form a "action-process-result" relationship around the task execution process, enabling subsequent models to learn task patterns, action patterns, and scene adaptation patterns based on more complete data. In this embodiment, embodied intelligence data is not a simple raw record, but a data product with training value formed after reference execution information migration, target scene adaptation, simulation execution, and data aggregation.

[0067] In this embodiment, source reference data is first received. This source reference data characterizes the reference execution information formed by the source execution entity around the target task. For example, it includes action information, trajectory information, control information, state information, or observation information corresponding to the source execution entity's actions such as approaching, grasping, transporting, and placing tasks in the source scene. Based on this, target scene representation data is further determined. This target scene representation data describes at least one of the following in the target scene corresponding to the target task: object pose information, environmental parameter information, scene observation information, and scene semantic information. This allows the data generation process to be integrated with the new task environment. Subsequently, based on the source reference data and the target scene representation data, target action representation data adapted to the target scene is generated. This allows the target action representation data to characterize the action process of the target embodied intelligent agent performing the target task in the target scene. The target action representation data may include at least one of the following: target action information, target trajectory information, target control information, and policy output information. This transfers the task execution rules contained in the source reference data to the target scene. Subsequently, based on the target action representation data, the target embodied agent executes the target task in the target scenario within the simulation environment, simultaneously acquiring process data corresponding to the task execution process and result data corresponding to the task execution result. Finally, based on at least one of the target action representation data, process data, and result data, embodied intelligence data that can be used for embodied intelligence training, evaluation, or optimization is generated, thus forming a complete data generation closed loop from reference execution information input to target scenario adaptation, and then to task execution and data accumulation output.

[0068] This embodiment receives source reference data and uses it to characterize the reference execution information of the source execution entity for the target task. This allows for the extraction of task action patterns, trajectory patterns, and control patterns from a small amount of reference execution results, providing a data foundation for generating embodied intelligence data in different scenarios. This reduces the reliance on large-scale manual data collection in the acquisition of related embodied intelligence data. By determining target scene representation data and using at least one of object pose information, environmental parameter information, scene observation information, and scene semantic information to characterize the target scene corresponding to the target task, the data generation process can establish a correspondence with new scene conditions, thereby improving the adaptability of the generated data to changes in different scenes. By determining target action representation data based on source reference data and target scene representation data, the task execution relationships in the reference execution information can be transferred to the target scene, enabling the target embodied intelligence agent to form a suitable action process in the target scene, thus improving the rationality of the generated data at both the task semantic and action execution levels. By enabling a target embodied intelligent agent to perform a target task in a target scene within a simulated environment based on target action representation data, and acquiring process data corresponding to the task execution process and result data corresponding to the task execution result, it is possible to obtain process feedback and result feedback corresponding to action execution while generating action representations, thereby improving the completeness and usability of embodied intelligent data. By generating embodied intelligent data based on at least one of the target action representation data, process data, and result data, training data samples that simultaneously cover action representation, execution process, and execution result can be formed, thereby enhancing the data support capability for subsequent embodied intelligent model training, testing, and optimization. This embodiment addresses the problems of high cost of acquiring high-quality reference execution data, insufficient data generalization ability under scene changes, and weak correlation between generated data and task execution process in related embodied intelligent training processes. It adopts the concept of collaboratively generating target action representation data based on source reference data and target scene representation data, and combining it with simulation environment execution results to generate embodied intelligent data, thereby improving the efficiency of embodied intelligent data generation and enhancing the scene adaptability, task correlation ability, and training application value of embodied intelligent data.

[0069] In some embodiments, determining the target scene representation data may include: randomizing at least one of the object pose, background environment, lighting conditions, and material style in the target scene to obtain the target scene representation data.

[0070] In some embodiments, randomizing the pose of objects in the target scene may include: resetting the target embodied agent to a reference pose and opening the gripper of the target embodied agent; traversing at least some objects in the target scene to obtain the current pose of the corresponding objects, and randomly offsetting at least some objects within a preset offset range to set a new pose of the corresponding objects based on the random offset result.

[0071] Target scene representation data can be parametric data, observational data, or semantic data. Parametric data includes object position coordinates, object pose angles, light source direction, and material parameters; observational data includes color images, depth images, and point cloud data; and semantic data includes object category labels, grabbable region labels, and task-related object identifiers.

[0072] Randomization refers to a processing method that modifies at least one scene element in a target scene according to preset rules, preset ranges, or preset probabilities. The focus of randomization is not on completely shuffling the scene without constraints, but rather on controlling the perturbation of object pose, background environment, lighting conditions, material styles, etc., within the allowable range of variation for the target task, thereby generating multiple different but still reasonable target scenes. Randomization can be implemented using various methods such as uniform sampling, normal distribution sampling, discrete candidate set sampling, and random combination under rule constraints. Through randomization, the ability of generated data to cover different scene variations can be improved, thereby reducing the dependence of embodied intelligence models on a single scene distribution.

[0073] Object pose refers to the position and orientation of an object in a target scene within space. Object pose can be composed of both translation and rotation information. Translation information includes, for example, the coordinates of the object's center point, the coordinates of the grab reference point, and the location of the contact area; rotation information includes, for example, Euler angles, rotation matrices, quaternions, and orientation angles. Object pose not only determines "where" the object is in space, but also "what it faces" and "at what angle it is placed."

[0074] Background environment refers to the environmental content within the target scene that constitutes the visual background or atmosphere of the scene. It can include various aspects such as wall color, table texture, floor texture, background patterns, background scenery, scene decorations, and environmental models. While the background environment may not directly participate in the contact actions between the target embodied agent and the target object, it significantly influences scene observation results, especially color images, depth images, and visual perception model inputs. Randomizing the background environment can improve the visual diversity of the generated data and reduce the problem of embodied intelligence models relying solely on fixed background features for learning tasks.

[0075] Lighting conditions refer to environmental factors in a target scene related to light source configuration and lighting effects. These conditions can include the number of light sources, their positions, directions, intensity, color temperature, shadow distribution, and local reflection effects. Changes in lighting conditions alter the light and shadow relationships on an object's surface, edge contours, shadow areas, and material reflection, thus significantly impacting visual observation results. In principle, lighting condition randomization can be achieved by adjusting light source parameters in the scene, changing the layout of main and auxiliary light sources, and switching between different lighting templates. By manipulating lighting conditions, the adaptability of generated data to different visual lighting environments can be improved.

[0076] Material style refers to the visual and physical representation of an object or environment surface in a target scene. Material style can include multiple dimensions such as color, roughness, gloss, transparency, metallic feel, texture mapping, and reflective properties. For example, material style can correspond to different representations such as a wooden tabletop, metal surface, frosted plastic surface, ceramic surface, and transparent glass surface. Changes in material style affect the texture features, boundary features, and reflective features in visual observation, and may also affect the contact performance related to physical simulation. By randomizing material styles, the diversity of generated data at the object appearance level can be increased, thereby mitigating the risk of overfitting embodied intelligence models to a single appearance condition.

[0077] The target embodied agent is not only the executor of task actions, but also a participant in the object pose randomization, scene reset, and task simulation processes. In this embodiment, the target embodied agent is reset to the reference pose and its grippers are opened before object pose randomization. This is to prepare a unified starting state for subsequent object position reset and task execution, thereby improving the consistency of starting conditions between different rounds of generation.

[0078] A reference pose refers to the baseline pose set to by the target embodied agent before executing a new round of scene randomization or task execution. The reference pose can be an initial standby pose, a safety reset pose, a task preparation pose, or a pre-configured standard execution starting point pose. The purpose of the reference pose is to provide the target embodied agent with a repeatable and alignable starting state, ensuring that object pose randomization and subsequent target task execution are based on a relatively unified robot state. By resetting the target embodied agent to the reference pose, the impact of residual states from the previous round of task execution on the next round of scene construction and task execution can be reduced.

[0079] Opening the gripper refers to adjusting the end effector of the target embodied intelligent agent to an open state, preventing the gripper from clamping the target object. Opening the gripper can be used in scene initialization, object reset, task preparation, or pre-grabbing preparation stages. Opening the gripper allows the target object to move freely without being restricted by the gripper and also reduces the probability of unexpected contact between the gripper and the object. In terms of implementation, opening the gripper can be achieved by setting the gripper opening degree, gripper position control values, and gripper actuator target values. Opening the gripper is a preparatory action in the target embodied intelligent agent reset process.

[0080] Traversing at least some objects refers to processing multiple objects in the target scene sequentially according to certain rules, rather than processing only a single object. Here, "at least some objects" can refer to all objects in the target scene, or any one or more of the following categories: objects directly related to the target task, movable objects, graspable objects, distracting objects, and background objects. The purpose of traversing at least some objects is that object pose randomization is not limited to a single target object, but can read the current pose of multiple objects one by one, perform offset processing on each one, and set new poses for each one, according to task requirements. By traversing at least some objects, the spatial layout of the target scene can be made to form richer variations.

[0081] The current pose refers to the object's original pose state in the target scene before randomization. The current pose can be seen as the reference pose for subsequent random offsets. By first obtaining the current pose and then applying an offset based on it, the object's pose adjustment can be based on a known spatial position, rather than directly generating a new position completely detached from the context. In this way, randomization can introduce variation while preserving the basic rationality of the original spatial layout of the target scene. The current pose also provides a basic reference for offset range determination, collision checking, and task executability assessment.

[0082] A preset offset range refers to a pre-defined spatial or parameter range used to constrain the amplitude of object pose changes. The preset offset range can be one-dimensional, two-dimensional, or three-dimensional, such as lateral and longitudinal offset ranges in a plane, or it can include height and rotation offset ranges. The logic for setting the preset offset range can be related to the execution space of the target task, object size, scene boundaries, robot reachability range, and collision restrictions. By using a preset offset range, randomization processing can be kept within a controllable range, thereby preventing objects from being moved to obviously unreasonable or unexecutable positions while introducing scene differences.

[0083] Random offset refers to a processing method that generates offsets for an object within a preset offset range and adjusts the object's pose based on these offsets. The advantage of random offset is that the offset is not a fixed value but is determined based on random sampling results, thus enabling the object to form different positional distributions across multiple data generation rounds. Random offset can adjust only the translation amount or simultaneously adjust the rotation amount; it can be performed on a single object or on multiple objects separately. The implementation process of random offset involves first reading the object's current pose, then randomly generating offset parameters based on a preset offset range, and finally superimposing the offset parameters onto the current pose. Random offset enhances the diversity of object distribution in the target scene.

[0084] A new pose refers to the updated pose state of an object after random offset processing. While the new pose has changed relative to the current pose, it should still remain within the preset offset range, the task's executable range, and the reasonable range of the scene. Setting a new pose can be understood as formally writing the random offset result into the target scene, enabling the object to participate in subsequent target action representation data generation and target embodied intelligent agent task execution with a new position and posture.

[0085] Setting a new pose for an object refers to actually applying the pose adjustment result obtained from random offset processing to the corresponding object in the target scene. Setting a new pose for an object is not only a data update action but also a scene construction action, because the spatial layout in the target scene has changed after the setting is completed. Setting a new pose for an object can be achieved through object pose interfaces, object state update interfaces, and physics engine state writing interfaces in the simulation scene. By setting a new pose for an object, randomization results can be transformed into a scene state that actually participates in subsequent task execution, thus providing a practical and effective new scene foundation for embodied intelligence data generation.

[0086] This embodiment randomizes at least one of the object pose, background environment, lighting conditions, and material style in the target scene, enabling the construction of multiple differentiated target scenes corresponding to the same target task. This enhances the coverage of target scene representation data for various environmental changes. By resetting the target embodied agent to a reference pose and opening its gripper, the target embodied agent returns to a relatively uniform task start state after each scene randomization, improving the consistency of initial conditions between different rounds of task execution. By traversing at least some objects in the target scene and obtaining the current pose of the corresponding objects, an object-level positional reference can be provided for object pose randomization, thereby improving the targeting of the target scene adjustment process. By randomly offsetting at least some objects within a preset offset range and setting new poses for the corresponding objects based on the random offset results, the target object and related objects can change their spatial positions, thereby improving the adaptability of subsequently generated data to changes in object position. By combining object pose randomization with random variations in background environment, lighting conditions, and material styles, the target scene can be expanded from both spatial layout and visual representation perspectives, thereby improving the generalization value of embodied intelligence data during training and evaluation. This embodiment addresses the problems of limited target scene diversity, insufficient variation in object spatial distribution, and inadequate coverage of visual environment changes during the generation of relevant embodied intelligence data. It employs a method of randomizing object pose, background environment, lighting conditions, and material styles within the target scene to construct target scene representation data, thereby improving the scene diversity of embodied intelligence data and enhancing its adaptability to different task environments.

[0087] In some embodiments, determining the target motion representation data corresponding to the target scene based on the source reference data and the target scene representation data may include: determining the end-effector pose in the target scene based on the relative pose relationship between the target object and the end-effector in the source scene and the pose of the target object in the target scene, so that the source scene matches the relative pose relationship between the target object and the end-effector in the target scene; wherein the relative pose relationship between the target object and the end-effector in the source scene is determined based on the source reference data, the target object pose in the target scene is determined based on the target scene representation data, and the target motion representation data is determined based on the end-effector pose in the target scene.

[0088] In some embodiments, determining the target action representation data corresponding to the target scene based on the source reference data and the target scene representation data may further include: adding a starting point corresponding to the current position of the target embodied agent; and performing trajectory interpolation processing based on the starting point and the end effector pose to generate a path point sequence.

[0089] The source scene refers to the task execution scenario corresponding to the generation of source reference data. The source scene includes not only the target object, the end effector, and the background environment, but also the spatial layout, object placement state, contact relationships, and task context formed during the source reference data acquisition process. The significance of the source scene lies in providing a reference context for the relative pose relationship between the target object and the end effector within the source scene. In other words, the source reference data does not exist independently of the scene, but is related to the target object's position, operation direction, grasping position, and contact method within the source scene.

[0090] The target scene refers to the environment in which the target embodied intelligent agent subsequently executes the target task and generates embodied intelligent data. The target scene may differ from the source scene in terms of object position, object pose, background environment, lighting conditions, material style, and other environmental factors. The target scene can be constructed through scene loading, object pose adjustment, environmental parameter setting, scene randomization, and other methods.

[0091] A target object can refer to any object that is manipulated, touched, grasped, moved, placed, or processed by the target embodied intelligent agent in a target task. A target object can be a single object or a key object among multiple related objects. Specific forms of target objects can include cups, boxes, tools, buttons, door handles, drawers, assemblies, containers, etc. The target object is not only the object on which task actions are performed, but also a key reference for establishing spatial relationships and trajectory transfer relationships. Whether determining relative pose relationships in the source scene or determining the pose of the end effector in the target scene, the target object exists as the central object for spatial mapping and action transfer.

[0092] An end effector is an actuator installed at the execution end of a target intelligent agent, directly contacting or interacting with the target object. End effectors can take various forms, such as grippers, suction cups, robotic arms, tool ends, push ends, and plug-in ends. During task execution, end effectors perform functions such as approach, alignment, contact, grasping, transporting, and releasing.

[0093] Relative pose relationships refer to the positional and orientational relationships between two objects. In this embodiment, the relative pose relationship focuses on the relative pose relationship between the target object and the end effector in the source scene, and the relative pose relationship between the target object and the end effector in the target scene. Relative pose relationships can be expressed in various ways, such as translation offsets, rotation relationships, pose transformation matrices, and relative coordinate representations. The purpose of relative pose relationships is not to simply copy the absolute trajectory in the source scene, but to maintain or match the operational relationship between the target object and the end effector. For example, in a grasping task, the relative pose relationship can be reflected in the approach direction, approach height, grasping angle, and contact position of the end effector relative to the target object. By maintaining relative pose relationships, the adaptability of the target motion representation data to changes in the target scene can be improved.

[0094] The relative pose relationship between the target object and the end effector in the source scene can refer to the spatial correspondence established between the target object and the end effector during the task execution process corresponding to the source reference data. The relative pose relationship can be calculated from the target object pose information and the end effector pose information in the source reference data, or it can be obtained directly from the relative pose records in the source reference data. The relative pose relationship can reflect key operational semantics in task execution experience, such as the direction from which to approach the target object, the height at which to grasp the target object, and the posture at which to contact the target object. Therefore, the absolute execution path in the source reference data is transformed into an operational relationship relative to the target object, so that the subsequent migration process does not rely on fixed absolute coordinates in the source scene, but on a more stable object-execution end relative relationship.

[0095] The pose of a target object can refer to its position and orientation in a target scene. The pose of a target object can be determined using target scene representation data, such as object coordinates, pose angles, rotation matrices, quaternions, scene annotation information, or perception and recognition results.

[0096] The end effector pose refers to the position and orientation of the end effector in space. The end effector pose can be represented as the spatial state at a single moment or as a pose sequence composed of multiple moments. In this embodiment, the end effector pose in the target scene is determined based on the relative pose relationship between the target object and the end effector in the source scene, as well as the pose of the target object in the target scene. The end effector pose determined in this way is not arbitrarily generated, but rather forms an operational relationship with the target object that matches the source reference data. The end effector pose can be directly used as part of the target action representation data, or it can be further used to generate trajectory interpolation results and pathpoint sequences.

[0097] The relative pose relationship between the target object and the end effector in the source scene matches the relative pose relationship between the target object and the end effector in the target scene. This can be used to describe the consistency or correspondence of the relative pose relationships between the target object and the end effector in the source scene and the target scene in terms of task semantics and spatial relationships. Matching does not necessarily require that the two be exactly the same numerically, but rather emphasizes that the action relationship in the target scene can inherit the effective operational intent in the source scene. For example, if the end effector in the source scene approaches the target object from above and grasps the target object at a certain angle, the end effector in the target scene can also establish a corresponding upward approach relationship and a corresponding grasping angle relationship around the target object. The specific implementation of "matching" can include various forms such as complete preservation, approximate preservation, preservation after proportional transformation, and preservation after constraint optimization.

[0098] The current position can refer to the spatial position or end effector spatial state of the target embodied agent at the current moment when generating new target action representation data. The current position can be the target embodied agent's current position after completing a scene reset, or it can be the current position after executing the previous sub-task. The significance of introducing the current position is to ensure that the subsequently generated path point sequence can naturally connect from the target embodied agent's current actual location, rather than starting directly from an intermediate point in the source reference data, thereby improving issues such as trajectory jumps, discontinuities, or disconnections in execution starting points. The current position can be represented either by the end effector pose or by the execution end position obtained from joint state mapping.

[0099] The starting point can correspond to the current position of the target embodied agent, providing a starting point constraint for subsequent trajectory interpolation processing. This ensures that the generated path point sequence smoothly transitions from the target embodied agent's current actual position to the end effector pose determined based on relative pose relationships within the target scene. The starting point can be a single pose point or a composite point marked with attitude, timestamp, gripper state, or control mode. Adding a starting point improves the connection between the newly generated trajectory and the current state of the target embodied agent.

[0100] Trajectory interpolation refers to the process of generating a continuous transition trajectory between existing keypoints, starting points, and target pose points. The main purpose of trajectory interpolation is to generate a smoother, denser, and more suitable set of trajectory points for execution between discrete key poses. Trajectory interpolation can be implemented using various methods, including linear interpolation, polynomial interpolation, spline interpolation, time-parameterized interpolation, and attitude interpolation. Interpolation can be performed not only on position but also synchronously on attitude, gripper opening / closing states, and velocity information. Through trajectory interpolation, problems such as sparse path points, abrupt trajectory changes, or discontinuous execution can be improved, thus providing a more stable path foundation for subsequent simulation execution.

[0101] A path point sequence is an ordered set of multiple sequentially arranged path points used to represent the spatial positions and attitude states that a target embodied intelligent agent should traverse sequentially when performing a target task. A path point sequence can consist of a starting point, key target points, interpolation points, and stage transition points. Each path point in the sequence can include end effector position, end effector attitude, gripper state, time information, stage identifiers, etc. The purpose of the path point sequence is to further organize the target action representation data into a progressively executable action path, enabling subsequent inverse kinematics solving, control output generation, and simulation execution to unfold sequentially along the path point sequence. By generating a path point sequence, the task execution rules in the source reference data can be concretized into executable motion processes in the target scene.

[0102] In this embodiment, the relative pose relationship that the target object and the end effector should maintain in the target scene is first determined, and then an action representation suitable for the current target scene is generated based on this. The target action representation data formed in this way not only inherits the task execution experience in the source reference data, but also establishes a connection with the current pose of the target object and the current state of the target embodied agent. Through this processing method, the spatial mismatch problem that occurs when the source reference data is directly reused in different scenes can be improved, and the executability and task relevance of the generated actions in the target scene can be enhanced.

[0103] In some embodiments, the source reference data may include source action information and / or source trajectory information in the source scene. Determining the target action representation data corresponding to the target scene based on the source reference data and the target scene representation data may include: dividing the reference action process corresponding to the target task into action stages based on the source action information and / or the source trajectory information; converting at least a portion of the source action information and / or source trajectory information into relative action representation data relative to the target object in the source scene; and mapping the relative action representation data to the target action representation data corresponding to the target scene based on the target object pose corresponding to the target scene representation data.

[0104] Source action information can be understood as the action layer information formed around the target task in the source scene. It is used to characterize "what actions the source executor performed, how the actions changed, and at what stage the actions occurred" during task execution. Source action information can include discrete action category information such as approach action information, grasping action information, lifting action information, carrying action information, placing action information, and releasing action information, as well as continuous action parameter information such as end effector movement direction information, gripper opening and closing information, speed adjustment information, and stage switching information. Source action information emphasizes the action semantics and action process of task execution and is an important foundation for subsequent action stage division, action transfer, and action mapping. By introducing source action information, action patterns directly related to the target task can be extracted from the source reference data, thereby improving the problem of insufficient semantic expression when relying solely on a single trajectory path for transfer.

[0105] Source trajectory information refers to the motion path information in the source scene corresponding to the target task, used to characterize the spatial motion state of the source executor at various moments during the execution of the target task. Source trajectory information can include multiple aspects such as end effector pose sequences, pathpoint sequences, joint motion sequences, gripper state changes over time sequences, and key point sequences for each stage. The focus of source trajectory information is to reflect the spatial path along which the task actions unfold and how the various key action points transition continuously during the task. Source trajectory information can be represented in the form of discrete pathpoints, continuous trajectory curves, or pose sequences with time labels. By utilizing source trajectory information, a spatial motion foundation can be provided for the generation of subsequent target action representation data, thereby improving the geometric rationality of the action transfer process.

[0106] A reference action process refers to the complete action flow formed by the source executor in the source scenario as it completes the task around the target task. A reference action process is not merely an isolated trajectory, but emphasizes the sequential, sequential, and constrained relationships between multiple action stages from the start to the end of the task. For grasping and placing tasks, the reference action process may include multiple stages such as approaching the target object, adjusting posture, performing grasping, lifting the target object, moving the target object, placing the target object, and withdrawing the end effector. For tasks such as pressing, plugging, opening and closing doors, and tool operations, the reference action process can also correspond to different stage division methods. The technical significance of the reference action process lies in providing a temporal framework for subsequent action stage division, ensuring that the generation of target action representation data considers not only spatial location changes but also the organizational logic of task actions in the temporal dimension.

[0107] Action phase segmentation refers to the process of breaking down a reference action process into multiple sub-stages with different task semantics and execution purposes based on source action information and source trajectory information. The criteria for action phase segmentation can include changes in end-effector speed, posture, distance to the target object, gripper opening / closing state, contact state, and task semantic labeling. For example, a grasping task can be divided into an approach phase, alignment phase, grasping phase, lifting phase, transport phase, and placement phase; a door opening task can be divided into an approach handle phase, handle gripping phase, handle rotation phase, and door opening / pushing phase. The purpose of action phase segmentation is to decompose the complete task process into several local action segments, enabling subsequent relative action representation, pose mapping, and trajectory reconstruction for different stages, thereby improving the lack of flexibility when directly transferring the overall trajectory.

[0108] Relative motion representation data refers to the motion representation result formed after at least some source motion information and source trajectory information have been transformed into a reference frame relative to the target object in the source scene. The core of relative motion representation data is not to retain absolute coordinates in the source scene, but to retain the relative relationship between the target object and the executed motion, such as the approach direction, approach distance, grasping angle, contact position, lifting direction, and placement offset of the end effector relative to the target object. Relative motion representation data can be represented in the form of relative displacement information, relative posture information, relative trajectory point sequences, relative control parameter sequences, and path information in a local coordinate system. By constructing relative motion representation data, the execution experience in the source scene can be decoupled from fixed scene coordinates, thereby improving the problem that actions cannot be directly reused when the target object's position changes.

[0109] The key role of the target object in the source scene is to provide a relative reference point, enabling source action and trajectory information to be re-expressed around the target object in the source scene. By converting actions and trajectories into expressions relative to the target object in the source scene, action representation can be extracted from a fixed world coordinate system, thereby improving the stability and scalability of actions when migrating to new scenes.

[0110] The target object pose refers to the position and orientation of a target object in a target scene, serving as a spatial reference for mapping relative motion representation data to the target scene. The target object pose can be determined using target scene representation data, specifically including the target object's center position, key contact area position, grasping reference point position, rotational posture, and orientation angle. The target object pose can be generated through scene randomization, obtained through an object state reading interface in a simulation environment, or identified through perception. The purpose of the target object pose is to reposition the relative motion relationships established around the target object in the source scene to the new object position in the target scene, thereby ensuring that the subsequently obtained target motion representation data adapts to the actual spatial state of the target object within the target scene.

[0111] Mapping to target action representation data corresponding to the target scene refers to combining relative action representation data with the pose of the target object in the target scene to convert it into an action expression result suitable for execution in the target scene. Here, "mapping" is not limited to a single mathematical transformation, but rather refers to the process of re-implementing the relative action relationships extracted from the source scene into the target scene. Mapping processing can include various methods such as relative displacement superposition, relative pose transformation, conversion from the local coordinate system to the target scene coordinate system, stage trajectory reconstruction, path point update, pose correction, and repositioning of action points based on the target object's pose. Through mapping processing, the action rules in the source scene can be transformed into executable target action representation data in the target scene, thereby improving the action mismatch problem caused by differences in object position, object pose, or environmental layout between the source and target scenes.

[0112] By simultaneously utilizing source action information and source trajectory information, both dimensions of "what the action is doing" and "how the action is completed in space" can be considered when generating target action representation data. For example, in a grasping task, it is necessary not only to know the action stages of approach, grasping, and placing, but also the pose change path of each stage; in an insertion task, it is necessary not only to know the task semantics of the insertion action, but also the insertion direction, alignment path, and insertion depth changes. By jointly utilizing source action information and source trajectory information, the problem of target action representation data only retaining local features while ignoring the overall task logic can be improved.

[0113] This embodiment divides the reference action process corresponding to the target task into action stages based on source action information and source trajectory information. This decomposes the reference action process into multiple stages with different task semantics and execution purposes, thereby improving the ability of the action generation process to express the internal stage logic of the target task. By converting at least part of the source action information and source trajectory information into relative action representation data relative to the target object in the source scene, the reference action process can be decoupled from its absolute spatial position in the source scene, thus improving the problem of difficulty in reusing actions when the target object's position changes. By mapping the relative action representation data to the target action representation data corresponding to the target scene based on the target object's pose corresponding to the target scene representation data, the action relationships formed in the source reference data can be transferred to the target scene, thereby improving the adaptability of the target action representation data to changes in the target scene. By combining action stage division, relative action representation, and target scene mapping, the task semantic consistency and spatial execution rationality of the reference action process can be maintained simultaneously, thereby improving the usability of subsequent embodied intelligence data in target task training, evaluation, and optimization. This embodiment addresses the problems of insufficient adaptability between reference action processes and target scenes during the generation of relevant embodied intelligence data, difficulty in action migration after changes in the target object's position, and difficulty in taking into account both task stage logic and spatial execution relationships in the generated actions. It adopts the concept of action stage division based on reference action processes, relative action representation construction, and action mapping oriented towards the target object's pose, thereby improving the scene adaptability, task semantic preservation, and execution rationality of the target action representation data.

[0114] In some embodiments, mapping the relative motion representation data to target motion representation data corresponding to the target scene based on the target object pose corresponding to the target scene representation data may include: adaptively adjusting the operation points corresponding to the relative motion representation data based on the target object pose and contact position change information, and generating corresponding target motion representation data.

[0115] Contact position change information refers to potential changes in the contact area within the target scene caused by changes in the target object's pose, shape, surface contactable area, scene interference, or task stage. This change can manifest as an offset of the contact area relative to the source scene, or as changes such as the original contact area becoming unreachable, requiring a new contact area selection, or a change in contact direction. Introducing contact position change information allows the motion mapping process to consider not only the overall pose of the target object but also changes in the specific contact relationship between the executed action and the target object, thereby improving the rationality of the target action representation data at the contact level.

[0116] A gripping point refers to the preferred location chosen by the target embodied intelligent agent to grasp, hold, adhere to, or extract a target object. A gripping point can be a single location on the target object, one of multiple gripping candidate locations, or a representative point within a gripping region. Specific forms of gripping points can include the contact center point of a gripper, the adhesion center point of a suction cup, a two-finger gripping reference point, or the force application point at the tool end. The determination of the gripping point can be related to the geometry of the target object, its center of gravity distribution, surface normal direction, reachability, and the task objective. By adaptively adjusting the gripping point, a more suitable gripping position can be re-determined when the target object's pose changes, thereby improving the success rate and stability of related gripping actions.

[0117] A contact point refers to the location where a target embodied intelligent agent makes contact with a target object or related objects during the execution of a target task. Contact points can correspond not only to gripping contact points in grasping scenarios, but also to positions of action in task scenarios such as pushing, pressing, touching, supporting, leaning, and inserting. A contact point can be an instantaneous contact location or a sequence of key positions along a contact path.

[0118] An operation point refers to the position selected by a target embodied intelligent agent to perform a specific task. Operation points can include grasping points, contact points, and various specific forms such as pressing points, pushing points, insertion points, rotational application points, force application points, and support points. For grasping tasks, the operation point may correspond to the closed position of the gripper; for pressing tasks, the operation point may correspond to the center position of the button; for opening a door, the operation point may correspond to the gripping position of the handle; and for plugging tasks, the operation point may correspond to the alignment position of the plug.

[0119] Adaptive adjustment refers to a processing method that dynamically corrects or redetermines the action point based on the target object's pose, changes in contact position, the reachability of the target embodied agent, the semantics of the target task, or scene constraints. Adaptive adjustment does not require completely replicating the action position in the source scene; instead, it emphasizes automatically determining a more suitable action position for the current execution environment while inheriting the task intent of the source scene and incorporating new conditions in the target scene. Adaptive adjustment can be implemented in various ways, including rule-based calculation, geometric mapping, constraint solving, scoring and filtering, and model prediction. Through adaptive adjustment, the positional deviation problem that occurs when directly reusing the original action point after a change in the target object's pose can be mitigated.

[0120] In some embodiments, after mapping the relative motion representation data to the target motion representation data corresponding to the target scene, the embodied intelligence data generation method may further include: optimizing the target motion representation data based on motion optimization processing so that the optimized target motion representation data satisfies at least one of kinematic constraints, dynamic constraints, smoothness constraints, and obstacle avoidance constraints.

[0121] Motion optimization refers to the process of adjusting constraints, correcting paths, enhancing executability, and improving trajectory quality after mapping relative motion representation data to target motion representation data corresponding to the target scene. Motion optimization can be implemented using analytical solutions, numerical optimization methods, constraint programming, sampling search, graph search, optimizer-based solutions, or learning-assisted optimization methods. The objects of motion optimization can be target motion information, target trajectory information, pathpoint sequences, end-effector pose sequences, or joint motion sequences. Through motion optimization, the execution quality of target motion representation data can be improved while preserving the task intent, making the target motion representation data more suitable for subsequent controller processing and simulation execution.

[0122] Kinematic constraints refer to the constraints established around the geometry and range of motion of the target embodied intelligent agent. Kinematic constraints can include limitations on joint angle range, joint displacement range, end effector reachability, singular pose constraints, and mechanical interference constraints. Kinematic constraints are primarily used to ensure that the motion path corresponding to the target motion representation data is geometrically reachable, solvable, and executable. By satisfying kinematic constraints, the problem of target motion representation data being semantically reasonable but structurally unrealizable can be addressed.

[0123] Dynamic constraints refer to the constraints established around the forces, velocities, accelerations, inertial responses, and driving capabilities of a target embodied intelligent agent during its motion. Dynamic constraints can include joint velocity limits, joint acceleration limits, driving force limits, load capacity limits, contact force variation limits, and stability constraints. The purpose of dynamic constraints is to ensure that the target's motion representation data is not only positionally accessible during execution but also within a feasible range in terms of physical response. By satisfying dynamic constraints, the problem of spatially feasible motion paths that are unstable, too fast, or exceed driving capabilities during physical execution can be improved.

[0124] Smoothness constraints refer to constraints established around the continuous changes in the target action representation data across the time and spatial dimensions. Smoothness constraints can manifest as requirements for continuous changes in position, attitude, velocity, acceleration, and gripper movements between adjacent path points. The focus of smoothness constraints is to reduce abrupt changes, jumps, and sharp transitions during the action process, making the target embodied agent's motion more natural, stable, and controllable. By satisfying smoothness constraints, the problem of discontinuous execution processes despite the target action representation data reaching the target position can be improved.

[0125] Obstacle avoidance constraints refer to conditions used to limit the unexpected collisions between a target intelligent agent and obstacles, target objects, supporting planes, environmental boundaries, or its own structure during the execution of a target task. Obstacle avoidance constraints can include end-effector obstacle avoidance constraints, robotic arm link obstacle avoidance constraints, safety distance constraints around the target object, collision constraints during gripper closure, and self-collision constraints. The introduction of obstacle avoidance constraints allows motion optimization processing to consider the spatial occupancy of the scene while generating or modifying target motion representation data, thereby improving the problem of the disconnect between path generation and scene obstacle distribution.

[0126] In some embodiments, the target action representation data may include target action information and / or target trajectory information, and the target embodied intelligent agent uses the control parameter configuration corresponding to the source reference data when performing the target task; wherein, the controller generates target control information based on the target action information and / or the target trajectory information, and in combination with the control parameter configuration.

[0127] A controller can refer to a control execution module used to convert target action information and target trajectory information into executable control outputs for the target embodied intelligent agent. The controller can be a position controller, velocity controller, force controller, impedance controller, hybrid controller, joint space controller, operand space controller, or other control execution mechanisms suitable for the target embodied intelligent agent. When the target embodied intelligent agent performs the target task, the controller can reuse the control parameter configuration corresponding to the source reference data. This means that the controller type and internal parameters corresponding to the source reference data acquisition or formation process are continued to be used when the target embodied intelligent agent performs the target task. This maintains the consistency of the control execution mechanism, ensuring that the control logic when the target action representation data is executed in the target scene is coordinated with the control logic during the source reference data formation process. Here, "reusing" does not equate to the control output remaining unchanged, but rather emphasizes that the controller framework and internal parameters remain unchanged. The specific target control information can change based on the target action information and target trajectory information. By reusing the control parameter configuration corresponding to the source reference data, the problem of inconsistent execution styles caused by differences in controller mechanisms can be improved.

[0128] Target control information refers to control-layer information generated by the controller based on target action information and / or target trajectory information, used to drive the target embodied intelligent agent to execute the target task. For example, the controller combines the task semantics corresponding to the target action information and the motion path corresponding to the target trajectory information to output a control result adapted to the current execution stage and current scene conditions. Through target control information, target action representation data can be further transformed into control content that the target embodied intelligent agent can directly execute.

[0129] This embodiment, by including at least one of target action information and target trajectory information in the target action representation data, can represent the target task from both the action semantic level and the motion path level, thereby improving the completeness of the description of the task execution process by the target action representation data. By using the control parameter configuration corresponding to the source reference data when the target embodied agent executes the target task, the task execution process in the target scene can continue to adopt a control execution mechanism consistent with the source reference data formation process, thus improving the inconsistency in control logic between the source reference data generation process and the target task execution process. By generating target control information based on the target action information and target trajectory information, the controller can output appropriate control results according to the action and path requirements in the target scene while maintaining consistency in its internal parameters, thereby improving the adaptability of the target control information to the specific execution conditions of the target task. By connecting the target action representation data with the target control information generated by the controller, the target action representation data can be further transformed into control content executable by the target embodied agent, thereby improving the execution rationality and stability of the target task in the simulation environment. This embodiment addresses the issues of insufficient connection between action representation and control execution during the generation of relevant embodied intelligent data, difficulty in continuing the control logic corresponding to the source reference data in the target scenario, and insufficient consistency of control mechanisms between the target task execution process and the reference execution process. It adopts the concept of using the control parameter configuration corresponding to the source reference data and having the controller generate target control information based on the target action information and target trajectory information, thereby improving the control consistency during the target task execution process and enhancing the effectiveness of converting target action representation data into the actual control process.

[0130] In some embodiments, the step of enabling the target embodied intelligent agent to perform the target task in the target scene of the simulation environment based on the target action representation data may include: sending the target scene representation data and the target action representation data to the simulation execution terminal through a communication interface, so that the simulation execution terminal constructs the target scene and drives the target embodied intelligent agent to perform the target task.

[0131] In some embodiments, acquiring the process data corresponding to the task execution process and / or the result data corresponding to the task execution result may include: receiving the process data and / or the result data from the simulation execution terminal. The process by which the simulation execution terminal acquires the process data and / or the result data may include: parsing the target trajectory information and gripper position information in the target motion representation data; performing inverse kinematics on the target pose corresponding to the target trajectory information to obtain joint control information; and, based on the joint control information and the gripper position information, driving the target embodied agent to execute the target task frame by frame, and recording the process data and / or the result data during execution.

[0132] In some embodiments, driving the target embodied agent to perform the target task frame by frame may include: in the grasping phase, establishing a connection between the target object and the gripper through an attachment process; and in the placement phase, detaching the connection between the target object and the gripper through a separation process.

[0133] A communication interface refers to an interactive channel used to transmit scene information, action information, control information, and execution result information between the data generation side and the simulation execution side. The communication interface's role extends beyond data transmission; it also handles task triggering, execution coordination, result feedback, and module decoupling. Communication interfaces can be implemented using remote procedure call interfaces, message queue interfaces, network socket interfaces, service call interfaces, or other interaction methods suitable for cross-module, cross-process, and cross-device data transmission. In this embodiment, the communication interface can be used to send target scene representation data and target action representation data, enabling the simulation execution end to construct the target scene and drive the target embodied intelligent agent to execute the target task, thereby mitigating the structural complexity caused by the mixing of data generation logic and simulation execution logic.

[0134] The simulation execution end can refer to an execution-side module or device that receives target scene representation data and target action representation data, and is responsible for constructing the target scene in the simulation environment, driving the target embodied intelligent agent to execute the target task, and recording the execution results. The simulation execution end can be a standalone simulation server, simulation process, simulation node, execution control module, or a comprehensive execution platform with scene construction, motion solving, and data recording capabilities. This transforms the target scene information and action information generated by the data generation side into an executable simulation action process, and simultaneously collects process data and result data during execution. By setting up a simulation execution end, the division of responsibilities between data generation and simulation execution can be improved, making the entire embodied intelligent data generation process more suitable for batch, modular, and distributed expansion. For example, the data generation side first determines the target scene representation data, and then sends it to the simulation execution end through a communication interface. The simulation execution end then loads scene elements, sets object states, and restores the task environment accordingly, thus providing a clear scene foundation for the subsequent execution of the target embodied intelligent agent's target task. The target action representation data is an intermediate result generated by the data generation side based on source reference data and target scene representation data, and is subsequently sent to the simulation execution end through a communication interface. The role of target motion representation data is not only to describe "what action to take", but also to describe "what path to take" and "how to enter the next stage of action", thus providing input basis for inverse kinematics solution and frame-by-frame driving.

[0135] After receiving the target trajectory information, the simulation execution end can use it as the target input for inverse kinematics solving, thereby further transforming the high-level motion representation into a specific joint control process. Gripper position information refers to information used to characterize the gripper's opening / closing state, gripping position state, or gripper-driven target state. Gripper position information can be represented by various examples such as gripper opening value, relative distance between the left and right grippers, gripper drive stroke value, gripping closure target value, and staged gripper state labels. Gripper position information can be used in conjunction with target trajectory information, enabling the simulation execution end to not only know where the end effector should go, but also whether the gripper should be in an open, closed, held, or released state at the corresponding path points. By introducing gripper position information, the problem of incomplete task execution when only spatial trajectory is available but lacks grasping execution details can be improved.

[0136] The target pose refers to the position and attitude state corresponding to each trajectory point or key motion point of the end effector in the target trajectory information. The target pose can be represented using three-dimensional coordinates and attitude angles, or it can be represented using a combination of pose matrices, quaternions, and displacement vectors. Therefore, the simulation actuator does not directly execute the abstract motion semantics, but first performs inverse kinematics solutions around the target pose, and then transforms the solution into joint-level control variables. The target pose can correspond to specific spatial states in multiple task stages such as approach pose, grasping pose, transport pose, placement pose, or evacuation pose, thus serving as a crucial intermediate bridge connecting trajectory information and joint control information.

[0137] Inverse kinematics (IK) refers to the process of calculating the appropriate states for each joint of a target embodied intelligent agent based on the target position and orientation of the end effector. The purpose of IK is to convert the target pose of the end effector from the target trajectory information into executable results in the joint control space. IK can be implemented using analytical methods, numerical iteration methods, constraint optimization methods, lookup table methods, or hybrid methods. For different types of target embodied intelligent agents, such as robotic arms, dual-arm robots, and humanoid robots, the number of joints, coupling relationships, and constraints considered in IK can vary. IK can improve the problem that the target pose of the end effector cannot directly affect the robot's actuator.

[0138] Joint control information refers to information derived from inverse kinematics solutions or other control calculations, used to drive the joints of a target embodied intelligent agent to perform actions. Joint control information can include various examples such as joint position control information, joint velocity control information, joint acceleration control information, joint torque control information, and joint target state sequences. The role of joint control information is to decompose the execution requirements corresponding to the target pose into individual joint levels, giving each joint a clear motion command. When combined with gripper position information, joint control information can form a complete execution control input, thereby improving the problem of simulation execution failure when only the end-effector target is available but specific driving details are lacking.

[0139] Frame-by-frame driving refers to a driving method that sequentially applies joint control information and gripper position information to the target embodied agent across multiple consecutive execution frames, following a chronological order. The advantage of frame-by-frame driving is that instead of issuing all control results at once and statically completing the task, it allows the target embodied agent to execute actions progressively as the simulation time advances, thus forming a continuous task process. Frame-by-frame driving can include various examples such as updating joint target positions frame-by-frame, updating gripper opening and closing states frame-by-frame, advancing physical simulation frame-by-frame, detecting contact states frame-by-frame, and recording data frame-by-frame. Frame-by-frame driving can improve issues such as discrete jumps in action execution, missing process data, or discontinuous task state transitions.

[0140] Attachment processing refers to the process of establishing a connection between the target object and the gripper during the grasping phase. The implementation process of attachment processing is as follows: when the target embodied agent reaches the grasping phase and meets preset conditions, such as the end effector reaching the target pose, the gripper closing, or the target object entering the attachable area, attachment processing is triggered, enabling the target object to move together with the gripper in the subsequent handling phase. Attachment processing can be implemented using various methods, including rigid attachment, constraint attachment, fitting attachment, and binding attachment. Attachment processing can improve the instability of the grasping process when relying solely on surface collision or simple gripper closure.

[0141] Separation processing refers to the method of detaching the target object from the gripper during the placement phase. The implementation process of separation processing is as follows: when the target embodied agent reaches the placement phase and meets preset conditions, such as the target object reaching the target placement area, the gripper opening to a preset position, or the object forming stable contact with the support surface, separation processing is triggered, causing the target object to detach from the gripper and remain in the target position. Separation processing can be implemented using various methods such as unbinding, constraint release, attachment cancellation, and independent object recovery. Separation processing can improve the problem of the target object remaining entangled with the gripper or being incompletely released after placement.

[0142] The grasping phase refers to the task phase in which the target embodied intelligent agent approaches, aligns, closes its gripper, and establishes a connection with the target object during the execution of the target task. The grasping phase can include multiple stages such as approach actions, posture adjustment actions, gripper closure actions, contact confirmation actions, and attachment processing trigger actions. The grasping phase plays a crucial role in the entire task flow, transitioning from an "untouched object" to a "controlled object." By explicitly configuring the attachment process within the grasping phase, the object state transition can correspond to the task phase logic, thereby improving the asynchronous problem between grasping behavior and object connection behavior.

[0143] The placement phase refers to the task phase in which the target embodied intelligent agent moves the target object to the target area and releases it during the execution of the target task. The placement phase can include multiple steps such as moving the object into place, adjusting its posture, aligning the target object, opening the gripper, and triggering the separation process. The purpose of the placement phase is to switch the target object from a "gripper-controlled" state to an "environment-supported" state. By explicitly configuring the separation process in the placement phase, the timing of the object's detachment from the gripper can be aligned with the task release logic, thereby improving issues such as premature detachment, delayed release, or unstable placement.

[0144] In this embodiment, based on target action representation data, the target embodied intelligent agent performs target tasks in a target scene within a simulation environment. This can be achieved by sending target scene representation data and target action representation data to the simulation execution end via a communication interface. The simulation execution end then constructs the corresponding target scene based on the target scene representation data and drives the target embodied intelligent agent to perform the target task based on the target action representation data. During simulation execution, the simulation execution end can parse the target trajectory information and gripper position information from the target action representation data, and perform inverse kinematics solving on the target pose corresponding to the target trajectory information to obtain joint control information. Subsequently, based on the joint control information and gripper position information, the target embodied intelligent agent is driven frame by frame to complete tasks such as approaching, grasping, carrying, placing, or releasing. Simultaneously, process data corresponding to the task execution process and result data corresponding to the task execution result are recorded during execution. Specifically, during the grasping phase, attachment processing can be used to establish a connection between the target object and the gripper to represent the following relationship after the target object is grasped. During the placement phase, separation processing can be used to release the connection between the target object and the gripper to represent the detachment relationship after the target object is released. After completing task execution and data recording at the simulation execution end, the process data and result data are returned to the data generation side, thus forming an embodied intelligent data generation closed loop consisting of target scene construction, action execution, process recording and result feedback.

[0145] This embodiment sends target scene representation data and target action representation data to the simulation execution end via a communication interface, decoupling the data generation side from the task execution side and enabling target scene construction and target task execution to be completed at the simulation execution end, thereby improving the modularity and execution flexibility of the embodied intelligence data generation process. By parsing the target trajectory information and gripper position information at the simulation execution end, and performing inverse kinematics solutions on the target pose corresponding to the target trajectory information, the target action representation data can be transformed into joint control information and gripper control information that can be executed by the target embodied intelligence, thereby improving the ability to convert target action representation data into an executable control process. By driving the target embodied intelligence to execute the target task frame by frame based on the joint control information and gripper position information, and recording process data and result data during execution, the generated embodied intelligence data can simultaneously cover the action execution process and task execution results, thereby improving the integrity and training application value of the embodied intelligence data. By employing attachment processing during the grasping phase to establish a connection between the target object and the gripper, and then using separation processing during the placement phase to detach the target object from the gripper, the controllability of the target object's state transitions during both phases can be improved, thereby enhancing the execution stability and result consistency of related tasks in the simulation environment. By returning process data and result data from the simulation execution end to the data generation end, a closed-loop processing chain can be formed, encompassing data generation, simulation execution, and result feedback, thus improving the efficiency of batch generation of embodied intelligent data. This embodiment addresses the problems of unstable transformation of motion representations into simulation execution processes, insufficient connection between task execution and data recording processes, and insufficient controllability of object state transitions during the grasping and placement phases in the related embodied intelligent data generation process. It adopts a design that sends target scene representation data and target motion representation data to the simulation execution end via a communication interface, where the simulation execution end completes inverse kinematics solving, frame-by-frame driving, attachment processing, separation processing, and process and result data feedback, thereby improving the execution stability, process integrity, and result usability of the embodied intelligent data generation process.

[0146] In some embodiments, generating embodied intelligence data based on at least one of the target action representation data, the process data, and the result data may include: repeatedly performing target scene representation data determination, target action representation data determination, task execution, and data acquisition processing until the generated embodied intelligence data reaches the target data volume.

[0147] In some embodiments, the repeated execution of target scene representation data determination, target action representation data determination, task execution, and data acquisition processing may include: dividing the data generation task corresponding to the target data volume into multiple sub-tasks; allocating the multiple sub-tasks to multiple simulation execution terminals for parallel execution; receiving embodied intelligence data returned by multiple simulation execution terminals, and summarizing the returned embodied intelligence data to obtain a target dataset corresponding to the target data volume.

[0148] In this embodiment, the process of generating embodied intelligence data can be carried out in a cyclical iterative manner. That is, around the target data volume as the generation target, the determination of target scene representation data, the determination of target action representation data, task execution, and data acquisition processing are repeatedly executed to continuously generate new embodied intelligence data samples. In each round of processing, the target scene representation data corresponding to the current task round is first determined. Then, based on the target scene representation data and source reference data, the target action representation data adapted to the current target scene is determined. Subsequently, the target embodied intelligence agent is driven to execute the target task in the simulation environment and acquire the corresponding process data and result data. Then, new embodied intelligence data is formed based on at least one of the target action representation data, process data, and result data. When it is necessary to improve the efficiency of data generation, the data generation task corresponding to the target data volume can be divided into multiple sub-tasks, and these sub-tasks can be assigned to multiple simulation execution terminals for parallel execution. This allows multiple simulation execution terminals to complete the target scene construction, target action execution, process data recording, and result data return, respectively. Then, the embodied intelligent data returned by multiple simulation execution terminals is received and aggregated to obtain the target dataset corresponding to the target data volume. This forms a batch generation mechanism for embodied intelligent data based on a combination of cyclic generation and parallel aggregation.

[0149] This embodiment, by repeatedly executing the determination of target scene representation data, target action representation data, task execution, and data acquisition processing, can continuously generate multiple embodied intelligence data samples around the same target task, thereby improving the automation level of the embodied intelligence data generation process. By using at least one of the target action representation data, process data, and result data to generate embodied intelligence data, the generated results can cover at least one dimension of action expression, execution process, and execution result, thereby improving the richness and training value of the embodied intelligence data samples. By dividing the data generation task corresponding to the target data volume into multiple sub-tasks, the overall data generation target can be decomposed into multiple independently executable local tasks, thereby improving the task organization capability of the data generation process. By distributing multiple sub-tasks to multiple simulation execution terminals for parallel execution, multiple simulation execution terminals can simultaneously carry out scene construction, task execution, and data recording, thereby improving the efficiency of embodied intelligence data generation and reducing the generation time required to reach the target data volume. By receiving embodied intelligence data returned from multiple simulation execution terminals and summarizing the returned embodied intelligence data, the scattered generated data samples can be integrated into a target dataset corresponding to the target data volume, thereby improving the integrity of embodied intelligence data management and batch output capabilities. This embodiment addresses the problems of limited sample generation per round, low overall generation efficiency, and insufficient organization in the large-scale dataset construction process during the relevant embodied intelligence data generation process. It adopts a concept of cyclically executing the data generation chain and combining parallel execution and summary output from multiple simulation execution terminals, thereby improving the batch generation capability of embodied intelligence data and enhancing the efficiency and scalability of embodied intelligence dataset construction.

[0150] In one application scenario, this application provides an embodied intelligent data generation system for executing any of the above-described embodied intelligent data generation methods. The embodied intelligent data generation system adopts a client-server architecture, where the client is primarily responsible for data generation logic, task orchestration, trajectory generation, and data stream management, while the server (as an example of a simulation execution end) is primarily responsible for task execution and trajectory recording in the simulation environment. The client and server communicate efficiently through a remote procedure call interface.

[0151] See Figure 2 , Figure 2 This is a flowchart illustrating an embodied intelligent data generation process provided in an embodiment of this application. The client may include a master editor, a scene manager, a data generator, and an interface layer.

[0152] like Figure 2As shown, the basic configuration (configuration) is first input into the main orchestrator's main loop. The main orchestrator connects to the SceneManager to perform scene loading and randomization. The SceneManager also connects to the DataGenerator, which generates waypoints and interacts with the server through the RPC Interface layer. The RPC Interface layer connects downwards to the RPCClient and gRPC Server, and further to the controller / service Arm&Object (the robotic arm and object controller / service), ultimately executing trajectory and physical simulation in IsaacSim (an example of a simulation execution end). This forms a complete processing chain of "basic configuration - main loop orchestration - scene construction - path generation - remote call - simulation execution - data recording".

[0153] See Figure 3 , Figure 3 This is a task orchestration flowchart provided in an embodiment of this application.

[0154] like Figure 3As shown, the main editor can construct subtask trajectories through the `generate_subtask_segment` process. Specifically, it first obtains the current scene state, including the robot state and object pose; then, it selects a source demo, which can be chosen either based on the "closest object position principle" or randomly; after selecting the source demo, it extracts key information from the source demo trajectory, including the end effector pose (EEF pose) and gripper action (i.e., the gripper position information mentioned earlier); based on this, it performs coordinate transformation to transform the relative operation relationships in the source demo to the new object position. An example formula is: `new_EEF = cur_obj @ inv(src_EEF) @ src_obj`, where `new_EEF` is the end effector pose in the target scene, `cur_obj` is the target object pose in the source scene, `src_EEF` is the end effector pose in the source scene, and `src_obj` is the target object pose in the target scene. After completing the relative transformation, starting points can be added to ensure the newly generated trajectory begins from the current position. Dense waypoints are then generated through trajectory interpolation, ultimately returning a WaypointSequence. In other words, the role of the master editor can be summarized as: extracting skills from the source demo and then generalizing those skills to new scenarios.

[0155] See Figure 4 , Figure 4 This is a scenario generalization flowchart provided in an embodiment of this application.

[0156] like Figure 4As shown, the SceneManager's scene generalization process can be implemented through `load_scene_with_randomization`. In this process, `load_scene` is first called to load the USD scene (Universal Scene Description scene); then `robot.reset()` is called to reset the robot to its initial pose; afterwards, `randomize_objects` is entered to randomize the object positions. During randomization, `robot.open_gripper` can be called to open the gripper, and `store_reset_pose` is used to store the reset pose. Then, all objects are traversed, the current object pose is obtained, and then a random offset is made within the range specified by `xy_offset_range`, and the new pose is set using `set_object_pose`. Thus, while preserving the basic scene framework, the spatial distribution of the target object and related objects in the scene can be changed, and this can be further extended to generalization of other scene conditions such as lighting and materials, thereby generating a new scene as the target scene.

[0157] See Figure 5 , Figure 5 This is a simulation execution flowchart provided in an embodiment of this application.

[0158] like Figure 5As shown, the server can complete the simulation processing flow through `handle_set_trajectory_list` (trajectory list setting processing). Specifically, the server first parses `trajectory_list` and `gripper_positions`, where `trajectory_list` represents the sequence of trajectory points transmitted by the client, and `gripper_positions` represents the corresponding gripper position sequence. Then, the server iterates through each target point and calls inverse kinematics (IK) for each target point to obtain the robotic arm joint information corresponding to the target pose. It then combines the robotic arm joint information with the current gripper position to form `cmd_state = arm_joints + gripper_positions[i]`, which is the combined joint positions `[arm_joints... + gripper_pos]`. Each `cmd_state` is stored sequentially in `cmd_list`. Subsequently, during the _on_every_frame execution, the server reads the control states from cmd_list frame by frame, applies joint position and / or velocity control through ArticulationAction, and completes the physical simulation in IsaacSim. The core functionality of this execution part can be summarized as receiving waypoints, performing inverse kinematics solutions, and executing frame by frame.

[0159] See Figure 6 , Figure 6 This is a communication flowchart between a client and a server provided in an embodiment of this application.

[0160] like Figure 6As shown, the communication process between the client and server based on subtasks can use gRPC communication (a remote procedure call method). For example, in the subtask 0: Approach phase, the client calls move_pose(pose_list, gripper_positions, arm="left") and sends "movement pose: end effector pose trajectory, gripper position, specified left arm execution" to the server. The server executes "inverse kinematics solution + trajectory execution" and then sends the result back to the client. In the subtask 1: Pick phase, the client calls move_pose(pose_list, arm="left") and sends "movement pose: end effector pose trajectory, specified left arm execution" to the server. The server continues to execute "inverse kinematics solution + trajectory execution". Subsequently, the client further calls attach_obj(target_object) to send an attachment object request to the server. The server attaches the target object to the gripper using a sphere fitting method and sends the result back to the client after attachment is complete. In the subtask 2+: Place phase, the client again calls `move_pose(pose_list, arm="left")` to send the placement trajectory, and the server performs "inverse kinematics solution + trajectory execution". Afterwards, the client calls `detach_obj()` to send a detach object request to the server, which completes the detachment process and sends it back to the client. Thus, the gRPC communication layer not only handles the transmission of ordinary trajectory execution instructions but also the interaction of task state switching instructions such as target object attachment and detachment, thereby supporting a complete approach, grasp, and placement task chain.

[0161] The above embodiments employ a basic architecture comprised of a master editor, a scene manager, an interface layer, and a server-side trajectory execution closed loop. Within this architecture, functional modules can be added, removed, or combined according to task requirements. For example, scene generalization processing can be added or omitted, control strategy alignment processing can be added or omitted, and distributed parallel deployment processing can be added. Even with different module combinations, the core objective remains the same: expanding a small amount of manually generated demonstration data into diverse trajectory data on a scale of tens of thousands, providing a scalable, high-quality data source for embodied intelligence training. In other words, while the embodiments may differ in module configuration and processing chains, they all revolve around the same technical concept and therefore each constitutes parallel and independently valid patentable content.

[0162] As can be seen, this embodiment proposes an intelligent augmentation mechanism for simulated teleoperation data based on the "few-to-many" paradigm. Specifically, a small amount of manually generated demonstration data can be input into the master orchestrator, which automatically executes task scheduling, data generation, main loop orchestration, and collaborative processing of various modules. This automatically augments, for example, 2-100 manually generated demonstration data into a high-quality, diverse trajectory dataset of tens of thousands. This processing method does not rely on long-term, large-scale, repetitive acquisition of real-world teleoperation data. Instead, it achieves data volume expansion through automated orchestration and simulation execution. Therefore, it can significantly improve the problems of high cost and low efficiency of related data acquisition methods and provide a near "infinitely scalable" data source for embodied intelligence training.

[0163] In terms of scene construction, this embodiment introduces a multi-dimensional scene joint randomization mechanism. Specifically, the scene manager can jointly randomize object pose, background environment, lighting conditions, and object material style, so that each data generation corresponds to a new simulation scene. Through this processing method, the target embodied agent no longer faces a fixed object layout and visual conditions in each round of task execution, but rather a target scene where both the visual and physical environments change. This encourages the subsequently learned strategy to focus more on the task actions themselves and object interaction relationships, rather than over-relying on single scene features. In this way, the binding degree of relevant training data to fixed scenes can be improved, and the visual generalization ability and physical environment adaptability of the training strategy under different object poses, different lighting conditions, and different material styles can be enhanced.

[0164] Regarding motion transfer, this embodiment proposes a trajectory transfer and adaptive grasping mechanism based on geometric semantics. Specifically, the source trajectory can be segmented according to the motion process, and then the segmented trajectory can be transformed into the object's local coordinate system to decouple the trajectory from the global pose, thereby extracting the geometric semantic relationship relative to the target object. In a new scene, the grasping point can be adaptively adjusted by combining the new object pose and contact position changes, and the local trajectory can be remapped onto the new object pose. When necessary, the generated trajectory can be optimized using processing modules such as CuRobo (motion optimizer) to make the generated result smoother and meet the requirements of dynamic feasibility. In this way, the same operational intent is no longer limited to a fixed absolute trajectory in the source scene, but can be flexibly transferred to target scenes corresponding to different object poses, different grasping positions, and different contact relationships.

[0165] Regarding control execution, this embodiment also proposes a control strategy alignment mechanism. Specifically, in the joint control phase after trajectory generation, the control parameter configuration corresponding to the GenieSim MC controller (lower-level controller) used during source data acquisition can be reused, ensuring consistency between the trajectory execution process in the target scene and the source data formation process at the controller mechanism level. Here, the controller itself and its internal parameters can remain unchanged, while the specific control information output by the controller can be adjusted based on the target motion information and target trajectory information. Through this processing method, the generated trajectory can maintain homogeneity with the source data in terms of underlying control logic, execution method, and dynamic characteristics. This helps to improve problems such as trajectory unsmoothness, force control response deviation, or inconsistent end effector behavior caused by control mechanism switching, thereby improving the quality of generated data and the usability of subsequent training.

[0166] See Figure 7 , Figure 7 This is a structural block diagram of an embodied intelligent data generation system provided in an embodiment of this application.

[0167] This application also provides an embodied intelligent data generation system, which includes a data receiving module, a scene determination module, an action determination module, a task execution module, and a data generation module.

[0168] The data receiving module is used to receive source reference data, which is used to characterize the reference execution information of the source execution entity for the target task.

[0169] The scene determination module is used to determine target scene representation data, which is used to represent the target scene corresponding to the target task. The target scene representation data includes at least one of object pose information, environmental parameter information, scene observation information, and scene semantic information.

[0170] The action determination module is used to determine target action representation data corresponding to the target scene based on the source reference data and the target scene representation data. The target action representation data is used to represent the action process of the target embodied intelligent agent performing the target task in the target scene, and the target action representation data includes at least one of target action information, target trajectory information, target control information and policy output information.

[0171] The task execution module is used to enable the target embodied intelligent agent to perform the target task in the target scene of the simulation environment based on the target action representation data, and to obtain the process data corresponding to the task execution process and / or the result data corresponding to the task execution result.

[0172] The data generation module is used to generate embodied intelligence data based on at least one of the target action representation data, the process data, and the result data.

[0173] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0174] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0175] The computer program product may be a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the computer program product of the embodiments of this application is not limited thereto, and the computer program product may be any combination of one or more computer-readable media.

[0176] See Figure 8 , Figure 8 This is a structural block diagram of a computer device provided in an embodiment of this application.

[0177] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0178] The embodiments of this application do not limit the computer device, which may be, for example, a local computer device, a cloud computer device, a distributed computer device, etc.

[0179] The computer device may include: a memory 110, a processor 120, and a communication interface 130. The memory 110, the processor 120, and the communication interface 130 are connected through internal connection paths.

[0180] The memory 110 is used to store computer programs, which in some implementations may include code for implementing the methods of the embodiments of this application.

[0181] The processor 120 executes the computer program stored in the memory 110 to control the communication interface 130 to receive input data and information, and output operation results and other data. In some implementations, when the solutions of the embodiments of this application are implemented by software or firmware, the computer program used to implement the solutions of the embodiments of this application can be stored in the processor 120 and executed by the processor 120.

[0182] The memory 110 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM). It should be noted that the memory 110 described herein is intended to include, but is not limited to, any memory of these and other suitable types. As an example, the memory 110 includes random access memory (RAM), cache memory, and read-only memory (ROM). The memory 110 stores a computer program that can be executed by processor 120, causing processor 120 to implement the steps of any of the methods described above.

[0183] The processor 120 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 120 can be any conventional processor.

[0184] In implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 120 or by instructions in software form. The method disclosed in the embodiments of this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor 120. The software modules can be located in mature storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in the memory 110, and the processor 120 reads the information in the memory 110 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0185] In some implementations, in addition to the hardware units described above, computer devices may also include software modules, such as operating systems, basic input / output systems (BIOS), and application software.

[0186] An operating system is used to manage the hardware and / or software resources of a computer device; it is the kernel and foundation of the computer. The operating system handles fundamental tasks such as managing and configuring memory, determining the priority of system resource allocation, controlling input and output devices, operating the network, and managing the file system. To facilitate user operation, most operating systems provide a user interface for interaction with the system.

[0187] The BIOS is used to perform hardware initialization during the power-on boot phase and to provide runtime services for the operating system and applications. In some implementations, the BIOS can also monitor and display processor temperature and execute temperature protection strategies.

[0188] Application software, also known as an application program, can be understood as software written for a specific user application purpose, and is one of the main categories of computer software. For example, application software can be a program used to achieve purposes such as power control and temperature management.

[0189] It is understood that the specific examples in this application are only intended to help those skilled in the art better understand the implementation of this application, and are not intended to limit the scope of protection of this application.

[0190] It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0191] It is understood that the various implementation methods described in this application can be implemented individually or in combination, and this application does not limit them.

[0192] Unless otherwise stated, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0193] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0194] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the embodiments described above can be referred to the corresponding processes in other embodiments, and will not be repeated here.

[0195] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0196] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the technical solution in this application, depending on actual needs.

[0197] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0198] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to related technologies, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0199] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating embodied intelligent data, characterized in that, The method for generating embodied intelligence data includes: Receive source reference data, which is used to characterize the reference execution information of the source execution entity for the target task; Determine target scene representation data, wherein the target scene representation data is used to represent the target scene corresponding to the target task, and the target scene representation data includes at least one of object pose information, environmental parameter information, scene observation information, and scene semantic information; Based on the source reference data and the target scene representation data, target action representation data corresponding to the target scene is determined. The target action representation data is used to represent the action process of the target embodied intelligent agent performing the target task in the target scene. The target action representation data includes at least one of target action information, target trajectory information, target control information, and policy output information. Based on the target action representation data, the target embodied intelligent agent performs the target task in the target scene of the simulation environment, and obtains the process data corresponding to the task execution process and / or the result data corresponding to the task execution result; Embodied intelligence data is generated based on at least one of the target action representation data, the process data, and the result data.

2. The embodied intelligence data generation method according to claim 1, characterized in that, The source reference data includes at least one of the following: source demonstration information, source action information, source trajectory information, source control information, source state information, and source observation information; The source execution entity includes at least one of physical embodied intelligent agents, virtual embodied intelligent agents, and human-machine teleoperation systems.

3. The embodied intelligence data generation method according to claim 1, characterized in that, The determined target scene representation data includes: At least one of the object pose, background environment, lighting conditions, and material style in the target scene is randomized to obtain the target scene representation data.

4. The embodied intelligence data generation method according to claim 3, characterized in that, Randomizing the poses of objects in the target scene includes: Reset the target embodied intelligent agent to the reference posture and open the gripper of the target embodied intelligent agent; Traverse at least some objects in the target scene to obtain the current pose of the corresponding objects, and randomly offset at least some objects within a preset offset range to set the new pose of the corresponding objects based on the random offset result.

5. The embodied intelligence data generation method according to claim 1, characterized in that, The step of determining the target action representation data corresponding to the target scene based on the source reference data and the target scene representation data includes: Based on the relative pose relationship between the target object and the end effector in the source scene, and the pose of the target object in the target scene, the pose of the end effector in the target scene is determined so that the relative pose relationship between the target object and the end effector in the source scene matches the relative pose relationship between the target object and the end effector in the target scene. The relative pose relationship between the target object and the end effector in the source scene is determined based on the source reference data, the pose of the target object in the target scene is determined based on the target scene representation data, and the target motion representation data is determined based on the pose of the end effector in the target scene.

6. The embodied intelligence data generation method according to claim 5, characterized in that, The step of determining the target action representation data corresponding to the target scene based on the source reference data and the target scene representation data further includes: Add a starting point corresponding to the current position of the target embodied intelligent agent; Trajectory interpolation is performed based on the starting point and the end effector pose to generate a path point sequence.

7. The embodied intelligence data generation method according to claim 1, characterized in that, The source reference data includes source action information and / or source trajectory information in the source scene. The step of determining the target action representation data corresponding to the target scene based on the source reference data and the target scene representation data includes: Based on the source action information and / or the source trajectory information, the reference action process corresponding to the target task is divided into action stages; At least a portion of the source action information and / or source trajectory information is converted into relative action representation data relative to the target object in the source scene; Based on the target object pose corresponding to the target scene representation data, the relative motion representation data is mapped to the target motion representation data corresponding to the target scene.

8. The embodied intelligence data generation method according to claim 7, characterized in that, The step of mapping the relative motion representation data to the target motion representation data corresponding to the target scene based on the target object pose corresponding to the target scene representation data includes: Based on the target object's pose and contact position change information, the operation points corresponding to the relative motion representation data are adaptively adjusted, and corresponding target motion representation data is generated.

9. The embodied intelligence data generation method according to claim 7, characterized in that, After mapping the relative action representation data to the target action representation data corresponding to the target scene, the embodied intelligence data generation method further includes: Based on motion optimization processing, the target motion representation data is optimized so that the optimized target motion representation data satisfies at least one of kinematic constraints, dynamic constraints, smoothness constraints, and obstacle avoidance constraints.

10. The method for generating embodied intelligent data according to claim 1, characterized in that, The target action representation data includes target action information and / or target trajectory information. When the target embodied intelligent agent performs the target task, it uses the control parameter configuration corresponding to the source reference data. The controller generates target control information based on the target action information and / or the target trajectory information, and in conjunction with the control parameter configuration.

11. The method for generating embodied intelligent data according to claim 1, characterized in that, The step of enabling the target embodied intelligent agent to perform the target task in the target scene within the simulation environment based on the target action representation data includes: The target scene representation data and the target action representation data are sent to the simulation execution terminal through the communication interface, so that the simulation execution terminal can construct the target scene and drive the target embodied intelligent agent to perform the target task.

12. The embodied intelligence data generation method according to claim 11, characterized in that, The acquisition of process data corresponding to the task execution process and / or result data corresponding to the task execution result includes: Receive the process data and / or result data from the simulation execution terminal; The process by which the simulation execution terminal acquires the process data and / or the result data includes: Analyze the target trajectory information and gripper position information in the target motion representation data; Inverse kinematics is performed on the target pose corresponding to the target trajectory information to obtain joint control information; Based on the joint control information and the gripper position information, the target embodied intelligent agent is driven to perform the target task frame by frame, and the process data and / or the result data are recorded during the execution.

13. The embodied intelligence data generation method according to claim 12, characterized in that, The frame-by-frame driving of the target embodied intelligent agent to execute the target task includes: During the gripping phase, the attachment process establishes a connection between the target object and the gripper. During the placement phase, the connection between the target object and the gripper is severed through a separation process.

14. The embodied intelligence data generation method according to claim 1, characterized in that, The process data includes at least one of image information, depth information, joint state information, end-effector pose information, and contact state information; The result data includes at least one of the following: task completion marker, quality score, and result status information.

15. The embodied intelligence data generation method according to claim 1, characterized in that, The generation of embodied intelligence data based on at least one of the target action representation data, the process data, and the result data includes: Repeatedly execute the process of determining target scene representation data, determining target action representation data, task execution, and data acquisition and processing until the generated embodied intelligence data reaches the target data volume.

16. The embodied intelligence data generation method according to claim 15, characterized in that, The process of repeatedly executing target scene representation data determination, target action representation data determination, task execution, and data acquisition and processing includes: The data generation task corresponding to the target data volume is divided into multiple sub-tasks; Multiple subtasks are assigned to multiple simulation execution terminals for parallel execution; The system receives embodied intelligence data returned from multiple simulation execution terminals and summarizes the returned embodied intelligence data to obtain a target dataset corresponding to the target data volume.

17. An embodied intelligent data generation system, characterized in that, The embodied intelligence data generation system includes: A data receiving module is used to receive source reference data, which is used to characterize the reference execution information of the source execution entity for the target task; A scene determination module is used to determine target scene representation data, wherein the target scene representation data is used to represent the target scene corresponding to the target task, and the target scene representation data includes at least one of object pose information, environmental parameter information, scene observation information, and scene semantic information; The action determination module is used to determine target action representation data corresponding to the target scene based on the source reference data and the target scene representation data. The target action representation data is used to represent the action process of the target embodied intelligent agent performing the target task in the target scene, and the target action representation data includes at least one of target action information, target trajectory information, target control information and policy output information. The task execution module is used to enable the target embodied intelligent agent to perform the target task in the target scene in the simulation environment based on the target action representation data, and to obtain process data corresponding to the task execution process and / or result data corresponding to the task execution result; The data generation module is used to generate embodied intelligence data based on at least one of the target action representation data, the process data, and the result data.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the embodied intelligence data generation method according to any one of claims 1 to 16.

19. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the embodied intelligence data generation method according to any one of claims 1 to 16.

Images