Data generation method and device for smart operation, equipment and storage medium

Abstract

The invention provides a data generation method, device and equipment for dexterous operation and a storage medium, and the method comprises the steps: obtaining at least one piece of demonstration data generated when a robot executes a dexterous demonstration task, the dexterous demonstration task comprises the initial pose of at least one target object, the demonstration data comprises a first robot action sequence and first observation data corresponding to the first robot action sequence; according to the first observation data, the corresponding first robot action sequence is divided into a motion section and a skill section corresponding to each target object; responding to multiple times of pose adjustment of the target object, performing pose adaptation on the motion section and the skill section corresponding to the target object, and generating a plurality of second robot action sequences; and executing the second robot action sequence in the simulation environment, collecting corresponding second observation data, and pairing the second observation data with the second robot action sequence to generate training data.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a data generation method, apparatus, device, and storage medium for dexterous manipulation. Background Technology

[0002] In the field of humanoid dexterous hand manipulation, the training of robot policies typically relies on high-quality demonstration data. However, the scale of demonstration data is usually limited, making it difficult to fully cover the diversity of task scenarios, which can easily lead to limited generalization ability of the trained policies. Summary of the Invention

[0003] This application provides a data generation method, apparatus, device, and storage medium for dexterous operation, to at least solve the above-mentioned technical problems existing in the prior art.

[0004] According to a first aspect of this application, a data generation method for dexterous operations is provided, the method comprising: Acquire at least one piece of demonstration data generated by the robot performing a dexterous demonstration task, wherein the dexterous demonstration task includes the initial pose of at least one target object, and the demonstration data includes a first robot action sequence and first observation data corresponding to the first robot action sequence; Based on the first observation data, the corresponding first robot action sequence is divided into motion segments and skill segments corresponding to each target object; In response to multiple pose adjustments of the target object, pose adaptation is performed on the corresponding motion segments and skill segments of the target object to generate multiple second robot action sequences; The second robot action sequence is executed in the simulation environment, the corresponding second observation data is collected, and the data is paired with the second robot action sequence to generate training data.

[0005] According to a second aspect of this application, a data generation apparatus for dexterous operation is provided, the apparatus comprising: The acquisition module is used to acquire at least one piece of demonstration data generated by the robot performing a dexterous demonstration task. The dexterous demonstration task includes the initial pose of at least one target object. The demonstration data includes a first robot action sequence and first observation data corresponding to the first robot action sequence. The segmentation module is used to divide the corresponding first robot action sequence into motion segments and skill segments corresponding to each target object based on the first observation data; The pose adaptation module is used to adapt the pose of the motion segment and skill segment corresponding to the target object in response to multiple pose adjustments of the target object, and generate multiple second robot action sequences. The pairing module is used to execute a second robot action sequence in the simulation environment, collect corresponding second observation data, pair it with the second robot action sequence, and generate training data.

[0006] According to a third aspect of this application, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in this application.

[0007] According to a fourth aspect of this application, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in this application.

[0008] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0009] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which: In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0010] Figure 1 This illustration shows an application scenario diagram of the data generation method for dexterous operations provided in an embodiment of this application; Figure 2 A schematic diagram illustrating the implementation flow of the data generation method for dexterous operations provided in an embodiment of this application is shown. Figure 3 This illustration shows a schematic diagram of the implementation flow of the sequence partitioning operation in the data generation method for dexterous operations provided in an embodiment of this application; Figure 4 This illustration shows a schematic diagram of the pose adaptation operation of the data generation method for dexterous operations provided in an embodiment of this application; Figure 5 A schematic diagram illustrating the pose adaptation process of the data generation method for dexterous operations provided in an embodiment of this application is shown. Figure 6 This illustration shows a schematic diagram of the implementation process of the training data generation operation of the data generation method for dexterous operations provided in an embodiment of this application; Figure 7 This invention provides a schematic diagram illustrating the structural composition of a data generation apparatus for dexterous operation according to an embodiment of this application. Figure 8 A schematic diagram of the composition structure of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0011] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] Figure 1 The illustration shows an application scenario diagram of the data generation method for dexterous operation provided in the embodiments of this application.

[0013] In the field of humanoid dexterous hand manipulation, data generation methods have become an important means to alleviate data acquisition costs and improve policy generalization capabilities. On the one hand, some methods use reinforcement learning and domain randomization techniques to expand data diversity in simulations, but these rely on manual parameter tuning and have limited task adaptability. On the other hand, some methods improve coverage by demonstrating trajectory slicing and segmentation, but these require manual annotation and ground truth poses, resulting in high deployment costs.

[0014] To solve the above problems, refer to Figure 1 This application first provides a robotic data acquisition platform for collecting demonstration data, serving as the implementation vehicle for the method of this application. The platform includes a robot consisting of a RealMan robotic arm and a CasBot dexterous hand, and is equipped with a RealSense depth camera, which is used to acquire global 3D point cloud data containing the scene and target objects.

[0015] To accurately record the process of experts performing dexterous demonstration tasks and collect the demonstration data generated during the process, the robot data acquisition platform is also equipped with a 3D mouse (SpaceMouse) and integrates an end-effector pose control scheme based on the SpaceMouse, as well as a data glove (SenseGlove). When experts perform demonstration tasks, the 3D mouse can precisely control the translation and posture of the robotic arm in real time. Simultaneously, the SensorGlove collects the finger servo angles corresponding to the expert's hand movements and drives the dexterous hand to synchronize its movements in real time through a mapping algorithm, thereby achieving natural human-computer interaction and completing the demonstration. The aforementioned dexterous demonstration task can be viewed as placing at least one target object to be grasped in a pre-set operating scenario on the platform, and then using the robotic arm and dexterous hand to perform the grasping operation on the target object.

[0016] For each dexterity demonstration task, a small amount of demonstration data can be collected, such as two data points, as the basis for generating subsequent robot training data. Furthermore, during the demonstration, the depth camera's acquisition frequency can be configured, for example, to 20Hz, to ensure that the demonstration's motion details are fully captured.

[0017] Figure 2 A schematic diagram illustrating the implementation flow of the data generation method for dexterous operations provided in an embodiment of this application is shown.

[0018] Based on the aforementioned robot data acquisition platform, this application provides a data generation method for dexterous manipulation, the method comprising: Operation 101: Obtain at least one piece of demonstration data generated by the robot performing a dexterous demonstration task. The dexterous demonstration task includes the initial pose of at least one target object. The demonstration data includes a first robot action sequence and first observation data corresponding to the first robot action sequence.

[0019] Based on the aforementioned robot data acquisition platform, at least one piece of demonstration data generated by the robot performing a dexterity demonstration task is first collected. This demonstration data includes the complete sequence of actions performed by the robot during the demonstration task, i.e., the first robot action sequence, and also includes scene data during the robot's performance of the dexterity demonstration task, i.e., the first observation data.

[0020] In this embodiment of the application, the first robot motion sequence is used to illustrate continuous motion data of the robot during task execution, such as angles or angle control commands arranged in sequence. The first observation data is used to illustrate the geometric features of the dexterity demonstration task scene, such as scene point cloud data and robot state data.

[0021] Operation 102: Based on the first observation data, divide the corresponding first robot action sequence into motion segments and skill segments corresponding to each target object.

[0022] Based on the spatial relationship characteristics between the robot and the target object shown in the first observation data, the action sequence can be analyzed into motion segments and skill segments, forming a structure in which motion segments and skill segments are arranged alternately. Among them, the motion segment is used to show the process of the robot adjusting from free space and approaching the target object, such as the process of the robotic arm moving from the initial position to the vicinity of the target object; the skill segment is used to show the process of the robot performing fine operations that interact with the target object, such as the process of a dexterous hand performing operations that directly contact the target object, such as grasping and rotating.

[0023] Operation 103 responds to multiple pose adjustments of the target object by performing pose adaptation on the corresponding motion segments and skill segments of the target object, generating multiple second robot action sequences.

[0024] Changes in the target object's position cause a shift in the pose at the end of the skill segment, further affecting adjacent motion segments and thus constructing a large number of different motion sequences within the reachable range. Therefore, in order to expand the number of motion sequences, embodiments of this application also expand multiple second robot motion sequences based on the pose adjustment of the target object.

[0025] Specifically, after receiving the user's pose adjustment for the target object, pose adaptation calculations are performed on the motion segments and skill segments for the current pose adjustment, generating a second robot motion sequence corresponding to the current pose adjustment. The pose adaptation calculation can be based on robot kinematics principles and path planning algorithms. By recalculating the endpoint positions and connecting paths of each trajectory segment, it is ensured that the generated new motion sequence is reachable within the robot's workspace and satisfies the physical constraints during the motion process.

[0026] Operation 104: Execute the second robot action sequence in the simulation environment, collect the corresponding second observation data, pair it with the second robot action sequence, and generate training data.

[0027] Considering the high time cost, equipment wear risk, and safety issues associated with executing a large number of action sequences on a real robot acquisition platform, this application embodiment chooses to perform this step in a simulation environment.

[0028] Specifically, multiple sequences of second robot actions are played in a simulation environment. During playback, corresponding second observation data are synchronously acquired using a virtual camera configured within the simulation environment. The second robot action sequences are then spatiotemporally aligned and paired with the second observation data to construct the final training data.

[0029] Thus, in this embodiment, by acquiring a small amount of high-quality demonstration data and combining it with observation data, the first robot motion sequence shown in the original demonstration data is divided into multiple skill segments and motion segments. Then, by perturbing the pose of the target object, diverse second robot motion sequences are generated, fundamentally expanding the scale of the data and the scope of the scene coverage. Furthermore, by replaying multiple second robot motion sequences in a simulation environment, the physical rationality of the data is ensured, providing a reliable data foundation for the subsequent learning of dexterous manipulation strategies.

[0030] In one embodiment of this application, the robot is further trained based on training data to enable it to acquire and perform corresponding dexterous manipulation tasks. Specifically, the training data is used as input, and robot strategy learning methods such as deep reinforcement learning or imitation learning are employed to train a control strategy model that takes observation data (such as scene point cloud data) as input and robot action sequences as output. The trained control strategy model is then deployed on a real robot hardware platform (such as a robot data acquisition platform), enabling the robot to perform stable and precise grasping and manipulation of target objects in diverse scenes, thus achieving a complete closed loop from automated data generation to effective skill learning.

[0031] In one embodiment of this application, the robot includes a multi-degree-of-freedom robotic arm and a multi-degree-of-freedom dexterous hand. The first observation data includes scene point cloud data and state data of the robotic arm and dexterous hand. The first robot action sequence includes action data of multiple time steps arranged in chronological order. The action data of each time step includes joint angles of the robotic arm and joint angles of the dexterous hand.

[0032] Specifically, the robot includes a multi-degree-of-freedom robotic arm and a multi-degree-of-freedom dexterous hand. The first observation data includes scene point cloud data acquired by a depth camera, as well as state data of the robotic arm and dexterous hand. The first robot motion sequence includes motion data for multiple time steps arranged in chronological order, with each time step's motion data including the joint angles of the robotic arm and the dexterous hand.

[0033] In one embodiment of this application, the robotic arm can be a 7-DOF Realman Gen72B robotic arm, and the dexterous hand can be a 10-DOF CasBot POS dexterous hand. The depth camera can be a RealSense Intel RealSense L515 RGB-D camera.

[0034] For example, the demonstration data structure collected by the robot data acquisition platform can be:

[0035] Where t represents the time step, This is the first observation data, including scene point clouds. Robotic arm status data and dexterity hand status data , Control commands (also known as joint angles), including robotic arm joint angles (also known as robotic arm joint control commands). And dexterity hand joint angles (also known as dexterity hand joint control commands). , This represents the initial poses of K target objects. Correspondingly, the first robot motion sequence can be extracted from the demonstration data:

[0036] Figure 3 This illustration shows a schematic diagram of the implementation flow of the sequence partitioning operation in the data generation method for dexterous operations provided in an embodiment of this application.

[0037] refer to Figure 3 In one embodiment of this application, the above-mentioned operation 102, based on the first observation data, divides the corresponding first robot action sequence into a motion segment and a skill segment corresponding to each target object, including: Operation 201: Calculate the distance between the dexterous hand and the geometric center of the target object based on scene point cloud data; Operation 202: When the distance is higher than or equal to a preset threshold, mark the action sequence of the corresponding time period as a motion segment; Operation 203: When the distance is lower than the preset threshold, mark the action sequence of the corresponding time period as a skill segment.

[0038] To improve the generalization of the generated data, the first robot action sequence needs to be replayed in a simulation environment. However, considering that treating the entire action sequence as a unified structure often makes it difficult to simultaneously achieve global spatial generalization and local interaction fineness, this application's embodiments introduce a trajectory segmentation mechanism, further dividing the action trajectory into motion segments and skill segments based on whether the robot interacts with an object: the process of the robot adjusting from free space and approaching the object is divided into motion segments; the process of the robot performing fine actions such as pre-grasping, contact, and manipulation is divided into skill segments.

[0039] Specifically, based on the scene point cloud data shown in the first observation data, we calculate the distance between the geometric center of the target object and the dexterous hand. When the distance is below a threshold, it is marked as a skill segment; otherwise, it is classified as a motion segment. Ultimately, the first robot motion sequence is parsed into a structure of alternating motion and skill segments:

[0040] in, Represents a set of multiple time steps. and These correspond to the motion segment when the robot approaches the k-th object and the skill segment when interacting with that object, respectively.

[0041] Figure 4 This illustration shows a schematic diagram of the pose adaptation operation of the data generation method for dexterous operations provided in an embodiment of this application.

[0042] In one embodiment of this application, the above-described operation 103, in response to multiple pose adjustments of the target object, performs pose adaptation on the motion segments and skill segments corresponding to the target object, generating multiple second robot action sequences, including: Operation 301: For each pose adjustment, calculate the pose transformation of the target object after the adjustment relative to its initial pose.

[0043] The first robot motion sequence can be viewed as a sequence of motion data at multiple consecutive time steps, where motion data can refer to joint angles. Since the robot has two components: a robotic arm and a dexterous hand, the motion data in this application can be understood as the angle combination of the joint angles corresponding to the robotic arm and the dexterous hand, respectively. The joint angles of the robotic arm and the dexterous hand include the angles of all their joints. For example, a 7-DOF robotic arm has 7 joints, so the joint angles of the robotic arm include the angle values ​​corresponding to the 7 joints. Specifically, the segmented motion sequence can be represented as follows:

[0044] in, This represents the skill segment and movement segment corresponding to the target object (1~k). This represents all the angles of the dexterous hand joints under the value i.

[0045] In this embodiment of the application, the pose can be regarded as a matrix, such as a 4×4 transformation matrix, which includes the position (X, Y, Z) coordinates and orientation (rotational attitude) of the corresponding object in three-dimensional space.

[0046] Specifically, in order to accurately map the pose transformation to the joint space and ensure the physical executability of the trajectory, the embodiments of this application further utilize forward kinematics to analyze the action sequence of the pose transformation, mapping the joint angles of the robotic arm to the end-effector pose. And calculate the transformation of the target object's pose after adjustment relative to its initial pose:

[0047] in, Indicates at time step At that time, the angles of all joints of the robotic arm; The positive kinematics function can be understood as a mathematical model defined according to the mechanical structure of the robot. The input is the joint angle, and the output is the pose of the end effector in three-dimensional space. Indicates at time step At that time, the pose of the end effector of the robotic arm includes the position (X, Y, Z) coordinates and orientation (rotational attitude) of the end effector in three-dimensional space. This represents the i-th target object. This represents the initial pose of the i-th object. This represents the adjusted pose of the i-th object. express The inverse matrix of , where K represents the total number of target objects.

[0048] Operation 302: For the robotic arm, for the skill segment, adjust the corresponding robotic arm end-effector pose according to the pose transformation, and generate multiple new robotic arm joint angles corresponding to the current skill segment by solving inverse kinematics; for the motion segment, based on the adjusted robotic arm end-effector pose of its adjacent skill segments, generate a smooth trajectory through motion planning algorithm, and convert it into multiple new robotic arm joint angles corresponding to the current motion segment.

[0049] The pose transformation of the target object directly affects the execution mode of the robotic arm and dexterous hand. Since the robotic arm and dexterous hand perform different functions in different trajectory segments (motion segment and skill segment), this application makes separate adjustments to the robotic arm and dexterous hand during the motion sequence synthesis process.

[0050] Specifically, in the robotic arm section, in the skill segment... In the middle, the end effector pose of the robotic arm It will change according to the pose Adjustments were made, and new robotic arm joint angles for each target object were obtained through inverse kinematics. :

[0051] in, Indicates the skill level Initial pose of the end effector of the internal robotic arm Let represent the inverse kinematics solution function, and k represent the number of target objects.

[0052] Movement segment In this process, motion planning is used to connect the end poses of adjacent skill segments, and inverse kinematics is used to convert them into new robotic arm joint angles corresponding to each target object. :

[0053] in, Indicates the first The adjusted end-effector pose sequence for each target object, including the poses of all target objects. Indicates the first The adjusted end-effector pose sequence for each skill segment. This represents a motion planning algorithm used to generate a smooth trajectory connecting two poses.

[0054] Operation 303, for dexterous hands, maintains the angles of the corresponding multiple dexterous hand joints unchanged for both the movement and skill segments.

[0055] In the dexterous hand section, the movements of the dexterous hand mainly involve contact and manipulation, and are not sensitive to spatial disturbances, thus maintaining consistency with the original sequence:

[0056] Operation 304 will alternately combine all motion segments and skill segments processed by the robotic arm and dexterous hand in a time sequence to generate the second robot motion sequence corresponding to each pose adjustment.

[0057] The adjusted joint angles of the robotic arm and the dexterous hand are alternately combined to obtain the second robot motion sequence:

[0058] Thus, through precise pose transformation and differentiated processing of the robotic arm's motion segments and skill segments, this embodiment of the application ensures the physical feasibility and spatial rationality of the newly generated second robot motion sequence while maintaining the integrity of the operational skills.

[0059] For example, see reference. Figure 5 , Figure 5 This illustration shows a pose adaptation process of the data generation method for dexterous manipulation provided in an embodiment of this application. In this example, two target objects, object 1 and object 2, are included. The generated trajectory shown represents the skill segment for object 1. With the movement segment Based on pose transformation And the skill segment of object 2 With the movement segment Based on pose transformation The second robot action sequence is a visual representation obtained after processing based on the aforementioned operations 301 to 304.

[0060] Figure 6 This illustration shows a schematic diagram of the implementation process of the training data generation operation of the data generation method for dexterous operations provided in the embodiments of this application.

[0061] refer to Figure 6 In one embodiment of this application, the above-mentioned operation 104 involves executing a second robot action sequence in a simulation environment, collecting corresponding second observation data, pairing it with the second robot action sequence, and generating training data, including: Operation 401: In the simulator, execute the second robot action sequence and simultaneously collect the second observation data generated during the execution process.

[0062] Given the dynamic constraints inherent in the IsaacLab simulator, which ensure the executability of the trajectory, and the fact that its virtual camera can cover observations of objects in different positions and postures, guaranteeing that the generated data has visual matching and dynamic rationality, this application generates a large number of second robot action sequences and then executes these sequences in the IsaacLab simulator to collect observation data that is strictly synchronized with each action data in the second robot action sequence, laying the foundation for building a high-quality training dataset in the future.

[0063] Operation 402 transforms the scene point cloud data in the second observation data from the world coordinate system to the base coordinate system of the robotic arm.

[0064] To reduce the migration error from simulation to reality, this application also transforms the scene point cloud data from the world coordinate system to the robot arm's base coordinate system, and combines cropping, clustering, and sampling to achieve robust data alignment.

[0065] Operation 403 pairs the scene point cloud data corresponding to each time step in the converted scene point cloud data with the action data of the same time step in the corresponding second robot action sequence to form training data including multiple training data pairs.

[0066] The processed point cloud data is spatiotemporally aligned and paired with action sequences to obtain multiple standard "observation-action" training data pairs. This generates training data.

[0067] Thus, the embodiments of this application realize a complete conversion process from the generated action sequence to high-quality training data, which significantly improves the data generation efficiency while ensuring data quality, and effectively solves the problem of training data acquisition in dexterity tasks.

[0068] In one embodiment of this application, the scene point cloud data corresponding to each time step in the converted scene point cloud data is paired with the action data of the same time step in the corresponding second robot action sequence to form training data including multiple training data pairs. This includes: preprocessing the converted scene point cloud data, and pairing the scene point cloud data corresponding to each time step in the preprocessed scene point cloud data with the action data of the same time step in the corresponding second robot action sequence to form initial training data including multiple initial training data pairs; injecting domain randomization noise and Gaussian noise into the initial training data pairs to obtain training data.

[0069] Specifically, to further reduce the migration error from simulation to real, this application embodiment also injects domain randomization and Gaussian noise into point cloud data and motion data (joint angles) during the data generation process to enhance the adaptability of the strategy to sensing errors and environmental disturbances.

[0070] In one embodiment of this application, for training data, this embodiment also automatically removes training data pairs corresponding to collisions or task execution failures in the simulation environment according to a preset task success determination function, and retains successful training data pairs to form the final training data.

[0071] Specifically, to ensure the quality of the training data, this embodiment of the application also sets a task success determination function for different demonstration tasks, automatically eliminating incomplete trajectories (i.e., second robot action sequences) caused by collisions or failures, and only retaining the "observation-action" pairs corresponding to successful second robot action sequences to obtain the final training data. The final training data generated by this embodiment of the application achieves a balance between consistency in three-dimensional space and dynamic feasibility, providing reliable support for subsequent policy learning.

[0072] Figure 7 A schematic diagram of the composition structure of the data generation apparatus for dexterous operation provided in an embodiment of this application is shown.

[0073] refer to Figure 7 This application also provides a data generation apparatus for dexterous operations, the apparatus comprising: The acquisition module 501 is used to acquire at least one piece of demonstration data generated by the robot performing a dexterous demonstration task. The dexterous demonstration task includes the initial pose of at least one target object. The demonstration data includes a first robot action sequence and first observation data corresponding to the first robot action sequence. The segmentation module 502 is used to divide the corresponding first robot action sequence into motion segments and skill segments corresponding to each target object based on the first observation data. The pose adaptation module 503 is used to adapt the pose of the motion segment and skill segment corresponding to the target object in response to multiple pose adjustments of the target object, and generate multiple second robot action sequences. The pairing module 504 is used to execute the second robot action sequence in the simulation environment, collect the corresponding second observation data, pair it with the second robot action sequence, and generate training data.

[0074] In one embodiment of this application, the partitioning module 502 includes: The first calculation submodule is used to calculate the distance between the dexterous hand and the geometric center of the target object based on scene point cloud data; The first marking submodule is used to mark the action sequence of the corresponding time period as a motion segment when the distance is higher than or equal to a preset threshold. The second marking submodule is used to mark the action sequence of the corresponding time period as a skill segment when the distance is lower than a preset threshold.

[0075] In one embodiment of this application, the pose adaptation module 503 includes: The second calculation submodule is used to calculate the pose transformation of the target object after the pose adjustment relative to its initial pose for each pose adjustment. The first solution submodule is used to, for the robotic arm, adjust the corresponding end-effector pose according to the pose transformation for the skill segment, and generate multiple new robotic arm joint angles corresponding to the current skill segment through inverse kinematics; for the motion segment, based on the adjusted end-effector pose of the robotic arm in the adjacent skill segment, generate a smooth trajectory through motion planning algorithm, and convert it into multiple new robotic arm joint angles corresponding to the current motion segment. The second solution submodule is used to keep the angles of multiple dexterous hand joints unchanged for both the movement segment and the skill segment for dexterous hands; The combination submodule is used to alternately combine all motion segments and skill segments processed by the robotic arm and dexterous hand in a time sequence to generate the second robot motion sequence corresponding to each pose adjustment.

[0076] In one embodiment of this application, the pairing module 504 includes: The execution submodule is used to execute the second robot action sequence in the simulator and synchronously collect the second observation data generated during the execution process; The transformation submodule is used to transform the scene point cloud data in the second observation data from the world coordinate system to the base coordinate system of the robotic arm; The pairing submodule is used to pair the scene point cloud data corresponding to each time step in the converted scene point cloud data with the action data of the same time step in the corresponding second robot action sequence to form training data including multiple training data pairs.

[0077] In one embodiment of this application, the conversion submodule includes: The pairing unit is used to preprocess the converted scene point cloud data and pair the scene point cloud data corresponding to each time step in the preprocessed scene point cloud data with the action data of the same time step in the corresponding second robot action sequence to form initial training data including multiple initial training data pairs. The injection unit is used to inject randomized noise and Gaussian noise into the initial training data to obtain the training data.

[0078] In one embodiment of this application, the device further includes: a rejection module, used to automatically reject training data pairs corresponding to collisions or task execution failures in the simulation environment according to a preset task success determination function, and retain successful training data pairs to form final training data.

[0079] Regarding the apparatus in the above embodiments, the specific manner in which each module and unit performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0080] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0081] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0082] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0083] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0084] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as data generation methods for dexterous operations. For example, in some embodiments, the data generation methods for dexterous operations may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data generation methods for dexterous operations described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform a data generation method for dexterous operations by any other suitable means (e.g., by means of firmware).

[0085] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0086] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0087] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0088] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0089] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0090] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0091] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0092] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0093] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data generation method for dexterous manipulation, characterized by, The method comprises: acquiring at least one demonstration data generated by a robot performing a dexterous demonstration task, the dexterous demonstration task containing an initial pose of at least one target object, the demonstration data comprising a first robot action sequence and first observation data corresponding to the first robot action sequence; dividing the corresponding first robot action sequence into a motion segment and a skill segment corresponding to each target object according to the first observation data; in response to multiple pose adjustments of the target object, performing pose adaptation on the motion segment and the skill segment corresponding to the target object to generate multiple second robot action sequences; executing the second robot action sequence in a simulation environment, collecting corresponding second observation data, pairing the second observation data with the second robot action sequence, and generating training data.

2. The method of claim 1, wherein, The robot comprises a multi-degree-of-freedom mechanical arm and a multi-degree-of-freedom dexterous hand, the first observation data comprises scene point cloud data and state data of the mechanical arm and the dexterous hand, and the first robot action sequence comprises action data of multiple time steps arranged in time sequence, each time step of action data comprising mechanical arm joint angles and dexterous hand joint angles.

3. The method of claim 2, wherein, The method comprises: based on the scene point cloud data, calculating the distance between the dexterous hand and the geometric center of the target object; when the distance is higher than or equal to a preset threshold, marking the action sequence of the corresponding time segment as a motion segment; when the distance is lower than the preset threshold, marking the action sequence of the corresponding time segment as a skill segment.

4. The method of claim 2, wherein, The method comprises: for each pose adjustment, calculating the pose transformation of the adjusted pose of the target object relative to its initial pose; for the mechanical arm, for the skill segment, adjusting the corresponding mechanical arm end pose according to the pose transformation, and generating multiple new mechanical arm joint angles corresponding to the current skill segment through inverse kinematics solving; for the motion segment, based on the adjusted mechanical arm end pose of its adjacent skill segment, generating a smooth trajectory through a motion planning algorithm and converting it into multiple new mechanical arm joint angles corresponding to the current motion segment; for the dexterous hand, for the motion segment and the skill segment, the corresponding multiple dexterous hand joint angles remain unchanged; combining all the motion segments and skill segments processed for the mechanical arm and the dexterous hand in time sequence to generate a second robot action sequence corresponding to each pose adjustment.

5. The method of claim 2, wherein, The method comprises: in the simulator, executing the second robot action sequence and synchronously collecting the second observation data generated in the execution process; converting the scene point cloud data in the second observation data from the world coordinate system to the base coordinate system of the mechanical arm; The scene point cloud data corresponding to each time step in the converted scene point cloud data is paired with the action data of the same time step in the corresponding second robot action sequence to form training data including a plurality of training data pairs.

6. The method of claim 5, wherein, The pairing of the scene point cloud data corresponding to each time step in the converted scene point cloud data with the action data of the same time step in the corresponding second robot action sequence to form training data including a plurality of training data pairs comprises: The converted scene point cloud data is preprocessed, and the scene point cloud data corresponding to each time step in the preprocessed scene point cloud data is paired with the action data of the same time step in the corresponding second robot action sequence to form initial training data including a plurality of initial training data pairs. The initial training data pairs are injected with domain randomization noise and Gaussian noise to obtain training data.

7. The method of claim 6, wherein, The method further comprises: According to a preset task success determination function, the training data pairs corresponding to collisions or task execution failures in the simulation environment are automatically removed, and the successful training data pairs are retained to form final training data.

8. A data generating apparatus for dexterous manipulation, characterized by, The device comprises: An acquisition module configured to acquire at least one demonstration data generated by a robot performing a dexterous demonstration task, the dexterous demonstration task including an initial pose of at least one target object, the demonstration data including a first robot action sequence and first observation data corresponding to the first robot action sequence; A division module configured to divide the corresponding first robot action sequence into a motion segment and a skill segment corresponding to each target object according to the first observation data; A pose adaptation module configured to generate a plurality of second robot action sequences by performing pose adaptation on the motion segment and the skill segment corresponding to the target object in response to a plurality of pose adjustments of the target object; A pairing module configured to execute the second robot action sequence in a simulation environment, collect corresponding second observation data, and pair the second observation data with the second robot action sequence to generate training data.

9. An electronic device, comprising: comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-7. The computer instructions are used to enable the computer to perform the method of any one of claims 1-7.

Images