Reinforcement learning method for multi-fingered hand operation based on human motion prediction model

Abstract

The application discloses a multi-fingered hand operation reinforcement learning method based on a human motion prediction model and belongs to the field of humanoid robot dexterous hand body operation. A human motion prediction model is trained by using data with human hand and object three-dimensional pose annotation, which is used to generate three-dimensional trajectories of human hand and object interaction in a generalizable manner. A general reward function is constructed by using the trajectory, which is used to train a dexterous hand reinforcement learning strategy in simulation. Then, the migration ability of the strategy from simulation to reality is improved through simulation parameter domain randomization and curriculum learning, and the trained reinforcement learning strategy is deployed on a real-world robot. The scheme introduces the understanding of the future pose of the object to be operated, predicts the interaction process of the human hand and the object as a whole, and constructs a unified, simple, efficient reward function based on this, which is suitable for different operation tasks and different types of dexterous hands, and has the advantages of strong generalization, high success rate and strong reward function universality.

Description

Technical Field

[0001] This invention relates to the field of humanoid robot dexterous hand manipulation, and more particularly to a reinforcement learning method for multi-finger dexterous hand manipulation based on a human motion prediction model. Background Technology

[0002] With the rise of embodied intelligence technology, how to endow humanoid robots with generalized operational skills through data-driven approaches has become a hot topic in current robotics research. Among these, multi-fingered dexterous hands, thanks to their humanoid design, can perform operations in various complex postures for different scenarios. Learning humanoid multi-fingered dexterous operations is a crucial step towards developing general-purpose humanoid robots.

[0003] In the field of data-driven multi-finger dexterity hand manipulation learning, existing methods can be divided into three categories. The first category employs traditional reinforcement learning methods, which rely on brute-force search guided by a reward function, typically requiring redesign of the reward function for different tasks. Inspired by the structural similarity between the human hand and a multi-finger dexterity hand, the second and third categories attempt to learn dexterity hand skills from human movements. The second category directly utilizes existing hand trajectories in the dataset, converting them into expert trajectories for the dexterity hand after redirection, and then imitating these expert trajectories through imitation learning. The third category trains a neural network that, given a hard-coded object trajectory, can predict the future trajectory of the dexterity hand, and then trains a network using residual reinforcement learning to optimize the underlying control of the dexterity hand.

[0004] However, existing technologies have significant drawbacks. First, multi-fingered dexterous hands, due to their numerous joints, far exceed common robotic end effectors (such as parallel grippers) in terms of spatial freedom and complexity. This makes the first type of method prone to slow training convergence and unstable success rates due to the excessively large exploration space. Furthermore, the complex nonlinear dynamics of the interaction between the dexterous hand and the object necessitate manually designing reward functions for different scenarios and tasks, making it difficult for the first type of method to achieve a universal reward function design. The second type of method relies excessively on memorizing human data, resulting in poor generalization performance, inability to handle entirely new objects and poses, and complete neglect of the perception and understanding of the manipulated object. The third type of method requires prior knowledge of the complete trajectory of the manipulated object, increasing the deployment cost of real-world robots in open worlds, and has extremely high requirements for the initial pose of the object, making it difficult to handle situations where the initial pose of the object deviates significantly from the training data.

[0005] In summary, previous methods focused solely on mimicking human hand behavior in data, neglecting the perception of the manipulated object and the understanding of the complex interaction between the hand and the object. This resulted in poor generalization across different types of objects, different poses of objects, and different tasks. Therefore, how to incorporate the perception and understanding of the manipulated object and how to predict the interaction between the hand and the object as a whole became the technical problems to be solved by this invention. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a multi-finger dexterity hand operation reinforcement learning method based on a human action prediction model. This method can simultaneously predict the future trajectories of both the human hand and objects, treating the interaction process between the human hand and objects as a whole for prediction. This improves the versatility, convergence speed, and success rate of multi-finger dexterity hand operation reinforcement learning, as well as enhancing the naturalness and anthropomorphism of the operation, thereby promoting the further development and application of multi-finger dexterity hand operation reinforcement learning.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] In a first aspect, this invention proposes a reinforcement learning method for multi-finger dexterous hand manipulation based on a human action prediction model, comprising the following steps:

[0009] Step 1: Train a human action prediction model using human operation data with 3D pose annotation. The human action prediction model can predict the human hand pose and object pose in future time steps in an autoregressive manner based on the input human hand pose, object pose and object geometric features.

[0010] Step 2: In the physical simulation environment, obtain the current object pose and dexterous hand state, convert the dexterous hand state into a human hand MANO parameter representation, and perform coordinate space normalization and rotation representation continuity processing together with the object pose and object geometric features to form an input format consistent with the training data; call the pre-trained human motion prediction model to synchronously generate the human hand pose and the pose of the manipulated object at future time steps, and then map the predicted human hand pose into joint angles and wrist poses of a dexterous hand suitable for robot control;

[0011] Step 3: Based on the joint angles and wrist pose of the dexterous hand predicted in Step 2, and the pose of the manipulated object, construct a general reward function for reinforcement learning. The general reward function includes a wrist following reward function, an object following reward function, and a contact reward function. Using the current observation state as input, guide the reinforcement learning algorithm to train an operation policy network based on the general reward function. During the training process, introduce simulation environment domain randomization technology and course learning technology for the initial pose of the object.

[0012] Step 4: Deploy the trained operation strategy network to a real robot platform.

[0013] Furthermore, the human motion prediction model includes a feature extraction network and a backbone network. The feature extraction network is used to extract the geometric features of the object. The backbone network takes the hand pose, the pose of the manipulated object, and the object's geometric features at the historical time step as input, and simultaneously predicts the hand pose and the pose of the manipulated object at the future time step. The hand pose is defined by MANO parameters.

[0014] Furthermore, the dataset processing for training the human action prediction model includes:

[0015] Sequences containing the complete process of human hand manipulation of objects are selected from publicly available datasets. The sequences include hand pose parameters based on the MANO model and 6D pose of the object, and the sequences are uniformly divided into continuous action segments.

[0016] Coordinate space normalization: A reference coordinate system is established based on the initial pose of the object, and the pose data of the hand and the object are uniformly transformed to the reference coordinate system; the 3D Mesh file of the object is read, and the Mesh vertices are first transformed to the world coordinate system using the pose of the object in frame 0, and then transformed to the reference coordinate system. The object point cloud is obtained through the farthest point sampling algorithm and the geometric features of the object are extracted based on the feature extraction network.

[0017] Rotation representation continuation processing: The axis-angle representation in the human hand pose and the quaternion representation in the object pose in the reference coordinate system are converted into 6D rotation representation, and zero-padding is performed on the first M frames to generate a dataset for training the human motion prediction model, where M is the length of the historical time step input to the backbone network.

[0018] Furthermore, the process of transforming the dexterous hand state into the optimal human hand MANO parameter representation includes:

[0019] Obtain the state of the dexterous hand, which includes the three-dimensional coordinate observations of the joints of the corresponding human hand in the dexterous hand model;

[0020] With the goal of minimizing the loss function of MANO parameter estimation, the three-dimensional coordinate observations are transformed into the optimal representation of the man's hand MANO parameters;

[0021] The loss function for the MANO parameter estimation includes a joint coordinate alignment term and a posture regularization term. The joint coordinate alignment term refers to the error calculation between the three-dimensional coordinates of the human hand joints calculated based on the currently estimated MANO parameters and the three-dimensional coordinates of the corresponding joints of the dexterous hand observed from the simulator. The posture regularization term is a regularization constraint on the estimated MANO parameters.

[0022] Furthermore, the wrist-following reward function includes a joint angle tracking reward, a wrist end position tracking reward, and a wrist end rotation orientation tracking reward.

[0023] The joint angle tracking bonus refers to the error calculation between the angles of each joint of the dexterous hand and the predicted reference joint angles.

[0024] The wrist end position tracking reward refers to the error calculation between the position of the dexterous hand's wrist end in three-dimensional space and the predicted reference wrist position;

[0025] The wrist end rotation orientation tracking bonus refers to the error calculation between the rotation orientation of the dexterous hand's wrist end and the predicted reference wrist rotation orientation.

[0026] Furthermore, the object following reward function includes an object translation tracking reward item and an object rotation tracking reward item;

[0027] The object translation tracking reward refers to the error calculation between the translation vector of the manipulated object in three-dimensional space and the predicted translation vector.

[0028] The object rotation tracking bonus refers to the error calculation between the rotational orientation of the manipulated object in three-dimensional space and the predicted rotational orientation.

[0029] Furthermore, the contact reward function refers to giving a one-time reward to a hand-object contact state that conforms to a preset contact pattern based on collision detection information from the simulation environment. The preset contact pattern requires that the thumb of the dexterous hand and at least one non-thumb finger simultaneously make contact with the object.

[0030] Furthermore, the course learning technique for the initial pose of an object specifically includes:

[0031] During reinforcement learning training, the success rate of the strategy is monitored in real time. When the success rate is below 30%, the initial pose randomization of the object is not applied. When the success rate is above 30%, the initial position offset of the object in the horizontal plane is gradually increased to a maximum of ±5cm, and the initial rotation range around the vertical axis is increased to a maximum of 360 degrees, according to the linear growth strategy.

[0032] Furthermore, the simulation environment domain randomization technique specifically includes:

[0033] Physical parameter noise injection: scaling the object collision mesh size from 0.96 to 1.04 times, scaling the object density from 0.50 to 2.00 times, scaling the object surface friction coefficient from 0.50 to 2.00 times, and scaling the object elastic coefficient from 0.90 to 1.10 times.

[0034] Noise injection into observation data: Add ±0.5cm noise to the observed object position values ​​in the X and Y directions, and add noise around the z-axis. The noise.

[0035] Secondly, the present invention proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-finger dexterity hand operation reinforcement learning method based on a human action prediction model as described above.

[0036] Thirdly, the present invention provides a computer electronic device, including a memory and a processor;

[0037] The memory is used to store computer programs;

[0038] The processor is configured to implement the multi-finger dexterity hand operation reinforcement learning method based on the human action prediction model as described above when executing the computer program.

[0039] The beneficial effects of this invention are:

[0040] (1) This invention introduces prior knowledge of the naturalness and intelligence of human operation into the training process of dexterity through a pre-trained human action prediction model. This model can simultaneously predict the future interaction trajectory between the human hand and the object in an autoregressive manner based on the current state (human hand pose, object pose, and object geometric features). Compared with traditional methods that only predict the human hand trajectory, this invention provides clear object operation goals and physical constraints for reinforcement learning by simultaneously predicting the motion trajectory of the object. This allows for more direct and effective guidance of the dexterity to complete anthropomorphic operations that conform to physical laws, significantly improving the efficiency and stability of skill learning.

[0041] (2) This invention utilizes predicted trajectories to construct a highly versatile reward function. Unlike traditional methods that require manually designing complex reward functions for each specific task, this invention designs wrist-following rewards and object-following rewards based on predicted dexterous hand trajectories and object trajectories, respectively. This reward mechanism does not depend on specific task objectives but encourages the agent to imitate predicted operational trajectories that conform to human experience, thereby achieving the generalization ability of a single reward function applicable to multiple tasks, greatly simplifying the application process and improving the versatility of the method.

[0042] (3) To ensure a high success rate of training results in real-world environments, this invention introduces simulation environment domain randomization (such as adding noise to physical parameters and observation data) and course learning for the initial pose of objects during the policy training phase, effectively improving the robustness of the policy to real-world uncertainties. This enables policies efficiently trained in simulation environments to be directly and stably deployed to real robot platforms. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating a multi-finger dexterity hand operation reinforcement learning method based on a human action prediction model proposed in this invention.

[0044] Figure 2 It is a flowchart of the implementation process of each step;

[0045] Figure 3 This is a schematic diagram illustrating the implementation process of a human motion prediction model;

[0046] Figure 4 This is a schematic diagram illustrating the framework from operational tasks to the actual deployment of robots;

[0047] Figure 5 This is a visualization of part of the experimental process. Detailed Implementation

[0048] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0049] The accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0050] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0051] like Figure 1 As shown, the present invention proposes a multi-finger dexterity hand operation reinforcement learning method based on a human action prediction model, which mainly includes four steps: predicting the trajectory of the object operated by the multi-finger dexterity hand, designing a reinforcement learning loss function, designing a reinforcement learning training strategy, and deploying the strategy. The contents of these four steps are described below.

[0052] In step S1, a human motion prediction model with human experience is learned from human operation data: the human operation data with three-dimensional pose annotation is preprocessed in batches, and then a human motion prediction model based on the Transformer autoregressive prediction network is trained; the human motion prediction model can predict the human hand pose and object pose in the future time step in an autoregressive manner according to the input human hand pose, object pose and object geometric features, forming the future pose trajectory of the human hand and the manipulated object, and then the predicted human hand pose is mapped to the joint angle and wrist pose of the dexterous hand suitable for robot control based on the fingertip redirection method, forming the trajectory of the multi-finger dexterous hand manipulating the object.

[0053] In step S2, in the reinforcement learning environment based on the physics simulator, a general reward function is designed using the dexterous hand trajectory and object trajectory generated in step S1 for efficient reinforcement learning in the simulator.

[0054] In step S3, during the reinforcement learning training process, simulation environment domain randomization technology and course learning technology for the initial pose of the object are introduced to further improve the generalization performance of the trained operation policy network for different poses of the object.

[0055] In step S4, the trained operation strategy network is deployed to the real robot platform.

[0056] The following combination Figure 2 The specific implementation process of each of the above steps will be explained.

[0057] Step S1 aims to learn a motion prediction model with human experience from human data, generate trajectories of human hands and objects, and convert them into dexterous hand manipulation trajectories. The entire process involves data preprocessing, model building, and inference transformation.

[0058] In one specific embodiment of the present invention, step S1 includes the following sub-steps:

[0059] S11, Filter human operation data with 3D pose annotations:

[0060] In this embodiment, sequences containing both 3D human hand poses and 6D object poses are selected from publicly available datasets such as Dex-YCB and HOI4D. These sequences completely record the process of human hand manipulation of objects. The original data is then uniformly divided into lengths... A continuous sequence of actions within a frame is denoted as a segment. Fragments with fewer than L frames remaining are truncated and discarded. (In world coordinate system) The following defines a standardized data format:

[0061] Define human hand position Using MANO parameters Indicates; where, parameter This represents the size and shape parameters of the hand. This represents the 16 axis angles and the global translation vector of the wrist. The 16 axis angles represent 16 joint rotation parameters, including 15 finger joint rotations and 1 global wrist rotation. Each rotation is represented by a three-dimensional vector, denoted as . The global translation vector of the wrist is represented by a three-dimensional vector, denoted as . The forward pass of the MANO model yields the rotation matrices for the 16 joints:

[0062]

[0063] Define object pose : Contains 3D global translation vector and 4D quaternion rotation representation The quaternion order is The rotation represented by the quaternion is then transformed into a rotation matrix. .

[0064] Define pose pairs : No. The pose pairs of frames are .

[0065] Human manipulation of datasets ,in, For the total number of sequences, each sequence .

[0066] S12, Coordinate space normalization based on the object's reference frame:

[0067] To eliminate the interference of the randomness of the absolute position and orientation of objects in the world coordinate system during data acquisition on model training, it is necessary to process the filtered data. Canonicalization is performed to unify the data to the bottom reference coordinate system of the object's 0th frame. Down.

[0068] The definition is as follows: Let the translation vector of the object in frame 0 in the world coordinate system be... The origin of the reference coordinate system is defined as... The basis vectors of this coordinate system are defined as follows: the x-axis is the positive forward direction of the world coordinate system, the y-axis is the positive left direction, and the z-axis is the positive upward direction (right-handed coordinate system). Next, a framework is constructed from the world coordinate system... To the reference coordinate system homogeneous transformation matrix Let the rotation matrix of the object in frame 0 in the world coordinate system be... Then the transformation matrix The calculation formula is as follows:

[0069]

[0070] Using the obtained homogeneous transformation matrix The MANO hand pose and object 6D pose of each frame in the dataset are uniformly transformed to this reference coordinate system:

[0071] Translation transformation: for any three-dimensional point in space (Such as the coordinates of a human hand joint, the coordinates of an object's center), and the resulting coordinates. for:

[0072]

[0073] Rotational transformation: for any rotational representation (such as the rotation matrix of a wrist or object) The transformed rotation matrix is:

[0074]

[0075] Obtain the processed data .

[0076] S13, Continuation mapping of rotation representation:

[0077] To address the discontinuity of axis angles and quaternions in neural network learning, this step employs 6D Rotation Representation to represent the processed data. Optimize.

[0078] Regarding manpower data, middle The axial angle vectors of the 16 joints Convert to rotation matrices respectively Then take the rotation matrix. The first two columns (i.e.) The first column vector Second column vector This forms 16 six-dimensional vectors, plus the global translation vector of the wrist. This makes the dimensions of the human hand pose 99.

[0079] For object data, middle Quaternions Convert to a rotation matrix and take the first two columns, then add the global translation vector of the object's center. This makes the object's pose dimension 9.

[0080] Obtain the processed data This representation has better Euclidean space continuity, which is beneficial for subsequent regression learning.

[0081] S14, Object point cloud feature injection:

[0082] Read the corresponding 3D mesh file of the object from the dataset metadata, and use the data... pose matrix of the object in frame 0 Transform the Mesh vertices to the world coordinate system, and then use the homogeneous transformation matrix from step S12. Transform to the reference coordinate system. Use the Farthest Point Sampling (FPS) algorithm to uniformly sample from the transformed mesh surface. A set of spatial coordinates recording these unordered points. This serves as the geometric condition input for the sequence. This step provides the model with geometric information about the object's structure, enhancing its generalization ability.

[0083] S15, Constructing a Human Action Prediction Model :

[0084] like Figure 3 As shown, the model consists of a feature extraction network. and backbone network Two parts:

[0085] Feature extraction network It adopts a standard three-layer PointNet architecture. The input is an unordered object point cloud. Each point in the point cloud is independently mapped to a feature vector through a weight-sharing multilayer perceptron. This multilayer perceptron consists of multiple fully connected layers connected in series, with dimensions of 64, 128, and 1024. Batch normalization and ReLU activation functions are applied between layers. This process maps low-dimensional geometric coordinates to high-dimensional feature vectors. Then Max pooling is applied to generate a global feature vector. Finally, the object's geometric features are projected onto a space that matches the dimensions of the hidden layers in the backbone network through a linear fully connected layer, yielding the final object geometric features. .

[0086] backbone network This network is used for autoregressive modeling of temporal dependencies, employing a Transformer decoder based on the GPT-2 architecture, with a certain number of layers. Attention count Hidden layer dimensions The model in Input of time From data Hand pose in the current frame Object pose and the extracted object point cloud features mentioned above It is pieced together:

[0087]

[0088] The model is based on a sequence of historical information from the past 10 frames. Predict the pose of the hand and objects in the next frame. Because the model lacks sufficient historical data for predicting frames 0-9, zero-padding is required during data loading. This is achieved using a method based on... Dataset after zero-padding The backbone network and feature extraction network are trained simultaneously using supervised learning. The loss function is defined as the predicted pose. Compared with the true pose Weighted mean square error between:

[0089]

[0090] in, and This is a hyperparameter used to balance the loss weights of the human hand and the object; in this embodiment, it is taken as... , .

[0091] S16, Model Inference:

[0092] In the simulation, during the network's inference phase, the current object pose is obtained based on the simulator's observation information. Three-dimensional coordinate observations of the joints of the human hand in the dexterous hand model The open-source software FrankMocap was used to simulate the MANO pose of the robot hand. It is estimated that, further based on the MANO model and Obtain the estimated three-dimensional coordinates of the joints corresponding to the human hand in the dexterous hand model. ,in It is a regularization term proposed by Frank Mocap to reduce solutions with unnatural shapes:

[0093]

[0094] in, These are the estimated three-dimensional coordinates of joint i. These are the three-dimensional coordinate observations of joint i.

[0095] according to Obtain the optimal MANO pose The above object pose and optimal MANO pose Perform the data preprocessing steps described above to obtain the data. Same format. Current observation status. Optimal MANO posture Point cloud features of the current object in the simulation Input Model Forward reasoning yields the predicted future trajectory. , It is the hand pose corresponding to the k-th predicted dexterity step. This is the observed state at the k-th prediction step. In actual execution, it is taken as... .

[0096] S17, Redirection from human hand to dexterous hand and transformation of object pose representation:

[0097] Redirection from human hand to dexterous hand: the reorientation of predicted trajectories manpower section Using an optimized retargeting algorithm from the open-source software dex-retargeting, it is mapped to the joint angle trajectory of a dexterous hand. And the 6D pose trajectory of the robot wrist .

[0098] Object pose representation transformation: for the object portion in the predicted trajectory (6D rotation representation), recovered as a standard rotation matrix using Gram-Schmidt orthogonalization. This is then converted into the quaternion corresponding to the rotation matrix of the object in the k-th prediction step. (The order of quaternions is) ), and the translation vector of the object in the k-th prediction step Together they form observation data The format of the input data is consistent with that of the input data and loaded into the simulator.

[0099] Based on the above calculations, step S1 ultimately outputs the predicted trajectory of the dexterous hand manipulating the object, including the joint angles of the dexterous hand at each predicted step. 6D wrist position of dexterous hand The quaternion corresponding to the object's transformation matrix Translation vector ( and Together they constitute the object's pose ).

[0100] like Figure 4 The diagram shown illustrates the framework from operational tasks to real robot deployment. First, observation information is obtained from the operational tasks. Then, the human motion prediction model in step S1 is called to generate the trajectory of the dexterous hand manipulating the object. Further, simulation and reinforcement learning, and real robot deployment are realized in steps S2 to S4.

[0101] The main purpose of step S2 is to establish an efficient reinforcement learning training framework in a physical simulation environment. By introducing trajectory information generated by a human action prediction model, a universal reward function is designed to solve the problems of low training efficiency and unstable success rate caused by the large exploration space in multi-finger dexterity hand operation.

[0102] In one specific embodiment of the present invention, step S2 includes the following sub-steps:

[0103] S21, Building a physical simulation environment for reinforcement learning:

[0104] In this embodiment, the simulation environment is built based on the open-source SAPIEN simulation platform, and the underlying physics engine calls the open-source NVIDIA PhysX. A seven-DOF robotic arm (such as xArm7) and a multi-fingered dexterous hand (such as ROBOTERA XHAND1 or Inspire Hand) are loaded into the simulator using the robot description file URDF. The robotic arm employs a joint control method, with the seven joint angles calculated from the position of the end effector through inverse kinematics. The dexterous hand's finger joints are controlled by position, and all joints use the PD control method. Furthermore, a three-dimensional environment (such as a desktop) is loaded into the simulator, and the reinforcement learning strategy for the manipulated object uses the standard PPO (Proximal Policy Optimization) method, employing an end-effector incremental control method.

[0105] The input to the policy network is the 6D pose of the manipulated object in the robot coordinate system in the current simulation environment. ), and the joint angles of dexterous hands in the current environment 6D wrist position of dexterous hand Output the Euler angle representation of the robotic arm's end effector (equivalent to a dexterous wrist) in the robot coordinate system (the three-dimensional coordinates of the end effector). This includes increments in roll, pitch, and yaw angles, as well as increments in the active free joints of the dexterous hand. The reinforcement learning process follows Markov decision-making. Within each fixed time step, the robot in the simulation environment receives environmental observations, performs inference through the reinforcement learning model, executes the action output by the reinforcement learning policy, receives the corresponding reward in the reward function based on changes in the environment, and then proceeds to the next step. This cycle continues until the task succeeds or fails. During this process, the parameters of the reinforcement learning network are continuously updated.

[0106] S22, Design a following reward function based on the robot wrist trajectory predicted in step S17. :

[0107]

[0108] in, For joint tracking, it encourages dexterous hand joint angles to follow reference angles predicted by the model:

[0109]

[0110] in, for Time-step robot hand joint angle, for Reference joint angle (predicted value) corresponding to the time step This is the scaling factor.

[0111] For wrist position following, this encourages the dexterous hand to keep its wrist position aligned with the reference position predicted by the model:

[0112]

[0113] in, for The wrist position of the time-stepping robot's hand. for The reference wrist position corresponding to the time step is obtained from the 6D pose of the wrist of the dexterous hand; This is the scaling factor.

[0114] For wrist rotation following, it encourages the dexterous hand's wrist orientation to follow the reference orientation predicted by the model:

[0115]

[0116] in, calculate The wrist of the time-stepping robot hand is oriented towards the axis angle. and the corresponding reference wrist orientation axis angle The angular distance between them is obtained from the 6D pose of the dexterous hand's wrist; This is the scaling factor.

[0117] In this embodiment, .

[0118] S23, Design the future pose of the object based on the prediction in step S17. Follow-up reward function :

[0119] Unlike traditional methods that use task-specific rewards (such as "lifting a certain height" or "opening a hinged object by a certain angle"), this method utilizes the predicted future object pose. As a general guiding signal, it gives the strategy stronger cross-task generalization and versatility. The reward function is defined as:

[0120]

[0121] The first calculation Time step object translation vector Translation vector of future objects predicted by the model The Euclidean distance error between them, the second term calculates the rotation error. It is a logarithmic mapping that converts quaternions into axis-angle representations. Represents the Hamiltonian product of quaternions. It is the L2 norm. In this embodiment, the scaling factor is... .

[0122] S24, Design a contact reward function to encourage exploration strategies. :

[0123] Encourage active finger exploration. Contact status is acquired through the collision detection function of the SAPIEN simulator, and a reward is given only when the thumb and at least one other finger (index, middle, ring, or little finger) simultaneously make contact with an object. .

[0124] S25, Task Success Assessment and Total Rewards:

[0125] Define the total reward function as:

[0126]

[0127] in, It's a wrist-following reward item. It is a reward item for following the target object. It's about accessing reward items. This is the reward coefficient; in this embodiment, .

[0128] Define the success conditions for the task (such as the object's height exceeding a certain range, or the object's final posture meeting certain requirements). When the success conditions are met, the current iteration round ends and the next iteration round begins.

[0129] The main purpose of step S3 is to introduce simulation environment domain randomization technology and course learning technology for the initial pose of objects into the above reinforcement learning training process.

[0130] S31, Course Learning and Randomization of Initial Object States:

[0131] To improve the strategy's adaptability to different initial conditions, the initial state of the object is randomized when resetting the simulation environment after each training iteration: the object's position is set to... Apply in plane The random position offset, and the object rotates around the z-axis. Random rotations are applied. Simultaneously, to ensure the stability of the reinforcement learning training process, a linearly increasing course learning strategy is adopted: a randomization threshold is defined. ,in This is the current time step. During the warm-up phase where the success rate is below 30%, let... =0, at which point the object's position is fixed, facilitating the robot's rapid learning of basic approach and grasping actions; once the success rate exceeds 30%, it increases linearly according to the formula. Until the maximum set value is reached (i.e., location) Rotation ):

[0132]

[0133] in, It's the success rate. Steps per hour It is a preset maximum number of training steps. In this embodiment, .

[0134] S32, Simulation Environment Domain Randomization and Observation Noise Injection:

[0135] To reduce the simulation-to-real gap, domain randomization is employed to add uniformly distributed noise to the physical parameters and observation data during the simulation. Specific randomization parameters are as follows:

[0136] Physical parameter noise injection: Scale the original object collision mesh by a factor of 0.96 to 1.04. Scale the original object density by a factor of 0.50 to 2.00. Scale the original static and kinetic friction coefficients of the object surface by a factor of 0.50 to 2.00. Scale the original restitution coefficient of the object material by a factor of 0.90 to 1.10.

[0137] Noise injection into the observation data: The object's position is increased in both the x and y directions. The observation noise is increased by the rotation of the object about the z-axis. Observation noise.

[0138] The main purpose of step S4 is to deploy the trained operation strategy network onto a real robot platform.

[0139] S41, Sim-to-Real Robot Deployment:

[0140] A dual-arm robot hardware platform was built, comprising two xArm-7 robotic arms, each equipped with a PSYONICAbility Hand, ROBOTERA XHAND1, or Inspire Robot Hand multi-fingered dexterous hand at the end effector. The perception module uses a RealSense D435i camera to acquire RGB-D data, and standard robotic arm hand-eye calibration is performed using EasyHEC++. The FoundationPose algorithm is employed for 6D pose estimation of the object in a model-based tracking mode.

[0141] The input to the policy network is the 6D pose of the manipulated object in the robot coordinate system in the real environment. ), and the joint angles of the current dexterous hand 6D wrist position of dexterous hand The system outputs the increments of the Euler angles of the end effector (equivalent to the dexterous wrist) in the robot coordinate system, as well as the increments of each active free joint of the dexterous hand, thus enabling control of both the dexterous hand and the robotic arm. All robotic arm joints employ the PD control method, using the open-source pyserial library for multi-threaded control. The control frequencies of both the dexterous hand and the robotic arm are maintained above 120 Hz, while the average inference frequency of the RL policy network is approximately 18 Hz.

[0142] To verify the effectiveness of the present invention, the following experiment was conducted:

[0143] Build your own dual-arm robot hardware platform, which includes two xArm-7 robotic arms, each equipped with a PSYONIC Ability Hand, ROBOTERA XHAND1, or Inspire Robot Hand multi-finger dexterous hand at the end.

[0144] Human motion prediction model Training on a single NVIDIA RTX 4090 GPU (24GB) and Intel i7-14700K CPU took approximately 2 hours to complete, with a learning rate of [missing information]. .

[0145] Simulation and reinforcement learning policy training were run on a single NVIDIA RTX 4090 GPU (24GB) and Intel i7-14700K CPU, with the number of parallel environments set to [value missing]. The average sampling rate can reach 10,000 steps / s, and the policy convergence time is approximately 25-30 minutes. The key hyperparameter used in policy learning is the maximum number of steps. Discount factor Learning rate During policy training, to improve the robustness of the reinforcement learning policy learned in simulation to the real world, the randomization parameters of the simulation environment and the injected noise parameters are as follows:

[0146] Scale the original object collision mesh by a factor of 0.96 to 1.04.

[0147] Scale the original object density by a factor of 0.50 to 2.00.

[0148] The original static and dynamic friction coefficients of the object surface are scaled from 0.50 to 2.00 times.

[0149] The original elastic modulus of the object's material is scaled by a factor of 0.90 to 1.10.

[0150] The observation of the object's position increases in both the x and y directions. Observation noise.

[0151] Increase the rotation of the object about the z-axis Observation noise.

[0152] In the task of grasping a single object, the motion prediction model can provide reasonable prediction results based on different object shapes. This makes the method of the present invention highly adaptable to different object shapes, and can achieve average performance in the simulation environment under different object and different initial pose conditions. The success rate is improved; at the same time, it benefits from the design based on the future pose trajectory of the object predicted in step S17. Follow-up reward function Unlike traditional methods that use task-specific rewards (such as "lifting a certain height" or "opening a hinged object by a certain angle"), this method utilizes predicted future object states. As a universal guiding signal, it gives the strategy stronger cross-task generalization and versatility. This allows the reinforcement learning strategy to be trained using the exact same reward function across different tasks. Therefore, the overall method exhibits strong cross-task generalization ability, achieving an average performance of [missing information] in the simulation environment across three tasks: grasping a single object, opening the upper part of a hinged object, and grasping a target object from multiple objects. The success rate, and visualization results of some experimental processes, such as... Figure 5As shown. Furthermore, the training of the policy benefits from the guidance of the model's predicted trajectory, drawing upon prior knowledge from human operations. This simplifies the extensive random exploration in the early stages of training in traditional reinforcement learning methods, thus requiring a shorter convergence time. This achieves... With a success rate as the target, the reinforcement learning strategy converges in approximately 25-30 minutes on different tasks, which is significantly faster than existing methods.

[0153] After being deployed on a real robot using a sim-to-real approach, the strategy achieves average performance across three tasks in a real robot environment with different initial poses: grasping a single object, opening the upper part of a hinged object, and grasping a target object among multiple objects. The high success rate demonstrates that the method of this invention exhibits good overall robustness against RGBD camera depth noise, object pose estimation noise, and robot joint noise in real-world environments. Furthermore, successful deployment on three multi-fingered dexterous hands—the PSYONIC Ability Hand, ROBOTERA XHAND1, and Inspire Robot Hand—verifies the versatility of the method.

[0154] It should also be noted that the multi-finger dexterity hand operation reinforcement learning method based on the human action prediction model in the above embodiments can essentially be executed by a computer program. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the method provided in the above embodiments, which includes a memory and a processor;

[0155] The memory is used to store computer programs;

[0156] The processor is configured to implement the multi-finger dexterity hand operation reinforcement learning method based on the human action prediction model in the above embodiments when executing the computer program.

[0157] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium.

[0158] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the method provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can realize the multi-finger dexterity hand operation reinforcement learning method based on the human action prediction model in the above embodiments.

[0159] It is understood that the computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0160] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A multi-fingered hand operation reinforcement learning method based on a human motion prediction model, characterized by, Includes the following steps: Step 1: Train a human action prediction model using human operation data with 3D pose annotation. The human action prediction model can predict the human hand pose and object pose in future time steps in an autoregressive manner based on the input human hand pose, object pose and object geometric features. Step 2: In the physical simulation environment, obtain the current object pose and dexterous hand state, convert the dexterous hand state into a human hand MANO parameter representation, and perform coordinate space normalization and rotation representation continuity processing together with the object pose and object geometric features to form an input format consistent with the training data; call the pre-trained human motion prediction model to synchronously generate the human hand pose and the pose of the manipulated object at future time steps, and then map the predicted human hand pose into joint angles and wrist poses of a dexterous hand suitable for robot control; Step 3: Based on the joint angles and wrist pose of the dexterous hand predicted in Step 2, and the pose of the manipulated object, construct a general reward function for reinforcement learning. The general reward function includes a wrist following reward function, an object following reward function, and a contact reward function. Using the current observation state as input, an operational policy network is trained based on a general reward function-guided reinforcement learning algorithm. During the training process, simulation environment domain randomization technology and course learning technology for the initial pose of the object are introduced. Step 4: Deploy the trained operation strategy network to a real robot platform.

2. The multi-finger dexterous hand manipulation reinforcement learning method based on a human action prediction model according to claim 1, characterized in that, The human motion prediction model includes a feature extraction network and a backbone network. The feature extraction network is used to extract the geometric features of the object. The backbone network takes the hand pose, the pose of the manipulated object and its geometric features at the historical time step as input, and simultaneously predicts the hand pose and the pose of the manipulated object at the future time step. The hand pose is defined by MANO parameters. 3.The multi-fingered hand operation reinforcement learning method based on human motion prediction model according to claim 2, wherein, The dataset processing steps used to train the human action prediction model include: Sequences containing the complete process of human hand manipulation of objects are selected from publicly available datasets. The sequences include hand pose parameters based on the MANO model and 6D pose of the object, and the sequences are uniformly divided into continuous action segments. Coordinate space normalization: A reference coordinate system is established based on the initial pose of the object, and the pose data of the hand and the object are uniformly transformed to the reference coordinate system; the 3D Mesh file of the object is read, and the Mesh vertices are first transformed to the world coordinate system using the pose of the object in frame 0, and then transformed to the reference coordinate system. The object point cloud is obtained through the farthest point sampling algorithm and the geometric features of the object are extracted based on the feature extraction network. Rotation representation continuation processing: The axis-angle representation in the human hand pose and the quaternion representation in the object pose in the reference coordinate system are converted into 6D rotation representation, and zero-padding is performed on the first M frames to generate a dataset for training the human motion prediction model, where M is the length of the historical time step input to the backbone network.

4. The multi-finger dexterous hand manipulation reinforcement learning method based on a human action prediction model according to claim 1, characterized in that, The process of converting the dexterous hand state into the optimal human hand MANO parameter representation includes: Obtain the state of the dexterous hand, which includes the three-dimensional coordinate observations of the joints of the corresponding human hand in the dexterous hand model; With the goal of minimizing the loss function of MANO parameter estimation, the three-dimensional coordinate observations are transformed into the optimal representation of the man's hand MANO parameters; The loss function for the MANO parameter estimation includes a joint coordinate alignment term and a posture regularization term. The joint coordinate alignment term refers to the error calculation between the three-dimensional coordinates of the human hand joints calculated based on the currently estimated MANO parameters and the three-dimensional coordinates of the corresponding joints of the dexterous hand observed from the simulator. The posture regularization term is a regularization constraint on the estimated MANO parameters.

5. The multi-finger dexterous hand manipulation reinforcement learning method based on a human action prediction model according to claim 1, characterized in that, The wrist-following reward function includes a joint angle tracking reward, a wrist end position tracking reward, and a wrist end rotation orientation tracking reward. The joint angle tracking bonus refers to the error calculation between the angles of each joint of the dexterous hand and the predicted reference joint angles. The wrist end position tracking reward refers to the error calculation between the position of the dexterous hand's wrist end in three-dimensional space and the predicted reference wrist position; The wrist end rotation orientation tracking bonus refers to the error calculation between the rotation orientation of the dexterous hand's wrist end and the predicted reference wrist rotation orientation. 6.The multi-fingered hand operation reinforcement learning method based on human motion prediction model according to claim 1, wherein, The object following reward function includes an object translation tracking reward and an object rotation tracking reward; The object translation tracking reward refers to the error calculation between the translation vector of the manipulated object in three-dimensional space and the predicted translation vector. The object rotation tracking bonus refers to the error calculation between the rotational orientation of the manipulated object in three-dimensional space and the predicted rotational orientation. 7.The multi-fingered hand operation reinforcement learning method based on human motion prediction model according to claim 1, wherein, The specific learning techniques for the initial pose of an object include: During reinforcement learning training, the success rate of the strategy is monitored in real time. When the success rate is below 30%, the initial pose randomization of the object is not applied. When the success rate is above 30%, the initial position offset of the object in the horizontal plane is gradually increased to a maximum of ±5cm, and the initial rotation range around the vertical axis is increased to a maximum of 360 degrees, according to the linear growth strategy. 8.The multi-fingered hand operation reinforcement learning method based on human motion prediction model according to claim 1, wherein, The simulation environment domain randomization technique specifically includes: Physical parameter noise injection: scaling the object collision mesh size from 0.96 to 1.04 times, scaling the object density from 0.50 to 2.00 times, scaling the object surface friction coefficient from 0.50 to 2.00 times, and scaling the object elastic coefficient from 0.90 to 1.10 times. Noise injection into observation data: Add ±0.5cm noise to the observed object position values ​​in the X and Y directions, and add noise around the z-axis. The noise.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the multi-finger dexterity hand operation reinforcement learning method based on a human action prediction model as described in any one of claims 1 to 8.

10. A computer electronic device, comprising: Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the multi-finger dexterity hand operation reinforcement learning method based on a human action prediction model as described in any one of claims 1 to 8.

Images