Functional grabbing method for five-finger mechanical dexterous hand based on progressive skill course learning

Abstract

The invention provides a five-finger mechanical dexterous hand functional grabbing method based on progressive skill course learning, which comprises the following steps of: controlling the actions of a dexterous hand in a simulation system by a human hand through a human hand posture detection and joint angle mapping module by utilizing a teleoperation teaching system, and grabbing functional parts of an object in the simulation system to obtain corresponding expert demonstration; defining different interaction tasks of the dexterous hand and the object according to the hand shape and the object category, and designing courses in different stages; starting progressive skill course learning, dividing a learning framework into primary, intermediate and advanced courses, and respectively obtaining a primary strategy set corresponding to a low-difficulty task, an intermediate strategy set corresponding to a medium-difficulty task and an advanced strategy corresponding to a high-difficulty task at the end of each stage; after progressive skill course learning, an advanced strategy grabs corresponding parts of different types of objects in correct hand shapes according to grabbed hand shape classification information and object point clouds in a direct execution or small-amount fine adjustment mode.

Description

Technical Field

[0001] The present invention relates to the technical field of five-finger manipulator operation, and in particular to a functional grasping method of a five-finger manipulator dexterous hand based on progressive skill course learning. Background Art

[0002] The research on dexterous robotic hands aims to develop multi-fingered robotic hands that can mimic the complex movements of human hands. Due to its high degree of freedom joint design and sophisticated sensor system, dexterous hands are able to perform flexible tasks in uncertain and dynamic environments. In the field of medical rehabilitation, they can be used to assist rehabilitation training or as prostheses to help patients restore hand function. In the manufacturing industry, such as complex assembly and precision operations, they can replace human hands to improve production efficiency and safety. There are also some service robots that perform complex grasping and manipulation tasks in scenarios such as housework, caring for the elderly and children.

[0003] Functional grasping means not only being able to grasp an object stably, but also preparing for the operation after grasping, such as placing the index finger on the handle of a watering can for use; dexterous hands have a high degree of freedom (24 to 30 degrees of freedom) and are difficult to control, which poses a great challenge to the control of dexterous functional grasping. Researchers at home and abroad have tried many research schemes on the learning of grasping strategies for dexterous hands. Some recent advances in reinforcement learning have demonstrated its ability to solve the high-dimensional challenges associated with dexterous hands. Reinforcement learning models dynamically utilize closed-loop feedback control based on visual and / or tactile perception to dynamically update strategies while accumulating new observation data. However, pure reinforcement learning faces challenges such as low sample efficiency and sparse rewards, especially in the case of high-dimensional dexterous hands. Combining reinforcement learning with imitation learning is an effective solution to these challenges, which can not only enable dexterous hands to imitate the actions of experts as much as possible, but also optimize and adjust the grasping posture through self-exploration to improve adaptability to changes in the task environment.

[0004] However, the above-mentioned dexterous grasping strategy learning methods usually adopt the method of learning a certain skill separately. When faced with new tasks, their generalization ability is insufficient and a large amount of data and time are required to learn from scratch. Summary of the invention

[0005] According to the technical problems raised above, a functional grasping method of a five-fingered mechanical dexterous hand based on progressive skill curriculum learning is proposed. The present invention combines imitation learning, reinforcement learning, meta-learning and curriculum learning techniques, uses human expert teleoperation technology, combines multi-task reinforcement learning based on hand shape classification, and designs a progressive skill curriculum learning framework. The framework gradually builds and enhances the robot's grasping skills when handling various objects through three stages of training: elementary, intermediate and advanced.

[0006] The technical means adopted by the present invention are as follows:

[0007] A functional grasping method for a five-finger dexterous mechanical hand based on progressive skill curriculum learning, comprising:

[0008] S1. Using the remote operation teaching system, through the human hand posture detection and joint angle mapping module, the human hand controls the movement of the dexterous hand in the simulation system, grasps the functional parts of the objects in the simulation system, obtains corresponding expert demonstrations, and prepares resources for course learning;

[0009] S2. Define different interactive tasks between dexterous hands and objects according to hand shapes and object categories, and design courses for different stages;

[0010] S3, start learning progressive skills courses. The learning framework is divided into three difficulty levels: elementary, intermediate and advanced courses. At the end of each stage, you will get a set of elementary strategies corresponding to low-difficulty tasks, an intermediate strategy set corresponding to medium-difficulty tasks, and an advanced strategy corresponding to high-difficulty tasks.

[0011] S4. After learning the progressive skill course, the advanced strategy grasps the corresponding parts of different categories of objects with the correct hand shape through direct execution or a small amount of fine-tuning based on the grasping hand shape classification information and object point cloud.

[0012] Furthermore, step S1 specifically includes:

[0013] S11. The remote operation teaching system includes two RGB cameras, one of which is placed at the front side of the operation space, and the other RGB camera is placed at the left side, providing a 60fps video stream input to obtain a video stream of real-time hand movements;

[0014] S12, using a human hand 2D posture detection model to detect a video stream of real-time hand movements to obtain a 2D hand posture detection result;

[0015] S13, using OpenCV to calibrate the two fixed RGB cameras, obtain their internal and external parameters and the conversion matrix between them, which are used to map the 2D gesture detection result of the human hand into the 3D space, and obtain the 3D gesture detection result of the human hand at each moment;

[0016] S14, establishing a coordinate system with the root joint of the middle finger as the origin, calculating the vectors between adjacent joints of the fingers, and then calculating the mapping angles of the angles between the vectors in different coordinate planes to obtain the rotation angles of the 15 joints of the human hand in different coordinate directions;

[0017] S15, using joint angle mapping to assign the human hand joint angle to the corresponding robot hand joint;

[0018] S16. Assign the relative change of the translation matrix T of the human hand in each frame to the manipulator, and convert the rotation matrix R into the Euler angle in the xyz direction and assign it to the manipulator. The remote-controlled manipulator follows the movement of the human hand and collects the expert teaching data.

[0019] Further, step S2 specifically includes:

[0020] S21. Design Elementary Course:

[0021] Each task is assigned to a primary policy, which learns to grasp an example object with a specific hand shape through a demonstration obtained by teleoperation;

[0022] S22, Intermediate Design Course:

[0023] Each task is assigned to an intermediate strategy, which uses the same grasping hand shape to grasp multiple objects of the same category, using the trajectories collected by the primary strategy exploration as demonstrations;

[0024] S23, Design Advanced Course:

[0025] A single high-level policy is trained using a meta-learning framework. The high-level policy grasps multiple object categories using different grasping hand shapes, and the mid-level policy explores the collected trajectories as demonstrations.

[0026] Further, step S3 specifically includes:

[0027] S31, Primary Course Learning Stage:

[0028] The method combines imitation learning and reinforcement learning. For each task, only one demonstration is used for behavior cloning, pre-training the primary strategy, and fine-tuning using the reinforcement learning algorithm.

[0029] S32, Intermediate Course Learning Stage:

[0030] Each intermediate strategy is an advanced strategy of the primary strategy, which successfully grasps similar parts of objects of the same category as the demonstration object with a single grasping hand shape;

[0031] To achieve the mid-level course learning goals, we expanded each interaction task from a single example object to 11 to 17 objects per category.

[0032] The primary strategy obtained from the primary stage training interacts online with the expanded new environment, namely the intermediate learning environment, to collect the grasping trajectory of the expanded object;

[0033] The object point cloud is encoded and embedded into the state space to enhance the understanding of the object's geometry;

[0034] During learning, we first use the primary strategy to explore the collected trajectories and clone the behaviors on the above 22 interactive tasks, and then use reinforcement learning for fine-tuning.

[0035] S33, advanced course learning stage:

[0036] Demonstrations in the high-level curriculum learning phase come from trajectories collected from online interactions between the intermediate policy and the environment; learning in the high-level curriculum learning phase employs a meta-learning algorithm, allowing the high-level policy to effectively handle a variety of objects and hand shapes after limited policy gradient updates.

[0037] Furthermore, step S31 specifically includes:

[0038] S311. Setting the state space of the primary course learning stage: The complete state space of each primary strategy contains three pieces of information, namely, the robot proprioception information {R}, the object posture information {O}, and the position coordinates of the five fingertips in the object coordinate system {F};

[0039] S312, setting the action space for the primary course learning stage: the action space consists of 30-dimensional motor control commands, expressed as the translation and rotation of the 6-DOF arm, and the 24-dimensional joint angles of the dexterous hand;

[0040] S313. Design the reward function for the primary course learning stage as follows:

[0041] r=r reach +r contact +r lift

[0042] Among them, r reach represents the reward function for approaching an object during the primary course learning phase, r reach =-λ r ∑||x finger -x coa ||2,λ r represents the reward r assigned to the object close to it reach The weight of x finger Indicates the position coordinates of the five fingertips in the object coordinate system, x coa Indicates the geometric center point of the functional part of the object; r contact represents the contact object reward function during the primary course learning stage, λ c represents the weight assigned to each contact point, c i Indicates whether the tactile sensor on each fingertip feels force, if yes, it is 1, otherwise it is 0, i represents the index of the five fingertips; r lift represents the reward function for grabbing objects during the primary course learning phase, r lift =λl max(0, z current -z initial ), λ l represents the weight assigned to the reward for grabbing the object, z current Indicates the z-axis coordinate of the current object's center of mass, z initial Represents the z-axis coordinate of the center of mass of the object in the initial state;

[0043] S314. Calculate the policy gradient in the primary course learning stage. The calculation formula is as follows:

[0044]

[0045] in, represents the advantage function of evaluating taking action a in state s relative to the primary strategy; represents the set of state-action pairs collected by the primary agent; represents the primary strategy; λ0 and λ1 represent the hyperparameters for adjusting the contribution of different gradient calculation items; k represents the number of iterations; It means to find the gradient of parameter θ. Represents the set of human expert state-action pairs collected during teleoperation.

[0046] Further, step S32 specifically includes:

[0047] S321. Setting the state space of the intermediate course learning stage: adding object point cloud encoding {P} to the primary strategy state space;

[0048] S322, setting the action space for the intermediate course learning stage: the action space consists of 30-dimensional motor control commands, expressed as translation and rotation of the 6-DOF arm, and 24-dimensional joint angles of the dexterous hand;

[0049] S323. Design the reward function for the intermediate course learning stage as follows:

[0050] r=r reach +r contact +r lift

[0051] Among them, r reach represents the reward function for approaching an object during the intermediate course learning phase, r reach =-λ r ∑||x finger -x coa ‖2,λ r represents the reward r assigned to the object close to it reach The weight of x finger Indicates the position coordinates of the five fingertips in the object coordinate system, x coaIndicates the geometric center point of the functional part of the object; r contact represents the contact object reward function during the intermediate course learning stage, λ c represents the weight assigned to each contact point, c i Indicates whether the tactile sensor on each fingertip feels force, if yes, it is 1, otherwise it is 0, i represents the index of the five fingertips; r lift represents the reward function for grasping objects during the intermediate course learning phase, r lift =λ l max(0, z current -z initial ), λ l represents the weight assigned to the reward for grabbing the object, z current Indicates the z-axis coordinate of the current object's center of mass, z initial Represents the z-axis coordinate of the center of mass of the object in the initial state;

[0052] S324. Calculate the policy gradient in the intermediate course learning stage. The calculation formula is as follows:

[0053]

[0054] in, represents the advantage function of evaluating taking action a in state s relative to the intermediate strategy; and denote the set of state-action pairs collected by the primary and intermediate agents, respectively; W k (s, a) represents the negative normalized log-likelihood, which is used to encourage the policy to learn trajectories that are difficult to replicate; represents the advantage function implemented by the multilayer perceptron, λ0, λ1, λ′0 represent the hyperparameters for adjusting the contribution of different gradient calculation terms; k represents the number of iterations.

[0055] Further, step S33 specifically includes:

[0056] S331, Task Settings:

[0057] Defining Task Distribution Each is a Markov decision process of length H; the strategy model f θ Expressed as π(a|s), the state s t Mapped to an action a at each time step t∈{1, 2, 3, ..., H} t distribution; in each task In , the goals of policy learning are as follows:

[0058]

[0059] Setting the state space for the advanced course learning stage: Adding the grasping hand type encoding {T} to the intermediate strategy state space, using one-hot encoding to help the agent distinguish and perform appropriate grasping actions;

[0060] Setting the action space for the advanced course learning phase: The action space consists of 30-dimensional motor control commands, represented as 6-DOF translation and rotation of the arm, and 24-dimensional joint angles of the dexterous hand;

[0061] Design the reward function for the advanced course learning stage as follows:

[0062] r=r contact +r lift

[0063] Among them, r contact represents the contact object reward function during the high-level curriculum learning stage, λ c represents the weight assigned to each contact point, c i Indicates whether the tactile sensor on each fingertip feels force, if yes, it is 1, otherwise it is 0, i represents the index of the five fingertips; r lift represents the reward function for grasping objects during the high-level course learning phase, r lift =λ l max(0, z current -z initial ), λ l represents the weight assigned to the reward for grabbing the object, z current Indicates the z-axis coordinate of the current object's center of mass, z initial Represents the z-axis coordinate of the center of mass of the object in the initial state;

[0064] S332, Behavior cloning pre-training:

[0065] Use the trajectories collected by the intermediate strategy to complete the behavior cloning and obtain the initial high-level strategy;

[0066] S333, Strategy training based on meta-learning:

[0067] Training includes internal and external cycles;

[0068] S334, Reasoning for new tasks:

[0069] Use the AffordPose dataset to train a grasping hand shape classification model. Input a video of a person grasping an object, output the grasping hand shape classification result, and embed it into the state space in the form of one-hot encoding.

[0070] Use the point cloud encoder to encode the object and embed the encoded features into the state space;

[0071] The high-level policy model trained in the meta-learning phase achieves successful grasping through zero or several gradient updates; in this process, the state space, action space, and reward function are consistent with the meta-learning phase, but the policy gradient update formula is changed to:

[0072]

[0073] Furthermore, the inner loop and the outer loop in step S333 specifically include:

[0074] Inner loop:

[0075] First, from the task distribution The sampling task, and then the high-level strategy currently being trained Interact with the environment and collect k training trajectories Using training tracks and the trajectories collected by the intermediate strategy Calculate the policy gradient g train , the calculation formula is as follows:

[0076]

[0077] After the inner loop, the strategy parameter update formula is as follows:

[0078] θ i =θ+αgtrain

[0079] Strategy after the inner cycle On Task Interact with the environment to generate verification trajectories

[0080] Outer loop:

[0081] In the outer loop, by Calculate the policy gradient g valid , the formula is as follows:

[0082]

[0083] The optimized advanced strategy parameter update formula is as follows:

[0084] θ=θ+βg valid

[0085] Get optimized strategy This process was optimized using the TRPO method.

[0086] Compared with the prior art, the present invention has the following advantages:

[0087] 1. The present invention defines the dexterous functional grasping task as a multi-task problem according to the grasping hand type classification, and develops a progressive skill course learning framework from simple to complex according to the task difficulty, so that the dexterous hand can gradually acquire functional grasping skills that can adapt to changes in the task environment. The learning process is divided into three stages: elementary, intermediate and advanced courses, so that the robot can learn an advanced strategy from multi-task teaching data, integrate multiple skills, and complete the corresponding functional grasping task directly or after a small amount of fine-tuning through a hand type classification information provided by a demonstration related to the new task.

[0088] 2. The dexterous hand functional grasping strategy learning method developed by the present invention includes a remote operation teaching system and a progressive skill course learning framework. After training, the dexterous hand can use a general strategy to complete the functional grasping of multiple hand shapes and multiple categories of objects with a high success rate.

[0089] 3. The present invention provides an advanced strategy for dexterous grasping obtained by combining imitation learning, reinforcement learning, meta-learning and curriculum learning, so that the robot dexterous hand can flexibly and effectively perform complex functional grasping tasks in different task environments, and when faced with new tasks, it can instantly or with a small amount of fine-tuning complete the corresponding functional grasping using only one piece of hand shape classification information provided by teaching.

[0090] Based on the above reasons, the present invention can be widely promoted in the fields of five-finger manipulator operation and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0092] Figure 1 The figure is a flowchart of the method of the present invention.

[0093] Figure 2 A flow chart of the primary course learning phase provided by an embodiment of the present invention.

[0094] Figure 3 A flow chart of the intermediate course learning phase provided by an embodiment of the present invention.

[0095] Figure 4 The demonstration of the advanced course learning stage provided for the embodiment of the present invention comes from the traces collected from the online interaction between the intermediate strategy and the environment.

[0096] Figure 5The high-level strategy provided for the embodiment of the present invention can quickly complete the multi-hand functional grasping simulation diagram. DETAILED DESCRIPTION

[0097] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0098] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0099] like Figure 1 As shown, the present invention provides a functional grasping method of a five-finger mechanical dexterous hand based on progressive skill course learning, comprising:

[0100] S1. Using the remote operation teaching system, through the human hand posture detection and joint angle mapping module, the human hand controls the movement of the dexterous hand in the simulation system, grasps the functional parts of the objects in the simulation system, obtains corresponding expert demonstrations, and prepares resources for course learning;

[0101] S2. Define different interactive tasks between dexterous hands and objects according to hand shapes and object categories, and design courses for different stages;

[0102] In this embodiment, tasks are first classified according to the grasping hand shape. According to the fine grasping division given in the AffordPose dataset, five functional grasping hand shapes are selected: twisting, cylindrical grip, operating handle, finger pressing and hooking. 22 interactive tasks are designed around these five hand shapes and ten types of objects, and a grasping demonstration trajectory is collected for each task.

[0103] S3, start learning progressive skills courses. The learning framework is divided into three difficulty levels: elementary, intermediate and advanced courses. At the end of each stage, you will get a set of elementary strategies corresponding to low-difficulty tasks, an intermediate strategy set corresponding to medium-difficulty tasks, and an advanced strategy corresponding to high-difficulty tasks.

[0104] S4. After learning the progressive skill course, the advanced strategy grasps the corresponding parts of different categories of objects with the correct hand shape through direct execution or a small amount of fine-tuning based on the grasping hand shape classification information and object point cloud.

[0105] In specific implementation, as a preferred embodiment of the present invention, step S1 specifically includes:

[0106] S11. The remote operation teaching system includes two RGB cameras, one of which is placed at the front side of the operation space, and the other RGB camera is placed at the left side, providing a 60fps video stream input to obtain a video stream of real-time hand movements;

[0107] S12, using a human hand 2D posture detection model to detect a video stream of real-time hand movements to obtain a 2D hand posture detection result;

[0108] S13, using OpenCV to calibrate the two fixed RGB cameras, obtain their internal and external parameters and the transformation matrix between them, which are used to map the 2D gesture detection results of the human hand into the 3D space, and obtain the 3D gesture detection results of the human hand at each moment, that is, the 3D coordinates of 21 key points;

[0109] S14, establishing a coordinate system with the root joint of the middle finger as the origin, calculating the vectors between adjacent joints of the fingers, and then calculating the mapping angles of the angles between the vectors in different coordinate planes to obtain the rotation angles of the 15 joints of the human hand in different coordinate directions;

[0110] S15, using joint angle mapping to assign the human hand joint angle to the corresponding robot hand joint;

[0111] S16. Assign the relative change of the translation matrix T of the human hand in each frame to the manipulator, and convert the rotation matrix R into the Euler angle in the xyz direction and assign it to the manipulator. The remote-controlled manipulator follows the movement of the human hand and collects the expert teaching data.

[0112] In specific implementation, as a preferred embodiment of the present invention, step S2 specifically includes:

[0113] S21. Design Elementary Course:

[0114] Each task is assigned to a primary policy, which learns to grasp an example object with a specific hand shape through a demonstration obtained by teleoperation;

[0115] S22, Intermediate Design Course:

[0116] Each task is assigned to an intermediate strategy, which uses the same grasping hand shape to grasp multiple objects of the same category, using the trajectories collected by the primary strategy exploration as demonstrations;

[0117] S23, Design Advanced Course:

[0118] A single high-level policy is trained using a meta-learning framework. The high-level policy grasps multiple object categories using different grasping hand shapes, and the mid-level policy explores the collected trajectories as demonstrations.

[0119] In specific implementation, as a preferred embodiment of the present invention, step S3 specifically includes:

[0120] S31, Primary Course Learning Stage:

[0121] Using a combination of imitation learning and reinforcement learning, such as Figure 2 As shown, for each task, only one demonstration is used for behavior cloning, pre-training the primary policy, and fine-tuning using the reinforcement learning algorithm;

[0122] S32, Intermediate Course Learning Stage:

[0123] Each intermediate strategy is an advanced strategy of the primary strategy, which successfully grasps similar parts of objects of the same category as the demonstration object with a single grasping hand shape; Figure 3 As shown, the process of learning the intermediate course is demonstrated.

[0124] To achieve the mid-level course learning goals, we expanded each interaction task from a single example object to 11 to 17 objects per category.

[0125] The primary strategy obtained from the primary stage training interacts online with the expanded new environment, namely the intermediate learning environment, to collect the grasping trajectory of the expanded object;

[0126] The object point cloud is encoded and embedded into the state space to enhance the understanding of the object's geometry;

[0127] During learning, we first use the primary strategy to explore the collected trajectories and clone the behaviors on the above 22 interactive tasks, and then use reinforcement learning for fine-tuning.

[0128] S33, advanced course learning stage:

[0129] like Figure 4 As shown, the demonstrations of the high-level curriculum learning stage come from trajectories collected from online interactions between the intermediate policy and the environment; the learning in the high-level curriculum learning stage adopts a meta-learning algorithm, allowing the high-level policy to effectively handle various objects and hand shapes after limited policy gradient updates.

[0130] In specific implementation, as a preferred embodiment of the present invention, step S31 specifically includes:

[0131] S311. Setting the state space of the primary course learning stage: The complete state space of each primary strategy contains three pieces of information, namely, the robot proprioception information {R}, the object posture information {O}, and the position coordinates of the five fingertips in the object coordinate system {F};

[0132] S312, setting the action space for the primary course learning stage: the action space consists of 30-dimensional motor control commands, expressed as the translation and rotation of the 6-DOF arm, and the 24-dimensional joint angles of the dexterous hand;

[0133] S313. Design the reward function for the primary course learning stage as follows:

[0134] r=r reach +r contact +r lift

[0135] Among them, r reach represents the reward function for approaching an object during the primary course learning phase, r reach =-λ r ∑||x finger -x coa ||2,λ r represents the reward r assigned to the object close to it reach The weight of x is set to 0.5 in this embodiment. finger Indicates the position coordinates of the five fingertips in the object coordinate system, x coa Indicates the geometric center point of the functional part of the object; r contact represents the contact object reward function during the primary course learning stage, λ c represents the weight assigned to each contact point. In this embodiment, it is set to 0.2. i Indicates whether the tactile sensor on each fingertip feels force, if yes, it is 1, otherwise it is 0, i represents the index of the five fingertips; r lift represents the reward function for grabbing objects during the primary course learning phase, r lift =λ l max(0, z current -z initial ), λ l represents the weight assigned to the reward for grabbing the object, z current Indicates the z-axis coordinate of the current object's center of mass, z initial Represents the z-axis coordinate of the center of mass of the object in the initial state;

[0136] S314. Calculate the policy gradient in the primary course learning stage. The calculation formula is as follows:

[0137]

[0138] in, represents the advantage function of evaluating taking action a in state s relative to the primary strategy; represents the set of state-action pairs collected by the primary agent; represents the state-action pairs collected by the primary agent; λ0, Represents the hyperparameters that adjust the contribution of different gradient calculation terms; It means to find the gradient of parameter θ. Represents the set of human expert state-action pairs collected during teleoperation.

[0139] In specific implementation, as a preferred embodiment of the present invention, step S32 specifically includes:

[0140] S321. Setting the state space of the intermediate course learning stage: adding object point cloud coding {P} to the primary strategy state space;

[0141] S322, setting the action space for the intermediate course learning stage: the action space consists of 30-dimensional motor control commands, expressed as translation and rotation of the 6-DOF arm, and 24-dimensional joint angles of the dexterous hand;

[0142] S323. Design the reward function for the intermediate course learning stage as follows:

[0143] r=r reach +r contact +r lift

[0144] Among them, r reach represents the reward function for approaching an object during the intermediate course learning phase, r reach =-λ r ∑||x finger -x coa ||2,λ r represents the reward r assigned to the object close to it reach The weight of x is set to 0.5 in this embodiment. finger Indicates the position coordinates of the five fingertips in the object coordinate system, x coa Indicates the geometric center point of the functional part of the object; r contact represents the contact object reward function during the primary course learning stage, λ c represents the weight assigned to each contact point. In this embodiment, it is set to 0.2. iIndicates whether the tactile sensor on each fingertip feels force, if yes, it is 1, otherwise it is 0, i represents the index of the five fingertips; r lift represents the reward function for grabbing objects during the primary course learning phase, r lift =λ l max(0, z current -z initial ), λ l represents the weight assigned to the reward for grabbing the object, z current Indicates the z-axis coordinate of the current object's center of mass, z initial Represents the z-axis coordinate of the center of mass of the object in the initial state;

[0145] S324. Calculate the policy gradient in the intermediate course learning stage. The calculation formula is as follows:

[0146]

[0147] in, represents the advantage function of evaluating taking action a in state s relative to the intermediate strategy; and denote the set of state-action pairs collected by the primary and intermediate agents, respectively; W k (s, a) represents the negative normalized log-likelihood, which is used to encourage the policy to learn trajectories that are difficult to replicate; represents the advantage function implemented by the multilayer perceptron, λ0, λ1, λ′0 represent the hyperparameters for adjusting the contribution of different gradient calculation terms; k is the number of iterations.

[0148] In specific implementation, as a preferred embodiment of the present invention, step S33 specifically includes:

[0149] S331, Task Settings:

[0150] Defining Task Distribution Each is a Markov decision process of length H; the strategy model f θ Expressed as π(a|s), the state s t Mapped to an action a at each time step t∈{1, 2, 3, ..., H} t distribution; in each task In , the goals of policy learning are as follows:

[0151]

[0152] Setting the state space for the advanced course learning stage: Adding the grasping hand type encoding {T} to the intermediate strategy state space, using one-hot encoding to help the agent distinguish and perform appropriate grasping actions;

[0153] Setting the action space for the advanced course learning phase: The action space consists of 30-dimensional motor control commands, represented as 6-DOF translation and rotation of the arm, and 24-dimensional joint angles of the dexterous hand;

[0154] Design the reward function for the advanced course learning stage as follows:

[0155] r=r contact +r lift

[0156] Among them, r contact represents the contact object reward function during the high-level curriculum learning stage, λ c represents the weight assigned to each contact point. In this embodiment, it is set to 0.2. i Indicates whether the tactile sensor on each fingertip feels force, if yes, it is 1, otherwise it is 0, i represents the index of the five fingertips; r lift represents the reward function for grabbing objects during the primary course learning phase, r lift =λ l max(0, z current -z initial ), λ l represents the weight assigned to the reward for grabbing the object, z current Indicates the z-axis coordinate of the current object's center of mass, z initial Represents the z-axis coordinate of the center of mass of the object in the initial state;

[0157] S332, Behavior cloning pre-training:

[0158] Use the trajectories collected by the intermediate strategy to complete the behavior cloning and obtain the initial high-level strategy;

[0159] S333, Strategy training based on meta-learning:

[0160] Training includes internal and external cycles;

[0161] S334, Reasoning for new tasks:

[0162] Use the AffordPose dataset to train a grasping hand shape classification model. Input a video of a person grasping an object, output the grasping hand shape classification result, and embed it into the state space in the form of one-hot encoding.

[0163] Use the point cloud encoder to encode the object and embed the encoded features into the state space;

[0164] The high-level policy model trained in the meta-learning phase achieves successful grasping through zero or several gradient updates; in this process, the state space, action space, and reward function are consistent with the meta-learning phase, but the policy gradient update formula is changed to:

[0165]

[0166] In specific implementation, as a preferred embodiment of the present invention, the inner loop and the outer loop in step S333 specifically include:

[0167] Inner loop:

[0168] First, from the task distribution The sampling task, and then the high-level strategy currently being trained Interact with the environment and collect k training trajectories Using training tracks and the trajectories collected by the intermediate strategy Calculate the policy gradient g train , the calculation formula is as follows:

[0169]

[0170] After the inner loop, the strategy parameter update formula is as follows:

[0171] θ i =θ+αg train

[0172] Strategy after the inner cycle On Task Interact with the environment to generate verification trajectories

[0173] Outer loop:

[0174] In the outer loop, by Calculate the policy gradient g valid , the formula is as follows:

[0175]

[0176] The optimized advanced strategy parameter update formula is as follows:

[0177] θ=θ+βg valid

[0178] Get optimized strategy This process was optimized using the TRPO method.

[0179] Example

[0180] Evaluation Metrics:

[0181] The grasping success rate is used to evaluate the performance of the method proposed in this invention.

[0182] Evaluation Methodology:

[0183] The test was conducted under 5 random seeds. Each object was initialized at a random position and grasped 100 times. A successful grasp was determined if the grasp lasted for more than 2 seconds and was 10 cm high. The average success rate was obtained.

[0184] Evaluation results:

[0185] Five ILAD grasping strategies and the advanced grasping strategy obtained by the present invention were tested and compared on five object categories of the ShapeNet dataset. Since only brute force grasping for the purpose of picking up and placing is considered in the ILAD method, only the experimental results of the advanced strategy of the present invention when executing the cylindrical handshake shape are compared. The grasping success rate is shown in Table 1. The first row indicates that the corresponding ILAD strategy trained separately is used for each object category to grasp objects of the same category. The success rate of each object category is very high, and the average success rate of all object categories is 0.93. If the ILAD strategy is trained on one object category but tested on other object categories, the success rate will be significantly reduced. The grasping results of the advanced strategy of the present invention are listed in the last row, and the success rate on each object category is higher than that of the ILAD method.

[0186] Table 1 Comparison of success rates with ILAD on five types of unseen objects on ShapeNet

[0187]

[0188] The grasping success rate is compared with Graff, a functional grasping strategy learning method for dexterous hands, on the 3DNet dataset. According to the information provided by the Graff method, 9 categories of objects that have not been seen in the 3DNet dataset are also selected as test sets. 2 to 4 instances are selected for each category, for a total of 25 new objects. Table 2 shows the comparison of the success rates of the present invention and the Graff method on the 3DNet dataset. The grasping strategy obtained by the present invention is significantly better than the grasping strategy obtained by the Graff method. Figure 5 It is demonstrated that the strategy obtained by the present invention not only allows grasping these categories of objects with a single strategy, but also supports multiple interaction modes, such as grasping the handle, mouth and body of a mug.

[0189] Table 2. Comparison of success rates with Graff on 9 types of unseen objects on 3DNet

[0190]

[0191] In summary, the present invention proposes a novel functional grasping learning framework for high-degree-of-freedom multi-finger dexterous robot hands, namely, a progressive skill curriculum learning framework, which enables advanced agents to acquire generalized strategies that can adapt to a variety of situations by increasing the difficulty of tasks in stages. The present invention designs multiple primary and intermediate agents to learn basic grasping skills based on the grasping hand shape and objects of human hands, and integrates the skills by advanced agents to achieve multifunctional anthropomorphic grasping based on a single demonstration.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A functional grasping method for a five-finger mechanical dexterous hand based on progressive skill course learning, characterized in that: include: S1. Using the remote operation teaching system, through the human hand posture detection and joint angle mapping module, the human hand controls the movement of the dexterous hand in the simulation system, grasps the functional parts of the objects in the simulation system, obtains corresponding expert demonstrations, and prepares resources for course learning; S2. Define different interactive tasks between dexterous hands and objects according to hand shapes and object categories, and design courses for different stages; S3, start learning progressive skills courses. The learning framework is divided into three difficulty levels: elementary, intermediate and advanced courses. At the end of each stage, you will get a set of elementary strategies corresponding to low-difficulty tasks, an intermediate strategy set corresponding to medium-difficulty tasks, and an advanced strategy corresponding to high-difficulty tasks. S4. After learning the progressive skill course, the advanced strategy grasps the corresponding parts of different categories of objects with the correct hand shape through direct execution or a small amount of fine-tuning based on the grasping hand shape classification information and object point cloud.

2. The functional grasping method of a five-finger mechanical dexterous hand based on progressive skill course learning according to claim 1, characterized in that: Step S1 specifically includes: S11. The remote operation teaching system includes two RGB cameras, one of which is placed at the front side of the operation space, and the other RGB camera is placed at the left side, providing a 60fps video stream input to obtain a video stream of real-time hand movements; S12, using a human hand 2D posture detection model to detect a video stream of real-time hand movements to obtain a 2D hand posture detection result; S13, using OpenCV to calibrate the two fixed RGB cameras, obtain their internal and external parameters and the conversion matrix between them, which are used to map the 2D gesture detection result of the human hand into the 3D space, and obtain the 3D gesture detection result of the human hand at each moment; S14, establishing a coordinate system with the root joint of the middle finger as the origin, calculating the vectors between adjacent joints of the fingers, and then calculating the mapping angles of the angles between the vectors in different coordinate planes to obtain the rotation angles of the 15 joints of the human hand in different coordinate directions; S15, using joint angle mapping to assign the human hand joint angle to the corresponding robot hand joint; S16. Assign the relative change of the translation matrix T of the human hand in each frame to the manipulator, and convert the rotation matrix R into the Euler angle in the xyz direction and assign it to the manipulator. The remote-controlled manipulator follows the movement of the human hand and collects the expert teaching data.

3. The functional grasping method of a five-finger mechanical dexterous hand based on progressive skill course learning according to claim 1 is characterized in that: Step S2 specifically includes: S21. Design Elementary Course: Each task is assigned to a primary policy, which learns to grasp an example object with a specific hand shape through a demonstration obtained by teleoperation; S22, Intermediate Design Course: Each task is assigned to an intermediate strategy, which uses the same grasping hand shape to grasp multiple objects of the same category, using the trajectories collected by the primary strategy exploration as demonstrations; S23, Design Advanced Course: A single high-level policy is trained using a meta-learning framework. The high-level policy grasps multiple object categories using different grasping hand shapes, and the mid-level policy explores the collected trajectories as demonstrations.

4. The functional grasping method of a five-finger mechanical dexterous hand based on progressive skill course learning according to claim 1, characterized in that: Step S3 specifically includes: S31, Primary Course Learning Stage: The method combines imitation learning and reinforcement learning. For each task, only one demonstration is used for behavior cloning, pre-training the primary strategy, and fine-tuning using the reinforcement learning algorithm. S32, Intermediate Course Learning Stage: Each intermediate strategy is an advanced strategy of the primary strategy, which successfully grasps similar parts of objects of the same category as the demonstration object with a single grasping hand shape; To achieve the mid-level course learning goals, we expanded each interaction task from a single example object to 11 to 17 objects per category. The primary strategy obtained from the primary stage training interacts online with the expanded new environment, namely the intermediate learning environment, to collect the grasping trajectory of the expanded object; The object point cloud is encoded and embedded into the state space to enhance the understanding of the object's geometry; During learning, we first use the primary strategy to explore the collected trajectories to clone the behaviors on the above 22 interactive tasks, and then use reinforcement learning for fine-tuning. S33, advanced course learning stage: Demonstrations in the high-level curriculum learning phase come from trajectories collected from online interactions between the intermediate policy and the environment; learning in the high-level curriculum learning phase employs a meta-learning algorithm, allowing the high-level policy to effectively handle a variety of objects and hand shapes after limited policy gradient updates.

5. The functional grasping method of a five-finger mechanical dexterous hand based on progressive skill course learning according to claim 4 is characterized in that: Step S31 specifically includes: S311. Setting the state space of the primary course learning stage: The complete state space of each primary strategy contains three pieces of information, namely, the robot proprioception information {R}, the object posture information {O}, and the position coordinates of the five fingertips in the object coordinate system {F}; S312, setting the action space for the primary course learning stage: the action space consists of 30-dimensional motor control commands, expressed as the translation and rotation of the 6-DOF arm, and the 24-dimensional joint angles of the dexterous hand; S313. Design the reward function for the primary course learning stage as follows: r＝r reach +r contact +r lift Among them, r reach represents the reward function for approaching an object during the primary course learning phase, r reach =-λ r ∑||x finger -x coa ||2,λ r represents the reward r assigned to the object close to it reach The weight of x finger Indicates the position coordinates of the five fingertips in the object coordinate system, x coa Indicates the geometric center position of the functional part of the object; r contact represents the contact object reward function during the primary course learning stage, λ c represents the weight assigned to each contact point, c i Indicates whether the tactile sensor on each fingertip feels force, if yes, it is 1, otherwise it is 0, i represents the index of the five fingertips; r lift represents the reward function for grabbing objects during the primary course learning phase, r lif t=λ l max(0, z current -z initial ), λ l represents the weight assigned to the reward for grabbing the object, z current Indicates the z-axis coordinate of the current object's center of mass, z initial Represents the z-axis coordinate of the center of mass of the object in the initial state; S314. Calculate the policy gradient in the primary course learning stage. The calculation formula is as follows: in, represents the advantage function of evaluating taking action a in state s relative to the primary strategy; represents the set of state-action pairs collected by the primary agent; represents the primary strategy; λ0 and λ1 represent the hyperparameters for adjusting the contribution of different gradient calculation items; k represents the number of iterations; It means to find the gradient of parameter θ. Represents the set of human expert state-action pairs collected during teleoperation.

6. The functional grasping method of a five-finger mechanical dexterous hand based on progressive skill course learning according to claim 4, characterized in that: Step S32 specifically includes: S321. Setting the state space of the intermediate course learning stage: adding object point cloud encoding {P} to the primary strategy state space; S322, setting the action space for the intermediate course learning stage: the action space consists of 30-dimensional motor control commands, expressed as translation and rotation of the 6-DOF arm, and 24-dimensional joint angles of the dexterous hand; S323. Design the reward function for the intermediate course learning stage as follows: r＝r reach +r contact +r lift Among them, r reach represents the reward function for approaching an object during the intermediate course learning phase, r reach =-λ r ∑||x finger -x coa ||2,λ r represents the reward r assigned to the object close to it reach The weight of x finger Indicates the position coordinates of the five fingertips in the object coordinate system, x coa Indicates the geometric center position of the functional part of the object; r contact represents the contact object reward function during the intermediate course learning stage, λ c represents the weight assigned to each contact point, c i Indicates whether the tactile sensor on each fingertip feels force, if yes, it is 1, otherwise it is 0, i represents the index of the five fingertips; r lift represents the reward function for grasping objects during the intermediate course learning phase, r lift =λ l max(0, z current -z initial ), λ l represents the weight assigned to the reward for grabbing the object, z current Indicates the z-axis coordinate of the current object's center of mass, z initial Represents the z-axis coordinate of the center of mass of the object in the initial state; S324. Calculate the policy gradient in the intermediate course learning stage. The calculation formula is as follows: in, represents the advantage function of evaluating taking action a in state s relative to the intermediate strategy; and denote the sets of state-action pairs collected by the primary and intermediate agents, respectively; w k (s, a) represents the negative normalized log-likelihood, which is used to encourage the policy to learn trajectories that are difficult to replicate; represents the advantage function implemented by the multilayer perceptron, λ0, λ1, λ′0 represent the hyperparameters for adjusting the contribution of different gradient calculation terms; k represents the number of iterations.

7. The functional grasping method of a five-finger mechanical dexterous hand based on progressive skill course learning according to claim 4, characterized in that: Step S33 specifically includes: S331, Task Settings: Defining Task Distribution Each is a Markov decision process of length H; the strategy model f θ Expressed as π(a|s), the state s t Mapped to an action a at each time step t∈{1, 2, 3, ..., H} t distribution; in each task In , the goals of policy learning are as follows: Setting the state space for the advanced course learning stage: Adding the grasping hand type encoding {T} to the intermediate strategy state space, using one-hot encoding to help the agent distinguish and perform appropriate grasping actions; Setting the action space for the advanced course learning phase: The action space consists of 30-dimensional motor control commands, represented as 6-DOF translation and rotation of the arm, and 24-dimensional joint angles of the dexterous hand; Design the reward function for the advanced course learning stage as follows: r＝r contact +r lift Among them, r contact represents the contact object reward function during the high-level curriculum learning stage, λ c represents the weight assigned to each contact point, c i Indicates whether the tactile sensor on each fingertip feels force, if yes, it is 1, otherwise it is 0, i represents the index of the five fingertips; r lift represents the reward function for grasping objects during the high-level course learning phase, r lift =λ l max(0, z current -z initial ), λ l represents the weight assigned to the reward for grabbing the object, z current Indicates the z-axis coordinate of the current object's center of mass, z initial Represents the z-axis coordinate of the center of mass of the object in the initial state; S332, Behavior cloning pre-training: Use the trajectories collected by the intermediate strategy to complete the behavior cloning and obtain the initial high-level strategy; S333, Strategy training based on meta-learning: Training includes internal and external cycles; S334, Reasoning for new tasks: Using AffordPos e The dataset trains a grasping hand shape classification model, inputs a video of a person grasping an object, and outputs the classification result of the grasping hand shape. e -hot encoding form is embedded into the state space; Use the point cloud encoder to encode the object and embed the encoded features into the state space; The high-level policy model trained in the meta-learning phase achieves successful grasping through zero or several gradient updates. In this process, the state space, action space, and reward function are the same as those in the meta-learning phase, but the policy gradient update formula is changed to:

8. The functional grasping method of a five-finger mechanical dexterous hand based on progressive skill course learning according to claim 7, characterized in that: The inner loop and outer loop in step S333 specifically include: Inner loop: First, from the task distribution The sampling task, and then the high-level strategy currently being trained Interact with the environment and collect k training trajectories Using training tracks and the trajectories collected by the intermediate strategy Calculate the policy gradient g train , the calculation formula is as follows: After the inner loop, the strategy parameter update formula is as follows: θ′ i =θ+αg train Strategy after the inner loop On Task Interact with the environment to generate verification trajectories Outer loop: In the outer loop, by Calculate the policy gradient g valid , the formula is as follows: The optimized advanced strategy parameter update formula is as follows: θ′=θ+βg valid Get optimized strategy This process was optimized using the TRPO method.

Images