A functional grasping method of five-finger mechanical dexterous hand based on progressive skill course learning

Abstract

The application provides a five-finger mechanical dexterous hand functional grasping method based on progressive skill course learning, comprising: using a teleoperation teaching system, using a human hand posture detection and joint angle mapping module to control the action of a dexterous hand in a human hand control simulation system, grasping the functional part of an object in the simulation system, and obtaining the corresponding expert demonstration; defining different interactive tasks of the dexterous hand and the object according to the hand shape and the object category, and designing courses in different stages; starting the progressive skill course learning, and dividing the learning framework into primary, intermediate and advanced courses; at the end of each stage, a primary strategy set corresponding to a low difficulty task, an intermediate strategy set corresponding to a medium difficulty task and an advanced strategy corresponding to a high difficulty task are obtained, respectively; after the progressive skill course learning, the advanced strategy directly executes or slightly adjusts the way according to the grasping hand shape classification information and the object point cloud, so as to correctly grasp the corresponding part of the object of different categories with the correct hand shape.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of five-fingered robot hand operation, in particular, and especially relates to a five-fingered robot hand functional grasping method based on progressive skill curriculum learning. BACKGROUND

[0002] The research of robot dexterous hand aims to develop a multi-fingered robot hand that can mimic the complex motions of human hand. Due to its high degree of freedom joint design and precise sensor system, dexterous hand can perform flexible tasks in uncertain and dynamic environments. In the field of medical rehabilitation, it can be used to assist rehabilitation training or as a prosthesis to help patients recover hand function. In manufacturing, such as complex assembly and precision operation, it can replace human hand to improve production efficiency and safety. There are also some service robots that perform complex grasping and manipulation tasks in scenarios such as household chores, taking care of the elderly and children, etc.

[0003] Functional grasping not only means to stably grasp the object, but also to prepare for the operation after grasping, for example, to place the index finger on the handle of the spray bottle for use. Dexterous hand has very high degree of freedom (24 to 30 degrees of freedom), which is difficult to control, bringing great challenges to dexterous functional grasping control. Researchers at home and abroad have tried many research schemes on dexterous hand grasping strategy learning. Some recent developments in reinforcement learning have proved that it can solve the high-dimensional challenges related to dexterous hand. Reinforcement learning model dynamically uses closed-loop feedback control based on visual and / or tactile perception, dynamically updating the strategy while accumulating new observation data. However, pure reinforcement learning faces challenges such as low sample efficiency and sparse rewards, especially in the case of high-dimensional dexterous hand. Combining reinforcement learning with imitation learning is an effective solution to these challenges, which not only allows the dexterous hand to try to imitate the actions of experts, but also optimizes and adjusts the grasping posture through self-exploration to improve adaptability to changes in task environment.

[0004] However, the above-mentioned dexterous hand grasping strategy learning method usually adopts the way of learning a certain skill separately, and when facing a new task, its generalization ability is insufficient, and it needs a lot of data and time to start learning from scratch. SUMMARY

[0005] In view of the above technical problems, a five-fingered robot hand functional grasping method based on progressive skill curriculum learning is proposed. The present application combines imitation learning, reinforcement learning, meta-learning and curriculum learning techniques, uses human expert teleoperation technology, combines multi-task reinforcement learning based on hand shape classification, and designs a progressive skill curriculum learning framework. Through the training of three stages of primary, intermediate and advanced, the framework gradually constructs and enhances the grasping skills of the robot when dealing with various objects.

[0006] The technical means adopted by the present application are as follows:

[0007] A five-finger mechanical dexterous hand functional grasping method based on progressive skill curriculum learning, comprising:

[0008] S1, using a teleoperation teaching system, through a hand posture detection and joint angle mapping module, controlling the action of the dexterous hand in the hand control simulation system, grasping the functional part of the object in the simulation system, obtaining the corresponding expert demonstration, and preparing resources for curriculum learning;

[0009] S2, defining different interactive tasks of the dexterous hand and the object according to the hand shape and the object category, and designing different stages of the curriculum;

[0010] S3, starting the progressive skill curriculum learning, the learning framework is divided into three difficulty levels of primary, intermediate and advanced courses, and at the end of each stage, a primary strategy set corresponding to a low difficulty task, an intermediate strategy set corresponding to a medium difficulty task and a high-level strategy corresponding to a high difficulty task are obtained;

[0011] S4, after the progressive skill curriculum learning, the high-level strategy grasps the corresponding part of the object of different categories according to the grasping hand shape classification information and the object point cloud through direct execution or a small amount of fine-tuning.

[0012] Further, step S1 specifically comprises:

[0013] S11, the teleoperation teaching system includes two RGB cameras, one of which is placed on the front side of the operation space, and the other is placed on the left side, providing 60fps video stream input to obtain the video stream of the real-time action of the hand;

[0014] S12, a hand 2D posture detection model is used to detect the video stream of the real-time action of the hand to obtain the 2D posture detection result of the hand;

[0015] S13, using OpenCV to calibrate the two fixed RGB cameras to obtain their respective internal and external parameters and the conversion matrix between them, which is used to map the 2D posture detection result of the hand to the 3D space to obtain the 3D posture detection result of the hand at each moment;

[0016] S14, a coordinate system is established with the middle finger root joint as the origin, the vectors between adjacent joints of the hand are calculated, and the mapping angles of the vectors in different coordinate planes are calculated to obtain the rotation angles of the 15 joints of the hand in different coordinate directions;

[0017] S15, using joint angle mapping, the joint angles of the hand are assigned to the corresponding mechanical hand joints;

[0018] S16, assign the relative change of the translation matrix T of each frame of the human hand to the robot hand, and convert the rotation matrix R into the Euler angle in the xyz direction to assign to the robot hand, remotely operate the robot hand to follow the human hand movement, and collect expert demonstration data.

[0019] Further, step S2 specifically comprises:

[0020] S21, design a primary course:

[0021] Each task is assigned to a primary strategy, and the primary strategy learns to grasp an example object with a specific hand shape through a demonstration obtained by a remote operation;

[0022] S22, design a middle course:

[0023] Each task is assigned to a middle strategy, and the middle strategy uses the same grasping hand shape to grasp multiple objects of the same category, and uses the trajectory collected by the primary strategy exploration as a demonstration;

[0024] S23, design a high-level course:

[0025] A single high-level strategy is trained using a meta-learning framework, and the high-level strategy uses different grasping hand shapes to grasp multiple object categories, and uses the trajectory collected by the middle strategy exploration as a demonstration.

[0026] Further, step S3 specifically comprises:

[0027] S31, primary course learning stage:

[0028] A method combining imitation learning and reinforcement learning is adopted, and for each task, only one demonstration is used for behavior cloning, the primary strategy is pre-trained, and the reinforcement learning algorithm is fine-tuned;

[0029] S32, middle course learning stage:

[0030] Each middle strategy is an advanced strategy of the primary strategy, and successfully grasps similar parts of objects of the same category as the demonstration object with a single grasping hand shape;

[0031] In order to achieve the learning goal of the middle course, each interactive task is expanded from a single example object to 11 to 17 objects of each category;

[0032] The primary strategy obtained by the primary stage training is online interacted with the expanded new environment, i.e. the middle learning environment, and the grasping trajectory of the expanded object is collected;

[0033] The object point cloud is encoded and embedded into the state space to enhance the understanding of the object geometry;

[0034] During learning, the collected trajectories are first explored by the primary policy to perform behavior cloning on the above 22 interaction tasks, and then fine-tuned by reinforcement learning.

[0035] S33, advanced course learning stage:

[0036] The demonstration of the advanced course learning stage comes from the trajectories collected by the intermediate policy interacting with the environment online; the learning of the advanced course learning stage adopts a meta-learning algorithm, allowing the advanced policy to effectively handle various objects and hand shapes after limited policy gradient updates.

[0037] Further, step S31 specifically comprises:

[0038] S311, setting the state space of the primary course learning stage: the complete state space of each primary policy contains three pieces of information, namely the robot body perception information {R}, the object pose information {O}, and the position coordinates of the five finger tips in the object coordinate system {F};

[0039] S312, setting the action space of the primary course learning stage: the action space is composed of 30-dimensional motor control commands, representing the translation and rotation of the 6-DOF arm, and the 24-dimensional joint angles of the dexterous hand;

[0040] S313, designing the reward function of the primary course learning stage, as follows:

[0041] r=r reach +r contact +r lift

[0042] wherein r reach represents the object-approaching reward function of the primary course learning stage, r reach =-λ r ∑||x finger -x coa ||2, λ r represents the weight assigned to the object-approaching reward r reach , x finger represents the position coordinates of the five finger tips in the object coordinate system, and x coa represents the geometric center point position of the object functional part; r contact represents the object-contacting reward function of the primary course learning stage, λ c represents the weight assigned to each contact point, c i represents whether the tactile sensor on each finger tip feels force, 1 if yes, otherwise 0, and i represents the index of the five finger tips; r lift represents the object-grasping reward function of the primary course learning stage, r lift =λl max(0, z current -z initial ), λ l represents the weight assigned to the reward for grasping the object, z current represents the z-axis coordinate of the current object centroid, z initial represents the z-axis coordinate of the object centroid in the initial state;

[0043] S314, calculate the strategy gradient of the primary course learning stage, the calculation formula is as follows:

[0044]

[0045] wherein, represents the advantage function of taking action a in state s relative to the primary strategy; represents the set of state-action pairs collected by the primary agent; represents the primary strategy; λ0, λ1 represents the hyperparameter for adjusting the contribution of different gradient calculation items; k represents the number of iterations; represents the gradient of the parameter θ, represents the set of state-action pairs collected by the human expert in the teleoperation process.

[0046] Further, step S32 specifically comprises:

[0047] S321, set the state space of the intermediate course learning stage: add the object point cloud code {P} to the primary strategy state space;

[0048] S322, set the action space of the intermediate course learning stage: the action space is composed of 30-dimensional motor control commands, representing the translation and rotation of the 6-DOF arm, and the joint angle of the dexterous hand 24 dimensions;

[0049] S323, design the reward function of the intermediate course learning stage, as follows:

[0050] r = r reach + r contact + r lift

[0051] wherein, r reach represents the reward function of the intermediate course learning stage for approaching the object, r reach = -λ r ∑||x finger -x coa ||2, λ r represents the weight assigned to the reward r reach for approaching the object, x finger represents the position coordinates of the five-finger fingertips in the object coordinate system, x coaGeometric center point position representing the functional part of the object; r contact Contact object reward function representing the intermediate course learning stage, λ c Weight assigned to each contact point, c i Whether the tactile sensor on each fingertip feels force, 1 if yes, 0 otherwise, i represents the index of the five fingertips; r lift Grasped object reward function representing the intermediate course learning stage, r lift = λ l · max(0, z current -z initial ), λ l Weight assigned to the grasped object reward, z current z-axis coordinate of the current object centroid, z initial z-axis coordinate of the object centroid in the initial state;

[0052] S324, calculate the policy gradient of the intermediate course learning stage, the calculation formula is as follows:

[0053]

[0054] wherein, represents the advantage function of taking action a in state s relative to the intermediate policy; and respectively represent the set of state-action pairs collected by the primary and intermediate intelligent agents; w k (s, a) represents the negative normalized log-likelihood, which is used to encourage the policy to learn trajectories that are difficult to replicate; represents the advantage function implemented by the multi-layer perceptron, λ0, λ1, λ'0 represent hyperparameters that adjust the contribution of different gradient calculation items; k represents the number of iterations.

[0055] Further, step S33 specifically includes:

[0056] S331, task setting:

[0057] Define task distribution Each is a Markov decision process with length H; policy model f θ represents π(a∣s), which maps state s t to the distribution of action a t at each time step t∈{1,2,3,…,H}; in each task , the goal of policy learning is as follows:

[0058]

[0059] Set the state space of the advanced curriculum learning stage: add the grasp hand shape code {T} to the intermediate strategy state space, use one-hot encoding to help the agent distinguish and perform appropriate grasping actions;

[0060] Set the action space of the advanced curriculum learning stage: the action space consists of 30-dimensional motor control commands, representing the translation and rotation of the 6-DOF arm, and the 24-dimensional joint angles of the dexterous hand;

[0061] Design the reward function of the advanced curriculum learning stage as follows:

[0062] r=r contact +r lift

[0063] where r contact represents the contact object reward function of the advanced curriculum learning stage, λ c represents the weight assigned to each contact point, c i represents whether the tactile sensor on each fingertip feels force, 1 if yes, otherwise 0, i represents the index of the five fingertips; r lift represents the grasp object reward function of the advanced curriculum learning stage, r lift =λ l ·max(0,z current -z initial ), λ l represents the weight assigned to the grasp object reward, z current represents the z-axis coordinate of the current object centroid, z initial represents the z-axis coordinate of the object centroid in the initial state;

[0064] S332, behavior cloning pre-training:

[0065] Use the trajectories collected by the intermediate strategy to complete behavior cloning to obtain the initial advanced strategy;

[0066] S333, meta-learning-based strategy training:

[0067] The training includes inner loop and outer loop;

[0068] S334, reasoning for new tasks:

[0069] Use the AffordPose dataset to train a grasp hand shape classification model, input the video of a human hand grasping an object, and output the classification result of the grasp hand shape, embedded in the state space in one-hot encoding form;

[0070] Use the point cloud encoder to encode the object and embed the encoded features into the state space;

[0071] The high-level strategy model trained in the meta-learning stage realizes successful grasping through zero or several gradient updates; in this process, the state space, action space and reward function are consistent with the meta-learning stage, but the policy gradient update formula is replaced by:

[0072]

[0073] Further, the inner loop and the outer loop in step S333 specifically include:

[0074] Inner loop:

[0075] First, a task is sampled from the task distribution , and then the current high-level strategy interacts with the environment, and k training trajectories are collected , and the training trajectories and the trajectories collected by the middle-level strategy are used to calculate the policy gradient g train , and the calculation formula is as follows:

[0076]

[0077] After the inner loop, the policy parameter update formula is as follows:

[0078] θ′ i = θ + αg train

[0079] The policy after the inner loop interacts with the environment in the task , and generates a validation trajectory

[0080] Outer loop:

[0081] In the outer loop, the policy gradient g is calculated valid , and the formula is as follows:

[0082]

[0083] After optimization, the high-level strategy parameter update formula is as follows:

[0084] θ′ = θ + βg valid

[0085] The optimized policy is obtained

[0086] Compared with the prior art, the present application has the following advantages:

[0087] 1. The present application classifies the dexterous function grasping task according to the hand shape type and defines it as a multi-task problem, and develops a progressive skill curriculum learning framework from simple to complex according to the task difficulty, so that the dexterous hand gradually acquires functional grasping skills that can adapt to changes in the task environment. The learning process is divided into three stages of primary, intermediate and advanced courses, so that the robot can learn a high-level strategy from multi-task demonstration data, integrate multiple skills, and complete the corresponding functional grasping task directly or with a small amount of fine-tuning through a hand shape classification information provided by a demonstration related to a new task.

[0088] 2. The dexterous hand functional grasping strategy learning method developed by the present application includes a set of teleoperation demonstration system and progressive skill curriculum learning framework. The trained dexterous hand can use a general strategy to complete functional grasping of multiple hand shapes and multiple categories of objects, with a high success rate.

[0089] 3. The present application provides a dexterous grasping high-level strategy obtained by combining imitation learning, reinforcement learning, meta-learning and curriculum learning, so that the dexterous hand of the robot can flexibly and effectively perform complex functional grasping tasks in different task environments, and can complete the corresponding functional grasping by using only a hand shape classification information provided by a demonstration when facing a new task.

[0090] Based on the above reasons, the present application can be widely popularized in the field of five-finger manipulator operation. BRIEF DESCRIPTION OF DRAWINGS

[0091] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0092] Figure 1 The method flowchart of the present application.

[0093] Figure 2 The primary course learning stage flowchart provided by the embodiment of the present application.

[0094] Figure 3 The intermediate course learning stage flowchart provided by the embodiment of the present application.

[0095] Figure 4 The demonstration of the advanced course learning stage provided by the embodiment of the present application is the trajectory collected from the online interaction of the intermediate strategy and the environment.

[0096] Figure 5The advanced strategy provided in the embodiment of the present application can quickly complete multi-hand functional grasping simulation graphs. DETAILED DESCRIPTION

[0097] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0098] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0099] As Figure 1 shown, the present application provides a five-finger mechanical dexterous hand functional grasping method based on progressive skill curriculum learning, comprising:

[0100] S1, using a teleoperation teaching system, through a hand posture detection and joint angle mapping module, controlling the action of a dexterous hand in a hand control simulation system, grasping the functional part of an object in the simulation system, obtaining the corresponding expert demonstration, and preparing resources for curriculum learning;

[0101] S2, defining different interactive tasks of the dexterous hand and the object according to the hand shape and the object category, and designing different stages of the curriculum;

[0102] In this embodiment, first, the grasping hand shape classification task is classified according to the fine grasping division given in the AffordPose dataset. Five functional grasping hand shapes are selected: screwing, cylinder wrapping, handle operation, finger pressing and hooking. Around these five hand shapes and ten categories of objects, 22 interactive tasks are designed, and a grasping demonstration trajectory is collected for each task.

[0103] S3, start the progressive skill course learning, the learning framework is divided into three difficulty levels of primary, intermediate and advanced courses, at the end of each stage, a primary strategy set corresponding to a low difficulty task, an intermediate strategy set corresponding to a medium difficulty task and a high-level strategy corresponding to a high difficulty task are obtained respectively;

[0104] S4, after the progressive skill course learning, the high-level strategy classifies the information of the grasping hand shape and the object point cloud, and directly executes or slightly adjusts to grasp the corresponding part of the object of different categories with the correct hand shape.

[0105] In specific implementation, as a preferred embodiment of the present application, step S1 specifically comprises:

[0106] S11, the teleoperation teaching system comprises two RGB cameras, one of which is placed on the front side of the operation space, and the other is placed on the left side, providing a 60fps video stream input to obtain a video stream of real-time hand action;

[0107] S12, a human hand 2D posture detection model is used to detect the video stream of real-time hand action to obtain a 2D posture detection result of the human hand;

[0108] S13, the two fixed RGB cameras are calibrated using OpenCV to obtain their respective internal and external parameters and the conversion matrix between them, which is used to map the 2D posture detection result of the human hand to the 3D space to obtain the 3D posture detection result of the human hand at each moment, i.e. the 3D coordinates of 21 key points;

[0109] S14, a coordinate system is established with the middle finger root joint as the origin, the vectors between adjacent joints of the fingers are calculated, and the mapping angles of the vectors in different coordinate planes are calculated to obtain the rotation angles of the 15 joints of the human hand in different coordinate directions;

[0110] S15, joint angle mapping is used to assign the joint angles of the human hand to the corresponding robot joints;

[0111] S16, the relative change of the translation matrix T of each frame of the human hand is assigned to the robot, and the rotation matrix R is converted into the Euler angles in the xyz direction and assigned to the robot, the teleoperation robot follows the movement of the human hand, and the expert teaching data is collected.

[0112] In specific implementation, as a preferred embodiment of the present application, step S2 specifically comprises:

[0113] S21, design the primary course:

[0114] Each task is assigned to a primary strategy, and the primary strategy learns to grasp the example object with a specific hand shape through a demonstration obtained by a teleoperation;

[0115] S22, design a middle-level course:

[0116] Each task is assigned to a middle-level policy, which uses the same grasp hand to grasp multiple objects of the same category, and uses the trajectories collected by the primary policy as demonstrations;

[0117] S23, design a high-level course:

[0118] A single high-level policy is trained using a meta-learning framework, which uses different grasp hands to grasp multiple object categories, and uses the trajectories collected by the middle-level policy as demonstrations.

[0119] In specific implementation, as a preferred embodiment of the present application, step S3 specifically comprises:

[0120] S31, primary course learning stage:

[0121] A method combining imitation learning and reinforcement learning is adopted, as shown in Figure 2 , for each task, only one demonstration is used for behavior cloning, the primary policy is pre-trained, and fine-tuned using a reinforcement learning algorithm;

[0122] S32, middle-level course learning stage:

[0123] Each middle-level policy is an advanced policy of the primary policy, which successfully grasps similar parts of objects of the same category as the demonstration object with a single grasp hand; as shown in Figure 3 , the process of middle-level course learning is shown.

[0124] In order to achieve the goal of middle-level course learning, each interaction task is expanded from a single example object to 11 to 17 objects of each category;

[0125] The primary policy obtained by the primary stage training is interacted online with the expanded new environment, i.e. the middle-level learning environment, to collect the grasping trajectories of the expanded objects;

[0126] The object point cloud is encoded and embedded into the state space to enhance the understanding of the object geometry;

[0127] During learning, the trajectories collected by the primary policy are first behavior cloned on the above 22 interaction tasks, and then fine-tuned using reinforcement learning.

[0128] S33, high-level course learning stage:

[0129] As shown in Figure 4 , the demonstration of the high-level course learning stage comes from the trajectories collected by the middle-level policy interacting online with the environment; the meta-learning algorithm is used for learning in the high-level course learning stage, which allows the high-level policy to effectively handle various objects and hands after limited policy gradient updates.

[0130] In particular implementation, as a preferred embodiment of the present application, step S31 specifically includes:

[0131] S311, setting the state space of the primary course learning stage: the complete state space of each primary strategy contains three pieces of information, which are robot body perception information {R}, object pose information {O}, and the position coordinates of the five finger tips in the object coordinate system {F};

[0132] S312, setting the action space of the primary course learning stage: the action space is composed of 30-dimensional motor control commands, representing the translation and rotation of the 6-DOF arm, and the joint angles of the dexterous hand with 24 dimensions;

[0133] S313, designing the reward function of the primary course learning stage, as follows:

[0134] r=r reach +r contact +r lift

[0135] wherein, r reach represents the object-approaching reward function of the primary course learning stage, r reach =-λ r ∑||x finger -x coa ||2, λ r represents the weight assigned to the object-approaching reward r reach , which is set to 0.5 in this embodiment, x finger represents the position coordinates of the five finger tips in the object coordinate system, and x coa represents the geometric center point position of the object functional part; r contact represents the object-contacting reward function of the primary course learning stage, λ c represents the weight assigned to each contact point, which is set to 0.2 in this embodiment, c i represents whether the tactile sensor on each finger tip senses force, 1 if yes, otherwise 0, and i represents the index of the five finger tips; r lift represents the object-grasping reward function of the primary course learning stage, r lift =λ l ·max(0,z current -z initial ), λ l represents the weight assigned to the object-grasping reward, z current represents the z-axis coordinate of the current object centroid, and z initial represents the z-axis coordinate of the object centroid in the initial state.

[0136] S314, calculate the strategy gradient of the primary course learning stage, the calculation formula is as follows:

[0137]

[0138] Wherein, represents the advantage function of taking action a in state s relative to the primary strategy; represents the set of state-action pairs collected by the primary agent; represents the state-action pair collected by the primary agent; λ0, represents a hyperparameter that adjusts the contribution of different gradient calculation items; represents the gradient of the parameter θ, represents the set of human expert state-action pairs collected during teleoperation.

[0139] In specific implementation, as a preferred embodiment of the present application, step S32 specifically includes:

[0140] S321, set the state space of the intermediate course learning stage: add the object point cloud point cloud encoding {P} to the primary strategy state space;

[0141] S322, set the action space of the intermediate course learning stage: the action space is composed of 30-dimensional motor control commands, representing the translation and rotation of the 6-DOF arm, and the joint angle of the dexterous hand 24 dimensions;

[0142] S323, design the reward function of the intermediate course learning stage, as follows:

[0143] r=r reach +r contact +r lift

[0144] Wherein, r reach represents the reward function of the intermediate course learning stage, r reach =-λ r ∑||x finger -x coa ||2, λ r represents the weight assigned to the reward r reach of approaching the object, which is set to 0.5 in this embodiment, x finger represents the position coordinates of the five-finger fingertips in the object coordinate system, x coa represents the geometric center point position of the object functional part; r contact represents the reward function of the primary course learning stage, λ c represents the weight assigned to each contact point, which is set to 0.2 in this embodiment, c i1 if the tactile sensor on each fingertip feels force, 0 otherwise, i denotes the index of the five fingertips; r lift represents the grasp object reward function of the primary course learning stage, r lift l ·max(0,z current -z initial ), λ l represents the weight assigned to the grasp object reward, z current represents the z-axis coordinate of the current object centroid, z initial represents the z-axis coordinate of the object centroid in the initial state;

[0145] S324, calculate the strategy gradient of the intermediate course learning stage, the calculation formula is as follows:

[0146]

[0147] wherein, represents the advantage function of taking action a in state s relative to the intermediate strategy; and respectively represent the set of state-action pairs collected by the primary and intermediate agents; w k (s,a) represents a negative normalized log-likelihood used to encourage the policy to learn hard-to-replicate trajectories; represents the advantage function implemented by a multi-layer perceptron, λ0, λ1, λ'0 represent hyperparameters that adjust the contribution of different gradient calculation items; k is the number of iterations.

[0148] In specific implementation, as a preferred embodiment of the present application, step S33 specifically comprises:

[0149] S331, task setting:

[0150] define the task distribution Each is a Markov decision process with length H; the policy model f θ represents π(a|s), which maps the state s t to the distribution of actions a t at each time step t∈{1,2,3,…,H}; in each task , the goal of policy learning is as follows:

[0151]

[0152] Set the state space of the advanced course learning stage: add the grasp hand shape code {T} to the intermediate policy state space, use one-hot encoding to represent, help the agent to distinguish and perform appropriate grasp actions;

[0153] ​Set the action space of the advanced course learning stage: the action space is composed of 30-dimensional motor control commands, representing the translation and rotation of the 6-DOF arm, and the joint angle of the dexterous hand with 24 dimensions;

[0154] Design the reward function of the advanced course learning stage as follows:

[0155] r=r contact +r lift

[0156] Wherein, r contact represents the contact object reward function of the advanced course learning stage, λ c represents the weight assigned to each contact point, which is set to 0.2 in this embodiment, c i represents whether the tactile sensor on each fingertip feels force, 1 if yes, otherwise 0, i represents the index of the five fingertips; r lift represents the grasp object reward function of the primary course learning stage, r lift =λ l ·max(0,z current -z initial ), λ l represents the weight assigned to the grasp object reward, z current represents the z-axis coordinate of the current object centroid, z initial represents the z-axis coordinate of the object centroid in the initial state;

[0157] S332, behavior cloning pre-training:

[0158] Complete behavior cloning with trajectories collected by the intermediate strategy to obtain an initial advanced strategy;

[0159] S333, strategy training based on meta-learning:

[0160] The training includes inner loop and outer loop;

[0161] S334, reasoning for new tasks:

[0162] A grasp hand shape classification model is trained using the AffordPose dataset, which inputs a video of a human hand grasping an object and outputs a classification result of the grasp hand shape, which is embedded into the state space in one-hot encoding form;

[0163] Use the point cloud encoder to encode the object and embed the encoded features into the state space;

[0164] The high-level policy model trained in the meta-learning phase achieves successful grasping through zero or a few gradient updates. In this process, the state space, action space, and reward function are consistent with those in the meta-learning phase, but the policy gradient update formula is changed to:

[0165]

[0166] In a specific implementation, as a preferred embodiment of the present invention, the inner loop and outer loop in step S333 specifically include:

[0167] Inner loop:

[0168] First, let's look at the task distribution. Mid-sampling task, then the current high-level policy being trained. Interact with the environment and collect k training trajectories Using training trajectory Trajectory collected by intermediate strategies Calculate the policy gradient g train The calculation formula is as follows:

[0169]

[0170] After the inner loop, the strategy parameter update formula is as follows:

[0171] θ′ i =θ+αg train

[0172] Strategy after inner loop In the mission Interact with the environment to generate verification trajectories

[0173] External circulation:

[0174] In the outer loop, through Calculate the policy gradient g valid The formula is as follows:

[0175]

[0176] The optimized formula for updating advanced strategy parameters is as follows:

[0177] θ′=θ+βg valid

[0178] Obtain the optimized strategy This process is optimized using the TRPO method.

[0179] Example

[0180] Evaluation indicators:

[0181] The performance of the method proposed in the application is evaluated by using the success rate of grasping.

[0182] Evaluation method:

[0183] The success rate is obtained by testing 5 random seeds, randomly initializing the position of each object to grasp 100 times, grasping for more than 2 seconds and grasping 10 cm to determine successful grasping.

[0184] Evaluation results:

[0185] Five ILAD grasping strategies and the advanced grasping strategy obtained in the application are compared on five object categories of the ShapeNet dataset. Since only violent grasping for the purpose of picking and placing is considered in the ILAD method, only the experimental results of the advanced strategy of the application performing cylindrical handshaking are compared. The success rate of grasping is shown in Table 1. The first row indicates that the corresponding ILAD strategy trained for each object category is used to grasp the objects of the same category. The success rate of each object category is high, and the average success rate of all object categories is 0.93. If the ILAD strategy is trained on one object category but tested on other object categories, the success rate will be significantly reduced. The advanced strategy grasping result of the application is listed in the last row, and the success rate on each object category is higher than that of the ILAD method.

[0186] Table 1. Comparison of success rates with ILAD on 5 unseen object categories of ShapeNet

[0187]

[0188] The success rate of the dexterous hand functional grasping strategy learning method Graff is compared on the 3DNet dataset. According to the information provided by the Graff method, 9 unseen objects are selected from the 3DNet dataset as the test set. 2 to 4 instances are selected for each category, for a total of 25 new objects. Table 2 shows the comparison of success rates of the application and the Graff method on the 3DNet dataset, and the grasping strategy obtained by the application is significantly better than that obtained by the Graff method. Figure 5 It is shown that the strategy obtained by the application not only allows grasping objects of these categories with a single strategy, but also supports multiple interaction modes, such as grasping the handle, cup mouth and cup body of the mug.

[0189] Table 2. Comparison of success rates with Graff on 9 unseen object categories of 3DNet

[0190]

[0191] In summary, the application proposes a novel high degree of freedom multi-fingered dexterous robot hand functional grasping learning framework - gradual skill curriculum learning framework, which gradually increases the task difficulty to enable the senior agent to obtain a general strategy that can adapt to various situations. The application designs multiple primary and intermediate agents to learn basic grasping skills according to the human hand grasping hand shape and objects, and integrates the skills by the senior agent to realize multi-functional anthropomorphic grasping based on a single demonstration.

[0192] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. A functional grasping method of a five-fingered dexterous hand based on progressive skill curriculum learning, characterized by, Comprise: S1, using a teleoperation teaching system, through a hand posture detection and joint angle mapping module, using a hand to control the action of a dexterous hand in a simulation system, grasping the functional part of an object in the simulation system, obtaining the corresponding expert demonstration, preparing resources for course learning; S2, define different interactive tasks of the dexterous hand and the object according to the hand shape and the object category, design different stage courses; according to the fine grasping division given in the AffordPose dataset, select five functional grasping hand shapes: screwing, cylinder wrapping, handle operation, finger pressing and hooking, design 22 interactive tasks around the five hand shapes and ten categories of objects, and collect one grasping demonstration trajectory for each task; S3, start the progressive skill course learning, the learning framework is divided into three difficulty levels of primary, intermediate and advanced courses, at the end of each stage, a primary strategy set corresponding to low difficulty tasks, an intermediate strategy set corresponding to medium difficulty tasks and a high-level strategy corresponding to high difficulty tasks are obtained, including: S31, primary course learning stage: Using a combination of imitation learning and reinforcement learning, for each task, only one demonstration is used for behavior cloning, the primary strategy is pre-trained, and the reinforcement learning algorithm is used for fine tuning; S32, intermediate course learning stage: Each intermediate strategy is an advanced strategy of the primary strategy, and successfully grasps the similar part of the object of the same category as the demonstration object with a single grasping hand shape; In order to achieve the goal of intermediate course learning, each interactive task is expanded from a single example object to 11 to 17 objects in each category; The primary strategy obtained in the primary stage is interacted online with the expanded new environment, i.e. the intermediate learning environment, to collect the grasping trajectories of the expanded objects; After the object point cloud is encoded and embedded into the state space, the understanding of the object geometry is enhanced; During learning, the trajectories collected by the primary strategy are first behavior cloned on the above 22 interactive tasks, and then fine-tuned using reinforcement learning; S33, advanced course learning stage: The demonstration of the advanced course learning stage comes from the trajectories collected by the online interaction of the intermediate strategy and the environment; the learning of the advanced course learning stage uses a meta-learning algorithm, which allows the advanced strategy to effectively handle various objects and hand shapes after limited strategy gradient updates; S4, after progressive skill course learning, the advanced strategy grasps the corresponding part of the object of different categories with the correct hand shape according to the grasping hand shape classification information and the object point cloud, through direct execution or a small amount of fine-tuning.

2. The functional grasping method of five-fingered dexterous hand based on progressive skill curriculum learning according to claim 1, characterized in that, Step S1, specifically comprising: S11, the teleoperation teaching system includes two RGB cameras, one of which is placed in the front of the operation space, and the other is placed on the left side, providing 60fps video stream input to obtain the real-time action of the hand video stream; S12, a hand 2D pose detection model is used to detect the real-time action of the hand video stream to obtain the 2D pose detection result of the hand; S13, calibrate the two fixed RGB cameras using OpenCV to obtain their respective internal and external parameters and the transformation matrix between them, which is used to map the 2D pose detection results of the human hand to the 3D space to obtain the 3D pose detection results of the human hand at each moment; S14, establish a coordinate system with the middle finger root joint as the origin, calculate the vectors between adjacent joints of the fingers, and then calculate the mapping angles of the vectors in different coordinate planes to obtain the rotation angles of the 15 joints of the human hand in different coordinate directions; S15, use joint angle mapping to assign the joint angles of the human hand to the corresponding joints of the manipulator; S16, assign the relative change of each frame translation matrix T of the human hand to the manipulator, and convert the rotation matrix R to the Euler angles in the xyz direction and assign it to the manipulator, so that the teleoperation manipulator follows the movement of the human hand and collects expert demonstration data.

3. The functional grasping method of five-fingered dexterous hand based on progressive skill curriculum learning according to claim 1, characterized in that, Step S2, specifically includes: S21, design a primary course: Each task is assigned to a primary strategy, which learns to grasp example objects with a specific hand shape through a demonstration obtained by teleoperation; S22, design a middle course: Each task is assigned to a middle strategy, which uses the same grasping hand shape to grasp multiple objects of the same category, and uses the trajectories collected by the primary strategy as demonstrations; S23, design an advanced course: Train a single advanced strategy using a meta-learning framework, which uses different grasping hand shapes to grasp multiple object categories, and uses the trajectories collected by the middle strategy as demonstrations.

4. The functional grasping method of a five-fingered dexterous hand based on progressive skill curriculum learning according to claim 1, wherein, Step S31, specifically includes: S311, set the state space of the primary course learning stage: the complete state space of each primary strategy contains three pieces of information, namely the robot body perception information {R}, the object pose information {O}, and the position coordinates of the five finger tips in the object coordinate system {F}; S312, set the action space of the primary course learning stage: the action space consists of 30-dimensional motor control commands, representing the translation and rotation of the 6-DOF arm, as well as the joint angles of the dexterous hand with 24 dimensions; S313, design the reward function of the primary course learning stage, as follows: r＝r reach +r contact +r lift where r reach represents the reward function of approaching objects in the primary course learning stage, r reach = -λ r ∑||x finger -X coa ||2, λ r represents the weight assigned to the reward r reach of approaching objects, x finger represents the position coordinates of the five fingertips in the object coordinate system, x coa represents the geometric center point position of the object functional part; r contact represents the reward function of contacting objects in the primary course learning stage, λ c represents the weight assigned to each contact point, c i represents whether the tactile sensor on each fingertip feels force, 1 if yes, otherwise 0, i represents the index of the five fingertips; r lift represents the reward function of grasping objects in the primary course learning stage, r lift = λ l ·max(0, z current -z initial ), λ l represents the weight assigned to the reward of grasping objects, z current represents the z-axis coordinate of the current object centroid, z initial represents the z-axis coordinate of the object centroid in the initial state; S314, calculate the strategy gradient of the primary course learning stage, the calculation formula is as follows: wherein, represents the advantage function of taking action a in state s with respect to the primary policy; represents a set of state-action pairs collected by the primary agent; represents the primary policy; λ0, λ1represent hyperparameters that regulate the contribution of different gradient computation terms; k represents the iteration number; represents the gradient with respect to the parameters θ, represents a set of state-action pairs collected by a human expert during teleoperation.

5. The functional grasping method of a five-fingered dexterous hand based on progressive skill curriculum learning according to claim 1, wherein, Step S32, specifically includes: S321, set the state space of the middle course learning stage: add the object point cloud encoding {P} to the primary strategy state space; S322, set the action space of the middle course learning stage: the action space consists of 30-dimensional motor control commands, representing the translation and rotation of the 6-DOF arm, as well as the joint angles of the dexterous hand with 24 dimensions; S323, design the reward function of the middle course learning stage, as follows: r＝r reach +r contact +r lift where r reach represents the reward function of approaching objects in the intermediate course learning stage, r reach = -λ r ∑||x finger -x coa ||2, λ r represents the weight assigned to the reward r reach of approaching objects, x finger represents the position coordinates of the five fingertips in the object coordinate system, x coa represents the geometric center point position of the object functional part; r contact represents the reward function of contacting objects in the intermediate course learning stage, λ c represents the weight assigned to each contact point, c i represents whether the tactile sensor on each fingertip feels force, 1 if yes, otherwise 0, i represents the index of the five fingertips; r lift represents the reward function of grasping objects in the intermediate course learning stage, r lift = λ l ·max(0, z current -z initial ), λ l represents the weight assigned to the reward of grasping objects, z current represents the z-axis coordinate of the current object centroid, z initial represents the z-axis coordinate of the object centroid in the initial state; S324, calculate the strategy gradient of the middle course learning stage, the calculation formula is as follows: where, represents the advantage function of taking action a in state s with respect to the intermediate policy; and represent the sets of state-action pairs collected by the primary and intermediate agents, respectively; w k (s, a) represents the negative normalized log-likelihood used to encourage the policy to learn trajectories that are difficult to replicate; represents the advantage function implemented by a multi-layer perceptron, λ0, λ1, λ'0represent hyperparameters that adjust the contribution of different gradient computation terms; k represents the iteration number.

6. The functional grasping method of a five-fingered dexterous hand based on progressive skill curriculum learning according to claim 1, wherein Step S33, specifically includes: S331, task setting: Defining task distribution Each is a Markov Decision Process of length H; policy model f θ is denoted as π(a|s), maps a state s t to a distribution over actions a t at each time step t ∈ {1, 2, 3, …, H}; in each task The goal of policy learning is as follows: Set the state space of the advanced course learning stage: add the grasping hand shape encoding {T} to the middle strategy state space, represented by one-hot encoding, to help the agent distinguish and perform appropriate grasping actions; Set the action space of the advanced course learning stage: the action space consists of 30-dimensional motor control commands, representing the translation and rotation of the 6-DOF arm, and the joint angles of the dexterous hand with 24 dimensions; Design the reward function of the advanced course learning stage as follows: r＝r contact +r lift where r contact represents the contact object reward function for the advanced curriculum learning phase, λ c represents the weight assigned to each contact point, c i represents whether the tactile sensor on each fingertip feels force, 1 if yes, 0 otherwise, i represents the index of the five fingertips; r lift represents the grasp object reward function for the advanced curriculum learning phase, r lift = λ l · max(0, z current - z initial ), λ l represents the weight assigned to the grasp object reward, z current represents the z-axis coordinate of the current object centroid, z initial represents the z-axis coordinate of the object centroid in the initial state; S332, behavior cloning pre-training: Complete behavior cloning with trajectories collected by the intermediate strategy to obtain an initial advanced strategy; S333, strategy training based on meta-learning: The training includes inner loop and outer loop; S334, reasoning for new tasks: A grasp hand shape classification model is trained using the AffordPose dataset, which takes a video of a human hand grasping an object as input and outputs the classification result of the grasp hand shape in one-hot encoding form and embeds it into the state space; Use the point cloud encoder to encode the object and embed the encoded features into the state space; The advanced strategy model trained in the meta-learning stage achieves successful grasping through zero or a few gradient updates; in this process, the state space, action space, and reward function are consistent with the meta-learning stage, but the policy gradient update formula is replaced with:

7. The functional grasping method of a five-fingered dexterous hand based on progressive skill curriculum learning according to claim 6, wherein, The inner loop and outer loop in step S333 specifically include: Inner loop: First sample tasks from the task distribution Then the current high-level policy Interacts with the environment, collects k training trajectories Uses the training trajectories And the trajectories collected by the middle-level policy Computes the policy gradient g train The formula is as follows: After the inner loop, the policy parameter update formula is as follows: θ' i = θ + αg train Post-inner loop strategy In the task Interact with the environment, generate verification trajectory Outer loop: In outer loop, by Computing the policy gradient g valid , which is given by the formula: After optimization, the advanced policy parameter update formula is as follows: θ' = θ + βg valid Obtaining an optimized policy This process is optimized using the TRPO method.

Images