A five-finger manipulator grasping and operating system based on single-track teaching

Abstract

The present invention provides a five-finger manipulator grasping and operating system based on single-track teaching, wherein a data set acquisition unit is used to acquire a data set; a remote operation unit is used to map human hand movements into simulation, control the movement of a dexterous hand, obtain corresponding expert teaching data after interacting with an object in the simulation, annotate the hand-object interaction key points and functional areas of the object used in the data, annotate the grasping hand type of the manipulator, and migrate the functional areas of the teaching object to the same category of objects through a point cloud registration method; a basic learning unit is used to perform rapid training of the expert teaching data in the basic learning stage to obtain a basic strategy; an extended learning unit is used to introduce an extended strategy, and perform joint training with the basic strategy when the functional area migration is completed, obtain an applicable dexterous hand strategy, and complete the grasping and operation of new objects. An indicator evaluation unit is used to evaluate the grasping and operation indicators of the system through the grasping success rate and learning efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of five-finger manipulator operation combining imitation learning with reinforcement learning, and in particular to a five-finger manipulator grasping and operating system based on single-track teaching. Background Art

[0002] The dexterous manipulation of a five-finger multi-DOF manipulator (hereinafter referred to as dexterous hand) based on imitation learning and reinforcement learning is a hot research topic in the field of embodied intelligent robots such as unstructured scenes, harsh working environments, and medical treatment, and there have been many excellent research results. However, so far, there are still many problems to be solved in the operation of dexterous hands. It is necessary to control multiple degrees of freedom to complete dexterous human-like operations, which brings huge challenges to the learning and application of dexterous hand control strategies. Therefore, it is of great significance to develop a method that is friendly to human teaching and easy to train dexterous hand strategies.

[0003] Domestic and foreign researchers have tried many research schemes on human-hand teaching and dexterous hand strategy learning. In human-hand teaching, Rajeswaran used VR to collect demonstrations of dexterous manipulation, and the user wore data gloves for teaching, and introduced the gradient enhancement [1][A.Rajeswaran, V.Kumar, A.Gupta, J.Schulman, E.Todorov, and S.Levine, "Learning complex dexterous manipulation with deep reinforcement learning and demonstrations," ArXiv, vol.abs / 1709.10087, 2017.] algorithm for reinforcement learning. However, this method uses VR to collect data, which requires more manpower and is expensive. There are many subsequent papers devoted to reducing the cost of expert teaching, mainly introducing visual methods. Literature [2] [A. Handa, KV Wyk, W. Yang, J. Liang, Y.-W. Chao, Q. Wan, S. Birchfield, N. D. Ratliff, and D. Fox, "Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system," 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 9164–9170, 2019.] uses multiple cameras to relocate human hand movements to the allegrohand four-finger robot online to collect teaching data, but it requires the construction of a cumbersome data collection system. Literature [3] [Li, Shuang, Xiaojian Ma, Hongzhuo Liang, Michael Philipp Ruppel, Bin Fang, Fuchun Sun and Jianwei Zhang. "Vision-based Teleoperation of Shadow Dexterous Hand using End-to-End Deep Neural Network." 2019 International Conference on Robotics and Automation (ICRA) (2018): 416-422.] Through the optimization method, a large number of data sets of human hand depth maps mapped to the shadowhand five-fingered robot were obtained. The posture mapping network from human hand to dexterous hand was trained using supervised learning methods, and then the interaction data was collected in conjunction with IMU equipment and handheld depth cameras. Qin[4][Y.Qin,Y.-H.Wu,S.Liu,H.Jiang,R.Yang,Y.Fu,and X.Wang,“Dexmv:Imitation learning for dexterous manipulation from human videos,”in European Conference on Computer Vision,2021.]et al. used multi-camera depth cameras to track and record the interaction trajectory between human hands and objects, extracted the human hand and object parameters in the trajectory, and then used the objective function optimization method to relocate the human hand in the data as a robotic arm. This method is processed offline. In their subsequent research, [5][Y. Qin, H. Su, and X. Wang, "From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera operation," IEEE Robotics and Automation Letters, vol. 7, pp. 10 873–10 881, 2022.] used a depth camera to perform real-time 30fps hand detection, mapped the detection results online to a standard humanoid manipulator in a simulation environment, completed the interaction with the object through teleoperation, and recorded the data and processed it in the manner of [4] to obtain expert teaching.In other literature [6][Y.-H.Wu,J.Wang,and X.Wang,“Learninggeneralizabledexterous manipulation from human grasp affordance,”ArXiv,vol.abs / 2204.02320,2022.], similar methods were also used to process imitation data.

[0004] Imitation learning from expert demonstrations can allow robots to have some of the experience of experts, reduce meaningless exploration in the reinforcement learning stage, make the behavior more humanized, and greatly reduce the complexity of the reward function, but the cost of expert data is high. Reference [7][Huang, Tao et al. "Demonstration-Guided Reinforcement Learning with Efficient Exploration for Task Automation of Surgical Robot." arXiv, 2023.] et al. In order to improve the efficiency of expert demonstrations and accelerate RL training, imitation learning is mainly used, and reinforcement learning is used to fine-tune and improve the learning process of the strategy. Reference [4] explored the impact of the number of expert demonstrations on strategy learning and proved that it is still a challenge to ensure that the strategy can successfully learn expert behavior with as few expert demonstrations as possible. In the literature [8][Arunachalam, Sridhar Pandian, Sneha Silwal, Ben Evans and Lerrel Pinto. "Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Deexterous Manipulation." 2023IEEE International Conference on Robotics and Automation (ICRA) (2022): 5954-5961.], the research focus is also placed on teaching and imitation learning, and the impact of different imitation learning methods on strategy learning is discussed. Summary of the invention

[0005] According to the technical problem raised above, a five-finger manipulator grasping and operating system based on single-track teaching is provided. The purpose of the present invention is to train the dexterous hand strategy to complete the functional grasping of the corresponding class using only one teaching as imitation learning data.

[0006] The technical means adopted by the present invention are as follows:

[0007] A five-finger manipulator grasping and operating system based on single-track teaching includes: a data set acquisition unit, a teleoperation unit, a basic learning unit, an extended learning unit and an indicator evaluation unit, wherein:

[0008] The data set acquisition unit is used to acquire a data set;

[0009] The remote operation unit is used to map the human hand movements to the simulation, control the dexterous hand in the simulation to move, and achieve the effect of remote control. After interacting with the objects in the simulation, the corresponding expert teaching data is obtained, and the hand-object interaction key points and functional areas of the objects used in the data are marked, the gripping hand shape of the manipulator is marked, and the functional areas of the teaching objects are migrated to other objects of the same category through the point cloud registration method;

[0010] The basic learning unit is used to perform rapid training in the basic learning phase on the expert teaching data obtained by the teleoperation unit to obtain a basic strategy;

[0011] The extended learning unit is used to introduce an extended strategy with better generalization ability based on the basic strategy, and conduct joint training with the basic strategy when the functional area migration is completed, so as to obtain an applicable dexterous hand strategy to complete the grasping and operation of new objects;

[0012] The indicator evaluation unit is used to evaluate the system's grasping and operation indicators by calculating the grasping success rate and learning efficiency.

[0013] Furthermore, the data set acquisition unit is used to acquire the data set, specifically including:

[0014] Select 12 types of objects from ShapeNet, DexFuncGrasp, and Grab datasets, and select at least 20 objects of different shapes for each type of object, including mugs, watering cans, pliers, and other tool-like objects.

[0015] Furthermore, the remote operation unit includes a hardware part and a software part, wherein:

[0016] The hardware part includes two RGB cameras, which are fixed at the front and left sides of a space respectively, to obtain the video stream of real-time hand movements. They are calibrated using OpenCV to obtain their internal and external parameters and the transformation matrix between them.

[0017] The software part includes hand detection module, simulation module, hand mapping and operation module, among which:

[0018] The hand detection module moves in a space, and is used to pass the acquired transformation matrix through the mediapipe 2D hand detection model to obtain the 2D detection result of the hand, and map the 2D detection result of the hand to the 3D space to obtain the 3D detection result of the hand at each moment, that is, the 3D coordinates of 21 key points;

[0019] The simulation module builds an operation platform based on Mujoco simulation, including a dexterous hand, a table and an object, and the dexterous hand in the simulation is operated by the human hand mapping and operation module;

[0020] The human hand mapping and operation module is used to represent the human hand detection result as shape and posture parameters in the form of a mano model, and uniformly establish a coordinate system with the middle finger root joint as the origin; calculate the vectors between adjacent key points of the fingers, and then calculate the mapping angles of the angles between the vectors in different planes to obtain the angles of 15 joints of the human hand in different coordinate directions; assign the human hand joint angles corresponding to the manipulator to the manipulator, assign the relative change of the translation matrix T of the human hand in each frame to the manipulator, convert the rotation matrix R into the Euler angle in the xyz direction and assign it to the manipulator, the remote-controlled manipulator follows the movement of the human hand, collects expert teaching data, and marks the hand-object interaction information on the teaching object, that is, the contact points between the five fingertips and the object, records the hand shape of the manipulator when grasping the object, and uses the KDTree algorithm to expand the five contact points to the parts of the object grasped by the dexterous hand.

[0021] Furthermore, the basic learning unit uses a method combining imitation learning and reinforcement learning to learn expert behavior using only one teaching data, wherein: the state space of a basic strategy obtained includes object information {P affordance}、Robot proprioceptive information {P palm , J angles}、Contact information between dexterous hand and object{P tips2tips}; In the class-level grasping task, object information is represented by the object contact point and the distance from the fingertip to the object annotation point, guiding the strategy to control the dexterous hand to contact the correct area; the action space output by the strategy model is a thirty-dimensional motor control command to control the joint movement.

[0022] Furthermore, the basic learning unit performs rapid training of the expert teaching data obtained by the teleoperation unit in the basic learning stage, specifically including:

[0023] Introducing gradient enhancement as a learning objective:

[0024]

[0025] Among them, the first term is the standard NPG algorithm gradient term, and the second term is the expert data enhancement term;

[0026] Design the reward function for reinforcement learning as follows:

[0027] R basic =r approach +r tips2tips +r contact +r penalty +r lift

[0028] Among them, r approach represents the reward function for approaching an object, r approach =-||P palm -P obj || 1 ; r tips2tips represents the reward function for grasping an object, r contact represents the reward function for contact between the fingertip and the object annotation point, r penalty represents the speed penalty reward function, r penalty =-λ 3 V palm ; r lift represents the reward function for grasping the object long enough, r lift =λ 4 H obj .

[0029] Furthermore, the extended learning unit is used to realize the migration of the grasped part of the object and the joint reinforcement learning training, wherein:

[0030] Migration of the grasped part of the object: The flexible registration network CPDNET is used to flexibly align the point cloud of the teaching object with the point cloud of the target object, so that the annotation of the grasped part of the teaching object can be migrated to the corresponding part of the target object;

[0031] Joint reinforcement learning training: A sparse reward function is used during training, expressed as:

[0032] R extension =r approach +r lift

[0033] Two gradient terms are used to train the base strategy and the extended strategy separately:

[0034]

[0035]

[0036] Among them, the first term is the NPG reinforcement learning gradient term using self-exploration data, the second term is the enhanced gradient term provided by the basic policy exploration data, and λ 5 , 6is a hyperparameter, k is the number of iterations, and the second gradient gradually disappears as k increases.

[0037] Furthermore, in the indicator evaluation unit, the system's grasping and operation indicators are evaluated by grasping success rate and learning efficiency, wherein:

[0038] Grasping success rate: The grasping success rate of non-teaching objects on 12 types of objects is counted. Each object is initialized and grasped ten times at a random position. If it is grasped for more than 5 seconds, it is considered a successful grasp, and the average success rate is obtained;

[0039] Learning efficiency: The number of iterations required for training to converge, with the success rate as the indicator in the task of grasping and resetting the object.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] 1. The present invention realizes the development of a set of dexterous hand grasping and operating systems that are easy to deploy and use manual remote operation teaching, and achieves the goal of learning dexterous hand operation from a teaching data. It does not require a large amount of imitation data as a drive, is highly efficient, and provides technical means for, for example, robot embodiment, and can achieve the effect of class-level function grasping with a high success rate.

[0042] 2. The five-finger manipulator grasping and operating system based on single-track teaching provided by the present invention has a low-cost teaching system, is easy to deploy, and is easy to operate in real time. It can achieve most human-like operation effects and perform hand repositioning processing online.

[0043] 3. The five-finger manipulator grasping and operating system based on single-track teaching provided by the present invention is based on the framework of experience transfer and joint training, which not only has the effect of completing training in rapid iteration, but also the dexterous hand grasping and operation strategy obtained has high generalization ability and good anthropomorphism.

[0044] Based on the above reasons, the present invention can be widely promoted in the fields of five-finger manipulator operation combining imitation learning with reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 This is a system structure block diagram of the present invention.

[0047] Figure 2 Schematic diagram of the remote control unit of the present invention.

[0048] Figure 3 This is a schematic diagram of the present invention learning expert behavior using only one piece of teaching data.

[0049] Figure 4 It is a diagram showing the migration effect of the grasped part of the object of the present invention.

[0050] Figure 5 This is a flow chart of the joint reinforcement learning training of the present invention.

[0051] Figure 6 This is a human-like rendering of the present invention.

[0052] Figure 7 The figure shows the human-like effect of the present invention in comparison with the ILAD method.

[0053] Figure 8 The learning efficiency of the present invention is compared with that of the DAPG algorithm in the object resetting task. DETAILED DESCRIPTION

[0054] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0055] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0056] like Figure 1 As shown, the present invention provides a five-finger manipulator grasping and operating system based on single-track teaching, including: a data set acquisition unit, a teleoperation unit, a basic learning unit, an extended learning unit and an indicator evaluation unit, wherein:

[0057] The data set acquisition unit is used to acquire a data set;

[0058] The remote operation unit is used to map the human hand movements to the simulation, control the dexterous hand in the simulation to move, and achieve the effect of remote control. After interacting with the objects in the simulation, the corresponding expert teaching data is obtained, and the hand-object interaction key points and functional areas of the objects used in the data are marked, the gripping hand shape of the manipulator is marked, and the functional areas of the teaching objects are migrated to other objects of the same category through the point cloud registration method;

[0059] The basic learning unit is used to perform rapid training in the basic learning phase on the expert teaching data obtained by the teleoperation unit to obtain a basic strategy;

[0060] The extended learning unit is used to introduce an extended strategy with better generalization ability based on the basic strategy, and conduct joint training with the basic strategy when the functional area migration is completed, so as to obtain an applicable dexterous hand strategy to complete the grasping and operation of new objects;

[0061] The indicator evaluation unit is used to evaluate the system's grasping and operation indicators by calculating the grasping success rate and learning efficiency.

[0062] In specific implementation, as a preferred embodiment of the present invention, the data set acquisition unit is used to acquire the data set, specifically including:

[0063] Select 12 types of objects from ShapeNet, DexFuncGrasp, and Grab datasets, and select at least 20 objects of different shapes for each type of object, including mugs, watering cans, pliers, and other tool-like objects.

[0064] In specific implementation, as a preferred embodiment of the present invention, the remote operation unit includes a hardware part and a software part, wherein:

[0065] The hardware part includes two RGB cameras, which are fixed at the front and left sides of a space respectively, to obtain the video stream of real-time hand movements. They are calibrated using OpenCV to obtain their internal and external parameters and the transformation matrix between them.

[0066] The software part includes hand detection module, simulation module, hand mapping and operation module, among which:

[0067] The hand detection module is about 0.5m 3 The obtained transformation matrix is ​​moved in the space to pass through the mediapipe 2D hand detection model to obtain the 2D detection result of the hand, and the 2D detection result of the hand is mapped to the 3D space to obtain the 3D detection result of the hand at each moment, that is, the 3D coordinates of the 21 key points;

[0068] The simulation module is based on the Mujoco simulation to build an operation platform, including a dexterous hand, a table and an object, wherein the hand joints have 22 degrees of freedom, the arm has 6 degrees of freedom, and the object attribute settings are consistent with reality. The dexterous hand in the simulation is operated by the human hand mapping and operation module;

[0069] The human hand mapping and operation module is used to represent the human hand detection results in the form of a mano model as shape and posture parameters, and uniformly establish a coordinate system with the middle finger root joint as the origin; calculate the vectors between adjacent key points of the fingers, and then calculate the mapping angles of the angles between the vectors in different planes to obtain the angles of the 15 joints of the human hand in different coordinate directions; assign the human hand joint angles corresponding to the manipulator to the manipulator, assign the relative change of the human hand translation matrix T in each frame to the manipulator, convert the rotation matrix R into the Euler angle in the xyz direction and assign it to the manipulator, the remote-controlled manipulator follows the movement of the human hand, collects expert teaching data, and marks the hand-object interaction information on the teaching object, that is, the contact points between the five fingertips and the object, records the manipulator hand shape when grasping the object, and uses the KDTree algorithm to expand the five contact points to the part of the object grasped by the dexterous hand. Figure 2 The figure shows a schematic diagram of a remote operation unit.

[0070] In specific implementation, as a preferred embodiment of the present invention, the basic learning unit uses a method combining imitation learning and reinforcement learning, and only uses one teaching data to learn expert behavior, wherein: the state space of a basic strategy obtained includes object information {P affordance}、Robot proprioceptive information {P palm , J angles}、Contact information between dexterous hand and object{P tips2tips}; In the class-level grasping task, the object information is represented by the object contact point and the distance from the fingertip to the object marking point, guiding the strategy to control the dexterous hand to contact the correct area; the action space output by the strategy model is a thirty-dimensional motor control command to control the joint movement. In this embodiment, the task environment in the basic stage is related to teaching, and the initialization state of a recorded teaching trajectory is applied to the basic strategy training. Each time the task environment is reset, the state of the object and the dexterous hand is the same as the initialization state, ensuring that the basic strategy learns the grasping logic and natural grasping hand shape, reducing the sample complexity in the reinforcement learning exploration process, and accelerating the strategy optimization iteration process. Figure 3 As shown in the figure, the basic learning stage uses a method that combines imitation learning and reinforcement learning, and only uses one piece of teaching data to learn expert behavior.

[0071] In specific implementation, as a preferred embodiment of the present invention, the basic learning unit performs rapid training of the expert teaching data obtained by the teleoperation unit in the basic learning stage, specifically including:

[0072] Introducing gradient enhancement as a learning objective:

[0073]

[0074] Among them, the first term is the standard NPG algorithm gradient term, and the second term is the expert data enhancement term;

[0075] Design the reward function for reinforcement learning as follows:

[0076] R basic =r approach +r tips2tips +r contact +r penalty +r lift

[0077] Among them, r approach represents the reward function for approaching an object, r approach =-||P palm -P obj || 1 ; r tips2tips represents the reward function for grasping an object, r contact represents the reward function for contact between the fingertip and the object annotation point, r penalty represents the speed penalty reward function, r penalty =-λ 3 V palm ; r lift represents the reward function for grasping the object long enough, r lift =λ 4 H obj .

[0078] In specific implementation, as a preferred embodiment of the present invention, the extended learning unit is used to realize the migration of the grasped part of the object and the joint reinforcement learning training, wherein:

[0079] Migration of the grasped part of the object: The flexible registration network CPDNET is used to flexibly align the point cloud of the teaching object with the point cloud of the target object, so that the annotation of the grasped part of the teaching object can be migrated to the corresponding part of the target object. The effect is as follows: Figure 4 shown.

[0080] Joint reinforcement learning training: Figure 5 As shown in the figure, the extended strategy is introduced, the extended strategy is combined with the basic strategy, and the extended strategy is obtained based on the strategy optimization of reinforcement learning; the basic strategy provides high-efficiency training and high-return exploration in the early stage of training, and uses experience transfer and strategy refinement to transfer its experience to the extended strategy and enhance it. During training, the state space of the extended strategy is different from that of the basic strategy, and its state space is represented by object information {P affordance}、Robot proprioceptive information {P palm , J angles}; Both are trained in a more realistic environment, where objects are randomly placed at a certain position when the environment is initialized; in order to make the learned expansion strategy more generalizable, a sparse reward function is used during training, expressed as:

[0081] R extension =r approach +r lift

[0082] In the joint training framework, the basic strategy is trained only with the data obtained by its own exploration, and the extended strategy is trained with the basic strategy data after state space distillation and the data generated by its own exploration, and gradually reduces the use of the data obtained by the basic strategy exploration in the later stages of training. Two gradient terms are used to train the basic strategy and the extended strategy respectively:

[0083]

[0084]

[0085] Among them, the first term is the NPG reinforcement learning gradient term using self-exploration data, the second term is the enhanced gradient term provided by the basic policy exploration data, and λ 5 , 6 is a hyperparameter, k is the number of iterations, and the second gradient gradually disappears as k increases. The basic strategy can complete the corresponding grasping task after training with mixed data. The final dexterous hand strategy has a good performance in the functional grasping task at the class level, and has the effect of grasping the correct part and anthropomorphic nature. Figure 6 shown.

[0086] In specific implementation, as a preferred embodiment of the present invention, in the indicator evaluation unit, the system's grasping and operation indicators are evaluated by grasping success rate and learning efficiency, wherein:

[0087] Grasping success rate: The grasping success rate of non-teaching objects on 12 types of objects is counted. Each object is initialized and grasped ten times at a random position. If it is grasped for more than 5 seconds, it is considered a successful grasp, and the average success rate is obtained;

[0088] Learning efficiency: The number of iterations required for training to converge, with the success rate as the indicator in the task of grasping and resetting the object.

[0089] In summary, the five-finger manipulator grasping and operating system based on single-track teaching provided by the present invention maps human hand movements into the simulation through human hand detection and a reasonable mapping module, controls the dexterous hand in the simulation to move, and achieves the effect of remote control. After interacting with the object in the simulation, the corresponding expert teaching data is obtained, and the key points and functional areas of the hand-object interaction of the objects used in the data are marked, and the grasping hand type of the manipulator is marked. The functional area of ​​the teaching object is migrated to other objects of the same category through the point cloud registration method. The expert teaching data is first quickly trained in the designed basic learning stage to obtain a basic strategy. The basic strategy does not have the ability to complete the task, but only learns how experts interact with objects flexibly. On this basis, an extended strategy with more generalization ability is introduced and jointly trained with the basic strategy when the functional area migration is completed, and finally an applicable dexterous hand strategy is obtained.

[0090] Example

[0091] The performance of the present invention is compared with the latest excellent results in recent years, including DAPG [A.Rajeswaran,V.Kumar,A.Gupta,J.Schulman,E.Todorov,and S.Levine,“Learning complex dexterous manipulation with deep reinforcement learning and demonstrations,”ArXiv,vol.abs / 1709.10087,2017.], ILAD [Wu,Yueh-Hua,Jiashun Wangand Xiaolong Wang.“Learning Generalizable Dexterous Manipulation from HumanGrasp Affordance.”ArXiv abs / 2204.02320(2022):n.pag.], GRAFF [Mandikal,Priyankaand Kristen Grauman.“Learning Dexterous Grasping with Object-Centric Visual Affordances.”2021IEEE International Conference on Robotics and Automation (ICRA) (2020): 6169-6176.]. Table 1 shows different methods for grasping given the functional area of ​​an object. The present invention has the best success rate and the best human-like operation effect. Figure 7The results of the comparison between the present invention and ILAD grasping human-like operation are shown. Under the premise of good grasping effect and success rate, the present invention also has a faster iteration rate, such as Figure 8 As shown, the learning efficiency of the present invention is significantly improved.

[0092] Table 1. Capture success rate of different methods

[0093] method Average crawl success rate DAPG 39％ ILAD 27％ GRAFF 62％ OURS 78％

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A five-finger manipulator grasping and operating system based on single-track teaching, characterized in that: include: Dataset acquisition unit, teleoperation unit, basic learning unit, extended learning unit and indicator evaluation unit, including: The data set acquisition unit is used to acquire a data set; The remote operation unit is used to map the human hand movements to the simulation, control the dexterous hand in the simulation to move, and achieve the effect of remote control. After interacting with the objects in the simulation, the corresponding expert teaching data is obtained, and the hand-object interaction key points and functional areas of the objects used in the data are marked, the gripping hand shape of the manipulator is marked, and the functional areas of the teaching objects are migrated to other objects of the same category through the point cloud registration method; The basic learning unit is used to perform rapid training in the basic learning phase on the expert teaching data obtained by the teleoperation unit to obtain a basic strategy; The basic learning unit uses a method combining imitation learning and reinforcement learning to learn expert behavior using only one teaching data, wherein: the state space of a basic strategy obtained includes object information {P affordance }、Robot proprioceptive information {P palm , J angles }、Contact information between dexterous hand and object{P tips2tips }; In the class-level grasping task, the object information is represented by the object contact point and the distance from the finger tip to the object annotation point, guiding the strategy to control the dexterous hand to contact the correct area; the action space output by the strategy model is a 30-dimensional motor control command to control the joint movement; The basic learning unit performs rapid training of the expert teaching data obtained by the teleoperation unit in the basic learning phase, specifically including: Introducing gradient enhancement as a learning objective: Among them, the first term is the standard NPG algorithm gradient term, and the second term is the expert data enhancement term; Design the reward function for reinforcement learning as follows: R basic ＝r approach +r tips2tips +r contact +r penalty +r lift Among them, r approach represents the reward function for approaching an object, r approach =-||P palm -P obj ||1;r tips2tips represents the reward function for grasping an object, r contact represents the reward function for contact between the fingertip and the object annotation point, r penalty represents the speed penalty reward function, r penalty =-λ3V palm ; r lift represents the reward function for grasping the object long enough, r lift =λ4H obj ; The extended learning unit is used to introduce an extended strategy with better generalization ability based on the basic strategy, and conduct joint training with the basic strategy when the functional area migration is completed, so as to obtain an applicable dexterous hand strategy to complete the grasping and operation of new objects; The extended learning unit is used to realize the migration of the grasped part of the object and the joint reinforcement learning training, wherein: Migration of the grasped part of the object: The flexible registration network CPDNET is used to flexibly align the point cloud of the teaching object with the point cloud of the target object, so that the annotation of the grasped part of the teaching object can be migrated to the corresponding part of the target object; Joint reinforcement learning training: Introduce the extended strategy, combine the extended strategy with the basic strategy, and obtain the extended strategy based on reinforcement learning strategy optimization; the basic strategy provides high-efficiency training and high-return exploration in the early stage of training, and uses experience transfer and strategy refinement to transfer experience to the extended strategy and enhance it. During training, the state space of the extended strategy is different from that of the basic strategy, and the state space is represented as object information {P affordance }、Robot proprioceptive information {P palm , J angles }; Both are trained in a more realistic environment, where objects are randomly placed at a certain position when the environment is initialized; in order to make the learned expansion strategy more generalizable, a sparse reward function is used during training, expressed as: R extension ＝r approach +r lift In the joint training framework, the basic strategy is trained only with the data obtained by its own exploration, and the extended strategy is trained with the basic strategy data after state space distillation and the data generated by its own exploration, and gradually reduces the use of the data obtained by the basic strategy exploration in the later stage of training; two gradient terms are used to train the basic strategy and the extended strategy respectively: Among them, the first term is the NPG reinforcement learning gradient term using self-exploration data, the second term is the enhanced gradient term provided by the basic strategy exploration data, λ5 and λ6 are hyperparameters, k is the number of iterations, and the second gradient gradually disappears as k increases; The indicator evaluation unit is used to evaluate the system's grasping and operation indicators by calculating the grasping success rate and learning efficiency.

2. The five-finger manipulator grasping and operating system based on single-track teaching according to claim 1 is characterized in that: The data set acquisition unit is used to acquire the data set, specifically including: Select 12 types of objects from ShapeNet, DexFuncGrasp, and Grab datasets, and select at least 20 objects of different shapes for each type of object, including mugs, watering cans, pliers, and other tool-like objects.

3. The five-finger manipulator grasping and operating system based on single-track teaching according to claim 1 is characterized in that: The remote operation unit includes a hardware part and a software part, wherein: The hardware part includes two RGB cameras, which are fixed at the front and left sides of a space respectively, to obtain the video stream of real-time hand movements. They are calibrated using OpenCV to obtain their internal and external parameters and the transformation matrix between them. The software part includes hand detection module, simulation module, hand mapping and operation module, among which: The hand detection module moves in a space, and is used to pass the acquired transformation matrix through the mediapipe 2D hand detection model to obtain the 2D detection result of the hand, and map the 2D detection result of the hand to the 3D space to obtain the 3D detection result of the hand at each moment, that is, the 3D coordinates of 21 key points; The simulation module builds an operation platform based on Mujoco simulation, including a dexterous hand, a table and an object, and the dexterous hand in the simulation is operated by the human hand mapping and operation module; The human hand mapping and operation module is used to represent the human hand detection result as shape and posture parameters in the form of a mano model, and uniformly establish a coordinate system with the middle finger root joint as the origin; calculate the vectors between adjacent key points of the fingers, and then calculate the mapping angles of the angles between the vectors in different planes to obtain the angles of 15 joints of the human hand in different coordinate directions; assign the human hand joint angles corresponding to the manipulator to the manipulator, assign the relative change of the translation matrix T of the human hand in each frame to the manipulator, convert the rotation matrix R into the Euler angle in the xyz direction and assign it to the manipulator, the remote-controlled manipulator follows the movement of the human hand, collects expert teaching data, and marks the hand-object interaction information on the teaching object, that is, the contact points between the five fingertips and the object, records the hand shape of the manipulator when grasping the object, and uses the KDTree algorithm to expand the five contact points to the parts of the object grasped by the dexterous hand.

4. The five-finger manipulator grasping and operating system based on single-track teaching according to claim 1 is characterized in that: In the indicator evaluation unit, the system's grasping and operation indicators are evaluated by grasping success rate and learning efficiency, where: Grasping success rate: The grasping success rate of non-teaching objects on 12 types of objects is counted. Each object is initialized and grasped ten times at a random position. If it is grasped for more than 5 seconds, it is considered a successful grasp, and the average success rate is obtained; Learning efficiency: The number of iterations required for training to converge, with the success rate as the indicator in the task of grasping and resetting the object.

Images