A skill learning method for arm-hand robots based on residual hierarchical reinforcement learning

Abstract

The present invention discloses a skill learning method for an arm-hand robot based on residual hierarchical reinforcement learning, which aims to solve the problems of slow reinforcement learning training speed and low task success rate of dexterous arm-hand robots, and relates to the technical field of intelligent control of dexterous arm-hand robots. The method includes building a scene for training dexterous arm-hand robots to complete door opening tasks and screw hole assembly tasks in the physical simulation engine MuJoCo; building a multifunctional primitive library and a teaching imitation learning network through visual teleoperation; building a residual-based hierarchical reinforcement learning framework; and performing HRL-GAIL reinforcement learning training, which can significantly improve the skill learning speed, standardization, and generalization ability of the dexterous arm-hand robot.

Description

Technical Field

[0001] The present invention relates to the technical field of robot arm-hand intelligent control, and in particular to an arm-hand robot skill learning method based on residual hierarchical reinforcement learning. Background Art

[0002] With the rapid development of science and technology, new end effectors such as dexterous hands have gradually been applied to industrial production and life, and their skill learning has become a new research hotspot. However, due to the high degree of freedom of dexterous hands, traditional modeling control is increasingly unable to meet its skill learning requirements. Deep reinforcement learning enables robots to complete tasks in complex environments through continuous trial and error, but traditional reinforcement learning has problems such as slow learning speed, difficulty in reward function design, or difficulty in convergence, and poor skill standardization. Therefore, it has spawned the research and application of more intelligent reinforcement learning methods.

[0003] Hierarchical reinforcement learning technology divides the model into two layers by defining sub-goals. The high layer is responsible for specifying goals or selecting low-level options, and the low layer is responsible for performing specific actions to complete complex tasks. This method can effectively improve the training speed and training effect, but it still requires a lot of trial and error training.

[0004] Imitation learning technology is a method of learning by observing expert demonstrations, which does not require a lot of trial and error training and can effectively alleviate the above problems.

[0005] Combining hierarchical reinforcement learning technology with imitation learning technology can significantly improve the skill learning speed, standardization and generalization ability of dexterous arm-hand robots. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a residual hierarchical reinforcement learning method based on visual teaching. Combining deep reinforcement learning with expert teaching for arm-hand skill learning provides a faster exploration method for complex task training of dexterous arm-hand robots.

[0007] To achieve the above object, the present invention adopts the following technical solution: a method for learning arm-hand robot skills based on residual hierarchical reinforcement learning, characterized in that it includes the following steps:

[0008] Step S1: Building a scene in the physical simulation engine MuJoCo for training a dexterous arm-hand robot to complete door opening tasks and screw hole assembly tasks;

[0009] Step S2: Use the scene in MuJoCo for visual teleoperation and build a multifunctional primitive library;

[0010] Step S3: Using the multifunctional primitive library to build a teaching imitation learning network and a residual-based hierarchical reinforcement learning framework;

[0011] Step S4: Perform HRL-GAIL reinforcement learning training under the residual-based hierarchical reinforcement learning framework.

[0012] Furthermore, the specific steps of step S1 are as follows:

[0013] Step S1-1: Create a robot arm and dexterous hand model in SOLIDWORKS software and import it into the MuJoCo simulation environment in the form of an XML file;

[0014] Step S1-2: Add objects to be interacted with in the MuJoCo simulation environment.

[0015] Furthermore, the specific steps of step S2 are as follows:

[0016] Step S2-1: collect teaching data through visual teleoperation and perform action segmentation on the teaching data. The segmentation format is as follows:

[0017]

[0018] where s t is the state at time t, r t is the reward value at time t, s t+1 is the state at time t+1, a t is the action at time t, is the hand movement at time t, is the arm movement at time t, are the low-level parameters of the primitive, is the name of the primitive. t There are 4+11 dimensions in total. The first four dimensions are the wrist space coordinates and azimuth angles, and the last 11 dimensions are the finger joint angles. First, the arm and hand need to be separated, that is, Then you need to label the primitive to which each action belongs;

[0019] Step S2-2: Through generative adversarial imitation learning, the strategy-state pairs generated by the generator are used as generated samples, and the taught strategy-state pairs are used as real samples for discrimination by the discriminator. The discriminator output value is used as the reward value in the reinforcement learning experience pool for positive reinforcement learning training, so that the generator distribution is close to the expert strategy distribution.

[0020] Furthermore, the specific steps of step S3 are as follows:

[0021] Step S3-1: The model framework consists of two parts: the dexterous hand hierarchical model and the robotic arm hierarchical model. Each part is divided into a high-level task layer and a low-level primitive layer. The high-level task layer outputs a unique hot encoding of the length of the low-level primitive layer, which is responsible for selecting specific primitives. The selected low-level primitives are responsible for outputting the mean and variance of the Gaussian distribution. The specific parameters are selected through Gaussian sampling and output to the actuator, so that the robot performs the corresponding action;

[0022] Step S3-2: Set reward functions for the door opening task and the screw hole assembly task. For the screw hole assembly task, the task requires screw hole assembly based on grasping, so the successful reward requires not only the lifting height of the object but also the detection of the coincidence of the screw hole center coordinate point and the screw center coordinate point, that is, the successful reward is as shown in the formula:

[0023]

[0024] where n episode Indicates the total number of steps completed. is the grab reward value, ||(x,y,z) cube -(x,y,z) hole ||<0.01 means that the distance between the three coordinates of the object and the three coordinates of the screw hole is less than 0.01. In order to guide the dexterous hand to align before placing after grasping, the basic reward function is set as follows:

[0025]

[0026] in is the grasping reward, β dreach and β stick is the weight coefficient. The reward is divided into three stages. The first stage ||p gripper -p cube || represents the bi-norm from the grasping center of the dexterous hand to the center of the object. The second segment || (x, z, a (x,y) ) cube -(x,z,a (x,y) ) hole || indicates the (x,z) axis coordinates and yaw angle a of the object after grabbing (x,y) Align the hole and finally adjust the y coordinate to insert it into the hole.

[0027] For the door opening task, the task requires adjusting the angle of the dexterous hand after grasping the door handle and then pushing and pulling the door to open a certain angle. The successful reward is as shown in the formula:

[0028]

[0029] where ||p gripper -p doornob || represents the distance between the grasping center of the dexterous hand and the door handle, is the angle between the door and the (y, z) plane. The basic function for guiding the dexterous arm-hand robot to grasp and open the door is designed as:

[0030]

[0031] where d th is the coefficient for regulating the exp value, β dreach and β doornob is the weight coefficient.

[0032] Furthermore, the specific steps of step S4 are as follows:

[0033] Step S4-1: Both the door opening task and the screw hole assembly task are trained for 5000 rounds. In both task environments, the tasks can be finally completed through the influence of the residual strategy.

[0034] Compared with the prior art, the present invention has at least the following advantages:

[0035] First, by separating the task objectives from the specific actions through hierarchical reinforcement learning technology, complex tasks can be completed effectively, and the training speed and training effect can be improved. Second, by combining imitation learning with hierarchical reinforcement learning based on visual teaching and residual reinforcement learning, a large amount of trial and error training is not required, which can significantly improve the skill learning speed, standardization and generalization ability of the dexterous arm-hand robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is the structural diagram of the HRL-GAIL algorithm of the present invention.

[0037] Figure 2 This is a network structure diagram of the HRL-GAIL algorithm strategy of the present invention.

[0038] Figure 3 This is a task scenario diagram of the dexterous arm-hand robot of the present invention.

[0039] Figure 4 It is a training reward value curve diagram of the present invention.

[0040] Figure 5 Schematic diagram of pseudo code of HRL-GAIL algorithm of the present invention. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0042] Figure 1The structure diagram of the HRL-GAIL algorithm of the present invention includes three parts: the policy network, the target value network, and the action value network. The specific execution process of the algorithm is: first initialize the policy network, the target value network, and the action value network, generate actions through the policy network to interact with the environment to obtain exploration experience, and store it in the experience replay pool, then sample from the experience replay pool to update the policy network, the target value network, and the action value network, and finally repeat the above-mentioned environmental interaction and network update steps until the preset number of iterations is reached or the stop condition is met. Finally, a trained policy network is obtained, which is used to generate the optimal strategy for decision-making; Figure 2 The network structure diagram of the HRL-GAIL algorithm strategy of the present invention includes a dexterous hand hierarchical model and a mechanical arm hierarchical model. The strategy obtained by generating imitation learning training through visual teleoperation as teaching data is used as the initial strategy part of residual hierarchical reinforcement learning. The reinforcement learning part performs random exploration as a residual strategy to supplement the generalization ability of the initial strategy. The actuator will eventually execute the weighted sum of the initial strategy and the reinforcement learning strategy. Figure 3 The dexterous arm-hand robot task scene diagram of the present invention includes door opening and screw hole assembly task scenes, and the task scene is built by importing the XML file of the model into the contact multi-joint dynamics (MuJoCo) physical simulation engine; Figure 4 It is a training reward value curve diagram of the present invention, including a door opening task and a screw hole assembly task; Figure 5 Schematic diagram of pseudo code of HRL-GAIL algorithm of the present invention.

[0043] The present invention proposes an arm-hand robot skill learning method based on residual hierarchical reinforcement learning, and the specific steps are as follows:

[0044] Step S1: Building a scene in the physical simulation engine MuJoCo for training a dexterous arm-hand robot to complete door opening tasks and screw hole assembly tasks;

[0045] Step S1-1: Create a robot arm and dexterous hand model in SOLIDWORKS software and import it into the MuJoCo simulation environment in the form of an XML file;

[0046] Step S1-2: Add objects to be interacted with in the MuJoCo simulation environment. The experimental platform environment is as follows: Figure 3 shown.

[0047] Step S2: constructing a multifunctional primitive library through visual teleoperation;

[0048] Step S2-1: collect teaching data through visual teleoperation and perform action segmentation on the teaching data. The segmentation format is as follows:

[0049]

[0050] where s t is the state at time t, r t is the reward value at time t, s t+1 is the state at time t+1, a t is the action at time t, is the hand movement at time t, is the arm movement at time t, is the corresponding primitive name. t There are 4+11 dimensions in total. The first four dimensions are the wrist space coordinates and azimuth angles, and the last 11 dimensions are the finger joint angles. First, the arm and hand need to be separated, that is, Then you need to label the primitive to which each action belongs;

[0051] Step S3: Using the multifunctional primitive library to build a teaching imitation learning network and a residual-based hierarchical reinforcement learning framework;

[0052] Step S3-1: Generate adversarial imitation learning by taking the strategy-state pairs generated by the generator as generated samples, taking the multifunctional primitive library as real samples, and then discriminating them by the discriminator, and using the discriminator output value as the reward value in the reinforcement learning experience pool for positive reinforcement learning training, so that the generator distribution is close to the expert strategy distribution;

[0053] Step S3-2: Use the trained teaching imitation learning network to build a residual-based hierarchical reinforcement learning framework, which mainly consists of two parts: the hand hierarchical model and the robotic arm hierarchical model. Each part is divided into a high-level task layer and a low-level primitive layer. The high-level task layer outputs a unique hot encoding of the length of the low-level primitive layer, which is responsible for selecting specific primitives. The selected low-level primitives are responsible for outputting the mean and variance of the Gaussian distribution. The specific parameters are selected through Gaussian sampling and output to the actuator, so that the robot performs the corresponding action;

[0054] Step S3-3: Set a reward function for the door opening task and the screw hole assembly task in the embodiment. For the screw hole assembly task, the task needs to assemble the screw hole on the basis of grasping, so the successful reward needs to detect the coincidence of the screw hole center coordinate point and the screw center coordinate point in addition to the object lifting height, that is, the successful reward is as shown in the formula:

[0055]

[0056] where n episode Indicates the total number of steps completed. is the grab reward value, ||(x,y,z) cube -(x,y,z) hole||<0.01 means that the distance between the three coordinates of the object and the three coordinates of the screw hole is less than 0.01. In order to guide the dexterous hand to align before placing after grasping, the basic reward function is set as follows:

[0057]

[0058] in is the grasping reward, β dreach and β stick is the weight coefficient. The reward is divided into three stages. The first stage ||p gripper -p cube || represents the bi-norm from the grasping center of the dexterous hand to the center of the object. The second segment || (x, z, a (x,y) ) cube -(x,z,a (x,y) ) hole || indicates the (x,z) axis coordinates and yaw angle a of the object after grabbing (x,y) Align the hole and finally adjust the y coordinate to insert it into the hole.

[0059] For the door opening task, the task requires adjusting the angle of the dexterous hand after grasping the door handle and then pushing and pulling the door to open a certain angle. The successful reward is as shown in the formula:

[0060]

[0061] where ||p gripper -p doornob || represents the distance between the grasping center of the dexterous hand and the door handle, is the angle between the door and the (y, z) plane. The basic function for guiding the dexterous arm-hand robot to grasp and open the door is designed as:

[0062]

[0063] where d th is the coefficient for regulating the exp value, β dreach and β doornob is the weight coefficient.

[0064] Step S4: Perform HRL-GAIL reinforcement learning training under the residual-based hierarchical reinforcement learning framework;

[0065] Step S4-1: Train the door opening task and screw hole assembly task for 5000 rounds. Figure 4 The training reward value curve of the present invention is shown. The orange line is the average reward value of the trained teaching network, which is used for comparative verification of the residual algorithm. The blue part is the residual hierarchical reinforcement learning method based on visual teaching proposed by the present invention. In the two task environments, the task can be completed by the influence of the residual strategy, such as Figure 3 shown.

Claims

1. A method for learning arm-hand robot skills based on residual hierarchical reinforcement learning, characterized in that: The steps include: Step S1: Building a scene in the physical simulation engine MuJoCo for training a dexterous arm-hand robot to complete door opening tasks and screw hole assembly tasks; Step S2: Use the scene in MuJoCo for visual teleoperation and build a multifunctional primitive library; Step S3: Using the multifunctional primitive library to build a teaching imitation learning network and a residual-based hierarchical reinforcement learning framework; Step S4: Perform HRL-GAIL reinforcement learning training under the residual-based hierarchical reinforcement learning framework; The specific steps of step S3 are as follows: Step S3-1: The model framework consists of two parts: a dexterous hand hierarchical model and a robotic arm hierarchical model; each part is divided into a high-level task layer and a low-level primitive layer; the high-level task layer outputs a unique hot encoding of the length of the low-level primitive layer, which is responsible for selecting specific primitives. The selected low-level primitives are responsible for outputting the mean and variance of the Gaussian distribution, and the specific parameters are selected through Gaussian sampling and output to the actuator, so that the robot performs the corresponding action; Step S3-2: Set reward functions for the door opening task and the screw hole assembly task; for the screw hole assembly task, the task needs to assemble the screw hole on the basis of grasping, so the successful reward needs to detect the coincidence of the center coordinate point of the screw hole and the center coordinate of the screw in addition to the lifting height of the object. The successful reward is shown as follows: where n episode Indicates the total number of steps completed. is the grab reward value, ||(x,y,z) cube -(x,y,z) hole ||<0.01 means that the distance between the three coordinates of the object and the three coordinates of the screw hole is less than 0.

01. In order to guide the dexterous hand to align before placing after grasping, the basic reward function is set as follows: in is the grasping reward, β dreach and β stick is the weight coefficient; the reward is divided into three sections, the first section || p gripper -p cube || represents the bi-norm from the grasping center of the dexterous hand to the center of the object. The second segment || (x, z, a (x,y) ) cube -(x,z,a (x,y) ) hole || indicates the adjustment of the (x,z) axis coordinates and yaw angle a of the object after grabbing (x,y) Align the hole and finally adjust the y coordinate to insert it into the hole; For the door opening task, the task requires adjusting the angle of the dexterous hand after grasping the door handle and then pushing and pulling the door to open a certain angle. The successful reward is shown in the formula: where ||p gripper -p doornob || represents the distance between the grasping center of the dexterous hand and the door handle, is the angle between the door and the (y, z) plane. The basic function for guiding the dexterous arm-hand robot to grasp and open the door is designed as: where d th is the coefficient for regulating the exp value, β dreach and β doornob is the weight coefficient.

2. The arm-hand robot skill learning method based on residual hierarchical reinforcement learning as claimed in claim 1, wherein step S1 comprises the following specific steps: Step S1-1: Create a robot arm and dexterous hand model in SOLIDWORKS software and import it into the MuJoCo simulation environment in the form of an XML file; Step S1-2: Add objects to be interacted with in the MuJoCo simulation environment.

3. According to the method for learning arm-hand robot skills based on residual hierarchical reinforcement learning as claimed in claim 1, the specific steps of step S2 are as follows: Step S2-1: collect teaching data through visual teleoperation and perform action segmentation on the teaching data. The segmentation format is as follows: where s t is the state at time t, r t is the reward value at time t, s t+1 is the state at time t+1, a t is the action at time t, is the hand movement at time t, is the arm movement at time t; a t There are 4+11 dimensions in total. The first four dimensions are the wrist space coordinates and azimuth angles, and the last 11 dimensions are the finger joint angles. First, the arm and hand need to be separated. Then you need to label the primitive to which each action belongs; Step S2-2: Through generative adversarial imitation learning, the strategy-state pairs generated by the generator are used as generated samples, and the taught strategy-state pairs are used as real samples for discrimination by the discriminator. The discriminator output value is used as the reward value in the reinforcement learning experience pool for positive reinforcement learning training, so that the generator distribution is close to the expert strategy distribution.

4. The arm-hand robot skill learning method based on residual hierarchical reinforcement learning as claimed in claim 1, wherein step S4 comprises the following specific steps: Step S4-1: Both the door opening task and the screw hole assembly task are trained for 5000 rounds. In both task environments, the tasks can be finally completed through the influence of the residual strategy.

Images