Reinforcement learning based collaborative control method for dexterous hand and robot arm based on heuristic trajectory

Abstract

The application belongs to the field of mechanical arm and dexterous hand control, and discloses a dexterous hand and mechanical arm reinforcement learning collaborative control method based on heuristic trajectory. The method firstly establishes a mechanical arm-dexterous hand collaborative control framework in a simulation environment, and adopts a PPO algorithm for control strategy learning. By designing a reward function, translation, attitude adjustment, rotation and other actions are considered to improve the adaptability of the control strategy. In the control process, a constrained reinforcement differential dynamic programming method is introduced to better consider the dynamics characteristics of the mechanical arm. Finally, through a hybrid gradient update method, the mechanical arm-dexterous hand collaborative control strategy is formed by combining heuristic samples and exploratory samples, thereby improving the learning efficiency and generalization.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of mechanical arm and dexterous hand control, and particularly relates to a dexterous hand and mechanical arm reinforcement learning collaborative control method based on heuristic trajectory. BACKGROUND

[0002] Most robots in current life are composed of mechanical arms and end effectors. The mechanical arms provide the functions of movement flexibility in the spatial range and target proximity, and the end effectors interact with specific objects or environments to complete the tasks set by the user. At present, the end effectors of most mechanical arms are usually two-fingered grippers or suction cups. The degrees of freedom of such effectors are limited, and they can often only perform translational operations on hard objects with large surfaces, which greatly limits the expansion of the types of objects and operation tasks.

[0003] With the rapid development of robot hardware platforms and intelligent technologies in recent years, the structure and control technology of robots are constantly updated, and their work capacity has been rapidly improved. The most representative one is the multi-joint, multi-degree-of-freedom end effector, i.e., the multi-fingered dexterous hand. The index of the dexterous hand is ≥3, and the degree of freedom is ≥9. It has a bionic structure, like a human being, can adapt to the surfaces of objects of various shapes and materials, adjust a wide range of postures, and finely complete operations such as stretching, grasping, and clamping, showing excellent precision, versatility, and generalization.

[0004] At present, the most promising application field of the dexterous hand is the unstructured scene such as the family and aerospace. With its operation adaptability to the surfaces of diversified objects and the possibility of performing diversified operation tasks, it can bring great convenience to household tasks such as room arrangement, article delivery, and space tasks such as equipment repair and part recycling. At the same time, these tasks also require its high-precision end operation ability, which brings higher control difficulty.

[0005] The dexterous hand has a high degree of freedom. When interacting with the environment and operating objects, it not only needs to consider the contact state with the object, but also needs to consider the rationality of its own posture. The above two have a strong coupling relationship with the high-degree-of-freedom joints, and it is difficult to optimize numerically through partial derivatives and gradient seeking methods during the operation process, which limits the performance of the model-based control method.

[0006] As mentioned above, due to the high-dimensional action space and complex dynamic model, the design of the control method is a challenging problem. At present, it can be divided into two categories based on model and learning based on model. The representative method based on model is: the model predictive path integral control (MP PI) method successfully performs the task of manipulating a cube; the improved MP PI method makes the handover task between two hands feasible. The learning-based method involves reinforcement learning, which simplifies the design process of the controller. The model-free method cooperates with a large amount of exploration and makes a lot of progress in the field of dexterous hand control: the reinforcement learning-based controller successfully controls the dexterous hand to reposition a block or a Rubik's cube in the hand. The learning system composed of the reinforcement learning method successfully repositions a large number of objects without shape information. Dexterous manipulation through demonstration learning from teleoperation or video shows good bionic effect. Although current research shows that reinforcement learning can efficiently and expandably learn in single-hand control, the collaborative control of dexterous hand-robotic arm for general tasks is still a difficult problem for model-free reinforcement learning.

[0007] Since the dexterous hand-robotic arm system has more degrees of freedom than the dexterous hand, and there are heterogeneous differences between the two, the effects of each degree of freedom are not the same. When exploring under the traditional reinforcement learning method, the problems of large data size, difficulty in selecting effective samples, and gradient effectiveness decline are faced, which easily fall into local traps and stop updating when the control strategy is not rationalized. Especially in the face of diversified tasks and diversified operation modes (translation, attitude adjustment, rotation, etc.), different operations and different objects have different requirements for pose and contact force, making it extremely difficult to form a general and effective control strategy. The existing method needs to collect expert data for guidance, which consumes a lot of manpower and time cost, but still cannot be generalized to different tasks, greatly limiting the improvement of dexterous hand work ability. SUMMARY

[0008] The present application aims to provide a dexterous hand and robotic arm reinforcement learning collaborative control method based on heuristic trajectory to solve the above technical problems.

[0009] To solve the above technical problems, the specific technical solutions of the dexterous hand and robotic arm reinforcement learning collaborative control method based on heuristic trajectory of the present application are as follows:

[0010] A dexterous hand and robotic arm reinforcement learning collaborative control method based on heuristic trajectory, comprising the following steps:

[0011] Step 1: Simulation environment and model creation: establish the mujocogym framework based on the gym library, establish the dexterous hand-robotic arm integrated model, and construct the connection relationship and motion constraint between the two;

[0012] Step 2: Establish a Markov decision process model for the coordinated control of the robot hand and the manipulator in the mujoco environment based on step 1;

[0013] Step 3: Design a phased reward function based on the characteristics of the environment and the task, which can fully express the completion of the task and reward the agent with more information;

[0014] Step 4: Generate heuristic trajectory samples based on the dynamics model and constraints using the CS-DDP method to guide the reinforcement learning training process;

[0015] Step 5: Use the previously generated samples and agent exploration samples to guide gradient updates with adaptive weights.

[0016] Further, step 1 establishes an integrated model of the dexterous hand and the manipulator in the mujoco environment, constructs the connection relationship and motion constraints between the two, imports related xml files of the environment, objects, robots, and sensors, forms a simulation environment, and completes the initialization of the state, task type, environment boundary, and state and action upper and lower bounds.

[0017] Further, for the established integrated model of the dexterous hand and the manipulator, there is a fixed base, and each joint has a clear coupling dynamics relationship during motion, not just a kinematic relationship in space. Each initialization of the environment will randomize the three-dimensional position of the base within a certain range.

[0018] Further, step 2 uses the tuple <S, A, P, R, γ> to represent the information at a certain time, where S and A are the state and action spaces, respectively, the reinforcement learning algorithm learns a policy π(a|s) to maximize the expected cumulative reward R(st,at) under that policy, starting from the initial state distribution u and following the transition probability P(s(t+1)|st,at), the discount factor γ is used to balance the trade-off between immediate rewards and future rewards, and Q(s,a,s') represents the estimated value of the state s(t+1) after state transition. The goal of this method is given by the following formula:

[0019] a=argmax(R(s,a)+γ∑ s′∈S Q(s,a,s′)P(s(t+1)|st;at))。

[0020] Further, a phased reward function is designed based on the characteristics of the environment and the task, and a combination method of using different reward functions at different stages is adopted, as shown below:

[0021] For the translation problem, the reward function is modified to incentivize the robot by taking a logarithmic amplification. The reward function in this stage is set as follows:

[0022] R=r(s,g)+r(log(s,g))+r(path)

[0023] Here, r(s, g) is the negative of the L2 norm of the distance between the robot hand and the target. The smaller the distance between the hand and the target, the greater the reward. In addition, the above L2 norm is logarithmically transformed to obtain r(log(s, g)), where the reward increases rapidly as the distance between the hand and the target decreases. The reward function also includes the number of steps multiplied by the negative coefficient r(path) to penalize more paths for the same distance.

[0024] For the posture adjustment problem, a method is adopted to compare the posture of the dexterous hand with the posture of the object and continuously optimize the difference between the two. The reward function is expressed in the following form:

[0025] R=r(s,g)+r(log(s,g))+r(path)+r(ori)+r(target)

[0026] The first three terms have the same meaning as the reward function in the translation phase. The fourth term, r(ori), represents the Euclidean distance between the quaternion of the dexterous hand's posture and the quaternion of the target's posture. This term is used to encourage the robot to adjust its posture toward the expected direction when using the dexterous hand to approach the target. Finally, r(rarget) directly captures the rotation reward of the target, which is a function obtained by multiplying the target's rotation angle by a certain coefficient, representing the reward feedback for the dexterous hand to rotate the target.

[0027] The reward function design for translation after the target has rotated to a certain angle is as follows:

[0028] R=r(s,g)+r(log(s,g))+r(path)+r(ori)+r(target)+r(move)

[0029] Here, the new r(move) reward term is directly related to the translation distance of the object that conforms to the constraints. Combined with the previously used r(target) which represents its rotation, it completes the full representation of the manipulated object.

[0030] Furthermore, in step 4: a constraint-enhanced differential dynamic programming method, CS-DDP, is proposed. Based on differential dynamic programming (DDP), a tight constraint term is added to the objective function that originally minimizes the deviation from the desired trajectory.

[0031] Let xtbe the state of the robot arm at time t, utbe the control input at time t, the dynamics of the system is described as: xt+1= f(xt, ut), where f is the nonlinear dynamics function of the robot arm, the goal is to find a series of control inputs ut, so as to minimize the objective function J under the dynamics and constraint conditions of the system:

[0032]

[0033]

[0034] Where L is the stage objective function, which penalizes deviation from the desired trajectory, Φ is the final state objective function, which penalizes deviation from the final state, is the desired initial state, is the desired final state, g is the constraint function that enforces system constraints, the constraint function is updated as:

[0035] g(x, u) = g0(x, u) + λv*gv(x, u) + λe*ge(x, u)

[0036] Where g0(x, u) is the original objective function without constraints, λvand λeare the penalty coefficients for velocity and position constraints, gv(x, u) and ge(x, u) are the penalty functions for velocity and position constraints, which are defined as

[0037] gv(x, u) = max(0, ||vk|| - vmax) 2

[0038] ge(x, u) = max(0, ||pk-pdes|| - emax) 2

[0039] During the execution of the algorithm, the target object attitude quaternion is sampled multiple times using the simulation environment to provide accurate position and attitude range, and the best range of dexterous hand position and attitude required for operation is determined using this information. That is, the maximum error e max And the best dexterous hand pose

[0040] Further, the step 5 updates the Markov decision framework to the following under the premise of considering the dexterous hand dynamics and operation attitude constraints:

[0041] a = argmax(R(s, a) + γ∑ s′∈S Q(s, a, s') P(s(t+1)|st; at))

[0042]

[0043] In the above formula, R(s, a) represents the reward obtained by performing action a in state s, Q(s, a, s') represents the Q value obtained by performing action a to transition from state s to state s', P represents the state transition probability, and the measurement function epsilon loc Quantify the distance difference between the current distance vector and the ideal distance vector, and ensure that it is less than delta, the measurement function epsilon pos Capture the difference between the current pose and the ideal pose, and the smaller the difference, the better the performance.

[0044] Further, the step 5 first uses the CS-DDP method to generate heuristic trajectory samples in the mujoco environment according to the dexterous hand and robot dynamics model and the target object pose constraint, establishes a parallel training environment and a sample buffer, and randomly initializes the policy pi in the training environment θ And the parameter lambda, secondly, set the number of cycles N iter In each cycle, set the number of interactions N step Fill the sample buffer 1 with heuristic trajectory samples in parallel environment 1, and establish a Markov chain of <S, A, P, R, gamma> based on this, and use pi θ As an exploration strategy, collect exploratory samples and fill them into the sample buffer 2, and cycle N step Step to complete sample collection, then update the gradient of the sample buffer 1 and the sample buffer 2 to the policy pi according to the PPO algorithm gradient update method, and the proportion of lambda and 1-lambda θ Above, after completing the gradient update, the gradient update amplitude contributed by the buffer 1 is reduced by the value of lambda, and thus a cycle is completed, and the cycle is repeated N iter Times, until the robot obtains a reasonable operation strategy in simulation.

[0045] The dexterous hand and robot reinforcement learning collaborative control method based on heuristic trajectory of the present application has the following advantages:

[0046] The method has the advantages of considering the dexterous hand-robot collaborative control problem, focusing on the two challenges of dexterous hand pose adjustment and effective heuristic sample collection, and strengthening the adjustment and optimization of the dexterous hand pose through the innovative design of the stage reward function; through the CS-DDP method to establish constraints and generate sample heuristic dexterous hand-robot system to make more reasonable attempts and improve the overall learning efficiency; through the mixed gradient update method, exploration and heuristic are jointly applied to the robot system, so that the robot system has efficient and reasonable behavior and generalization of free exploration of reinforcement learning. The overall framework does not need to collect expert data manually, the model constraint is derived from the simulation environment and obtained by sampling, which ensures the lightness and flexibility of the overall framework. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 A structure diagram of the dexterous hand-robot arm integrated system of the present application;

[0048] Figure 2 An algorithm framework schematic diagram of the embodiment of the present application;

[0049] Figure 3 An algorithm specific flowchart of the embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to better understand the purpose, structure and function of the present application, the dexterous hand and robot arm reinforcement learning collaborative control method based on heuristic trajectory of the present application is further described in detail below in combination with the drawings.

[0051] The dexterous hand and robot arm reinforcement learning collaborative control method based on heuristic trajectory proposed by the present application comprises the following steps:

[0052] Step 1: Simulation environment and model creation: establish the mujocogym framework based on the gym library in the mujoco environment, as shown in Figure 1 The dexterous hand-robot arm integrated model is established, the connection relationship and motion constraint of the two are constructed, and the related environment, object, robot, sensor, etc. xml file is imported to form a simulation environment, and the initialization state, task type, environment boundary and state action upper and lower bound setting are completed. For the established dexterous hand-robot arm integrated model, there is a fixed base, and each joint has a clear coupled dynamics relationship in the motion process, not only the kinematics relationship in space. Each initialization of the environment will randomize the three-dimensional position of its base within a certain range.

[0053] Step 2: On the basis of step 1, a robot arm-dexterous hand collaborative control Markov decision process model under the mujoco environment is established, and a tuple <S, A, P, R, γ> is used to represent the information at a certain time, wherein S and A are state and action spaces respectively. The reinforcement learning algorithm aims to learn a policy π(a|s) so that the expected cumulative sum of the reward R(st,at) obtained under the policy is maximized, starting from the initial state distribution u and following the transition probability P(s(t+1)|st,at). The discount factor γ is used to balance the trade-off between immediate reward and future reward, and Q(s,a,s') represents the estimated value of the state s(t+1) after state transition. The goal of the present method is given by the following formula:

[0054] a=argmax(R(s,a)+γ∑ s′∈S Q(s,a,s′)P(s(t+1)|st;at))

[0055] Step 3: Design the reward function in stages according to the characteristics of the environment and the task, so that it can fully express the completion of the task. By using more information to reward the agent, learning can be accelerated and the convergence of its behavior can be accelerated. Considering the diversity of actions, this method uses a combination of different reward functions in different stages, making it more adaptable to tasks such as translation, pose adjustment, and rotation. The specific implementation is as follows:

[0056] For the most common translation problem, the most effective way to reach the target is to minimize the distance to the final target in all three dimensions. To emphasize the importance of samples at close distances, this method modifies the reward function by taking the logarithmic amplification form to encourage the robot to approach the target as directly and efficiently as possible. This stage reward function is set as follows:

[0057] R = r(s, g) + r(log(s, g)) + r(path)

[0058] where the value of r(s, g) is the negative value of the two-norm of the distance between the robot's hand and the target. The smaller the distance between the hand and the target, the greater the reward obtained. In addition, by taking the logarithmic transformation of the above two-norm, we get r(log(s, g)), where the reward increases rapidly as the distance between the hand and the target decreases. To encourage the agent to approach the target more efficiently, the reward function also includes r(path) multiplied by a negative coefficient, which punishes more paths for the same distance.

[0059] For the pose adjustment problem, a method is used to compare the pose of the dexterous hand with the pose of the object and continuously optimize the difference between the two. The reward function can be expressed as follows:

[0060] R = r(s, g) + r(log(s, g)) + r(path) + r(ori) + r(target)

[0061] where the first three terms have the same meaning as the translation stage reward function, the fourth term r(ori) represents the Euclidean distance between the quaternion of the dexterous hand's pose and the quaternion of the target's pose. This term is used to encourage the robot to adjust its pose towards the expected direction when approaching the target using the dexterous hand. In this way, the task of rotating the target is performed using a reasonable hand pose, avoiding the use of poor gestures such as wrist or back of the hand to manipulate the object. Finally, r(rarget)) directly captures the reward for rotating the target, which is a function obtained by multiplying the target rotation angle by a certain coefficient, representing the reward feedback for the dexterous hand rotating the target.

[0062] Since the rotation of the object is often accompanied by a translation movement, such as pushing the door open after rotating the door handle, opening the cabinet door after rotating the cabinet lock, and the like, the method further proposes a reward function design when the target needs to be translated after rotating to a certain angle:

[0063] R=r(s,g)+r(log(s,g))+r(path)+r(ori)+r(target)+r(move)

[0064] Here, the new r(move) reward term is directly related to the translation distance of the object conforming to the constraint, combined with the previous r(target) representing the rotation, to complete the complete representation of the operating object.

[0065] Step 4: Integrating the dexterous hand and the robot arm for integrated control faces the problem of high degrees of freedom, and only guided exploration sampling by the above reward function is not enough to obtain a reasonable control strategy, therefore, it is necessary to provide a certain number of heuristic trajectory samples to provide effective guidance for the robot in the early stage of reinforcement learning. In order to make the sample as much as possible to conform to the robot dynamics, kinematics constraints and initial and final states, the present application proposes a constraint reinforcement differential dynamic programming method (CS-DDP) based on differential dynamic programming (DDP), which adds a tight constraint term to the original objective function of minimizing the deviation from the expected trajectory, which is closer to the dynamics of the dexterous hand system.

[0066] Let xt be the state of the robot arm at time t, and ut be the control input at time t. The dynamics of the system can be described as: xt+1=f(xt,ut), where f is the nonlinear dynamics function of the robot arm. The goal is to find a series of control inputs ut that minimize the objective function J under the dynamics and constraint conditions of the system:

[0067]

[0068]

[0069] Where L is the stage objective function, which penalizes deviation from the expected trajectory, and Φ is the final state objective function, which penalizes deviation from the final state, is the expected initial state, is the expected final state, and g is the constraint function that enforces system constraints. Considering the above different operations, g is an additional constraint that must be considered when updating the trajectory and dynamics objective function linear approximation to penalize violations of velocity and end effector pose constraints. In the present method (CS-DDP), the constraint function is updated as:

[0070] g(x,u)=g0(x,u)+λv*gv(x,u)+λe*ge(x,u)

[0071] where g0(x, u) is the original objective function without constraints, λvand λeare the penalty coefficients for velocity and position constraints, and gv(x, u) and ge(x, u) are the penalty functions for velocity and position constraints. These penalty functions can be defined as

[0072] gv(x, u) = max(0, ||vk|| - vmax) 2

[0073] ge(x, u) = max(0, ||pk- pdes|| - emax) 2

[0074] To generate the optimal grasp pose of the dexterous hand for the object, the method samples the object's pose quaternion multiple times in the simulation environment during the algorithm execution to provide accurate position and pose ranges. With this information, the optimal range of the dexterous hand's position and pose required for the operation can be determined. That is, the maximum error emaxand the optimal dexterous hand pose

[0075] Step 5: Use the previously generated samples and the agent exploration samples to guide the gradient update in an adaptive weight manner, as follows:

[0076] Under the premise of considering the dexterous hand dynamics and operation pose constraints, the Markov decision framework is updated as follows:

[0077] a = argmax(R(s, a) + γ∑ s′∈S Q(s, a, s') P(s(t+1)|st; at))

[0078]

[0079] In the above formula, R(s, a) represents the reward obtained by performing action a in state s, Q(s, a, s') represents the Q value obtained by performing action a to transition from state s to state s', and P represents the state transition probability. The measurement function loc quantifies the distance difference between the current distance vector and the ideal distance vector, ensuring that it is less than δ. Similarly, the measurement function pos captures the difference between the current pose and the ideal pose, and the smaller the difference, the better the performance. For example, Figure 2As shown, the algorithm aims to effectively introduce generated sample data in the reinforcement learning process. CS-DDP generated sample learning and Scratch exploration sample learning are simultaneously performed during the training process, and the differentiated information of the two is fully utilized. In some training fields, the generated trajectory of CS-DDP is introduced, the heuristic behavior is used to guide the hand operation to approach the ideal state, and an adaptive coefficient λ<1 is introduced as the use proportion of the CS-DDP sample, which gradually decreases to 0 with the increase of the training iteration number. In other training fields, an exploratory method of starting from zero is adopted to improve the generalization ability and eliminate the false association of the strategy under the guidance of the behavior. This gradual transition from heuristic to policy gradient helps to traverse the action space, improve the final policy performance, and maintain the generalization characteristics of reinforcement learning. Multi-step experience replay is used in the application to reduce the error between the sample value and the true distribution.

[0080] Specifically, first, heuristic trajectory samples are generated in the mujoco environment according to the dexterous hand and robot arm dynamics model and target object pose constraints using the CS-DDP method, a parallel training environment and a sample buffer are established, and the policy π θ and the parameter λ. Second, the number of cycles N iter In each cycle, the number of interactions N step The heuristic trajectory samples are used to fill the sample buffer 1 in the parallel environment 1, and a Markov chain of <S, A, P, R, γ> is established, and the policy π θ is used as an exploration strategy to collect exploratory samples and fill into the sample buffer 2. This cycle is repeated N step times. After the sample collection is completed, the gradients calculated by the samples in the buffer 1 and the samples in the buffer 2 are updated to the policy π θ according to the PPO algorithm gradient update method in the proportion of λ and 1-λ. After the gradient update is completed, the value of λ is reduced according to the gradient update amplitude contributed by the buffer 1, and thus a cycle is completed. Repeat the cycle N iter times until the robot obtains a reasonable operation strategy in the simulation.

[0081] It can be understood that the present application is described by some embodiments, and those skilled in the art can make various changes or equivalent replacements to these features and embodiments without departing from the spirit and scope of the present application. In addition, under the guidance of the present application, these features and embodiments can be modified to adapt to specific conditions and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application are within the scope of the present application.

Claims

1. A heuristic trajectory based dexterous hand and robot arm reinforcement learning collaborative control method, characterized in that, Comprise the following steps: Step 1: simulation environment and model creation: establish the mujocogym framework based on the gym library, establish the dexterous hand-robot arm integrated model, construct the connection relationship and motion constraint of the two; Step 2: on the basis of step 1, establish the robot arm-dexterous hand cooperative control Markov decision process model under the mujoco environment; Step 3: according to the characteristics of the environment and the task, design the stage reward function, which can fully express the completion of the task, and use more abundant information to reward the agent; Step 4: use the CS-DDP method to generate heuristic trajectory samples based on the dynamics model and constraints to guide the reinforcement learning training process; A differential dynamic programming method combined with constraint enforcement, CS-DDP, is proposed. On the basis of differential dynamic programming (DDP), a tight constraint term is added to the original objective function that minimizes the deviation from the desired trajectory. Let x t be the state of the robot manipulator at time t, u t be the control input at time t, and the dynamics of the system be described as: x t+1 = f(x t , u t ), where f is the nonlinear dynamics function of the robot manipulator. The goal is to find a series of control inputs u t that minimize the objective function J under the dynamics and constraint conditions of the system: where L is a stage objective function penalizing deviation from a desired trajectory, Φ is a final state objective function penalizing deviation from a desired final state, is a desired initial state, is a desired final state, g is a constraint function enforcing system constraints, the constraint function is updated as: g(x,u)=g0(x,u)+λv*gv(x,u)+λe*ge(x,u) where g0(x, u) is the original objective function without constraints, λvand λeare the penalty coefficients for the velocity and position constraints, gv(x, u) and ge(x, u) are the penalty functions for the velocity and position constraints, which are defined as gv(x, u) = max(0, ||vk|| - v max ) 2 ge(x, u) = max(0, ||pk-pdes|| - e max ) 2 During the execution of the algorithm, the simulation environment is used to sample the target object's quaternion several times to provide an accurate position and orientation range, and using this information to determine the optimal range of positions and orientations of the dexterous hand required for the operation, i.e. the maximum error e max and the desired final state Step 5: use the samples generated before and the agent exploration samples to guide the gradient update in an adaptive weight manner.

2. The heuristic trajectory based dexterous hand and robot arm reinforcement learning collaborative control method according to claim 1, characterized in that, The step 1 establishes the dexterous hand-robot arm integrated model in the mujoco environment, constructs the connection relationship and motion constraint of the two, imports the xml files of related environment, object, robot and sensor, forms the simulation environment, and completes the setting of the initialization state, task type, environment boundary and state action upper and lower bound.

3. The heuristic trajectory based dexterous hand and robot arm reinforcement learning collaborative control method of claim 2, wherein, For the established dexterous hand-robot arm integrated model, there is a fixed base, and each joint has a clear coupled dynamics relationship in the motion process, not just a kinematics relationship in space. Each initialization of the environment will randomize the three-dimensional position of the base within a certain range.

4. The heuristic trajectory based dexterous hand and robot arm reinforcement learning collaborative control method of claim 1, wherein, The step 2 uses the tuple <S, A, P, R, gamma> to represent the information at a certain time, wherein S and A are state and action spaces respectively, the reinforcement learning algorithm learns a strategy pi(a|s) to maximize the expected cumulative reward R(st,at) under the strategy, starting from the initial state distribution u and following the transition probability P(s(t+1)|st,at), the discount factor gamma is used to balance the trade-off between immediate reward and future reward, Q(s,a,s') represents the estimated value of state s(t+1) after state transition, and the target is given by the following formula: a = argmax(R(s, a) + γ∑ s′∈S Q(s, a, s') P(s(t+1) | st; at).

5. The heuristic trajectory based dexterous hand and robot arm reinforcement learning collaborative control method of claim 1, wherein, According to the characteristics of the environment and the task, the stage reward function is designed, and a combination method of using different reward functions in different stages is adopted, as shown below: For the translation problem, the reward function is modified by taking the logarithmic amplification form to encourage the robot, and the reward function in this stage is set as follows: R=r(s,g)+r(log(s,g))+r(path) Wherein, the value of r(s,g) is the negative value of the two norm of the distance between the dexterous hand and the target, the smaller the distance between the hand and the target, the greater the reward, in addition, the logarithmic transformation of the above two norm is obtained r(log(s,g)), wherein the reward increases rapidly with the decrease of the distance between the hand and the target, and the reward function also includes r(path) of the step number multiplied by a negative coefficient, which is used to punish more paths for the same distance; For the pose adjustment problem, the method of comparing the pose of the dexterous hand with the pose of the object and constantly optimizing the difference between them is adopted, and the reward function is expressed in the following form: R = r(s, g) + r(log(s, g)) + r(path) + r(ori) + r(rarget) Where the first three terms have exactly the same meaning as the reward function in the translation stage, the fourth term r(ori) represents the Euclidean distance between the quaternion of the pose of the dexterous hand and the quaternion of the pose of the target, and this term is used to encourage the robot to adjust the pose of the dexterous hand to the expected direction when it approaches the target, r(rarget) directly captures the rotation reward of the target, which is a function obtained by multiplying the target rotation angle by a certain coefficient, indicating the reward feedback for the dexterous hand to rotate the target; After the target is rotated to a certain angle, the reward function design when translating is as follows: R = r(s, g) + r(log(s, g)) + r(path) + r(ori) + r(target) + r(move) The new r(move) reward term is directly related to the translation distance of the object meeting the constraint, combined with the previous r(target) representing the rotation, to complete the complete representation of the operating object. 6.The heuristic trajectory based dexterous hand and robot arm reinforcement learning collaborative control method of claim 4, wherein, The step 5 updates the Markov decision framework to the following under the premise of considering the dexterous hand dynamics and operating pose constraints: a = argmax(R(s, a) + γ∑ s′∈S Q(s, a, s') P(s(t+1) | st; at) R(s, a) in the above equation represents a reward obtained by performing action a in state s, Q(s, a, s') represents a Q value obtained by performing action a to transit from state s to state s', P represents a state transition probability, and a measurement function ∈ loc Quantify the distance difference between the current distance vector and the ideal distance vector, ensuring that it is less than δ, measurement function ∈ pos Capture the difference between the current pose and the ideal pose. The smaller the difference, the better the performance.

7. The heuristic trajectory based dexterous hand and robot arm reinforcement learning collaborative control method of claim 1, wherein, The step 5 first uses the CS-DDP method to generate heuristic trajectory samples in the mujoco environment according to the dexterous hand and robot dynamics model and target object pose constraints, establishes a parallel training environment and a sample buffer, and randomly initializes the strategy π in the training environment θ With the parameter λ, secondly, the number of cycles N is set iter In each cycle, the number of interaction steps N is set step In parallel environment one, the heuristic trajectory samples are used to fill the sample buffer one, and a Markov chain of <S, A, P, R, γ> is established, and in parallel environment two, π θ As an exploration strategy, exploratory sample collection is carried out, and is filled into the sample buffer two, and the cycle N is repeated step After the sample collection is completed, the gradient obtained by calculating the samples in the sample buffer one and the gradient obtained by calculating the samples in the sample buffer two are updated to the strategy π according to the PPO algorithm gradient update mode, and the value of λ is reduced according to the gradient update amplitude contributed by the sample buffer one θ After the gradient update is completed, the value of λ is reduced according to the gradient update amplitude contributed by the sample buffer one, and thus a cycle is completed, and the cycle is repeated N iter times, until the robot obtains a reasonable operation strategy in the simulation.

Images