Thumb independent optimization five-finger manipulator control method based on hierarchical strategy control

Abstract

The invention relates to the field of robot technology and artificial intelligence, and particularly discloses a thumb independent optimization five-finger manipulator control method based on hierarchical strategy control. The method comprises the following steps: S1, designing a hierarchical strategy control algorithm framework, and obtaining current target object information; s2, dividing a high-level strategy network and a low-level strategy network, wherein the high-level strategy network determines whether to activate the action of a thumb; the low-layer strategy network outputs the action probability of each finger joint based on the decision of the high-layer strategy network; s3, evaluating a strategy optimization effect by using an evaluation module; S4, carrying out training and optimization; s5, repeating the training process in S4; s6, updating the strategy according to the collected experience after the action is executed every time; s7, after training is completed, the model of each finger and strategy parameters learned in the training process are stored; and S8, verifying the performance of the trained model in a simulation environment. According to the invention, the problem of uncoordinated actions caused by excessive coupling between fingers can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of robotics and artificial intelligence technology, and in particular to a thumb-independent optimized five-finger manipulator control method based on hierarchical strategy control. Background Art

[0002] In the biomechanics of human fingers, the thumb can provide stronger grip than other fingers due to the unique configuration of joints and muscles, improving the efficiency of clamping and grasping objects. Biological studies have shown that fingers with thumbs can increase their grip by 25% to 50%, and the flexibility of the thumb enables humans to perform a variety of fine operations, such as using tools and writing. Relevant data show that the participation of the thumb can increase the success rate of fine operation tasks by 70%.

[0003] However, most of the existing five-finger robotic arms use a unified control network for coordination, but in actual applications the movement requirements of the thumb and other fingers are quite different, which makes it impossible to accurately adapt to complex task requirements.

[0004] In addition, traditional manipulator control methods usually rely on a single control strategy or low-level PID control, ignoring the delicate balance between collaboration and independence between fingers, and are unable to meet the control requirements of multi-degree-of-freedom manipulators in complex dynamic tasks. Summary of the invention

[0005] In response to the above-mentioned problems in the prior art, the present invention provides a five-finger manipulator control method with independent thumb optimization based on hierarchical strategy control, which can effectively solve the problem of uncoordinated movements caused by excessive coupling between fingers, improve the operating accuracy and flexibility of the overall manipulator, better cope with task changes in complex environments, and ensure the stability and accuracy of the manipulator in dynamic scenarios.

[0006] The technical solution adopted by the present invention is as follows:

[0007] A thumb-independent optimized five-finger manipulator control method based on hierarchical strategy control, the steps are as follows:

[0008] S1. Design a hierarchical strategy control algorithm framework to obtain the current target object's positioning information for subsequent action decision calculations;

[0009] S2. Use PyTorch to divide the high-level policy network and the low-level policy network. The high-level policy network receives the state of the target object as input, and generates a binary action probability output through a three-layer fully connected neural network to decide whether to activate the thumb action; the low-level policy network is based on the decision of the high-level policy network, uses a multi-layer fully connected neural network and outputs the action probability of each finger joint to refine the action decision of each finger;

[0010] S3. Use the evaluation module PPOCritic network to evaluate the effect of strategy optimization and calculate the value function of each strategy. The PPOCritic network is divided into two parts: high-level and low-level evaluation.

[0011] S4, training and optimization based on the PPO algorithm, wherein the optimization process uses reward signals and state value estimation to optimize the strategy through multiple iterations;

[0012] S5, repeat the training process of S4 until the preset training target or convergence condition is reached;

[0013] S6. After each action is executed, the agent updates its strategy through the PPO algorithm based on the collected experience;

[0014] S7. After the training is completed, the Actor model of each finger and the strategy parameters learned during the training process are saved for subsequent reasoning or continued training;

[0015] S8. Use simulation and reasoning to verify the performance of the trained model in the simulation environment.

[0016] Preferably, in S2, the high-level strategy network architecture consists of an input layer that accepts 6-dimensional features of the state, two layers of 64-dimensional fully connected hidden layers using ReLU activation function, and a binary classification output layer. The high-level strategy network uses Softmax activation function to calculate the action probability and generates the final action decision a by random sampling. thumb , the calculation formula is as follows:

[0017] a thumb =softmax(W3·ReLu(W2·ReLu(W1·St)));

[0018] Among them, W1, W2, and W3 are the weight matrices of the network, W3 is the weight matrix shared by other fingers in the low-level strategy network, and ReLU is the activation function.

[0019] Preferably, in S2, an independent network is designed in the low-level policy network to process the action space of the thumb, and other fingers share the same network for action decision making.

[0020] Preferably, in S2, the architecture of the low-level strategy network consists of an input layer that receives 6-dimensional features of the current state, two fully connected hidden layers of size 64, and an output layer that outputs the action probability of each finger joint through a Softmax function, wherein the thumb strategy network is:

[0021] a thumb =softmax(W thumb·ReLu(W2·ReLu(W1·S t )));

[0022] Where W thumb It is the weight matrix of the thumb strategy network, which is responsible for processing the action decision of the thumb in a specific state; W3 in the above calculation process is for the convenience of description and needs to be modified later.

[0023] Other finger strategy networks:

[0024] a finger =softmax(W3·ReLu(W2·ReLu(W1·S t ))).

[0025] Preferably, in S3, the high-level evaluation network is used to evaluate the overall state of the task and output a scalar value, indicating the overall value of the current task state; the low-level evaluation network is used to evaluate the local state of the finger joints and output a scalar value, indicating the value of the finger joint state; the architecture of the high-level evaluation network and the low-level evaluation network are both composed of two fully connected layers, and the output is a scalar value, indicating the value of the corresponding state, wherein the high-level evaluation network:

[0026] V task =f task (S t );

[0027] Low-level evaluation network:

[0028] V joint =f joint (S t ).

[0029] Preferably, in S4, the training process is carried out in the MuJoCo simulation environment in which the environment and the agent (robot finger) interact, and the agent performs the target object grasping task. The execution of each action is decided based on the current state St and the action probability output by the network.

[0030] Preferably, in S6, the PPO model updates the strategy according to the experience replay, splitting the advantage calculation, TD target, and loss function into different levels; the experience replay includes the state, action, and reward of the agent.

[0031] Preferably, in S6, the specific steps of updating the strategy include:

[0032] Step 1: Use the advantage function of the PPO algorithm to calculate the difference between the expected return in the current state and the state value;

[0033] Step 2: Based on the immediate reward and the value of the next state, calculate the time difference TD target, obtain the expected return of the agent in the current state, compare the difference between the current strategy and the old strategy, and calculate the reward based on whether the round ends;

[0034] Step 3: Measure the difference between the current state value estimate and the TD target, calculate the policy loss, value loss, and entropy loss, use the entropy regularization term to encourage exploration, and control the balance between exploration and exploitation by adjusting the hyperparameters;

[0035] Step 4: Use back propagation to calculate the gradient of the loss function, and use the gradient descent method to update the parameters of the policy network, so that the PPO model can continuously adjust the strategy during training and gradually optimize the control action of the finger.

[0036] Preferably, in a hierarchical structure, the policy network selects the optimal action at each time step by calculating the probability distribution of each action, and decides whether to switch to a low-level policy to handle more specific action selection based on the high-level decision of the current task.

[0037] Preferably, the action is selected according to the PPO algorithm, and the high-level strategy determines the overall planning; the low-level control network uses a PID controller and calculates a control signal to adjust the joint angle according to the error between the current joint angle value and the target value corresponding to the control signal.

[0038] The present invention has the following characteristics and advantages:

[0039] (1) The present invention adopts a hierarchical policy control framework based on Proximal Policy Optimization (PPO), which divides the control task into two levels: high-level and low-level. The high-level network is responsible for learning the global policy, while the low-level network focuses on fine-grained motion control. The hierarchical design enables the manipulator to effectively learn and adapt at multiple levels, improving the control accuracy and stability, so that it can better cope with task changes in complex environments and ensure the stability and accuracy of the manipulator in dynamic scenarios.

[0040] (2) It improves the mechanical accuracy of grasping and pushing and pulling operations, and effectively solves the problem of uncoordinated movements caused by excessive coupling between fingers, thereby improving the flexibility and precision of the five-finger dexterous manipulator. In the absence of external experience data, the manipulator can autonomously learn and optimize its motion strategies in different environments. The enhanced adaptive capability enables the manipulator to better perform high-precision tasks and provides a more reliable control solution for applications in various complex scenarios.

[0041] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments, but the embodiments should not be construed as limiting the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a hierarchical strategy control algorithm framework diagram of a thumb independent optimization five-finger manipulator control method based on hierarchical strategy control of the present invention;

[0043] Figure 2 It is a network architecture diagram of a thumb-independent optimized five-finger manipulator control method based on hierarchical strategy control of the present invention. DETAILED DESCRIPTION

[0044] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0045] In order to make the technical solutions, advantages and purposes of the present invention clearer, the technical solutions of the embodiments of the present invention are clearly and completely described below. The described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of this application.

[0046] This paper proposes a thumb-independent optimization five-finger manipulator control method based on hierarchical strategy control and designs a hierarchical strategy control algorithm framework. Figure 1 As shown, the following steps are included:

[0047] S1: In the implementation process, a five-finger dexterous manipulator driven by tendons is used, through high simulation and precise control. First, the first and last three-dimensional coordinates [x1, y1, z1...x2, y2, z2] of the current target object are obtained to represent the positioning information of the object, which is used for subsequent action decision calculations.

[0048] S2, such as Figure 2 As shown, in the policy control module, PyTorch is used to divide the high-level policy network and the low-level policy network. The high-level policy network receives the state S of the target object. t As input, a two-class action probability output is generated through a three-layer fully connected neural network to decide whether to activate the thumb action. The Softmax activation function is used to calculate the action probability, and the final action decision a is generated through random sampling. thumb， The calculation formula is:

[0049] a thumb =softmax(W3·ReLu(W2·ReLu(W1·S t )))

[0050] Among them, W1, W2, W3 are the weight matrices of the network, W3 is the weight matrix shared by other fingers in the low-level strategy network, and ReLU is the activation function;

[0051] The low-level policy network uses a multi-layer fully connected neural network based on the decision of the high-level policy network and outputs the action probability of each finger joint to refine the action decision of each finger;

[0052] In S2, an independent network is designed in the low-level policy network to process the action space of the thumb, and other fingers share the same network for action decision-making. That is, the low-level policy network uses the same control strategy for other fingers and adopts an independent action strategy for the thumb.

[0053] The architecture of the low-level policy network consists of an input layer that receives the 6-dimensional features of the current state, two fully connected hidden layers of size 64, and an output layer that outputs the movement probability of each finger joint through the Softmax function. The low-level policy network controls the movement of each finger through a three-layer fully connected neural network to ensure coordination between fingers.

[0054] Among them, the thumb strategy network is:

[0055] a thumb =softmax(W thumb ·ReLu(W2·ReLu(W1·S t )));

[0056] Where W thumb is the weight matrix of the thumb policy network, which is responsible for processing the action decision of the thumb in a specific state;

[0057] Other finger strategy networks:

[0058] a finger =softmax(W3·ReLu(W2·ReLu(W1·S t ))).

[0059] S3. Use the evaluation module PPOCritic network to evaluate the effect of strategy optimization and calculate the value function of each strategy. The PPOCritic network is divided into two parts: high-level evaluation and low-level evaluation. The network is mainly used to evaluate the optimization effect of the current strategy and calculate the value function V of each state, that is, the value estimation of the current state.

[0060] The high-level evaluation network is used to evaluate the overall state of the task and output a scalar value to indicate the overall value of the current task state. The low-level evaluation network is used to evaluate the local state of the finger joints and output a scalar value to indicate the value of the finger joint state. The architectures of the high-level evaluation network and the low-level evaluation network are both composed of two fully connected layers, and the output is a scalar value to indicate the value of the corresponding state. The high-level evaluation network:

[0061] V task =f task (S t );

[0062] Low-level evaluation network:

[0063] V joint =f joint (S t ).

[0064] In a specific embodiment, the current state S is input t , the next state S after executing the action t+1 , reward signal R t , output the state value estimate:

[0065] V current =f(S t ), V next＝ f(S t+1 );

[0066] Among them, V current Indicates the current state S t Value estimate: The estimated value of the expected long-term return (long-term cumulative reward), reflecting the agent's t When , the total reward that can be obtained in the future is estimated according to the current strategy; V next Indicates the next state S t+1 The value estimate of : The next state S that the policy enters after performing an action t+1 The estimated long-term return reflects the agent's t+1 , the total reward that can be obtained in the future; f(·) represents the state value estimation function, and then the advantage function A is calculated based on the obtained parameters t for:

[0067] A t =R t +γV next –V current ;

[0068] Among them, γ is the discount factor, which represents the decay of future rewards.

[0069] In the hierarchical structure, the policy network selects the optimal action at each time step by calculating the probability distribution of each action, and decides whether to switch to the low-level policy to handle more specific action selection based on the high-level decision of the current task.

[0070] S4, training and optimization based on the PPO algorithm, wherein the optimization process uses reward signals and state value estimation to optimize the strategy through multiple iterations;

[0071] The training process is carried out in the MuJoCo simulation environment, where the agent performs the task of grasping the target object. The execution of each action is based on the current state S t The decision is made based on the action probability output by the network.

[0072] In each training round, MuJoCo updates the simulation based on the current finger movements, object positions, and the state of the manipulator, and provides environmental feedback to the PPO model. The agent selects appropriate actions based on the state information of the object, and can also provide different levels of feedback based on the different task levels, which helps the agent to learn in a more staged manner according to the task level.

[0073] S5, repeat the training process of S4 until the preset training target or convergence condition is reached;

[0074] S6. After each action is executed, the agent updates its strategy through the PPO algorithm based on the collected experience. The PPO model updates its strategy based on the experience playback (the state, action, and reward of the agent), splitting the advantage calculation, TD target, and loss function into different levels.

[0075] In S6, the specific steps of updating the strategy include:

[0076] Step 1: Use the advantage function of the PPO algorithm to calculate the difference between the expected return in the current state and the state value;

[0077] Step 2: Based on the immediate reward and the value of the next state, calculate the time difference TD target, obtain the expected return of the agent in the current state, compare the difference between the current strategy and the old strategy, and calculate the reward based on whether the round ends;

[0078] Step 3: Measure the difference between the current state value estimate and the TD target, calculate the policy loss, value loss, and entropy loss, use the entropy regularization term to encourage exploration, and control the balance between exploration and exploitation by adjusting the hyperparameters;

[0079] Step 4: Use back propagation to calculate the gradient of the loss function, and use the gradient descent method to update the parameters of the policy network, so that the PPO model can continuously adjust the strategy during training and gradually optimize the control action of the finger.

[0080] The entire training process is repeated over multiple rounds, and in each round the agent collects new experience data and updates its strategy through the PPO algorithm. In each round, the model collects information about the state, action, reward, and next state through interaction with the environment. By collecting these experience data, the strategy is updated and the training process is accelerated. By observing the fluctuation trend of the reward, it is possible to determine whether the training process has converged and whether the strategy has reached the optimal or near-optimal state.

[0081] The action is selected according to the PPO algorithm, the high-level strategy determines the overall planning, and the low-level control network uses the PID controller to calculate a control signal based on the error between the current joint angle value and the target value corresponding to the control signal to smoothly adjust the joint angle.

[0082] By continuously updating these control signals, it is ensured that the joints of each finger can smoothly transition to the target angle, avoiding the phenomenon of jumping or oscillating joint movements caused by incoherent control signals.

[0083] S7. After the training is completed, the Actor model of each finger and the strategy parameters learned during the training process are saved for subsequent reasoning or continued training;

[0084] S8. Use simulation and reasoning to verify the performance of the trained model in the simulation environment. Load the PPOActor model corresponding to each finger after training, initialize the scene through the MuJoCo simulation engine, use the loaded PPO model to select actions and apply them to the simulation environment, and simulate and evaluate the grasping success rate as an evaluation of the specific effect.

[0085] In this embodiment, the training process is carried out in the MuJoCo simulation environment, using a 22-DOF tendon-driven five-fingered dexterous manipulator. The manipulator passes through the state S of the target object. t Select the best grasping action, based on the current action a t , the reward R returned after execution t , and update the next state S t+1 Calculate the training error. The optimization target is updated by calculating the probability distribution of the current action and the state value, using the objective function:

[0086] L(θ)=E t [min(r t (θ)At,clip(r t (θ),1-∈,1+∈)A t )];

[0087] Among them, r t (θ) is the probability ratio of the current strategy to the old strategy:

[0088]

[0089] ∈ is a small constant used to limit the scope of the update, avoid excessive policy updates, and ensure the stability of training.

[0090] In addition, the entropy regularization term is introduced into the loss function to increase the exploration ability and avoid falling into the local optimum:

[0091]

[0092] In the formula, H(π θ ) is the entropy regularization term, a represents the current action a t .

[0093] Compared with other fingers, the joint actions of the thumb in the loss function have higher weights and entropy coefficients, which allows the thumb to explore more during optimization and makes the strategy update more flexible during the optimization process.

[0094] The final loss function is:

[0095] L total =L(θ)-βH(π θ );

[0096] Among them, β is a hyperparameter used to balance the policy loss and the entropy regularization term.

[0097] According to the above process, the training method effectively trained the tendon-driven five-fingered dexterous manipulator through the hierarchical policy control framework and the PPO reinforcement learning algorithm. Based on the uniqueness of thumb control, an independent policy network was designed, which enabled the thumb to make independent decisions in grasping tasks and coordinate with other fingers, improving control accuracy and flexibility. Combining hierarchical strategies, efficient training in the MuJoCo simulation environment, and a balance between exploration and stability, the control method has strong generalization ability and can adapt to a variety of grasping tasks, making effective use of the training of the five-fingered dexterous manipulator in the simulation environment.

[0098] Therefore, the present invention proposes a method for controlling a five-finger manipulator with thumb independent optimization based on hierarchical policy control. The present invention adopts a hierarchical policy control framework based on Proximal Policy Optimization (PPO), and optimizes the control task by dividing it into two levels, high-level and low-level. The high-level network is responsible for learning the global policy, while the low-level network focuses on fine-grained motion control. The hierarchical design enables the manipulator to effectively learn and adapt at multiple levels, improving control accuracy and stability, so that it can better cope with task changes in complex environments and ensure the stability and accuracy of the manipulator in dynamic scenarios.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A thumb-independent optimized five-finger manipulator control method based on hierarchical strategy control, characterized in that: Here are the steps: S1. Design a hierarchical strategy control algorithm framework to obtain the current target object's positioning information for subsequent action decision calculations; S2. Use PyTorch to divide the high-level policy network and the low-level policy network. The high-level policy network receives the state of the target object as input, and generates a binary action probability output through a three-layer fully connected neural network to decide whether to activate the thumb action; the low-level policy network is based on the decision of the high-level policy network, uses a multi-layer fully connected neural network and outputs the action probability of each finger joint to refine the action decision of each finger; S3. Use the evaluation module PPOCritic network to evaluate the effect of strategy optimization and calculate the value function of each strategy. The PPOCritic network is divided into two parts: high-level and low-level evaluation. S4, training and optimization based on the PPO algorithm, wherein the optimization process uses reward signals and state value estimation to optimize the strategy through multiple iterations; S5, repeat the training process of S4 until the preset training target or convergence condition is reached; S6. After each action is executed, the agent updates its strategy through the PPO algorithm based on the collected experience; S7. After the training is completed, the Actor model of each finger and the strategy parameters learned during the training process are saved for subsequent reasoning or continued training; S8. Use simulation and reasoning to verify the performance of the trained model in the simulation environment.

2. According to claim 1, a thumb-independent optimized five-finger manipulator control method based on hierarchical strategy control is characterized in that: In S2, the high-level policy network architecture consists of an input layer that accepts the 6-dimensional features of the state, two layers of 64-dimensional fully connected hidden layers using the ReLU activation function, and a binary classification output layer. The high-level policy network uses the Softmax activation function to calculate the action probability and generates the final action decision a through random sampling. thumb , the calculation formula is as follows: your thumb =softmax(W3·ReLu(W2·ReLu(W1·St))); Among them, W1, W2, and W3 are the weight matrices of the network, W3 is the weight matrix shared by other fingers in the low-level strategy network, and ReLU is the activation function.

3. According to claim 1, a thumb-independent optimized five-finger manipulator control method based on hierarchical strategy control is characterized in that: In S2, an independent network is designed in the low-level policy network to process the action space of the thumb, while other fingers share the same network for action decision making.

4. According to claim 1, a thumb-independent optimized five-finger manipulator control method based on hierarchical strategy control is characterized in that: In S2, the architecture of the low-level strategy network consists of an input layer that receives the 6-dimensional features of the current state, two fully connected hidden layers of size 64, and an output layer that outputs the action probability of each finger joint through a Softmax function, where the thumb strategy network is: to thumb =softmax(W thumb ·ReLu(W2·ReLu(W1·S t ))); Where W thumb is the weight matrix of the thumb policy network, which is responsible for processing the action decision of the thumb in a specific state; Other finger strategy networks: your finger =softmax(W3·ReLu(W2·ReLu(W1·S t )))。 5. According to claim 1, a thumb-independent optimized five-finger manipulator control method based on hierarchical strategy control is characterized in that: In S3, the high-level evaluation network is used to evaluate the overall state of the task and output a scalar value to indicate the overall value of the current task state; the low-level evaluation network is used to evaluate the local state of the finger joints and output a scalar value to indicate the value of the finger joint state; the architecture of the high-level evaluation network and the low-level evaluation network are both composed of two fully connected layers, and the output is a scalar value to indicate the value of the corresponding state, where the high-level evaluation network: V task =f task (S t ); Low-level evaluation network: V joint =f joint (S t )。 6. According to claim 1, a thumb-independent optimized five-finger manipulator control method based on hierarchical strategy control is characterized in that: In S4, the training process is carried out in the MuJoCo simulation environment in which the environment and the agent (robot finger) interact. The agent also performs the task of grasping the target object. The execution of each action is based on the current state S t The decision is made based on the action probability output by the network.

7. The method for controlling a five-finger manipulator with thumb independent optimization based on hierarchical strategy control according to claim 1 is characterized in that: In S6, the PPO model updates the strategy based on experience replay, splitting the advantage calculation, TD target, and loss function into different levels; the experience replay includes the state, action, and reward of the agent.

8. The method for controlling a five-finger manipulator with thumb independent optimization based on hierarchical strategy control according to claim 1 is characterized in that: In S6, the specific steps of updating the strategy include: Step 1: Use the advantage function of the PPO algorithm to calculate the difference between the expected return in the current state and the state value; Step 2: Based on the immediate reward and the value of the next state, calculate the time difference TD target, obtain the expected return of the agent in the current state, compare the difference between the current strategy and the old strategy, and calculate the reward based on whether the round ends; Step 3: Measure the difference between the current state value estimate and the TD target, calculate the policy loss, value loss, and entropy loss, use the entropy regularization term to encourage exploration, and control the balance between exploration and exploitation by adjusting the hyperparameters; Step 4: Use back propagation to calculate the gradient of the loss function, and use the gradient descent method to update the parameters of the policy network, so that the PPO model can continuously adjust the strategy during training and gradually optimize the control action of the finger.

9. The method for controlling a five-finger manipulator with thumb independent optimization based on hierarchical strategy control according to claim 1 is characterized in that: In the hierarchical structure, the policy network selects the optimal action at each time step by calculating the probability distribution of each action, and decides whether to switch to the low-level policy to handle more specific action selection based on the high-level decision of the current task.

10. The method for controlling a five-finger manipulator with thumb independent optimization based on hierarchical strategy control according to claim 1 is characterized in that: The action is selected according to the PPO algorithm, and the high-level strategy determines the overall planning; the low-level control network uses the PID controller and calculates a control signal to adjust the joint angle based on the error between the current joint angle value and the target value corresponding to the control signal.

Images