Reinforcement learning control method with optimal global performance for smart operation of robot
By employing reinforcement learning and recursive control strategies, the optimal angle for robot dexterity is obtained, solving the global optimal control problem for robot dexterity and improving both dexterity and stability.
Patent Information
- Application Number
- CN202511523395.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-23
- Publication Date
- 2026-01-16
AI Technical Summary
Existing technologies struggle to achieve globally optimal control for dexterous robot operation in the absence of prior knowledge of the task. Traditional methods can easily lead to robots getting stuck in local areas of extremely high operability rather than optimal values.
A reinforcement learning method is used to obtain the optimal angle strategy for the robot's current control cycle. By combining the end-point tracking error and the desired trajectory, a real-time control model is established, and the real-time control quantity is obtained through a recursive control strategy to avoid getting trapped in local optima.
It achieves global optimization of robot dexterity, ensuring the system's dexterity and stability, avoiding local optima traps, and possessing theoretical traceability of dexterity performance.
Smart Images
Figure CN121340255A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to robot control technology, specifically to a reinforcement learning control method for achieving optimal global performance in robot dexterity operation. Background Technology
[0002] With the development of robotics, especially humanoid robots, dexterous and precise manipulation has become an important research direction and task requirement for robotic operations. Especially in uncertain and complex task scenarios, high-precision dexterous manipulation is needed even without prior task knowledge. Traditional control methods to ensure robot flexibility mainly fall into two categories: one is a planning and control method based on offline programming, but this method is difficult to apply when there is a lack of prior task knowledge. The second method typically uses indicators such as maneuverability as guidance, with gradient descent as its core. It introduces a control component based on the maneuverability gradient while controlling the robot to optimize its flexibility. However, this type of gradient-based method often leads the system into local optima rather than global optima. Summary of the Invention
[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a reinforcement learning control method with optimal global performance for robot dexterity operation, so that the robot end effector can perform operation tasks with high precision and achieve global optimization of operability.
[0004] To achieve the above objectives, the technical solution of the present invention is as follows: A reinforcement learning control method for achieving optimal global performance in robot dexterity manipulation includes: The optimal angle strategy for dexterity in the current control cycle of the robot is obtained by using reinforcement learning methods. The corrected speed command for the robot's end effector is obtained based on the robot's end-effector tracking error and the desired trajectory of dexterous manipulation. A real-time control model is established based on the dexterity-optimal angle strategy of the current control cycle and the correction speed command of the robot end effector. Based on the recursive control strategy, the real-time control quantity of the real-time control model is obtained.
[0005] Optionally, the optimal angle strategy for dexterity in the current control cycle of the robot is obtained through reinforcement learning: Within each job optimization cycle, the optimal angle strategy for the current control cycle that satisfies the robot's job accuracy requirements and has the best global flexibility index is obtained through reinforcement learning.
[0006] Optionally, the step of finding the optimal robot configuration that satisfies the robot's operational accuracy requirements and has the best global flexibility index in each job optimization cycle through reinforcement learning includes: Reinforcement learning state space Defined as:
[0007] in, This represents the position of the end effector in the base coordinate system. Action space Defined as:
[0008] in, For the first The target angle of each joint is directly output by the Actor network; These are the upper and lower limits of the joint angle; reward function Designed as follows:
[0009] in, These are the weighting coefficients. The distance between the end effector and the target position is the error distance. For flexibility index functions; Q-value update in Critic network:
[0010] in, The loss function for the Critic network is... For the parameters of the Critic and Actor networks, , For the target network parameters of Critic and Actor, Represents the expected value, indicating the expected value for the experience replay pool. The average of the middle sample; For the Critic network's Q-value prediction of state-action pairs, The target Q value; As a discount factor, it balances current and future rewards. The immediate reward value; the Critic learns accurate action value assessment by minimizing temporal difference error; the Actor's policy gradient is:
[0011] in, Let the gradient of the policy objective function with respect to the Actor parameters be denoted as . The deterministic motion, i.e., joint angles, output by the Actor network. The gradient of the Q-value with respect to the action. The Actor network outputs the gradient with respect to its own parameters; the Actor network adjusts its strategy along the Q-value growth direction to select the joint angle combination that results in a higher Critic score. An exploration mechanism is introduced, initially exploring a wide range of joint torque combinations, then gradually converging to a stable strategy, ultimately obtaining the angle strategy with optimal dexterity for the current control cycle. .
[0012] Optionally, the introduced exploration mechanism is as follows:
[0013]
[0014] in, For time t noise variance at time, These represent the upper and lower boundaries of the noise variance. It is a decaying exponential function. The attenuation coefficient controls the rate at which noise decreases. To explore noise, it follows a normal distribution.
[0015] Optionally, obtaining the corrected speed command for the robot's end effector based on the robot's end-effector tracking error and the desired trajectory of dexterous manipulation includes: The robot's end-effector tracking error is The expected trajectory of dexterous manipulation is , The derivative is The corrected speed command obtained from the robot's end effector is: , This is a positive control parameter.
[0016] Optionally, the real-time control model is:
[0017] For the joint angles of the robot, for The angular velocity of the robot's joints.
[0018] Optionally, based on the recursive control strategy, the real-time control quantity of the real-time control model is obtained as follows:
[0019] in, For the joint angular acceleration of the robot, As dual variables, It is a positive control parameter. for The derivative of .
[0020] Compared with the prior art, the advantages of this invention are as follows: The method of this invention establishes a real-time control model based on the optimal angle strategy for dexterity in the current control cycle and the corrected speed command of the robot's end effector. It obtains the real-time control quantity of the real-time control model based on a recursive control strategy. Under the premise of ensuring dexterity operation of the end effector, it achieves global optimization of robot operability, thereby ensuring the dexterity of the robot system. It can effectively avoid the robot getting trapped in local operability maxima instead of optimal values due to traditional gradient-based control methods. Compared with traditional reinforcement learning methods, it has the advantages of theoretical traceability of dexterity operation performance and stability guarantee. Attached Figure Description
[0021] Figure 1 The main flowchart of the reinforcement learning control method for achieving optimal global performance of robot dexterity operation provided in the embodiments of this application is shown. Detailed Implementation
[0022] Example: The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0023] See Figure 1 As shown, the reinforcement learning control method for optimal global performance of robot dexterity operation provided in this embodiment mainly includes the following steps: The optimal angle strategy for dexterity in the current control cycle of the robot is obtained by using reinforcement learning methods. In this step, reinforcement learning is used to obtain the optimal angle strategy for the robot's current control cycle. This effectively avoids the problem of the robot getting stuck in a local operability maxima instead of the optimal value caused by traditional gradient-based control methods. It has the advantages of theoretically traceable dexterity performance and guaranteed stability.
[0024] 120. The corrected speed command for the robot's end effector is obtained based on the robot's end-effector tracking error and the desired trajectory of dexterous operation.
[0025] 130. A real-time control model is established based on the dexterity-optimal angle strategy of the current control cycle and the correction speed command of the robot end effector.
[0026] 140. Based on the recursive control strategy, the real-time control quantity of the real-time control model is obtained.
[0027] Therefore, this method establishes a real-time control model based on the optimal angle strategy for dexterity in the current control cycle and the corrected speed command of the robot's end effector. It also obtains the real-time control quantity of the real-time control model based on a recursive control strategy. Under the premise of ensuring dexterity operation at the end effector, it achieves global optimization of the robot's maneuverability, thereby ensuring the dexterity of the robot system. It can also effectively avoid the robot getting trapped in local maneuverability maxima rather than optimal values due to traditional gradient-based control methods. Compared with traditional reinforcement learning methods, it has the advantages of theoretical traceability of dexterity operation performance and stability assurance.
[0028] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.
[0029] In one specific embodiment, the strategy for obtaining the optimal dexterity angle for the current control cycle of the robot using reinforcement learning includes: Within each job optimization cycle, reinforcement learning is used to find the optimal angle strategy for the current control cycle that satisfies the robot's job accuracy requirements and has the best global flexibility index. More specifically, this includes: DDPG (Deep Deterministic Policy Gradient) reinforcement learning state space Defined as:
[0030] in, This represents the position of the end effector in the base coordinate system.
[0031] Action space is defined as:
[0032] in, For the first The target angles of each joint are directly output by the Actor network. These are the upper and lower limits of the joint angle.
[0033] The reward function is designed as follows:
[0034] in, These are the weighting coefficients. The distance between the end effector and the target position is the error distance. This is a flexibility index function.
[0035] Q-value update in Critic network:
[0036] in, The loss function for the Critic network is... For the parameters of the Critic and Actor networks, , For the target network parameters of Critic and Actor, Represents the expected value, indicating the expected value for the experience replay pool. The average of the middle sample; For the Critic network's Q-value prediction of state-action pairs, The target Q value. As a discount factor, it balances current and future rewards. The immediate reward value; the Critic learns accurate action value assessment by minimizing temporal difference error; the Actor's policy gradient is:
[0037] in, Let the gradient of the policy objective function with respect to the Actor parameters be denoted as . The deterministic motion, i.e., joint angles, output by the Actor network. The gradient of the Q-value with respect to the action. The Actor network outputs gradients with respect to its own parameters. The Actor network adjusts its strategy along the Q-value growth direction, selecting joint angle combinations that result in higher Critic scores.
[0038] Introducing an exploration mechanism to prevent getting trapped in local optima:
[0039]
[0040] in, For time t noise variance at time, These represent the upper and lower boundaries of the noise variance. It is a decaying exponential function. The attenuation coefficient controls the rate at which noise decreases. To explore noise, a normal distribution is followed. Initially, a wide range of joint torque combinations are explored, gradually converging to a stable strategy, ultimately obtaining the dexterity-optimal angle strategy for the current control cycle. .
[0041] Thus, based on the above operations, it is possible to effectively avoid the robot getting stuck in a local operability maxima rather than the optimal value due to traditional gradient-based control methods. At the same time, compared with traditional reinforcement learning methods, it has the advantages of theoretically traceable dexterous operation performance and guaranteed stability.
[0042] In one specific embodiment, obtaining the corrected speed command for the robot's end effector based on the robot's end-effector tracking error and the desired trajectory of dexterous manipulation includes: To address the precision requirements of dexterous manipulation, the robot's end-effector tracking error is defined as... The expected trajectory of dexterous manipulation is , The derivative is The corrected speed command for the robot's end effector is designed as follows: ;in, This is a positive control parameter.
[0043] In this way, through the above operations, the corrected speed command of the robot end effector (equivalent to the robot end effector controller) is obtained, thereby ensuring that the robot end effector can perform the operation task with high precision.
[0044] Finally, a real-time control model is established based on the optimal angle strategy for dexterity in the current control cycle and the corrective speed command from the robot's end effector.
[0045] For the joint angles of the robot, for The angular velocity of the robot's joints.
[0046] Based on the recursive control strategy, the real-time control quantity of the real-time control model is obtained as follows:
[0047] in, For the joint angular acceleration of the robot, As dual variables, It is a positive control parameter. for The derivative of .
[0048] In this way, we can obtain the reference configuration (i.e., the real-time control quantity) that is globally optimal in terms of dexterity, and achieve global optimization of operability through the reference configuration.
[0049] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.
Claims
1. A reinforcement learning control method for robot dexterous manipulation global performance optimization, characterized in that, The method comprises the following steps: obtaining a dexterity optimal angle strategy of a current control cycle of the robot by a reinforcement learning method; obtaining a corrected speed instruction of the robot end based on an end tracking error of the robot and a desired trajectory of the dexterous operation; establishing a real-time control model according to the dexterity optimal angle strategy of the current control cycle and the corrected speed instruction of the robot end; obtaining a real-time control amount of the real-time control model based on a recursive control strategy. 2.The robot dexterous operation global performance optimal reinforcement learning control method of claim 1, wherein, The method of obtaining the dexterity optimal angle strategy of the current control cycle of the robot by the reinforcement learning method comprises the following steps: in each operation optimization cycle, a dexterity optimal angle strategy of a current control cycle that meets the robot operation accuracy requirement and has an optimal global flexibility index is obtained by a reinforcement learning method. 3.The robot dexterous operation global performance optimal reinforcement learning control method of claim 2, wherein, The method of obtaining the dexterity optimal angle strategy of the current control cycle that meets the robot operation accuracy requirement and has the optimal global flexibility index in the current cycle by the reinforcement learning method in each operation optimization cycle comprises the following steps: Reinforcement learning state space is defined as: ； wherein, is the position of the end effector in the base coordinate frame; Action space is defined as: ； wherein, is the target angle of the jth joint directly output by the Actor network; is the target angle of the jth joint directly output by the Actor network; are the upper and lower limits of the joint angle. Reward function Designed to: ； wherein, is a weight coefficient, is an error distance of the end effector from the target position, is a flexibility index function; Q value updating in the Critic network: ； where, is the Critic network loss function, are the Critic and Actor network parameters, , are the Critic and Actor target network parameters, represents the expected value, indicating the average over samples from the experience replay pool is the Q-value prediction of the Critic network for a state-action pair, is the target Q-value; is the discount factor, balancing current and future rewards, is the immediate reward value; the Critic learns an accurate action value estimate by minimizing the temporal difference error; the policy gradient of the Actor is: ； wherein, is the gradient of the policy objective function with respect to the actor parameters, is the deterministic action, i.e. joint angles, output by the actor network, is the gradient of the Q value with respect to the action, is the gradient of the actor network output with respect to its own parameters; the actor network adjusts the policy in the direction of increasing Q value, selecting joint angle combinations that result in higher critic scores. The exploration mechanism is introduced to explore the joint torque combination in a large range in the initial stage, to gradually converge to a stable strategy in the later stage, and to finally obtain the dexterity optimal angle strategy in the current control period . 4.The robot dexterous operation global performance optimal reinforcement learning control method of claim 3, wherein, The exploration mechanism comprises the following steps: ； ； where, is the time t is the noise variance at time is the upper and lower bound of the noise variance, is the decay exponential function, is the decay coefficient, controlling the speed of noise reduction, is the exploration noise, subject to a normal distribution.
5. The robot dexterous manipulation global performance optimal reinforcement learning control method of claim 4, wherein, The method of obtaining the corrected speed instruction of the robot end based on the end tracking error of the robot and the desired trajectory of the dexterous operation comprises the following steps: The end tracking error of the robot is , the desired trajectory of the dexterous operation is , , the derivative of the desired trajectory of the dexterous operation is , and the modified velocity command of the end of the robot is , is a positive control parameter.
6. The robot dexterous manipulation global performance optimal reinforcement learning control method of claim 5, wherein, The real-time control model comprises the following steps: ； joint angle of the robot, The method of obtaining the real-time control amount of the real-time control model based on the recursive control strategy comprises the following steps: joint angular velocity of the robot.
7. The robot dexterous manipulation global performance optimal reinforcement learning control method of claim 6, wherein, ； wherein is the joint angle acceleration of the robot, is the dual variable, is the positive control parameter, is the derivative.