Multi-style parallel intelligent strategy learning method for wheel-legged humanoid robot

Abstract

The invention discloses a multi-style parallel intelligent strategy learning method for a wheel-legged humanoid robot, and the method comprises the steps: carrying out robot simulation training on a computer, training parallel strategies A and B through independent customized observation values A and B and customized rewards A and B, and obtaining optimal parallel strategies A and B through parallel reinforcement learning processes A and B; the optimal parallel strategies A and B are implanted into a robot industrial personal computer in a neural network mode, forward propagation operation is carried out, local skills A and B cooperating with each other are generated respectively, and after the local skills A and B are converted through an arm joint PD controller and a wheel-leg joint PD controller, the robot obtains an agile auxiliary arm skill A and a basic robustness wheel-leg skill B. A special exploration space is provided for the arm, the defect of over-fitting of the arm can be effectively avoided, and consumption and conflicts among different targets can be avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of wheel-legged robots, and in particular relates to a multi-style parallel intelligent strategy learning method for wheel-legged humanoid robots. Background Art

[0002] The reinforcement learning theory was first proposed by Google's DEEPMIND company and turned into a technology commercial product. It aims to enable the controlled object to interact with the environment continuously, gain useful experience from it, abandon harmful decisions, and ultimately achieve continuous self-tuning of the controlled object. The first well-known product of this theory is the AlphaGo Go robot that defeated the human champion.

[0003] In the context of reinforcement learning, since the arms of the wheel-legged humanoid robot do not directly serve as actuators for balancing movements, the (arm's) intelligent exploration space is squeezed by the wheel-legged balancing task. The current single-strategy reinforcement learning framework is very prone to strategy overfitting, which is manifested as the arms converging to strange positions and not participating in auxiliary center of gravity adjustment tasks.

[0004] In addition, the wheel-legged humanoid robot is a complex multi-rigid body system. The existing single-strategy reinforcement learning framework is not compatible with reward functions that are too complex, which results in inevitable conflicts and consumption between different reward function items. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a multi-style parallel intelligent strategy learning method for a wheel-legged humanoid robot.

[0006] The present invention achieves the above technical objectives through the following technical means.

[0007] Multi-style parallel intelligent strategy learning method for wheel-legged humanoid robot:

[0008] Conduct robot simulation training on a computer, rely on independent customized observations A and B, customized rewards A and B, train parallel strategies A and B, and obtain the optimal parallel strategies A and B through parallel reinforcement learning processes A and B;

[0009] The optimal parallel strategies A and B are implanted into the robot's industrial computer in the form of neural networks, and forward propagation operations are performed to generate local skills A and B that collaborate with each other. After conversion by the arm joint PD controller and the wheel-leg joint PD controller, the robot acquires the agile auxiliary arm skill A and the basic robust wheel-leg skill B.

[0010] Furthermore, the customized reward A is obtained by weighted summation of the contact force distribution reward, the recovery distance reward, and the arm basic reward.

[0011] Furthermore, the contact force distribution reward is:

[0012]

[0013] in, represents the contact force distribution reward, and are the support forces of the robot’s left and right wheels on the ground, and K A is the kernel function.

[0014] Furthermore, the recovery distance reward is:

[0015]

[0016] in, Indicates the recovery distance reward, and are the average speed of the robot's two wheels at time t and the reference average speed respectively.

[0017] Furthermore, the basic reward of the arm is:

[0018]

[0019] in, Represents the basic reward of the arm, K A is the kernel function, φ i arm is the actual position of the i-th joint of the current arm, is the desired position of the arm's joint No. i, U is the robot's rated voltage, τ i arm is the torque of the arm joint No. i, k i is the torque-current proportionality factor, P torso is the robot upper body posture vector.

[0020] Furthermore, the customized reward B is obtained by weighted summing the wheel speed following reward, the wheel-leg posture reward, and the altitude maintaining reward.

[0021] Furthermore, the customized observation value A is the robot's torso posture, arm joint position and speed, and six-dimensional force sensor value.

[0022] Furthermore, the customized observation value B is the position and speed of the whole body joints of the wheel-legged robot.

[0023] Furthermore, the parallel strategy is specifically a neural network, and its training and working processes are both Markov decision processes.

[0024] Furthermore, the training and operation of parallel strategies A and B are parallel Markov decision processes:

[0025]

[0026] Among them, π * A and π * B are the optimal reinforcement learning strategies for controlling the robot arm and wheel leg skills, respectively. a and b represent the ordinal variables of A and B, respectively. t is the current time step of reinforcement learning. At and r Bt are the rewards at time step t in two parallel Markov decision processes, γ t A represents the use of the reward discount factor of strategy A to reduce the impact of future rewards, γ t B represents the use of the reward discount factor of strategy B to reduce the impact of future rewards, and γ t A , γ t B ∈(0,1).

[0027] The beneficial effects of the present invention are:

[0028] (1) The present invention obtains the agile auxiliary arm skill A from the parallel strategy A, provides a dedicated exploration space for the arm, and can effectively avoid the defect of overfitting of the arm; the present invention provides a set of additional, customized rewards for the arm, which can avoid conflicts with the wheel-leg motion reward items, and finally provides a stable, diverse, and reasonable parallel motion control strategy for the wheel-leg humanoid robot.

[0029] (2) The parallel strategy of the present invention allows users to customize their movement styles (including observation values ​​and reward symbols) according to their needs. The customized style can give full play to the exploration performance of reinforcement learning while avoiding ablation (consumption) and conflict between different goals, and ultimately achieve collaborative and unified parallel local skills. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a block diagram of the multi-style parallel intelligent strategy learning method for the wheel-legged humanoid robot described in the present invention. DETAILED DESCRIPTION

[0031] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0032] Markov decision process (MDP) is the core model of reinforcement learning tasks. A single reinforcement learning framework only needs to establish an MDP for the overall motion. In the present invention, a parallel Markov decision process PX-MDPs is proposed and established in the same multi-rigid body system (i.e., wheel-legged humanoid robot):

[0033]

[0034] Among them, π * A and π * B are the optimal reinforcement learning strategies (in the form of fully connected neural networks) for controlling the robot's arm and wheel-leg skills, respectively; t is the current time step of reinforcement learning; γ is the reward discount factor, which is used to reduce the impact of rewards in the distant future; γ t A represents the use of the reward discount factor of strategy A to reduce the impact of future rewards, γ t B represents the use of the reward discount factor of strategy B to reduce the impact of future rewards, and γ t A , γ t B ∈(0,1); a and b represent the ordinal variables of A and B respectively; r At and r Bt They are the rewards in the two parallel MDPs at time step t. The expectation (E) of the cumulative rewards can evaluate the optimization degree of the strategy and finally explore the optimal parallel strategy.

[0035] like Figure 1 As shown, robot simulation training is performed on a high-performance workstation (i.e., a computer), and parallel strategies (A and B) are trained based on independent customized observations (A and B) and customized rewards (A and B). Specifically, the customized observation A can be the robot trunk posture, arm joint position and speed, and six-dimensional force sensor value, while the customized observation B can be the position and speed of the whole body joints of the wheel-leg robot. In addition, the specific definition of the customized reward (A and B) will be explained in the form of a formula below. Under a unified time scale, through the parallel reinforcement learning process (A and B), the optimal parallel strategy (A and B) is finally obtained. After that, the trained parallel strategy (A and B) is implanted into the robot industrial computer in the form of a neural network. Using the C++ programming language, the parallel strategy (fully connected neural network form) is forward propagated under multi-threaded conditions, thereby generating local skills (A and B) that cooperate with each other. After the conversion of the arm joint PD controller and the wheel-leg joint PD controller, the robot obtains the agile auxiliary arm skill A and the basic robust wheel-leg skill B respectively.

[0036] Specifically, the significance of customized observation values ​​is that parallel strategies (A and B) can observe the robot states that users prefer according to their needs, avoid redundant observations of the whole body state, and optimize observation efficiency and rationality; in addition, customized rewards (A and B) can set independent and mutually balanced reward styles to avoid conflicts in the parallel strategy training process and achieve more robust whole body coordination characteristics. In the specific implementation process, the parallel strategy proposed in the present invention provides more state space and style space for reward design; more importantly, through the above customized observation values ​​(A and B) and customized rewards (A and B), parallel strategies (A and B) can be trained through the reinforcement learning process (A and B) and obtain different local skills of wheel-legged humanoid robots, and the independence and collaboration of these local skills can be adjusted through customization of different styles. The customization of different styles is reflected in the customization of observation values ​​and rewards. The customization of observation values ​​is selected according to user needs, and the customization of rewards is the adjustment of weights and constants in each reward item.

[0037] As for the specific implementation process, the present invention uses the PhysX physics engine optimized for the wheel-legged humanoid robot to realize robot dynamics simulation (NVIDIA open source engine, the present invention optimizes and models the wheel-legged humanoid robot) in the simulation training stage (high-performance workstation simulation training), and adopts python and c++ language programming to integrate the physical dynamics simulation data and the reinforcement learning training framework to achieve collaborative work, and finally make the parallel strategies (A and B) in the form of neural networks converge separately and realize collaborative whole-body control.

[0038] The specific modeling process is to build the same digital twin model in a high-performance workstation simulation environment according to the limb size, mass, inertia, joint friction and degree of freedom configuration of the real robot. High-precision simulation modeling helps to smoothly implement the reinforcement learning process; for the "wheel-ground" contact characteristics of the wheel-legged robot, the mesh contact characteristics of PhysX are optimized, and the polygonal characteristics of the wheels are removed to make the simulation closer to the contact characteristics of the actual wheels.

[0039] In the implementation process, customized rewards are a very critical link. The following is the customized arm reward proposed by the present invention, and its preferred styles are: agility, collaboration and low power consumption. Specifically:

[0040] First, the present invention uses a kernel function to make the reward term bounded:

[0041]

[0042] Among them, e x is a natural exponential function, and the most common exp notation is used below.

[0043] Then there is the support distribution reward, which aims to help the lower limb end support force achieve dynamic balance through the autonomous adjustment of the arm. and The support forces of the left and right wheels are:

[0044]

[0045] The next item is the recovey distance reward, which aims to restore the robot to a controllable balance state as quickly as possible by adjusting the center of gravity of the arm when the robot is subjected to external collision or disturbance. and are the average speed of the two wheels at time t and the reference average speed, in rad / s. It should be noted that in order to skip the acceleration process of the robot caused by disturbance, The sampling time is t = 0.2s when the disturbance occurs.

[0046]

[0047] The last item is the arm base reward, which aims to guide the arm to autonomously learn reasonable auxiliary movements when encountering disturbances, while consuming as little energy as possible and taking full account of the overall balance; is the desired position of the i-th joint of the arm, set by the user; φ i arm is the actual position of the i-th joint of the current arm; k i is the torque-current proportionality coefficient, U is the rated voltage of the robot, τ i arm is the torque of the arm's joint No. i; P torso is the robot upper body posture vector.

[0048]

[0049] The contact force distribution reward, recovery distance reward, and arm basic reward are weighted summed to get the customized arm reward:

[0050]

[0051] Among them, w A1 、w A2 、w A3 are the weights of the corresponding reward items.

[0052] As for customized wheel-leg rewards, the emphasis is on overall balance, and the goal is very clear.

[0053] First, use a kernel function similar to the one used in the customized arm reward to perform bounded processing:

[0054]

[0055] Then, the wheel speed following reward is introduced, with the user command v * To achieve the desired result, the actual wheel speed is as close to the user's instruction as possible:

[0056]

[0057] Where v represents the linear velocity vector of the robot's center of mass at the current moment;

[0058] In addition, in order to maintain the balance of the robot's torso posture, the three-dimensional values ​​of roll, pitch, and yaw of the posture sensor are used here to construct the wheel-leg posture reward:

[0059]

[0060] Finally, in order to make the robot maintain a reasonable torso height, a height maintenance reward is introduced, where h * torso is the desired torso height, h torso is the actual current torso height of the robot:

[0061]

[0062] The wheel speed following reward, wheel leg posture reward, and altitude holding reward are weighted summed to obtain the customized wheel leg reward:

[0063]

[0064] Among them, w B1 、w B2 、w B3 are the weights of the corresponding reward items.

[0065] The training process of the neural network uses reinforcement learning methods, specifically the PPO algorithm (Proximal Policy Optimization). Through the actor-critic training method and the policy gradient optimization method, the parallel strategy (i.e., the fully connected neural network) is continuously trained until the reward convergence requirement is met. In this case, the parallel strategy is considered to have learned the specified skills.

[0066] The input of the neural network is customized observation values ​​(A and B), and the output is the target value for the arm joint PD controller and the wheel-leg joint PD controller. After the above two types of PD controllers, it is finally converted into joint torque. After continuous execution over time, local skills are obtained. Because of the strong intelligence and exploratory nature of the reinforcement learning method, local skills are naturally collaborative. At the same time, the design of parallel strategies (A and B) also makes local skills independent.

[0067] According to the parallel strategy and customized style proposed in the present invention, the wheel-legged humanoid robot can acquire excellent upper and lower limb local skills, and achieve stable balance movements and agile arm-assisted movements through collaborative work.

[0068] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.

Claims

1. A multi-style parallel intelligent strategy learning method for a wheel-legged humanoid robot, characterized by: Conduct robot simulation training on a computer, rely on independent customized observations A and B, customized rewards A and B, train parallel strategies A and B, and obtain the optimal parallel strategies A and B through parallel reinforcement learning processes A and B; The optimal parallel strategies A and B are implanted into the robot's industrial computer in the form of neural networks, and forward propagation operations are performed to generate local skills A and B that collaborate with each other. After conversion by the arm joint PD controller and the wheel-leg joint PD controller, the robot acquires the agile auxiliary arm skill A and the basic robust wheel-leg skill B.

2. The multi-style parallel intelligent strategy learning method for wheel-legged humanoid robots according to claim 1 is characterized in that: The customized reward A is obtained by weighted summing the contact force distribution reward, the recovery distance reward, and the arm basic reward.

3. The multi-style parallel intelligent strategy learning method for wheel-legged humanoid robots according to claim 2 is characterized in that: The contact force distribution reward is: in, represents the contact force distribution reward, and are the support forces of the robot’s left and right wheels on the ground, K A is the kernel function.

4. The multi-style parallel intelligent strategy learning method for wheel-legged humanoid robots according to claim 2 is characterized in that: The recovery distance reward is: in, Indicates the recovery distance reward, and are the average speed of the robot's two wheels at time t and the reference average speed respectively.

5. The multi-style parallel intelligent strategy learning method for wheel-legged humanoid robots according to claim 2 is characterized in that: The basic rewards for the arm are: in, Represents the basic reward of the arm, K A is the kernel function, is the actual position of the i-th joint of the current arm, is the desired position of the arm's joint No. i, U is the robot's rated voltage, τ i arm is the torque of the arm joint No. i, k i is the torque-current proportionality factor, P torso is the robot upper body posture vector.

6. The multi-style parallel intelligent strategy learning method for wheel-legged humanoid robots according to claim 1 is characterized in that: The customized reward B is obtained by weighted summing the wheel speed following reward, the wheel-leg posture reward, and the height keeping reward.

7. The multi-style parallel intelligent strategy learning method for wheel-legged humanoid robots according to claim 1 is characterized in that: The customized observation value A is the robot's torso posture, arm joint position and speed, and six-dimensional force sensor value.

8. The multi-style parallel intelligent strategy learning method for wheel-legged humanoid robots according to claim 1 is characterized in that: The customized observation value B is the position and speed of the joints of the wheel-legged robot.

9. The multi-style parallel intelligent strategy learning method for wheel-legged humanoid robots according to claim 1, characterized in that: The parallel strategy is specifically a neural network, and its training and working processes are both Markov decision processes.

10. The multi-style parallel intelligent strategy learning method for wheel-legged humanoid robots according to claim 9, characterized in that: The training and operation process of parallel strategies A and B is a parallel Markov decision process: Among them, π * A and π * B are the optimal reinforcement learning strategies for controlling the robot arm and wheel leg skills, respectively. a and b represent the ordinal variables of A and B, respectively. t is the current time step of reinforcement learning. At and r Bt are the rewards at time step t in two parallel Markov decision processes, γ t A represents the use of the reward discount factor of strategy A to reduce the impact of future rewards, γ t B represents the use of the reward discount factor of strategy B to reduce the impact of future rewards, and γ t A , γ t B ∈(0,1).

Images