Wheel-legged humanoid robot multi-style parallel intelligent policy learning method

Abstract

The application discloses a multi-style parallel intelligent strategy learning method for a wheel-legged humanoid robot, and robot simulation training is carried out on a computer; independent customized observation values A and B, customized rewards A and B are relied on; parallel strategies A and B are trained; through parallel reinforcement learning processes A and B, optimal parallel strategies A and B are obtained; the optimal parallel strategies A and B are implanted into a robot engineering control computer in the form of a neural network, and forward propagation operation is carried out, so that local skills A and B which cooperate with each other are respectively generated; after conversion through an arm joint PD controller and a wheel-legged joint PD controller, the robot obtains agile auxiliary arm skills A and basic robust wheel-legged skills B. The application provides a dedicated exploration space for the arm, can effectively avoid the defects of arm overfitting, and can avoid consumption and conflicts between different targets.

Description

Technical Field

[0001] The present invention belongs to the technical field of wheel-legged robots, and in particular relates to a multi-style parallel intelligent strategy learning method for wheel-legged humanoid robots. Background Art

[0002] Reinforcement learning theory, pioneered by Google's DeepMind and developed into commercial technology products, aims to enable a controlled object to continuously interact with its environment, gaining beneficial experience and discarding harmful decisions, ultimately achieving continuous self-tuning. The first well-known product of this theory was the AlphaGo Go-playing robot, which defeated a human champion.

[0003] In the context of reinforcement learning, since the arms of wheel-legged humanoid robots do not directly serve as actuators for balancing movements, the (arms') intelligent exploration space is squeezed by the wheel-legged balancing task. The current single-strategy reinforcement learning framework is prone to strategy overfitting, which manifests as the arms converging to strange positions and not participating in the auxiliary center of gravity adjustment task.

[0004] In addition, the wheel-legged humanoid robot is a complex multi-rigid body system. The existing single-strategy reinforcement learning framework is not compatible with reward functions that are too complex, which manifests as inevitable conflicts and consumption between different reward function items. Summary of the Invention

[0005] In view of the shortcomings in the prior art, the present invention provides a multi-style parallel intelligent strategy learning method for a wheel-legged humanoid robot.

[0006] The present invention achieves the above technical objectives through the following technical means.

[0007] Multi-style parallel intelligent strategy learning method for wheel-legged humanoid robots:

[0008] Conduct robot simulation training on a computer, relying on independent customized observations A and B, customized rewards A and B, to train parallel strategies A and B. Through parallel reinforcement learning processes A and B, the optimal parallel strategies A and B are obtained.

[0009] The optimal parallel strategies A and B are implanted into the robot's industrial computer in the form of neural networks, and forward propagation operations are performed to generate local skills A and B that collaborate with each other. After conversion through the arm joint PD controller and the wheel-leg joint PD controller, the robot acquires the agile auxiliary arm skill A and the basic robust wheel-leg skill B.

[0010] Furthermore, the customized reward A is obtained by weighted summation of the contact force distribution reward, the recovery distance reward, and the arm basic reward.

[0011] Furthermore, the contact force distribution reward is:

[0012]

[0013] in, represents the contact force distribution reward, and are the support forces exerted on the left and right wheels of the robot by the ground, K A is the kernel function.

[0014] Furthermore, the recovery distance reward is:

[0015]

[0016] in, Indicates the recovery distance bonus, and are the average speed of the robot's two wheels at time t and the reference average speed respectively.

[0017] Furthermore, the basic rewards for the arm are:

[0018]

[0019] in, Indicates the basic reward of the arm, K A is the kernel function, φ i arm is the actual position of the current arm joint No. i, is the desired position of the arm's joint No. i, U is the robot's rated voltage, τ i arm is the torque of the arm joint No. i, k i is the torque-current proportionality coefficient, P torso is the robot upper body posture vector.

[0020] Furthermore, the customized reward B is obtained by weighted summation of the wheel speed following reward, the wheel-leg posture reward, and the altitude maintenance reward.

[0021] Furthermore, the customized observation value A is the robot's torso posture, arm joint position and speed, and six-dimensional force sensor value.

[0022] Furthermore, the customized observation value B is the position and velocity of the joints of the wheel-legged robot.

[0023] Furthermore, the parallel strategy is specifically a neural network, and its training and working processes are both Markov decision processes.

[0024] Furthermore, the training and operation of parallel strategies A and B are parallel Markov decision processes:

[0025]

[0026] Among them, π * A and π * B are the optimal reinforcement learning strategies for controlling the robot’s arm and wheel-leg skills, respectively. a and b represent the ordinal variables of A and B, respectively. t is the current time step of reinforcement learning, and r At and r Bt are the rewards at time step t in two parallel Markov decision processes, γ t A represents the use of the reward discount factor of strategy A to reduce the impact of future rewards, γ t B represents the use of the reward discount factor of strategy B to reduce the impact of future rewards, and γ t A , γ t B ∈(0,1).

[0027] The beneficial effects of the present invention are:

[0028] (1) The present invention obtains the agile auxiliary arm skill A from the parallel strategy A, provides a dedicated exploration space for the arm, and can effectively avoid the defect of overfitting of the arm; the present invention provides a set of additional, customized rewards for the arm, which can avoid conflicts with the wheel-leg motion reward items, and finally provides a stable, diverse and reasonable parallel motion control strategy for the wheel-leg humanoid robot.

[0029] (2) The parallel strategy of the present invention allows users to customize their movement styles (including observation values ​​and reward symbols) according to their needs. The customized style can give full play to the exploration performance of reinforcement learning while avoiding ablation (consumption) and conflict between different goals, and ultimately achieve collaborative and unified parallel local skills. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a block diagram of the multi-style parallel intelligent strategy learning method for the wheel-legged humanoid robot described in the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0032] The Markov decision process (MDP) is the core model of reinforcement learning tasks. A single reinforcement learning framework only needs to establish an MDP for the overall motion. In this paper, a parallel Markov decision process PX-MDPs is proposed and established in the same multi-body system (i.e., a wheel-legged humanoid robot):

[0033]

[0034] Among them, π * A and π * B are the optimal reinforcement learning strategies (fully connected neural network form) for controlling the robot's arm and wheel-leg skills, respectively; t is the current time step of reinforcement learning; γ is the reward discount factor used to reduce the impact of rewards in the distant future; γ t A represents the use of the reward discount factor of strategy A to reduce the impact of future rewards, γ t B represents the use of the reward discount factor of strategy B to reduce the impact of future rewards, and γ t A , γ t B ∈(0,1); a, b represent the ordinal variables of A and B respectively; r At and r Bt are the rewards at time step t in the two parallel MDPs respectively. The expectation (E) of the cumulative rewards can be used to evaluate the degree of optimization of the strategy and ultimately explore the optimal parallel strategy.

[0035] like Figure 1 As shown, robot simulation training is performed on a high-performance workstation (i.e., a computer). Parallel policies (A and B) are trained based on independent customized observations (A and B) and customized rewards (A and B). Specifically, customized observations A can be the robot's torso posture, arm joint positions and velocities, and six-dimensional force sensor values, while customized observations B can be the positions and velocities of the entire body joints of the wheel-legged robot. The specific definitions of customized rewards (A and B) are explained in formula form below. Under a unified timescale, the optimal parallel policies (A and B) are ultimately obtained through parallel reinforcement learning processes (A and B). The trained parallel policies (A and B) are then embedded in the robot's industrial computer in the form of neural networks. Using the C++ programming language and multi-threaded operation, forward propagation operations are performed on the parallel policies (in the form of fully connected neural networks), generating collaborative local skills (A and B). After conversion between the arm joint PD controller and the wheel-leg joint PD controller, the robot acquires the agile auxiliary arm skill A and the basic robust wheel-leg skill B, respectively.

[0036] Specifically, the significance of customized observation values ​​lies in that the parallel strategies (A and B) can observe the robot states that the user prefers according to their needs, avoid redundant observations of the whole body state, and optimize observation efficiency and rationality; in addition, customized rewards (A and B) can set independent and mutually balanced reward styles to avoid conflicts during the parallel strategy training process and achieve more robust whole-body collaborative characteristics. In the specific implementation process, the parallel strategy proposed in the present invention provides more state space and style space for reward design; more importantly, through the above customized observation values ​​(A and B) and customized rewards (A and B), the parallel strategies (A and B) can be trained through the reinforcement learning process (A and B) and obtain different local skills of the wheel-legged humanoid robot, and the independence and collaboration of these local skills can be adjusted through customization of different styles. The customization of different styles is reflected in the customization of observation values ​​and rewards. The customization of observation values ​​is selected according to user needs, and the customization of rewards is the adjustment of weights and constants in each reward item.

[0037] As for the specific implementation process, the present invention uses the PhysX physics engine optimized for wheel-legged humanoid robots to realize robot dynamics simulation (NVIDIA open source engine, the present invention optimizes and models wheel-legged humanoid robots) in the simulation training stage (high-performance workstation simulation training), and adopts Python and C++ language programming to integrate the physical dynamics simulation data and the reinforcement learning training framework to achieve collaborative work, and finally make the parallel strategies (A and B) in the form of neural networks converge separately and achieve collaborative whole-body control.

[0038] The specific modeling process is to build the same digital twin model in a high-performance workstation simulation environment based on the limb size, mass, inertia, joint friction and degree of freedom configuration of the real robot. High-precision simulation modeling helps to smoothly implement the reinforcement learning process; for the "wheel-ground" contact characteristics of the wheel-legged robot, the mesh contact characteristics of PhysX are optimized and the polygonal characteristics of the wheels are removed, making the simulation closer to the contact characteristics of the actual wheels.

[0039] In the implementation process, customized rewards are a very critical link. The following is the customized arm reward proposed by this invention, whose preferred styles are: agility, collaboration and low power consumption. Specifically:

[0040] First, the present invention uses a kernel function to make the reward term bounded:

[0041]

[0042] Among them, e x is the natural exponential function, and the most common exp notation is used below.

[0043] Then there is the support distribution reward, which aims to help the lower limb end support force achieve dynamic balance through the autonomous adjustment of the arm. and The support forces of the left and right wheels are respectively:

[0044]

[0045] The next item is the recovey distance reward, which aims to restore the robot to a controllable equilibrium state as quickly as possible by adjusting the center of gravity of the arm when the robot is subjected to external collision or disturbance. and are the average speed of the two wheels at time t and the reference average speed, in rad / s. It should be noted that in order to skip the acceleration process of the robot caused by the disturbance, The sampling time is t = 0.2s when the disturbance occurs.

[0046]

[0047] The last item is the arm base reward, which aims to guide the arm to autonomously learn reasonable auxiliary movements when encountering disturbances, while consuming as little energy as possible and taking full account of the overall balance; is the desired position of the arm's joint No. i, set by the user; φ i arm is the actual position of the current arm joint i; k i is the torque-current proportional coefficient, U is the rated voltage of the robot, τ i arm is the torque of the arm's joint No. i; P torso is the robot upper body posture vector.

[0048]

[0049] The contact force distribution reward, recovery distance reward, and arm basic reward are weighted summed to obtain the customized arm reward:

[0050]

[0051] Among them, w A1 、w A2 、w A3 are the weights of the corresponding reward items.

[0052] As for customized wheel-leg rewards, the focus is on overall balance, and the goal is very clear.

[0053] First, we use a kernel function similar to the one used in the custom arm reward to perform bounded processing:

[0054]

[0055] Then, the wheel speed following reward is introduced to calculate the user command v * To achieve the desired result, the actual wheel speed is as close to the user's command as possible:

[0056]

[0057] Where v represents the linear velocity vector of the robot's center of mass at the current moment;

[0058] In addition, in order to maintain the balance of the robot's torso posture, the three-dimensional values ​​of roll, pitch, and yaw of the posture sensor are used to construct the wheel and leg posture rewards:

[0059]

[0060] Finally, in order to make the robot maintain a reasonable torso height, a height maintenance reward is introduced, where h * torso is the desired torso height, h torso is the actual current torso height of the robot:

[0061]

[0062] The weighted sum of the wheel speed following reward, wheel leg posture reward, and altitude holding reward gives the customized wheel leg reward:

[0063]

[0064] Among them, w B1 、w B2 、w B3 are the weights of the corresponding reward items.

[0065] The neural network training process uses reinforcement learning methods, specifically the PPO algorithm (Proximal Policy Optimization). Through the actor-critic training method and policy gradient optimization method, the parallel strategy (i.e., fully connected neural network) is continuously trained until the reward convergence requirement is met. The parallel strategy is considered to have learned the specified skill.

[0066] The input of the neural network is customized observation values ​​(A and B), and the output is the target value for the arm joint PD controller and the wheel-leg joint PD controller. After the above two types of PD controllers, it is finally converted into joint torque. After continuous execution over time, local skills are obtained. Because of the strong intelligence and exploratory nature of the reinforcement learning method, local skills are naturally collaborative. At the same time, the design of parallel strategies (A and B) also makes local skills independent.

[0067] According to the parallel strategy and customized style proposed in the present invention, the wheel-legged humanoid robot can acquire excellent local skills of the upper and lower limbs, and achieve stable balance movements and agile arm-assisted movements through collaborative work.

[0068] The embodiments described are preferred implementations of the present invention, but the present invention is not limited to the above implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essence of the present invention are within the scope of protection of the present invention.

Claims

1. A multi-style parallel intelligent strategy learning method for a wheel-legged humanoid robot, characterized by: Conduct robot simulation training on a computer, relying on independent customized observations A and B, customized rewards A and B, to train parallel strategies A and B. Through parallel reinforcement learning processes A and B, the optimal parallel strategies A and B are obtained. The optimal parallel strategies A and B are embedded in the robot's industrial computer in the form of neural networks, and forward propagation operations are performed to generate collaborative local skills A and B, respectively. After conversion through the arm joint PD controller and the wheel-leg joint PD controller, the robot acquires the agile auxiliary arm skill A and the basic robust wheel-leg skill B. The customized reward A is obtained by weighted summation of the contact force distribution reward, the recovery distance reward, and the arm basic reward; The contact force distribution reward is: in, represents the contact force distribution reward, and are the support forces exerted on the left and right wheels of the robot by the ground, K A It is the kernel function. Among them, e x is the natural exponential function; The customized reward B is obtained by weighted summation of the wheel speed following reward, the wheel leg posture reward, and the altitude maintenance reward; The customized observation value A is the robot's torso posture, arm joint position and speed, and six-dimensional force sensor value; The customized observation value B is the position and speed of the joints of the wheel-legged robot.

2. The multi-style parallel intelligent strategy learning method for wheel-legged humanoid robots according to claim 1 is characterized in that: The recovery distance reward is: in, Indicates the recovery distance bonus, and are the average speed of the robot's two wheels at time t and the reference average speed respectively.

3. The multi-style parallel intelligent strategy learning method for wheel-legged humanoid robots according to claim 1 is characterized in that: The basic rewards for the arm are: in, Indicates the basic reward of the arm, K A is the kernel function, is the actual position of the current arm joint No. i, is the desired position of the arm's joint No. i, U is the robot's rated voltage, τ i arm is the moment of the arm joint No. i, k i is the torque-current proportionality coefficient, P torso is the robot upper body posture vector.

4. The multi-style parallel intelligent strategy learning method for wheel-legged humanoid robots according to claim 1 is characterized in that: The parallel strategy is specifically a neural network, and its training and working processes are both Markov decision processes.

5. The multi-style parallel intelligent strategy learning method for a wheel-legged humanoid robot according to claim 4 is characterized in that: The training and operation process of parallel strategies A and B is a parallel Markov decision process: Among them, π * A and π * B are the optimal reinforcement learning strategies for controlling the robot’s arm and wheel-leg skills, respectively. a and b represent the ordinal variables of A and B, respectively. t is the current time step of reinforcement learning, and r At and r Bt are the rewards at time step t in two parallel Markov decision processes, Indicates using the reward discount factor of strategy A to reduce the impact of future rewards, represents the use of the reward discount factor of strategy B to reduce the impact of future rewards, and

Images