Biped robot reinforcement learning control method

Abstract

The invention discloses a biped robot reinforcement learning control method which comprises the following steps: after a model prediction controller receives a walking instruction, outputting expected angle data of each joint of a robot to a reinforcement learning neural network, and meanwhile, returning joint angle data of an actual strategy of the reinforcement learning neural network to the neural network by a sensing system at a bottom layer; a joint angle taking time as a sequence and planned by model prediction is compared with a joint angle actually generated by a neural network, and model prediction control is fused into a reinforcement learning training process by setting a reward function for punishment, so that the reinforcement learning training efficiency is improved, and a stable gait is more quickly achieved. Excellent gaits planned by the MPC are transplanted into reinforcement learning control, and control robustness can be improved under the condition that the excellent gaits of the MPC are reserved; and meanwhile, the joint angle data which is obtained by taking time as a sequence and is obtained by taking MPC as a planner can accelerate the reinforcement learning training process, so that the training speed and the control effect of reinforcement learning are greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a biped robot reinforcement learning control method. Background Art

[0002] With the development of bipedal robot technology, its humanoid appearance has given it unique advantages in assisting humans in completing tasks. This biomimetic design not only enhances the intimacy of human-robot interaction, making it more easily accepted and trusted by humans, but also allows it to better adapt to environments and facilities designed for humans. However, this humanoid structure also brings significant control challenges, primarily due to the multi-degree-of-freedom and strongly coupled nature of bipedal robots. From a kinematic perspective, a bipedal robot's legs typically have multiple joints, and the movement of each joint affects the other joints as well as the overall posture and position. The complexity of this high-dimensional state space makes motion control of bipedal robots particularly difficult.

[0003] Among existing control methods, model predictive control (MPC) is widely used for high-precision motion control. Through rolling-horizon optimization and feedback correction mechanisms, MPC predicts the output for a period of time in the future based on the current system state within each control cycle, and solves the optimal control input sequence through an optimization algorithm. Ultimately, only the first control input in the sequence is executed. This method can adapt to the dynamic changes of the system in real time, and performs particularly well in scenarios with high stability requirements such as precise trajectory tracking and anti-disturbance control. However, the performance of MPC is highly dependent on accurate dynamic modeling. The dynamic system of a bipedal robot is complex and is affected by factors such as mass distribution, joint friction, and external disturbances. Any model error will be amplified through the prediction link, resulting in a significant decrease in control performance. For example, ignoring the influence of joint friction will lead to prediction deviation. As the prediction time increases, the deviation will accumulate, and ultimately the robot will be unable to accurately track the target trajectory.

[0004] Furthermore, reinforcement learning (RL) offers a solution that eliminates the need for precise dynamics models. By autonomously learning control strategies, it can adapt to complex environments. However, RL training relies on a large amount of interaction data, resulting in long training times and high costs. Furthermore, the design of a reasonable reward function is crucial; improper reward settings can cause the learning strategy to crash or produce unexpected behavior. Furthermore, RL strategies suffer from poor interpretability, posing challenges for safety verification.

[0005] To address the above issues, this paper proposes a control method that combines MPC with RL. MPC, leveraging its advantage in known models, provides RL with initial joint angle planning, serving as the starting point for RL learning. This reduces the blindness of RL exploration and reduces the interaction data and time required for training. At the same time, RL can leverage its ability to adapt to complex non-system environments. Summary of the Invention

[0006] The purpose of the present invention is to provide a biped robot reinforcement learning control method to address the deficiencies of the existing technology.

[0007] The object of the present invention is achieved through the following technical solution: A bipedal robot reinforcement learning control method comprising the following steps:

[0008] (1) During the training process, receiving walking instructions from the user or the upper controller; the walking instructions include the desired average yaw value, foot end position value, and target speed of the bipedal robot;

[0009] (2) Based on the current state information of the bipedal robot and the received walking instructions, the MPC controller predicts the robot's behavior, generates a reference joint angle trajectory, and transmits it to the reinforcement learning neural network;

[0010] (3) At the same time, the joint velocity and position sensors collect the actual joint angle data of the biped robot in real time and transmit it to the reinforcement learning neural network. The reference joint angle trajectory is compared with the actual joint angle data to construct a reward function.

[0011] (4) Optimizing the reinforcement learning neural network by maximizing the reward function to obtain an optimized reinforcement learning neural network;

[0012] (5) The control action output by the reinforcement learning neural network is converted into actual joint control signals through the proportional-differential controller; and the action signal is executed on each joint of the robot through the proportional-differential controller;

[0013] (6) Joint velocity and position sensors collect the robot’s motion state data in real time, including the angle and velocity information of each joint; this data is fed back to the system to update the robot’s current state information; the updated state vector provides new input for the reinforcement learning network to make decisions in the next control cycle;

[0014] (7) During the training process, the entire control system continuously trains and updates the policy network through continuous state feedback and reward feedback; each iteration optimizes the reinforcement learning policy network based on the new state data and reward feedback to gradually improve the control accuracy and ultimately achieve the optimal control strategy.

[0015] Furthermore, the step (2) specifically includes the following sub-steps:

[0016] (2.1) Construct a simplified single rigid body dynamic model of the biped robot. The biped robot is considered as a rigid body with concentrated mass. The external torque generated by the hip and ankle joints is considered to form the biped robot body acceleration. Rate of change of angular momentum about the center of mass And the control input vector u=[u1,u2] T =[F1,F2,M1,M2] T The linear relationship between them is as follows:

[0017]

[0018] Where g represents the gravitational acceleration vector; u i =[F i ,M i ], F i =[F i,x ,F i,y ,F i,z ] T , M i =[M i,x ,M i,y ] T , i = 1, 2, F1 represents the three-dimensional force on the left leg, F2 represents the three-dimensional force on the right leg, M1 represents the two-dimensional moment on the left leg, and M2 represents the two-dimensional moment on the right leg; m represents the mass of the robot, p1-p c Represents the distance vector between the robot's center of mass in the world coordinate system and the left foot position in the world coordinate system, p2-p c Represents the distance vector between the robot's center of mass position in the world coordinate system and the right foot position in the world coordinate system; (p1-p c )×and(p2-p c )×represents a skew-symmetric matrix, which is used to calculate (p i -p c )×F i The cross product of I G represents the moment of inertia of the robot's center of mass in the world coordinate system, Represents the angular acceleration of the robot in the world coordinate system; I 3×3 represents the 3×3 identity matrix, 0 3×2 represents a 3×2 zero matrix;

[0019] The robot posture is represented by the rotation matrix R, which can be converted into Euler angles Θ = [φ,θ,ψ] T , where φ is the roll angle, θ is the pitch angle, and ψ is the yaw angle;

[0020] Assuming the roll and pitch angles are zero, the rate of change of the Euler angles The relationship with the angular velocity ω (x, y, z, robot as a whole) can be approximated as:

[0021]

[0022] in, Indicates the rate of change of the roll angle; Indicates the rate of change of pitch angle; Indicates the rate of change of yaw angle;

[0023] Taking gravity as an additional state variable, we get the state vector Among them, p c represents the robot's center of mass position vector, represents the velocity vector of the robot's center of mass, and g represents the acceleration due to gravity; so that the dynamics formula can be written as a continuous time matrix and The linear state space form of :

[0024]

[0025] in,

[0026] I b Represents the moment of inertia of the robot body in the body coordinate system;

[0027] (2.2) The MPC controller predicts the future state in the form of discrete time steps in each control cycle, and expresses the linear dynamic equation in the form of discrete time step j as follows:

[0028]

[0029] Among them, the discrete time matrix is a constant matrix, from It is calculated as the average yaw value during the entire reference trajectory; In addition to the matrix It is based on the current state of the robot, and the other moment matrices are based on Calculate,using the expected average yaw and foot-end position values;

[0030] Get the state vector X[j] and control input vector u[j] of N intervals, and construct the objective function J as follows:

[0031]

[0032] Where X[j] represents the state vector of the jth interval; u[j] represents the control input vector of the jth interval; X[j]-X[j] ref Indicates the difference between the actual state X[j] and the reference state X[j] in the jth interval ref The deviation betweenj The weight vector representing the center of mass state error; R j The weight vector representing the ground contact force and contact torque input to the system;

[0033] The objective function J must satisfy the following dynamic constraints and inequality constraints:

[0034] -μF iz ≤F ix ≤μF iz ;

[0035] -μF iz ≤F iy ≤μF iz ;

[0036] 0<F min ≤F iz ≤F max ;

[0037] |τ i |≤τ max ;

[0038] Where μ represents the friction coefficient; F min Indicates the minimum contact force between the robot foot and the ground; F max represents the maximum contact force between the robot foot and the ground; τ max Indicates the maximum value of the joint torque;

[0039] (2.3) The solution of the MPC optimal control solution problem is finally transformed into a QP problem:

[0040]

[0041] stCU≤d;

[0042] Among them, U contains the control input vector of N interval numbers; h represents a symmetric positive definite matrix; f represents a vector of linear coefficients of the control input in an objective function; C represents the constraint matrix; d represents the constraint vector;

[0043] The optimal control input vectors for multiple future control intervals are quickly calculated using the QP solver and mapped to the joint torques of each leg. The controller inputs for each leg are mapped to their joint torques in the following way:

[0044]

[0045] Among them, J i represents the Jacobian matrix of the robot's leg i, J v and J ω They are the Jacobian matrix J iThe linear velocity component and angular velocity component of

[0046] (3.4) The force of the bipedal robot's swinging legs is calculated by considering the foot as connected to a virtual spring-damper system. According to the PD control law, the foot force can be expressed as:

[0047]

[0048] Among them, K P and K D is the PD control gain coefficient, is the desired foot position in world coordinates, is the actual foot position in the world coordinate system; represents the force applied to the foot when the robot's leg swings;

[0049] Mujoco software was then used for simulation to obtain the joint angle data set in time sequence when the biped robot moved at the target speed, and the data was merged and saved as an NPY format file as the reference joint angle trajectory.

[0050] Furthermore, the step (3) specifically includes the following sub-steps:

[0051] (3.1) Joint velocity and position sensors simultaneously collect the actual joint angle data of the bipedal robot in real time and transmit it to the reinforcement learning neural network. The reference joint angle trajectory is compared with the actual joint angle data to construct a reward function.

[0052] The reward function is r t =r cmd +r diff +r tq +r track +r T , where r cmd To control the target reward, r diff is the action change reward, r tq is the torque penalty, r track is the trajectory tracking reward, r T To stop the punishment;

[0053] The control target reward r cmd for in, represents the target control speed of the robot in the x and y directions at time t, represents the actual speed of the robot in the x and y directions at time t; represents the target yaw angle of the robot at time t, represents the actual yaw angle of the robot at time t; k xy 、k wis the scaling factor, σ xy ∈(0,1],σ w ∈(0,1];

[0054] The action change reward r diff For r diff =k diff ||A t -A t-1 ||, where A t is the current action vector, A t-1 is the action vector at the previous moment, k diff is the scaling factor;

[0055] The moment penalty r tq For r tq =k tq ||τ t ||, where τ t is the sum of the joint motor torques at time t, k tq is the scaling factor;

[0056] The trajectory tracking reward r track For r track =exp(k track ·||p act -p track ||), where p act Indicates the actual joint angle data, p track represents the reference joint angle trajectory, k track represents the scaling factor;

[0057] The suspension penalty r T for Among them, h com Represents the height of the center of mass of the biped robot.

[0058] Furthermore, the reinforcement learning neural network includes a strategy network and a value network;

[0059] The strategy network adopts a multi-layer fully connected neural network architecture, specifically including a strategy network input layer, a strategy network first hidden layer, a strategy network second hidden layer and a strategy network output layer;

[0060] The dimension of the policy network input layer is consistent with the dimension of the state vector, and is used to receive the state vector of the biped robot, wherein the state information includes the trunk posture, linear velocity, angular velocity, angles and angular velocities of each joint, foot contact state and environmental information;

[0061] The first hidden layer of the strategy network is a fully connected layer with 256 nodes and uses the ReLU activation function;

[0062] The second hidden layer of the strategy network is a fully connected layer with 128 nodes and uses the ReLU activation function;

[0063] The dimension of the output layer of the strategy network is consistent with the dimension of the action vector, and the action vector includes the target angle value of each joint motor of the biped robot;

[0064] The value network adopts a multi-layer fully connected neural network architecture, specifically including a value network input layer, a value network first hidden layer, a value network first hidden layer and a value network output layer;

[0065] The dimension of the value network input layer is consistent with the dimension of the state vector, and is used to receive the state vector of the robot;

[0066] The first hidden layer of the value network is a fully connected layer with 256 nodes and uses the ReLU activation function;

[0067] The second hidden layer of the value network is a fully connected layer with 128 nodes and uses the ReLU activation function;

[0068] The dimension of the output layer of the value network is 1, which is used to output the value estimate of the current state vector.

[0069] The beneficial effects of the present invention are: transplanting the excellent gait planned by MPC into reinforcement learning control, while retaining the excellent gait of MPC and improving the robustness of control; at the same time, the joint angle data in time sequence obtained by MPC as a planner can accelerate the reinforcement learning training process, greatly improving the training speed and control effect of reinforcement learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 The actual hybrid biped robot designed for the present invention and its equivalent series model diagram;

[0071] Figure 2 A flow chart for implementing the model predictive control of the present invention;

[0072] Figure 3 This is a diagram of the reinforcement learning control framework with model predictive control added to the present invention;

[0073] Figure 4 This is a flowchart for implementing reinforcement learning training of the present invention. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to illustrate the present invention, rather than to represent all embodiments. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0075] See also Figure 1 The object of implementation of the present invention is a hybrid biped robot independently developed. In order to facilitate subsequent control research, the present invention converts the parallel joints into a more feasible equivalent series joint experiment while keeping the number and properties of the degree of freedom of the mechanism unchanged. That is, the hip and ankle parallel joints of the two legs are replaced with vertically intersecting revolute pairs to obtain a full-body series equivalent model of the robot.

[0076] The present invention provides a biped robot reinforcement learning control method, comprising the following steps:

[0077] First, after receiving the walking command, the model predictive controller outputs the expected angle data of each joint of the robot to the reinforcement learning neural network. At the same time, the underlying sensor system returns the joint angle data of the actual strategy of the reinforcement learning neural network to the neural network. The joint angles planned in time series by the model prediction are compared with the joint angles actually generated by the neural network. By setting a reward function as a penalty, the model predictive control is integrated into the reinforcement learning training process, thereby accelerating the efficiency of reinforcement learning training and enabling it to achieve a stable gait more quickly.

[0078] like Figure 2 As shown in the figure, model predictive control planning is performed on the biped robot, which involves the derivation of a single rigid body dynamics model, MPC controller design, QP planning, and swing leg control. Finally, joint angle data in a time series is obtained through simulation.

[0079] The implementation process is as follows:

[0080] 1. Derivation of the robot's single rigid body dynamics model

[0081] A simplified single-rigid-body dynamic model of a bipedal robot is constructed. This model treats the robot as a concentrated rigid body, considers the external torques generated by the hip and ankle joints, and describes the linear relationship between the robot's center of mass acceleration, the rate of change of angular momentum, and the contact forces and torques.

[0082] The external torque generated by the hip and ankle joints is also included in the robot dynamics to form the bipedal robot body acceleration Rate of change of angular momentum about the center of mass And the control input vector u=[F1,F2,M1,M2]T The linear relationship between them is as follows:

[0083]

[0084] Where g represents the gravitational acceleration vector; F i =[F i,x ,F i,y ,F i,z ] T , M i =[M i,x ,M i,y ] T , i = 1, 2, F1 represents the three-dimensional force on the left leg (sole), F2 represents the three-dimensional force on the right leg (sole), M1 represents the two-dimensional moment on the left leg, and M2 represents the two-dimensional moment on the right leg; m represents the mass of the robot, p1-p c Represents the distance vector between the robot's center of mass in the world coordinate system and the left foot position in the world coordinate system, p2-p c Represents the distance vector between the robot's center of mass position in the world coordinate system and the right foot position in the world coordinate system; (p1-p c )×and(p2-p c )×represents a skew-symmetric matrix, which is used to calculate (p i -p c )×F i The cross product of I G represents the moment of inertia of the robot's center of mass in the world coordinate system, Represents the angular acceleration of the robot in the world coordinate system; I 3×3 represents the 3×3 identity matrix, 0 3×2 Represents a 3×2 zero matrix.

[0085] The present invention uses the rotation matrix R to represent the robot posture, which can be converted into the Euler angle Θ = [φ,θ,ψ] T , where φ is the roll angle, θ is the pitch angle, and ψ is the yaw angle (of the robot as a whole).

[0086] To simplify the calculation, assuming that the roll and pitch angles are zero, the Euler angle change rate is The relationship with the angular velocity ω (x, y, z, robot as a whole) is approximately:

[0087]

[0088] in, Indicates the rate of change of the roll angle; Indicates the rate of change of pitch angle; Indicates the rate of change of yaw angle.

[0089] Taking gravity as an additional state variable, we get the state vector (13 dimensions), where p c represents the robot's center of mass position vector, represents the velocity vector of the robot's center of mass. This allows the dynamics formula to be written as a continuous time matrix and The linear state space form of :

[0090]

[0091] in,

[0092] I b Represents the moment of inertia of the robot body in the body coordinate system.

[0093] 2. MPC Problem Formulation

[0094] Based on a simplified dynamic model, the present invention formulates the MPC problem as a constrained optimization problem. The objective function consists of minimizing the center-of-mass state error and the magnitude of the ground contact force / torque, while satisfying system dynamic constraints and inequality constraints. The MPC controller predicts future states in discrete time steps within each control cycle and solves for the optimal control input sequence.

[0095] The present invention expresses the linear dynamics equation in the form of discrete time steps j as follows:

[0096]

[0097] Among them, the discrete time matrix is a constant matrix, from It is calculated as the average yaw value during the entire reference trajectory; In addition to the matrix It is based on the current state of the robot, and the other moment matrices are based on Calculate, using the desired average yaw and foot position values. This is also the dynamic constraint of the MPC problem.

[0098] Based on the simplified dynamic model, N represents the number of control intervals and prediction intervals, and the MPC problem is formulated as a constrained optimization problem. The objective function is:

[0099]

[0100] Where X[j] represents the state vector of the jth interval; u[j] represents the control input vector of the jth interval; X[j]-X[j] ref Indicates the difference between the actual state X[j] and the reference state X[j] in the jth interval ref The deviation between j The weight vector representing the center of mass state error; R j A weight vector representing the ground contact forces and contact moments input to the system.

[0101] The MPC controller solves for the optimal ground contact force and torque based on the dynamic constraints and the following inequality constraints:

[0102] -μF iz ≤F ix ≤μF iz ;

[0103] -μF iz ≤F iy ≤μF iz ;

[0104] 0<F min ≤F iz ≤F max ;

[0105] |τ i |≤τ max ;

[0106] Where μ represents the friction coefficient; F min Indicates the minimum contact force between the robot foot and the ground; F max represents the maximum contact force between the robot foot and the ground; τ1 represents the joint torque of the left leg, τ2 represents the joint torque of the right leg, and τ max Indicates the maximum value of the joint torque.

[0107] Via-μF iz ≤F ix ≤μF iz and -μF iz ≤F iy ≤μF iz Control the contact force in the x-direction and y-direction to be within the friction cone; by 0<F min ≤F iz ≤F max The contact force in the z direction also has its upper and lower limits, where 0 < F min Indicates that the robot is always in contact with the ground; |τ i |≤τ max The joint torques τ (5 joints each for the left and right legs) were constrained to be within the range of the physical motors.

[0108] 3. QP Planning

[0109] The MPC problem is converted into a quadratic programming (QP) form, and the optimal control input is obtained by solving the QP problem in real time. The present invention writes the dynamic constraints of the entire prediction range into a matrix form and constructs the objective function and the constraint matrix.

[0110] The MPC controller can be written in the following quadratic programming (QP) form:

[0111]

[0112] stCU≤d;

[0113] Here, stCU≤d means that CU≤d is satisfied; U contains the control input vectors at all time steps within the prediction horizon and is the decision variable in the optimization problem. h represents a symmetric positive definite matrix, f represents the vector of linear coefficients of the control input in an objective function, and C represents the constraint matrix. It defines the linear inequality constraints that the control input must satisfy. d represents the constraint vector. Together with the constraint matrix C, it defines the upper limit of each constraint. Specifically, for each constraint, CU≤d means that the result of the control input U transformed by the matrix C must be less than or equal to the corresponding value.

[0114] The above quadratic programming problem can be solved by the QP solver to quickly calculate the optimal control input vector for multiple future control intervals and map it to the joint torque of each leg.

[0115] The controller input for each leg is mapped to its joint torques in the following way:

[0116]

[0117] Among them, J i represents the Jacobian matrix of the robot's leg i, J v and J ω They are the Jacobian matrix J i The linear and angular velocity components.

[0118] 4. Swinging leg control

[0119] The force of the robot's swinging legs is calculated by treating the feet as connected to a virtual spring-damper system. Since the weight of the feet is very small compared to the robot body, they can be reasonably ignored. According to the PD control law, the foot force can be expressed as:

[0120]

[0121] Among them, K P and K D is the PD control gain coefficient, is the desired foot position in world coordinates, is the actual foot position in the world coordinate system; Represents the force applied to the foot when the robot's leg swings.

[0122] After completing the above steps, an MPC simulation is performed. The gait generator determines the stance or swing phase for each leg within a fixed gait cycle and assigns an appropriate controller to the corresponding leg. The stance leg uses the MPC controller to achieve accurate center of mass trajectory tracking and dynamic balance, while the swing leg uses the PD controller to complete foot trajectory tracking.

[0123] The simulation platform of the embodiment of the present invention adopts the Ubuntu20.04 system, and uses Mujoco software to perform simulation according to the received walking instructions. The maximum forward movement speed of the bipedal robot is set to 0.5m / s and the center of mass height is 1.03m. In the Mujoco simulation environment, a set of joint angle data in a time sequence is obtained when the robot moves at the target speed.

[0124] In this embodiment, each joint angle data set includes 10 joint angle data.

[0125] After acquiring the robot's joint angle data, we saved it as an NPY file for subsequent processing in the Python code framework. The time-series joint angles generated by the model predictive control planning in the simulation environment were incorporated into the reinforcement learning framework by designing a reward function.

[0126] In the Python code framework, use the Numpy library to import the saved NPY format file into the reinforcement learning code framework.

[0127] In an embodiment of the present invention, a reinforcement learning method based on the PPO algorithm is used to train the control strategy of the biped robot, and the simulation platform adopts the Isaacgym platform.

[0128] The reinforcement learning control framework built in this paper is as follows Figure 3 As shown in the figure, after the model predictive control planner outputs joint angle data in a time series, it is compared with the robot's actual joint angle data in the reinforcement learning simulation environment. This data is then incorporated into the training framework by designing a reward function. During training, the reinforcement learning neural network inputs the robot's state in the environment, and outputs the robot's actions, namely the angles of the motors in each joint.

[0129] The reinforcement learning control implementation process of the present invention can be found in Figure 4The reinforcement learning framework designed in this invention mainly includes state space and action space design, reward function design, neural network construction and training. During the training phase, the Actor and Critic networks work simultaneously, and reward values ​​are obtained by continuously interacting with the environment.

[0130] The specific steps are as follows:

[0131] 1) Design state vector and action vector

[0132] State vector S t Including but not limited to: robot torso posture, linear velocity, angular velocity, angles and angular velocities of each joint, contact status and relative position of the foot end.

[0133] Motion vector A t is the target angle value of each joint motor. The joint motors of the robot in the present invention adopt PD control, and the PD parameters here are fixed values.

[0134] 2) Design reward function:

[0135] The core innovation of this invention is to integrate the time-series joint angle data generated by MPC into the reinforcement learning training framework to guide the bipedal robot to form a regular and stable walking gait.

[0136] The combination of MPC reference trajectories and reinforcement learning brings high precision and adaptability to the control of bipedal robots. The planned MPC reference trajectory provides the target joint angles for robot motion, ensuring high precision and repeatability. Reinforcement learning, by learning the optimal strategy for the environment and task, enables the robot to cope with dynamic changes and complex tasks.

[0137] This paper aims at the walking problem of a biped robot and sets the reward function as follows:

[0138] r t =r cmd +r diff +r tq +r track +r T ;

[0139] Among them, r cmd To control the target reward, r diff is the action change reward, r tq is the torque penalty, r track is the trajectory tracking reward, r T To stop the punishment.

[0140] Control target reward r cmd In order to make the robot walk at the desired speed and direction, the specific function is as follows:

[0141]

[0142] in, represents the target control speed of the robot in the x and y directions at time t, represents the actual speed of the robot in the x and y directions at time t; represents the target yaw angle of the robot at time t, represents the actual yaw angle of the robot at time t; k xy 、k w is the scaling factor, σ xy ∈(0,1],σ w ∈(0,1].

[0143] Action change reward r diff This is to improve the robot's jitter and limit the robot's ability to generate actions with excessive variations. The specific definition is as follows:

[0144] r diff =k diff ||A t -A t-1 ||;

[0145] Among them, A t is the current action vector, A t-1 is the action vector at the previous moment, k diff is the scaling factor.

[0146] The present invention adds a torque penalty r tq To improve energy efficiency, the torque is the sum of the torques of the active drives, which is defined as follows:

[0147] r tq =k tq ||τ t ||;

[0148] Among them, τ t is the sum of the joint motor torques at time t, k tq is the scaling factor.

[0149] Trajectory tracking reward r track It rewards and punishes the swinging legs and supporting legs of the bipedal robot, mainly to encourage the robot's movements to fit the MPC planning gait, thereby improving the training speed of reinforcement learning.

[0150] This is achieved by first loading the reference trajectory data from the NPY file and extracting the information of specific joints, including but not limited to the hip pitch, knee pitch, and ankle pitch angles of each leg.

[0151] The training framework calculates the index based on the current time step to obtain the target joint angle in real time and use it as a reference input. The relevant reward function is designed and defined as follows:

[0152] r track =exp(k track ·||p act -p track ||);

[0153] Among them, p act 、p track They represent the actual robot motion trajectory joint angles and the MPC reference motion trajectory angles, respectively, track Indicates the scaling factor.

[0154] When the robot falls, it is given a suspension penalty of -1, and 0 at other times. The specific definition is as follows:

[0155]

[0156] Among them, h com Indicates the height of the robot's center of mass.

[0157] 3) Designing the Strategy Network (Actor) and the Value Network (Critic)

[0158] Policy network (Actor): The policy network is used to generate joint control instructions based on the current state of the robot. Its input is the state vector and its output is the action vector.

[0159] The policy network adopts a multi-layer fully connected neural network (MLP) architecture, specifically including a policy network input layer, a policy network first hidden layer, a policy network second hidden layer and a policy network output layer.

[0160] The policy network input layer has the same dimensions as the state vector and is used to receive the bipedal robot's state vector. This state information includes trunk posture, linear velocity, angular velocity, joint angles and angular velocities, foot contact state, and environmental information. The policy network input layer serves as a data entry point and does not include an activation function. This design ensures normalization of the state vector, improving network training stability and convergence efficiency.

[0161] The first hidden layer of the policy network is a fully connected layer with 256 nodes and a ReLU activation function. This layer is responsible for extracting high-level features from the input state and enhancing the network's expressive power through nonlinear transformations. It also effectively alleviates the vanishing gradient problem and accelerates the training process.

[0162] The second hidden layer of the policy network is a fully connected layer with 128 nodes and the same ReLU activation function. This layer further extracts and combines features to provide a more refined feature representation for the output layer. The number of nodes is designed to gradually compress information, reducing computational complexity and improving the model's generalization ability.

[0163] The output layer of the policy network, whose dimensions match those of the action vectors, generates standardized control actions, representing the target angles for each joint. The activation function uses the Tanh (hyperbolic tangent) function, limiting the output to the range [-1, 1] to facilitate subsequent mapping to the actual action space and ensure the rationality and stability of the control actions.

[0164] The present invention utilizes the multi-layer, fully connected neural network structure to efficiently extract features from the state space and generate standardized control actions. The strategy outputs are then scaled and mapped to the desired angles for each joint.

[0165] The value network also uses a multi-layer, fully connected neural network architecture, specifically comprising a value network input layer, a first hidden layer, a second hidden layer, and a value network output layer. The value network input layer has the same dimension as the state vector and is used to receive the robot's state vector. The value network output layer has a dimension of 1 and outputs an estimated value for the current state vector, representing the potential cumulative reward the robot will receive in the given state.

[0166] The first hidden layer of the value network and the second hidden layer of the value network use the same settings as the first hidden layer of the policy network and the second hidden layer of the policy network.

[0167] 4) Training Process

[0168] Please combine Figure 3 , the specific training process of the present invention is as follows:

[0169] (1) Walking command input

[0170] During the training process, a walking instruction is received from a user or an upper controller; the walking instruction includes the desired average yaw value, foot end position value and target speed of the bipedal robot.

[0171] (2) MPC model predictive controller generates reference trajectory

[0172] Based on the current state of the bipedal robot and received walking commands, the MPC controller predicts the robot's behavior, generates a reference joint angle trajectory, and transmits it to the reinforcement learning neural network. This trajectory provides high-precision control of the robot through rolling-horizon optimization and feedback mechanisms.

[0173] (3) Reward function design and calculation

[0174] Simultaneously, joint velocity and position sensors collect the bipedal robot's actual joint angle data in real time and transmit it to a reinforcement learning neural network. Reference joint angle trajectories are compared with the actual joint angle data to construct a reward function. This reward function reflects the deviation between the robot's actual behavior and the target prediction and is used to guide the optimization of the reinforcement learning strategy. Behaviors with smaller errors receive higher rewards, prompting the robot to learn more precise control strategies.

[0175] (4) Reinforcement learning neural network strategy training

[0176] The reinforcement learning neural network is optimized by maximizing the reward function, resulting in an optimized reinforcement learning neural network. The reinforcement learning neural network receives state information from the environment and outputs corresponding control actions. Through continuous interaction with the environment, the reinforcement learning network outputs the optimal control decision based on the current state. By maximizing the reward function, the reinforcement learning neural network optimizes its control strategy to achieve long-term stable behavior.

[0177] (5) PD controller performs actions

[0178] The control actions output by the reinforcement learning neural network are further processed by a proportional-derivative (PD) controller and converted into actual joint control signals. The PD controller is responsible for accurately executing the action signals to each joint of the robot, ensuring the stability and responsiveness of the control system.

[0179] (6) Sensor feedback and status update

[0180] Joint velocity and position sensors collect real-time data on the robot's motion state, including the angles and velocities of each joint. This data is fed back to the system to update the robot's current state. The updated state vector provides new input to the reinforcement learning network for decision-making in the next control cycle.

[0181] (7) Loop Iteration Optimization Strategy

[0182] During training, the entire control system continuously trains and updates the policy network through continuous state and reward feedback. Each iteration optimizes the reinforcement learning policy network based on the new state data and reward feedback, gradually improving control accuracy and ultimately achieving the optimal control strategy.

[0183] Experimental verification of the embodiments of the present invention shows that the present invention effectively improves the efficiency of reinforcement learning training after adding model prediction and planning trajectory.

[0184] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A reinforcement learning control method for a bipedal robot, characterized in that: The following steps are involved: (1) During the training process, receiving walking instructions from the user or the upper controller; the walking instructions include the desired average yaw value, foot end position value, and target speed of the bipedal robot; (2) Based on the current state information of the bipedal robot and the received walking instructions, the MPC controller predicts the robot's behavior, generates a reference joint angle trajectory, and transmits it to the reinforcement learning neural network; (3) At the same time, the joint velocity and position sensors collect the actual joint angle data of the biped robot in real time and transmit it to the reinforcement learning neural network. The reference joint angle trajectory is compared with the actual joint angle data to construct a reward function. (4) Optimizing the reinforcement learning neural network by maximizing the reward function to obtain an optimized reinforcement learning neural network; (5) The control action output by the reinforcement learning neural network is converted into actual joint control signals through the proportional-differential controller; and the action signal is executed on each joint of the robot through the proportional-differential controller; (6) Joint velocity and position sensors collect the robot’s motion state data in real time, including the angle and velocity information of each joint; this data is fed back to the system to update the robot’s current state information; the updated state vector provides new input for the reinforcement learning network to make decisions in the next control cycle; (7) During the training process, the entire control system continuously trains and updates the policy network through continuous state feedback and reward feedback; Each iteration optimizes the reinforcement learning policy network based on the new state data and reward feedback to gradually improve the control accuracy and ultimately achieve the optimal control strategy.

2. A bipedal robot reinforcement learning control method according to claim 1, characterized in that: The step (2) specifically includes the following sub-steps: (2.1) Construct a simplified single rigid body dynamic model of the biped robot. The biped robot is considered as a rigid body with concentrated mass. The external torque generated by the hip and ankle joints is considered to form the biped robot body acceleration. Rate of change of angular momentum about the center of mass And the control input vector u=[u1,u2] T =[F1,F2,M1,M2] T The linear relationship between them is as follows: Where g represents the gravitational acceleration vector; u i =[F i ,M i ], F i =[F i,x ,F i,y ,F i,z ] T , M i =[M i,x ,M i,y ] T , i = 1, 2, F1 represents the three-dimensional force on the left leg, F2 represents the three-dimensional force on the right leg, M1 represents the two-dimensional moment on the left leg, and M2 represents the two-dimensional moment on the right leg; m represents the mass of the robot, p1-p c Represents the distance vector between the robot's center of mass in the world coordinate system and the left foot position in the world coordinate system, p2-p c Represents the distance vector between the robot's center of mass position in the world coordinate system and the right foot position in the world coordinate system; (p1-p c )×and(p2-p c )×represents a skew-symmetric matrix, which is used to calculate (p i -p c )×F i The cross product of I G represents the moment of inertia of the robot's center of mass in the world coordinate system, Represents the angular acceleration of the robot in the world coordinate system; I 3×3 represents the 3×3 identity matrix, 0 3×2 represents a 3×2 zero matrix; The robot posture is represented by the rotation matrix R, which can be converted into Euler angles Θ = [φ,θ,ψ] T , where φ is the roll angle, θ is the pitch angle, and ψ is the yaw angle; Assuming the roll and pitch angles are zero, the rate of change of the Euler angles The relationship with the angular velocity ω (x, y, z, robot as a whole) can be approximated as: in, Indicates the rate of change of the roll angle; Indicates the rate of change of pitch angle; Indicates the rate of change of yaw angle; Taking gravity as an additional state variable, we get the state vector Among them, p c represents the robot's center of mass position vector, represents the velocity vector of the robot's center of mass, and g represents the acceleration due to gravity; so that the dynamics formula can be written as a continuous time matrix and The linear state space form of : in, ； I b Represents the moment of inertia of the robot body in the body coordinate system; (2.2) The MPC controller predicts the future state in the form of discrete time steps in each control cycle, and expresses the linear dynamic equation in the form of discrete time step j as follows: Among them, the discrete time matrix is a constant matrix, from It is calculated as the average yaw value during the entire reference trajectory; In addition to the matrix It is based on the current state of the robot, and the other moment matrices are based on Calculate,using the expected average yaw and foot-end position values; Get the state vector X[j] and control input vector u[j] of N intervals, and construct the objective function J as follows: Where X[j] represents the state vector of the jth interval; u[j] represents the control input vector of the jth interval; X[j]-X[j] ref Indicates the difference between the actual state X[j] and the reference state X[j] in the jth interval ref The deviation between j The weight vector representing the center of mass state error; R j The weight vector representing the ground contact force and contact torque input to the system; The objective function J must satisfy the following dynamic constraints and inequality constraints: -μF iz ≤F ix ≤μF iz ; -μF iz ≤F iy ≤μF iz ; 0<F min ≤F iz ≤F max ; |t i |≤τ max ; Where μ represents the friction coefficient; F min Indicates the minimum contact force between the robot foot and the ground; F max represents the maximum contact force between the robot foot and the ground; τ max Indicates the maximum value of the joint torque; (2.3) The solution of the MPC optimal control solution problem is finally transformed into a QP problem: Among them, U contains the control input vector of N interval numbers; h represents a symmetric positive definite matrix; f represents a vector of linear coefficients of the control input in an objective function; C represents the constraint matrix; d represents the constraint vector; The optimal control input vectors for multiple future control intervals are quickly calculated using the QP solver and mapped to the joint torques of each leg. The controller inputs for each leg are mapped to their joint torques in the following way: Among them, J i represents the Jacobian matrix of the robot's leg i, J v and J ω They are the Jacobian matrix J i The linear velocity component and angular velocity component of (3.4) The force of the bipedal robot's swinging legs is calculated by considering the foot as connected to a virtual spring-damper system. According to the PD control law, the foot force can be expressed as: Among them, K P and K D is the PD control gain coefficient, is the desired foot position in world coordinates, is the actual foot position in the world coordinate system; represents the force applied to the foot when the robot's leg swings; Mujoco software was then used for simulation to obtain the joint angle data set in time sequence when the biped robot moved at the target speed, and the data was merged and saved as an NPY format file as the reference joint angle trajectory.

3. The reinforcement learning control method for a bipedal robot according to claim 1, characterized in that: The step (3) specifically includes the following sub-steps: (3.1) Joint velocity and position sensors simultaneously collect the actual joint angle data of the bipedal robot in real time and transmit it to the reinforcement learning neural network. The reference joint angle trajectory is compared with the actual joint angle data to construct a reward function. The reward function is r t =r cmd +r diff +r tq +r track +r T , where r cmd To control the target reward, r diff is the action change reward, r tq is the torque penalty, r track is the trajectory tracking reward, r T To stop the punishment; The control target reward r cmd for in, represents the target control speed of the robot in the x and y directions at time t, represents the actual speed of the robot in the x and y directions at time t; represents the target yaw angle of the robot at time t, represents the actual yaw angle of the robot at time t; k xy 、k w is the scaling factor, σ xy ∈(0,1],σ w ∈(0,1]; The action change reward r diff For r diff =k diff ||A t -A t-1 ||, where A t is the current action vector, A t-1 is the action vector at the previous moment, k diff is the scaling factor; The moment penalty r tq For r tq =k tq ||τ t ||, where τ t is the sum of the joint motor torques at time t, k tq is the scaling factor; The trajectory tracking reward r track For r track =exp(k track ·||p act -p track ||), where p act Indicates the actual joint angle data, p track represents the reference joint angle trajectory, k track represents the scaling factor; The suspension penalty r T for Among them, h com Represents the height of the center of mass of the biped robot.

4. A bipedal robot reinforcement learning control method according to claim 1, characterized in that: The reinforcement learning neural network includes a strategy network and a value network; The strategy network adopts a multi-layer fully connected neural network architecture, specifically including a strategy network input layer, a strategy network first hidden layer, a strategy network second hidden layer and a strategy network output layer; The dimension of the policy network input layer is consistent with the dimension of the state vector, and is used to receive the state vector of the biped robot, wherein the state information includes the trunk posture, linear velocity, angular velocity, angles and angular velocities of each joint, foot contact state and environmental information; The first hidden layer of the strategy network is a fully connected layer with 256 nodes and uses the ReLU activation function; The second hidden layer of the strategy network is a fully connected layer with 128 nodes and uses the ReLU activation function; The dimension of the output layer of the strategy network is consistent with the dimension of the action vector, and the action vector includes the target angle value of each joint motor of the biped robot; The value network adopts a multi-layer fully connected neural network architecture, specifically including a value network input layer, a value network first hidden layer, a value network first hidden layer and a value network output layer; The dimension of the value network input layer is consistent with the dimension of the state vector, and is used to receive the state vector of the robot; The first hidden layer of the value network is a fully connected layer with 256 nodes and uses the ReLU activation function; The second hidden layer of the value network is a fully connected layer with 128 nodes and uses the ReLU activation function; The dimension of the output layer of the value network is 1, which is used to output the value estimate of the current state vector.

Images