Multi-gait biped motion control method and device based on deep reinforcement learning

By using the multi-gait bipedal motion control method with deep reinforcement learning in a simulation environment, the problem of high training costs and susceptibility to external disturbances in a humanoid robot is solved, and the expected gait motion is simulated on the humanoid robot, reducing hardware damage Risk and improve control accuracy.

CN118938645APending Publication Date: 2024-11-12INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202410855321.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-28
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the prior art, humanoid robots are highly costly in training control and are susceptible to external disturbances, resulting in losing balance and dumping during walking simulation in actual environments, unable to collect effective gait data, and may cause damage to the hardware system.

Method used

Using a multi-gait bipedal motion control method based on deep reinforcement learning, in the simulation environment, by determining the state vector of the humanoid robot, calling the preset control strategy neural network to generate the output vector, performing linear transformation to obtain the desired joint angle, and using The proportional differential controller calculates the output torque, simulates the trajectory movement of the humanoid robot, collects trajectory data and updates the network weights to achieve the desired gait movement.

Benefits of technology

Simulating the movement trajectory of the humanoid robot in the simulation environment reduces training costs, avoids the influence of external disturbances, improves control accuracy and stability, reduces the risk of damage to the hardware system, and achieves effective gait data collection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118938645A_ABST
    Figure CN118938645A_ABST
Patent Text Reader

Abstract

The invention provides a multi-gait biped motion control method and device based on deep reinforcement learning, and the method comprises the steps: determining a state vector of a current humanoid robot, the state vector comprising a joint parameter, a hip parameter and a biped phase parameter; calling a preset control strategy neural network in a simulation environment, and determining an output vector of the preset control strategy neural network based on the state vector; performing linear transformation based on the output vector to obtain an expected joint angle; a proportional differential controller is called based on the expected joint angle, and the output torque is determined; simulating the trajectory motion of the humanoid robot in a simulation environment based on the output torque to obtain sample data of the trajectory data; and when the number of the sample data is greater than a preset minimum sample value, updating a network weight of a preset control strategy neural network based on the trajectory data to obtain an updated control strategy neural network so as to control the humanoid robot to perform expected gait motion.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of humanoid robot control technology, and in particular to a multi-gait bipedal motion control method and device based on deep reinforcement learning. Background Art

[0002] With the advancement of science and technology and the increasing demand for robot functions, the research on humanoid robots has gradually become an interdisciplinary field, involving multiple fields such as mechanical engineering, control theory, computer vision, and human-computer interaction.

[0003] Since humanoid robots need to simulate complex human movements, they have high requirements for materials and processes, and therefore their manufacturing costs are relatively high. In related technologies, when controlling a humanoid robot to simulate walking in an actual environment, the humanoid robot is prone to lose balance and fall over when subjected to large external disturbances, making it impossible to collect effective gait data and possibly causing damage to the humanoid robot's hardware system.

[0004] It can be seen that the multi-gait bipedal motion control method based on deep reinforcement learning in the related technology has the problem that the training and control cost of humanoid robots is high and is easily affected by external disturbances. Summary of the invention

[0005] The present invention provides a multi-gait bipedal motion control method and device based on deep reinforcement learning, which are used to solve the defects in the prior art that the training and control cost of humanoid robots is high and they are easily affected by external disturbances, and realize controlling the humanoid robot to move according to the desired gait in a simulation environment.

[0006] The present invention provides a multi-gait bipedal motion control method based on deep reinforcement learning, comprising the following steps: determining the state vector of the current humanoid robot, wherein the state vector includes joint parameters, hip parameters and leg phase parameters; calling a preset control strategy neural network in a simulation environment based on the state vector to determine the output vector of the preset control strategy neural network; performing a linear transformation based on the output vector to obtain the desired joint angle; calling a proportional differential controller based on the desired joint angle to determine the output torque; simulating the trajectory motion of the humanoid robot in the simulation environment based on the output torque to obtain sample data of the trajectory data; when the number of the sample data is greater than the preset minimum sample value, updating the network weight of the preset control strategy neural network based on the trajectory data to obtain an updated control strategy neural network to control the humanoid robot to perform the desired gait motion.

[0007] According to a multi-gait bipedal motion control method based on deep reinforcement learning provided by the present invention, each leg of the humanoid robot includes m joints, wherein the m joints include n active joints. Before determining the state vector of the current humanoid robot, the method also includes: constructing a neural network based on a long short-term memory network including two hidden layers to obtain a preset control strategy neural network, wherein the preset control strategy neural network includes an input layer, a hidden layer and an output layer; the input layer is used to input joint parameters, hip parameters and double-leg phase parameters, wherein the joint parameters include 2m joint angles and 2m joint angle angular velocities, the hip parameters include 3-dimensional hip motion line acceleration, 4-dimensional hip rotation angle, and 3-dimensional hip angular velocity, and the double-leg phase parameters include 2-dimensional double-leg phase; the hidden layer includes 2 hidden layer units, each with 128 units; the output layer includes 2n output units.

[0008] According to a multi-gait bipedal motion control method based on deep reinforcement learning provided by the present invention, the trajectory motion of the humanoid robot is simulated in the simulation environment based on the output torque to obtain sample data of trajectory data, including: applying the output torque to the corresponding active joint angle driver based on the simulation environment to obtain the next moment state vector of the humanoid robot; determining the single-step reward value of the humanoid robot based on the next moment state vector, wherein the single-step reward value includes a gait reward, an overall motion reward, and a smooth control reward; calling a preset comment network to evaluate the state vector of the humanoid robot to obtain a state evaluation value corresponding to the state vector; and using the state vector, the output vector, the single-step reward value, and the state evaluation value as sample data of trajectory data.

[0009] According to a multi-gait bipedal motion control method based on deep reinforcement learning provided by the present invention, the proportional differential controller is called based on the expected joint angle to determine the output torque, including: determining a first difference between the expected joint angle and the current joint angle, and taking the product of the first difference and a preset proportional gain as the gain joint angle; determining a second difference between the expected joint angle angular velocity and the current joint angle angular velocity, and taking the product of the second difference and a preset differential gain as the gain joint angle velocity; and determining the sum of the gain joint angle and the gain joint angle velocity as the output torque.

[0010] According to a multi-gait bipedal motion control method based on deep reinforcement learning provided by the present invention, the network weights of the preset control strategy neural network are updated based on the trajectory data to obtain an updated control strategy neural network, including: determining the current advantage function based on the error between the single-step reward system and the state evaluation value of each sample data in the trajectory data; determining the loss function of the preset control strategy neural network based on the current advantage function; and updating the network weights of the preset control strategy neural network based on the loss function of the preset control strategy neural network to obtain an updated control strategy neural network.

[0011] According to a multi-gait bipedal motion control method based on deep reinforcement learning provided by the present invention, the method also includes: determining the product of a preset discount function and the single-step reward value of each sample data in the trajectory data as the actual single-step reward value of each sample data; determining the minimum mean square error between the state evaluation value of each sample data in the trajectory data and the actual single-step reward value of each sample data; determining the loss function of a preset comment network based on the minimum mean square error; and updating the network weights of the preset comment network based on the loss function of the preset comment network to obtain an updated comment network.

[0012] The present invention also provides a multi-gait bipedal motion control device based on deep reinforcement learning, comprising the following modules: an acquisition module, used to determine the state vector of the current humanoid robot, wherein the state vector includes joint parameters, hip parameters and leg phase parameters; a first determination module, used to call a preset control strategy neural network in a simulation environment based on the state vector to determine the output vector of the preset control strategy neural network; a transformation module, used to perform a linear transformation based on the output vector to obtain an expected joint angle; a second determination module, used to call a proportional differential controller based on the expected joint angle to determine the output torque; a simulation module, used to simulate the trajectory motion of the humanoid robot in the simulation environment based on the output torque to obtain sample data of the trajectory data; an update module, used to update the network weights of the preset control strategy neural network based on the trajectory data when the number of the sample data is greater than a preset minimum sample value, to obtain an updated control strategy neural network to control the humanoid robot to perform the expected gait motion.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements any of the multi-gait bipedal motion control methods based on deep reinforcement learning as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multi-gait bipedal motion control methods based on deep reinforcement learning as described above.

[0015] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the multi-gait bipedal motion control methods based on deep reinforcement learning as described above.

[0016] The multi-gait biped motion control method based on deep reinforcement learning provided by the present invention determines the state vector of the current humanoid robot, calls a preset control strategy neural network in a simulation environment based on the state vector, determines the output vector of the preset control strategy neural network; performs linear transformation based on the output vector to obtain the expected joint angle; calls a proportional differential controller based on the expected joint angle to determine the output torque; simulates the trajectory motion of the humanoid robot in the simulation environment based on the output torque to obtain sample data of the trajectory data, thereby being able to simulate the actual motion trajectory of the humanoid robot in the simulation environment to obtain the trajectory data; when the number of sample data is greater than the minimum sample value, updates the preset control strategy neural network, thereby being able to obtain an updated control strategy neural network with better deep learning ability, so as to control the humanoid robot to perform the expected gait motion through the updated control strategy neural network; thereby solving the problem that the multi-gait biped motion control method based on deep reinforcement learning in the related art has the high training and control cost of the humanoid robot and is easily affected by external disturbances. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the multi-gait bipedal motion control method based on deep reinforcement learning provided by the present invention.

[0019] Figure 2 It is a schematic diagram of a process of simulating trajectory data in a simulation environment provided by the present invention.

[0020] Figure 3 It is a schematic diagram of the flow of updating the network weights of the preset control strategy neural network provided by the present invention.

[0021] Figure 4It is a structural schematic diagram of a multi-gait bipedal motion control device based on deep reinforcement learning provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] It has always been the dream of relevant researchers to create robots that are similar to humans in appearance and function, and to achieve robots that imitate and surpass human behavior and motor ability, so as to provide more diverse and efficient services and support for human society. With the advancement of science and technology and the increasing demand for robot functions, the research on humanoid robots has gradually become an interdisciplinary field, involving mechanical engineering, control theory, computer vision, human-computer interaction and other fields. Researchers strive to simulate the structure and function of the human skeleton, muscle and nervous system, and explore the dynamics, motion planning and control methods of robots to achieve robust movement, coordinated action and intelligent decision-making of robots.

[0025] The models of humanoid robots are often high-dimensional, mixed, and nonlinear systems. Among them, bipedal motion control, as a basic, core, and challenging problem, has attracted long-term research. With the rapid development of technologies such as neural networks in recent years, bipedal robot motion control methods based on deep reinforcement learning have achieved certain results. This type of method does not require the robot to meet a specific expected trajectory, which enhances the diversity of strategies. At the same time, there is no need to accurately model the robot's dynamic model. Instead, the neural network provides the ability to handle high-dimensional and nonlinear problems, and has achieved good results in bipedal motion control. In addition, training control strategies in a simulation environment can facilitate debugging without damaging the robot hardware system, which improves safety.

[0026] The present invention is aimed at a humanoid robot system that performs motion control based on a two-leg controller, and proposes a method for performing deep reinforcement learning training in a simulation environment so that the robot can learn motion control strategies under multiple gaits. The present invention uses the Mujoco simulation environment (a general physics engine designed to promote robotics, biomechanics, graphics and animation, machine learning, and other fields that require rapid and accurate simulation of articulated structures that interact with their environment) to build a robot and a corresponding environment. The robot state and some control instructions obtained by the sensor are used as inputs, and the corresponding joint angle angle / speed / torque and other outputs are generated through the strategy network. After a simple controller is input to the robot to perform a single-step action and obtain a corresponding reward, the algorithm randomly samples and updates the strategy network in the simulation environment so that it converges to a stable control strategy.

[0027] refer to Figure 1 , Figure 1 is a flow chart of a multi-gait bipedal motion control method based on deep reinforcement learning provided by the present invention, such as Figure 1 As shown, the method includes the following steps.

[0028] Step 101: Determine a state vector of the current humanoid robot, wherein the state vector includes joint parameters, hip parameters, and leg phase parameters.

[0029] In the embodiment of the present invention, the real-time state vector of the humanoid robot is obtained by using a state sensor pre-set in the humanoid robot. Here, the state vector includes joint parameters, hip parameters and double-leg phases.

[0030] In some embodiments, the gait information of the humanoid robot can also be obtained through state sensors pre-set in the humanoid robot, for example, the gait of the humanoid robot such as stepping, walking, jumping, and corresponding parameters such as joint angle, joint angle velocity, and joint angle change.

[0031] Through the embodiments provided by the present invention, the state vector of the humanoid robot can be obtained through the sensors arranged in the humanoid robot, so as to facilitate subsequent simulation control in a simulation environment.

[0032] Step 102: Invoke a preset control strategy neural network in a simulation environment based on a state vector to determine an output vector of the preset control strategy neural network.

[0033] Among them, the simulation environment can be a general-purpose physics engine, such as the Mujoco simulation environment, which is used to facilitate robotics, biomechanics, graphics and animation, machine learning and other fields that require fast and accurate simulation of articulated structures interacting with their environment.

[0034] In an embodiment of the present invention, a strategy network is used to control the robot to interact with the simulation environment. In the Mujoco simulation environment, the state vector of the humanoid robot required by the preset control strategy neural network input layer is read, and after being input into the preset control strategy neural network, 2n values ​​in the range of [-1, 1] are output.

[0035] Step 103: Perform a linear transformation based on the output vector to obtain a desired joint angle.

[0036] The above preset control strategy neural network outputs 2n values ​​in the range of [-1, 1]. After linear transformation, the desired joint angle is obtained, which can be expressed by the following formula (1): q di =0.5(upperbound i -lowerbound i )(output i +1)+lowerbound i (1)

[0037] Among them, q di represents the expected angle of the i-th joint angle (expected joint angle), upperbound i ,lowerbound i Respectively represent the upper and lower bounds of the i-th joint angle, output i Represents the i-th value of the output layer of the preset control strategy neural network.

[0038] In an embodiment of the present invention, the angles of the humanoid robot's hip roll joint (a joint that allows the robot to rotate in the front-to-back direction (usually around the X-axis)) and the humanoid robot's hip yaw joint (a joint that allows the robot to rotate in the horizontal direction (usually around the Z-axis)) are additionally restricted to a smaller range (for example, the range value is -0.1, 0.1]).

[0039] Step 104: Call a proportional-differential controller based on the desired joint angle to determine the output torque.

[0040] The proportional-derivative control is then used to calculate the torque applied to each joint angle, which can be expressed by the following formula (2):

[0041] Where τ represents the torque applied to each joint angle, kp represents the proportional gain, and k d represents the differential gain, q, Respectively represent the current joint angle and the current joint angle velocity, q d , Represent the expected joint angle and the expected joint angle velocity respectively. In practical applications,

[0042] Among them, the proportional-derivative (PD) controller is a controller with a proportional-derivative control law. It calculates the control output based on the system error and the rate of change of the error to achieve control of the stability and response speed of the system.

[0043] Step 105: Simulate the trajectory motion of the humanoid robot in a simulation environment based on the output torque to obtain sample data of the trajectory data.

[0044] In an embodiment of the present invention, a Mujoco simulation environment is used to construct a humanoid robot and a corresponding environment. The state vector of the humanoid robot obtained by the sensor and the control command of the user are used as input, and the corresponding joint angle / speed / torque and other outputs are generated through a preset control strategy neural network. The preset controller (for example, a proportional differential controller) is input to the humanoid robot to perform a single-step action and obtain a corresponding reward. The algorithm randomly samples and updates the preset control strategy neural network in the simulation environment to converge it to a stable control strategy.

[0045] In the embodiment of the present invention, each sample data in the trajectory data corresponding to the humanoid robot includes a state vector, an output vector, a single-step reward value, and a state evaluation value.

[0046] Through the embodiments provided by the present invention, by determining the state vector of the current humanoid robot and calling a preset control strategy neural network in a simulation environment to read the above control vector, the motion trajectory of the actual humanoid robot can be simulated in the simulation environment to obtain multiple trajectory data, thereby avoiding the problem of high training control cost of the humanoid robot and susceptibility to external disturbances in actual scenarios.

[0047] Step 106: When the number of sample data is greater than a preset minimum sample value, the network weights of the preset control strategy neural network are updated based on the trajectory data to obtain an updated control strategy neural network to control the humanoid robot to perform a desired gait movement.

[0048] In an embodiment of the present invention, the minimum sample size N required for a single network weight update is set (in this specific embodiment, N=50000), trajectory data is collected cyclically until the sum of all trajectory data sample sizes is not less than N, the network weights of the preset control strategy neural network are updated, and an updated control strategy neural network is obtained. The humanoid robot is controlled to perform the desired gait movement through the updated control strategy neural network.

[0049] It should be noted that when the network weights of the preset control strategy neural network are updated, the network weights of the preset comment network may also be updated.

[0050] Through the above steps of the embodiment of the present invention, by determining the state vector of the current humanoid robot, wherein the state vector includes joint parameters, hip parameters and leg phase parameters; calling the preset control strategy neural network in the simulation environment based on the state vector, determining the output vector of the preset control strategy neural network; performing linear transformation based on the output vector to obtain the desired joint angle; calling the proportional differential controller based on the desired joint angle to determine the output torque; simulating the trajectory motion of the humanoid robot in the simulation environment based on the output torque to obtain sample data of the trajectory data; when the number of sample data is greater than the preset minimum sample value, updating the network weights of the preset control strategy neural network based on the trajectory data to obtain the updated control strategy neural network to control the humanoid robot to perform the desired gait motion. Thus, the multi-gait bipedal motion control method based on deep reinforcement learning in the related art is solved, and the problem that the training and control cost of the humanoid robot is high and it is easily affected by external disturbances is solved.

[0051] According to a multi-gait bipedal motion control method based on deep reinforcement learning provided by the present invention, each leg of the humanoid robot includes m joints, wherein the m joints include n active joints. Before determining the current state vector of the humanoid robot, the method further includes: A neural network is constructed based on a long short-term memory network including two hidden layers to obtain a preset control strategy neural network, wherein the preset control strategy neural network includes an input layer, a hidden layer, and an output layer; The input layer is used to input joint parameters, hip parameters and leg phase parameters, where the joint parameters include 2m joint angles and 2m joint angle velocities, the hip parameters include 3D hip motion line acceleration, 4D hip rotation angle, 3D hip angular velocity, and the leg phase parameters include 2D leg phase; the hidden layer includes 2 hidden layer units with 128 units each; the output layer includes 2n output units.

[0052] In an embodiment of the present invention, a neural network describing the robot control strategy is constructed, that is, a long short-term memory network (LSTM) containing two hidden layers is used to describe the humanoid robot control strategy. Here, it is assumed that each leg of the humanoid robot contains m joints, of which n are active joints. The input layer of the neural network is composed of 2m joint angles of both legs, 2m joint angle angular velocities, 3D hip motion line acceleration, 4D hip rotation angle (quaternion), 3D hip angular velocity and 2D leg phase, forming a 4m+12-dimensional vector. The output layer of the neural network is composed of 2n units, and is scaled to the range of [-1,1] by the tanh activation function, and the number of units in the two hidden layers is 128.

[0053] It should be noted that in the embodiment of the present invention, the algorithm adopts an actor-critic architecture, and uses a preset comment network (i.e., Critic) to evaluate the above-mentioned preset control strategy neural network (i.e., Actor), wherein the preset comment network is the same as the above-mentioned preset control strategy neural network (structure), but the last layer does not use the tanh activation function, but uses a linear layer to map to a scalar. The neural network parameters are randomly initialized.

[0054] Here, the actor-critic architecture is a reinforcement learning algorithm. Its core idea is to combine the policy gradient and the value function to solve the reinforcement learning problem in continuous action space and high-dimensional state space. The actor-critic architecture combines the two components of actor and critic. The actor component (corresponding to the preset control policy neural network mentioned above) is responsible for learning the strategy and generating actions; while the critic component (corresponding to the preset comment network mentioned above) is responsible for learning the value function and evaluating the value of the state or state-action pair.

[0055] refer to Figure 2 , Figure 2 It is a schematic diagram of a process of simulating trajectory data in a simulation environment provided by the present invention.

[0056] According to a multi-gait bipedal motion control method based on deep reinforcement learning provided by the present invention, the trajectory motion of a humanoid robot is simulated in a simulation environment based on the output torque to obtain sample data of trajectory data, including: Step 201 : applying the output torque to the corresponding active joint angle actuator based on the simulation environment to obtain the state vector of the humanoid robot at the next moment.

[0057] Step 202, determining a single-step reward value of the humanoid robot based on the state vector at the next moment, wherein the single-step reward value includes a gait reward, an overall motion reward, and a smooth control reward.

[0058] Step 203: Call a preset review network to evaluate the state vector of the humanoid robot to obtain a state evaluation value corresponding to the state vector.

[0059] Step 204: Use the state vector, output vector, single-step reward value, and state evaluation value as sample data of trajectory data.

[0060] In the embodiment of the present invention, the output torque τ is input into the simulation environment and applied to the actuators corresponding to the active joint angles for single-step simulation to obtain the robot state at the next moment and calculate the corresponding reward. The reward function mainly includes gait reward, overall motion reward, and smooth control reward.

[0061] The gait reward specifically includes: for periodic gaits such as stepping, walking, and jumping, within a cycle, each leg state is divided into a stance phase and a swing phase; in the stance phase, the robot's foot movement speed is penalized, and in the swing phase, the robot's foot force is penalized. It can be expressed by the following formula (3):

[0062] Among them, r bipedal For gait reward, I left swing -1 during the left leg swing phase and 0 during the rest of the phases. -1 during the left leg stance phase and 0 during the rest of the phases. left is the left leg plantar force, S left is the left foot movement speed. The meanings of the parameters of the right leg are the same as those of the left leg, so they will not be repeated here.

[0063] The overall motion reward includes: for gaits such as stepping, walking, and jumping, it is expected that the robot should avoid lateral deviation, that is, the expected lateral speed is 0; for stepping gaits, the expected forward speed is also 0; for gaits such as walking and jumping, different expected forward speeds can be given. In addition, during the movement, it is expected that the upper body remains upright, that is, the hips face forward. It can be expressed by the following formula (4):

[0064] Among them, r command For overall movement reward, v x ,v y are the forward and lateral velocities of the robot hip, v xd , v yd is the expected forward and lateral velocity of the robot, O Tis the robot hip orientation (quaternion), O d is the quaternion (1, 0, 0, 0) corresponding to the front.

[0065] The stable control reward includes: during the robot movement, it is expected to maintain the stability of the robot body. On the one hand, the change of the robot's single-step joint angle is reduced to avoid large torque; on the other hand, the swing amplitude of the body is reduced. It can be expressed by the following formula (5):

[0066] Among them, r smooth To smoothly control the reward, a t represents the desired joint angle at step t (time t), τ represents the torque calculated by the PD controller, and p acc , p avel Represents the linear acceleration and angular velocity of the fuselage.

[0067] The final single-step reward can be expressed by the following formula (6): r=0.6r bipedal +0.3r command +0.1r smooth +3 (6)

[0068] Among them, r is the single-step reward, r bipedal is the gait reward, r command For overall exercise rewards, r smooth Reward for smooth control.

[0069] Subsequently, multiple trajectory data are collected and the network weights are updated; the initial state of the humanoid robot in the simulation environment, including the whole body joint angles, angular velocities and initial positions, is set so that the humanoid robot presents a two-foot standing posture, and the preset control strategy neural network is used cyclically to control the robot to interact with the simulation environment until the termination condition is met; the 4m+12-dimensional state vector s of the humanoid robot is recorded each time it is executed t , the preset control strategy neural network 2n-dimensional output vector a t , single-step reward value r t , the state evaluation value v of the preset comment network for the current state t ; To obtain a trajectory data obtained by the interaction between the humanoid robot and the simulation environment

[0070] In this specific embodiment, the termination conditions are composed of the following three parts: (1) The humanoid robot's hip height is lower than 0.6m or higher than 1.2m (the robot's initial hip height is 1m). (2) The humanoid robot's body position is greater than 0.6m from the expected position. (3) The number of interactions between the humanoid robot and the environment is greater than or equal to 300 (i.e., the length of a single trajectory does not exceed 300).

[0071] According to a multi-gait bipedal motion control method based on deep reinforcement learning provided by the present invention, a proportional differential controller is called based on a desired joint angle to determine the output torque, including: Determine a first difference between the desired joint angle and the current joint angle, and use the product of the first difference and a preset proportional gain as the gain joint angle; Determine a second difference between the desired joint angle velocity and the current joint angle velocity, and use the product of the second difference and a preset differential gain as the gain joint angle velocity; The sum of the gain joint angle and the gain joint angle velocity is determined as the output torque.

[0072] In the embodiment of the present invention, a strategy network is used to control the robot to interact with the simulation environment. In the Mujoco simulation environment, the state vector of the humanoid robot required by the preset control strategy neural network input layer is read, and after inputting into the preset control strategy neural network, 2n values ​​in the range of [-1, 1] are output. The linear transformation of the value obtains the desired joint angle, which can be expressed by the following formula (7): q di =0.5(upperbound i -lowerbound i )(output i +1)+lowerbound i (7)

[0073] Among them, q di represents the expected angle of the i-th joint angle (expected joint angle), upperbound i ,lowerbound i Respectively represent the upper and lower bounds of the i-th joint angle, output i Represents the i-th value of the output layer of the preset control strategy neural network.

[0074] In an embodiment of the present invention, the angles of the humanoid robot's hip roll joint (a joint that allows the robot to rotate in the front-to-back direction (usually around the X-axis)) and the humanoid robot's hip yaw joint (a joint that allows the robot to rotate in the horizontal direction (usually around the Z-axis)) are additionally restricted to a smaller range (for example, the range value is -0.1, 0.1]).

[0075] The proportional-derivative control is then used to calculate the torque applied to each joint angle, which can be expressed by the following formula (8):

[0076] Among them, τ represents the torque applied to each joint angle, k p represents the proportional gain, k d represents the differential gain, q, Respectively represent the current joint angle and the current joint angle velocity, q d , Represent the expected joint angle and the expected joint angle velocity respectively. In practical applications,

[0077] According to a multi-gait bipedal motion control method based on deep reinforcement learning provided by the present invention, the network weights of a preset control strategy neural network are updated based on trajectory data to obtain an updated control strategy neural network, including: determining a current advantage function based on the error between a single-step reward system and a state evaluation value of each sample data in the trajectory data; determining a loss function of a preset control strategy neural network based on the current advantage function; The network weights of the preset control strategy neural network are updated based on the loss function of the preset control strategy neural network to obtain an updated control strategy neural network.

[0078] The minimum sample size N required for a single network update is set (in this specific embodiment, N=50000), and trajectory data is collected cyclically until the sum of all trajectory data sample sizes is not less than N, and then the network weights of the preset control strategy neural network are updated according to the following steps: refer to Figure 3 , Figure 3 : is a schematic diagram of the flow chart of updating the network weights of the preset control strategy neural network provided by the present invention, such as Figure 3 As shown, the method comprises the following steps: Step 301: Calculate the advantage function using the generalized advantage estimation method.

[0079] Among them, generalized advantage estimation (GAE) is a technique used in reinforcement learning to estimate the advantage of each state and action. It combines the characteristics of Monte Carlo method and TD error method, aiming to reduce estimation variance and improve sample efficiency. Generalized advantage estimation combines advantage estimates of multiple time steps by exponentially weighted averaging to obtain a more accurate estimate. This method allows the algorithm to reuse the same data samples in multiple updates, thereby improving data efficiency.

[0080] The advantage function is calculated using the generalized advantage estimation method, which can be specifically expressed by the following formulas (9) and (10):

[0081] in, is the TD error of the current state (step t) (i.e., the error between the actual value and the predicted value of each trajectory data), v t is the state evaluation value of the preset comment network for the current state (step t), r t is the single-step reward value, γ is the discount factor, v t+1 It is the state evaluation value of the preset comment network for the next state (t+1 step).

[0082]

[0083] Among them, A t is the advantage function of the current state (step t), γ is the discount coefficient, λ is the coefficient for balancing variance and deviation, and l is the index variable for summation, which is used to traverse all future time steps starting from the current time step t. is the TD error at t+1 steps, where v represents the value function.

[0084] In actual calculations, the advantage value A at the end of the trajectory is usually calculated first n (the advantage value of the nth step), and then iterate to solve the advantage value of the entire trajectory. Specifically, it can be expressed by the following formula (11):

[0085] Among them, A t is the advantage function of the current state (step t), γ is the discount coefficient, λ is the coefficient for balancing variance and deviation, and l is the index variable for summation, which is used to traverse all future time steps starting from the current time step t. is the TD error at t+1 steps, is the TD error of the current state (step t), is the TD error at t+1+1 steps, A t+1 is the advantage function of the next state (step t+1), where v represents the value function.

[0086] It should be noted that the network weight update includes k epochs (epoch means a traversal of the model over the entire training dataset), and in each epoch all samples are divided into multiple batches (batch means a subset of data selected from the entire dataset and used for a forward / backward propagation when training or evaluating a neural network). Each batch updates the network weights of the control strategy neural network and the comment network.

[0087] In the embodiment of the present invention, the number of epochs is 4, and each batch contains 64 trajectories.

[0088] Step 302: Determine the loss function of the preset control strategy neural network based on the current advantage function to update the network weights of the preset control strategy neural network.

[0089] In the embodiment of the present invention, the Adam (Adaptive Moment Estimation) optimizer is used to update the parameters, wherein the Adam optimizer is a widely used deep learning optimization algorithm, and the loss function of the preset control strategy neural network can be specifically expressed by the following formula (12): L actor (θ) = E t [min(r t (θ)A t ,clip(r t (θ), 1-∈, 1+∈)A t )] (12)

[0090] Among them, L actor (θ) represents the loss function of the network weight update of the preset control strategy neural network, θ is the strategy network parameter vector, E t represents the expectation for t, r t (θ) represents the ratio. For details, please refer to formula (13). A t Denotes the advantage function used to evaluate the state s t Take action a t Relative to the average performance, clip(r t (θ), 1-∈, 1+∈) is a clipping function used to t The value of (θ) is restricted to [1-∈, 1+∈], where ∈ is the cutoff coefficient (usually 0.2).

[0091] r t The (θ) ratio can be expressed by the following formula (13):

[0092] Among them, r t (θ) represents the ratio of the policy network parameter vector, π θ is the policy network, s t represents the state at step t (time), a t represents the action taken at step t.

[0093] In neural network training, the loss function is used to measure the difference between the model's predicted value and the actual value. The purpose of updating the neural network (i.e. adjusting its weights and biases) is to minimize this loss function. This process is usually implemented through backpropagation and gradient descent or their variants.

[0094] In an embodiment of the present invention, trajectory data collection and network weight updating steps are executed cyclically until the preset control strategy neural network converges to obtain an updated control strategy neural network. The strategy network of the (converged) updated control strategy neural network takes the current state of the humanoid robot as input and outputs the desired joint angle in the current state. After the PD controller outputs the corresponding torque, the robot can be controlled to move according to the desired gait.

[0095] According to a multi-gait bipedal motion control method based on deep reinforcement learning provided by the present invention, the method further includes: Determine the product of a preset discount function and the single-step reward value of each sample data in the trajectory data as the actual single-step reward value of each sample data; Determine the minimum mean square error between the state evaluation value of each sample data in the trajectory data and the actual single-step reward value of each sample data; Determine the loss function of the preset review network based on the minimum mean square error; Based on the loss function of the preset comment network, the network weights of the preset comment network are updated to obtain an updated comment network.

[0096] In the embodiment of the present invention, the Adam optimizer is used to update the parameters, and the loss function of the preset review network can be expressed by the following formula (14):

[0097] Among them, L critic (θ) represents the loss function for updating the network weights of the preset review network, E t represents the expectation for all time steps t, MSE is the minimum mean square error, v t represents the value function of the tth step, k is a summation index variable used to traverse all future time steps starting from the current time step t, γ krepresents the discount function, r t+k+1 represents the reward obtained at time step t+k+1.

[0098] In neural network training, the loss function is used to measure the difference between the model's predicted value and the actual value. The purpose of updating the neural network (i.e. adjusting its weights and biases) is to minimize this loss function. This process is usually implemented through backpropagation and gradient descent or their variants.

[0099] The present invention is aimed at a humanoid robot system that performs motion control based on a two-leg controller, and proposes a method for performing deep reinforcement learning training in a simulation environment, so that the robot can learn motion control strategies under multiple gaits. The present invention uses the Mujoco simulation environment to build the robot and the corresponding environment, and uses the robot state and some control instructions obtained by the sensor as input. The corresponding outputs such as joint angle / speed / torque are generated through the strategy network. A simple controller is used to input the robot to perform a single-step action and obtain the corresponding reward. The algorithm randomly samples and updates the strategy network in the simulation environment, so that it converges to a stable control strategy.

[0100] The multi-gait biped motion control device based on deep reinforcement learning provided by the present invention is described below. The multi-gait biped motion control device based on deep reinforcement learning described below and the multi-gait biped motion control method based on deep reinforcement learning described above can be referenced to each other.

[0101] refer to Figure 4 , Figure 4 is a schematic diagram of the structure of a multi-gait bipedal motion control device based on deep reinforcement learning provided by the present invention. Figure 4As shown, the device includes: an acquisition module 401, a first determination module 402, a transformation module 403, a second determination module 404, a simulation module 405 and an update module 406, wherein the acquisition module 401 is used to determine the state vector of the current humanoid robot, wherein the state vector includes joint parameters, hip parameters and leg phase parameters; the first determination module 402 is used to call the preset control strategy neural network in the simulation environment based on the state vector to determine the output vector of the preset control strategy neural network; the transformation module 403 is used to perform linear transformation based on the output vector to obtain the expected joint angle; the second determination module 404 calls the proportional differential controller based on the expected joint angle to determine the output torque; the simulation module 405 is used to simulate the trajectory motion of the humanoid robot in the simulation environment based on the output torque to obtain sample data of the trajectory data; the update module 406 is used to update the network weights of the preset control strategy neural network based on the trajectory data when the number of sample data is greater than the preset minimum sample value to obtain the updated control strategy neural network to control the humanoid robot to perform the expected gait motion.

[0102] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a multi-gait bipedal motion control method based on deep reinforcement learning, the method comprising: determining the state vector of the current humanoid robot, wherein the state vector includes joint parameters, hip parameters and leg phase parameters; calling a preset control strategy neural network in a simulation environment based on the state vector to determine the output vector of the preset control strategy neural network; performing a linear transformation based on the output vector to obtain the desired joint angle; calling a proportional differential controller based on the desired joint angle to determine the output torque; simulating the trajectory motion of the humanoid robot in the simulation environment based on the output torque to obtain sample data of the trajectory data; when the number of sample data is greater than the preset minimum sample value, updating the network weight of the preset control strategy neural network based on the trajectory data to obtain an updated control strategy neural network to control the humanoid robot to perform the desired gait motion.

[0103] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0104] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-gait bipedal motion control method based on deep reinforcement learning provided by the above methods, the method including: determining the state vector of the current humanoid robot, wherein the state vector includes joint parameters, hip parameters and leg phase parameters; calling a preset control strategy neural network in a simulation environment based on the state vector to determine the output vector of the preset control strategy neural network; performing a linear transformation based on the output vector to obtain the expected joint angle; calling a proportional differential controller based on the expected joint angle to determine the output torque; simulating the trajectory motion of the humanoid robot in the simulation environment based on the output torque to obtain sample data of the trajectory data; when the number of sample data is greater than the preset minimum sample value, updating the network weights of the preset control strategy neural network based on the trajectory data to obtain an updated control strategy neural network to control the humanoid robot to perform the expected gait motion.

[0105] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multi-gait bipedal motion control method based on deep reinforcement learning provided by the above-mentioned methods, the method comprising: determining the state vector of the current humanoid robot, wherein the state vector comprises joint parameters, hip parameters and leg phase parameters; calling a preset control strategy neural network in a simulation environment to determine an output vector of the preset control strategy neural network based on the state vector; performing a linear transformation based on the output vector to obtain a desired joint angle; calling a proportional differential controller based on the desired joint angle to determine the output torque; simulating the trajectory motion of the humanoid robot in the simulation environment based on the output torque to obtain sample data of the trajectory data; when the number of sample data is greater than a preset minimum sample value, updating the network weights of the preset control strategy neural network based on the trajectory data to obtain an updated control strategy neural network to control the humanoid robot to perform the desired gait motion.

[0106] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-gait bipedal motion control method based on deep reinforcement learning, characterized in that: include: Determine a state vector of the current humanoid robot, wherein the state vector includes joint parameters, hip parameters, and leg phase parameters; Calling a preset control strategy neural network in a simulation environment based on the state vector to determine an output vector of the preset control strategy neural network; Performing a linear transformation based on the output vector to obtain a desired joint angle; Invoking a proportional-derivative controller based on the desired joint angle to determine an output torque; Simulating the trajectory motion of the humanoid robot in the simulation environment based on the output torque to obtain sample data of trajectory data; When the number of the sample data is greater than a preset minimum sample value, the network weights of the preset control strategy neural network are updated based on the trajectory data to obtain an updated control strategy neural network to control the humanoid robot to perform a desired gait movement.

2. The multi-gait bipedal motion control method based on deep reinforcement learning according to claim 1, characterized in that: Each leg of the humanoid robot includes m joints, wherein the m joints include n active joints. Before determining the current state vector of the humanoid robot, the method further includes: Constructing a neural network based on a long short-term memory network including two hidden layers to obtain a preset control strategy neural network, wherein the preset control strategy neural network includes an input layer, a hidden layer, and an output layer; The input layer is used to input joint parameters, hip parameters and double-leg phase parameters, wherein the joint parameters include 2m joint angles and 2m joint angle velocities, the hip parameters include 3-dimensional hip motion line acceleration, 4-dimensional hip rotation angle, and 3-dimensional hip angular velocity, and the double-leg phase parameters include 2-dimensional double-leg phase; the hidden layer includes 2 hidden layer units, each with 128 units; the output layer includes 2n output units.

3. The multi-gait bipedal motion control method based on deep reinforcement learning according to claim 1, characterized in that: The step of simulating the trajectory motion of the humanoid robot in the simulation environment based on the output torque to obtain sample data of trajectory data includes: Applying the output torque to the corresponding active joint angle driver based on the simulation environment to obtain a state vector of the humanoid robot at a next moment; Determining a single-step reward value of the humanoid robot based on the next-moment state vector, wherein the single-step reward value includes a gait reward, an overall motion reward, and a smooth control reward; Calling a preset review network to evaluate the state vector of the humanoid robot to obtain a state evaluation value corresponding to the state vector; The state vector, the output vector, the single-step reward value, and the state evaluation value are used as sample data of trajectory data.

4. The multi-gait bipedal motion control method based on deep reinforcement learning according to claim 1, characterized in that: The step of calling a proportional differential controller based on the desired joint angle to determine the output torque includes: Determine a first difference between the desired joint angle and the current joint angle, and use the product of the first difference and a preset proportional gain as a gain joint angle; Determine a second difference between the desired joint angle velocity and the current joint angle velocity, and use the product of the second difference and a preset differential gain as the gain joint angle velocity; The sum of the gain joint angle and the gain joint angle velocity is determined as the output torque.

5. The multi-gait bipedal motion control method based on deep reinforcement learning according to claim 1, characterized in that: The updating of the network weights of the preset control strategy neural network based on the trajectory data to obtain an updated control strategy neural network includes: Determining a current advantage function based on an error between a single-step reward system and a state evaluation value for each sample data in the trajectory data; Determine the loss function of the preset control strategy neural network based on the current advantage function; The network weights of the preset control strategy neural network are updated based on the loss function of the preset control strategy neural network to obtain an updated control strategy neural network.

6. The multi-gait bipedal motion control method based on deep reinforcement learning according to claim 5, characterized in that: The method further comprises: Determine the product of a preset discount function and the single-step reward value of each sample data in the trajectory data as the actual single-step reward value of each sample data; Determine the minimum mean square error between the state evaluation value of each sample data in the trajectory data and the actual single-step reward value of each sample data; Determine a loss function of a preset review network based on the minimum mean square error; The network weights of the preset comment network are updated based on the loss function of the preset comment network to obtain an updated comment network.

7. A multi-gait bipedal motion control device based on deep reinforcement learning, characterized in that: include: An acquisition module, used to determine a state vector of the current humanoid robot, wherein the state vector includes joint parameters, hip parameters, and leg phase parameters; A first determination module is used to call a preset control strategy neural network in a simulation environment to determine an output vector of the preset control strategy neural network based on the state vector; A transformation module, used for performing a linear transformation based on the output vector to obtain a desired joint angle; A second determination module calls a proportional differential controller based on the desired joint angle to determine an output torque; A simulation module, used for simulating the trajectory motion of the humanoid robot in the simulation environment based on the output torque to obtain sample data of trajectory data; An updating module is used to update the network weights of the preset control strategy neural network based on the trajectory data when the number of the sample data is greater than a preset minimum sample value, so as to obtain an updated control strategy neural network to control the humanoid robot to perform a desired gait movement.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the multi-gait bipedal motion control method based on deep reinforcement learning as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-gait bipedal motion control method based on deep reinforcement learning as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-gait bipedal motion control method based on deep reinforcement learning as described in any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • Biped robot complex terrain adaptive gait planning method and biped robot

    CN119644704A

  • Adaptive gait planning method for biped robot on complex terrain and biped robot

    CN119644704B

  • Control method and device for straight knee walking of humanoid robot and storage medium

    CN120116236A

  • A control method, device, and storage medium for a humanoid robot to walk with straight knees

    CN120116236B

  • Ammonia escape concentration real-time prediction and control system based on data analysis

    CN120671532A