Quadruped robot motion control method based on adaptive reinforcement learning

Abstract

The invention discloses a quadruped robot motion control method based on adaptive reinforcement learning, and the method comprises the following steps: constructing an adaptive module and a strategy network, and generating a plurality of simulated terrains and robot instances; acquiring the historical state of the robot, inputting the historical state into an adaptive module for imitation learning training, and outputting the linear velocity estimation of the robot at the current moment, the leg lifting height estimation of each leg of the robot and the potential feature estimation of the surrounding terrain; the output of the self-adaptive module and the current robot state are obtained, a strategy network is input for reinforcement learning training, target angles of multiple motor joints are output, and the target angles are converted into target torque; the robot executes the target torque, calculates a current reward according to the reward function, and updates the state of the robot; and forming a motion controller of the quadruped robot by the trained adaptive module and the strategy network, and performing motion control on the quadruped robot. According to the invention, the robot can be controlled to realize stable and agile movement in a complex terrain.

Description

Technical Field

[0001] The present invention relates to the technical field of robot motion control, and in particular to a quadruped robot motion control method based on adaptive reinforcement learning. Background Art

[0002] In order to take into account the complex kinematic structure, high-dimensional state space and the need for adaptive behavior in different terrains of quadruped robots, it is not easy to develop a robust or agile controller for legged robots. At present, many methods rely on computationally intensive environmental perception methods to achieve agile and stable motion in complex terrains (such as stairs, grass, and slopes). Cameras and lidars are used to perceive the surrounding terrain and plan the trajectory and footholds of the feet. These front-end perception parts are easily affected by lighting conditions and weather conditions, which may affect back-end calculations such as foothold planning. In contrast, proprioceptive sensors such as joint encoders and inertial sensors IMU (Inertial Measurement Unit) are more powerful and reliable than external perception sensors (cameras, lidars), but it is difficult to achieve stable and agile motion in complex terrains using only proprioceptive sensors.

[0003] The controller of the quadruped robot is designed based on the model predictive control method (MPC). The dynamic equations are used to describe the robot's leg motion, joint torque, and the interaction between the foot and the ground to build a mathematical model that can predict the robot's motion state (such as position, speed, and force). At the same time, physical constraints (joint range of motion, torque limit, foot contact conditions, etc.) and dynamic balance (keeping the robot's center of mass in the stable area of ​​the support surface) are considered. The motion trajectory in the future is predicted and the optimal control input (such as joint torque) that can meet the goal is found through the optimization algorithm. However, due to the uncertainty of unforeseen events, the blind motion controller designed using the model prediction algorithm can often only walk on flat ground and cannot adapt to complex terrain.

[0004] At present, there are many methods that use reinforcement learning to design blind motion controllers for quadruped robots. Reinforcement learning (RL) is a unified framework that manually designs various rewards, allows robots to try different actions in the environment, observes state changes and reward values, encourages agents to obtain the maximum reward, and finally obtains the optimal control strategy. Reinforcement learning can easily design blind motion controllers. However, reinforcement learning algorithms require continuous trial and error and exploration in the robot simulation environment, and require a long period of exploration training to converge. Moreover, the actions after convergence are often not natural enough. Therefore, existing methods often use multi-stage training, introduce expert demonstration data, let robots imitate the performance of experts, speed up iterations, and avoid unnatural movements. However, multi-stage training brings higher training costs. At the same time, the introduction of expert demonstration data will limit the robot's movement performance to expert demonstrations, fall into local optimal situations, and make it difficult to adapt to various complex terrains. Summary of the invention

[0005] In order to overcome the defects and shortcomings of the prior art, the present invention provides a quadruped robot motion control method based on adaptive reinforcement learning. The present invention designs a quadruped robot blind motion controller only through proprioceptive sensors, and uses proprioceptive sensor data to estimate the robot's own state and external terrain, allowing the robot to "explore" and adapt to various complex terrains (such as stairs, grass, slopes, bushes, etc.) and achieve stable and reliable movement. At the same time, a reasonable reward function is designed. Through end-to-end training, it can be deployed in an actual robot after 2000 iterations in one hour of training in a simulation environment, thereby reducing the training cost while improving the generalization ability of the controller.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention provides a quadruped robot motion control method based on adaptive reinforcement learning, comprising the following steps:

[0008] Based on the multi-layer perceptron, an adaptive module and a policy network are constructed respectively. Different simulation terrains and multiple robot instances are generated based on the dynamics simulator, and the robot instances are randomly placed in different simulation terrains.

[0009] The robot's historical state is obtained and input into the adaptive module for imitation learning training. The adaptive module outputs the robot's current linear velocity estimate, the current leg height estimate of each leg of the robot, and the potential feature estimate of the surrounding terrain.

[0010] Obtain the output of the adaptive module and the current robot state, input the policy network for reinforcement learning training, output the robot action, which is the target angle of multiple motor joints, and convert the target angle into the target torque;

[0011] The robot executes the target torque, calculates the current reward based on the reward function, and updates the robot state;

[0012] The termination conditions of iterative training are set, and the trained adaptive module and strategy network are combined into a motion controller of the quadruped robot to control the motion of the quadruped robot.

[0013] As a preferred technical solution, the adaptive module and the strategy network are composed of a multi-layer perception machine of a three-layer network.

[0014] As a preferred technical solution, the acquisition of the robot's historical state is specifically expressed as follows:

[0015]

[0016] Among them, t-5:t represents the historical state of the robot in the first five time steps from time t-5 to the current time t, ω t-5:t , g t-5:t 、cmd t-5:t ,θ t-5:t , and a t-6:t-1 They represent the robot's body angular velocity, gravity projection vector, velocity tracking command, joint angle, joint angular velocity and action at the previous moment in the first five time steps from time t-5 to the current time t.

[0017] As a preferred technical solution, the adaptive module obtains the historical state of the robot through a proprioceptive sensor, and the proprioceptive sensor includes a joint encoder and an inertial sensor.

[0018] As a preferred technical solution, the target angle is converted into a target torque, specifically by PD control, the target angle is converted into a target torque, which is expressed as:

[0019]

[0020] Among them, τ t represents the target torque, kp and kd represent the proportional and differential coefficients, θ t and Represents the current joint position and angular velocity.

[0021] As a preferred technical solution, the reward function includes the target speed tracking reward r t v,ω , balance reward r t ori , body height and leg lift height maintenance reward r t h , Anti-slip reward t slipand joint smoothness reward r t smooth , calculate the current reward according to the reward function, which is specifically expressed as:

[0022] r t =2r t v,ω +2r t ori +5r t h +0.05r t slip +0.01r t smooth

[0023] Among them, r t Indicates the current reward.

[0024] As a preferred technical solution, the target speed tracking reward r t v,ω It is expressed as:

[0025]

[0026] in, represents the linear velocity of the robot along the xy axis, represents the angular velocity on the z-axis, Indicates the line speed command, Indicates the angular velocity command;

[0027] The balance reward r t ori It is expressed as:

[0028]

[0029] in, represents the projected gravity of the robot base along the xy axis, Indicates the linear velocity on the z-axis;

[0030] The body height and leg lift height maintain the reward r t h It is expressed as:

[0031]

[0032] in, and Indicates the current base height and base target height of the robot. and Indicates the robot's current plantar height and plantar target height;

[0033] The anti-slip rewardt slip It is expressed as:

[0034]

[0035] in, Represents the linear velocity of the sole along the xy axis;

[0036] The joint smoothness reward r t smooth It is expressed as:

[0037]

[0038] Among them, a t 、a t-1 and a t-2 represents the robot's actions at the current moment, the previous moment, and the previous two moments, τ t Indicates the torque of the robot motor at the current moment.

[0039] As a preferred technical solution, in the training phase, the policy network parameters are first frozen, and the adaptive module is trained by supervised learning using a regression model to update the adaptive module parameters. The loss function is:

[0040]

[0041] in, It represents the estimated linear velocity of the robot along the xy axis at the current moment. It represents the estimated height of each leg of the robot at the current moment. represents the potential feature estimation of the surrounding terrain, Indicates the actual linear velocity of the robot along the xy axis at the current moment. Indicates the actual height of each leg of the robot at the current moment, l t Represents the true quantity of potential features of the surrounding terrain;

[0042] Refreeze the adaptive module network parameters, unfreeze the policy network parameters, use the deep reinforcement learning algorithm to train the policy network, and update the policy network parameters.

[0043] The present invention also provides a quadruped robot motion control system based on adaptive reinforcement learning, comprising: a controller network construction module, a simulation module, an adaptive module training, a strategy network training module, a target torque conversion module, an action execution module, a reward calculation module, a robot state update module, a condition setting module, and a motion control module;

[0044] The controller network building module is used to build an adaptive module and a strategy network based on a multi-layer perceptron.

[0045] The simulation module generates different simulation terrains and multiple robot instances based on a dynamics simulator, and randomly places the robot instances in different simulation terrains;

[0046] The adaptive module training is used to obtain the historical state of the robot, input the adaptive module for imitation learning training, and the adaptive module outputs the current moment linear velocity estimation of the robot, the current moment leg lift height estimation of each leg of the robot and the potential feature estimation of the surrounding terrain;

[0047] The strategy network training module is used to obtain the output of the adaptive module and the current robot state, input the strategy network for reinforcement learning training, and output the robot action as the target angle of multiple motor joints;

[0048] The target torque conversion module is used to convert the target angle into a target torque;

[0049] The action execution module is used to execute the target torque;

[0050] The reward calculation module is used to calculate the current reward according to the reward function;

[0051] The robot status update module is used to update the robot status;

[0052] The condition setting module is used to set the termination condition of iterative training;

[0053] The motion control module is used to combine the trained adaptive module and the strategy network into a motion controller of the quadruped robot to perform motion control on the quadruped robot.

[0054] The present invention also provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the above-mentioned quadruped robot motion control method based on adaptive reinforcement learning.

[0055] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0056] (1) Most existing technologies rely on external perception sensors such as vision and lidar for movement in complex terrains. However, these external sensors are easily blocked or affected by light, while proprioceptive perception sensors IMU and joint encoders are more powerful and reliable. The present invention only relies on robot proprioceptive sensing to design a blind trainer, without relying on external sensors such as vision and radar, and can achieve stable and agile movement in various complex terrains.

[0057] (2) The prior art uses a model predictive control algorithm to design a blind sensor. However, due to the uncertainty of unforeseen events, a blind motion controller designed using a model predictive control algorithm can often only walk on flat ground and cannot adapt to complex terrain. The present invention trains a controller based on an adaptive module combined with reinforcement learning. The adaptive module allows the robot to perceive its own external state and external terrain only by its own sensors. When the robot can adaptively adjust its own movements in different terrains, such as actively tilting itself and raising the height of its feet on the stairs, the blind controller trained in this way can enable the robot to adapt to various complex terrains. The robot can achieve agile and stable movements on various terrains such as stairs, slopes, grass, and sand.

[0058] (3) The present invention designs a corresponding reward function for a quadruped robot, which can reduce the time and cost of reinforcement learning training and converge to obtain natural and smooth movements.

[0059] (4) The present invention is based on an adaptive module and a strategy network to form a quadruped robot motion controller, which is obtained through end-to-end training and does not require multi-stage training. It allows the robot to fully explore in a simulation environment, improve the generalization ability of the controller, and stably move in various complex terrains. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flow chart of the motion control method of a quadruped robot based on adaptive reinforcement learning of the present invention;

[0061] Figure 2 Schematic diagram of a simulation of multiple non-interfering robot instances moving through various terrains generated by the simulator. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0063] Example 1

[0064] like Figure 1As shown, this embodiment provides a quadruped robot motion control method based on adaptive reinforcement learning, which uses a neural network to construct an adaptive module and a strategy network, which together constitute a quadruped robot motion controller, wherein the adaptive module uses the data collected by the proprioceptive sensor to dynamically estimate the robot's own state (including its own linear velocity and foot lift height) and surrounding terrain features. The proprioceptive sensor of this embodiment includes a joint encoder and an inertial sensor IMU. The input of the strategy network is the proprioceptive sensor data, the adaptive module's estimation of its own state and terrain, and the speed command, and outputs the target angle a of the twelve joint motors. t , and then obtain the target torque τ of the twelve joint motors through PD control t The present invention uses the simulation platform IsaacGym for training, wherein the adaptive module performs supervised learning through a regression model, and the strategy network performs reinforcement learning, allowing the robot to continuously interact with the environment. The two are trained simultaneously, and finally form a quadruped robot motion controller together. The method specifically includes the following steps:

[0065] S1: Build an adaptive module. The adaptive module consists of a three-layer MLP (Multilayer Perceptron) network with a network structure of [512, 256, 128]. The input of the adaptive module network is the historical state of the robot in the previous five time steps. where ω t-5:t , g t-5:t 、cmd t-5:t ,θ t-5:t , and a t-6:t-1 They represent the robot's body angular velocity, gravity projection vector, velocity tracking command, joint angles, joint angular velocity and action at the previous moment from t-5 to the current moment t, respectively. They come from the measurements of the inertial sensor IMU and joint encoder, user input and policy network output.

[0066] The output of the adaptive module obtained through imitation learning training is the estimated linear velocity of the robot along the xy axis at the current moment Estimated height of each leg of the robot at the current moment Estimation of potential features of the surrounding terrain The corresponding real quantity is and l t .

[0067] Among them, the terrain features t Representation: A terrain elevation map consisting of 187 points around the robot, which is then reduced in dimension from 187 dimensions to 16 dimensions through PCA (Principal Component Analysis).

[0068] S2: Construct a policy network. The policy network consists of three layers of MLP. The network structure is the same as the adaptive module, which is [512, 256, 128]. The input of the policy network is the current robot state o t and the output of the adaptive module The policy network is trained through reinforcement learning, and the adaptive module is trained through imitation learning, and finally outputs the quadruped robot action a t , which specifically means the target angles of the 12 motor joints, and then the target angles are converted into target torques τ through PD control t , whose expression is: where kp and kd represent the proportional and differential coefficients, θ t and Represents the current joint position and angular velocity.

[0069] S3: Use the dynamics simulator Isaac Gym to perform parallel robot simulation. Different simulation terrains are generated in the dynamics simulator IsaacGym, including smooth slopes of various slopes, uneven rough slopes and steps of different heights, which are used to simulate various complex terrains in real scenes.

[0070] like Figure 2 As shown in the figure, 4096 non-interfering robot instances are generated in the dynamics simulator Isaac Gym, and these quadruped robots are randomly placed in different simulated terrains. The robots are trained on different terrains to improve the generalization ability of the controller and allow the robots to adapt to various complex terrains.

[0071] The dynamics simulator Isaac Gym in this embodiment may also be replaced by other dynamics simulators, such as simulator Mujoco and simulator Pybullet.

[0072] When the robot falls or reaches the maximum simulation step length of 1000 steps, the simulation is terminated and the robot is randomly placed on a new terrain for the next round of simulation. 4096 robot instances are simulated simultaneously for a total of 2000 rounds.

[0073] S4: The adaptive module is based on the robot's historical state o of the first five time steps from t-5 to the current time t t-5:t , estimate the robot's own linear velocity Height of each leg lift Latent variables related to the terrain around the robot

[0074] S5: The policy network receives the output of the adaptation module and the current robot state o t , output robot action a t, represents the target angles of the 12 motor joints, and the target angles are converted into target torques τ through PD control t .

[0075] S6: The robot executes the target torque τ in the simulation t , the target torque acts on the actuator motor, the robot performs the action, and calculates the current reward r according to the reward function setting t , and update the robot status to get o t+1 .

[0076] The reward function designed by the present invention includes the target speed tracking reward r t v,ω , balance reward r t ori , body height and leg lift height maintenance reward r t h , Anti-slip reward t slip and joint smoothness reward r t smooth :

[0077] r t =2r t v,ω +2r t ori +5r t h +0.05r t slip +0.01r t smooth

[0078] Among them, the target speed tracking reward r t v,ω Encourage the robot's linear velocity along the xy axis Angular velocity on the z-axis Separate and line speed commands Angular velocity command Stay consistent:

[0079]

[0080] Balance Rewards t ori Penalize the robot's body tilt and shaking when moving:

[0081]

[0082] in, represents the projected gravity of the robot base along the xy axis, Indicates the linear velocity on the z-axis.

[0083] Body height and leg lift height maintenance reward t h Encourage the robot to move with a center of gravity height and leg lift height that are consistent with the preset values Be consistent and penalize when you deviate from a set value:

[0084]

[0085] in, and Indicates the current base height and base target height of the robot. and Indicates the robot's current foot height and foot target height.

[0086] Anti-slip reward t slip The linear velocity of the sole of the foot along the xy axis when the sole of the foot contacts the ground during the movement of the penalty robot:

[0087]

[0088] in, Represents the linear velocity of the sole along the xy axis;

[0089] Joint smoothness bonus t smooth Penalize the strategy network for actions that differ too much from before and after, and penalize the joint motor for excessive torque:

[0090] r t smooth =-||a t -a t-1 || 2 -||a t -2a t-1 +a t-2 || 2 -0.002||τ t || 2

[0091] Among them, a t 、a t-1 and a t-2 represents the robot's actions at the current moment, the previous moment, and the previous two moments, τ t Indicates the torque of the robot motor at the current moment.

[0092] S7: Collecting data from simulation Used to train the adaptive module and policy network. In the training phase, first freeze the policy network parameters, use the regression model to perform supervised learning training on the adaptive module, and update the adaptive module parameters. The loss function is:

[0093]

[0094] By minimizing the loss function to make the estimated value close to the true value, the adaptive module can accurately estimate the robot's own linear speed, the height of each leg and the external terrain. Then freeze the adaptive module network parameters, unfreeze the policy network parameters, use the deep reinforcement learning algorithm to train the policy network, and update the policy network parameters so that the actions generated by the policy network can obtain the maximum reward.

[0095] S8: Determine whether the robot falls or reaches the maximum simulation step length of 1000 steps. In the simulator, one simulation step corresponds to 0.02s and 1000 steps corresponds to 20s. When the robot reaches 1000 steps, the simulation is restarted and it is randomly placed in a new terrain. If the robot falls during the simulation, the simulation is terminated early and a new round of simulation is performed again.

[0096] S9: Determine whether the simulation has reached 2000 rounds. If so, end the training. The policy network and the adaptive module together form a quadruped robot motion controller, which is deployed on the physical robot.

[0097] Example 2

[0098] This embodiment provides a quadruped robot motion control system based on adaptive reinforcement learning, which is used to implement the quadruped robot motion control method based on adaptive reinforcement learning in the above-mentioned embodiment 1, and the system includes: a controller network construction module, a simulation module, an adaptive module training, a strategy network training module, a target torque conversion module, an action execution module, a reward calculation module, a robot state update module, a condition setting module, and a motion control module;

[0099] In this embodiment, the controller network construction module is used to construct the adaptive module and the policy network respectively based on the multi-layer perceptron;

[0100] In this embodiment, the simulation module generates different simulation terrains and multiple robot instances based on the dynamics simulator, and randomly places the robot instances in different simulation terrains;

[0101] In this embodiment, the adaptive module training is used to obtain the historical state of the robot, input the adaptive module for imitation learning training, and the adaptive module outputs the current moment linear velocity estimation of the robot, the current moment leg lift height estimation of each leg of the robot and the potential feature estimation of the surrounding terrain;

[0102] In this embodiment, the strategy network training module is used to obtain the output of the adaptive module and the current robot state, input the strategy network for reinforcement learning training, and output the robot action, which is the target angle of multiple motor joints;

[0103] In this embodiment, the target torque conversion module is used to convert the target angle into a target torque;

[0104] In this embodiment, the action execution module is used to execute the target torque;

[0105] In this embodiment, the reward calculation module is used to calculate the current reward according to the reward function;

[0106] In this embodiment, the robot state update module is used to update the robot state;

[0107] In this embodiment, the condition setting module is used to set the termination condition of iterative training;

[0108] In this embodiment, the motion control module is used to combine the trained adaptive module and the strategy network into a motion controller of the quadruped robot to perform motion control on the quadruped robot.

[0109] Example 3

[0110] This embodiment provides a storage medium, which may be a ROM, RAM, disk, CD or other storage medium, and the storage medium stores one or more programs. When the program is executed by the processor, the quadruped robot motion control method based on adaptive reinforcement learning of embodiment 1 is implemented.

[0111] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A quadruped robot motion control method based on adaptive reinforcement learning, characterized in that: The steps include: Based on the multi-layer perceptron, an adaptive module and a policy network are constructed respectively. Different simulation terrains and multiple robot instances are generated based on the dynamics simulator, and the robot instances are randomly placed in different simulation terrains. The robot's historical state is obtained and input into the adaptive module for imitation learning training. The adaptive module outputs the robot's current linear velocity estimate, the current leg height estimate of each leg of the robot, and the potential feature estimate of the surrounding terrain. Obtain the output of the adaptive module and the current robot state, input the policy network for reinforcement learning training, output the robot action, which is the target angle of multiple motor joints, and convert the target angle into the target torque; The robot executes the target torque, calculates the current reward based on the reward function, and updates the robot state; The termination conditions of iterative training are set, and the trained adaptive module and strategy network are combined into a motion controller of the quadruped robot to control the motion of the quadruped robot.

2. The quadruped robot motion control method based on adaptive reinforcement learning according to claim 1 is characterized in that: The adaptive module and the strategy network are composed of a multi-layer perception mechanism of a three-layer network.

3. The quadruped robot motion control method based on adaptive reinforcement learning according to claim 1, characterized in that: The acquisition of the robot's historical status is specifically expressed as follows: Among them, t-5:t represents the historical state of the robot in the first five time steps from time t-5 to the current time t, ω t-5:t , g t-5:t 、cmd t-5:t ,θ t-5:t , and a t-6:t-1 They represent the robot's body angular velocity, gravity projection vector, velocity tracking command, joint angle, joint angular velocity and action at the previous moment in the first five time steps from time t-5 to the current time t.

4. The quadruped robot motion control method based on adaptive reinforcement learning according to claim 1, characterized in that: The adaptive module obtains the historical state of the robot through a proprioceptive sensor, wherein the proprioceptive sensor includes a joint encoder and an inertial sensor.

5. The quadruped robot motion control method based on adaptive reinforcement learning according to claim 1, characterized in that: The target angle is converted into the target torque, specifically, the target angle is converted into the target torque through PD control, which is expressed as: Among them, τ t represents the target torque, kp and kd represent the proportional and differential coefficients, θ t and Represents the current joint position and angular velocity.

6. The quadruped robot motion control method based on adaptive reinforcement learning according to claim 1, characterized in that: The reward function includes the target speed tracking reward r t v,ω , balance reward r t ori , body height and leg lift height maintenance reward r t h , Anti-slip reward t slip and joint smoothness reward r t smooth , calculate the current reward according to the reward function, which is specifically expressed as: r t ＝2r t v,ω +2r t ori +5r t h +0.05r t slip +0.01r t smooth Among them, r t Indicates the current reward.

7. The quadruped robot motion control method based on adaptive reinforcement learning according to claim 6, characterized in that: The target speed tracking reward r t v,ω It is expressed as: in, represents the linear velocity of the robot along the xy axis, represents the angular velocity on the z-axis, Indicates the line speed command, Indicates the angular velocity command; The balance reward r t ori It is expressed as: in, represents the projected gravity of the robot base along the xy axis, Indicates the linear velocity on the z-axis; The body height and leg lift height maintain the reward r t h It is expressed as: in, and Indicates the current base height and base target height of the robot. and Indicates the robot's current plantar height and plantar target height; The anti-slip reward t slip It is expressed as: in, represents the linear velocity of the sole along the xy axis; The joint smoothness reward r t smooth It is expressed as: r t smooth =-||a t -a t-1 || 2 -||a t -2a t-1 +a t-2 || 2 -0.002||τ t || 2 Among them, a t 、a t-1 and a t-2 represents the robot's actions at the current moment, the previous moment, and the previous two moments, τ t Indicates the torque of the robot motor at the current moment.

8. The quadruped robot motion control method based on adaptive reinforcement learning according to claim 1, characterized in that: In the training phase, the policy network parameters are first frozen, and the adaptive module is trained by supervised learning using the regression model. The adaptive module parameters are updated, and the loss function is: in, It represents the estimated linear velocity of the robot along the xy axis at the current moment. It represents the estimated height of each leg of the robot at the current moment. represents the potential feature estimation of the surrounding terrain, Indicates the actual linear velocity of the robot along the xy axis at the current moment. Indicates the actual height of each leg of the robot at the current moment, l t Represents the true quantity of potential features of the surrounding terrain; Refreeze the adaptive module network parameters, unfreeze the policy network parameters, use the deep reinforcement learning algorithm to train the policy network, and update the policy network parameters.

9. A quadruped robot motion control system based on adaptive reinforcement learning, characterized in that: include: Controller network construction module, simulation module, adaptive module training, strategy network training module, target torque conversion module, action execution module, reward calculation module, robot state update module, condition setting module, motion control module; The controller network building module is used to build an adaptive module and a strategy network based on a multi-layer perceptron. The simulation module generates different simulation terrains and multiple robot instances based on a dynamics simulator, and randomly places the robot instances in different simulation terrains; The adaptive module training is used to obtain the historical state of the robot, input the adaptive module for imitation learning training, and the adaptive module outputs the current moment linear velocity estimation of the robot, the current moment leg lift height estimation of each leg of the robot and the potential feature estimation of the surrounding terrain; The strategy network training module is used to obtain the output of the adaptive module and the current robot state, input the strategy network for reinforcement learning training, and output the robot action as the target angle of multiple motor joints; The target torque conversion module is used to convert the target angle into a target torque; The action execution module is used to execute the target torque; The reward calculation module is used to calculate the current reward according to the reward function; The robot status update module is used to update the robot status; The condition setting module is used to set the termination condition of iterative training; The motion control module is used to combine the trained adaptive module and the strategy network into a motion controller of the quadruped robot to perform motion control on the quadruped robot.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by the processor, the quadruped robot motion control method based on adaptive reinforcement learning as described in any one of claims 1 to 8 is implemented.

Images