Robot reinforcement learning control method and system based on gating circulation unit

Abstract

The invention discloses a robot reinforcement learning control method and system based on a gating circulation unit, and relates to the technical field of artificial intelligence. Based on a reinforcement learning state machine, a trained student network is deployed to a real quadruped robot in a zero mapping manner, and movement control of the quadruped robot is realized; the training process of the student network is as follows: building a simulation training terrain environment of the robot, and setting different levels of terrain difficulty for each terrain; based on the distillation learning thought, establishing a teacher network and a student network based on a gating circulation unit, constructing a total loss function, and based on the constructed total loss function, fitting a motor joint angle output by the teacher network through an omnibearing simulation training terrain environment, a large-scale parallel training method and the student network; according to the robot reinforcement learning control method and system, robust motion of the robot in an unstructured terrain is realized, and the robot reinforcement learning control method and system have good anti-interference capability and self-recovery capability for a dynamic environment in which traditional control cannot be competent.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a robot reinforcement learning control method and system based on a gated recurrent unit. Background Art

[0002] Quadruped robots are well suited for traversing complex unstructured terrains in urban environments due to their superior flexibility and stability. However, the flexibility and stability of existing reinforcement learning-based quadruped robots in these terrains are still not ideal. Traditional control methods usually require multiple links such as state estimation, trajectory generation, gait optimization, and actuator control, and when faced with different environments, these controllers must perform precise environment-specific adaptive adjustments. This complex design usually requires tedious manual modeling and detailed parameter adjustments. In addition, robots are prone to losing control in unmodeled environments.

[0003] In recent years, the application of reinforcement learning to quadruped robots has significantly improved their maneuverability and robustness. Many advanced methods use multiple sensors, such as cameras and lidar systems. Although these external sensors can enhance the robot's perception capabilities, they also reduce the robot's overall robustness. For example, cameras perform poorly in low-light conditions, such as at night or in fog, and lidar may not work properly in soft terrain, such as snow or thick grass.

[0004] Therefore, it is necessary to propose a robust reinforcement learning control algorithm for a quadruped robot equipped only with proprioception sensors such as inertial measurement units and joint encoders to improve the safety of the quadruped robot in unstructured environments.

[0005] Existing proprioception-based reinforcement learning methods mainly rely on a two-stage teacher-student learning framework, where the student imitates the teacher's performance through knowledge distillation and supervised learning. However, since the two components are trained separately, the student often fails to learn from the teacher's failure states in the early stages of training. Therefore, how to learn the motor joint angles end-to-end from sensor data is also a key issue. Summary of the invention

[0006] Based on the technical problems existing in the background technology, the present invention proposes a robot reinforcement learning control method and system based on gated recurrent units, which can realize the robust motion of the robot in unstructured terrain and has good anti-interference and self-recovery capabilities in dynamic environments that traditional control cannot cope with.

[0007] The robot reinforcement learning control method based on gated recurrent unit proposed in the present invention deploys the trained student network zero mapping to a real quadruped robot based on the reinforcement learning state machine to realize the motion control of the quadruped robot;

[0008] The training process of the student network is as follows:

[0009] Step 1: Build a simulated training terrain environment for the robot and set different levels of terrain difficulty for each terrain;

[0010] Step 2: Based on the idea of ​​distillation learning, a teacher network and a student network based on gated recurrent units are built, and a total loss function is constructed. The teacher network maps the proprioceptive information and privileged information to the motor joint angles by learning a multi-layer perceptron strategy network π, thereby outputting a potential feature vector The student network learns the latent feature vector output by the teacher network Realize the prediction of motor joint angle;

[0011] Step 3: Based on the constructed total loss function, through a comprehensive simulation training terrain environment and large-scale parallel training methods, the student network fits the motor joint angles output by the teacher network.

[0012] Furthermore, in step one, the complete map of the simulated training terrain environment is composed of 10×10 sub-terrains, each sub-terrain represents one type of training terrain, and the same type of terrains increases in difficulty from left to right to form a row of the complete map, and each column of the complete map represents a different type of terrain of the same difficulty.

[0013] Further, in step 2, the teacher network includes an Actor network and a Critic network, and the Actor network includes a multi-layer perceptron encoder μ 1 , Multilayer Perceptron Encoder μ 2 , a multilayer perceptron policy network π, and privileged information is passed through a multilayer perceptron encoder μ 1 Get the potential feature vector z t , the latent feature vector z t The proprioceptive information is combined with the multi-layer perceptron policy network π to output the motor joint angle a t , the surrounding sub-terrain discrete heights are passed through a multi-layer perceptron encoder μ 2 The vectors are respectively related to the potential feature vector z t , the ontological perception information is input into the Critic network to output the evaluation value;

[0014] The student network consists of a multilayer perceptron encoder φ based on a gated recurrent unit 1 and a multilayer perceptron policy network The information before the current moment in the historical proprioceptive information is passed through the multi-layer perceptron encoder φ 1 Output latent feature vector The information of the current moment in the historical ontological perception information is combined with the latent feature vector Through the multi-layer perceptron policy network Output motor joint angle Based on the latent feature vector z t , latent feature vector Motor joint angle a t , motor joint angle Construct the student-side loss function.

[0015] Furthermore, in step 2, the total loss function includes a teacher-side loss function and a student-side loss function. The teacher-side loss function is constructed using the PPO algorithm, and the student-side loss function is specifically as follows:

[0016]

[0017] Among them, L is the loss function.

[0018] Furthermore, step three is specifically as follows:

[0019] S31, initializing all robots on various sub-terrains of the first three terrain difficulties in the simulation training terrain environment, and the robots are reset and move with random initial commands;

[0020] S32, the robot obtains proprioceptive information, privileged information, and the discrete height of the surrounding sub-terrain as the input of the teacher network, and outputs the motor joint angle a through the Actor network t , realize the interaction between the robot and the simulated training terrain environment;

[0021] S33, evaluate the output evaluation value through the Critic network to evaluate the motor joint angle a t The robot's actions are good or bad, and guide the robot's next decision;

[0022] S34, calculate the teacher-side loss function through the PPO algorithm and update the Actor network and Critic network;

[0023] S35, the student network obtains historical ontology perception information, evaluates the loss difference between the student network output and the Actor network output according to the constructed student-side loss function, and then updates the student network;

[0024] S26. Repeat steps S32 to S25 until the reward function becomes stable.

[0025] Further, in step S32, the robot learns movement skills in the current sub-terrain, and determines in sequence whether it has passed through the current sub-terrain, whether it has passed through 25% of the next sub-terrain, and whether it has reached 85% of the target linear speed;

[0026] If both are true, increase the difficulty of the sub-terrain;

[0027] If yes, determine whether the current training round is over. If not, return the robot to the current sub-terrain to continue learning sports skills. If it is over, determine whether it has passed through 50% of the terrain. If it has passed, enter step S31, the robot resets and moves with a random initial command. If it has not passed, reduce the difficulty of the sub-terrain.

[0028] Furthermore, the proprioception data includes the robot's gravity vector, base angular velocity, motor joint angle, motor joint speed, command speed and the previous output joint angle of the teacher network; the privileged information includes base linear velocity, ground friction coefficient, ground restitution coefficient, foot end contact state, robot mass and center of mass position, and the surrounding sub-terrain discrete height is the vertical distance from the surrounding sub-terrain to the robot's base coordinate system.

[0029] Furthermore, the reward function includes a trunk xy axis velocity tracking reward, a trunk heading angular velocity tracking reward, a trunk z axis velocity penalty, a trunk xy axis angular velocity penalty, a gravity vector penalty, a joint power penalty, a joint acceleration penalty, a body height penalty, a collision penalty, a gait constraint penalty, a motion change rate penalty, and a second-order motion change rate penalty.

[0030] Furthermore, the trained student network zero mapping is deployed on the real quadruped robot as follows:

[0031] Convert the trained student network into a JIT model through PyTorch, and deploy the model on the embedded platform based on LibTorch.

[0032] Based on the reinforcement learning state machine template, the robot can switch between squatting, standing, and reinforcement learning motion control states.

[0033] Parse the control instructions given by the handle, obtain the historical proprioception information required for the student model input, implement student model reasoning, and obtain 12 motor joint angles. The robot tracks the obtained 12 motor joint angles through PD control to realize robot motion control under real terrain.

[0034] A robot reinforcement learning control system based on a gated recurrent unit, including deploying the trained student network zero mapping to a real quadruped robot based on a reinforcement learning state machine to achieve quadruped robot motion control;

[0035] The training process of the student network is as follows:

[0036] Step 1: Build a simulated training terrain environment for the robot and set different levels of terrain difficulty for each terrain;

[0037] Step 2: Based on the idea of ​​distillation learning, a teacher network and a student network based on gated recurrent units are built, and a total loss function is constructed. The teacher network maps the proprioceptive information and privileged information to the motor joint angles by learning a multi-layer perceptron strategy network π, thereby outputting a potential feature vector The student network learns the latent feature vector output by the teacher network Realize the prediction of motor joint angle;

[0038] Step 3: Based on the constructed total loss function, through a comprehensive simulation training terrain environment and large-scale parallel training methods, the student network fits the motor joint angles output by the teacher network.

[0039] The advantages of the robot reinforcement learning control method and system based on gated recurrent units provided by the present invention are: the robust motion of the quadruped robot in unstructured terrain is achieved through a teacher-end to student-end reinforcement learning method, and no additional external sensors are required. It has good anti-interference and self-recovery capabilities for dynamic environments that traditional control cannot cope with. Compared with other reinforcement learning methods, this embodiment has powerful motion performance and can climb continuous stairs and higher discrete obstacles without visual input; it can recover the potential information of high-dimensional space from historical proprioceptive information, and improve the stability and accuracy of student network output; the all-round terrain course ensures that the quadruped robot has a good tracking effect on control instructions in any direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the process of the present invention;

[0041] Figure 2 It is the structural framework diagram of the asymmetric teacher-student network;

[0042] Figure 3 Training flow chart for the full range of terrain courses. DETAILED DESCRIPTION

[0043] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0044] like Figures 1 to 3 As shown, the robot reinforcement learning control method based on gated recurrent unit proposed in the present invention deploys the trained student network zero mapping to a real quadruped robot based on the reinforcement learning state machine to realize the motion control of the quadruped robot;

[0045] The training process of the student network is as follows:

[0046] Step 1: Build a simulated training terrain environment for the robot and set different levels of terrain difficulty for each terrain;

[0047] The simulated training terrain environment for the quadruped robot built in Isaac Gym generates flat ground, gravel roads, slopes, stairs, stairs, and discrete obstacles through triangular meshes. Different levels of terrain difficulty are set for each terrain, allowing the robot to learn robust motion performance.

[0048] The complete map consists of 10×10 sub-terrains, each of which represents one type of training terrain. The same type of terrain increases in difficulty from left to right to form a row of the complete map. Each column of the complete map represents a different type of terrain of the same difficulty, which is randomly sampled from the terrain type at a certain ratio, for example, at a ratio of 0.1, 0.1, 0.35, 0.25, and 0.2 from the five terrain types of gravel road, slope, stairs, stairs, and discrete obstacles.

[0049] To simulate the roughness on the flat terrain, noise ranging from ±0.2cm to ±2cm was introduced. The inclination of the slope varied from 0 to 45 degrees, accompanied by ±0.5cm noise. For the stair terrain, the width of the steps was always 30cm, and the height increased from 2cm to 30cm. 20 rectangular obstacles were set on the discrete obstacle terrain, with the obstacle height increasing from 2cm to 30cm and the area from 1m 2 Increase to 2m 2 .

[0050] Step 2: Based on the idea of ​​distillation learning, a teacher network and a student network based on gated recurrent units are built, and a total loss function is constructed. The teacher network maps the proprioceptive information and privileged information to the motor joint angles by learning a multi-layer perceptron strategy network π, thereby outputting a potential feature vector The student network learns the latent feature vector output by the teacher network Realize the prediction of motor joint angle;

[0051] like Figure 2 , the process of building the teacher network and the student network is as follows S21 to S23:

[0052] S21. Multilayer perceptron encoder μ for privileged information processing 1 ,Combined with the proprioceptive information input dimension, the teacher-side Actor network is designed, and the Actor network outputs 12-dimensional motor joint angles;

[0053] Specifically, by using the proximal policy optimization algorithm, the teacher network will learn a multi-layer perceptron policy network π to map the proprioceptive information and privileged information to the target joint angles. The Actor network consists of three components: a multi-layer perceptron (MLP) encoder μ 1 and a multi-layer perceptron policy network π. Multiple encoders compress the high-dimensional privileged information into a low-dimensional potential representation, reducing the data dimension. That is, the privileged information is passed through the multi-layer perceptron encoder μ 1 Get the potential feature vector z t , the latent feature vector z t With proprioceptive information t Output the motor joint angle a through the multi-layer perceptron policy network π t , the input-output relationship is as follows:

[0054] z t =μ 1 (e t );

[0055] a t =π(o t ,z t ).

[0056] S22, based on the input in step S21, an additional terrain height perception encoder (i.e., a multilayer perceptron encoder μ 2 ), design the Critic network according to the Actor network dimension, output the evaluation value, and combine it with the architecture in step S21 to form an asymmetric teacher network architecture.

[0057] S23: Design a gated recurrent unit based on the dimension of historical proprioceptive information and build a gated recurrent unit decoder φ with the same output dimension as the encoder of the teacher network 1 , combined with the Actor network to form a student network architecture, outputting 12-dimensional motor joint angles;

[0058] Specifically, the student network can only access the proprioceptive information, and the student network imitates the actions of the teacher network through supervised learning. This method is based on assuming that the latent feature vector It can be approximately recovered from the proprioceptive observation of the time series. The characteristics of the time series data are effectively captured through the gated recurrent unit (GRU) network. There is a multi-layer perceptron network behind the gated recurrent unit network to form a multi-layer perceptron encoder φ 1 Through supervised learning, the student network simulates the multilayer perceptron policy network And reconstructed by φ 1 Generated latent feature vector The student-side loss function L is defined as:

[0059]

[0060] By training the student network and the teacher network in parallel, the student network can effectively learn from the early failure experience of the teacher network, thereby improving the robustness of the algorithm. 1 is randomly initialized. The student network is trained by using historical ontology perception information and latent feature vector , combined with the sample labels of the teacher network, train the GRU-MLP encoder (i.e., the multi-layer perceptron encoder φ 1 ), thereby realizing the prediction of the motor joint angle. The input and output relationship of the student network is as follows:

[0061]

[0062]

[0063] Among them, t-1:t-k is the proprioceptive information before the current moment in the historical proprioceptive information, o t It is the proprioceptive information at the current moment.

[0064] Step 3: Based on the constructed total loss function, through a comprehensive simulation training terrain environment and a large-scale parallel training method, the student network fits the motor joint angles output by the teacher network, and is robust to unstructured terrain;

[0065] At the beginning of training, all robots are uniformly placed on a simple sub-terrain. Due to different initial velocity directions, the robots may cross the current sub-terrain from different directions. In order to implement sub-terrain difficulty updates, when a robot passes through a sub-terrain, one robot will be transferred to the leftmost side of the subsequent sub-terrain. At the same time, the robot's quaternion, velocity vector, and gravity are adjusted through a rotation matrix so that the coordinate system of the robot after the move is aligned with the local coordinate system before the move, ensuring that the robot can move normally in the next terrain. Through the design of a full range of terrain courses, the robot can learn to pass through the current sub-terrain with commands in different directions. When the robot successfully crosses one quarter of the next sub-terrain, the current sub-terrain coordinate origin will be updated to the new position, ensuring that the robot will start from the new sub-terrain when initialized next time.

[0066] Specifically, refer to Figure 3As shown, the robot will enter a more challenging sub-terrain only after successfully adapting to the current sub-terrain. The robot will be updated to a more difficult sub-terrain only when it can reach at least 85% of the set average linear speed and successfully pass through the current sub-terrain. Conversely, if a robot cannot traverse half of the current sub-terrain at the specified speed at the end of the round, it will be downgraded to a simpler terrain. To prevent skill forgetting, once the robot passes through the most difficult sub-terrain, it will be randomly teleported to a sub-terrain of a certain difficulty in the same category.

[0067] Step 3 specifically includes S31 to S36:

[0068] S31, initializing all robots on various sub-terrains of the first three terrain difficulties in the simulation training terrain environment, and the robots are reset and move with random initial commands;

[0069] Initialize the quadruped robots in Isaac Gym so that they are evenly distributed in various terrains of the first three terrain difficulties.

[0070] S32, the robot obtains proprioceptive information, privileged information, and the discrete height of the surrounding sub-terrain as the input of the teacher network, and outputs the motor joint angle a through the Actor network t , realize the interaction between the robot and the simulated training terrain environment;

[0071] The proprioceptive data include the robot's gravity vector, base angular velocity, motor joint angle, motor joint speed, command speed and the previous output joint angle of the teacher network; the privileged information includes base linear velocity, ground friction coefficient, ground restitution coefficient, foot end contact state, robot mass and center of mass position, and the surrounding sub-terrain discrete height is composed of 187 discrete sampling points around the robot, that is, the vertical distance from the surrounding sub-terrain to the robot's base coordinate system.

[0072] The interaction between the robot and the simulation training terrain environment is as follows: the robot learns sports skills in the current sub-terrain, and determines in turn whether it has passed through the current sub-terrain, whether it has passed through 25% of the next sub-terrain, and whether it has reached 85% of the target linear speed. If so, the difficulty of the sub-terrain is increased; if not, it is determined whether the current training round has ended. If not, the robot returns to the current sub-terrain to continue learning sports skills. If it has ended, it is determined whether it has passed through 50% of the terrain. If so, it enters step S31, the robot is reset and moves with a random initial command. If not, the difficulty of the sub-terrain is reduced.

[0073] S33, evaluate the output evaluation value through the Critic network to evaluate the motor joint angle a t The robot's actions are good or bad, and guide the robot's next decision;

[0074] S34, calculate the teacher-side loss function through the PPO algorithm and update the Actor network and Critic network;

[0075] S35, the student network obtains historical ontology perception information, evaluates the loss difference between the student network output and the Actor network output according to the constructed student-side loss function, and then updates the student network;

[0076] S26. Repeat steps S32 to S25 until the reward function becomes stable; the reward function includes trunk xy axis velocity tracking reward, trunk heading angular velocity tracking reward, trunk z axis velocity penalty, trunk xy axis angular velocity penalty, gravity vector penalty, joint power penalty, joint acceleration penalty, body height penalty, collision penalty, gait constraint penalty, motion change rate penalty, and second-order motion change rate penalty.

[0077] In this embodiment, based on the reinforcement learning state machine, the trained student network zero mapping is deployed on the real quadruped robot to realize the motion control of the quadruped robot, which specifically includes S41 to S43:

[0078] S41. Convert the trained student network into a jit model through pytorch, and implement model deployment on the embedded platform based on libtorch;

[0079] S42. Based on the reinforcement learning state machine template, the robot can switch between squatting, standing, and reinforcement learning motion control states.

[0080] S43. Analyze the control instructions given by the handle, obtain the historical proprioception information required for the student model input, implement the student model reasoning, and obtain 12 motor joint angles. The robot tracks the obtained 12 motor joint angles through PD control to realize robot motion control under real terrain.

[0081] This embodiment first builds different types of simulated training terrain environments in Isaac Gym to gradually increase the terrain difficulty of robot training; then, based on the idea of ​​distillation learning, a teacher-student network framework based on gated recurrent units is built; then, the student network gradually learns to fit the potential feature vector z of the teacher network through a full range of terrain courses and large-scale parallel training methods. t , realize the motor joint angle prediction; finally, export the JIT model, use libtorch to infer the model on the embedded platform, and realize the robot reinforcement learning motion control. This embodiment is applicable to different robots, especially to quadruped robots.

[0082] This embodiment has the following beneficial effects:

[0083] 1. This embodiment can improve the robustness of the quadruped robot's motion: This embodiment achieves the robust motion of the quadruped robot in unstructured terrain through a teacher-end to student-end reinforcement learning method. It has good anti-interference and self-recovery capabilities for dynamic environments that traditional control cannot cope with. Compared with other reinforcement learning methods, this embodiment has powerful motion performance and can climb continuous stairs and higher discrete obstacles without visual input.

[0084] 2. This embodiment can improve the fitting accuracy of the teacher-student network: This embodiment selects input quantities with strong recoverability through the design of an asymmetric teacher-student network, realizes the recovery of potential information in high-dimensional space from historical ontological perception information, and improves the stability and accuracy of the student network output.

[0085] 3. This embodiment can optimize the efficiency of model training: This embodiment greatly accelerates the training speed through large-scale parallel training. The Sim2Sim verification operation based on Isaac Gym to Gazebo ensures the physical accuracy of the deployed model. The all-round terrain course ensures that the quadruped robot has a good tracking effect on control instructions in any direction.

[0086] 4. This embodiment can enhance the interpretability of latent space vectors: This embodiment encodes the proprioceptive information into latent space vector information through an asymmetric teacher-student network framework based on gated recurrent units, thereby realizing adaptive motion control for different terrains. The latent space vectors can be decoupled in high-dimensional space, and have good interpretability for different terrains.

[0087] 5. This embodiment has innovations and advantages in many aspects such as technical implementation, application effects and social benefits, and provides strong support for the development and practical application research of quadruped robot motion control technology.

[0088] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A robot reinforcement learning control method based on gated recurrent units, characterized in that: Based on the reinforcement learning state machine, the trained student network zero mapping is deployed on the real quadruped robot to realize the motion control of the quadruped robot; The training process of the student network is as follows: Step 1: Build a simulated training terrain environment for the robot and set different levels of terrain difficulty for each terrain; Step 2: Based on the idea of ​​distillation learning, a teacher network and a student network based on gated recurrent units are built, and a total loss function is constructed. The teacher network maps the proprioceptive information and privileged information to the motor joint angles by learning a multi-layer perceptron strategy network π, thereby outputting a potential feature vector The student network learns the latent feature vector output by the teacher network Realize the prediction of motor joint angle; Step 3: Based on the constructed total loss function, through a comprehensive simulation training terrain environment and large-scale parallel training methods, the student network fits the motor joint angles output by the teacher network.

2. The robot reinforcement learning control method based on gated recurrent unit according to claim 1 is characterized in that: In step 1, the complete map of the simulated training terrain environment is composed of 10×10 sub-terrains, each of which represents one type of training terrain. The same type of terrains increases in difficulty from left to right to form a row of the complete map, and each column of the complete map represents a different type of terrain of the same difficulty.

3. The robot reinforcement learning control method based on gated recurrent unit according to claim 1 is characterized in that: In step 2, the teacher network includes an Actor network and a Critic network. The Actor network includes a multilayer perceptron encoder μ1, a multilayer perceptron encoder μ2, and a multilayer perceptron policy network π. The privileged information is obtained through the multilayer perceptron encoder μ1 to obtain a potential feature vector z t , the latent feature vector z t The proprioceptive information is combined with the multi-layer perceptron policy network π to output the motor joint angle a t , the surrounding sub-terrain discrete heights are obtained through the multi-layer perceptron encoder μ2 and the vectors are respectively combined with the potential feature vector z t , the proprioceptive information is input into the Critic network to output the evaluation value; The student network includes a multilayer perceptron encoder φ1 based on a gated recurrent unit and a multilayer perceptron policy network The information before the current moment in the historical ontological perception information is output as a potential feature vector through the multi-layer perceptron encoder φ1 The information of the current moment in the historical ontology perception information is combined with the potential feature vector Through the multi-layer perceptron policy network Output motor joint angle Based on the latent feature vector z t , latent feature vector Motor joint angle a t , motor joint angle Construct the student-side loss function.

4. The robot reinforcement learning control method based on gated recurrent unit according to claim 3 is characterized in that: In step 2, the total loss function includes a teacher-side loss function and a student-side loss function. The teacher-side loss function is constructed using the PPO algorithm, and the student-side loss function is as follows: Among them, L is the loss function.

5. The robot reinforcement learning control method based on gated recurrent unit according to claim 4 is characterized in that: Step three is as follows: S31, initializing all robots on various sub-terrains of the first three terrain difficulties in the simulation training terrain environment, and the robots are reset and move with random initial commands; S32, the robot obtains proprioceptive information, privileged information, and the discrete height of the surrounding sub-terrain as the input of the teacher network, and outputs the motor joint angle a through the Actor network t , realize the interaction between the robot and the simulated training terrain environment; S33, evaluate the output evaluation value through the Critic network to evaluate the motor joint angle a t The robot's actions are good or bad, and guide the robot's next decision; S34, calculate the teacher-side loss function through the PPO algorithm and update the Actor network and Critic network; S35, the student network obtains historical ontology perception information, evaluates the loss difference between the student network output and the Actor network output according to the constructed student-side loss function, and then updates the student network; S26. Repeat steps S32 to S25 until the reward function becomes stable.

6. The robot reinforcement learning control method based on gated recurrent unit according to claim 5 is characterized in that: In step S32, the robot learns movement skills in the current sub-terrain, and determines in sequence whether it has passed through the current sub-terrain, whether it has passed through 25% of the next sub-terrain, and whether it has reached 85% of the target linear speed; If both are true, increase the difficulty of the sub-terrain; If yes, determine whether the current training round is over. If not, return the robot to the current sub-terrain to continue learning sports skills. If it is over, determine whether it has passed through 50% of the terrain. If it has passed, enter step S31, the robot resets and moves with a random initial command. If it has not passed, reduce the difficulty of the sub-terrain.

7. The robot reinforcement learning control method based on gated recurrent unit according to claim 5 is characterized in that: The proprioception data includes the robot's gravity vector, base angular velocity, motor joint angle, motor joint speed, command speed and the previous output joint angle of the teacher network; the privileged information includes base linear velocity, ground friction coefficient, ground restitution coefficient, foot end contact state, robot mass and center of mass position, and the surrounding sub-terrain discrete height is the vertical distance from the surrounding sub-terrain to the robot's base coordinate system.

8. The robot reinforcement learning control method based on gated recurrent unit according to claim 5 is characterized in that: The reward function includes trunk xy axis velocity tracking reward, trunk heading angular velocity tracking reward, trunk z axis velocity penalty, trunk xy axis angular velocity penalty, gravity vector penalty, joint power penalty, joint acceleration penalty, body height penalty, collision penalty, gait constraint penalty, motion change rate penalty, and second-order motion change rate penalty.

9. The robot reinforcement learning control method based on gated recurrent unit according to claim 1, characterized in that: Deploy the trained student network zero mapping to the real quadruped robot as follows: Convert the trained student network into a JIT model through PyTorch, and deploy the model on the embedded platform based on LibTorch. Based on the reinforcement learning state machine template, the robot can switch between squatting, standing, and reinforcement learning motion control states. Parse the control instructions given by the handle, obtain the historical proprioception information required for the student model input, implement student model reasoning, and obtain 12 motor joint angles. The robot tracks the obtained 12 motor joint angles through PD control to realize robot motion control under real terrain.

10. A robot reinforcement learning control system based on gated recurrent units, characterized in that: This includes deploying the trained student network zero mapping to a real quadruped robot based on a reinforcement learning state machine to achieve quadruped robot motion control; The training process of the student network is as follows: Step 1: Build a simulated training terrain environment for the robot and set different levels of terrain difficulty for each terrain; Step 2: Based on the idea of ​​distillation learning, a teacher network and a student network based on gated recurrent units are built, and a total loss function is constructed. The teacher network maps the proprioceptive information and privileged information to the motor joint angles by learning a multi-layer perceptron strategy network π, thereby outputting a potential feature vector The student network learns the latent feature vector output by the teacher network Realize the prediction of motor joint angle; Step 3: Based on the constructed total loss function, through a comprehensive simulation training terrain environment and large-scale parallel training methods, the student network fits the motor joint angles output by the teacher network.

Images