Robot reinforcement learning control method and system based on gated recurrent unit

Abstract

The application discloses a robot reinforcement learning control method and system based on a gated recurrent unit, relates to the technical field of artificial intelligence, and is based on a reinforcement learning state machine, deploys a trained student network zero mapping to a real quadruped robot, and realizes quadruped robot motion control; the training process of the student network is as follows: a simulation training terrain environment of the robot is built, different levels of terrain difficulty are set for each kind of terrain; based on the distillation learning idea, a teacher network and a student network based on the gated recurrent unit are built, and a total loss function is constructed; based on the constructed total loss function, through omnidirectional simulation training terrain environment and a large-scale parallel training method, the student network fits the motor joint angle output by the teacher network; the robot reinforcement learning control method and system realize the robust motion of the robot on unstructured terrain, and have good anti-interference ability and self-recovery ability in a dynamic environment which cannot be well handled by traditional control.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a robot reinforcement learning control method and system based on gated loop units. Background Technology

[0002] Quadruped robots, with their exceptional flexibility and stability, are well-suited for navigating complex, unstructured terrain in urban environments. However, the flexibility and stability of quadruped robots based on existing reinforcement learning methods in these terrains remain less than ideal. Traditional control methods typically involve multiple stages, including state estimation, trajectory generation, gait optimization, and actuator control. Furthermore, these controllers must undergo precise environment-specific adaptive adjustments to suit different environments, a complex design often requiring tedious manual modeling and detailed parameter tuning. Moreover, in unmodeled environments, robots are prone to losing control.

[0003] In recent years, the application of reinforcement learning to quadruped robots has significantly improved their mobility and robustness. Many advanced methods employ multiple sensors, such as cameras and lidar systems. While these external sensors can enhance the robot's perception capabilities, they also reduce its overall robustness. For example, cameras perform poorly in low-light conditions, such as at night or in fog; and lidar may not function properly in soft terrain, such as snow or thick grass.

[0004] Therefore, a robust reinforcement learning control algorithm is needed for quadruped robots equipped only with proprioception sensors such as inertial measurement units and joint encoders to improve the safety of quadruped robots in unstructured environments.

[0005] Existing ontology-based reinforcement learning methods primarily rely on a two-stage teacher-student learning framework, where students mimic the teacher's performance through knowledge distillation and supervised learning. However, because the training of the two components is separate, students often fail to learn from the teacher's failures in the early stages of training. Therefore, how to learn motor joint angles end-to-end from sensor data is also a critical issue. Summary of the Invention

[0006] Based on the technical problems existing in the background technology, this invention proposes a robot reinforcement learning control method and system based on gated loop units, which realizes robust robot movement in unstructured terrain and has good anti-interference ability and self-recovery ability in dynamic environments that traditional control cannot handle.

[0007] The robot reinforcement learning control method proposed in this invention is based on a gated loop unit. It uses a reinforcement learning state machine to deploy a trained student network onto a real quadruped robot with zero mapping, thereby realizing the motion control of the quadruped robot.

[0008] The training process for the student network is as follows:

[0009] Step 1: Build a simulated training terrain environment for the robot and set different levels of terrain difficulty for each type of terrain;

[0010] Step 2: Based on the distillation learning concept, construct a teacher network and a student network based on gated recurrent units, and build a total loss function. The teacher network learns a multilayer perceptron policy network π to map ontology perception information and privileged information to motor joint angles, thereby outputting a latent feature vector. The student network learns the latent feature vectors output by the teacher network. To predict the joint angle of the motor;

[0011] Step 3: Based on the constructed total loss function, the student network fits the motor joint angles output by the teacher network through comprehensive simulation training terrain environment and large-scale parallel training methods.

[0012] Furthermore, in step one, the complete map of the simulated training terrain environment consists of 10×10 sub-terrains, each sub-terrain representing one type of training terrain. Terrains of the same type increase in difficulty from left to right to form a row of the complete map, and each column of the complete map represents different types of terrain of the same difficulty.

[0013] Furthermore, in step two, the teacher network includes an Actor network and a Critic network. The Actor network includes a multilayer perceptron encoder μ1, a multilayer perceptron encoder μ2, and a multilayer perceptron policy network π. Privileged information is used to obtain the latent feature vector z through the multilayer perceptron encoder μ1. t latent feature vector z t The proprioceptive information is used to output the motor joint angle α through a multilayer perceptron policy network π. t The discrete heights of the surrounding sub-terrain are obtained as vectors by the multilayer perceptron encoder μ2 and compared with the latent feature vector z. t The ontological perception information is input into the Critic network to output an evaluation value;

[0014] The student network includes a multilayer perceptron encoder φ1 based on a gated recurrent unit and a multilayer perceptron policy network. Information prior to the current moment in the historical ontology perception information is used to output a latent feature vector through a multilayer perceptron encoder φ1. Combine the current moment's information with the latent feature vector from the historical ontology perception information. Through multi-layer perceptron policy network Output motor joint angle Based on the latent feature vector z tlatent feature vectors Motor joint angle a t Motor joint angle Construct the student-side loss function.

[0015] Furthermore, in step two, the total loss function includes a teacher-side loss function and a student-side loss function. The teacher-side loss function is constructed using the PPO algorithm, and the student-side loss function is as follows:

[0016]

[0017] Where L is the loss function.

[0018] Furthermore, step three specifically involves:

[0019] S31. Initialize all robots on various sub-terrains of the first three terrain difficulties in the simulation training terrain environment. Reset the robots and move with random initial commands.

[0020] S32. The robot acquires proprioceptive information, privileged information, and discrete heights of the surrounding sub-terrain, which are used as input to the teacher network. The motor joint angle α is then output through the Actor network. t This enables the robot to interact with the simulated training terrain environment;

[0021] S33. Evaluate the output evaluation value through the Critic network to assess the motor joint angle α. t The quality of the robot's actions is assessed, and this information guides the robot's next decision.

[0022] S34. Calculate the teacher-side loss function using the PPO algorithm, and update the Actor network and Critic network;

[0023] S35. The student network acquires historical ontology perception information, evaluates the loss difference between the student network output and the Actor network output based on the constructed student-side loss function, and then updates the student network.

[0024] S26. Repeat steps S32 to S25 until the reward function tends to stabilize.

[0025] Further, in step S32, the robot learns movement skills in the current sub-terrain and sequentially determines whether it has passed through the current sub-terrain, whether it has passed 25% of the next sub-terrain, and whether it has reached 85% of the target linear velocity.

[0026] If both are true, then increase the difficulty of the sub-terrain;

[0027] If the current training round has not ended, the robot returns to the current sub-terrain and continues to learn movement skills. If the current training round has ... sub-terrain has ended, the robot returns to the current sub-terrain and continues to learn movement skills. If the

[0028] Furthermore, the ontological perception data includes the robot's gravity vector, base angular velocity, motor joint angles, motor joint velocities, command velocities, and the output joint angle of the previous time on the teacher network; the privileged information includes base linear velocity, ground friction coefficient, ground recovery coefficient, foot contact state, robot mass, and the position of the center of mass; and the discrete height of the surrounding sub-terrain is the vertical distance from the surrounding sub-terrain to the robot's base coordinate system.

[0029] Furthermore, the reward function includes a trunk xy axis velocity tracking reward, a trunk yaw angular velocity tracking reward, a trunk z axis velocity penalty, a trunk xy axis angular velocity penalty, a gravity vector penalty, a joint power penalty, a joint acceleration penalty, a body height penalty, a collision penalty, a gait constraint penalty, a motion change rate penalty, and a second-order motion change rate penalty.

[0030] Furthermore, the trained student network is deployed to a real quadruped robot with zero mapping, specifically as follows:

[0031] The trained student network is converted into a JIT model using PyTorch, and the model is deployed on an embedded platform using libtorch.

[0032] Based on the reinforcement learning state machine template, the robot can switch between three states: crouching, standing, and reinforcement learning motion control.

[0033] The control commands given by the handle are analyzed to obtain the historical ontology perception information required for the student model input, enabling the student model to reason and obtain the 12 motor joint angles. The robot tracks the obtained 12 motor joint angles through PD control to achieve robot motion control in real terrain.

[0034] A robot reinforcement learning control system based on a gated loop unit includes a reinforcement learning state machine that deploys a trained student network onto a real quadruped robot with zero mapping to achieve quadruped robot motion control.

[0035] The training process for the student network is as follows:

[0036] Step 1: Build a simulated training terrain environment for the robot and set different levels of terrain difficulty for each type of terrain;

[0037] Step 2: Based on the distillation learning concept, construct a teacher network and a student network based on gated recurrent units, and build a total loss function. The teacher network learns a multilayer perceptron policy network π to map ontology perception information and privileged information to motor joint angles, thereby outputting a latent feature vector. The student network learns the latent feature vectors output by the teacher network. To predict the joint angle of the motor;

[0038] Step 3: Based on the constructed total loss function, the student network fits the motor joint angles output by the teacher network through comprehensive simulation training terrain environment and large-scale parallel training methods.

[0039] The advantages of the robot reinforcement learning control method and system based on gated loop units provided by this invention are as follows: Robust movement of the quadruped robot in unstructured terrain is achieved through a teacher-to-student reinforcement learning method, without the need for additional external sensors. It exhibits excellent anti-interference and self-recovery capabilities in dynamic environments where traditional control methods are inadequate. Compared to other reinforcement learning methods, this embodiment demonstrates superior motion performance, enabling the quadruped robot to climb continuous stairs and tall discrete obstacles without visual input. It also recovers latent information in high-dimensional space from historical ontological perception information, improving the stability and accuracy of student network output. The omnidirectional terrain curriculum ensures excellent tracking performance of control commands in any direction for the quadruped robot. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the process of the present invention;

[0041] Figure 2 A structural framework diagram of an asymmetric teacher-student network;

[0042] Figure 3 This is a flowchart for a comprehensive terrain course training program. Detailed Implementation

[0043] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0044] like Figures 1 to 3 As shown, the robot reinforcement learning control method based on gated loop unit proposed in this invention uses a reinforcement learning state machine to deploy the trained student network onto a real quadruped robot with zero mapping, thereby realizing the motion control of the quadruped robot.

[0045] The training process for the student network is as follows:

[0046] Step 1: Build a simulated training terrain environment for the robot and set different levels of terrain difficulty for each type of terrain;

[0047] The quadruped robot's simulated training terrain environment was built in Isaac Gym, generating flat ground, gravel roads, slopes, stairs, stairs, and discrete obstacles using triangular meshes. Different levels of terrain difficulty were set for each type of terrain, enabling the robot to learn robust movement performance.

[0048] The complete map consists of 10×10 sub-terrains, each representing one type of training terrain. Terrains of the same type increase in difficulty from left to right, forming a row of the complete map. Each column of the complete map represents different types of terrain of the same difficulty, randomly sampled from terrain types at a certain ratio. For example, it randomly samples from five terrain types—gravel roads, slopes, stairs, stairs, and discrete obstacles—at ratios of 0.1, 0.1, 0.35, 0.25, and 0.2.

[0049] To simulate the roughness of flat terrain, noise ranging from ±0.2 cm to ±2 cm was introduced. Slopes with inclinations ranging from 0 to 45 degrees were accompanied by noise of ±0.5 cm. For staircase terrain, the step width remained constant at 30 cm, while the height increased from 2 cm to 30 cm. Twenty rectangular obstacles were placed on discrete obstacle terrain, with obstacle heights increasing from 2 cm to 30 cm and areas ranging from 1 m². 2 Increase to 2m 2 .

[0050] Step 2: Based on the distillation learning concept, construct a teacher network and a student network based on gated recurrent units, and build a total loss function. The teacher network learns a multilayer perceptron policy network π to map ontology perception information and privileged information to motor joint angles, thereby outputting a latent feature vector. The student network learns the latent feature vectors output by the teacher network. To predict the joint angle of the motor;

[0051] like Figure 2 The process of building the teacher network and student network is as follows: S21 to S23:

[0052] S21. A multilayer perceptron encoder μ1 for processing privileged information is built using a multilayer perceptron. Combined with the input dimension of the ontological perception information, an Actor network is designed for the teacher. The Actor network outputs 12-dimensional motor joint angles.

[0053] Specifically, by using a proximal policy optimization algorithm, the teacher network learns a multilayer perceptron policy network π, mapping ontology perception information and privileged information to target joint angles. The Actor network consists of three components: a multilayer perceptron (MLP) encoder μ1 and the multilayer perceptron policy network π. Multiple encoders compress high-dimensional privileged information into a low-dimensional latent representation, reducing the data dimensionality. That is, the privileged information is passed through the multilayer perceptron encoder μ1 to obtain the latent feature vector z. t latent feature vector z t With proprioceptive information o t The motor joint angle 'a' is output through a multilayer perceptron policy network π. t The input-output relationship is as follows:

[0054] z t =μ1(e t );

[0055] a t =π(o) t ,z t ).

[0056] S22. Based on the input in step S21, a terrain height perception encoder (i.e., multilayer perceptron encoder μ2) is built using a multilayer perceptron. A Critic network is designed according to the Actor network dimension, and the output evaluation value is combined with the architecture in step S21 to form an asymmetric teacher network architecture.

[0057] S23: Based on the dimension of historical ontology perception information, design a gated loop unit, build a gated loop unit decoder φ1 with the encoder output dimension consistent with the teacher network, combine it with the Actor network to form a student network architecture, and output 12-dimensional motor joint angles.

[0058] Specifically, the student network can only access ontology-aware information, and it imitates the teacher network's actions through supervised learning. This method is based on hypothesized latent feature vectors. It can be approximately recovered from ontological observations of time series data. Features of the time series data are effectively captured using a gated recurrent unit (GRU) network. Following the GRU network is a multilayer perceptron network, forming the multilayer perceptron encoder φ1. Through supervised learning, the student network simulates the multilayer perceptron policy network. And reconstruct the latent feature vector generated by φ1 The student-side loss function L is defined as follows:

[0059]

[0060] By training the student and teacher networks in parallel, the student network can effectively learn from the early failures of the teacher network, thereby improving the robustness of the algorithm. During the training phase, the multilayer perceptron encoder φ1 is randomly initialized. The student network uses historical ontology perception information and latent feature vectors. By combining the sample labels from the teacher network, a GRU-MLP encoder (i.e., a multilayer perceptron encoder φ1) is trained to predict the joint angles of the motor. The input-output relationship of the student network is as follows:

[0061]

[0062]

[0063] Among them, o t-1:t-k For historical ontology perception information prior to the current moment, o t This refers to the ontology perception information at the current moment.

[0064] Step 3: Based on the constructed total loss function, the student network fits the motor joint angles output by the teacher network through comprehensive simulation training of terrain environment and large-scale parallel training method, which is robust to unstructured terrain.

[0065] At the start of training, all robots are uniformly placed on simple subterraneans. Due to different initial velocity directions, robots may traverse the current subterranean from different directions. To update the subterranean difficulty, when a robot traverses a subterranean, it is moved to the leftmost position of the subsequent subterranean. Simultaneously, the robot's quaternions, velocity vector, and gravity are adjusted using a rotation matrix to align the robot's coordinate system after movement with its local coordinate system before movement, ensuring the robot can move normally in the next subterranean. Through the design of the omnidirectional terrain curriculum, the robot can learn to traverse the current subterranean using commands in different directions. When a robot successfully traverses one-quarter of the next subterranean, the origin of the current subterranean coordinate system is updated to the new position, ensuring that the robot starts from the new subterranean on the next initialization.

[0066] Specifically, refer to Figure 3 As shown, a robot will only move to a more challenging subterranean terrain after successfully adapting to the current one. A robot will only be moved to a more difficult subterranean terrain when it reaches at least 85% of the set average linear speed and successfully traverses the current subterranean terrain. Conversely, if a robot fails to traverse half of the current subterranean terrain at the specified speed by the end of the round, it will be downgraded to an easier terrain. To prevent skill amnesia, once a robot has traversed the most difficult subterranean terrain, it will be randomly transferred to a subterranean terrain of the same category and difficulty.

[0067] Step three specifically includes S31 to S36:

[0068] S31. Initialize all robots on various sub-terrains of the first three terrain difficulties in the simulation training terrain environment. Reset the robots and move with random initial commands.

[0069] Initialize the quadruped robots in Isaac Gym and distribute them evenly across the various terrains of the first three difficulty levels.

[0070] S32. The robot acquires proprioceptive information, privileged information, and discrete heights of the surrounding sub-terrain, which are used as input to the teacher network. The motor joint angle α is then output through the Actor network. t This enables the robot to interact with the simulated training terrain environment;

[0071] The body perception data includes the robot's gravity vector, base angular velocity, motor joint angles, motor joint speeds, command speeds, and the output joint angles of the previous time on the teacher network; the privileged information includes base linear velocity, ground friction coefficient, ground recovery coefficient, foot contact state, robot mass, and the position of the center of mass; the discrete height of the surrounding sub-terrain consists of 187 discrete sampling points around the robot, which is the vertical distance from the surrounding sub-terrain to the robot's base coordinate system.

[0072] The interaction between the robot and the simulated training terrain environment is as follows: The robot learns movement skills in the current sub-terrain and sequentially determines whether it has passed through the current sub-terrain, whether it has passed 25% of the next sub-terrain, and whether it has reached 85% of the target linear velocity. If all of these are true, the difficulty of the sub-terrain is increased. If none of these are true, it determines whether the current training round has ended. If it has not ended, the robot returns to the current sub-terrain to continue learning movement skills. If it has ended, it determines whether it has passed 50% of the terrain. If it has passed, it proceeds to step S31, where the robot resets and moves with a random initial command. If it has not passed, the difficulty of the sub-terrain is reduced.

[0073] S33. Evaluate the output evaluation value through the Critic network to assess the motor joint angle α. t The quality of the robot's actions is assessed, and this information guides the robot's next decision.

[0074] S34. Calculate the teacher-side loss function using the PPO algorithm, and update the Actor network and Critic network;

[0075] S35. The student network acquires historical ontology perception information, evaluates the loss difference between the student network output and the Actor network output based on the constructed student-side loss function, and then updates the student network.

[0076] S26. Repeat steps S32 to S25 until the reward function tends to stabilize; the reward function includes trunk xy axis velocity tracking reward, trunk yaw angular velocity tracking reward, trunk z axis velocity penalty, trunk xy axis angular velocity penalty, gravity vector penalty, joint power penalty, joint acceleration penalty, body height penalty, collision penalty, gait constraint penalty, motion change rate penalty, and second-order motion change rate penalty.

[0077] In this embodiment, based on a reinforcement learning state machine, the trained student network is deployed to a real quadruped robot with zero mapping to achieve motion control of the quadruped robot, specifically including S41 to S43:

[0078] S41. Convert the trained student network into a JIT model using PyTorch, and deploy the model on an embedded platform using libtorch.

[0079] S42. Based on the reinforcement learning state machine template, realize the switching between three states of the robot: squatting, standing, and reinforcement learning motion control;

[0080] S43. Analyze the control commands given by the handle, obtain the historical ontology perception information required for the student model input, realize the student model reasoning, and obtain the 12 motor joint angles. The robot tracks the obtained 12 motor joint angles through PD control to realize robot motion control under real terrain.

[0081] This embodiment first builds different types of simulated training terrain environments in Isaac Gym, gradually increasing the terrain difficulty for robot training; then, based on the distillation learning concept, it constructs a teacher-student network framework based on gated recurrent units; next, the student network gradually learns to fit the latent feature vector z of the teacher network through a comprehensive terrain curriculum and large-scale parallel training methods. t This process involves predicting motor joint angles, deriving the JIT model, and then using libtorch to infer the model on an embedded platform to achieve reinforcement learning-based motion control for the robot. This embodiment is applicable to various robots, but is particularly suitable for quadruped robots.

[0082] This embodiment has the following beneficial effects:

[0083] 1. This embodiment can improve the robustness of quadruped robot movement: This embodiment achieves robust movement of quadruped robot in unstructured terrain through teacher-to-student reinforcement learning method. It has good anti-interference ability and self-recovery ability in dynamic environments that traditional control cannot handle. Compared with other reinforcement learning methods, this embodiment has strong motion performance and can climb continuous stairs and high discrete obstacles without visual input.

[0084] 2. This embodiment can improve the fitting accuracy of the teacher-student network: This embodiment, through the design of an asymmetric teacher-student network, selects inputs with strong recoverability, realizes the recovery of potential information in high-dimensional space from historical ontology perception information, and improves the stability and accuracy of the student network output.

[0085] 3. This embodiment can optimize model training efficiency: This embodiment greatly accelerates the training speed through large-scale parallel training, the Sim2Sim verification operation based on Isaac Gym to Gazebo ensures the physical accuracy of the deployed model, and the comprehensive terrain course ensures that the quadruped robot has a good tracking effect on control commands in any direction.

[0086] 4. This embodiment can enhance the interpretability of latent space vectors: This embodiment encodes ontology perception information into latent space vector information through an asymmetric teacher-student network framework based on gated recurrent units, realizing adaptive motion control for different terrains. In high-dimensional space, latent space vectors can be decoupled, and have good interpretability for different terrains.

[0087] 5. This embodiment has innovations and advantages in many aspects, such as technical implementation, application effect and social benefits, and provides strong support for the development and practical application research of quadruped robot motion control technology.

[0088] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A robot reinforcement learning control method based on gated recurrent units, characterized in that, Based on the reinforcement learning state machine, the trained student network is deployed to the real quadruped robot with zero mapping to achieve the motion control of the quadruped robot; The training process for the student network is as follows: Step 1: Build a simulated training terrain environment for the robot and set different levels of terrain difficulty for each type of terrain; Step 2: Based on the distillation learning concept, construct a teacher network and a student network based on gated recurrent units, and build a total loss function. The teacher network learns a multilayer perceptron policy network. The ontology perception information and privileged information are mapped to the motor joint angles, thereby outputting a latent feature vector. The student network learns the latent feature vectors output by the teacher network. This enables the prediction of motor joint angles; Step 3: Based on the constructed total loss function, the student network fits the motor joint angles output by the teacher network through comprehensive simulation training terrain environment and large-scale parallel training methods. In step two, the teacher network includes an Actor network and a Critic network, and the Actor network includes a multilayer perceptron encoder. Multilayer sensor encoder Multilayer Perceptron Policy Network Privileged information is transmitted through the multilayer perceptron encoder. Obtain latent feature vectors latent feature vectors Information about the subject is obtained through a multilayer perceptron policy network. Output motor joint angle The discrete height of the surrounding sub-terrain is obtained through a multilayer perceptron encoder. The obtained vectors are compared with the latent feature vectors. The ontological perception information is input into the Critic network to output an evaluation value; The student network includes a multilayer perceptron encoder based on a gated loop unit. and multilayer perceptron policy network The historical ontology perception information prior to the current moment is processed by a multilayer perceptron encoder. Output latent feature vector This involves combining the current moment's information from historical ontology perception information with potential feature vectors. Through multi-layer perceptron policy network Output motor joint angle Based on latent feature vectors latent feature vectors Motor joint angle Motor joint angle Construct the student-side loss function.

2. The robot reinforcement learning control method based on gated recurrent units according to claim 1, characterized in that, In step one, the complete map of the simulated training terrain environment consists of 10×10 sub-terrains, each sub-terrain representing one type of training terrain. Terrains of the same type increase in difficulty from left to right to form a row of the complete map, and each column of the complete map represents different types of terrain of the same difficulty.

3. The robot reinforcement learning control method based on gated recurrent units according to claim 1, characterized in that, In step two, the total loss function includes a teacher-side loss function and a student-side loss function. The teacher-side loss function is constructed using the PPO algorithm, and the student-side loss function is as follows: in, This is the loss function.

4. The robot reinforcement learning control method based on gated recurrent units according to claim 3, characterized in that, Step three specifically involves: S31. Initialize all robots on various sub-terrains of the first three terrain difficulties in the simulation training terrain environment. Reset the robots and move with random initial commands. S32. The robot acquires proprioceptive information, privileged information, and discrete heights of the surrounding sub-terrain, which serve as input to the teacher network. The motor joint angles are then output through the Actor network. This enables the robot to interact with the simulated training terrain environment; S33. Evaluate the output values ​​using the Critic network to assess the motor joint angle. The quality of the robot's actions is assessed, and this information guides the robot's next decision. S34. Calculate the teacher-side loss function using the PPO algorithm, and update the Actor network and Critic network; S35. The student network acquires historical ontology perception information, evaluates the loss difference between the student network output and the Actor network output based on the constructed student-side loss function, and then updates the student network. S26. Repeat steps S32 to S25 until the reward function tends to stabilize.

5. The robot reinforcement learning control method based on gated recurrent units according to claim 4, characterized in that, In step S32, the robot learns movement skills in the current sub-terrain and sequentially determines whether it has passed through the current sub-terrain, whether it has passed 25% of the next sub-terrain, and whether it has reached 85% of the target linear velocity. If both are true, then increase the difficulty of the sub-terrain; If the current training round has not ended, the robot returns to the current sub-terrain and continues to learn movement skills. If the current training round has ... sub-terrain has ended, the robot returns to the current sub-terrain and continues to learn movement skills. If the 6. The robot reinforcement learning control method based on gated recurrent units according to claim 4, characterized in that, The body perception information includes the robot's gravity vector, base angular velocity, motor joint angles, motor joint velocities, command velocities, and the output joint angle of the previous time on the teacher network; the privileged information includes base linear velocity, ground friction coefficient, ground recovery coefficient, foot contact state, robot mass, and the position of the center of mass; the discrete height of the surrounding sub-terrain is the vertical distance from the surrounding sub-terrain to the robot's base coordinate system.

7. The robot reinforcement learning control method based on gated recurrent units according to claim 4, characterized in that, The reward function includes trunk xy axis velocity tracking reward, trunk heading angular velocity tracking reward, trunk z axis velocity penalty, trunk xy axis angular velocity penalty, gravity vector penalty, joint power penalty, joint acceleration penalty, body height penalty, collision penalty, gait constraint penalty, motion change rate penalty, and second-order motion change rate penalty.

8. The robot reinforcement learning control method based on gated recurrent units according to claim 1, characterized in that, The trained student network is deployed to a real quadruped robot with zero mapping, specifically as follows: The trained student network is converted into a JIT model using PyTorch, and the model is deployed on an embedded platform using libtorch. Based on the reinforcement learning state machine template, the robot can switch between three states: crouching, standing, and reinforcement learning motion control. The control commands given by the handle are analyzed to obtain the historical ontology perception information required for the student model input, enabling the student model to reason and obtain the 12 motor joint angles. The robot tracks the obtained 12 motor joint angles through PD control to achieve robot motion control in real terrain.

9. A robot reinforcement learning control system based on a gated recurrent unit, characterized in that, This includes deploying a trained student network onto a real quadruped robot with zero mapping based on a reinforcement learning state machine, thereby achieving motion control of the quadruped robot; The training process for the student network is as follows: Step 1: Build a simulated training terrain environment for the robot and set different levels of terrain difficulty for each type of terrain; Step 2: Based on the distillation learning concept, construct a teacher network and a student network based on gated recurrent units, and build a total loss function. The teacher network learns a multilayer perceptron policy network. The ontology perception information and privileged information are mapped to the motor joint angles, thereby outputting a latent feature vector. The student network learns the latent feature vectors output by the teacher network. This enables the prediction of motor joint angles; Step 3: Based on the constructed total loss function, the student network fits the motor joint angles output by the teacher network through comprehensive simulation training terrain environment and large-scale parallel training methods. In step two, the teacher network includes an Actor network and a Critic network, and the Actor network includes a multilayer perceptron encoder. Multilayer sensor encoder Multilayer Perceptron Policy Network Privileged information is transmitted through the multilayer perceptron encoder. Obtain latent feature vectors latent feature vectors Information about the subject is obtained through a multilayer perceptron policy network. Output motor joint angle The discrete height of the surrounding sub-terrain is obtained through a multilayer perceptron encoder. The obtained vectors are compared with the latent feature vectors. The ontological perception information is input into the Critic network to output an evaluation value; The student network includes a multilayer perceptron encoder based on a gated loop unit. and multilayer perceptron policy network The historical ontology perception information prior to the current moment is processed by a multilayer perceptron encoder. Output latent feature vector This involves combining the current moment's information from historical ontology perception information with potential feature vectors. Through multi-layer perceptron policy network Output motor joint angle Based on latent feature vectors latent feature vectors Motor joint angle Motor joint angle Construct the student-side loss function.

Images