A training method and system for quadruped robots based on deep reinforcement learning

Abstract

This invention discloses a training method and system for quadruped robots based on deep reinforcement learning. The AMP training framework significantly enhances the robot's adaptability to complex environments. By distinguishing the robot's current motion sample distribution from that of a reference sample, the training aims to maximize the probability of misjudgment and the reward for reinforcement learning, enabling the robot to learn motion patterns that meet actual needs. When the environment changes, the robot can adjust its strategy promptly based on new information without remodeling, solving the problem of traditional methods being sensitive to environmental changes. Furthermore, this framework introduces imitation learning into reinforcement learning, providing effective guidance for robot learning, reducing the blindness of random exploration, accelerating the learning speed of effective strategies, and improving training efficiency. Simultaneously, the integrated hierarchical LSTM Actor network can extract and aggregate motion features at multiple time scales, allowing the robot to generate actions that consider both short-term gait and long-term trends, resulting in more natural and fluid movements. It also fully utilizes existing sample information, improving sample utilization and reducing training costs.

Description

Technical Field

[0001] This invention relates to the field of quadruped robot training technology, and in particular to a quadruped robot training method and system based on deep reinforcement learning. Background Technology

[0002] In the field of robotics, quadruped robots have broad application prospects due to their ability to walk and perform tasks in complex terrains, such as disaster relief, field exploration, and logistics transportation. Traditional quadruped robot control methods mainly rely on model-based control algorithms, such as zero-torque point (ZMP) control and inverted pendulum model (LIPM) control. These methods typically require accurate robot dynamics models and are highly sensitive to environmental changes. In practical applications, the environments in which quadruped robots operate are complex and varied, with uneven ground and obstacles being common. This makes it difficult for model-based control methods to accurately model the robots, leading to decreased motion stability and even task failure. Furthermore, traditional control methods often require extensive manual parameter adjustments and experience-based design, which not only increases development costs and time but also limits the robot's adaptability and flexibility in different scenarios. For complex motion tasks, such as running quickly and turning flexibly, traditional methods struggle to achieve efficient and natural motion control. Summary of the Invention

[0003] In view of this, the present invention proposes a training method and system for quadruped robots based on deep reinforcement learning, which can effectively solve the shortcomings of existing technologies, such as difficulty in accurate modeling and difficulty in achieving efficient and natural motion control.

[0004] The technical solution of this invention is implemented as follows:

[0005] A training method for quadruped robots based on deep reinforcement learning, specifically including:

[0006] Collect state observation information of the quadruped robot, including angular velocity, gravity vector, joint position, joint velocity, previous strategy output, and user-expected velocity command;

[0007] The state observation information is input into a hierarchical LSTM Actor network to generate action vectors, and the quadruped robot is controlled to perform corresponding actions based on the action vectors.

[0008] The AMP training framework is used to distinguish the robot's current motion distribution from the reference motion sample distribution, and training is conducted with the goal of maximizing the misjudgment probability and reinforcement learning reward.

[0009] The AMP training framework parameters are updated alternately at a preset frequency until training converges.

[0010] As a further alternative to the aforementioned deep reinforcement learning-based quadruped robot training method, the specific structure of the hierarchical LSTM Actor network includes:

[0011] The input layer is used to adjust the dimensionality of state observation information based on the Unsqueeze and Transpose modules;

[0012] The hierarchical LSTM Actor network layer is used to extract motion features at different time scales and aggregate multi-scale temporal information based on the adjusted state observation information.

[0013] The output layer is used to reduce the feature dimension layer by layer and generate the final action vector based on the fully connected layer and ELU activation layer.

[0014] As a further optional scheme for the deep reinforcement learning-based quadruped robot training method, the input layer adjusts the dimension of the state observation information based on the Unsqueeze and Transpose modules, specifically including:

[0015] By inserting a single-dimensional channel into the original tensor of the state observation information using the Unsqueeze module, the original three-dimensional tensor with shape [B,T,C] is transformed into a four-dimensional tensor of shape [B,T,1,C], where B represents the batch size, T represents the time step, and C represents the number of feature channels.

[0016] The Transpose module rearranges the dimensions of the transformed four-dimensional tensor to generate an intermediate tensor of shape [B,1,T,C].

[0017] As a further optional scheme for the training method of quadruped robots based on deep reinforcement learning, the hierarchical LSTM Actor network layer includes three LSTM modules, each with 256 hidden units, which process sequence features sequentially; the hidden states of each LSTM module are concatenated and transmitted through the Slice module and the Concat module to achieve multi-scale temporal information aggregation.

[0018] As a further alternative to the aforementioned deep reinforcement learning-based quadruped robot training method, the AMP training framework includes:

[0019] The discriminator takes the quadruped robot's posture, velocity, and the hidden state information of the LSTM Actor network as input and is used to distinguish the current motion distribution of the quadruped robot from the distribution of reference motion samples.

[0020] The policy network is trained with the goal of maximizing the discriminator's misclassification probability and reinforcement learning rewards.

[0021] As a further optional scheme of the aforementioned deep reinforcement learning-based quadruped robot training method, the policy network is trained with the goal of maximizing the discriminator's misclassification probability and reinforcement learning reward, specifically including:

[0022] The discriminator's misclassification probability is achieved by minimizing the discriminator's cross-entropy loss function, making the motion distribution generated by the policy network approximate the reference motion sample;

[0023] Reinforcement learning rewards are calculated based on the policy gradient algorithm and are used to optimize the long-term benefits of the policy network in the environment.

[0024] As a further optional scheme of the aforementioned deep reinforcement learning-based quadruped robot training method, the cross-entropy loss function is specifically:

[0025] ;

[0026] in, Represented as the total loss function, This indicates losses related to rewards for reinforcing the main learning task. This is to counteract the loss caused by imitation. This is represented as a motion smoothing term, used to constrain motion differences between adjacent frames. Represented as the policy entropy regularization term, , , These are represented as the corresponding weights.

[0027] A quadruped robot training system based on deep reinforcement learning includes:

[0028] The acquisition module is used to acquire the state observation information of the quadruped robot. The state observation information includes angular velocity, gravity vector, joint position, joint velocity, previous strategy output, and user-expected velocity command.

[0029] The hierarchical LSTM Actor network module is used to input the state observation information into the hierarchical LSTM Actor network, generate action vectors, and control the quadruped robot to perform corresponding actions based on the action vectors;

[0030] The deep reinforcement learning module is used to distinguish the robot’s current motion distribution from the reference motion sample distribution using the AMP training framework, and is trained with the goal of maximizing the misjudgment probability and reinforcement learning reward.

[0031] The parameter update module is used to alternately update the AMP training framework parameters at a preset frequency until training converges.

[0032] A computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described deep reinforcement learning-based quadruped robot training methods.

[0033] A computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the steps of the above-described quadruped robot training method based on deep reinforcement learning.

[0034] The beneficial effects of this invention are as follows: First, the use of the AMP training framework enhances the robot's adaptability to complex environments. By distinguishing the robot's current motion distribution from the reference motion sample distribution and training with the goal of maximizing the misjudgment probability and reinforcement learning rewards, the robot can learn motion patterns that better meet the needs of the actual environment. Even when the environment changes, the robot can adjust its strategy in a timely manner based on new state observation information without remodeling, effectively solving the problem of sensitivity to environmental changes in traditional methods. Second, by introducing imitation learning into the reinforcement learning framework and using the AMP training framework to allow the robot to imitate the reference motion sample distribution, this method provides effective guidance for the robot's learning, reduces the blindness of random exploration, and enables the robot to learn effective motion strategies more quickly, greatly improving training efficiency. In addition, the hierarchical LSTM Actor network can extract motion features at different time scales and perform multi-scale temporal information aggregation. This allows the robot to comprehensively consider short-term gait rhythm and long-term motion trends when generating action vectors, thereby generating more natural and smooth movements. At the same time, it makes full use of existing motion sample information, improving sample utilization and reducing training costs. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart illustrating a quadruped robot training method based on deep reinforcement learning according to the present invention.

[0037] Figure 2 This is a schematic diagram of the components of a quadruped robot training system based on deep reinforcement learning according to the present invention.

[0038] Figure 3 This is a schematic diagram of the composition of a computing device according to the present invention. Detailed Implementation

[0039] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] refer to Figures 1 to 3 A training method for quadruped robots based on deep reinforcement learning, specifically including:

[0041] The state observation information of the quadruped robot is collected. The state observation information includes angular velocity, gravity vector, joint position, joint velocity, previous strategy output and user-expected velocity command, but does not include explicit terrain information.

[0042] Specifically, various sensors are installed on the robot, such as gyroscopes to measure angular velocity and encoders to acquire information such as joint position and joint velocity. At the same time, the robot receives the user's desired velocity command through the user interface. The state observation information collected by these sensors in real time (angular velocity, gravity vector, joint position, joint velocity, previous policy output, and user's desired velocity command) is transmitted to the hierarchical LSTM Actor network in the training system. Since the state observation information does not contain explicit terrain information, the robot is not controlled based on a specific terrain model. Instead, it makes decisions by sensing the state in real time through its own sensors and combining the learned policies. This allows the robot to flexibly adjust its motion strategy based on real-time state observation information when facing different terrains (such as flat ground, slopes, gravel roads, etc.), without the need for precise pre-modeling of each terrain, greatly improving its adaptability to complex and varied terrains.

[0043] The state observation information is input into a hierarchical LSTM Actor network to generate action vectors, and the quadruped robot is controlled to perform corresponding actions based on these action vectors. In some embodiments, the specific structure of the hierarchical LSTM Actor network includes:

[0044] The input layer is used to adjust the dimensionality of state observation information based on the Unsqueeze and Transpose modules;

[0045] The hierarchical LSTM Actor network layer is used to extract motion features at different time scales and aggregate multi-scale temporal information based on the adjusted state observation information.

[0046] The output layer is used to reduce the feature dimension layer by layer and generate the final action vector based on the fully connected layer and ELU activation layer.

[0047] Specifically, the dimensionality of the state observation information is adjusted based on the Unsqueeze and Transpose modules to adapt it to the input requirements of the subsequent hierarchical LSTM Actor network. The dimensionalized state observation information is then fed into the hierarchical LSTM Actor network, where three LSTM modules (LSTM_10, LSTM_20, and LSTM_30) process the input sequence features sequentially. Each LSTM module extracts motion features at different time scales during processing. For example, the first LSTM layer may primarily capture short-term gait rhythm changes, while subsequent layers can focus on longer-term motion trends. The information processed by the hierarchical LSTM Actor network is then passed to the output layer. The output layer first performs a linear transformation on the features through a fully connected layer, and then introduces non-linear factors using an ELU activation layer to progressively reduce the feature dimensionality, ultimately generating motion vectors corresponding to the 12 joint commands of the quadruped robot. The robot controls the movement of each joint based on the generated motion vectors to achieve actions such as forward movement and turning. During the training phase, combined with the AMP training framework, the relevant parameters are alternately updated to maximize the discriminator's misclassification probability and reinforcement learning rewards, continuously optimizing the hierarchical LSTM. The strategy for generating the Actor network continues until training converges.

[0048] Thus, the hierarchical LSTM Actor network structure is designed to effectively handle the temporal features in the state observation information of the quadruped robot. By sequentially processing the sequence features and aggregating multi-scale temporal information through multi-layer LSTM modules, the robot can comprehensively consider the motion state at different time points, such as the joint movements of the past few steps and the trend of speed changes, thereby more accurately predicting and generating appropriate actions and improving its adaptability to dynamic environments. The output layer, through the combination of fully connected layers and ELU activation layers, can progressively reduce the feature dimension and generate accurate motion vectors. This ensures that the generated motion commands can accurately control the various joints of the quadruped robot, achieving complex and precise movements, such as flexible turning and stable climbing, thus improving the robot's motion performance and flexibility.

[0049] In some embodiments, the input layer adjusts the dimension of the state observation information based on the Unsqueeze module (dimensionality expansion module) and the Transpose module (transpose module), specifically including:

[0050] By inserting single-dimensional channels into the original tensor of state observation information using the Unsqueeze module (dimension expansion module), the original three-dimensional tensor with shape [B,T,C] is transformed into a four-dimensional tensor [B,T,1,C], where B represents the batch size, T represents the time step, and C represents the number of feature channels.

[0051] The Transpose module rearranges the dimensions of the transformed four-dimensional tensor to generate an intermediate tensor of shape [B,1,T,C].

[0052] Specifically, the quadruped robot is equipped with various sensors, such as gyroscopes, joint position sensors, and velocity sensors, which continuously collect state observation information of the robot. Assume that at a certain moment, the three-dimensional tensor shape formed by the raw state observation data such as angular velocity and joint position collected by the sensors is [64, 10, 8], where the batch size B = 64 (i.e., 64 sets of data samples are processed simultaneously), the time step T = 10 (meaning each sample contains information from 10 time steps), and the number of feature channels C = 8 (covering multiple features such as angular velocity and different joint positions);

[0053] The input layer first uses the Unsqueeze module (dimensional expansion module) to convert the original three-dimensional tensor of shape [64,10,8] into a four-dimensional tensor of shape [64,10,1,8] in an appropriate dimension (e.g., inserting a single-dimensional channel before the feature channel dimension). This step prepares for subsequent dimension rearrangement and data adaptation to the hierarchical LSTM Actor network.

[0054] Next, the dimensionality of the transformed four-dimensional tensor is rearranged using the Transpose module. Based on the input requirements of the hierarchical LSTM Actor network layers, an intermediate tensor with the shape [64, 1, 10, 8] is generated. After this adjustment, the dimensional order of the tensor conforms to the input specifications of the LSTM network for batch, single-sample extended dimensions, time steps, and feature channels, so that the state observation information can be successfully input into the subsequent hierarchical LSTM Actor network for feature extraction and action generation.

[0055] Thus, by adjusting the dimensions of state observation information through the Unsqueeze module (dimensionality expansion module) and the Transpose module (transpose module), the original irregular or non-compliant data can be converted into a suitable format, ensuring that the state observation information can be smoothly input into the hierarchical LSTM Actor network, laying the foundation for subsequent feature extraction and action generation. Reasonable dimensional adjustment helps optimize the flow and processing of data in the computation graph. After converting the data into dimensions suitable for network processing, tensor operations can be performed more efficiently in subsequent network computations, reducing unnecessary computational redundancy, improving the efficiency of the entire training and inference process, and shortening the response time of quadruped robot motion control.

[0056] In some embodiments, the hierarchical LSTM Actor network layer includes three LSTM modules, each with 256 hidden units, which process sequence features sequentially; the hidden states of each LSTM module are spliced ​​and transmitted through the Slice module and the Concat module to achieve multi-scale temporal information aggregation.

[0057] Specifically, the state observation sequence first enters the first LSTM module of the hierarchical LSTM Actor network layer. This module has 256 hidden units. The first LSTM module performs preliminary feature extraction on the input sequence, capturing short-term temporal features in the state observation information, such as the trend of obstacle changes that the robot may encounter in the next very short time, and the need for minor adjustments in its own joint movements. Its hidden state contains these preliminarily extracted feature information. Next, the hidden state of the first LSTM module is concatenated and passed through the Slice module and the Concat module. The Slice module can slice the hidden state according to preset rules to extract information from key parts. Then, the Concat module concatenates these sliced ​​information with other possible relevant information to form a new feature representation, which is then passed to the second LSTM layer. The first LSTM module has 256 hidden units. Based on the features extracted in the first layer, it further processes the sequence features, focusing on medium-term time-scale information, such as how the robot plans its path to avoid obstacles and how it adjusts its body posture to maintain balance in the next few steps. Its hidden state is then processed again by the Slice and Concat modules before being passed to the third LSTM module. The third LSTM module continues to extract and aggregate features, focusing on long-term time-scale information, such as the robot's overall strategy in the entire obstacle avoidance process and how to safely and efficiently reach the target position from the current position. Finally, the feature information output by the third LSTM module is processed to generate motion vectors, which control the movement of each joint of the quadruped robot to achieve dynamic obstacle avoidance.

[0058] Thus, the three-layer LSTM module processes sequential features sequentially, with each layer focusing on information at different time scales. The first layer captures short-term features, enabling rapid response to immediate changes in the environment; the second layer focuses on medium-term features, helping to plan the robot's actions in the next few steps; and the third layer focuses on long-term features, ensuring the overall strategy of the robot is reasonable throughout the task. This aggregation of multi-scale temporal information allows the robot to fully understand environmental dynamics and changes in its own state, making decisions that are more in line with reality. Furthermore, the hidden states of each LSTM module are concatenated and transferred through the Slice module and the Concat module, achieving fine-grained feature processing. With effective aggregation, the Slice operation can extract key feature fragments and avoid interference from redundant information; the Concat operation integrates feature information from different sources, enriching the representation dimension of features. This processing method enables the network to learn more complex and representative features, improving the understanding and utilization of state observation information. In addition, based on multi-scale temporal features and rich feature representations, the generated motion vectors can more accurately control the joint movement of the quadruped robot. In dynamic obstacle avoidance scenarios, the robot can more flexibly adjust its gait, turning angle and speed to accurately avoid obstacles while maintaining body balance and stability, improving the accuracy and reliability of motion control.

[0059] The AMP training framework is used to distinguish between the robot's current motion distribution and the reference motion sample distribution, and training is performed with the goal of maximizing the misclassification probability and reinforcement learning reward; in some embodiments, the AMP training framework includes:

[0060] The discriminator takes the quadruped robot's posture, velocity, and the hidden state information of the LSTM Actor network as input and is used to distinguish the current motion distribution of the quadruped robot from the distribution of reference motion samples.

[0061] The policy network is trained with the goal of maximizing the discriminator's misclassification probability and reinforcement learning rewards.

[0062] Specifically, the discriminator in the AMP training framework receives the quadruped robot's posture, velocity, and hidden state information from the LSTM Actor network. By learning the characteristics of the distribution of reference motion samples, the discriminator attempts to distinguish whether the robot's current motion comes from real reference samples or is generated by the policy network. For example, the discriminator can analyze whether the robot's current joint angle change pattern matches the joint angle change pattern in the reference samples.

[0063] The policy network is trained with the goal of maximizing the discriminator's misclassification probability and reinforcement learning rewards. Reinforcement learning rewards are set based on the robot's motion performance; for example, approaching the reference speed yields a positive reward, while excessive swaying yields a negative reward. During training, the policy network continuously adjusts its parameters, generating different action policies to enable the robot to perform corresponding actions. As training progresses, the motion generated by the policy network gradually approaches the reference motion sample while simultaneously satisfying the requirements of reinforcement learning rewards, making it increasingly difficult for the discriminator to distinguish between the current motion and the generated motion.

[0064] The training system updates the parameters of the discriminator and the policy network alternately at a preset frequency. For example, for every one iteration of the policy network, two iterations of the discriminator are trained. Through this alternating optimization method, the motion policy generated by the policy network is continuously improved until the training converges.

[0065] Thus, the existence of the discriminator prompts the policy network to generate motions that are as close as possible to the distribution of reference motion samples. By continuously maximizing the discriminator's misjudgment probability, the policy network can learn key features and patterns in the reference motion, enabling the quadruped robot to imitate motion actions that are closer to the reference, thereby improving the effectiveness of imitation learning. Secondly, the reinforcement learning reward mechanism guides the policy network to further optimize the robot's motion performance based on imitating the reference motion. For example, by rewarding actions that approach the desired speed and maintain a stable posture, the policy network can generate policies that make the robot run faster and more stably, improving the robot's efficiency and stability in actual motion. In addition, the AMP training framework combines the advantages of imitation learning and reinforcement learning, enabling the policy network not only to learn the features of the reference motion samples but also to explore and optimize during the reinforcement learning process. This gives the trained policies a certain degree of generalization ability. When the robot encounters a scenario that is slightly different from the training environment, it can still generate reasonable motion strategies based on state observation information and adapt to the new environment.

[0066] In some embodiments, the policy network is trained with the goal of maximizing the discriminator's misclassification probability and reinforcement learning rewards, specifically including:

[0067] The discriminator's misclassification probability is achieved by minimizing the discriminator's cross-entropy loss function, making the motion distribution generated by the policy network approximate the reference motion sample;

[0068] Reinforcement learning rewards are calculated based on the policy gradient algorithm and are used to optimize the long-term benefits of the policy network in the environment.

[0069] Specifically, when training the policy network, the goal is to maximize the discriminator's misclassification probability and the reinforcement learning reward. The discriminator's misclassification probability is achieved by minimizing the discriminator's cross-entropy loss function. For example, when the discriminator has a high accuracy in judging whether the robot's current motion is a reference sample or a generated motion, the cross-entropy loss value is large. The policy network will adjust its own parameters based on this feedback, so that the generated motion distribution gradually approaches the reference motion sample, thereby reducing the discriminator's judgment accuracy and increasing the misclassification probability.

[0070] Reinforcement learning rewards are calculated based on the policy gradient algorithm. Reward rules are set according to the robot's walking performance on different terrains. For example, a certain positive reward is given for walking steadily on flat grass, an extra reward is given for successfully climbing a slope, and a negative reward is given for falling or excessive shaking. The policy network adjusts its parameters according to these reward signals through the policy gradient algorithm to optimize the long-term benefits in the environment. During the training process, relevant parameters are updated alternately at a certain frequency. After a large number of iterations, the policy network gradually converges.

[0071] Thus, by minimizing the discriminator's cross-entropy loss function to maximize the misclassification probability, the policy network enables the generated motion distribution to approximate the reference motion sample, thereby accurately mimicking the walking movements of professionals on complex terrain. Simultaneously, by combining reinforcement learning rewards to optimize the policy, the robot can adjust its motion strategy based on actual terrain and task requirements, improving the adaptability and efficiency of its movement. Secondly, the reinforcement learning rewards calculated based on the policy gradient algorithm guide the policy network to optimize long-term gains in the environment. This allows the robot to not only focus on its current motion performance when facing complex terrain but also consider subsequent action planning, thereby adopting a more stable and sustainable motion strategy, reducing instability such as falls, and improving the stability and reliability of its movement. Furthermore, the alternating parameter updates and the combination of two training objectives allow the discriminator and policy network to mutually promote and counterbalance each other. Optimization of the discriminator provides a clear learning direction for the policy network, while improvements in the policy network drive the discriminator to continuously improve its performance, accelerating the convergence of the entire training process, reducing the time and resources required for training, and improving training efficiency.

[0072] In some embodiments, the cross-entropy loss function is specifically:

[0073] ;

[0074] in, Represented as the total loss function, This indicates losses related to rewards for reinforcing the main learning task. This is to counteract the loss caused by imitation. This is represented as a motion smoothing term, used to constrain motion differences between adjacent frames. Represented as the policy entropy regularization term, , , These are represented as the corresponding weights.

[0075] Specifically, during training, the total loss function is calculated, where the loss related to the reward of the main reinforcement learning task is included. The loss value is set based on indicators such as the accuracy and smoothness of the robot's actions. For example, if the robot can accurately complete the specified actions, a lower loss value is given; if there are delays or errors in the actions, the loss value is increased accordingly.

[0076] Counteracting imitation loss This is used to measure the difference between the motion distribution generated by the policy network and the distribution of reference motion samples. A discriminator determines whether the robot's current motion comes from the reference sample or the generated motion. If the discriminator can accurately distinguish between them, it means that the generated motion differs significantly from the reference motion. A larger value indicates a larger value, and vice versa.

[0077] Motion smoothing item Constraining the differences in actions between adjacent frames requires that the robot's movements transition naturally and smoothly. Excessive changes in joint angles or sudden speed changes between adjacent frames will lead to problems. The value will increase to make the generated motion smoother;

[0078] Policy entropy regularization term The policy network is encouraged to explore. In the early stages of training, a higher policy entropy helps the robot try different action combinations and avoid getting stuck in local optima. As training progresses, its weights are adjusted appropriately to achieve a balance between exploration and exploitation.

[0079] By continuously adjusting the weights , , And based on the total loss function Backpropagation updates the parameters of the policy network and discriminator. Through extensive iterative training, the robot gradually learns complex and fluid movements.

[0080] The AMP training framework parameters are updated alternately at a preset frequency until training converges.

[0081] Specifically, alternating parameter updates at a preset frequency (e.g., two discriminator iterations for every one policy network iteration) avoids the instability caused by unilaterally over-optimizing a part of the network (discriminator or policy network). For example, if the policy network is updated multiple times without updating the discriminator, the discriminator may not be able to provide timely and effective feedback, causing the policy network's learning direction to deviate. Conversely, alternating updates ensure a dynamic balance between the two during training, making the training process more stable.

[0082] A quadruped robot training system based on deep reinforcement learning includes:

[0083] The acquisition module is used to acquire the state observation information of the quadruped robot. The state observation information includes angular velocity, gravity vector, joint position, joint velocity, previous strategy output, and user-expected velocity command.

[0084] The hierarchical LSTM Actor network module is used to input the state observation information into the hierarchical LSTM Actor network, generate action vectors, and control the quadruped robot to perform corresponding actions based on the action vectors;

[0085] The deep reinforcement learning module is used to distinguish the robot’s current motion distribution from the reference motion sample distribution using the AMP training framework, and is trained with the goal of maximizing the misjudgment probability and reinforcement learning reward.

[0086] The parameter update module is used to alternately update the AMP training framework parameters at a preset frequency until training converges.

[0087] A computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described deep reinforcement learning-based quadruped robot training methods.

[0088] A computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the steps of the above-described quadruped robot training method based on deep reinforcement learning.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A training method for a quadruped robot based on deep reinforcement learning, characterized in that, Specifically, it includes: Collect state observation information of the quadruped robot, including angular velocity, gravity vector, joint position, joint velocity, previous strategy output, and user-expected velocity command; The state observation information is input into a hierarchical LSTM Actor network to generate action vectors, and the quadruped robot is controlled to perform corresponding actions based on the action vectors. The AMP training framework is used to distinguish the robot's current motion distribution from the reference motion sample distribution, and training is conducted with the goal of maximizing the misjudgment probability and reinforcement learning reward. The AMP training framework parameters are updated alternately at a preset frequency until training converges.

2. The quadruped robot training method based on deep reinforcement learning according to claim 1, characterized in that, The specific structure of the hierarchical LSTM Actor network includes: The input layer is used to adjust the dimensionality of state observation information based on the Unsqueeze and Transpose modules; The hierarchical LSTM Actor network layer is used to extract motion features at different time scales and aggregate multi-scale temporal information based on the adjusted state observation information. The output layer is used to reduce the feature dimension layer by layer and generate the final action vector based on the fully connected layer and ELU activation layer.

3. The quadruped robot training method based on deep reinforcement learning according to claim 2, characterized in that, The input layer adjusts the dimensions of the state observation information based on the Unsqueeze and Transpose modules, specifically including: By inserting a single-dimensional channel into the original tensor of the state observation information using the Unsqueeze module, the original three-dimensional tensor with shape [B,T,C] is transformed into a four-dimensional tensor of shape [B,T,1,C], where B represents the batch size, T represents the time step, and C represents the number of feature channels. The Transpose module rearranges the dimensions of the transformed four-dimensional tensor to generate an intermediate tensor of shape [B,1,T,C].

4. The quadruped robot training method based on deep reinforcement learning according to claim 3, characterized in that, The hierarchical LSTM Actor network layer includes three LSTM modules, each with 256 hidden units, which process sequence features sequentially. The hidden states of each LSTM module are concatenated and transmitted through the Slice and Concat modules to achieve multi-scale temporal information aggregation.

5. The quadruped robot training method based on deep reinforcement learning according to claim 4, characterized in that, The AMP training framework includes: The discriminator takes the quadruped robot's posture, velocity, and the hidden state information of the LSTM Actor network as input and is used to distinguish the current motion distribution of the quadruped robot from the distribution of reference motion samples. The policy network is trained with the goal of maximizing the discriminator's misclassification probability and reinforcement learning rewards.

6. The quadruped robot training method based on deep reinforcement learning according to claim 5, characterized in that, The policy network is trained with the goal of maximizing the discriminator's misclassification probability and reinforcement learning rewards, specifically including: The discriminator's misclassification probability is achieved by minimizing the discriminator's cross-entropy loss function, making the motion distribution generated by the policy network approximate the reference motion sample; Reinforcement learning rewards are calculated based on the policy gradient algorithm and are used to optimize the long-term benefits of the policy network in the environment.

7. The quadruped robot training method based on deep reinforcement learning according to claim 6, characterized in that, The cross-entropy loss function is specifically as follows: ； in, Represented as the total loss function, This indicates losses related to rewards for reinforcing the main learning task. This is to counteract the loss caused by imitation. This is represented as a motion smoothing term, used to constrain motion differences between adjacent frames. Represented as the policy entropy regularization term, , , These are represented as the corresponding weights.

8. A training system for a quadruped robot based on deep reinforcement learning, characterized in that, include: The acquisition module is used to acquire the state observation information of the quadruped robot. The state observation information includes angular velocity, gravity vector, joint position, joint velocity, previous strategy output, and user-expected velocity command. The hierarchical LSTM Actor network module is used to input the state observation information into the hierarchical LSTM Actor network, generate action vectors, and control the quadruped robot to perform corresponding actions based on the action vectors; The deep reinforcement learning module is used to distinguish the robot’s current motion distribution from the reference motion sample distribution using the AMP training framework, and is trained with the goal of maximizing the misjudgment probability and reinforcement learning reward. The parameter update module is used to alternately update the AMP training framework parameters at a preset frequency until training converges.

9. A computing device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the deep reinforcement learning-based quadruped robot training method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the quadruped robot training method based on deep reinforcement learning as described in any one of claims 1-7.

Images