A method for training a bipedal robot gait network

Abstract

This invention discloses a method for training a bipedal robot gait network, constructing a dual-channel deep reinforcement learning architecture; and collecting data. X The simulation describes the operation of a bipedal robot in a simplified model within a simulated environment. Based on the current operation state, a reward is obtained, and all information is merged into a Markov decision process and stored in the experience replay area. When the number of Markov decision processes in the experience replay area exceeds a preset threshold... n A predetermined number of Markov decision processes are randomly selected from these processes to update the parameters of the main network and the adversary network. The parameters of the main network are perturbed, and the effects of human memory curves on the hyperparameters of the main network are simulated. clip Make continuous adjustments; move the bipedal robots that have achieved a stable walking time of the preset time to terrains with a higher difficulty level, and repeat the process of updating each network parameter; continuously conduct course learning until the cumulative reward information and stable walking time of all bipedal robots reach the preset values.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and specifically to a method for training a bipedal robot gait network. Background Technology

[0002] In recent years, with the rapid development of deep reinforcement learning technology, training the gait and motion control of bipedal robots using reinforcement learning has become a research hotspot. Traditional methods usually use a single master network for decision-making and use algorithms such as Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) to guide the robot in training in a simulation environment. The main process of these methods usually includes: (1) deploying a single or a small number of robot instances in the simulation platform and collecting their respective state data; (2) using deep neural networks (usually including Actor and Critic) to evaluate the robot's state and output action commands; (3) after the robot performs the action, adjusting the network parameters through reward information from the environment (such as motion tracking, stability, energy efficiency, etc.) to gradually optimize the control strategy.

[0003] While existing training methods have driven the development of bipedal robot gait control technology to some extent, several problems still exist in practical applications: Traditional methods often employ a single-channel master network structure. Due to the high dimensionality of both the state and action spaces, sample utilization is low, requiring a large amount of simulation data for training, and the convergence speed is slow. Relying solely on a single master network, the training process is prone to getting trapped in local optima, lacking sufficient exploratory power. Especially when facing complex dynamic equilibrium and varied terrain, a single-channel network cannot provide comprehensive exploration guidance, thus limiting the robot's mobility and robustness. Current reinforcement learning algorithms are unstable when dealing with complex environmental disturbances, exhibiting performance fluctuations in areas such as gait switching, shock absorption, and balance control. Furthermore, the "simulation-physical gap" between the simulation environment and the real world makes it difficult to directly transfer training results to practical applications. Traditional algorithms are highly sensitive to the setting of hyperparameters (such as clipping parameters), and improper parameter adjustments can lead to instability or performance degradation during training. Additionally, there is a lack of effective means to control the exploration strategy in the early stages of training. Summary of the Invention

[0004] In view of this, the present invention provides a bipedal robot gait network training method to at least solve the problems of low sample efficiency, insufficient exploration, poor stability and hyperparameter sensitivity in the prior art.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for training a bipedal robot gait network includes the following steps:

[0007] S1. A dual-channel deep reinforcement learning architecture is constructed based on the same policy deep reinforcement learning PPO algorithm as the gait network of a bipedal robot, including a main network with the same network structure but different initial parameters and an opponent network.

[0008] S2. Collect the running status of X bipedal robots in a simple shape in the simulation environment, and train the main network and the opponent network based on the running status;

[0009] Construct a Markov decision process and obtain N samples from it. Input the samples into the main network and the opponent network, calculate the KL divergence between the main network and the opponent network, construct the composite loss function of the main network based on the mean square error between the output action value and the target action value and the KL divergence, update the parameters of the main network according to the composite loss function, construct the loss function of the opponent network according to the KL divergence and update the parameters of the opponent network, apply a perturbation to the main network, and update the clip value of the hyperparameter in the main network according to the forgetting curve.

[0010] S3. When the preset performance indicators are met, the simple terrain will be switched to complex terrain, and the main network and the opponent network will be further trained until the cumulative reward information and stable walking time of all bipedal robots reach the preset network performance indicators, and the trained bipedal robot gait network is obtained.

[0011] Preferably, the specific content of the training includes:

[0012] S21. Input the collected current running state into the main network. The main network outputs action commands and converts them into motor control vectors, which are then sent back to the bipedal robot for execution. Further collect the next running state of the bipedal robot after execution, and calculate the current reward value based on the next running state and the preset target state. Combine the current running state, action commands, next running state, and current reward value into a Markov decision process expressed in tuples and store it in the experience playback area.

[0013] S22. Repeat the content of S21 until the number of Markov decision processes in the experience replay area is greater than the preset threshold n;

[0014] S23. Randomly select a preset number of N samples from the experience replay area, input the samples into the main network, calculate the output action value based on each action command and the current running state, and calculate the target action value by combining the current reward value and the action command output by the main network in the next running state; calculate the KL divergence between the action probability distributions output by the main network and the opponent network under the same input, construct the composite loss function of the main network based on the mean square error between the output action value and the target action value and the KL divergence; update the parameters of the main network according to the composite loss function; and update the parameters of the opponent network by constructing the loss function of the opponent network according to the KL divergence.

[0015] S24. Apply a perturbation to the main network, update the hyperparameter clip value in the main network according to the forgetting curve, and after completing the parameter perturbation and hyperparameter update, regenerate the motor control vector using the updated main network and send it back to the bipedal robot for execution.

[0016] Preferably, the network structures of both the main network and the adversary network include a strategy subnetwork and a value subnetwork, wherein the strategy subnetwork and the value subnetwork of the main network are respectively referred to as the first strategy network and the first value network, and the strategy subnetwork and the value subnetwork of the adversary network are respectively referred to as the second strategy network and the second value network.

[0017] The first strategy network interacts with the parallel simulation environment, receives the reduced and normalized operating state, outputs the probability distribution of motor control actions based on the current operating state, and generates control commands based on the probability distribution; the first value network estimates the expected return of the operating state, participates in the calculation of the advantage function and the construction of PPO loss, thereby optimizing the main network strategy.

[0018] The second policy network receives the same operating state as the first policy network without directly interacting with the simulation environment, and independently generates the action probability distribution. By comparing the probability distributions output by the first and second policy networks, the KL divergence between them is calculated, and the KL divergence is used as a loss term to update the parameters of the second policy network. During the optimization process, the first policy network and the first value network simultaneously integrate the PPO algorithm loss and KL divergence for joint gradient updates.

[0019] Preferably, the operating status of the bipedal robot includes the pose and speed information of the freely movable joints of the bipedal robot; the reward information includes the bipedal robot's motion tracking performance reward, stability and balance reward, gait and contact reward, safety and hardware protection reward, motion efficiency and smoothness reward, command response and behavior reward.

[0020] Preferably, each policy subnetwork and value subnetwork in the main network and the adversary network adopts a multilayer perceptron structure, including an input layer, one or more hidden layers and an output layer;

[0021] The input layer is used to receive the current dimensionality-reduced running status and historical action instructions of the bipedal robot, and transmits the input data to the hidden layer after preliminary encoding.

[0022] The hidden layer consists of several fully connected neurons. It uses an activation function to perform a non-linear transformation on the input signal, thereby extracting features from the complex relationship between states and actions.

[0023] The output layer is used to generate policy distributions, that is, to output the estimated value or probability distribution of the corresponding action based on the feature information extracted from the hidden layer, so as to guide the robot to generate motor control commands.

[0024] Wherein, the input layer dimension is equal to the state dimension of the bipedal robot after dimensionality reduction, and the output layer dimension is equal to the number of joints controlled by the robot;

[0025] The input layer has the same dimension as the policy network, while the output layer has a dimension of 1 and is used to estimate the state value corresponding to the current state.

[0026] Preferably, the composite loss function of the main network is:

[0027] ;

[0028] ;

[0029] ;

[0030] ;

[0031] in, Indicates the total loss. This represents the shear constraint loss of the first policy network update. This indicates the loss in the first value network. This represents the first value network loss weight. Denotes KL divergence, Indicates the weight of the KL divergence loss. Indicates time step The expected value of a sample of Markov decision processes. This represents the probability ratio between the new and old policies, where the new policy refers to the policy network updated during the current training process, and the old policy refers to the policy network used to generate the current training data, i.e., the policy before the update. This represents generalized advantage estimation, clip This represents the clipping hyperparameter in the PPO algorithm. Indicates at time step The clipping threshold at that time This indicates the first value network's operational state. The predicted value, This represents the target value estimated through discount rewards and generalized advantage. This represents the standard deviation of the action probability distribution output by the second policy network. This represents the standard deviation of the action probability distribution output by the first policy network. This represents the mean of the action probability distribution output by the second policy network. This represents the mean of the action probability distribution output by the first policy network.

[0032] Preferably, the loss function of the adversary network is:

[0033] ;

[0034] in, Denotes KL divergence, This represents the standard deviation of the action probability distribution output by the second policy network. This represents the standard deviation of the action probability distribution output by the first policy network. This represents the mean of the action probability distribution output by the second policy network. This represents the mean of the action probability distribution output by the first policy network.

[0035] Preferably, the method for perturbing the main network parameters in S24 is as follows:

[0036] ;

[0037] ;

[0038] in, This indicates the updated network parameters. This indicates the network parameters before the update. This indicates an update parameter discount. This represents the noise term that follows a Gaussian distribution. Indicates a Gaussian distribution. This indicates the intensity of the noise disturbance.

[0039] Preferably, the specific content of continuously adjusting the hyperparameter clip of the main network by simulating the human memory curve in S24 includes:

[0040] The hyperparameter clip in the main network is the clipping threshold in the PPO algorithm. The hyperparameter `clip` is used to control the magnitude of policy updates to balance training stability and efficiency. The method for adjusting the hyperparameter `clip` to simulate the human memory curve is as follows:

[0041] ;

[0042] in, The clip value updated at time t. This is the initial clip value. For the decay period, This represents the decay rate.

[0043] The preferred course learning method is:

[0044] ;

[0045] ;

[0046] in, For bipedal robots Have you completed the target distance indicator for the current difficulty terrain? For bipedal robots The maximum distance traveled under the current terrain difficulty. This is the threshold for travel distance set under the current terrain difficulty. For bipedal robots The difficulty level of the current terrain. This represents the increase in terrain difficulty.

[0047] Preferably, in S3, when the preset performance indicators are met, the simple terrain will be switched to complex terrain to further train the main network and the adversary network:

[0048] The bipedal robot is trained in a simulation environment to achieve a preset performance index of stable walking under the current terrain difficulty level. That is, during continuous training, the simulated walking time continuously exceeds the preset stable duration threshold. When the condition is met, the simulation environment is automatically switched to complex terrain.

[0049] In complex terrain, continue to train both the main network and the adversary network.

[0050] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a bipedal robot gait network training method, which has the following beneficial effects:

[0051] This invention is applied to a bipedal robot gait training system based on deep reinforcement learning. By deploying multiple bipedal robots for parallel training in a simulation environment, it significantly improves sample collection efficiency and training throughput, and substantially shortens the time required for policy convergence. The dual-channel architecture consisting of a main network and an adversary network not only maintains consistency in policy learning but also introduces an enhanced exploration mechanism, effectively avoiding getting trapped in local optima and improving the robustness and generalization ability of the policy. Dimensionality reduction and normalization of the running states reduce state space complexity and improve network learning efficiency. Meanwhile, the refined multi-dimensional reward function comprehensively covers key performance indicators in gait control, making the training process more directional and accurate. The introduction of a perturbation mechanism and dynamic adjustment of the clip hyperparameter memory curve during the main network parameter update process further enhances the stability and flexibility of training. Through a course learning mechanism, robots with stable walking ability are gradually transferred to more challenging terrains for continuous training, achieving hierarchical and progressive capability improvement and effectively addressing the challenges of environmental complexity. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of the overall model structure of a bipedal robot gait network training method provided by the present invention;

[0054] Figure 2 A flowchart illustrating a bipedal robot gait network training method is provided.

[0055] Figure 3 This is a schematic diagram illustrating the structure and perturbation method of the main network and the adversary network in an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of the hyperparameter memory curve adjustment method in an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] This invention provides a method for training a bipedal robot gait network, such as... Figure 1-2 As shown, it includes the following steps:

[0059] S1. A dual-channel deep reinforcement learning architecture is constructed based on the same policy deep reinforcement learning PPO algorithm as the gait network of a bipedal robot, including a main network with the same network structure but different initial parameters and an opponent network.

[0060] S2. Collect the running status of X bipedal robots in a simple shape in the simulation environment as environmental interaction data in the early stage of training, and train the main network and the opponent network according to the running status;

[0061] Construct a Markov decision process and obtain N samples from it. Input the samples into the main network and the opponent network, calculate the KL divergence between the main network and the opponent network, construct the composite loss function of the main network based on the mean square error between the output action value and the target action value and the KL divergence, update the parameters of the main network according to the composite loss function, construct the loss function of the opponent network according to the KL divergence and update the parameters of the opponent network, apply a perturbation to the main network, and update the clip value of the hyperparameter in the main network according to the forgetting curve.

[0062] S3. When the preset performance indicators are met, the simple terrain will be switched to complex terrain, and the main network and the opponent network will be further trained until the cumulative reward information and stable walking time of all bipedal robots reach the preset network performance indicators, and the trained bipedal robot gait network is obtained.

[0063] To further implement the above technical solution, the specific training content includes:

[0064] S21. Input the collected current running state into the main network. The main network outputs action commands and converts them into motor control vectors, which are then sent back to the bipedal robot for execution. Further, collect the next running state of the bipedal robot after execution, and calculate the current reward value based on indicators such as the deviation between the next running state and the preset target state, motion stability, energy consumption, or gait continuity. Combine the current running state, action commands, next running state, and current reward value into a Markov decision process expressed in tuples and store it in the experience playback area.

[0065] S22. Repeat the content of S21 until the number of Markov decision processes in the experience replay area is greater than the preset threshold n;

[0066] S23. Randomly select a preset number of N samples from the experience replay area, input the samples into the main network, calculate the output action value based on each action command and the current running state, and calculate the target action value by combining the current reward value and the action command output by the main network in the next running state; calculate the KL divergence between the action probability distributions output by the main network and the opponent network under the same input, construct the composite loss function of the main network based on the mean square error between the output action value and the target action value and the KL divergence; update the parameters of the main network according to the composite loss function; and update the parameters of the opponent network by constructing the loss function of the opponent network according to the KL divergence.

[0067] S24. During the main network parameter update process, a small perturbation is applied to the main network to enhance the policy exploration capability. This perturbation affects the main network parameters, enabling the main network to have stronger local exploration capabilities in the policy space. The perturbation is applied to the main network, and the hyperparameter clip value in the main network is updated according to the forgetting curve, causing this parameter to gradually decrease during training. This tightens the clipping range during policy updates, limits the policy update amplitude, and controls policy fluctuations. After completing the parameter perturbation and hyperparameter update, the updated main network is used to regenerate the motor control vector and feed it back to the bipedal robot for execution.

[0068] It should be noted that the above method can be implemented by computer equipment.

[0069] In this embodiment, simple terrain refers to a flat surface without obstacles, while complex terrain refers to adding terrain elements such as rocks, undulations, and slopes to a basic flat surface. In actual use, complex terrain can be used to train the bipedal robot's gait network step by step according to the level of difficulty.

[0070] The simulation environment for the bipedal robot was built using Isaac Gym, which can be configured to accommodate terrains of varying difficulty. The simulation model of the bipedal robot was constructed using the Unified Robot Description Format (URDF), which includes the kinematic and dynamic descriptions, geometric representation, and collision model of the bipedal robot. A dual-channel deep reinforcement learning network architecture based on the PPO algorithm was constructed, where each policy subnetwork and value subnetwork of the main network and opponent network is a 5-layer multilayer perceptron.

[0071] The simulation model of the bipedal robot interacts within the simulation environment, allowing for the sampling of the robot's operational state. This operational state comprises the pose and velocity information of each joint. After dimensionality reduction and normalization of all information, the operational state space is obtained. ,Will The input is fed into the Actor layer of the main network to obtain the action output. The control vectors are then converted into motor control vectors and fed back to the bipedal robot. The bipedal robot interacts with the environment through these control vectors to obtain its next operational state space. And receive multi-dimensional reward information. All the above information is then constructed into a Markov decision process tuple. The tuple is then stored in the experience replay pool.

[0072] The course learning mechanism is continuously implemented. Through multiple rounds of simulation training and progressively increasing terrain difficulty, the gait stability and adaptability of bipedal robots in complex environments are continuously improved until the cumulative reward information and stable walking time of all bipedal robots reach the preset performance indicators, thus completing the entire training process.

[0073] To further implement the above technical solution, the network structures of both the main network and the adversary network include a strategy sub-network and a value sub-network. The strategy sub-network and value sub-network of the main network are respectively referred to as the first strategy network and the first value network, and the strategy sub-network and value sub-network of the adversary network are respectively referred to as the second strategy network and the second value network.

[0074] The first strategy network interacts with the parallel simulation environment, receives the reduced and normalized operating state, outputs the probability distribution of motor control actions based on the current operating state, and generates control commands based on the probability distribution; the first value network estimates the expected return of the operating state, participates in the calculation of the advantage function and the construction of PPO loss, thereby optimizing the main network strategy.

[0075] The second policy network receives the same operating state as the first policy network without directly interacting with the simulation environment, and independently generates the action probability distribution. By comparing the probability distributions output by the first and second policy networks, the KL divergence between them is calculated, and the KL divergence is used as a loss term to update the parameters of the second policy network. During the optimization process, the first policy network and the first value network simultaneously integrate the PPO algorithm loss and KL divergence for joint gradient updates.

[0076] To further implement the above technical solutions, the operational status of the bipedal robot includes the pose and speed information of the freely movable joints of the bipedal robot; the reward information includes the bipedal robot's motion tracking performance reward, stability and balance reward, gait and contact reward, safety and hardware protection reward, motion efficiency and smoothness reward, command response and behavior reward.

[0077] To further implement the above technical solution, each policy subnetwork and value subnetwork in the main network and the adversary network adopts a multilayer perceptron structure, including an input layer, one or more hidden layers and an output layer.

[0078] The input layer is used to receive the current dimensionality-reduced running status and historical action instructions of the bipedal robot, and transmits the input data to the hidden layer after preliminary encoding.

[0079] The hidden layer consists of several fully connected neurons. It uses an activation function to perform a non-linear transformation on the input signal, thereby extracting features from the complex relationship between states and actions.

[0080] The output layer is used to generate policy distributions, that is, to output the estimated value or probability distribution of the corresponding action based on the feature information extracted from the hidden layer, so as to guide the robot to generate motor control commands.

[0081] Wherein, the input layer dimension is equal to the state dimension of the bipedal robot after dimensionality reduction, and the output layer dimension is equal to the number of joints controlled by the robot;

[0082] The input layer has the same dimension as the policy network, while the output layer has a dimension of 1 and is used to estimate the state value corresponding to the current state.

[0083] To further implement the above technical solution, the composite loss function of the main network is:

[0084] ;

[0085] ;

[0086] ;

[0087] ;

[0088] in, Indicates the total loss. This represents the shear constraint loss of the first policy network update. This indicates the loss in the first value network. This represents the first value network loss weight. Denotes KL divergence, Indicates the weight of the KL divergence loss. Indicates time step The expected value of a sample of Markov decision processes. This represents the probability ratio between the new and old policies, where the new policy refers to the policy network updated during the current training process, and the old policy refers to the policy network used to generate the current training data, i.e., the policy before the update. This represents generalized advantage estimation, clip This represents the clipping hyperparameter in the PPO algorithm. Indicates at time step The clipping threshold at that time This indicates the first value network's operational state. The predicted value, This represents the target value estimated through discount rewards and generalized advantage. This represents the standard deviation of the action probability distribution output by the second policy network. This represents the standard deviation of the action probability distribution output by the first policy network. This represents the mean of the action probability distribution output by the second policy network. This represents the mean of the action probability distribution output by the first policy network.

[0089] To further implement the above technical solution, the loss function of the adversary network is:

[0090] ;

[0091] in, Denotes KL divergence, This represents the standard deviation of the action probability distribution output by the second policy network. This represents the standard deviation of the action probability distribution output by the first policy network. This represents the mean of the action probability distribution output by the second policy network. This represents the mean of the action probability distribution output by the first policy network.

[0092] To further implement the above technical solution, the method for perturbing the main network parameters in S24 is as follows:

[0093] ;

[0094] ;

[0095] in, This indicates the updated network parameters. This indicates the network parameters before the update. This indicates an update parameter discount. This represents the noise term that follows a Gaussian distribution. Indicates a Gaussian distribution. This indicates the intensity of the noise disturbance.

[0096] To further implement the above technical solution, the specific content of continuously adjusting the hyperparameter clip of the main network by simulating the human memory curve in S24 includes:

[0097] The hyperparameter clip in the main network is the clipping threshold in the PPO algorithm. The hyperparameter `clip` is used to control the magnitude of policy updates to balance training stability and efficiency. The method for adjusting the hyperparameter `clip` to simulate the human memory curve is as follows:

[0098] ;

[0099] in, The clip value updated at time t. This is the initial clip value. For the decay period, This represents the decay rate.

[0100] It should be noted that:

[0101] During the main network parameter update process, a perturbation mechanism is introduced to moderately intervene in its parameters, enhancing the policy exploration capability. Furthermore, by simulating the human forgetting curve, the key hyperparameter in the main network—the clipping parameter—is dynamically adjusted to achieve a parameter optimization process that better aligns with learning principles. This mechanism helps the model maintain sensitivity to key policy changes during long-term training, avoiding getting trapped in local optima.

[0102] Building upon this foundation, a curriculum learning approach is employed to gradually transfer bipedal robots that have already achieved a preset walking duration in the simulation environment to more challenging terrain environments. During the transfer process, the parameter update process for the main network and the adversary network is repeatedly executed to ensure that the model possesses sufficient adaptability and generalization ability when facing more complex tasks.

[0103] To further implement the above technical solutions, the course learning method is as follows:

[0104] ;

[0105] ;

[0106] in, For bipedal robots Have you completed the target distance indicator for the current difficulty terrain? For bipedal robots The maximum distance traveled under the current terrain difficulty. This is the threshold for travel distance set under the current terrain difficulty. For bipedal robots The difficulty level of the current terrain. This represents the increase in terrain difficulty.

[0107] It should be noted that:

[0108] This course-based training process will continue until all bipedal robots can reach the predetermined cumulative reward threshold and stable walking time target in each terrain level, thereby achieving the comprehensive achievement of the training objectives and improving the robustness and environmental adaptability of the overall system.

[0109] like Figure 3 As shown, the Actor and Critic subnetworks of the main network and the adversary network are 5-layer multilayer perceptrons, including one input layer, three hidden layers, and one output layer. The dimension of the Actor network's input layer is the number of running states obtained after dimensionality reduction of the bipedal robot, and the dimension of its output layer is the number of joints of the bipedal robot. The dimension of the Critic network's input layer is the number of running states obtained after dimensionality reduction of the bipedal robot, and the dimension of its output layer is 1. The hidden layer dimensions of both the Actor and Critic networks are 512, 256, and 128, respectively. After each round of main network parameter update, a parameter perturbation operation is introduced to enhance the policy's adaptability to the plasticity loss problem caused by terrain changes.

[0110] like Figure 4 As shown, in the Proximal Policy Optimization (PPO) algorithm, the clip hyperparameter is one of its core design features. It is used to construct the clipped surrogate objective to limit the magnitude of policy updates, thereby ensuring the stability and reliability of the training process. This embodiment provides a method for adjusting hyperparameters based on the human memory curve. Initially, the clip hyperparameter is set relatively large to encourage the policy to explore more extensively in the early stages of training. As training progresses, the clip value gradually decreases according to a decay function that simulates the forgetting curve of human memory, thereby enhancing the stability and fine-tuning of the learned policy in the later stages and improving the convergence performance and generalization ability of the policy in complex environments.

[0111] To further implement the above technical solution, when the preset performance indicators are met, S3 will switch from simple terrain to complex terrain to further train the main network and the adversary network:

[0112] The bipedal robot is trained in a simulation environment to achieve a preset performance index of stable walking under the current terrain difficulty level. That is, during continuous training, the simulated walking time continuously exceeds the preset stable duration threshold. When the condition is met, the simulation environment is automatically switched to complex terrain.

[0113] In complex terrain, continue to train both the main network and the adversary network.

[0114] This invention is applied to a bipedal robot gait training system based on deep reinforcement learning. By deploying multiple bipedal robots for parallel training in a simulation environment, it significantly improves sample collection efficiency and training throughput, and substantially shortens the time required for policy convergence. The dual-channel architecture consisting of a main network and an adversary network not only maintains consistency in policy learning but also introduces an enhanced exploration mechanism, effectively avoiding getting trapped in local optima and improving the robustness and generalization ability of the policy. Dimensionality reduction and normalization of the running states reduce state space complexity and improve network learning efficiency. Meanwhile, the refined multi-dimensional reward function comprehensively covers key performance indicators in gait control, making the training process more directional and accurate. The introduction of a perturbation mechanism and dynamic adjustment of the clip hyperparameter memory curve during the main network parameter update process further enhances the stability and flexibility of training. Through a course learning mechanism, robots with stable walking ability are gradually transferred to more challenging terrains for continuous training, achieving hierarchical and progressive capability improvement and effectively addressing the challenges of environmental complexity.

[0115] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for training a bipedal robot gait network, characterized in that, Includes the following steps: S1. A dual-channel deep reinforcement learning architecture is constructed based on the same policy deep reinforcement learning PPO algorithm as the gait network of a bipedal robot, including a main network with the same network structure and a rival network with different initial parameters; Both the main network and the adversary network have a strategy subnetwork and a value subnetwork. The strategy subnetwork and the value subnetwork of the main network are denoted as the first strategy network and the first value network, respectively, and the strategy subnetwork and the value subnetwork of the adversary network are denoted as the second strategy network and the second value network, respectively. The first strategy network interacts with the parallel simulation environment, receives the reduced and normalized operating state, outputs the probability distribution of motor control actions based on the current operating state, and generates control commands based on the probability distribution; the first value network estimates the expected return of the operating state, participates in the calculation of the advantage function and the construction of PPO loss, thereby optimizing the main network strategy. The second policy network receives the same operating state as the first policy network without directly interacting with the simulation environment, and independently generates action probability distributions. By comparing the probability distributions output by the first and second policy networks, the KL divergence between them is calculated, and the KL divergence is used as a loss term to update the parameters of the second policy network. During the optimization process, the first policy network and the first value network simultaneously integrate the PPO algorithm loss and KL divergence for joint gradient updates. S2. Collection X The operation status of a bipedal robot in a simplified simulation environment is used to train the main network and the opponent network based on the operation status. The specific content of the training includes: Construct a Markov decision process to obtain... N For each sample, input the sample into the main network and the opponent network, calculate the KL divergence between the main network and the opponent network, construct the composite loss function of the main network based on the mean square error between the output action value and the target action value and the KL divergence; update the parameters of the main network according to the composite loss function; construct the loss function of the opponent network according to the KL divergence to update the parameters of the opponent network; apply a perturbation to the main network, and update the hyperparameter clip value in the main network according to the forgetting curve; S3. When the preset performance indicators are met, the simple terrain will be switched to complex terrain, and the main network and the opponent network will be further trained until the cumulative reward information and stable walking time of all bipedal robots reach the preset network performance indicators, and the trained bipedal robot gait network is obtained.

2. The bipedal robot gait network training method according to claim 1, characterized in that, The specific training content in S2 includes: S21. Input the collected current running state into the main network. The main network outputs action commands and converts them into motor control vectors, which are then sent back to the bipedal robot for execution. Further collect the next running state of the bipedal robot after execution, and calculate the current reward value based on the next running state and the preset target state. Combine the current running state, action commands, next running state, and current reward value into a Markov decision process expressed in tuples and store it in the experience playback area. S22. Repeat the steps in S21 until the number of Markov decision processes in the experience replay area exceeds a preset threshold. n ; S23. Randomly select a preset number from the experience replay area. N For each set of samples, the samples are input into the main network. Based on each action command and the current running state, the output action value is calculated. Combined with the current reward value and the action command output by the main network in the next running state, the target action value is calculated. The KL divergence between the action probability distributions output by the main network and the opponent network under the same input is calculated. Based on the mean squared error between the output action value and the target action value and the KL divergence, a composite loss function of the main network is constructed. The parameters of the main network are updated according to the composite loss function. The parameters of the opponent network are updated according to the loss function constructed based on the KL divergence. S24. Apply a perturbation to the main network, update the hyperparameter clip value in the main network according to the forgetting curve, and after completing the parameter perturbation and hyperparameter update, regenerate the motor control vector using the updated main network and send it back to the bipedal robot for execution.

3. The bipedal robot gait network training method according to claim 2, characterized in that, The operational status of the bipedal robot includes the pose and velocity information of the freely movable joints of the bipedal robot; the reward information includes the bipedal robot's motion tracking performance reward, stability and balance reward, gait and contact reward, safety and hardware protection reward, motion efficiency and smoothness reward, command response and behavior reward.

4. The bipedal robot gait network training method according to claim 1, characterized in that, Each policy subnetwork and value subnetwork in the main network and the adversary network adopts a multilayer perceptron structure, including an input layer, one or more hidden layers, and an output layer; The input layer is used to receive the current dimensionality-reduced running status and historical action instructions of the bipedal robot, and transmits the input data to the hidden layer after preliminary encoding. The hidden layer consists of several fully connected neurons. It uses an activation function to perform a non-linear transformation on the input signal, thereby extracting features from the complex relationship between states and actions. The output layer is used to generate policy distributions, that is, to output the estimated value or probability distribution of the corresponding action based on the feature information extracted from the hidden layer, so as to guide the robot to generate motor control commands. Wherein, the input layer dimension is equal to the state dimension of the bipedal robot after dimensionality reduction, and the output layer dimension is equal to the number of joints controlled by the robot; The input layer has the same dimension as the policy network, while the output layer has a dimension of 1 and is used to estimate the state value corresponding to the current state.

5. The bipedal robot gait network training method according to claim 1, characterized in that, The composite loss function of the main network is: ； ； ； ； in, Indicates the total loss. This represents the shear constraint loss of the first policy network update. Indicates the first value network loss. This represents the first value network loss weight. Denotes KL divergence, Indicates the weight of the KL divergence loss. Indicates time step The expected value of a sample of Markov decision processes. This represents the probability ratio between the new and old policies, where the new policy refers to the policy network updated during the current training process, and the old policy refers to the policy network used to generate the current training data, i.e., the policy before the update. This represents generalized advantage estimation, clip This represents the clipping hyperparameter in the PPO algorithm. Indicates at time step The clipping threshold at that time This indicates the first value network's operational state. The predicted value, This represents the target value estimated through discount rewards and generalized advantage. This represents the standard deviation of the action probability distribution output by the second policy network. This represents the standard deviation of the action probability distribution output by the first policy network. This represents the mean of the action probability distribution output by the second policy network. This represents the mean of the action probability distribution output by the first policy network.

6. The bipedal robot gait network training method according to claim 1, characterized in that, The loss function of the adversary network is: ； in, Denotes KL divergence, This represents the standard deviation of the action probability distribution output by the second policy network. This represents the standard deviation of the action probability distribution output by the first policy network. This represents the mean of the action probability distribution output by the second policy network. This represents the mean of the action probability distribution output by the first policy network.

7. The bipedal robot gait network training method according to claim 2, characterized in that, The method for perturbing the main network parameters is as follows: ； ； in, This indicates the updated network parameters. This indicates the network parameters before the update. This indicates an update parameter discount. This represents the noise term that follows a Gaussian distribution. Indicates a Gaussian distribution. Indicates the intensity of noise disturbance; The specific details of continuously adjusting the hyperparameter clip of the main network by simulating the human memory curve include: The hyperparameter clip in the main network is the clipping threshold in the PPO algorithm. The hyperparameter `clip` is used to control the magnitude of policy updates to balance training stability and efficiency. The method for adjusting the hyperparameter `clip` to simulate the human memory curve is as follows: ； in, The clip value updated at time t. This is the initial clip value. For the decay period, This represents the decay rate.

8. The bipedal robot gait network training method according to claim 1, characterized in that, The course learning method is as follows: ； ； in, For bipedal robots Have you completed the target distance indicator for the current difficulty terrain? For bipedal robots The maximum distance traveled under the current terrain difficulty. This is the threshold for travel distance set under the current terrain difficulty. For bipedal robots The difficulty level of the current terrain. This represents the increase in terrain difficulty.

9. The bipedal robot gait network training method according to claim 1, characterized in that, In S3, when the preset performance metrics are met, the simple terrain will be switched to complex terrain for further training of both the main network and the adversary network. The bipedal robot is trained in a simulation environment to achieve a preset performance index of stable walking under the current terrain difficulty level. That is, during continuous training, the simulated walking time continuously exceeds the preset stable duration threshold. When the condition is met, the simulation environment is automatically switched to complex terrain. In complex terrain, continue to train both the main network and the adversary network.

Images