Reinforcement learning biped robot gait control method based on multi-layer self-adaptive exploration mechanism

Abstract

The invention discloses a reinforcement learning biped robot gait control method based on a multilayer adaptive exploration mechanism, and belongs to the field of biped robot reinforcement learning control. The method aims at solving the problems that in existing gait control based on a near-end strategy optimization (PPO) algorithm, strategy convergence is unstable, and exploration and utilization balance is insufficient. According to the method, a multi-layer adaptive exploration mechanism is introduced on the basis of a standard PPO framework: firstly, a state-dependent standard deviation network is designed to realize adaptive noise adjustment of a space layer for dynamically controlling action disturbance intensity; secondly, constructing an entropy regularization mechanism based on advantage weighting, and realizing exploration intensity adaptive control of a direction layer; thirdly, a dynamic entropy coefficient is designed according to the uncertainty of a value function, and exploration rhythm self-adaptive adjustment of a time layer is achieved. Through the synergistic effect of three layers of mechanisms, the strategy realizes dynamic balance of exploration and convergence in different training stages, so that the stability of the training process and the strategy convergence quality are improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and robot control, and more specifically, relates to a reinforcement learning-based bipedal robot gait control method based on a multi-layer adaptive exploration mechanism. Background Technology

[0002] With the deep integration of robotics, artificial intelligence, and automatic control theory, humanoid bipedal robots, as an important branch of intelligent robotics, have shown broad application prospects in scenarios such as human-robot collaboration, disaster relief, and complex environment operations due to their human-like locomotion and flexible environmental adaptability. However, the motion control of bipedal robots involves complex characteristics such as high-dimensional state space, nonlinear dynamics, and multi-constraint coordination, making their control in terms of dynamic balance and gait stability significantly more difficult than that of wheeled or quadrupedal robot systems.

[0003] Traditional bipedal robot control methods are primarily based on model-driven strategies, such as zero-moment point control, linear inverted pendulum model control, and optimization-based trajectory planning methods. These methods heavily rely on accurate robot dynamics modeling and prior environmental knowledge, facing numerous challenges in practical applications: controller performance tends to degrade significantly when there are changes in ground friction, external disturbances, or uncertainties in model parameters, making it difficult to maintain a stable gait. Furthermore, model-driven methods are highly parameter-dependent, have high design complexity, and exhibit poor adaptability and robustness across different tasks and environments.

[0004] In recent years, breakthroughs in deep reinforcement learning technology have provided new solutions to continuous control problems. The Proximal Policy Optimization (PPO) algorithm, as an efficient and stable reinforcement learning algorithm, has been widely used in robot control. PPO effectively balances policy update magnitude and training stability by introducing a shearing objective function and KL divergence constraints. However, the standard PPO algorithm still has significant limitations in bipedal robot gait control: First, facing high-dimensional state spaces and sparse reward functions, the algorithm is prone to getting trapped in local optima in the early stages of training, leading to unstable policy convergence; second, its exploration mechanism relies on fixed entropy coefficients and noise settings, making it unable to adaptively adjust the exploration intensity according to the training process and state characteristics, resulting in an imbalance between exploration and utilization, manifesting as premature convergence or continuous oscillation; third, in cross-task migration and external disturbance environments, the algorithm's generalization ability is insufficient, and the robustness of the control policy is poor.

[0005] There is an inherent contradiction between policy convergence and exploration during reinforcement learning training: overexploration, while helping to avoid local optima, introduces policy fluctuations; while insufficient exploration limits policy diversity and weakens the robot's adaptability to complex terrain and dynamic disturbances. Existing PPO algorithms exhibit problems such as large fluctuations in reward curves, excessively rapid entropy decay, and slow convergence speed in bipedal robot control. The root cause lies in the algorithm's use of a fixed exploration mechanism, lacking the ability to intelligently regulate the training process and making it difficult to adapt to the dynamic characteristics of different training stages and state spaces.

[0006] In summary, existing PPO-based bipedal robot control methods still suffer from inherent shortcomings when dealing with high-dimensional nonlinear and dynamically uncertain environments, including unstable policy convergence, lack of adaptability in the exploration mechanism, and insufficient environment generalization ability. Currently, how to construct a multi-layered adaptive exploration mechanism capable of coordinated exploration and control at the state, orientation, and time levels while maintaining the stability of the PPO algorithm's trust domain optimization framework remains an unsolved problem and has become a key technical challenge for improving the gait control performance of bipedal robots. Summary of the Invention

[0007] This invention proposes a reinforcement learning-based bipedal robot gait control method (Multi-layer Adaptive Exploration Proximal Policy Optimization, MLAE-PPO) to overcome the problems of unstable policy convergence and insufficient balance between exploration and exploitation in existing PPO-based gait control methods under complex continuous action spaces. The technical solution adopted in this invention is as follows:

[0008] A reinforcement learning-based gait control method for bipedal robots based on a multi-layer adaptive exploration mechanism is proposed, with the following specific steps:

[0009] Step 1: Establish a motion model for the bipedal robot and select a simulation environment:

[0010] A reinforcement learning training model was established based on the humanoid bipedal walking mechanism. The standard continuous control environment Walker2d-v5 or Humanoid-v5, based on the MuJoCo physics engine of the Gymnasium platform, was selected; both environments can be represented as time-varying environments. Markov decision process ;

[0011] in, The state space includes the body's posture, link posture, and joint kinematics observations. For continuous motion space, corresponding to the target torque or velocity command for each controllable joint; The state transitions are generated by the MuJoCo dynamics solver; Provide immediate environmental benefits; Discount factor;

[0012] The agent's interaction with the environment yields a trajectory:

[0013]

[0014] in, For state, For action, In return, These are the termination and cutoff flags, respectively; the overall goal of strategy optimization is to maximize the cumulative return from the discount.

[0015]

[0016] in, Represents mathematical expectation, For parameters The random policy distribution;

[0017] Step 2: Construct a two-branch Actor-Critic neural network structure:

[0018] In the aforementioned dual-branch Actor-Critic neural network structure, the Actor (policy) network consists of a mean branch. With variance branch In parallel processing, the two branches of the Actor network and the Critic (value) network both receive state vectors from the simulation environment as input. This vector contains the robot's position at time [time]. Multidimensional observation information, including joint angles, angular velocities, body posture, centroid linear velocity, and contact status;

[0019] The Actor network's mean branch employs a four-layer fully connected structure, with the first three layers serving as feature extraction layers. The activation function for each layer is the hyperbolic tangent (tanh). The number of nodes in the output layer corresponds to the action space dimension, which in turn corresponds to the number of controllable joints in the robot. The output is then averaged using the hyperbolic tangent (tanh). Compress to the interval [-1, 1];

[0020] The variance branch of the Actor network uses the same four-layer fully connected structure, with the first three layers serving as feature extraction layers. The activation function is the hyperbolic tangent (tanh). The output layer does not contain an activation function and is used to output the logarithmic standard deviation of each action dimension. The clip function is used to limit the output to the interval [-4, 1], ensuring that the standard deviation is always within a reasonable range.

[0021]

[0022] The two branches of the Actor network output in parallel to form a Gaussian policy distribution:

[0023]

[0024] Represents the state at time t Take action The probability, which follows a Gaussian distribution. , Represents a diagonal matrix function;

[0025] The loss function of the Actor network is optimized using the policy gradient method and combined with the advantage function to maximize the expected reward. Its loss function is:

[0026]

[0027] in, For shearing parameters, The probability ratio between the current policy and the old policy is expressed as:

[0028]

[0029] The advantage function is used to measure the current policy in the state. The performance improvement compared to the old strategy is expressed as:

[0030]

[0031] The timing difference error is calculated using the following formula:

[0032]

[0033] In the state The instant rewards obtained below As a discount factor, As the dominant smoothing factor, The state value function;

[0034] The Critic network employs the same four-layer fully connected structure, uses the hyperbolic tangent function tanh as the activation function, and has a single neuron output layer to estimate the state value function for a given state. To obtain the expected long-term return under a given state;

[0035] Critic fits state values ​​by minimizing temporal difference errors, constructing a mean squared error loss function:

[0036]

[0037] in, In the state The instant rewards obtained below Using the discount factor, by minimizing the above Critic loss function, a smooth estimate of the state value can be obtained, providing a stable value benchmark for the Actor network.

[0038] Step 3: Construct a multi-layer adaptive exploration mechanism and integrate it with the Actor network:

[0039] Introducing a three-layer adaptive exploration within the standard PPO framework:

[0040] Firstly, the spatial layer achieves adaptive noise adjustment for different states through state-related variance branches;

[0041] The spatial layer is located in the variance branch of the Actor network, and for the input state... The Actor network simultaneously outputs the action mean. With log standard deviation Then, the range was clipped to limit the time. The range of values ​​for is obtained as follows:

[0042]

[0043] The standard deviation can be obtained through exponential mapping as follows:

[0044]

[0045] The action distribution of a strategy is composed of both the mean and variance:

[0046]

[0047] in, Indicates action strategy, Represents the Gaussian distribution function. This represents a diagonal matrix function.

[0048] Thus, by adaptively adjusting the standard deviation of action noise in a state-dependent manner, the spatial layer effectively controls the exploration intensity while ensuring training stability, thereby realizing a state-based uncertainty exploration mechanism.

[0049] Secondly, in the direction layer, a preference-weighted entropy regularization is introduced to dynamically adjust the weight of the entropy term based on the preference of the sample.

[0050] State-action pair at each time step Treat it as a single sample. Each time the policy is updated, first calculate the advantage function value for that sample based on the Critic network output. By performing intra-batch normalization on the dominance values, the normalized dominance is obtained:

[0051]

[0052] in, It is the mean of the current batch of sample advantages. It is the standard deviation of the advantage of this batch of samples.

[0053] Then, the orientation weighting factor for the sample set is constructed:

[0054]

[0055] in, The normalized dominance function value. Orientation adjustment factor; sample set orientation weighting factor In the entropy regularization term Introduced as a dynamic coefficient:

[0056]

[0057] in, Indicates the state Lower policy distribution entropy, This indicates the current policy at the th... The state obtained by sampling at each time step The expected value of the corresponding trajectory data distribution is calculated, that is, the expected value of the data at time step is calculated. The average value of the batch samples is used to represent the expectation of the entropy regularization term at that time step;

[0058] When the sample performance is better than the average level ( )hour, Smaller values ​​reduce the weight of the corresponding entropy term, causing the strategy to shrink its distribution near high-dominance actions; when the sample performance is worse than average ( )hour, Increasing the value strengthens the weight of the entropy term, encouraging more exploration in low-advantage areas;

[0059] Third, at the time level, a dynamic entropy coefficient is set based on the uncertainty of value, and the exploration rhythm is adaptively adjusted according to the training process.

[0060] By monitoring the temporal difference residuals (TD residuals) in each round of training. The variance of the current estimate is used to measure its instability, and a dynamic entropy weight decay coefficient is constructed accordingly; the empirical variance of the current batch TD residuals is denoted as... First compress it to a range The dimensionless uncertainty is obtained:

[0061]

[0062] Then update the entropy regularization coefficient using the following formula:

[0063]

[0064] in, Basic entropy weight coefficient , This is a magnification factor used to control the sensitivity of uncertainty to the entropy coefficient. and These are the lower and upper bounds for safety, used to prevent premature convergence due to excessively small entropy weights or divergence during training due to excessively large entropy weights.

[0065] When the policy estimation is unstable in the early stages of training and the variance of the TD residual is large, entropy weighting... The amplification of the error enhances the global exploration capability; as training gradually converges, the TD residual decreases. Automatically decays to near the base entropy weight or safe lower bound This reduces random disturbances, causes the policy distribution to shrink, and improves stability.

[0066] Step 4: Strategy update and optimization.

[0067] An improved loss function is formed by employing a sheared PPO objective and combining it with multi-layered exploration terms, which includes probability ratios. The optimization process includes a value regression term and an entropy regularization term. An early stopping criterion based on KL divergence is set to suppress excessive policy updates, and an adaptive learning rate optimizer is used to complete parameter iteration.

[0068] (1) Environmental interaction and strategy execution.

[0069] At each time step The agent, based on the current observation state The mean and variance branches of the Actor network are input to obtain the Gaussian policy distribution for continuous actions:

[0070]

[0071] Among them, the mean The log-standard deviation is output from the mean branch of the Actor network. Output from the variance branch of the Actor network and pruned to The standard deviation is then obtained through exponential mapping. This allows for the control of motion noise and exploration intensity.

[0072] Actor networks generate actions through reparameterized sampling. When applied to the environment, the environment returns to its state at the next moment. and instant rewards The Critic network then provides a state value estimate. .

[0073] (2) Experience storage and trajectory sampling. The states, actions, rewards, and end markers generated during the above interactions are written into the experience buffer in chronological order to form a multi-environment parallel trajectory set.

[0074]

[0075] Buffer capacity by step The data is stored cyclically. Once a full round of sampling is completed, it can be expanded over time and randomly sampled in small batches during subsequent optimization stages to calculate the loss function and update the network parameters.

[0076] (3) Advantage estimation and target return calculation. After the empirical buffer is filled, the first step is based on the one-step time-series difference error:

[0077]

[0078] The generalized advantage estimation (GAE) method is used to calculate the advantage function estimate at each time step:

[0079]

[0080] in As a discount factor, This is the GAE smoothing coefficient; when the next state is a termination state, a termination flag is used to stop the accumulation of subsequent discount terms. Then, according to...

[0081]

[0082] The target return is constructed using a value function regression to measure the fitting error of the Critic network to the long-term return. The calculation of the advantage and return described above is performed in the implementation by the advantage buffer module iterating backwards over time.

[0083] (4) Trajectory Unfolding and Mini-Batch Sampling. All environmental trajectories collected within an update cycle are flattened along the time and environment dimensions to obtain a length of... The sample sequence, and the corresponding state ,action Log probability of the old strategy Advantage estimation With target return The training dataset is constructed. After randomly shuffling the index of the dataset, it is divided into several small batches according to a preset batch size. Gradient updates are performed several times on each batch to improve sample utilization and reduce estimation variance.

[0084] (5) Strategy improvement and value assessment.

[0085] In each mini-batch, the log probability and value estimate of the sample are first recalculated using the current Actor network to obtain the probability ratio between the old and new policies:

[0086]

[0087] Construct the cutoff probability ratio objective function:

[0088]

[0089] in, The clipping boundary is defined. The combined value function regression loss is used. Combined with the entropy regularization term obtained based on the multi-layer adaptive exploration mechanism, the comprehensive loss function can be obtained:

[0090]

[0091] in, For the value function loss weights, For the current policy in state The entropy below, The sample-level entropy weights are constructed from the advantage estimation. The entropy coefficient, dynamically adjusted by the time layer based on the TD residual variance, realizes the collaborative constraint of the exploration rights of the spatial, directional, and temporal layers. (6) Parameter update and strategy iteration. The above comprehensive loss function is applied to the policy parameters. With state value function parameters The gradient is calculated, and the Adam optimizer is used for backpropagation and gradient pruning. During each update round, the KL divergence between the old and new policies is monitored. When the KL value exceeds a preset threshold, the iteration of the current round is terminated early to prevent the policy from collapsing due to excessively large update steps. After completing several rounds of alternating "environment interaction - experience storage - advantage estimation - policy evaluation and update", an improved PPO policy that meets the requirements of adaptive exploration can be obtained.

[0092] Step 5: Strategy training and simulation verification.

[0093] Policy training and evaluation were performed under parallel sampling settings, and comparative verification was completed in Walker2d-v5 and Humanoid-v5 environments respectively. During training, TensorBoard was used to record and monitor entropy, total loss, and mean reward to determine the exploration intensity, policy convergence magnitude, and reward improvement trend, thereby verifying the stability and training feasibility of the method.

[0094] Furthermore, in step 2, the Actor-Critic dual-branch neural network structure adjusts the state input dimension according to the complexity of different control tasks: the input dimension is 17 in the Walker2d-v5 environment and 376 in the Humanoid-v5 environment.

[0095] The number of neurons in the fully connected layer is 128 in the Walker2d-v5 environment and 512 in the Humanoid-v5 environment.

[0096] The beneficial effects of this invention are as follows:

[0097] (1) A state-related noise adjustment mechanism is proposed, which enables the policy network to adaptively adjust the exploration intensity according to the state complexity, avoids the policy oscillation problem caused by fixed noise, and effectively improves the convergence stability;

[0098] (2) The advantage-weighted entropy regularization mechanism is introduced to dynamically allocate exploration weights according to the sample advantages, thereby achieving a balance between exploration and utilization and significantly improving the sample utilization rate and strategy convergence speed.

[0099] (3) A dynamic entropy coefficient adjustment method based on the uncertainty of the value function was designed, which can adaptively control the exploration rhythm in different training stages, thereby enhancing the robustness and generalization ability of the algorithm.

[0100] (4) The method of the present invention achieves adaptive coordination of exploration intensity, direction and rhythm while maintaining the simplicity of PPO structure, and can obtain more stable and efficient humanoid bipedal robot gait control effect in complex situations. Attached Figure Description

[0101] Figure 1 This is an overall flowchart of the present invention;

[0102] Figure 2 This is a schematic diagram of a bipedal robot's motion model.

[0103] Figure 3 To improve the action network structure of the PPO algorithm;

[0104] Figure 4To improve the evaluation network structure of the PPO algorithm;

[0105] Figure 5 A block diagram of a multi-layer adaptive exploration mechanism;

[0106] Figure 6 To enhance the training and updating framework for learning strategies;

[0107] Figure 7 Training curves (policy entropy, total loss, reward value) in the Walker2d-v5 environment;

[0108] Figure 8 The training curves (policy entropy, total loss, and reward value) are shown in the Humanoid-v5 environment. Detailed Implementation

[0109] The invention will now be described in detail with reference to the accompanying drawings and embodiments, further verifying the effectiveness of the described technical solution. It should be noted that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0110] This invention discloses a reinforcement learning-based gait control method for bipedal robots based on a multi-layer adaptive exploration mechanism. This method, building upon the traditional Proximal Policy Optimization (PPO) algorithm, introduces a multi-layer adaptive exploration mechanism. By jointly adjusting the spatial, directional, and temporal layers, it achieves hierarchical adaptive control of exploration intensity, direction, and pace in complex dynamic systems, thereby significantly improving the stability and convergence performance of humanoid bipedal robots in high-dimensional continuous motion spaces. Figure 1 As shown, the overall process includes five main stages: environment construction and state acquisition, construction of a dual-branch Actor-Critic network structure, construction of a multi-layer adaptive exploration mechanism, policy update and optimization, and reinforcement learning training and simulation verification.

[0111] Phase 1: Environment Setup and Status Acquisition

[0112] During the environment setup phase, the system initializes two simulation environments in the Gymnasium platform: Walker2d-v5 and Humanoid-v5, corresponding to 2D and 3D bipedal robot walking tasks, respectively. Both environments are based on the MuJoCo physics engine to achieve a high-fidelity mechanical simulation process, with the state vector at time t... Including continuously observed variables such as joint angles, angular velocity, center of mass linear velocity, body posture, and ground contact state, the motion vector at time t. The target torque command for each joint; the trajectory sample generated at each interaction step can be represented as

[0113]

[0114] in, The instantaneous reward at time t. and These represent the natural termination and truncation signals, respectively. The above design provides the data foundation for subsequent policy network training and the implementation of adaptive mechanisms.

[0115] The bipedal robot model is based on a simplified humanoid structure. For example... Figure 2 As shown, Walker2d employs a planar bipedal structure, including the torso, legs, and ground contact point, possessing 6 degrees of freedom. Humanoid gait is achieved by applying control torques to the main drive joints, such as the legs. Humanoid is a full-body humanoid model, comprising the torso, legs, arms, and head, with a total of 17 controllable degrees of freedom. The hip, knee, and ankle joints are the main drive joints, connected in series to achieve humanoid gait. The observation signals from the environment and the dynamic feedback together constitute the input space for reinforcement learning. The system achieves closed-loop control between the agent and the physical environment through state-action interaction.

[0116] Phase Two: Construction of the Dual-Branch Actor-Critic Network Structure

[0117] The reinforcement learning control network proposed in this invention adopts an independent dual-branch Actor-Critic framework, such as... Figure 3 and Figure 4 As shown, the Actor network generates joint control commands based on the robot's current state, while the Critic network predicts the long-term cumulative reward of the current state. This structure maintains the decoupling of policy optimization and value estimation, and improves the adaptability of motion distribution modeling and the stability of policy evaluation through a two-branch independent design.

[0118] In the specific design, both the Actor network and the Critic network adopt a multilayer perceptron (MLP) architecture, but innovative improvements have been made in the parameter sharing method and output format. Unlike the single-branch structure of traditional PPOs that share a feature extraction layer, the Actor network of this invention adopts an independent dual-branch structure design, namely, the Actor network mean branch and the Actor network variance branch. These two branches are completely independent in the input layer and hidden layer, and do not share parameters with each other. This decouples the policy's mean output from the noise distribution during the feature extraction process. This structure avoids the interference of action variance on the learning of the mean distribution and improves the convergence stability of the policy in complex state spaces.

[0119] In this embodiment, the MLP architecture of the Actor network and Critic network is as follows:

[0120] I. Actor Network:

[0121] (1) Input layer

[0122] The input to the Actor network is the robot's state vector at the current moment. The state variables include the robot's position, linear velocity, angular velocity, and the angles of each joint. The input layer dimension is consistent with the state space dimension. In this embodiment of the invention, the state dimension is 17 for the Walker2d environment and 376 for the Humanoid environment. All input data is normalized before entering the network to ensure the balance of features with different dimensions and improve training stability.

[0123] (2) Hidden layer and output layer:

[0124] a. Mean branch of the Actor network:

[0125] This branch is used to output the mean vector of the action distribution. That is, the robot in a state The optimal action trend under the given circumstances. For example... Figure 3 As shown, the hidden layer of the mean branch point contains three fully connected layers. The number of neurons in each layer is set according to the environment. It is set to 512 in the Humanoid environment and 128 in the Walker2d environment. The activation function is the hyperbolic tangent function tanh.

[0126] The output layer of this branch also uses a fully connected layer, with the number of neurons equal to the dimension of the action space. The output range is constrained by an activation function. Internally, ensure that the amplitude of the motion output is controllable and conforms to physical constraints.

[0127] b. Variance branch of the Actor network:

[0128] This branch independently receives the same state input. Log-standard deviation used to predict motion noise This allows for adaptive adjustment of the state of the exploration strategy. For example... Figure 3 As shown, the network structure of the variance branch is the same as that of the mean branch. Both the hidden and output layers are composed of fully connected layers, and the number of neurons is also the same as in the mean branch; however, the output layer is not constrained by the hyperbolic tangent function tanh. To avoid training instability caused by excessive or insufficient noise in the initial stage, the logarithmic standard deviation of the output is... Restricted to the range Within this range, the appropriateness of early exploration is ensured. The standard deviation is obtained through exponential mapping as follows:

[0129]

[0130] (3) The action strategy is ultimately modeled by a Gaussian distribution:

[0131]

[0132] in, Indicates action strategy, Represents the Gaussian distribution function. This represents a diagonal matrix function. It uses Gaussian distribution modeling to achieve probability sampling of actions, outputting a smooth and differentiable control signal.

[0133] (4) Loss function:

[0134] The loss function of the Actor network is optimized using the policy gradient method and combined with the advantage function to maximize the expected reward. The loss function is:

[0135]

[0136] in, This indicates the current policy at the th... The state obtained by sampling at each time step The expected value of the corresponding trajectory data distribution is calculated, which is the statistical average of all samples within that time step. This average is used to characterize the current policy at that time step. The expected value of the lower loss term; This is a shearing parameter used to limit the magnitude of policy updates and prevent policy oscillations caused by excessively large update steps. For the current strategy Compared to the old strategy The probability ratio between them is expressed as:

[0137]

[0138] The advantage function is used to measure the current policy in the state. The performance improvement compared to the old strategy is expressed as:

[0139]

[0140] in, As the dominant smoothing factor, For timing difference error:

[0141]

[0142] In the state The instant rewards obtained below As a discount factor, This is the state value function.

[0143] II. Critic Network:

[0144] like Figure 4 As shown, the Critic network employs a four-layer fully connected multilayer perceptron, with its input layer being consistent with the Actor network, where the input is a state vector. For the Humanoid environment, the number of neurons in the fully connected layer is 512, and for the Walker2d environment it is 128. The activation function is the hyperbolic tangent function tanh. The output layer is a single neuron, and the output is the state value. It does not use an activation function and directly outputs a scalar value.

[0145] To make the value estimate approximate the true return of the environment, Critic fits the state value by minimizing the temporal difference error. This embodiment uses a temporal difference objective. As a monitoring signal, construct the mean squared error loss function:

[0146]

[0147] in, In the state The instant rewards obtained below Using the discount factor, by minimizing the above Critic loss function, a smooth estimate of the state value can be obtained, providing a stable value benchmark for the Actor network.

[0148] Phase 3: Construction of a Multi-Layer Adaptive Exploration Mechanism

[0149] The multi-layer adaptive exploration mechanism proposed in this invention consists of three parts: a spatial layer, a directional layer, and a temporal layer. These three layers work together and act sequentially on the parameter update process of the Actor network to dynamically adjust the exploration intensity and directionality of the agent, thereby achieving stable learning and efficient convergence under complex gait tasks. Figure 5 As shown, the specific steps are as follows:

[0150] Step 1: Design of spatial layer adaptive exploration mechanism.

[0151] The spatial layer is primarily responsible for adjusting the variance of the action distribution at the state level. Located in the variance prediction branch of the Actor network, its role is to adaptively adjust the scale of the action distribution based on the current state in the continuous action space, thereby achieving state-based uncertainty-based exploratory control. For the input state... The Actor network simultaneously outputs the action mean. With log standard deviation Then, the range was clipped to limit the time. The range of values ​​for is obtained as follows:

[0152]

[0153] The standard deviation can be obtained through exponential mapping as follows:

[0154]

[0155] The action distribution of a strategy is composed of both the mean and variance:

[0156]

[0157] in, Indicates action strategy, Represents the Gaussian distribution function. This represents a diagonal matrix function.

[0158] Thus, by adaptively adjusting the standard deviation of action noise in a state-dependent manner, the spatial layer effectively controls the exploration intensity while ensuring training stability, thereby realizing a state-based uncertainty exploration mechanism.

[0159] At each time step, the Actor network generates the mean and variance of the action distribution based on the current input state vector, where the variance is output by the variance prediction branch and constrained to... Its role has already been demonstrated in the variance prediction part of the Actor network.

[0160] Step 2: Design of the adaptive exploration mechanism for the direction layer.

[0161] The directional layer is used to distinguish the direction of influence of different trajectories on policy updates at the sample level. This layer proposes an advantage-weighted entropy (AW-Entropy) term to achieve advantage-driven directional exploration regulation. Figure 5 As shown, this embodiment pairs the state and action at each time step. Treat it as a single sample. Each time the policy is updated, first calculate the advantage function value for that sample based on the value network output. By performing intra-batch normalization on the dominance values, the normalized dominance is obtained:

[0162]

[0163] in, It is the mean of the current batch of sample advantages. It is the standard deviation of the advantage of this batch of samples.

[0164] Then, the orientation weighting factor for the sample set is constructed:

[0165]

[0166] in, The normalized dominance function value. This is a direction adjustment factor. This weight is in the entropy regularization term. Introduced as a dynamic coefficient:

[0167]

[0168] in, Indicates the state Lower policy distribution entropy, This indicates the weighting applied to the entropy term of the sample.

[0169] When the sample performance is better than the average level ( )hour, Smaller values ​​reduce the weight of the corresponding entropy term, causing the strategy to shrink its distribution near high-dominance actions; when the sample performance is worse than average ( )hour, Increasing the value strengthens the weight of the entropy term, encouraging more exploration in low-advantage regions. Through this directional layer adaptive exploration mechanism, the influence of different trajectories on the policy entropy regularization term can be dynamically adjusted according to sample advantage, achieving automatic adjustment of the policy update direction and a balance between exploration and exploitation.

[0170] Step 3: Design of dynamic entropy decay mechanism at the time level.

[0171] The temporal layer is primarily used to dynamically adjust the overall exploration intensity based on the uncertainty of the value function at the training iteration scale. This embodiment monitors the temporal difference residuals in each training round. The variance is used to measure the instability of the current estimate, and a dynamic entropy weight decay coefficient is constructed accordingly.

[0172] Let the empirical variance of the current batch of TD residuals be . First compress it to a range The dimensionless uncertainty is obtained:

[0173]

[0174] Then update the entropy regularization coefficient using the following formula:

[0175]

[0176] in, The basic entropy weighting coefficient is taken in this embodiment. , , This is a magnification factor used to control the sensitivity of uncertainty to the entropy coefficient; in this embodiment, it is taken as... , and These are the lower and upper bounds, used to prevent premature convergence due to excessively small entropy weights or training divergence due to excessively large entropy weights. In this embodiment, they are set as follows: .

[0177] When the policy estimation is unstable in the early stages of training and the TD residual variance is large, entropy weighting... The amplification of the error enhances the global exploration capability; as training gradually converges, the TD residual decreases. Automatically decays to near the base entropy weight or safe lower bound This reduces random perturbations, causing the policy distribution to shrink and improving stability. Through this time-layer dynamic entropy decay mechanism, adaptive exploration-utilization equilibrium based on value function uncertainty can be effectively achieved while ensuring training stability.

[0178] The spatial, directional, and temporal layers influence the policy update process from the perspectives of state, sample, and time, respectively. During the training phase, the signals from these three layers participate in the PPO objective function through a weighted entropy regularization term, ensuring that the total loss function maintains a stable balance among the policy clipping constraint, the value error term, and the entropy regularization term.

[0179] Phase Four: Strategy Update and Optimization

[0180] like Figure 6 As shown, in the policy update and optimization stage, this invention establishes a dynamic and interactive proximal policy optimization process based on the aforementioned dual-branch Actor-Critic neural network and multi-layer adaptive exploration mechanism. The agent generates trajectory data through repeated interactions with the environment, estimates the advantage function and target reward using experience storage and trajectory sampling mechanisms, and completes policy evaluation and parameter updates accordingly. Specifically, the process includes the following steps:

[0181] (1) Environmental interaction and strategy execution.

[0182] At each time step The agent, based on the current observation state The mean and variance branches of the Actor network are input to obtain the Gaussian policy distribution for continuous actions:

[0183]

[0184] Among them, the mean Log-standard deviation output from the mean network of the Actor Output from variance network and cropped to The standard deviation is then obtained through exponential mapping. This allows for the control of motion noise and exploration intensity. The strategy generates motions through reparameterized sampling. When applied to the environment, the environment returns to its state at the next moment. and instant rewards The value network provides a state value estimate. .

[0185] (2) Experience storage and trajectory sampling. The states, actions, rewards, and end markers generated during the above interactions are written into the experience buffer in chronological order to form a multi-environment parallel trajectory set.

[0186]

[0187] Buffer capacity by step The data is stored cyclically. Once a full round of sampling is completed, it can be expanded over time and randomly sampled in small batches during subsequent optimization stages to calculate the loss function and update the network parameters.

[0188] (3) Advantage estimation and target return calculation. After the empirical buffer is filled, the first step is based on the one-step time-series difference error:

[0189]

[0190] The generalized advantage estimation (GAE) method is used to calculate the advantage function estimate at each time step:

[0191]

[0192] in As a discount factor, This is the GAE smoothing coefficient; when the next state is a termination state, a termination flag is used to stop the accumulation of subsequent discount terms. Then, according to...

[0193]

[0194] The target return is constructed using a value function regression to measure the fitting error of the Critic network to the long-term return. The calculation of the advantage and return described above is performed in the implementation by the advantage buffer module iterating backwards over time.

[0195] (4) Trajectory Unfolding and Mini-Batch Sampling. All environmental trajectories collected within an update cycle are flattened along the time and environment dimensions to obtain a length of... The sample sequence, and the corresponding state ,action Log probability of the old strategy Advantage estimation With target return The training dataset is constructed. After randomly shuffling the index of the dataset, it is divided into several small batches according to a preset batch size. Gradient updates are performed several times on each batch to improve sample utilization and reduce estimation variance.

[0196] (5) Strategy improvement and value assessment.

[0197] In each mini-batch, the log probability and value estimate of the sample are first recalculated using the current Actor network to obtain the probability ratio between the old and new policies:

[0198]

[0199] Construct the cutoff probability ratio objective function:

[0200]

[0201] in, The clipping boundary is defined. The combined value function regression loss is used. Combined with the entropy regularization term obtained based on the multi-layer adaptive exploration mechanism, the comprehensive loss function can be obtained:

[0202]

[0203] in, For the value function loss weights, For the current policy in state The entropy below, The sample-level entropy weights are constructed from the advantage estimation. The entropy coefficient, dynamically adjusted by the time layer based on the TD residual variance, realizes the collaborative constraint of the exploration rights of the spatial, directional, and temporal layers. (6) Parameter update and strategy iteration. The above comprehensive loss function is applied to the policy parameters. With value function parameters The gradient is calculated, and the Adam optimizer is used for backpropagation and gradient pruning. During each update round, the KL divergence between the old and new policies is monitored. When the KL value exceeds a preset threshold, the iteration of the current round is terminated early to prevent the policy from collapsing due to excessively large update steps. After completing several rounds of alternating "environment interaction - experience storage - advantage estimation - policy evaluation and update", an improved PPO policy that meets the requirements of adaptive exploration can be obtained.

[0204] The key hyperparameter settings for the improved PPO algorithm in this embodiment are shown in Table 1.

[0205] Table 1

[0206]

[0207] This invention was trained and validated in the Walker2d-v5 environment, and the results are as follows: Figure 7 As shown, the Walker-2d environment has a small state space dimension and a 6-dimensional action space dimension, primarily examining low-dimensional gait coordination and ground contact stability. During training, the improved PPO algorithm exhibits a smoother reward increase curve and lower loss fluctuation amplitude, indicating that the three-layer exploration mechanism effectively avoids the policy getting trapped in local optima. Simultaneously, the entropy change curve shows that the algorithm maintains a high exploration intensity in the early stages of training, gradually weakening it in the mid-to-late stages to improve convergence speed, verifying the effectiveness of the orientation and temporal adjustment mechanisms.

[0208] This invention further validates the scalability of the algorithm in the Humanoid-v5 environment, with results as follows: Figure 8 As shown, this environment has a 376-dimensional state space and a 17-dimensional continuous action space, simulating the balance and coordination control task of a high-dimensional dynamic system. In this environment, the standard PPO algorithm exhibits strong training instability and policy oscillations. However, the multi-layer adaptive exploration mechanism of this invention effectively reduces the variance of the policy gradient through state-related variance adjustment and dynamic entropy weight control, resulting in a continuous increase in average reward in the later stages of training. Curve comparisons show that the improved algorithm has a significantly faster convergence speed and a significantly improved final reward value, verifying its stability and robustness in complex gait tasks.

[0209] During training, a parallel sampling and mixed-precision computing mechanism was employed to improve sample utilization and computational efficiency. The number of parallel environments was set to 32, the batch size to 256, and each batch performed 10 mini-batch updates. The optimizer was Adam, and the learning rate was set to 3 × 10⁻⁶. -4 The total number of training rounds is 1000, and the random seed is fixed at 42 to ensure the reproducibility of the results.

[0210] During the training phase, TensorBoard is used to visualize and monitor various metrics. The main monitored variables include:

[0211] (1) Entropy: reflects the intensity and uncertainty of the strategy's exploration and measures the balance between exploration and exploitation;

[0212] (2) Total loss (loss / total): reflects the overall optimization convergence trend;

[0213] (3) Mean / reward: Reflects the overall trend of policy performance and is used to evaluate convergence speed and final performance.

[0214] The results show that, with the same number of training steps, the method of this invention significantly outperforms the standard PPO algorithm in terms of the stability of the reward curve, the convergence smoothness of the loss function, and the dynamic characteristics of entropy adjustment. Especially in the Humanoid environment, the strategy of this invention maintains stable gait output even in complex environments, demonstrating high generalization ability and anti-interference capability.

[0215] In summary, this invention achieves stable convergence of reinforcement learning in complex gait control tasks for bipedal robots by introducing a multi-layer adaptive exploration mechanism. The three-layer mechanism adjusts the exploration behavior of the strategy at different levels: the spatial layer regulates the exploration intensity, the orientation layer guides the exploration direction, and the temporal layer controls the exploration rhythm. This mechanism implements hierarchical adaptive exploration control within the PPO framework, significantly improving the algorithm's performance in high-dimensional continuous control tasks.

Claims

1. A reinforcement learning-based gait control method for bipedal robots based on a multi-layer adaptive exploration mechanism, characterized in that, The specific steps of this method are as follows: Step 1: Establish a motion model for the bipedal robot and select a simulation environment: A reinforcement learning training model was established based on the humanoid bipedal walking mechanism. The standard continuous control environment Walker2d-v5 or Humanoid-v5, based on the MuJoCo physics engine of the Gymnasium platform, was selected; both environments can be represented as time-varying environments. Markov decision process ; in, The state space includes the body's posture, link posture, and joint kinematics observations. For continuous motion space, corresponding to the target torque or velocity command for each controllable joint; The state transitions are generated by the MuJoCo dynamics solver; Provide immediate environmental benefits; Discount factor; The agent's interaction with the environment yields a trajectory: in, For state, For action, In return, These are the termination and cutoff flags, respectively; the overall goal of strategy optimization is to maximize the cumulative return from the discount. in, Represents mathematical expectation, For parameters The random policy distribution; Step 2: Construct a two-branch Actor-Critic neural network structure: In the described dual-branch Actor-Critic neural network structure, both the Actor network and the Critic network employ a multilayer perceptron (MLP) architecture; and the Actor network consists of mean branches. With variance branch In parallel processing, the two branches of the Actor network and the Critic network both receive state vectors from the simulation environment as input. This vector contains the robot's position at time [time]. Multidimensional observation information, including joint angles, angular velocities, body posture, centroid linear velocity, and contact status; The two branches of the Actor network output in parallel to form a Gaussian policy distribution: Represents the state at time t Take action The probability, which follows a Gaussian distribution. , Represents a diagonal matrix function; The input to the Critic network is a state vector. The output is the state value. ; Step 3: Construct a multi-layer adaptive exploration mechanism and integrate it with the Actor network: Introducing a three-layer adaptive exploration within the standard PPO framework: Firstly, the spatial layer achieves adaptive noise adjustment for different states through state-related variance branches; The spatial layer is located in the variance branch of the Actor network, and for the input state... The Actor network simultaneously outputs the action mean. With log standard deviation Then, the range was clipped to limit the time. The range of values ​​for is obtained as follows: The standard deviation can be obtained through exponential mapping as follows: The Gaussian strategy distribution is composed of the mean and variance. Secondly, in the direction layer, a preference-weighted entropy regularization is introduced to dynamically adjust the weight of the entropy term based on the preference of the sample. State-action pair at each time step Treat it as a single sample; each time the policy is updated, first calculate the advantage function value of that sample based on the Critic network output. By performing intra-batch normalization on the dominance values, the normalized dominance is obtained: in, It is the mean of the current batch of sample advantages. This is the standard deviation of the advantage of this batch of samples; Then, the orientation weighting factor for the sample set is constructed: in, The normalized dominance function value. Orientation adjustment factor; sample set orientation weighting factor In the entropy regularization term Introduced as a dynamic coefficient: in, Indicates the state Lower policy distribution entropy, This indicates the current policy at the th... The state obtained by sampling at each time step The expected value of the corresponding trajectory data distribution is calculated, that is, the expected value of the data at time step is calculated. The average value of the batch samples is used to represent the expectation of the entropy regularization term at that time step; Third, at the time level, a dynamic entropy coefficient is set based on the uncertainty of value, and the exploration rhythm is adaptively adjusted according to the training process. By monitoring the TD residuals in each round of training The variance of the current batch of TD residuals is denoted as the empirical variance. First compress it to a range The dimensionless uncertainty is obtained: Then update the entropy regularization coefficient using the following formula: in, Basic entropy weight coefficient , This is a magnification factor used to control the sensitivity of uncertainty to the entropy coefficient. and These are the lower and upper bounds for safety, used to prevent premature convergence due to excessively small entropy weights or divergence during training due to excessively large entropy weights. Step 4: Strategy update and optimization; An improved loss function is formed by employing a sheared PPO objective and combining it with multi-layered exploration terms, which includes probability ratios. Value regression term and entropy regularization term; during the optimization process, an early stopping criterion based on KL divergence is set to suppress excessive policy updates, and an adaptive learning rate optimizer is used to complete parameter iteration; Step 5: Policy training and simulation verification; Policy training and evaluation were performed under parallel sampling settings, and comparative verification was completed in Walker2d-v5 and Humanoid-v5 environments respectively. During training, entropy, total loss and average reward were recorded and monitored using TensorBoard to determine the exploration intensity, policy convergence magnitude and reward improvement trend, thus verifying the stability and training feasibility of the method.

2. The reinforcement learning-based bipedal robot gait control method based on a multi-layer adaptive exploration mechanism according to claim 1, characterized in that, In step 2, the mean branch of the Actor network adopts a four-layer fully connected structure, where the first three layers are feature extraction layers. The activation function for all layers is the hyperbolic tangent function (tanh). The number of nodes in the output layer is consistent with the dimension of the action space, which corresponds to the number of controllable joints of the robot. The output is then averaged using the hyperbolic tangent function (tanh). The data is compressed to the interval [-1, 1]. The variance branch of the Actor network uses the same four-layer fully connected structure, with the first three layers used as feature extraction layers. The activation function is the hyperbolic tangent function tanh. The output layer does not contain an activation function and is used to output the logarithmic standard deviation of each action dimension. The clip function restricts the output to the range [-4, 1]. The loss function of the Actor network is optimized using the policy gradient method and combined with the advantage function to maximize the expected reward. Its loss function is: in, For shearing parameters, The probability ratio between the current policy and the old policy is expressed as: The advantage function is used to measure the current policy in the state. The performance improvement compared to the old strategy is expressed as: The timing difference error is calculated using the following formula: In the state The instant rewards obtained below As a discount factor, As the dominant smoothing factor, This is the state value function.

3. The reinforcement learning-based bipedal robot gait control method based on a multi-layer adaptive exploration mechanism according to claim 2, characterized in that, In step 2, the Critic network adopts the same four-layer fully connected structure as the Actor network, with the hyperbolic tangent function tanh as the activation function. The output layer is a single neuron, used to estimate the state value function for a given state. To obtain the expected long-term return under a given state; The Critic network fits the state value by minimizing the temporal difference error, constructing a mean squared error loss function: in, In the state The instant rewards obtained below Using the discount factor, by minimizing the above Critic network loss function, a smooth estimate of the state value can be obtained, providing a stable value benchmark for the Actor network.

4. The reinforcement learning-based bipedal robot gait control method based on a multi-layer adaptive exploration mechanism according to claim 3, characterized in that, Step 4: The steps for strategy updating and optimization are as follows: (1) Environmental interaction and strategy execution; At each time step The agent, based on the current observation state The mean and variance branches of the Actor network are input to obtain the Gaussian policy distribution for continuous actions: Among them, the mean Log-standard deviation output from the mean network of the Actor Output from variance network and cropped to The standard deviation is then obtained through exponential mapping. This allows for the control of motion noise and exploration intensity; the Actor network generates motion through reparameterized sampling. When an action is applied to the environment, the environment returns to its state at the next moment. and instant rewards The state value function is given by the Critic network. ; (2) Experience storage and trajectory sampling: The states, actions, rewards and end flags generated during the above interaction process are written into the experience buffer in chronological order to form a trajectory set in multiple environments. Buffer capacity by step The data is stored cyclically. Once a full round of sampling is completed, it can be expanded over time and randomly sampled in small batches during subsequent optimization stages to calculate the loss function and update the network parameters. (3) Advantage estimation and target return calculation; after the empirical buffer is filled, the first step is based on the one-step time-series difference error: The generalized dominance estimation method is used to calculate the dominance function estimate for each time step: in As a discount factor, The smoothing coefficient for GAE is used; when the next state is a termination state, a termination flag is used to terminate the accumulation of subsequent discount terms; subsequently, according to... The target return of the value function regression is constructed to measure the fitting error of the Critic network to the long-term return; the calculation process of the above advantage and return is completed by the advantage buffer module in reverse traversal over time in the implementation. (4) Trajectory Unfolding and Mini-Batch Sampling: All environmental trajectories collected in one update cycle are flattened in the time and environment dimensions to obtain a length of... The sample sequence and the corresponding state ,action Log probability of the old strategy Advantage estimation With target return The training dataset is constructed; after randomly shuffling the index of the dataset, it is divided into several small batches according to the preset batch size, and gradient updates are performed several times on each batch to improve sample utilization and reduce estimation variance. (5) Strategy improvement and value assessment; In each mini-batch, the log probability and value estimate of the sample are first recalculated using the current Actor network to obtain the probability ratio between the old and new policies: Construct the cutoff probability ratio objective function: in, To cut the boundary; combine the value function regression loss. Combined with the entropy regularization term obtained based on the multi-layer adaptive exploration mechanism, the comprehensive loss function can be obtained: in, For the value function loss weights, For the current policy in state The entropy below, The sample-level entropy weights are constructed from the advantage estimation. The entropy coefficient, dynamically adjusted by the time layer based on the TD residual variance, realizes the coordinated constraint of the exploration rights of the spatial, directional, and temporal layers. (6) Parameter update and strategy iteration; The above comprehensive loss function is applied to the policy parameters. With the parameters of the state value function The gradient is calculated, and the Adam optimizer is used for backpropagation and gradient pruning. During each update round, the KL divergence between the old and new policies is monitored. When the KL value exceeds a preset threshold, the iteration of the current round is terminated in advance to prevent the policy update step from being too large and causing the policy to collapse. After completing several rounds of alternating "environment interaction-experience storage-advantage estimation-policy evaluation and update", an improved PPO policy that meets the requirements of adaptive exploration can be obtained.

5. The reinforcement learning-based bipedal robot gait control method based on a multi-layer adaptive exploration mechanism according to claim 4, characterized in that, In step 2, the Actor-Critic dual-branch neural network structure adjusts the state input dimension according to the complexity of different control tasks: the input dimension is 17 in the Walker2d-v5 environment and 376 in the Humanoid-v5 environment; the number of neurons in the fully connected layer is 128 in the Walker2d-v5 environment and 512 in the Humanoid-v5 environment.

Images