Robot walking training method combining minimum free energy principle

Abstract

The application provides a robot reinforcement learning walking training method combined with a minimum free energy principle, and comprises the following steps: S1, performing exponential moving average on an error term in time; S2, introducing the exponentially moving averaged error term into an adaptive weight parameter to construct a minimum free energy reward function; and S3, introducing a regularization term into the minimum free energy reward function to obtain a total reward function. Therefore, the robot reinforcement learning walking training method combined with the minimum free energy principle reduces the dependence on artificial parameter adjustment and improves the training efficiency and convergence speed through a unified free energy target reward function and an adaptive weight adjustment mechanism.

Description

Robot reinforcement learning walking training method based on the principle of minimum free energy Technical Field

[0001] This invention relates to the field of robot motion control technology, specifically to a robot reinforcement learning walking training method that incorporates the principle of minimum free energy. Background Technology

[0002] Currently, in the field of humanoid robot motion control, end-to-end walking training methods based on reinforcement learning are commonly used. These methods typically use robot joint angles, IMU (Integrated Mutual Assist Unit) state information as input and robot joint torque or joint position increments as control output, directly generating walking control commands through a policy network. To guide the policy network in learning stable walking behavior, existing methods heavily rely on manually designed reward functions. These reward functions usually consist of multiple independent reward terms, including but not limited to velocity tracking rewards, posture stability rewards, foot lift height rewards, foot contact constraint rewards, energy consumption penalties, and motion smoothness penalties. Each reward term characterizes different local indicators of walking behavior, and its weight parameters are usually manually set and remain fixed during training. However, the above-mentioned technical approach still has the following shortcomings in practical applications:

[0003] 1. The walking behavior of humanoid robots is decomposed into a large number of local reward indicators, and there may be explicit or implicit conflicts between different reward items. For example, it is often difficult to balance speed tracking rewards with energy consumption penalties, and stability rewards with gait flexibility rewards. The number of reward functions is large and the coupling relationship is complex. Existing methods lack a unified optimization objective or theoretical explanation.

[0004] 2. When there are a large number of reward items that are interdependent, the design of each reward weight parameter is difficult and highly dependent on human experience. This typically requires repeated adjustments to the reward weights and multiple resuming of the training state during training, resulting in extremely high parameter tuning costs. Especially at different stages of walk training, the importance requirements for reward items vary significantly. A fixed-weight reward function cannot simultaneously meet the stability requirements of the early training phase and the performance optimization requirements of the later training phase, leading to long training cycles and low convergence efficiency. Summary of the Invention

[0005] This invention is made to solve the above-mentioned problems, and aims to provide a robot reinforcement learning walking training method that combines the principle of minimum free energy.

[0006] This invention provides a robot reinforcement learning walking training method based on the principle of minimum free energy, characterized by the following steps: Step S1, performing an exponential moving average over time on the error term; Step S2, introducing the error term after the exponential moving average into adaptive weight parameters to construct a minimum free energy reward function; Step S3, introducing a regularization term into the minimum free energy reward function to obtain the total reward function.

[0007] The robot reinforcement learning walking training method based on the principle of minimum free energy provided by this invention may also have the following features: the error terms include: joint reference error, velocity tracking error, gait error, and standing error.

[0008] The robot reinforcement learning walking training method based on the principle of minimum free energy provided in this invention may also have the following feature: wherein the formula for joint reference error is:

[0009]

[0010] in, Indicates the robot's first The actual angle of each joint Indicates the robot's first Reference angles for each joint.

[0011] The robot reinforcement learning walking training method based on the principle of minimum free energy provided in this invention may also have the following feature: wherein the formula for velocity tracking error is:

[0012]

[0013] in, This represents the actual linear velocity of the robot's base on the horizontal plane. Indicates the desired horizontal surface linear velocity command;

[0014]

[0015] in, This indicates the actual yaw rate of the robot base. This indicates the desired yaw rate command.

[0016] The robot reinforcement learning walking training method based on the principle of minimum free energy provided in this invention may also have the following feature: wherein the formula for gait error is:

[0017]

[0018] in, Indicates the robot's first The actual contact state of each foot tip Indicates the robot's first Ideal contact state at each foot tip;

[0019]

[0020] in, Indicates the robot's first The expected minimum lift height of the foot during the swing phase. Indicates the robot's first The actual height of each foot relative to the ground.

[0021] The robot reinforcement learning walking training method based on the principle of minimum free energy provided in this invention may also have the following feature: where the formula for the standing error is:

[0022]

[0023] in, This indicates the robot's first standing position. The actual angle of each joint Indicates the robot's first The default reference angle for each joint.

[0024] The robot reinforcement learning walking training method based on the principle of minimum free energy provided in this invention may also have the following feature: wherein the formula for the exponential moving average is:

[0025]

[0026] in, Indicates the error term The smoothing statistic, This represents the EMA update factor.

[0027] The robot reinforcement learning walking training method based on the principle of minimum free energy provided in this invention may also have the following feature: The formula for the regularization term is:

[0028]

[0029] in, Indicates the adaptive weight parameters. These are the prior parameters for accuracy.

[0030] The robot reinforcement learning walking training method based on the principle of minimum free energy provided in this invention may also have the following feature: wherein the calculation formula for the accuracy prior parameter is:

[0031]

[0032] in, Indicates the first The scaling parameter of each error term, This represents a very small positive number to prevent numerical divergence; the formula for calculating the adaptive weight parameters is:

[0033]

[0034] in, Error term The smoothing statistics.

[0035] The robot reinforcement learning walking training method based on the principle of minimum free energy provided in this invention may also have the following feature: the formula for the total reward function is:

[0036]

[0037] in, Indicates the first The free energy value at each time step. Indicates the first The training progress weights of each error term. This represents the adaptive weight of the error term. This indicates that the error term occurs at time step [time step]. Instantaneous error, This represents the adaptive weight of the error term. Relative to its benchmark value The regularization term.

[0038] The role and effect of invention

[0039] The robot reinforcement learning walking training method based on the principle of minimum free energy according to the present invention includes: step S1, performing an exponential moving average over time on the error term; step S2, introducing the error term after exponential moving average into adaptive weight parameters to construct a minimum free energy reward function; and step S3, introducing a regularization term into the minimum free energy reward function to obtain the total reward function. Therefore, the robot reinforcement learning walking training method based on the principle of minimum free energy of the present invention reduces the dependence on manual parameter tuning and improves training efficiency and convergence speed through a unified free energy target reward function and adaptive weight adjustment mechanism. Attached Figure Description

[0040] Figure 1 is a flowchart illustrating the robot reinforcement learning walking training method based on the principle of minimum free energy in an embodiment of the present invention.

[0041] Figure 2 is a schematic diagram of the framework of the robot reinforcement learning walking training method based on the principle of minimum free energy in an embodiment of the present invention. Detailed Implementation

[0042] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the robot reinforcement learning walking training method based on the principle of minimum free energy.

[0043] Example

[0044] Figure 1 is a flowchart illustrating the robot reinforcement learning walking training method based on the principle of minimum free energy in an embodiment of the present invention.

[0045] Figure 2 is a schematic diagram of the framework of the robot reinforcement learning walking training method based on the principle of minimum free energy in an embodiment of the present invention.

[0046] As shown in Figures 1 and 2, this embodiment provides a robot reinforcement learning walking training method based on the principle of minimum free energy, including:

[0047] Step S1: Perform an exponential moving average over time on the error terms. The error terms in the free energy are used to describe the deviation between the robot's actual behavior and its expected behavior. These error terms include: joint reference error, velocity tracking error, gait error, and standing error.

[0048] The formula for the joint reference error is as follows:

[0049]

[0050] in, Indicates the robot's first The actual angle of each joint Indicates the robot's first Reference angles for each joint.

[0051] The formula for speed tracking error is as follows:

[0052]

[0053] in, This represents the actual linear velocity of the robot's base on the horizontal plane. Indicates the desired horizontal surface linear velocity command;

[0054]

[0055] in, This indicates the actual yaw rate of the robot base. This indicates the desired yaw rate command.

[0056] The formula for gait error is:

[0057]

[0058] in, Indicates the robot's first The actual contact state of each foot tip Indicates the robot's first Ideal contact state at each foot tip;

[0059]

[0060] in, Indicates the robot's first The expected minimum lift height of the foot during the swing phase. Indicates the robot's first The actual height of each foot relative to the ground.

[0061] The formula for standing error is:

[0062]

[0063] in, This indicates the robot's first standing position. The actual angle of each joint Indicates the robot's first The default reference angle for each joint.

[0064] Step S2 involves incorporating the error term from the exponential moving average into the adaptive weight parameters to construct the minimum free energy reward function. The formula for the exponential moving average is:

[0065]

[0066] in, Indicates the error term The smoothing statistic, This represents the EMA update factor.

[0067] Step S3: A regularization term is introduced into the minimum free energy reward function to obtain the total reward function. This is to prevent adaptive weight parameters... Deviation from its baseline during training If the value is too large, a regularization term based on Kullback-Leibler divergence is introduced to constrain the offset of the adaptive weight parameters.

[0068] The formula for the regularization term is:

[0069]

[0070] in, Indicates the adaptive weight parameters. These are the prior parameters for accuracy.

[0071] The formula for calculating the prior accuracy parameter is as follows:

[0072]

[0073] in, Indicates the first The scaling parameter of each error term, This represents the smallest positive number to prevent numerical divergence. This prior parameter of accuracy is used to characterize the reasonable constraint strength of the corresponding error term when there is no observational error interference.

[0074] The formula for calculating the adaptive weight parameters is:

[0075]

[0076] in, Error term The smoothing statistic is used to describe the formula. This formula can be viewed as fusing prior information with current error statistics in the precision space, thereby automatically weakening the constraint strength when the error is large and gradually strengthening the constraint strength as the error converges.

[0077] The formula for the total reward function is as follows:

[0078]

[0079] in, Indicates the first The free energy value at each time step. Indicates the first The training progress weights of each error term. This represents the adaptive weight of the error term. This indicates that the error term occurs at time step [time step]. Instantaneous error, This represents the adaptive weight of the error term. Relative to its benchmark value The regularization term.

[0080] Considering that the reward requirement varies at different stages of the humanoid robot's walking, a training progress variable is introduced. The formula for calculating the training progress variable is as follows:

[0081]

[0082] in, This is the preset number of annealing training steps.

[0083] The weights of some error terms can be determined according to To perform a smoothing adjustment, use the following formula:

[0084]

[0085] in, For the first Each error term during training time The effective accuracy parameters, For the first The conservative accuracy parameters of the error term in the early stages of training. For the first The target accuracy parameter for each error term in the later stages of training.

[0086] The minimum free energy value calculated at the current time step Transformed into the reward value required for reinforcement learning via an exponential mapping function. Its form is as follows:

[0087]

[0088] in, Indicates the first The reinforcement learning reward value at each time step. Under this mapping method, the smaller the free energy value, the larger the corresponding reward value, thereby guiding the policy network to optimize in the direction of minimizing the free energy objective.

[0089] In this embodiment, to ensure the numerical stability and convergence of the minimum free energy reward function during the reinforcement learning training process, the values ​​of the key parameters are explained as follows.

[0090] Exponential moving average update coefficient The smoothing statistic used to control the trade-off between historical and current errors in the error term typically ranges from 0.01 to 0.05. Values ​​within this range ensure the smoothness of the error statistic while avoiding an overly slow response to error changes; when… When the value is too large, the error statistic is easily affected by instantaneous noise. If the value is too small, it will cause the weight update to be delayed.

[0091] Scale parameters The physical scale used to characterize different error terms can be set according to the dimensions and allowable deviation range of the corresponding error term. For example, for speed tracking error terms (including base linear velocity error and yaw rate error). Typical values ​​range from 0.1 to 0.3. For joint reference errors and standing errors, which are related to joint angles, Typical values ​​range from 0.2 to 0.5; for positional error terms such as foot lift height, The typical value ranges from 0.02 to 0.1. For discrete error terms such as foot contact state, the scale parameter... The typical value is 0.3 to 0.7.

[0092] Minimal positive number To prevent division by zero or numerical divergence during the calculation of accurate a priori parameters, its typical value is 1×10. -6 .

[0093] Annealing steps related to training progress Used to control the smooth transition of error term weights from the early to the later stages of training, its typical value is 1×10. 5 ~3×10 5 The training time step is set. By setting a training progress variable, the policy network focuses on learning basic walking patterns in the early stage of training, and gradually strengthens the constraints on stability and fine control in the later stage of training, thereby improving the overall training efficiency and convergence stability.

[0094] Adaptive weight parameters It is determined jointly by the prior parameters of accuracy and the error statistics. In specific implementations, it can be determined by... Set reasonable upper and lower limits, with the lower limit being 1×10. -3 The upper limit can be 1×10 2 This is to prevent the weights from being too large or too small under abnormal error conditions, thereby ensuring the numerical stability of the training process.

[0095] Training progress weights This is used to adjust the relative importance of different error terms at different training stages. Its values ​​are typically normalized to the range of 0-1, reflecting the manually assigned importance of each error term at different training stages. In this embodiment, when there is no need for staged scheduling of error term weights, the training progress weight of each error term... It can be set to a constant value of 1, so that each error term maintains a constant weight throughout the training process.

[0096] The above parameter values ​​represent a typical configuration in this embodiment. Those skilled in the art can adjust the parameters within the above range according to the specific robot model, training task, and simulation environment without affecting the implementation of the technical solution of this invention.

[0097] The total reward function in this embodiment also includes a fall penalty, which refers to the punishment given to the robot when it falls or is about to fall in the simulation environment.

[0098] In this embodiment, the training process steps of the robot reinforcement learning walking training method based on the principle of minimum free energy on the robot are as follows:

[0099] Step T1: Before reinforcement learning training begins, initialize the robot simulation environment, policy network parameters, and value network parameters; initialize the robot joint states, base states, and reference motion commands required for training.

[0100] In step T2, at each training time step, the current observation state of the robot is acquired, including the robot's joint angles, joint angular velocities, base posture, base velocity, and other ontological perception information, which is used as input to the policy network.

[0101] In step T3, the policy network outputs action commands based on the current observation state. These action commands are used to control the robot's joint positions or joint torques, driving the robot to perform walking actions in the environment.

[0102] Step T4: Calculate the joint reference error, velocity tracking error, gait error, and standing error based on the robot's actual motion state and desired motion state.

[0103] In step T5, the error terms obtained in step T4 are processed by exponential moving average, and the minimum free energy reward function is calculated by combining the adaptive weight parameters; at the same time, a regularization term is introduced to obtain the total reward function for the current time step.

[0104] Step T6: Based on the total reward function, the policy network parameters are updated using a reinforcement learning optimization algorithm, so that the policy network gradually learns a walking control strategy that can minimize the free energy objective.

[0105] Step T7: Repeat steps T2 to T6 until the training reaches the preset number of iterations or the convergence condition is met, and obtain the trained walking control strategy.

[0106] In this embodiment, the robot reinforcement learning walking training method based on the principle of minimum free energy is performed in a physical simulation environment. This simulation environment is used to simulate the robot's dynamic behavior during walking and the contact process between its feet and the ground, thereby providing the information needed for state feedback and reward calculation during reinforcement learning training.

[0107] The robot model used is a bipedal robot lower limb model with 12 degrees of freedom, each leg containing 6 degrees of freedom, and the joint actuation method is position control. In the simulation environment, state variables such as robot joint position, joint velocity, base posture, base linear velocity and angular velocity are defined to describe the robot's motion state during walking.

[0108] In this embodiment, the input to the policy network is the observation information acquired by the robot based on its proprioception. This observation information includes state variables such as robot joint positions, joint velocities, base pose, base linear velocity, and base angular velocity. The policy network input is constructed using a historical observation stacking method, where observation vectors from multiple consecutive time steps are concatenated in chronological order and used as the policy network input to enhance its ability to express the dynamic characteristics of the system. Specifically, the dimension of a single-frame observation vector is 45, and the number of historical observation stacked frames is set to 66, thus forming a policy network input vector with a dimension of 2970.

[0109] The policy network employs a multi-layer feedforward neural network structure, containing three fully connected hidden layers with 512, 256, and 128 neurons respectively. ReLU non-linear activation functions are used between the layers. The output layer dimension of the policy network corresponds to the number of degrees of freedom of the robot joints and is used to generate control commands for each joint.

[0110] In this embodiment, the value network adopts an asymmetric structure. Its input includes not only the ontology-aware observation information used by the policy network, but also privileged state information obtainable in the simulation environment to improve the accuracy of value estimation. The privileged state information includes the robot's actual base speed, environmental parameters, and auxiliary state information obtainable during training. This privileged state information is only used for value network calculation during the training phase and does not participate in the policy network's input during the policy deployment phase.

[0111] The value network also adopts a multi-layer feedforward neural network structure, which contains three fully connected hidden layers with 768, 256 and 128 neurons in each hidden layer, respectively. The ReLU non-linear activation function is used between each layer, and the output layer is a single neuron used to output the value estimation result corresponding to the current state.

[0112] During training, both the policy network and the value network use gradient descent-based optimization methods for parameter updates. The optimizer employs the Adam optimization algorithm, with a learning rate set to 1×10⁻⁻⁴. 5 In reinforcement learning training, the discount factor γ was set to 0.994, and the generalized advantage estimation parameter λ was set to 0.9. During each policy update, multiple iterations were performed based on the training data sampled from the parallel simulation environment, with two iterations per update. The training samples were divided into four mini-batches for optimization.

[0113] In this embodiment, the control time step of the simulation environment is set to 0.02 seconds, and the maximum duration of each training round is 20-30 seconds. During training, multiple simulation environment instances can be run in parallel simultaneously to improve data sampling efficiency; the number of parallel environments can be set to 512-4096. At each training time step, the simulation environment updates the robot state according to the action instructions output by the policy network and returns the motion state information of the current time step for error term calculation and the construction of the minimum free energy reward function, thereby forming a closed-loop interactive process for reinforcement learning training.

[0114] The role and effect of the embodiments

[0115] According to the robot reinforcement learning walking training method based on the principle of minimum free energy involved in this embodiment, it includes: step S1, performing an exponential moving average over time on the error term; step S2, introducing the error term after exponential moving average into adaptive weight parameters to construct a minimum free energy reward function; and step S3, introducing a regularization term into the minimum free energy reward function to obtain the total reward function. Therefore, the robot reinforcement learning walking training method based on the principle of minimum free energy of this invention reduces the dependence on manual parameter tuning and improves training efficiency and convergence speed through a unified free energy target reward function and adaptive weight adjustment mechanism.

[0116] This embodiment introduces a training progress variable to perform phased smooth scheduling of error term weights. This can reduce the constraint strength of some error terms in the early stage of training, and prioritize guiding the policy network to learn the basic walking pattern. In the later stage of training, the constraint ability of relevant error terms is gradually enhanced to achieve more stable and finer walking control, thereby shortening the overall training cycle and improving the convergence speed.

[0117] This embodiment also introduces a regularization constraint based on Kullback-Leibler divergence to prevent the adaptive weight parameters from deviating too much from their baseline values ​​during training, and to avoid a single error term having a dominant influence on the training objective during the abnormal phase, thereby improving the stability and robustness of the reinforcement learning training process.

[0118] Those skilled in the art should understand that this invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to this invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A robot reinforcement learning walking training method combining the principle of minimum free energy, characterized in that, include: Step S1: Perform an exponential moving average over time on the error term; Step S2: Introduce the error term from the exponential moving average into the adaptive weight parameters to construct the minimum free energy reward function; Step S3: Introduce a regularization term into the minimum free energy reward function to obtain the total reward function; wherein, the formula for the regularization term is: in, Indicates the adaptive weight parameters. Let be the prior parameters for accuracy; where the formula for calculating the prior parameters for accuracy is: in, Indicates the first The scaling parameter of each error term, This represents a very small positive number to prevent numerical divergence; the formula for calculating the adaptive weight parameters is: in, Error term The smoothing statistic; wherein, the formula for the total reward function is: in, Indicates the first The free energy value at each time step. Indicates the first The training progress weights of each error term. This represents the adaptive weight of the error term. This indicates that the error term occurs at time step [time step]. Instantaneous error, This represents the adaptive weight of the error term. Relative to its benchmark value The regularization term.

2. The robot reinforcement learning walking training method based on the principle of minimum free energy as described in claim 1, characterized in that: in, The error terms include: joint reference error, velocity tracking error, gait error, and stance error.

3. The robot reinforcement learning walking training method based on the principle of minimum free energy as described in claim 2, characterized in that: in, The formula for the joint reference error is: in, Indicates the robot's first The actual angle of each joint Indicates the robot's first Reference angles for each joint.

4. The robot reinforcement learning walking training method based on the principle of minimum free energy according to claim 2, characterized in that: wherein, The formula for the speed tracking error is: in, This represents the actual linear velocity of the robot's base on the horizontal plane. Indicates the desired horizontal surface linear velocity command; in, This indicates the actual yaw rate of the robot base. This indicates the desired yaw rate command.

5. The robot reinforcement learning walking training method based on the principle of minimum free energy according to claim 2, characterized in that: wherein, The formula for the gait error is: in, Indicates the robot's first The actual contact state of each foot tip Indicates the robot's first Ideal contact state at each foot tip; in, Indicates the robot's first The expected minimum lift height of the foot during the swing phase. Indicates the robot's first The actual height of each foot relative to the ground.

6. The robot reinforcement learning walking training method based on the principle of minimum free energy according to claim 2, characterized in that: wherein, The formula for the standing error is: in, This indicates the robot's first standing position. The actual angle of each joint Indicates the robot's first The default reference angle for each joint.

7. The robot reinforcement learning walking training method based on the principle of minimum free energy as described in claim 1, characterized in that: in, The formula for the exponential moving average is: in, Indicates the error term The smoothing statistic, This represents the EMA update factor.

Images