A reinforcement learning training method and system for a robot motion control strategy

Abstract

The application discloses a kind of reinforcement learning training methods for robot motion control strategy, comprising: obtaining robot ontology observation, historical observation and privileged observation in simulation training environment;In the teacher stage, the privileged observation is used to train the teacher strategy, so that the privileged encoder generates privileged latent variable capable of representing terrain, contact and disturbance state;In the adaptation stage, load the teacher strategy checkpoint, freeze the action network parameter, only train the history encoder, and through the latent variable alignment, action alignment, value information retention and smoothness constraint, make the history latent variable approximate the privileged latent variable;In the fine-tuning stage, remove the privileged observation, continue reinforcement learning fine-tuning with the ontology observation and the history observation available when the robot is deployed, and obtain deployable motion control strategy.The application can reduce the observation difference between the privileged training strategy and the real deployment strategy, improve the strategy migration stability, action smoothness and deployment consistency.

Description

Technical Field

[0001] This invention belongs to the field of robot motion control technology, specifically a reinforcement learning training method and system for robot motion control strategies. Background Technology

[0002] Robot motion control strategy refers to a strategy network used to output joint, wheel speed, or combined wheel-leg control quantities based on robot sensor observations and velocity, posture, or task commands. When legged, wheeled, and hybrid mobile robots move in complex terrain, they need to simultaneously meet requirements such as velocity tracking, posture stability, contact switching, smooth motion, and realistic deployment feasibility. Reinforcement learning can learn motion control strategies through large-scale interaction in a simulation environment. However, when directly training deployable strategies, the strategy can only utilize the ontological and historical observations provided by the real robot's sensors, making it difficult to fully perceive implicit factors such as terrain, contact states, and external disturbances. This leads to slow training convergence, unstable gait quality, or insufficient capability in complex terrain.

[0003] Current practices typically introduce privileged observations during the simulation training phase, enabling teacher policies to utilize terrain parameters, contact states, or perturbation information that are not directly available during real deployment, thereby learning stronger movement capabilities. However, teacher policies rely on privileged observations and cannot be directly executed on the real robot. If only ordinary distillation methods are used to fit teacher actions to student policies, the student policies tend to learn only the surface form of the actions, failing to retain the environmental perception and value information in the latent variables of the teacher policies. This may lead to problems such as motion jitter, gait degradation, and decreased terrain adaptability during subsequent fine-tuning or real deployment. When continuing to train deployment policies directly from teacher policies, the old optimizer state, learning rate scheduling, and privileged observation distribution may interfere with subsequent training, resulting in the final policy performing well on simulation metrics but lacking stability under real deployment observation conditions.

[0004] Therefore, it is necessary to propose a new reinforcement learning training method to improve the training stability and deployment consistency of robot motion control strategies. Summary of the Invention

[0005] Purpose of the invention: To address the problems existing in the prior art, this invention proposes a reinforcement learning training method and system for robot motion control strategies. Through teacher-guided three-stage reinforcement learning, high-ability learning, information transfer, and deployment consistency are processed separately, forming a reusable, traceable, and scalable training process.

[0006] Technical solution:

[0007] This invention proposes a reinforcement learning training method for robot motion control strategies, comprising:

[0008] Construct a simulation training environment that includes a robot dynamics model to obtain the robot's ontological observations, historical observations, and privileged observations in different scenarios;

[0009] A policy network is provided, which includes a privileged encoder, a history encoder, an action network, and an evaluation network. The privileged encoder is a neural network, and the history encoder is a temporal encoder.

[0010] The proximal policy optimization algorithm is used to train the policy network in the teacher phase, adaptation phase and fine-tuning phase in sequence to obtain a deployable robot motion control policy model.

[0011] The training in the teacher phase includes: activating the privileged encoder, action network, and evaluation network; inputting ontology observations and privileged observations; training the privileged encoder, action network, and evaluation network; and sampling teacher policy checkpoints at preset intervals during the training process until preset stopping conditions are met.

[0012] The training in the adaptation phase includes: activating the historical encoder, loading the teacher policy checkpoint as input, freezing the parameters of the teacher phase privilege encoder, action network and evaluation network, training only the historical encoder, and extracting historical adaptation checkpoints at preset intervals during the training process until the preset stopping condition is met.

[0013] The training in the fine-tuning phase includes: initializing the policy network, activating the historical encoder, action network, and evaluation network, removing the privileged encoder, loading historical adaptation checkpoints as input, fine-tuning the historical encoder and action network until the deployment conditions are met, and outputting the robot motion control policy model.

[0014] Furthermore, the ontology observations are parameters that are input during the actual deployment of the strategy, including joint positions, joint velocities, body angular velocities, body posture projections, speed commands, control actions from the previous moment, wheel speed states, and gait phase information output by the robot dynamics model; the historical observations are a sequence of ontology observations within several consecutive control cycles; and the privileged observations are parameters that are not input during the actual deployment of the strategy.

[0015] Furthermore, the teacher strategy checkpoint is a strategy model file saved in a certain training round during the teacher phase, including privileged encoder parameters, action network parameters, comment network parameters, training rounds, observation dimension, action dimension, and identifier; the historical adaptation checkpoint is a strategy model file saved in a certain training round during the adaptation phase, including historical encoder parameters, action network parameters, comment network parameters, training rounds, observation dimension, action dimension, and identifier.

[0016] Furthermore, the teacher training phase also includes:

[0017] Privileged observations are input into the privileged encoder to generate privileged latent variables. The ontological observations and privileged latent variables are concatenated and input into the action network to generate teacher control actions.

[0018] Design an objective function, evaluate the teacher value estimate of the network executing the teacher's control action based on the objective function value, and use a proximal policy optimization algorithm to update the parameters of the privileged encoder and action network.

[0019] Furthermore, the training in the adaptation phase also includes:

[0020] In a simulation training environment, teacher control actions are executed at teacher policy checkpoints to collect training samples; corresponding historical observations are extracted from the training samples, input into the historical encoder, and historical latent variables are obtained.

[0021] The action network uses the parameters of the privileged encoder and action network in the teacher phase to generate student control actions based on historical latent variables;

[0022] A loss function is designed by combining the teacher value estimation at the teacher stage, the loss function value of executing the student's control action is calculated, and the parameters of the historical encoder are updated by using the proximal policy optimization algorithm.

[0023] Furthermore, the loading of teacher strategy checkpoints as input includes:

[0024] Before loading, all extracted teacher strategy checkpoints are scored, and those that meet the preset scoring criteria are selected as input for the adaptation phase.

[0025] The scoring rules include: for any checkpoint, calculating all evaluation indicators for a certain scenario, normalizing all evaluation indicators to obtain the current scenario score; and taking a weighted average of all scenario scores obtained for that checkpoint to obtain the final score.

[0026] Furthermore, the normalization of all evaluation indicators includes:

[0027] If the evaluation index is an error-related index, including speed error, yaw error, rate of change of maneuver, rate of change of maneuver acceleration, slip rate, and fuselage sway amplitude, the normalization function is a negative correlation function;

[0028] If the evaluation indicators are benefit-related indicators, including step completion rate, front wheel clearance height, phase lock score, diagonal support ratio, and clearance consistency, the normalization function is a positive correlation function.

[0029] Furthermore, the training in the fine-tuning phase also includes:

[0030] Initialize the strategy model, reset the strategy model state during the adaptation phase, and load historical adaptation checkpoints.

[0031] In the simulation training environment, student control actions of historical adaptation checkpoints are executed, training samples are collected, and corresponding historical observation inputs are extracted from the training samples and fed into the historical encoder to obtain historical latent variables.

[0032] By inputting the current ontological observations and historical latent variables into the action network, the motion control strategy and the fine-tuning control action for the next moment are obtained.

[0033] Design a reward function, perform fine-tuning of the reward function value for the control action in a simulation training environment, and use a proximal policy optimization algorithm to update the parameters of the historical encoder and action network until the deployment conditions are met.

[0034] Furthermore, the deployment conditions include:

[0035] The fine-tuning phase training rounds reach the preset maximum training rounds; the sampling ratio of each scenario is not lower than the preset lower limit; the error-related indicators are not higher than the preset upper limit; the benefit-related indicators are not lower than the preset lower limit; the observation dimension and action dimension of the motion control strategy are exported and matched with the observation dimension and action dimension in the robot interface.

[0036] On the other hand, this invention also proposes a reinforcement learning training system for robot motion control strategies, comprising:

[0037] The simulation module is used to build a simulation training environment that includes a robot dynamics model, and to obtain the robot's ontological observations, historical observations, and privileged observations in different scenarios;

[0038] The policy network includes a privileged encoder, a history encoder, an action network, and an evaluation network, wherein the privileged encoder is a neural network and the history encoder is a temporal encoder;

[0039] The phased training module is used to train the policy network sequentially through teacher phase, adaptation phase and fine-tuning phase by using the proximal policy optimization algorithm to obtain a deployable robot motion control policy model.

[0040] The training in the teacher phase includes: activating the privileged encoder, action network, and evaluation network; inputting ontology observations and privileged observations; training the privileged encoder, action network, and evaluation network; and sampling teacher policy checkpoints at preset intervals during the training process until preset stopping conditions are met.

[0041] The training in the adaptation phase includes: activating the historical encoder, loading the teacher policy checkpoint as input, freezing the parameters of the teacher phase privilege encoder, action network and evaluation network, training only the historical encoder, and extracting historical adaptation checkpoints at preset intervals during the training process until the preset stopping condition is met.

[0042] The training in the fine-tuning phase includes: initializing the policy network, activating the historical encoder, action network, and evaluation network, removing the privileged encoder, loading historical adaptation checkpoints as input, fine-tuning the historical encoder and action network until the deployment conditions are met, and outputting the robot motion control policy model.

[0043] Beneficial effects:

[0044] This invention constructs an actor-critic policy network, collects privileged observations, ontology observations, and historical observations for phased training, and finely decouples privileged training, historical observation adaptation, and deployment mode.

[0045] The Teacher phase uses privileged paths, enabling the strategy to fully utilize terrain, contact, and disturbance information in the simulation environment, thereby gaining strong mobility and gait structure in complex terrain.

[0046] In the adaptation phase, the action network is frozen, and the history encoder is trained separately. Privileged latent variables are transferred to historical latent variables, avoiding the direct application of teacher policies dependent on privileged observations to deployment, thus reducing the gap between simulation training capabilities and real deployment observations. Through a combination of latent variable alignment, action alignment, value information preservation, and smoothness constraints, the history encoder not only fits the teacher's actions but also retains the environmental awareness, value-related information, and temporal continuity in the teacher's latent variables, which helps reduce motion jitter and fine-tuning degradation.

[0047] The fine-tuning phase switches to the historical latent variable path and the deployment mode with unprivileged observations. Training continues to ensure that the final policy is consistent with the observation conditions on the robot's end, thereby improving the deployability and traceability of the final policy. Attached Figure Description

[0048] Figure 1 This is a flowchart of the method of the present invention;

[0049] Figure 2 A flowchart for generating privileged implicit variables for the Teacher stage;

[0050] Figure 3 A flowchart for historical latent variable alignment training in the Adapt phase;

[0051] Figure 4 A flowchart for deploying available observations for the Finetune phase. Detailed Implementation

[0052] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments. This embodiment uses a wheeled robot as the controlled object. The wheeled robot is a type of robot with wheels at the ends of its legs, combining the speed of wheeled movement with the flexibility of legged movement. In terms of hardware, the wheeled robot includes a main body, multi-directional rotating joints, wheels providing locomotion, a controller, and corresponding sensors for each part. The motion states of the wheeled robot in this invention include forward movement, lateral movement, turning movement, movement over complex terrain, and transitional movement.

[0053] like Figure 1 The diagram shown is a flowchart of a reinforcement learning training method for robot motion control strategies according to the present invention, which is a teacher-guided three-stage approach. Specifically, it includes the following steps:

[0054] S1: Construct a robot simulation training environment to obtain robot body observations, historical observations, and privileged observations.

[0055] The simulation training environment includes a robot dynamics model, simulated terrain, termination conditions, and randomized parameters. The simulated terrain includes flat ground, rough terrain, steps, or areas with different friction materials. Randomized parameters include robot mass, center of mass, inertia, friction, external disturbances, and execution delay. Control commands are issued to the robot dynamics model to control the robot's forward velocity, lateral velocity, and yaw rate within the simulation training environment.

[0056] Ontology observation refers to observations that can be directly obtained by the robot's own sensors or controllers during actual deployment, including joint positions, joint velocities, body angular velocities, body posture projections, velocity commands, previous control actions, wheel speed states, and gait phase information. Ontology observation does not include ground truth values ​​of terrain height, simulated contact, or simulated internal perturbations that cannot be directly used as policy inputs during actual deployment.

[0057] Historical observations are a sequence of ontological observations spanning multiple consecutive control cycles. Their purpose is to infer terrain, contact, and perturbation trends using temporal context. In this embodiment, the historical window length is 15 control cycles, the total dimension of historical observations is 855, and the observation dimension for each historical step is 57. The historical window length can be set according to the control frequency, sensor latency, and robot dynamics response speed, for example, covering one or more gait phase change cycles.

[0058] Privileged observations are observations that can be directly obtained during simulation training but are not directly input into the strategy during actual deployment. These include at least one of the following: true value of body linear velocity, terrain height scan, reference contact pattern, gait subphase, scene identifier, wheel-ground material parameters, wheel-ground contact force, wheel-ground slippage rate, robot mass ratio, body center of mass offset, and body inertia ratio. These privileged observations can be calculated by the simulator's state interface, terrain generator, and contact sensors. During actual deployment, it is not required for the robot to directly measure all of these privileged observations; instead, the corresponding implicit states are estimated from the ontology's observation history through a historical encoder.

[0059] The motion space of the robot's dynamics model corresponds to the robot's hardware actuators. In this embodiment, for an M20 four-wheeled legged robot, the motion space includes twelve leg joint position targets and four wheel joint velocity targets, totaling sixteen dimensions. The twelve leg joints are fl_hipx_joint, fl_hipy_joint, fl_knee_joint, fr_hipx_joint, fr_hipy_joint, fr_knee_joint, hl_hipx_joint, hl_hipy_joint, hl_knee_joint, hr_hipx_joint, hr_hipy_joint, and hr_knee_joint; the four wheel joints are fl_wheel_joint, fr_wheel_joint, hl_wheel_joint, and hr_wheel_joint. The hip swing joint of the leg uses a motion scaling factor of 0.125, other leg joints use a motion scaling factor of 0.25, the wheel joint speed uses a motion scaling factor of 5.0, and the motion clipping range is set to -100 to 100.

[0060] S2: Construct a policy network and enter the Teacher stage, training teacher policies based on ontology observation and privileged observation.

[0061] First, an actor-critic policy network is constructed, where the actor network is used for policy learning and the critic network is used for value estimation. Structurally, the actor network includes a privileged encoder and a history encoder in addition to its core action network. The history encoder exists as a reserved module during the Teacher phase and is not activated.

[0062] Figure 2 This corresponds to the privileged latent variable generation and teacher control action output process in the Teacher stage. The simulation training environment built in step S1 outputs ontological observations in each control cycle. Privileged observation Privileged observation Input privileged encoder of actor network, output privileged hidden variables , recorded as Action network receives ontology observations privileged hidden variables The concatenated vectors are output, and the teacher's control actions are output. , recorded as The teacher controls the actions applied to the simulated robot to generate the state of the next moment.

[0063] The Teacher phase employs a proximal policy optimization algorithm for training. The objective function of this algorithm includes a policy pruning term, a value loss term, and an entropy regularization term. The objective function is used as the evaluation observation. The critic network receives evaluation observations and outputs value estimates. This represents the estimated cumulative reward based on the discount when executing the current strategy from the current state.

[0064] In this embodiment, the privileged encoder uses a multilayer perceptron with the privileged observation dimension as the input dimension, 128 and 64 hidden layers, and a 16-dimensional privileged latent variable as the output. The action network uses a multilayer perceptron with the concatenation of ontological observations and 16-dimensional latent variables as the input, 512, 256 and 128 hidden layers, and a 16-dimensional control action as the output. The critic network uses a multilayer perceptron with the evaluation observations as the input, 512, 256 and 128 hidden layers, and a 1-dimensional value estimate as the output. The network activation function is ELU.

[0065] Taking a single implementation as an example, each simulation training environment samples twenty-four control steps. Each update uses four mini-batches and several learning epochs. The discount factor γ is 0.99, the generalized advantage estimation parameter λ is 0.95, the pruning parameter is 0.2, and the maximum gradient norm is 1.0. The learning rate for the basic Teacher stage can be set to 3.0 × 10⁻⁶. -4 The entropy coefficient can be set to 0.003, and the target KL divergence can be set to 0.008; in the fast gait stabilization teacher strategy, the learning rate can be set to 2.6 × 10⁻⁶. -4 The entropy coefficient can be set to 0.0045, the target KL divergence can be set to 0.009, and the maximum number of training rounds can be set to 15500. The above values ​​are parameters for the example and can be adjusted according to the robot size and control frequency.

[0066] S3: Select teacher strategy checkpoints based on multi-scenario evaluation indicators.

[0067] During training, multiple teacher policy checkpoints are saved at preset intervals; in this embodiment, the saving interval is 250 rounds. A teacher policy checkpoint refers to a teacher policy model file saved at a specific training round, which includes at least privileged encoder parameters, action network parameters, critic parameters, action distribution parameters, training round, observation dimension, action dimension, and training configuration identifier. After loading the teacher policy checkpoints, the privileged encoder and action network can reproduce the teacher control actions of the corresponding training round given ontology observations and privileged observations.

[0068] Multiple teacher strategy checkpoints were evaluated across various scenarios. This multi-scenario scoring covered all types of movement states of the wheeled robot, preventing the transfer of teacher strategies that performed well in a single scenario but exhibited significant degradation in lateral movement, turning, or transitional actions to later stages. The specific scoring rules are as follows:

[0069] (1) For any checkpoint, let the k-th multi-scenario evaluation index be denoted as . The corresponding weight is The metric is calculated from data recorded in the logs of the simulation training environment.

[0070] In this embodiment, the evaluation indicators selected for different motion states are as follows: Forward motion indicators include planar velocity error, yaw rate error, fuselage sway amplitude, motion change rate, and attitude deviation; Complex terrain motion indicators include step forward speed error, step yaw error, fuselage pitch alignment, front wheel obstacle clearance height, and step completion rate; Lateral motion indicators include lateral velocity tracking error, yaw leakage, forward leakage, lateral phase lock score, and lateral pause ratio; Turning motion indicators include yaw rate tracking error, planar velocity leakage, yaw ground contact proxy, left and right wheel speed asymmetry, and turning phase lock score.

[0071] (2) If For error-related indicators, including speed error, yaw error, rate of change of maneuver, rate of change of maneuver acceleration, slip rate, and fuselage roll amplitude, the following method is used: Normalization function that increases and decreases ;like For revenue-related indicators, including step completion rate, front wheel clearance height, phase lock score, diagonal support ratio, and clearance consistency, a randomized method is used. The normalization function increases with the increase .

[0072] (3) The scene score of the checkpoint is expressed as follows: Calculate the score for all action states. Then, a comprehensive score is calculated and expressed as follows: ,in For scene weights.

[0073] The comprehensive score for each teacher strategy checkpoint in S2 is calculated according to the above rules. Teacher strategy checkpoints that meet the preset scoring criteria are selected as input for the next stage.

[0074] In addition to calculating the overall score, this invention also sets a minimum ability threshold and a maximum risk threshold. The minimum ability threshold ensures that the teacher's strategy has formed a transferable motor skill, while the maximum risk threshold excludes checkpoints with excessively forceful movements, excessive slippage, excessive machine shaking, or unstable transition processes. These minimum ability threshold and maximum risk threshold are hard filtering conditions that must be met before the teacher's strategy checkpoints enter the Adapt stage.

[0075] The minimum capability threshold in this embodiment includes: the course level in the training log corresponding to the teacher checkpoint reaches the preset target level, and the appearance ratio of the corresponding scene in the sampling window is not lower than the preset lower limit. When using a five-level course, the complex terrain course, lateral movement course, and turning course can be required to reach level 5; the sampling ratio of complex terrain, lateral movement, and turning scenes can be required to be not less than 0.18, or can be set to 0.20, 0.22, or 0.24 depending on the task stage.

[0076] The maximum risk thresholds in this embodiment include: for forward motion, the planar velocity error is no higher than 0.30, the yaw error is no higher than 0.24, and the attitude error is no higher than 0.24; for lateral movement, the lateral velocity error is no higher than 0.30, the yaw leakage is no higher than 0.24, the forward leakage is no higher than 0.18, the fuselage sway is no higher than 0.96, the phase error is no higher than 0.36, and the rate of change of motion acceleration is no higher than 2.20; for turning motion, the yaw error is no higher than 0.30, the planar velocity leakage is no higher than 0.18, the fuselage sway is no higher than 0.88, the phase error is no higher than 0.34, and the rate of change of motion acceleration is no higher than 2.10; and for complex terrain movement, the step forward speed error is no higher than 0.30, and the front wheel obstacle clearance height is no less than 0.30.

[0077] In the actual screening process, we first eliminated teacher strategy checkpoints that did not cover the target scenario or were of insufficient course level based on the minimum capability threshold, then eliminated teacher strategy checkpoints that had obvious deployment risks based on the highest risk threshold, and finally compared the comprehensive scores among the remaining teacher strategy checkpoints.

[0078] S4: Enter the Adapt phase, activate and train the historical encoder.

[0079] Load the teacher policy checkpoint selected by S3 and activate the historical encoder in the actor-critic policy network. Freeze the privileged encoder, action network, and critic parameters, and update only the historical encoder parameters. Optionally, construct an auxiliary value probe parameter to constrain historical latent variables to carry information related to long-term feasibility. Figure 3 The corresponding historical latent variable alignment training process in the Adapt phase includes the following steps:

[0080] (1) Perform teacher-controlled actions in a simulation training environment and collect training samples.

[0081] Using the frozen teacher policy, the teacher control action corresponding to the teacher policy checkpoint obtained in S2 is executed through the privileged latent variable path. The control actions executed at this time are collected as training samples, ensuring that the state, command, and terrain distribution of the collected training samples are consistent with the state distribution during the teacher policy execution in the Teacher phase. The training samples are from the simulation training environment rollout, serving as the supervision target for the training and updating of the historical encoder.

[0082] (2) Take historical observations from the training samples, input them into the historical encoder, and output historical hidden variables; the action network calculates student control actions based on historical hidden variables.

[0083] Using the history encoder Historical observations obtained from step S1 Generate historical latent variables , recorded as When the historical observations are time series, the historical encoder employs a one-dimensional convolutional structure. In this embodiment, the historical observations for fifteen control cycles are organized into a tensor of 15 steps multiplied by 57 dimensions per step. This tensor first passes through a one-dimensional convolutional layer with 128 channels and then a one-dimensional convolutional layer with 64 channels. It is then flattened and input into a multilayer perceptron with 256 and 128 hidden layers, outputting 16-dimensional historical latent variables. Alternatively, if one-dimensional convolution is not used, the historical observations can be directly concatenated and input into the multilayer perceptron to output historical latent variables.

[0084] (3) Design a loss function, and the trainer updates the historical encoder based on the latent variables.

[0085] When updating the historical encoder, the same batch of training samples is used to compute four types of supervision signals:

[0086] First, privileged latent variables are obtained using the privileged encoder. Historical latent variables are obtained using a historical encoder. Through the square norm 2 Constrain the two to be close:

[0087]

[0088] Second, by inputting ontological observations and historical latent variables into the action network, student control actions are obtained. By inputting ontological observations and privileged latent variables into the action network, teacher-controlled actions can be obtained. ,pass Consistent constraints on actions.

[0089] Third, use critics to convey the value of teachers. Set the training period auxiliary value probe P, so that and All of these are close to the value of teachers, thus constituting a loss in the retention of value information. .

[0090] Fourth, use historical latent variables at adjacent time steps. Constrain time smoothness.

[0091] Combining the above four types of supervision signals, the total loss function is expressed as follows: In this embodiment, It is 1.0. It is 0.5 or 0.45. It is 0.25. The learning rate is set to 0.05 or 0.06; the learning rate for the Adapt phase is set to 1.0 × 10⁻³, and for the fast gait adaptation phase it is set to 7.5 × 10⁻⁴. Twenty-four steps are sampled per simulation training environment, and each update uses four mini-batches and five learning epochs. The hidden layers of the auxiliary value probe are 128 and 64. The auxiliary value probe only participates in training during the Adapt phase to constrain historical latent variables to carry information related to long-term rewards, and is not part of the final robot-side inference path.

[0092] After the Adapt phase, a historical adaptation checkpoint is generated. This checkpoint retains the action network parameters, privileged encoder parameters, critic network parameters, and post-trained historical encoder parameters of the teacher's policy. Simultaneously, it deletes or resets the optimizer state and records `checkpoint_kind` as the adaptation phase identifier. The purpose of deleting or resetting the optimizer state is to ensure that the Finetune phase does not inherit the old momentum and learning rate schedules from the Teacher or Adapt phases, thereby re-establishing the gradient update state in a deployed manner.

[0093] S5: Enter the Finetune phase, remove privileged observation inputs, and fine-tune the deployment pattern.

[0094] The initialization strategy for historical adaptation checkpoints generated from the S4 Adapt phase sets the input to the action network as ontology observations and historical latent variables, and no longer provides privileged observations to the action network; that is, the deployment path is switched to... , Then, the process enters the Finetune stage. In this stage, the history encoder and the action network continue to refine the process through reinforcement learning, ensuring that the policy maintains velocity tracking, pose stability, transition stability, and smooth motion under real-world observation conditions.

[0095] Figure 4 The diagram illustrates the fine-tuning deployment process during the Finetune phase. Specifically, after loading historical adaptation checkpoints, privileged observation inputs are disabled; historical observations are fed into the historical encoder to obtain historical latent variables, and the current ontology observation and historical latent variables are fed into the action network to output control actions; the simulation environment updates the state based on the control actions and returns the reward function. The training path during the Finetune phase is consistent with the actual robot's inference path, and the reward function is only used for simulation training updates, without requiring additional privileged observation inputs during actual deployment.

[0096] In this embodiment, the Finetune phase reward function comprises: first, a speed tracking reward, used to constrain forward speed, lateral speed, and yaw rate tracking commands; second, an attitude stability reward, used to constrain fuselage pitch, roll, base height, and fuselage sway; third, a motion smoothing reward, used to constrain the rate of change of motion and the rate of change of motion acceleration in adjacent control cycles; fourth, a wheel-to-ground slip suppression reward, used to constrain the slip of the supporting foot relative to the ground, wheel speed error, and wheel-to-ground slip ratio; fifth, a complex terrain reward, used to constrain step forward error, front wheel obstacle clearance height, fuselage pitch alignment, and step completion rate; and sixth, a transition stability reward, used to constrain the rate of change of transition motion acceleration, the number of transition stability steps, and the residual yaw at the stop when transitioning from a stop to a start, from a general motion to a lateral shift or turn, and from a lateral shift or turn back to a general motion.

[0097] For lateral movement and turning scenarios, the Finetune stage can also introduce micro-gait rewards as a refinement of the above reward function. The micro-gait rewards include at least one of the following: lateral velocity continuity reward, yaw angular velocity continuity reward, directional leakage penalty, planar velocity leakage penalty, diagonal support reward, phase lock reward, yaw wheel contact penalty, contact lock reward, phase hold reward, pause ratio penalty, left and right wheel speed coordination reward, and yaw rubbing penalty.

[0098] The Finetune phase still uses the proximal policy optimization algorithm for training, but the learning rate or exploration intensity is set lower than in the Teacher phase to reduce disruption to the learned gait structure. Taking the training process of a fast gait as an example, the maximum number of training epochs in the Finetune phase is 6000, the save interval is 100, the old optimizer state is not loaded when loading historical adaptation checkpoints, the learning rate is 1.5 × 10^-4, the entropy coefficient is 0.0020, the target KL divergence is 0.0040, and the number of learning epochs is 4. For single-gait links that emphasize deployment consistency, the maximum number of training epochs in the Finetune phase can be 7000, the learning rate can be 1.0 × 10^-4, the entropy coefficient can be 0.0015, and the target KL divergence can be 0.0035.

[0099] The completion conditions for the Finetune phase include: reaching the preset maximum number of training rounds; under unprivileged observation input conditions, the sampling ratio of forward, lateral, turning, and complex terrain scenarios is not lower than the preset lower limit; speed tracking error, body sway, motion change rate, motion jerk, wheel speed error, and slippage proxy quantity are not higher than the preset upper limit; phase lock score, obstacle clearance consistency, or step completion rate are not lower than the preset lower limit; and the observation dimension and motion dimension are matched in the robot's inference interface after policy export. If multiple checkpoints meet the conditions, the checkpoint with the lowest overall risk and the most stable deployment indicators is selected according to the multi-scenario scoring described in S3.

[0100] S6: After all stages of training are completed, the history encoder and action network of the policy network are exported separately as reasoning models for the robot.

[0101] In one deployable embodiment, the final export adopts a two-stage inference structure: the history encoder file `adaptation.onnx` receives 855-dimensional historical observations and outputs 16-dimensional historical latent variables; the action network file `policy.onnx` receives 75-dimensional inputs formed by concatenating 59-dimensional current ontology observations and 16-dimensional historical latent variables, and outputs 16-dimensional control actions. The exported metadata records the following dimensions: action network input observation dimension 75, value network input observation dimension 251, control action dimension 16, current ontology observation dimension 59, deployment phase privileged observation dimension 0, history observation dimension 855, latent variable dimension 16, history observation time steps 15, action network hidden layer dimensions 512, 256, 128, value network hidden layer dimensions 512, 256, 128, history encoder hidden layer dimensions 256, 128, and the history encoder type is a one-dimensional convolutional encoder.

[0102] To illustrate that the present invention is applicable to fast gait training, in an embodiment of fast gait policy generation, the Teacher phase uses an RMA policy architecture with privileged observation dimension 217, historical observation dimension 855, latent variable dimension 16, and action dimension 16; the historical observation window is 15 steps; the Teacher phase is trained in a mixed scenario of forward movement, step movement, lateral movement, and turning; the Adapt phase trains the historical encoder using latent variable alignment, action alignment, value information preservation, and smoothness constraints; the Finetune phase continues training using unprivileged observations and derives a two-stage model.

[0103] The simulation training environment logs metrics such as the percentage of lateral movement scenarios, the percentage of turning scenarios, lateral movement error, turning error, phase lock score, and step completion rate. For example, near a checkpoint in a single export, the percentage of lateral movement scenarios is approximately 0.2721, the percentage of turning scenarios is approximately 0.2853, the lateral movement error is approximately 0.0985, the lateral movement phase lock score is approximately 0.5023, the turning yaw error is approximately 0.1425, the turning phase lock score is approximately 0.5570, and the step completion rate is approximately 0.7543. These values ​​are used to illustrate the recordability and evaluation methods of the embodiment; specific thresholds can be set by the inventor based on the robot model, terrain, and testing requirements.

[0104] Example 2

[0105] The present invention also proposes a reinforcement learning training system for motion control strategies of wheeled robots. The system stores a computer program, and running the program completes all the steps of the method described in Example 1.

[0106] Specifically, the reinforcement learning training system includes: a simulation environment construction module for building an actor-critic policy network and a simulation training environment, outputting ontology observations, historical observations, and privileged observations; a teacher training module for training the privileged encoder, action network, and critic network of the policy network, generating teacher policy checkpoints; a checkpoint selection module for reading logs and model files, selecting teacher policy checkpoints based on multi-scenario metrics; an adapt training module for freezing the teacher policy and training the historical encoder of the policy network; a fine-tuning module for continuing to fine-tune the policy network for reinforcement learning using unprivileged observations; and a policy export module for outputting action network files, historical encoder files, and metadata.

[0107] Without departing from the core idea of ​​this invention, those skilled in the art can replace the privileged encoder with other neural network structures, replace the historical encoder with a recurrent network, a Transformer encoder or other temporal encoders, replace the reinforcement learning algorithm with other actor-critic algorithms, or adjust the motion space and observation space according to the robot hardware.

[0108] This invention stages privileged training, history adaptation, and deployment fine-tuning, clarifying the source of privileged observations, the training method for historical latent variables, the generation methods for teacher policy checkpoints and history adaptation checkpoints, the meaning of preserving value information, and the reasoning path that does not use privileged observations during final deployment. Those skilled in the art can adaptively adjust the observation dimensions, network size, and evaluation thresholds according to the specific robot platform.

Claims

1. A reinforcement learning training method for robot motion control strategies, characterized in that, include: Construct a simulation training environment that includes a robot dynamics model to obtain the robot's ontological observations, historical observations, and privileged observations in different scenarios; A policy network is provided, which includes a privileged encoder, a history encoder, an action network, and an evaluation network. The privileged encoder is a neural network, and the history encoder is a temporal encoder. The proximal policy optimization algorithm is used to train the policy network in the teacher phase, adaptation phase and fine-tuning phase in sequence to obtain a deployable robot motion control policy model. The training in the teacher phase includes: activating the privileged encoder, action network, and evaluation network; inputting ontology observations and privileged observations; training the privileged encoder, action network, and evaluation network; and sampling teacher policy checkpoints at preset intervals during the training process until preset stopping conditions are met. The training in the adaptation phase includes: activating the historical encoder, loading the teacher policy checkpoint as input, freezing the parameters of the teacher phase privilege encoder, action network and evaluation network, training only the historical encoder, and extracting historical adaptation checkpoints at preset intervals during the training process until the preset stopping condition is met. The training in the fine-tuning phase includes: initializing the policy network, activating the historical encoder, action network, and evaluation network, removing the privileged encoder, loading historical adaptation checkpoints as input, fine-tuning the historical encoder and action network until the deployment conditions are met, and outputting the robot motion control policy model.

2. The reinforcement learning training method according to claim 1, characterized in that, The ontology observations are parameters that are input during the actual deployment of the strategy, including joint positions, joint velocities, body angular velocities, body posture projections, speed commands, control actions from the previous moment, wheel speed states, and gait phase information output by the robot dynamics model; the history observations are a sequence of ontology observations within several consecutive control cycles; the privileged observations are parameters that are not input during the actual deployment of the strategy.

3. The reinforcement learning training method according to claim 2, characterized in that, The teacher strategy checkpoint is a strategy model file saved in a certain training round during the teacher phase, including privileged encoder parameters, action network parameters, comment network parameters, training rounds, observation dimension, action dimension, and labels; the historical adaptation checkpoint is a strategy model file saved in a certain training round during the adaptation phase, including historical encoder parameters, action network parameters, comment network parameters, training rounds, observation dimension, action dimension, and labels.

4. The reinforcement learning training method according to claim 3, characterized in that, The teacher training phase also includes: Privileged observations are input into the privileged encoder to generate privileged latent variables. The ontological observations and privileged latent variables are concatenated and input into the action network to generate teacher control actions. Design an objective function, evaluate the teacher value estimate of the network executing the teacher's control action based on the objective function value, and use a proximal policy optimization algorithm to update the parameters of the privileged encoder and action network.

5. The reinforcement learning training method according to claim 4, characterized in that, The training during the adaptation phase also includes: In a simulation training environment, teacher control actions are executed at teacher policy checkpoints to collect training samples; corresponding historical observations are extracted from the training samples, input into the historical encoder, and historical latent variables are obtained. The action network uses the parameters of the privileged encoder and action network in the teacher phase to generate student control actions based on historical latent variables; A loss function is designed by combining the teacher value estimation at the teacher stage, the loss function value of executing the student's control action is calculated, and the parameters of the historical encoder are updated by using the proximal policy optimization algorithm.

6. The reinforcement learning training method according to claim 5, characterized in that, The loading of teacher strategy checkpoints as input includes: Before loading, all extracted teacher strategy checkpoints are scored, and those that meet the preset scoring criteria are selected as input for the adaptation phase. The scoring rules include: for any checkpoint, calculating all evaluation indicators for a certain scenario, normalizing all evaluation indicators to obtain the current scenario score; and taking a weighted average of all scenario scores obtained for that checkpoint to obtain the final score.

7. The reinforcement learning training method according to claim 6, characterized in that, The normalized evaluation metrics include: If the evaluation index is an error-related index, including speed error, yaw error, rate of change of maneuver, rate of change of maneuver acceleration, slip rate, and fuselage sway amplitude, the normalization function is a negative correlation function; If the evaluation indicators are benefit-related indicators, including step completion rate, front wheel clearance height, phase lock score, diagonal support ratio, and clearance consistency, the normalization function is a positive correlation function.

8. The reinforcement learning training method according to claim 7, characterized in that, The training during the fine-tuning phase also includes: Initialize the strategy model, reset the strategy model state during the adaptation phase, and load historical adaptation checkpoints. In the simulation training environment, student control actions of historical adaptation checkpoints are executed, training samples are collected, and corresponding historical observation inputs are extracted from the training samples and fed into the historical encoder to obtain historical latent variables. By inputting the current ontological observations and historical latent variables into the action network, the motion control strategy and the fine-tuning control action for the next moment are obtained. Design a reward function, perform fine-tuning of the reward function value for the control action in a simulation training environment, and use a proximal policy optimization algorithm to update the parameters of the historical encoder and action network until the deployment conditions are met.

9. The reinforcement learning training method according to claim 8, characterized in that, The deployment conditions include: The fine-tuning phase training rounds reach the preset maximum training rounds; the sampling ratio of each scenario is not lower than the preset lower limit; the error-related indicators are not higher than the preset upper limit; the benefit-related indicators are not lower than the preset lower limit; the observation dimension and action dimension of the motion control strategy are exported and matched with the observation dimension and action dimension in the robot interface.

10. A reinforcement learning training system for robot motion control strategies, characterized in that, include: The simulation module is used to build a simulation training environment that includes a robot dynamics model, and to obtain the robot's ontological observations, historical observations, and privileged observations in different scenarios; The policy network includes a privileged encoder, a history encoder, an action network, and an evaluation network, wherein the privileged encoder is a neural network and the history encoder is a temporal encoder; The phased training module is used to train the policy network sequentially through teacher phase, adaptation phase and fine-tuning phase by using the proximal policy optimization algorithm to obtain a deployable robot motion control policy model. The training in the teacher phase includes: activating the privileged encoder, action network, and evaluation network; inputting ontology observations and privileged observations; training the privileged encoder, action network, and evaluation network; and sampling teacher policy checkpoints at preset intervals during the training process until preset stopping conditions are met. The training in the adaptation phase includes: activating the historical encoder, loading the teacher policy checkpoint as input, freezing the parameters of the teacher phase privilege encoder, action network and evaluation network, training only the historical encoder, and extracting historical adaptation checkpoints at preset intervals during the training process until the preset stopping condition is met. The training in the fine-tuning phase includes: initializing the policy network, activating the historical encoder, action network, and evaluation network, removing the privileged encoder, loading historical adaptation checkpoints as input, fine-tuning the historical encoder and action network until the deployment conditions are met, and outputting the robot motion control policy model.

Images