A quadruped robot reinforcement learning control method for simulation-to-reality migration

Abstract

This invention discloses a reinforcement learning control method for quadruped robots oriented towards simulation-to-real-world transfer, belonging to the field of intelligent control technology. This invention constructs a time-series observation using a historical sliding window, employing only non-privileged state information from the robot's own sensors as input to the policy network; it dynamically adjusts the complexity of the simulated terrain based on the robot's horizontal walking ability assessment results, injecting seven types of multi-source uncertainties into the parallel simulation environment; it calculates discounted cumulative reward and advantage estimation using state transition data from the parallel environment, iteratively updating the policy and value network parameters through deep reinforcement learning; during the deployment phase, it loads the convergence network to generate control commands, while simultaneously recording operational data for subsequent optimization. This invention integrates non-privileged observation modeling, curriculum learning, and multi-source perturbation injection, significantly improving the stability, robustness, and simulation-to-real-world transfer performance of quadruped robots in non-ideal environments, reducing deployment costs, and enhancing long-term operational feasibility.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control technology, and in particular relates to a reinforcement learning control method for quadruped robots that is oriented towards simulation-to-real-world transfer. Background Technology

[0002] Current research on robot control algorithms is gradually shifting from traditional model-based control methods to data-driven intelligent control methods. Traditional methods, such as PID control and model predictive control (MPC), demonstrate good stability and real-time performance in scenarios with simple structures and well-defined system models. However, their limitations become increasingly apparent when facing robot tasks with high-dimensional state spaces, nonlinear dynamics, and complex environmental interactions. For example, traditional methods rely on high-precision modeling and fine-tuning, making it difficult to adapt to dynamic terrain changes, multi-source external disturbances, and policy generalization requirements, resulting in unstable control performance and poor deployment adaptability.

[0003] In contrast, Deep Reinforcement Learning (DRL) learns control strategies through interactions between the agent and its environment. It can autonomously explore optimal action outputs for complex tasks without relying on explicit system modeling. DRL demonstrates strong flexibility and generalization capabilities in robot control problems such as terrain adaptation, balance control, and multi-objective optimization. It guides policy optimization through reward function design, supports simultaneous optimization of multiple metrics including speed, stability, and energy consumption, and possesses the ability to continuously learn and self-adjust in disturbed environments. Therefore, reinforcement learning has become an important research direction in the field of intelligent robot control.

[0004] However, most current reinforcement learning control strategies are still trained based on idealized simulation environments, making it difficult to cover various uncertainties in real-world deployments, such as sensor errors, execution delays, and terrain changes. The training sample distribution is often too homogeneous, making the strategies prone to overfitting specific observation states. They also lack stability when faced with non-ideal initial conditions or external disturbances, resulting in simulation training effects that are difficult to reproduce on real robots, thus limiting their practical application potential.

[0005] Therefore, there is an urgent need for a reinforcement learning control method with systematic perturbation modeling capabilities, non-privileged observation constraint mechanisms, and deployable inference structures to address the shortcomings of existing methods in policy generalization, control robustness, and practical deployment, and to promote the practical deployment of deep reinforcement learning in quadruped robots in complex terrain. Especially for control scenarios such as quadruped robots standing up, stabilizing, and adjusting at low speeds in complex terrain, due to rapid changes in body posture, complex and variable contact states between the feet and different ground materials, and significant fluctuations in instantaneous joint output, existing methods often struggle to simultaneously consider the available observation conditions of the body, execution link perturbations, and changes in contact states. This leads to control policies trained in simulations easily exhibiting problems such as instability, jittery movements, or a significant decrease in control effectiveness after foot slippage on physical platforms. Summary of the Invention

[0006] The purpose of this invention is to provide a reinforcement learning control method for quadruped robots that is adapted from simulation to real-world applications. This method addresses the shortcomings of existing methods in terms of policy generalization, control robustness, and real-world deployment. In particular, it addresses the issues of decreased standing stability, deteriorated motion continuity, and unstable control performance in quadruped robots under complex terrain and non-ideal disturbance conditions. This improves the practical deployment performance of deep reinforcement learning in quadruped robots in complex terrain.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is a reinforcement learning control method for quadruped robots oriented towards simulation-to-real-world transfer, comprising the following steps:

[0008] S1. Obtain non-privileged observation information output by the quadruped robot's body sensors. Cache and stitch the non-privileged observations, including the body linear velocity, body angular velocity, joint position, joint velocity, and joint control commands from the previous control cycle, according to a preset historical sliding window to form a time-series state input, which is used to characterize the dynamic state changes of the quadruped robot during posture adjustment, foot contact changes, and continuous control processes.

[0009] S2, In the simulation environment, the horizontal walking ability assessment results are calculated based on the displacement of the quadruped robot body on the horizontal plane within the preset evaluation time window, and the terrain complexity level of the simulation environment is adjusted accordingly to generate course terrain parameters, so that the quadruped robot can gradually adapt to contact changes and body disturbances under different terrain conditions during the training process.

[0010] S3. In the simulation environment, multi-source disturbances are synchronously injected according to preset random rules to construct a multi-source uncertainty model covering control links, dynamic parameters and external disturbances. The multi-source disturbances correspond to common control lag, joint output fluctuation, foot slippage, ground contact changes and external impacts in the actual operation of quadruped robots.

[0011] S4. Construct multiple parallel simulation environments. In each parallel simulation environment, use the time series state input, the course terrain parameters, and the multi-source disturbances to perform interactive sampling, collect state and control-related data, and use a deep reinforcement learning algorithm to iteratively update the policy network based on the data to obtain a policy network for quadruped robot control.

[0012] S5. During the actual operation of the quadruped robot, a policy network that has been trained and converged is loaded. The non-privileged time-series state input, consistent with that in the training phase, is used as the network input to generate joint control commands. At the same time, the non-privileged observation information, control commands, and key feedback data during the operation are recorded.

[0013] Furthermore, in step S1, the single-frame observation dimension of the historical sliding window is 45, the number of historical observation frames is 10, and the total dimension of the time series state input is 450; at each time step, the current observation vector is obtained and inserted at the beginning and end of the historical sliding window, and the original observation data is shifted to the right in chronological order to form the observation time series. :

[0014] ;

[0015] in, Represents the set of real numbers. express 3D real vector space, Indicates all A vector space composed of 3D real matrices; The dimension of the observation vector. The length of the sliding window. For time step index, For time step The observation vector.

[0016] Furthermore, in step S1, the non-privileged observation information comes only from the inertial measurement unit and joint encoder of the quadruped robot, and does not include global information such as terrain height map, relative pose, and external positioning, to ensure the consistency of observation information between the training phase and the actual deployment phase.

[0017] Furthermore, in step S2, the terrain complexity level is a discrete level variable with a preset level upper limit of 5, and the initial training terrain is flat land corresponding to level 0. When the horizontal mobility assessment result is higher than the preset increase threshold and the current terrain complexity level is lower than the level upper limit, the terrain complexity level is increased by one level. When the horizontal mobility assessment result is lower than the preset decrease threshold and the current terrain complexity level is greater than 0, the terrain complexity level is decreased by one level. In other cases, the level remains unchanged.

[0018] Furthermore, in step S2, the course terrain parameters include terrain height undulation, obstacle placement density, and obstacle height range. Each parallel simulation environment is assigned an independent terrain complexity level, and the corresponding course terrain parameters are updated according to its own level.

[0019] Furthermore, in step S3, the multi-source disturbances include motion delay disturbances, torque disturbances, leg slippage disturbances, friction coefficient disturbances, mass disturbances, initial state disturbances, and external thrust disturbances. The disturbance parameters of each parallel simulation environment are independently and randomly sampled to simulate non-ideal operating conditions such as control delay, actuator output error, foot slippage, ground friction changes, load changes, initial attitude deviations, and external impacts.

[0020] Furthermore, the motion delay perturbation reuses the previous action with a probability of 0.33 to replace the current strategy output action; the torque perturbation introduces a perturbation coefficient uniformly distributed in [-0.06, 0.06] to correct the execution torque; the leg slippage perturbation randomly selects a single leg with a probability of 0.01 and applies a scaling factor of 0.1 to its joint torque; the friction coefficient perturbation is a random value uniformly distributed in [0.5, 1.25]; and the mass perturbation is a perturbation amount uniformly distributed in [-1.0kg, 1.0kg] superimposed on the robot's body mass.

[0021] Furthermore, in step S4, 256 parallel simulation environments are constructed, each of which interacts independently with the current policy network. State transition tuples containing state, action, reward, next state, and termination flag are collected and formed. The state transition tuples are stored in the experience buffer pool. Based on the data in the experience buffer pool, an instantaneous reward function is constructed, which consists of linear velocity tracking, angular velocity control, attitude offset suppression, joint acceleration constraint, action change rate constraint, and non-foot collision penalty weighting.

[0022] Furthermore, in step S4, the deep reinforcement learning algorithm is a policy gradient-based algorithm. It calculates the discounted cumulative reward and the generalized advantage estimate based on the immediate reward function, using the discounted cumulative reward as the learning objective of the value network and the generalized advantage estimate for updating the parameters of the policy network; the learning rates of the policy network and the value network are set to... The reward discount factor is set to 0.99, the generalized advantage estimation coefficient is set to 0.95, the policy shearing threshold is set to 0.2, and the entropy regularization coefficient is set to 0.01. Iterative updates are performed until the maximum number of training iterations of 1000 or the policy network converges.

[0023] Furthermore, in step S5, after loading the converged policy network, gradient calculation is turned off and the network is switched to inference mode. The collected actual operation non-privileged observation information is processed and normalized in the same way as in the training phase to construct time series state input. The generated joint control commands are target joint position, target joint velocity, target torque or a combination thereof, which are sent to the quadruped robot drive layer to form closed-loop motion control. The recorded operation data is written to a non-volatile storage medium for subsequent failure mode analysis, performance evaluation or policy network retraining.

[0024] Compared with existing technologies, the beneficial effects of this invention are as follows: By constructing an observation input mechanism with a historical sliding window structure, this invention can fully utilize the time-series information of the robot's sensors, enhancing the modeling ability of dynamic behavior. Simultaneously, the use of a non-privileged observation structure avoids the inconsistency between training and deployment observations, improving the transferability of the strategy in real-world environments. By introducing a course learning mechanism based on horizontal movement capability assessment, the terrain complexity can be dynamically adjusted from easy to difficult during training, gradually expanding the state space distribution encountered by the strategy. This allows the trained strategy to adapt to more diverse terrain conditions, significantly improving the strategy's generalization and terrain adaptability. By synchronously injecting seven types of disturbances—action delay, torque disturbance, leg slippage, friction coefficient change, mass disturbance, initial state disturbance, and external thrust disturbance—into the simulation environment, non-ideal operating scenarios in actual deployment can be comprehensively simulated without increasing the risk of real robot testing. This covers the full-link uncertainty of control links, physical parameters, and environmental disturbances, effectively enhancing the robustness and anti-interference ability of the strategy in real deployments. By driving multiple parallel simulation environments to interact synchronously and independently, the efficiency of collecting diverse samples can be accelerated, shortening the convergence cycle of reinforcement learning training. Simultaneously, the parallel sampling mechanism ensures the independence and diversity of samples, preventing the policy from getting trapped in local optima and improving the stability and sample utilization of the training process. By deploying and recording operations to record the observed state, control actions, and feedback data in real time, traceability of the control process can be achieved without adding extra sensors. This provides a data foundation for offline analysis and iterative optimization of policy failure modes, significantly improving deployment diagnostic efficiency and the ability to continuously improve the policy. Especially for control scenarios with high requirements for posture continuity, such as quadruped robot recovery, stable standing, and low-speed adjustments in complex terrain, this invention can improve the control stability and practical usability of the policy on a physical platform without relying on external positioning information. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is an overall flowchart of the reinforcement learning control method for quadruped robots oriented towards simulation-to-real-world transfer, as described in this invention.

[0027] Figure 2 This is a schematic diagram of the observation construction and non-privileged sliding window structure of the present invention;

[0028] Figure 3 This is a schematic diagram of the course terrain adjustment mechanism of the present invention;

[0029] Figure 4 This is a schematic diagram of the multi-source disturbance modeling and injection structure of the present invention;

[0030] Figure 5 This is a flowchart of the parallel simulation and reinforcement learning training process of this invention;

[0031] Figure 6 This is a flowchart illustrating the deployment control and data recording process of the present invention;

[0032] Figure 7 This is a graph showing the return curves of the parallel simulation training phase under different ablation settings of the present invention.

[0033] Figure 8 This is a statistical chart showing the success rate of the fall-recovery task during the actual deployment phase of this invention under different ablation settings. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] This embodiment provides a reinforcement learning control method for quadruped robots that is adapted from simulation to reality. Specifically, it is a reinforcement learning control method and system for quadruped robots that is adapted from simulation to reality. It solves the problems of traditional control architectures, such as difficulty in building stable strategies, poor adaptability to multi-source disturbances, and difficulty in stable deployment in real-world environments. This enables quadruped robots to still have adaptive perception and robust motion control capabilities under non-ideal conditions.

[0036] In some specific implementation methods, such as Figure 1 Taking the self-developed Mini quadruped robot as an example, the proposed reinforcement learning control method for quadruped robots, oriented towards simulation-to-real-world transfer, is used to achieve rapid recovery and stationary standing after a fall, i.e., both the desired linear velocity and the desired angular velocity are zero. In other task scenarios, by adjusting the desired linear velocity and desired angular velocity parameters, this method can also be used for motion control tasks such as walking on flat ground and running on complex terrain. In this embodiment, the control method first completes policy training in the IsaacGym simulation environment, and then deploys the converged policy network on the physical robot to achieve transfer control from simulation to reality.

[0037] S1. Observation Construction and Input Operation: A time-series state input with a historical sliding window structure is constructed, using only non-privileged observations from the quadruped robot's sensors as input to the policy network. Specifically, in each control cycle, the robot's linear velocity, angular velocity, joint positions, joint velocities, and joint control commands from the previous control cycle are collected. The current observation vector is inserted at the beginning of the sliding window and the existing data is shifted to the right in chronological order to form a time-series state input of length K (used to characterize the quadruped robot's dynamic state changes during posture adjustment, foot contact changes, and continuous control processes). This enhances the modeling capability of the robot's dynamic behavior and avoids introducing external privileged information unavailable during the deployment phase.

[0038] S2. Course Terrain Adjustment Operation: A course terrain mechanism is introduced to dynamically adjust the terrain complexity level based on the quadruped robot's horizontal movement capability assessment results in simulated terrain. The robot's displacement on the horizontal plane is accumulated within a preset assessment time window to obtain the horizontal movement capability assessment result; the terrain complexity level is set as a discrete level variable. And assign independent [resources] to each parallel simulation environment When the evaluation result is higher than the preset increase threshold and If the result is less than the maximum level, it will be raised by one level; if the evaluation result is lower than the preset decrease threshold, it will be lowered by one level. If it is greater than the minimum level, reduce it by one level; and according to the updated Select or generate corresponding course terrain parameters, including at least one of terrain height undulation, obstacle placement density, and obstacle height range, so as to realize progressive terrain training from easy to difficult, enabling the quadruped robot to gradually adapt to contact changes and body disturbances under different terrain conditions during the training process.

[0039] S3. Multi-source disturbance modeling and injection: Multiple types of disturbances (multi-source disturbances) are synchronously injected into the simulation environment to construct a multi-source uncertainty model covering the control link, dynamic parameters, and external disturbances. These disturbances include at least seven types: motion delay disturbances, torque disturbances, leg slippage disturbances, friction coefficient disturbances, mass disturbances, initial state disturbances, and external thrust disturbances. The disturbance parameters for each parallel simulation environment are sampled independently. By simulating non-ideal factors such as control delay, actuator output error, foot slippage, changes in ground friction, load changes, initial attitude deviations, and external impacts, the strategy is optimized under rich disturbance conditions during training, thereby improving its robustness and anti-interference capability in real-world deployment.

[0040] S4. Parallel Simulation and Policy Training Operation: Multiple parallel simulation environments are constructed. Under given course terrain parameters and perturbation configurations, each parallel environment interacts independently with the current policy network, collecting time-series state inputs, control commands, and environmental feedback to form state transition data. Based on the state transition data, an instantaneous evaluation signal for evaluating control performance is constructed, and a learning signal for measuring long-term benefits is constructed accordingly. A deep reinforcement learning algorithm based on policy gradients is used to iteratively update the parameters of the policy network and the value estimation network to obtain a quadruped robot control policy network for suppressing attitude deviation, reducing motion jitter, and improving control stability under abnormal contact conditions. In one embodiment, the instantaneous evaluation signal can be composed of weighted sub-items such as linear velocity tracking, angular velocity control, attitude maintenance, motion smoothing, and collision suppression. By synchronously sampling and batch updating in multiple parallel simulation environments, the training convergence time is shortened and the generalization ability of the policy under complex terrain and multiple perturbation conditions is improved.

[0041] S5. Deployment Control and Data Recording: During the actual operation of the quadruped robot, the converged training policy network is loaded. Using the same non-privileged time-series state input as during the training phase, and without relying on external positioning systems or high-precision environmental sensors, control commands for each joint are generated and sent to the robot body to achieve closed-loop motion control of the quadruped robot in the actual environment. In this embodiment, the control method supports multiple motion tasks, including walking, running, fall recovery, and static holding. Simultaneously, non-privileged observations, control commands, and key feedback data can be recorded during deployment for subsequent failure mode analysis and policy network retraining.

[0042] In one embodiment, the control system of the present invention includes a processor and a memory. The memory stores a computer program for executing the above steps S1 to S5. When the processor executes the computer program, it implements each step of the above control method. The storage medium may be a non-volatile storage medium such as a solid-state drive, used to store disturbance configuration parameters, course level thresholds, and log data recorded during the deployment phase.

[0043] The above process will be explained in detail below through examples.

[0044] Example

[0045] This embodiment provides a control method for a quadruped robot (taking a Mini robotic dog as an example) based on the aforementioned control framework, used to achieve rapid standing and maintaining a stationary state after a fall. The embodiment is trained in the IssacGym simulation environment and deployed on a physical robot, forming a complete simulation-to-real-world transfer closed loop. In this scenario, the quadruped robot exhibits rapid posture changes, frequent switching of foot contact relationships, and significant instantaneous load fluctuations in the joints during the standing-up and stabilization process. If only a single-frame observation input or idealized training conditions are used, the trained strategy is unlikely to achieve a stable and usable control effect on the physical robot.

[0046] S1, such as Figure 2 The initial setup for the quadruped robot's control task is performed, including defining the dimensions of the observation space and motion space, constructing a sliding window observation structure, and continuously acquiring multiple sensor data frames to form a time-series state input, thereby enhancing the modeling capability of the quadruped robot's dynamic behavior. A non-privileged observation input method is adopted, using only the body's linear velocity, angular velocity, joint position, joint velocity, and historical movements from onboard sensors such as the robot's inertial measurement unit and joint encoders, ensuring consistency between observations during the training and actual deployment phases. This input method can preserve the continuous state changes of the quadruped robot during its standing, adjustment, and stabilization processes, which is beneficial for improving the strategy's adaptability to posture changes and joint response processes.

[0047] Among them, such as Figure 1 Step S1 may include the following sub-steps: S101 to S103.

[0048] S101: Define the state space and action space parameters for reinforcement learning training. The single-frame observation dimension is... Historical observation frame count This forms the total observed input dimension. The action space dimension is set to The maximum number of rounds is Initialization includes observation buffer, reward buffer, and environment reset flag buffer, which are used to support subsequent parallel sampling and training processes.

[0049] S102: At each time step Get the current observation vector And insert the first and second elements of the sliding window to form the observation time series:

[0050]

[0051] in, Represents the set of real numbers. express 3D real vector space, Indicates all A vector space composed of 3D real matrices; The dimension of the observation vector. The length of the sliding window. For time step index, For time step The observation vector is used. By shifting old observations to the right and updating the window, temporal correlation modeling is achieved, improving the policy's ability to perceive dynamic state changes. For the quadruped robot control task in this embodiment, this method helps to reflect the continuity of posture evolution and joint state changes during the recovery and stabilization process.

[0052] S103: Construct a non-privileged observation input structure, relying solely on state information obtained from the robot's onboard sensors:

[0053] Specifically, in one embodiment of the present invention, the aforementioned state information includes the linear velocity of the body. angular velocity Joint position Joint velocity Historical actions This structure contains no global information from terrain height maps, relative poses, external positioning, etc., ensuring consistency of observation information between the training and deployment phases.

[0054] S2, This implementation method introduces, as follows Figure 3 The course terrain mechanism automatically adjusts the terrain level based on the quadruped robot's forward displacement during training, allowing the strategy to gradually adapt from flat ground to more complex and rugged terrain, thereby improving its generalized control capability under different terrain conditions. Before each training round, the quadruped robot in all simulation environments undergoes state initialization and course evaluation, resetting the observation cache and environmental variables to ensure state consistency and sample independence during parallel training. This course design allows the strategy to gradually adapt to more complex terrain contact conditions after forming basic control capabilities, helping to avoid unstable control effects due to excessive terrain changes in the early stages of training.

[0055] Step S2 may include the following sub-steps: S201 to S202.

[0056] S201: Set the maximum level for terrain courses The initial training terrain is set to flat ground, corresponding to the course level of [level missing]. During training, the robot's forward horizontal displacement distance is statistically analyzed within an evaluation interval of a preset time window. and based on Update the course levels. Course Levels Take discrete values ​​and satisfy Its update strategy is as follows:

[0057]

[0058] in, Indicates at time step The terrain course level at that time This indicates the course level for the next time step, updated based on the displacement statistics within the evaluation window. This represents the forward displacement threshold used to upgrade the course level; This represents the forward displacement threshold used to reduce the course level, and is typically set to no greater than [amount missing]. To establish a stable level switching mechanism.

[0059] The curriculum mechanism is used to gradually increase the complexity of the training terrain, thereby improving the generalization ability and adaptability of the strategy in complex scenarios.

[0060] S202: Before the start of each training round, perform the following initialization procedure for all parallel simulation environments: obtain the current state of the environment. Used for course assessment and reset determination; updating course level variables. Clear the sliding window cache and complete the construction of the starting state for the current training round.

[0061] S3, such as Figure 4 To enhance the robustness and anti-interference capability of quadruped robots in real-world deployment environments, this implementation introduces a multi-source perturbation mechanism during training to simulate common anomalies such as control link delays, joint execution errors, local failures, terrain friction changes, load uncertainties, initial state disturbances, and periodic external force interventions. These perturbations are independently injected into multiple simulation environments, enabling the strategy to possess stronger generalization ability and control stability in actual deployment. Without modeling these factors, the trained strategy is prone to discontinuous movements, increased body posture fluctuations, or unstable recovery processes when encountering execution delays, foot slippage, or external disturbances on the physical robot.

[0062] Step S3 may include the following sub-steps: S301 to S307.

[0063] S301: Based on probability Apply an action delay disturbance, and reuse the action from the previous moment with a certain probability in the current control cycle. Replace the current policy output Otherwise, the current policy output will be executed. This mechanism is used to simulate the communication or execution delays of quadruped robots in actual deployments, improving the adaptability of the policy under control lag conditions.

[0064]

[0065] Indicates at time The motion vector actually executed by the robot after motion delay perturbation; For the time The generated random variable, and ; This represents a continuous uniform distribution on the interval from 0 to 1, therefore This indicates that the random event is true, and its probability of occurrence is... .

[0066] In one embodiment of the present invention, This represents the action vector output by the policy network at the current moment. This represents the cached action from the previous moment, with a latency probability. This means that a delayed perturbation is applied approximately every 3 steps.

[0067] S302: Output execution torque Introducing the actuator output uncertainty disturbance coefficient during the process This makes the actual torque applied. satisfy It is used to simulate the physical uncertainties of the actuator and the motor response error during the output phase.

[0068] In one embodiment of the present invention, Indicates time step The original joint torque output, This indicates the actual torque output of the actuator joint after the perturbation is applied. Indicates time step The random perturbation variable obtained from sampling, Indicates time step The relative error coefficient of the actuator torque obtained from sampling is a dimensionless variable, and is taken as... ,in Indicates in Uniform sampling is performed within the specified range; the above-mentioned value range in this embodiment can improve the output fluctuation, motor response error and control stability of the target quadruped robot joint actuator in the face of the control link.

[0069] S303: Based on probability (Approximately every 100 steps) Randomly select one leg and apply a scaling factor to all its joint torques. The perturbation is constructed as follows:

[0070]

[0071] This mechanism is used to simulate situations where a robot experiences local execution anomalies such as slippage or failure of a single leg under complex terrain or unstable contact conditions.

[0072] In one embodiment of the present invention, Indicates the first leg of the quadruped robot Control torque of each joint; weakening coefficient The probability of the disturbance occurring is This indicates that the quadruped robot's legs are slipping; This is the joint torque output after the leg slips.

[0073] S304: The coefficient of friction between the foot and the ground in the foot contact model of the simulator. Apply random perturbation, so that This is used as an environmental parameter in the calculation of foot contact force to simulate the impact of friction variations on foot contact stability under different surface material conditions. In one embodiment of the invention, the friction coefficient is set at the beginning of each training round. Sampling is performed and the results remain unchanged throughout the round. Follows uniform distribution ,in Indicates in Uniform sampling is performed within the range. The value range of this implementation can cover the foot contact conditions of different ground materials in the forward working conditions, so that the strategy can learn stable foot landing and anti-slip control under different contact conditions.

[0074] S305: Robot body mass parameters in the simulator's dynamic model Inject a disturbance, and let the mass parameter after the disturbance be... and will The mass parameters used in this training round are incorporated into the dynamics calculation to simulate the impact of load changes on the quadruped robot's dynamic characteristics, thereby enhancing load robustness in actual deployment. In one embodiment of the invention, The mass of the quadruped robot body. The disturbance term is uniform and satisfies ,in Indicates in The disturbance quantity corresponding to the actual load change level obtained by continuous and uniform sampling within the range.

[0075] S306: Set three types of perturbations for the robot's initial state: position, attitude, and velocity. Among them, the position perturbation... An attitude perturbation is applied to the robot's first two coordinates (X, Y). Angular disturbances acting on the roll, pitch, and yaw axes; velocity disturbances It acts on linear velocity and angular velocity.

[0076] In one embodiment of the invention, the initial position perturbation attitude disturbance velocity disturbance .

[0077] S307. Apply periodic external disturbance signals to the robot to simulate environmental shocks or control interference during actual operation. Specifically, set to: linear velocity disturbance. angular velocity disturbance The three-dimensional linear velocity and angular velocity components are applied to the root of the robot, and a perturbation update is performed once in each preset period to simulate external force impact or inertial change, thereby improving the strategy's response capability and robustness to sudden disturbances.

[0078] S4. For the motion control task of quadruped robots in complex terrain and disturbed environments, a reinforcement learning method based on policy gradient is adopted to iteratively optimize the policy network and value network. First, key training hyperparameters, including learning rate, discount factor, GAE coefficient, shearing threshold, and entropy regularization coefficient, are set to control the training convergence speed and policy update magnitude. During training, multiple parallel simulation environments for quadruped robots are constructed to synchronously sample data, collecting state, action, reward, next state, and termination flag data, forming state transition samples and writing them into the experience buffer. After reaching a predetermined sampling length, the advantage function and discount cumulative reward are calculated for each trajectory segment, serving as supervision signals for policy learning. Subsequently, a mini-batch sampling strategy is adopted to construct a policy loss function including shearing probability ratio and entropy regularization term, and a value loss function in the form of mean squared error. Gradient backpropagation is then performed to update the policy parameters and value parameters respectively. During training, experience collection and network updates are continuously alternated until the loss converges or the maximum number of training rounds is reached. Finally, the converged policy parameters and value parameters are output for control inference in the actual deployment stage of the quadruped robot, ensuring the policy has stability, generalization ability, and practical control adaptability. Therefore, the strategy training in this embodiment can not only improve the rewards in the simulation environment, but also improve the posture stability and motion continuity of the quadruped robot under physical deployment conditions.

[0079] Among them, such as Figure 5 Step S4 may include the following sub-steps: S401 to S408.

[0080] Before calculating the policy gradient, the processor preferentially executes a reward processing sub-step to construct an instant reward function structure composed of multiple sub-items. By applying rewards or penalties to behavioral features such as linear velocity tracking, angular velocity control, posture shift suppression, joint acceleration constraints, smoothness of motion changes, and non-foot collisions, the policy is guided to achieve the desired action goal during continuous control. Each sub-reward item is scaled according to its physical meaning and relative importance and subjected to pruning to suppress the interference of abnormal observations or errors on the training process. All sub-rewards are nonlinearly processed and linearly weighted to form a total reward value, which serves as the basic feedback signal for policy optimization. During training, the system constructs an advantage function and a discounted reward estimate based on the instant rewards, which serve as the update targets for the policy network and value network, respectively, to improve the gait stability, motion accuracy, and control robustness of the quadruped robot under complex terrain and disturbance conditions.

[0081] S401. Construct an instant reward function structure consisting of multiple sub-items, wherein the instant reward... It consists of six sub-reward items It is composed of linear weighted combinations and is used to comprehensively characterize the overall performance of the robot in the control process.

[0082] Linear velocity tracking reward The linear velocity of the robot's root in the local coordinate system With expected speed The Euclidean distance between them is calculated using exponential decay to encourage the robot to move along the target direction;

[0083] Angular velocity tracking reward Using the Z-axis angular velocity as the core feedback quantity, calculate the actual and expected angular velocities around the Z-axis. and The difference is processed by an exponential function and then weighted to penalize the angular velocity deviation, and a limiting term is introduced to avoid excessive rotation.

[0084] Attitude deviation penalty The sum of squares of the projections of the gravity vector in the body coordinate system onto the horizontal plane (x, y axes) is used as a metric, and the degree of posture deviation is measured using an exponential decay function, thereby guiding the robot to maintain an upright posture.

[0085] Joint acceleration penalty The joint acceleration is obtained based on the difference between the current joint velocity and the previous time step, and a squared penalty is applied to suppress movement jumps.

[0086] Action change rate penalty For two consecutive actions , The squared difference term is penalized to constrain policy output changes and improve control smoothness.

[0087] Collision Penalty : Detect the contact force or contact object between the non-footed parts of a quadruped robot and the ground, apply penalties, and prevent falls or abnormal contact behaviors.

[0088] S402: Apply scaling factors to each sub-item and perform pruning to suppress the impact of outlier errors on training stability; obtain the total reward by linear weighting after non-linear processing. This serves as the basis for the advantage function and return estimation.

[0089] Specifically, in one embodiment of the present invention

[0090] Linear velocity tracking reward ;

[0091] Angular velocity tracking reward ;

[0092] Attitude deviation penalty ;

[0093] Joint acceleration penalty ;

[0094] Action change rate penalty ;

[0095] Collision Penalty ;

[0096] Total Rewards

[0097] Specifically, in one embodiment of the present invention, This represents the actual linear velocity of the root of the quadruped robot in the local coordinate system. Its expected target speed; and These represent the current and target angular velocities of the aircraft around the Z-axis, respectively. This represents a truncation function used to restrict input values ​​to a preset range. Indicates calculation first Then truncate the value to the interval. Inside, that is, when When -1 is taken, Take 0 in the first case, and take 0 in the other cases. itself. , These represent the components of the gravity vector in the X and Y directions of the body coordinate system, respectively; and They represent the first The joint velocities of each joint at the current time step and the previous time step. The time interval between adjacent time steps. The number of contact points used to detect non-foot-end collisions; , These represent the actions output by the strategy at the current time step and the previous time step, respectively. This represents the output torque of the j-th joint; This represents the contact force between the robot's non-footed parts and the ground, where i is the index of the non-footed collision contact point; This is a logical indicator function, which is 1 when the robot's non-foot part collides with the ground, and 0 otherwise; The threshold value is 0.1N.

[0098] , , , , , , , These are the scaling factors for each sub-reward item. Specifically, in one embodiment of the present invention, , , , , , , =-1.0.

[0099] S403: Based on total reward Calculate the discounted reward and advantage estimate at each time step, and use the advantage estimate to update the policy network parameters. Fit the discount return to the value network parameters. The learning objectives are thus achieved, thereby enabling goal-oriented strategy optimization.

[0100] S404. Set the core hyperparameters for training the quadruped robot for policy and value networks.

[0101] In one embodiment of the present invention, , , , , .in, , These are the learning rates for the policy network and the value network, respectively. As a reward discount factor; The coefficients are the generalized advantage estimate (GAE) coefficients. This is the policy shearing threshold, used to limit the update magnitude; This is the entropy regularization coefficient, used to encourage exploratory strategies. The batch sample size is [size missing] for each network update. The process is divided into several mini-batches for gradient updates.

[0102] S405, Construction The quadruped robots interact synchronously in a parallel simulation environment, with each environment operating independently until a preset number of interaction steps are accumulated. The robot control system generates actions and interacts with the environment based on the policy function, collects interaction data from the simulator, constructs state transition tuples, and stores them in a buffer pool.

[0103] In one embodiment of the invention, a quadruped robot is constructed. Each of the parallel simulation environments executes a parallel simulation environment. At each time step, use the strategy function. Perform sampling and generate state transition tuples. ,in For state, For action, As a reward, For the next state, This is the round termination flag. All state transition tuples are written to the experience buffer. This serves as the source of training data for subsequent optimization.

[0104] S406, the quadruped robot interacts with the environment, and after reaching the predetermined time step, it moves to the buffer pool. Each trajectory segment in the data is processed, and its time step is calculated. Corresponding cumulative discount With advantage function .

[0105] In one embodiment of the present invention, the method for calculating the cumulative return of the discount and the advantage function is as follows:

[0106]

[0107]

[0108] in, =0.99, =0.95 The above quantities are respectively used as the regression targets of the value function. With strategy optimization direction It is used to supervise network training.

[0109] S407: Use a random sampler to extract data from the trajectory buffer. Multiple extractions of scale Mini-batch data is used for policy network processing. and value network The loss function is calculated, and the policy parameters are obtained through gradient backpropagation. and value parameters Update.

[0110] In one embodiment of the invention, scale Define the policy loss function With value loss function :

[0111]

[0112]

[0113] in, Representational Policy Network The set of network parameters, Value network The set of network parameters, To truncate the threshold, Indicates will Limited to Inside, that is, less than Time to take greater than Time to take The rest take the original value. The weight of the entropy regularization term. This represents the empirical expectation of the set of sampling time steps. For the estimation of the advantage function, For return estimates, The probability ratio of the strategies. This is the policy entropy term.

[0114] After calculating the loss function, the policy parameters are updated using gradient backpropagation. and value parameters :

[0115] ,

[0116] S408: Continuously execute the sampling process (S405) and the network update process (S406) (S407), forming an iterative optimization loop until the maximum number of training iterations or the loss convergence condition is reached. Finally, output the converged policy parameters. With value parameters This serves as the input to the control model during the deployment phase.

[0117] In one embodiment of the present invention, the training termination condition is:

[0118]

[0119] The final output is the optimized policy parameters. With value parameters It is used to deploy stage controller calls to achieve highly robust motion generation for the robot.

[0120] S5. After policy training is complete, the converged policy parameters are exported and loaded during the deployment phase for real-time action inference and closed-loop control. During the deployment phase, the processor disables all gradient calculations and sets the policy network to inference mode to ensure determinism and computational efficiency in the inference process. The observation input structure received during the deployment phase is consistent with that of the training phase. The observations are constructed from a sliding window to form time-series observations and only contain non-privileged information sources available during the training phase, thereby ensuring consistency between the training and deployment structures and policy generalization performance.

[0121] Among them, such as Figure 6 As shown, step S5 may include the following sub-steps: S501 to S509.

[0122] S501, Deployment phase begins. The processor loads the convergence strategy network parameters from the storage medium or an externally exported file. Turn off gradient calculation and switch the policy network and its related inference components to inference mode to establish an inference and deployment process consistent with the control cycle.

[0123] S502. Acquire observation information from the body sensors. The processor acquires body sensor data of the quadruped robot in each control cycle. The body sensor data includes at least the output of the inertial measurement unit and the output of the joint encoder.

[0124] S503. Obtain the state variables required for the strategy input. The processor calculates or processes the linear velocity of the body based on the observation information from the body sensors. Angular velocity of the body Joint position Joint velocity and the action at the previous moment The above quantities are used as the basic elements for constructing time series observations.

[0125] S504. Update the sliding window and construct time series observations. The processor will control the current cycle. , , , , Write the data into a sliding window and stitch them together according to the preset window length and time order to form a time series observation. And guarantee The dimensions, dimensional processing, and normalization methods are consistent with those in the training phase.

[0126] S505, Execute policy inference. The processor will analyze the time series observations. The input policy reasoning module calls the policy network to perform forward reasoning and outputs continuous action signals. The continuous motion signals are used to characterize the target control quantities of each joint or control dimension of the quadruped robot.

[0127] S506 generates and issues joint control commands. The processor bases these commands on continuous motion signals. Generate joint control commands, which may be target joint position, target joint velocity, target torque, or a combination thereof, and send the joint control commands to the lower-level controller or drive layer for execution.

[0128] S507. The quadruped robot performs motion and generates feedback. The quadruped robot executes motion control processes according to joint control commands, generating new body observation states and contact information. These new body observation states and contact information serve as the input source for the next control cycle, thereby forming a closed-loop control cycle in the deployment phase.

[0129] S508. Determine whether to record runtime data. The processor determines whether to enable the runtime data recording process based on a preset switch; if the determination result is not to record, skip data recording and enter the next control cycle, repeating S502 to S507.

[0130] S509. Record the running data and write it to the storage medium. When the judgment result is to record, the processor calls the data recording module to record the information of the robot's sensor body and write the recorded data to the storage medium for subsequent failure analysis, performance evaluation or retraining data construction; after the recording is completed, the next control cycle is entered, and S502 to S507 are repeated until the deployment task is completed.

[0131] To verify the technical effectiveness of the "reinforcement learning control framework for quadruped robots oriented towards simulation-to-real-world transfer," this embodiment designs an ablation comparison experiment: In the IsaacGym simulation environment, under unified training and deployment conditions, the performance differences of the complete framework and the framework after removing key modules are tested respectively, and its transfer capability on quadruped robots is further verified. Specifically, four experimental configurations are set up:

[0132] (1) Standard configuration: includes historical sliding window observation structure, terrain course module and multi-source disturbance mechanism;

[0133] (2) Remove the sliding window: only retain the ontology observation at the current moment, and do not use the historical sliding window structure;

[0134] (3) Remove terrain courses: train on fixed medium-difficulty terrain, without dynamically increasing the terrain complexity according to the robot's movement ability;

[0135] (4) Removal of multiple disturbances: Multiple sources of disturbances, such as motion delays, torque disturbances, leg slippage, friction coefficient changes, mass disturbances, initial state disturbances, and external thrust interference, are no longer injected into the simulation environment. Except for the ablated modules, the remaining network structure, hyperparameters, and training process remain consistent with the framework of this invention to ensure the fairness and comparability of the comparison results. In this embodiment, all cases are trained and deployed using the same Mini quadruped robot model under the same environment.

[0136] Figure 7 The training phase reward curves and their 95% confidence intervals are shown under different ablation settings. Figure 7 It can be seen that, under the standard configuration, after a brief initial fluctuation, the training reward quickly improves and stabilizes at a high level, with small curve fluctuations, exhibiting the best convergence speed and stability. Removing the sliding window results in a significant drop in reward in the early stages of training. Although it gradually recovers to near zero, the final convergence reward is significantly lower than the standard configuration and exhibits greater fluctuations. Removing terrain lessons results in rewards remaining in the negative range for a long period during training, indicating that without guidance from easy to difficult terrain lessons, the robot struggles to learn effective fall recovery strategies on complex terrains. Removing multiple perturbations results in training rewards that fall between the standard configuration and the configuration without terrain lessons, which is better than the case without lessons but still significantly lower than the standard configuration. Figure 7 It is evident that historical sliding window observation, topographic curriculum, and multi-source perturbation mechanism all contribute to improving the convergence speed and stability of policy learning.

[0137] After completing the simulation training, the strategies obtained from the four configurations were deployed to the actual Mini quadruped robot platform. The fall-recovery task was triggered multiple times under the same field conditions, and the task success rate was statistically analyzed. The results are as follows: Figure 8 As shown. By Figure 8 It can be seen that the standard configuration achieves a fall recovery success rate of approximately 98%, significantly higher than the three comparison configurations with the sliding window removed (approximately 58%), terrain lessons removed (approximately 3%), and multiple perturbations removed (approximately 52%). Specifically, when the terrain lessons are removed, the robot is almost unable to achieve effective fall recovery; when the sliding window or multiple perturbations are removed, the success rate decreases by approximately 40%–46% compared to the standard configuration, indicating that the historical sliding window structure and multi-source perturbation mechanism play a crucial role in improving robustness during actual deployment. Figure 7 and Figure 8 It can be confirmed that the synergistic effect of the historical sliding window observation structure, terrain course module and multi-source perturbation mechanism proposed in this invention can significantly improve the learning performance of the quadruped robot's fall recovery strategy in the simulation training stage and its transfer effect in the real environment.

[0138] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0139] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A reinforcement learning control method for quadruped robots oriented towards simulation-to-real-world transfer, characterized in that, Includes the following steps: S1. Obtain non-privileged observation information output by the quadruped robot's body sensors. Cache and stitch together the non-privileged observations, including the body linear velocity, body angular velocity, joint position, joint velocity, and joint control commands from the previous control cycle, according to a preset historical sliding window to form a time-series state input. S2, In the simulation environment, the horizontal movement capability assessment results are calculated based on the displacement of the quadruped robot body on the horizontal plane within the preset evaluation time window, and the terrain complexity level of the simulation environment is adjusted accordingly to generate course terrain parameters. S3, In the simulation environment, multi-source disturbances are synchronously injected according to preset random rules to construct a multi-source uncertainty model covering control links, dynamic parameters and external disturbances; S4. Construct multiple parallel simulation environments. In each parallel simulation environment, use the time series state input, the course terrain parameters, and the multi-source disturbances to perform interactive sampling, collect state and control-related data, and use a deep reinforcement learning algorithm to iteratively update the policy network based on the data to obtain a policy network for quadruped robot control. S5. During the actual operation of the quadruped robot, a policy network that has been trained and converged is loaded. The non-privileged time-series state input, consistent with that in the training phase, is used as the network input to generate joint control commands. At the same time, the non-privileged observation information, control commands, and key feedback data during the operation are recorded.

2. The reinforcement learning control method for quadruped robots oriented towards simulation-to-real-world transfer, as described in claim 1, is characterized in that... In step S1, the single-frame observation dimension of the historical sliding window is 45, the number of historical observation frames is 10, and the total dimension of the time series state input is 450. At each time step, the current observation vector is obtained and inserted at the beginning and end of the historical sliding window, and the original observation data is shifted to the right in chronological order to form the observation time series. : ； in, Represents the set of real numbers. express 3D real vector space, Indicates all A vector space composed of 3D real matrices; The dimension of the observation vector. The length of the sliding window. For time step index, For time steps The observation vector.

3. The reinforcement learning control method for quadruped robots oriented towards simulation-to-real-world transfer, as described in claim 1, is characterized in that... In step S1, the non-privileged observation information comes only from the inertial measurement unit and joint encoder of the quadruped robot, and does not include global information such as terrain height map, relative pose, and external positioning, so as to ensure the consistency of observation information between the training stage and the actual deployment stage.

4. The reinforcement learning control method for quadruped robots oriented towards simulation-to-real-world transfer, as described in claim 1, is characterized in that... In step S2, the terrain complexity level is a discrete level variable with a preset level upper limit of 5. The initial training terrain is flat land, corresponding to level 0. When the horizontal mobility assessment result is higher than the preset increase threshold and the current terrain complexity level is lower than the level upper limit, the terrain complexity level is increased by one level. When the horizontal mobility assessment result is lower than the preset decrease threshold and the current terrain complexity level is greater than 0, the terrain complexity level is decreased by one level. In other cases, the level remains unchanged.

5. The reinforcement learning control method for quadruped robots oriented towards simulation-to-real-world transfer, as described in claim 1, is characterized in that... In step S2, the course terrain parameters include terrain height undulation, obstacle placement density, and obstacle height range. Each parallel simulation environment is assigned an independent terrain complexity level and updates the corresponding course terrain parameters according to its own level.

6. The reinforcement learning control method for quadruped robots oriented towards simulation-to-real-world transfer as described in claim 1, characterized in that, In step S3, the multi-source disturbances include motion delay disturbances, torque disturbances, leg slippage disturbances, friction coefficient disturbances, mass disturbances, initial state disturbances, and external thrust disturbances. The disturbance parameters of each parallel simulation environment are independently and randomly sampled to simulate non-ideal operating conditions such as control delay, actuator output error, foot slippage, ground friction change, load change, initial attitude deviation, and external impact.

7. The reinforcement learning control method for quadruped robots oriented towards simulation-to-real-world transfer as described in claim 6, characterized in that, The motion delay perturbation reuses the previous action with a probability of 0.33 to replace the current strategy output action; the torque perturbation introduces a perturbation coefficient uniformly distributed in [-0.06, 0.06] to correct the execution torque; the leg slippage perturbation randomly selects a single leg with a probability of 0.01 and applies a scaling factor of 0.1 to its joint torque; the friction coefficient perturbation is a random value uniformly distributed in [0.5, 1.25]; and the mass perturbation is a perturbation amount uniformly distributed in [-1.0kg, 1.0kg] superimposed on the robot's body mass.

8. The reinforcement learning control method for quadruped robots oriented towards simulation-to-real-world transfer as described in claim 1, characterized in that, In step S4, 256 parallel simulation environments are constructed. Each parallel simulation environment interacts independently with the current policy network, and collects and forms a state transition tuple containing state, action, reward, next state, and termination flag. The state transition tuple is stored in the experience buffer pool. Based on the data in the experience buffer pool, an instantaneous reward function is constructed, which consists of linear velocity tracking, angular velocity control, attitude offset suppression, joint acceleration constraint, action change rate constraint, and non-foot collision penalty weighting.

9. A reinforcement learning control method for quadruped robots oriented towards simulation-to-real-world transfer, as described in claim 1, characterized in that, In step S4, the deep reinforcement learning algorithm is a policy gradient-based algorithm. It calculates the discounted cumulative reward and the generalized advantage estimate based on the immediate reward function. The discounted cumulative reward is used as the learning objective of the value network, and the generalized advantage estimate is used for parameter updates of the policy network. The learning rates of the policy network and the value network are set to... The reward discount factor is set to 0.99, the generalized advantage estimation coefficient is set to 0.95, the policy shearing threshold is set to 0.2, and the entropy regularization coefficient is set to 0.

01. Iterative updates are performed until the maximum number of training iterations of 1000 or the policy network converges.

10. A reinforcement learning control method for quadruped robots oriented towards simulation-to-real-world transfer, as described in claim 1, characterized in that, In step S5, after loading the converged policy network, gradient calculation is turned off and the network is switched to inference mode. The collected actual non-privileged observation information is processed and normalized in the same way as in the training phase to construct time series state input. The generated joint control commands are target joint position, target joint velocity, target torque or a combination thereof, which are sent to the quadruped robot drive layer to form closed-loop motion control. The recorded running data is written to a non-volatile storage medium for subsequent failure mode analysis, performance evaluation or policy network retraining.

Images