A reinforcement learning-based stair climbing control method and system for a wheel-legged robot, a terminal, and a storage medium

Abstract

The application relates to the technical field of robot control and discloses a stair climbing control method, system, terminal and storage medium for an upright wheel-legged robot based on reinforcement learning. The method comprises the following steps: establishing a parameterized stair scene and loading a robot model in a physical simulation platform to obtain a simulation environment; a Markov decision process is constructed, the Markov decision process comprising a state space, an action space and a reward function; according to the Markov decision process, a domain randomization technology is introduced into the simulation environment, and a reinforcement learning algorithm is used to iteratively train a strategy network; the strategy network after training convergence is deployed to a computing platform of a real robot, the computing platform generates a control instruction according to real-time sensor data, and the real robot performs a stair climbing action through the control instruction. The application can reduce the dependence of the upright wheel-legged robot on the modeling precision of a stair environment, and effectively improve the continuity of the stair climbing action of the robot.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a method, system, terminal, and computer-readable storage medium for controlling a vertical wheeled robot to climb stairs based on reinforcement learning. Background Technology

[0002] Wheel-legged robots combine the efficiency of wheeled locomotion with the obstacle-crossing capabilities of legged motion, showing great potential for adapting to complex terrain. Among them, upright wheel-legged robots have become a research hotspot due to their compact structure and flexible movement. However, enabling them to stably and adaptively climb stairs is an extremely challenging task.

[0003] Currently, the closest existing technical solution is a model-based control method. This method typically includes the following steps: First, the edges of the stairs are accurately identified using a visual sensor (such as Kinect), and the geometric parameters (height, depth) of the steps are calculated; second, the robot is simplified into a physical model such as an inverted pendulum, and the robot's zero-torque point (ZMP) is calculated to evaluate stability; finally, based on preset physical formulas and ZMP stability criteria, motor control commands are derived to guide the robot to complete the stair-climbing action step by step.

[0004] However, the above-mentioned existing technical solutions have the following significant drawbacks: 1. High requirements for environmental precision: Control performance heavily relies on the accurate measurement of the staircase's geometry. Sensor recognition errors or non-standard or damaged staircases can cause the preset physical model to fail, leading to the robot misstepping or falling.

[0005] 2. Poor adaptability (low robustness): Model-based control is usually based on ideal assumptions such as rigid surfaces and constant friction coefficients. Once it encounters carpets, slippery surfaces, or changes in friction, the control parameters (such as PID parameters) require a lot of manual readjustment and cannot automatically adapt to environmental changes.

[0006] 3. Stiff and disjointed movements: Finite state machines (FSMs) are usually used to divide the stair climbing stages. When switching between stages, it is easy to feel a sense of stagnation, the movement is not smooth, and it is difficult to achieve dynamic and efficient climbing.

[0007] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0008] The main objective of this invention is to provide a method, system, terminal, and computer-readable storage medium for controlling a vertical wheeled robot to climb stairs based on reinforcement learning, aiming to solve the problems of strong environmental dependence, poor adaptability, and disjointed movements in the control of vertical wheeled robots climbing stairs in the prior art.

[0009] To achieve the above objectives, the present invention provides a reinforcement learning-based control method for a vertical wheeled robot climbing stairs, the method comprising the following steps: A parametric staircase scene was created in a physical simulation platform, and a robot model was loaded to obtain the simulation environment. Construct a Markov decision process, wherein the Markov decision process includes a state space, an action space, and a reward function; Based on the Markov decision process, domain randomization is introduced into the simulation environment, and reinforcement learning algorithm is used to iteratively train the policy network. The converged policy network is deployed to the computing platform of a real robot. The computing platform generates control commands based on real-time sensor data and controls the real robot to perform the climbing action through the control commands.

[0010] Furthermore, the step height and depth of the staircase scene are randomly generated within a preset range.

[0011] Furthermore, the state space includes a policy observation vector and an evaluation observation vector. The policy observation vector includes robot body state information, which is used for real-time control of the real robot. The robot body state information includes the base attitude, base angular velocity and linear acceleration calculated by the inertial measurement unit, the joint angle and joint velocity measured by the encoder, and the base linear velocity obtained by state estimation. The evaluation observation vector includes privileged environment perception information, which is used for policy network training in a simulation environment. The privileged environment perception information includes a local terrain height map of the robot's feet and surrounding area.

[0012] Furthermore, the motion space is configured to output motor control commands for the robot. These motor control commands include: 4D absolute position commands, 2D relative position commands, and 2D angular velocity commands. The 4D absolute position commands, the 2D relative position commands, and the 2D angular velocity commands work together to execute a composite stair-climbing action, specifically: The 4D absolute position commands correspond to the hip and knee joints of the robot's left and right legs, respectively, and are used to adjust the robot's posture support relative to the step. The two-dimensional relative position commands correspond to the triangular star-shaped wheel frame flipping mechanism of the robot's left and right legs, respectively. They are configured to output angle increments to drive the star-shaped wheel frame to dynamically adapt to the edge of the step, thereby realizing the continuous flipping of the star-shaped wheel frame and stair climbing. The 2D angular velocity commands correspond to the wheel drive joints of the robot's left and right legs, respectively, and are used to provide continuous propulsion for the robot's climbing process.

[0013] Furthermore, the reward function is a composite reward function, and the expression of the composite reward function is: ; in, For a composite reward function, Progress rewards are based on the speed of progress. To maintain a balanced survival reward, To address the energy consumption penalty for motor output torque, To address the issue of smoothing out sudden changes in action commands, , , and These are the weighting coefficients for each sub-item.

[0014] Furthermore, the domain randomization technique includes randomly perturbing at least one of the robot parameters, environmental parameters, and sensor parameters at the beginning of each training cycle, wherein the robot parameters are the robot's total mass and center of mass position; the environmental parameters are the friction coefficients of the ground and stair surfaces; and the sensor parameters are the noise added to the sensor measurement data.

[0015] Furthermore, the process of deploying the converged policy network to the computing platform of the real robot, wherein the computing platform generates control commands based on real-time sensor data, and controls the real robot to perform the stair-climbing action through the control commands, further includes: Real-time monitoring of the robot's posture angles; When the attitude angle exceeds the safety threshold, an emergency stop or protective action is triggered.

[0016] Furthermore, to achieve the above objectives, the present invention also provides a reinforcement learning-based control system for a vertical wheeled robot climbing stairs. This reinforcement learning-based control system is used to implement the reinforcement learning-based control method for a vertical wheeled robot climbing stairs as described above. The reinforcement learning-based control system for a vertical wheeled robot climbing stairs includes: The simulation environment construction module is used to create a parametric staircase scene and load the robot model in the physical simulation platform to obtain the simulation environment; A decision process construction module is used to construct a Markov decision process, wherein the Markov decision process includes a state space, an action space, and a reward function; The policy network training module is used to introduce domain randomization technology into the simulation environment according to the Markov decision process, and to iteratively train the policy network using a reinforcement learning algorithm. The policy network deployment module is used to deploy the trained and converged policy network to the computing platform of the real robot. The computing platform generates control commands based on real-time sensor data and controls the real robot to perform the climbing action through the control commands.

[0017] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a reinforcement learning-based stair climbing control program for an upright wheeled robot stored in the memory and executable on the processor. When the reinforcement learning-based stair climbing control program for an upright wheeled robot is executed by the processor, it implements the steps of the reinforcement learning-based stair climbing control method for an upright wheeled robot as described above.

[0018] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a reinforcement learning-based control program for a vertical wheeled robot climbing stairs, and when the reinforcement learning-based control program for a vertical wheeled robot climbing stairs is executed by a processor, it implements the steps of the reinforcement learning-based control method for a vertical wheeled robot climbing stairs as described above.

[0019] The beneficial effects of this invention are as follows: This invention achieves both safety and efficiency in the training process by constructing a high-fidelity simulation environment. The simulation environment allows for large-scale, repetitive trials in a virtual setting, avoiding the risk of damage to real robot hardware and significantly reducing time and cost. The parameterized staircase scenario ensures the diversity of training data, exposing the policy network to various staircase variations, thereby improving the generalization and adaptability of the final policy and reducing dependence on specific environmental parameters.

[0020] This invention formalizes the robot's stair-climbing task as a Markov decision process, providing a structured framework for reinforcement learning. By defining the state space, action space, and reward function, the policy network can learn optimal behavior through interaction. This avoids the complexity of relying on manual derivation of physical models in traditional methods, promotes end-to-end learning, and thus improves the coherence and intelligence of control, enabling the robot to autonomously adapt to dynamic environments.

[0021] This invention effectively narrows the gap between simulation and the real world through domain randomization technology, enhancing the robustness and anti-interference ability of the policy network. Combined with iterative training using reinforcement learning algorithms, the policy network can learn from diverse virtual experiences and automatically optimize its behavior to cope with uncertainty. This improves the system's adaptability, reduces the manual debugging required by traditional methods, and enables the robot to operate stably in unknown environments.

[0022] This invention enables seamless migration from simulation to real-world environments, allowing trained policy networks to be directly applied to real robots. By generating control commands based on real-time sensor data, the robot can dynamically respond to environmental changes, achieving closed-loop control. This reduces deployment complexity, improves system usability and efficiency, while maintaining policy performance, ensuring the robot's reliability and autonomy in real-world scenarios. Attached Figure Description

[0023] Figure 1 This is a flowchart of a preferred embodiment of the reinforcement learning-based stair climbing control method for an upright wheeled robot according to the present invention; Figure 2 This is a structural diagram of a preferred embodiment of the reinforcement learning-based stair-climbing control system for an upright wheeled robot of the present invention; Figure 3 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0024] This application provides a method, system, terminal, and storage medium for controlling a vertical wheeled robot to climb stairs based on reinforcement learning. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description, with reference to the accompanying drawings and embodiments, further illustrates the application. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0026] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0027] The preferred embodiment of the present invention describes a reinforcement learning-based stair-climbing control method for an upright wheeled robot. Figure 1 As shown, the reinforcement learning-based stair climbing control method for an upright wheeled robot includes the following steps: S10. Establish a parametric staircase scene in the physical simulation platform and load the robot model to obtain the simulation environment.

[0028] The purpose of this step is to construct a high-fidelity virtual simulation environment for the safe and efficient training of reinforcement learning policy networks. Training robots in real-world environments carries risks of hardware damage, is time-consuming, and makes it difficult to repeat experiments. In contrast, a simulation environment can mimic various staircase scenarios and physical conditions, allowing for massive parallel training and rapid iteration. By parameterizing the staircase scenario, diverse training data can be generated, improving the generalization ability of the policy network and laying the foundation for subsequent reinforcement learning training. Furthermore, the simulation environment allows for precise control of the robot model and environmental parameters, facilitating debugging and optimization.

[0029] In this embodiment, NVIDIA Isaac Sim is selected as the physics simulation platform, which supports high-precision physics engines (such as PhysX) and robot model import. First, the URDF (Unified Robot Description Format) file or USD (Universal Scene Description) file of the upright wheeled robot is exported from robot design software (such as SolidWorks or the URDF tool) and loaded into Isaac Sim. The robot model includes mechanical structure, mass attributes, joint constraints, and sensor configuration (such as IMU, encoder, and depth camera). Next, a parametric staircase scene is constructed: the staircase consists of multiple steps, with the height and depth of each step randomly generated within a preset range. For example, the step height varies randomly between 15cm and 20cm, and the step depth varies randomly between 25cm and 30cm; the staircase slope can be calculated based on the height and depth. This parametric design ensures the diversity of training data and avoids overfitting. The simulation environment also sets physical parameters such as gravity and air resistance to simulate the real world. To enhance realism, this embodiment may add textures and lighting effects, but this is not necessary for training. During training, the robot starts from the bottom of the stairs, aiming to climb to the top using a policy network. The simulation environment runs in real-time or accelerated mode, allowing for adjustments to the time step to improve training efficiency.

[0030] It should be noted that the choice of simulation platform is not limited to Isaac Sim; other platforms such as Gazebo, PyBullet, or Mujoco can also be used for similar purposes. The parameterization range should be adjusted according to the actual application scenario: for example, for indoor stairs, the step height may be concentrated between 15-18cm; while for outdoor stairs, the range may be wider. The accuracy of the robot model directly affects the training effect; therefore, it is necessary to ensure that the URDF file accurately reflects the dynamic characteristics of the real robot. Furthermore, the simulation environment should support API interfaces to integrate with reinforcement learning frameworks (such as rsl_rl) and achieve automated training processes.

[0031] Furthermore, the step height and depth of the staircase scene are randomly generated within a preset range.

[0032] The purpose here is to improve the adaptability of the policy network to different staircase structures by randomizing the staircase geometry parameters. Real-world staircases are often not standardized and may have inconsistent heights, damage, or non-standard dimensions. Randomly generating step parameters can simulate these variations, enabling the trained policy network to handle unknown staircase scenarios, reducing reliance on precise environmental measurements, and enhancing robustness.

[0033] In this embodiment, the randomization of step height and depth is implemented via script. At the start of each training episode, the simulation environment generates a new staircase: step heights are sampled from a uniform distribution U(0.15, 0.20) meters, and step depths are sampled from U(0.25, 0.30) meters. The total length of the staircase is fixed (e.g., 10 steps), but the parameters vary randomly. Furthermore, obstacles, such as broken step edges or additional protrusions, may be randomly added to further increase the difficulty. The randomization algorithm can be extended to be based on probability distributions (e.g., Gaussian distributions) or user-defined rules to simulate specific scenarios (e.g., old staircases). The parameter range should be set according to the target application: for example, for residential staircases, the height range might be 15-18 cm; for industrial staircases, it might be 18-22 cm.

[0034] It should be noted that parameter randomization is part of domain randomization, but here we focus on environmental geometry. The degree of randomization needs to be balanced: too broad a range may lead to training difficulties, while too narrow a range reduces generalization ability. It is recommended to adjust the range experimentally to ensure that the policy network adapts to variations while maintaining stability. Furthermore, randomization is not limited to step size; it can also include visual changes such as stair material and color, but this embodiment prioritizes the impact of geometric parameters on motion control.

[0035] S20. Construct a Markov decision process, wherein the Markov decision process includes a state space, an action space, and a reward function.

[0036] The purpose of this step is to formalize the robot's stair-climbing task as a Markov Decision Process (MDP), providing a mathematical framework for reinforcement learning. The MDP defines a state space (environmental observation), an action space (control output), and a reward function (learning objective), enabling the agent (policy network) to learn the optimal policy through interaction with the environment. The state space captures the robot's own state and environmental perception information, ensuring the policy has sufficient context; the action space defines the format of motor control commands; and the reward function guides the policy towards optimizing the desired behavior (such as fast, stable stair climbing). The construction of the MDP is fundamental to reinforcement learning, directly impacting training efficiency and final performance.

[0037] In this embodiment, the MDP is constructed as follows: The state space is divided into policy observation vectors and evaluation observation vectors. Policy observation vectors (Actor network input): These have a total dimension of 41 and contain only the robot's ontological state information acquired through sensors and historical control feedback, used for real-time deployment of the actual robot. The ontological state information is acquired from sensors: the base attitude, base angular velocity, and linear acceleration calculated by the inertial measurement unit, and the base linear velocity obtained through state estimation; the encoder measures the angle (rad) and velocity (rad / s) of each joint. Historical control feedback includes the 8-dimensional action commands executed in the previous moment, used to provide temporal context for implicitly inferring terrain features. Evaluation observation vector (Critic network input): The total dimension is 231, encompassing all dimensions of the policy observation vector, and additionally introducing privileged information available only in the simulation environment to aid training. This includes privileged environment awareness information (187 dimensions): a local terrain height map centered on the robot base obtained through ray detection in the simulation platform, where each grid value represents the height difference between the terrain surface and the robot base; and real motion state (3 dimensions): including the actual linear velocity of the base provided by the simulation environment, used to accurately evaluate movement efficiency. The reward function is a composite form, including progress reward, survival reward, energy consumption penalty, and smoothing penalty, with weights adjusted experimentally. The time step of the MDP is set to 0.01 seconds (100Hz), consistent with the control frequency of the real robot.

[0038] It should be noted that the design of MDP requires a trade-off between complexity and learning efficiency. The state space should contain relevant information but avoid redundancy. The choice of action space affects control accuracy: torque control is more direct but requires hardware protection; position control is safer but may have a slower response. The reward function weights need to be carefully adjusted to avoid local optima (such as the robot remaining still for survival rewards). This embodiment uses the Proximal Policy Optimization (PPO) algorithm, but other algorithms (such as SAC) can also be used.

[0039] Furthermore, the state space includes a policy observation vector and an evaluation observation vector. The policy observation vector includes robot body state information, which is used for real-time control of the real robot. The robot body state information includes the base attitude, base angular velocity and linear acceleration calculated by the inertial measurement unit, the joint angle and joint velocity measured by the encoder, and the base linear velocity obtained by state estimation. The evaluation observation vector includes privileged environment perception information, which is used for training the policy network in a simulated environment. This privileged environment perception information includes a local terrain height map of the robot's feet and surrounding area. The purpose of this limitation is to ensure that the state space comprehensively captures the robot's internal state and control history, providing sufficient information for the policy network to make intelligent decisions. The ontological state information reflects the robot's posture and motion dynamics.

[0040] In this embodiment, the state space (state vector) is functionally divided into policy observation vectors and evaluation observation vectors. The policy observation vectors are used to control the real-time action generation of the robot, and have a total dimension of 41, specifically comprising: Base motion state (9D): This includes base angular velocity (3D), projected gravity vector (3D), and velocity command (3D). The base angular velocity is acquired via an inertial measurement unit (IMU) and characterizes the rotational rate of the robot base in three axes. The gravity vector [0, 0, -1] in the world coordinate system is projected onto the current robot base coordinate system using a rotation matrix to characterize the real-time tilt attitude of the base relative to the ground. The velocity command is the target linear velocity (x, y directions) and target angular velocity (yaw direction) issued by the host computer or operator.

[0041] Joint status information (24-dimensional): This includes the current angles (12-dimensional) and current angular velocities (12-dimensional) of all 12 driven joints of the robot. In this embodiment, the robot adopts a wheel-leg composite structure. The wheel-leg composite structure includes a triangular star-shaped wheel frame (hereinafter referred to as the star-shaped wheel frame), the center of which is connected to the leg via a drive shaft, and wheel drive units that can rotate synchronously are installed at its three vertices. Each leg contains 6 physical drive joints: the hip joint, the knee joint, the star-shaped wheel frame rotation joint, and three wheel drive joints. By monitoring the real-time feedback of all physical joints, the system can accurately sense the contact state between the star-shaped wheel assembly and the ground.

[0042] Historical action feedback (8-dimensional): Retains the control command vector (actions, 8-dimensional) executed in the previous time step. This dimension corresponds to the robot's core control degrees of freedom, with each leg assigned a 4-dimensional control signal. Specifically, the three wheel drive joints of each leg employ synchronous control logic, meaning all three wheels share a single control signal component. This design effectively reduces the output complexity of the neural network while ensuring gripping power when climbing stairs.

[0043] During the training phase, the evaluation observation vector is used to assess the value of the current state. It contains richer global information than the policy observation vector, with a total dimension of 231, including all dimensions of the policy observation vector, and additionally introduces privileged information available only in the simulation environment to assist training. Base true linear velocity (3D): Includes the base true linear velocity provided by the simulation environment, used to accurately evaluate movement efficiency.

[0044] Privileged environment perception information (187-dimensional): A local terrain height map centered on the robot base is obtained through ray detection in the simulation platform. Each grid value represents the height difference between the terrain surface and the robot base.

[0045] The data preprocessing and generation logic is as follows: Projected gravity calculation: The gravity vector in the global coordinate system is projected onto the robot's base coordinate system through a transformation matrix, enabling the network to perceive the robot's tilt posture.

[0046] Normalization processing: All raw sensor data are standardized and scaled according to preset mean and variance before being input into the neural network to accelerate model convergence.

[0047] k-step historical observation time series enhancement (extension of the example): Construction method: The system maintains a sliding window buffer, which concatenates the original observation O_t at the current time t with the historical records of the past k-1 time steps to form a time series feature matrix of shape [k, 41].

[0048] Addressing Delay and Uncertainty: To address signal delays caused by visual perception or communication during stair climbing, the time-series observation provides a predictive effect similar to "Kalman filtering," making the output of control commands more predictable and smoother in abrupt environments such as stair edges.

[0049] Gait recognition and latching: For wheeled robots, the flipping of the star-shaped wheel frame is a temporally continuous action. k-step observations help the network determine which specific stage the robot is currently "crossing the stairs," avoiding control jitter caused by single-frame noise.

[0050] Data preprocessing includes normalization (scaling the raw sensor values ​​to a mean of 0 and a variance of 1) and time alignment (ensuring that all sensor data are synchronized).

[0051] It should be noted that the state dimension can be adjusted according to the robot's complexity. For more complex robots (such as multi-legged systems), joint information may need to be added; however, excessively high dimensionality may slow down training. The history length k affects policy memory: k that is too small may ignore long-term dependencies, while k that is too large will increase the computational burden.

[0052] Furthermore, the motion space is configured to output motor control commands for the robot. The motor control commands include: 4D absolute position commands, 2D relative position commands, and 2D angular velocity commands. The 4D absolute position commands, the 2D relative position commands, and the 2D angular velocity commands work together to execute a composite stair-climbing action, specifically: The 4D absolute position commands correspond to the hip and knee joints of the robot's left and right legs, respectively, and are used to adjust the robot's posture support relative to the step. The two-dimensional relative position commands correspond to the triangular star-shaped wheel frame flipping mechanism of the robot's left and right legs, respectively. They are configured to output angle increments to drive the star-shaped wheel frame to dynamically adapt to the edge of the step, thereby realizing the continuous flipping of the star-shaped wheel frame and stair climbing. The 2D angular velocity commands correspond to the wheel drive joints of the robot's left and right legs, respectively, and are used to provide continuous propulsion for the robot's climbing process.

[0053] In other words, the motion space of this embodiment includes 8-dimensional control vectors, and the robot climbs stairs through a heterogeneous control strategy, specifically: First control mode: 4-dimensional absolute position commands for the robot's hip and knee joints (2-dimensional for each leg) to adjust the robot's posture support relative to the step. Second control mode: For the 2-dimensional relative position command (1 dimension for each of the left and right legs) of the triangular star wheel frame flipping mechanism, the star wheel frame is dynamically adapted to the edge of the step by outputting the angle increment, so as to realize the continuous flipping of the star wheel frame and the climbing of the stairs; The third control mode: This mode provides continuous propulsion for the robot during the climbing process by issuing 2D angular velocity commands to the wheel drive joints (1D for each leg), and works in conjunction with the first and second control modes to complete the composite stair-climbing motion.

[0054] Furthermore, the reward function is a composite reward function, and the expression of the composite reward function is: ; in, For a composite reward function, Progress rewards are based on the speed of progress. To maintain a balanced survival reward, To address the energy consumption penalty for motor output torque, To address the issue of smoothing out sudden changes in action commands, , , and These are the weighting coefficients for each sub-item.

[0055] The purpose here is to guide the policy network to learn fast, stable, energy-efficient, and smooth stair-climbing behavior through a multi-objective reward function. Progress rewards encourage the robot to move upwards, survival rewards prevent falls, energy consumption penalties improve efficiency, and smoothness penalties reduce hardware wear. The composite reward balances competing objectives and prevents the policy from getting trapped in local optima (e.g., only surviving without moving).

[0056] It should be noted that the reward function can be designed with different components depending on the task. For example, an alignment reward can be added to encourage the robot to face the stairs, or a time penalty can be added to encourage quick completion. The weights need to be determined experimentally: excessively high survival rewards may lead to conservative behavior. This function has been effective in actual testing, but it can be optimized.

[0057] S30. Based on the Markov decision process, domain randomization is introduced into the simulation environment, and reinforcement learning algorithm is used to iteratively train the policy network.

[0058] The purpose of this step is to reduce the simulation-to-real-world gap through domain randomization and to train a policy network using reinforcement learning algorithms, enabling it to learn robust policies that adapt to various uncertainties in a virtual environment. Domain randomization simulates real-world variations by adding random perturbations to robot, environmental, and sensor parameters, making the policy network independent of specific simulation conditions and thus improving its performance in real-world environments. Reinforcement learning algorithms (such as PPO) optimize the policy through trial and error, maximizing cumulative rewards. The training process is conducted efficiently within the simulation, avoiding the risks associated with using real robots.

[0059] In this embodiment, the training process is as follows: The Proximal Policy Optimization (PPO) algorithm is used due to its stability and efficiency. The policy network is a neural network structure: the input layer corresponds to the state space (41 dimensions), the hidden layer includes three fully connected layers (the number of neurons in the hidden layer is set to 512, 256, and 128 respectively, forming a progressively converging feature extraction structure. The hidden layers uniformly use the ELU activation function), and the output layer corresponds to the action space (8 dimensions, with the tanh activation function to limit the output range). The value network has a similar structure. Training hyperparameters: learning rate 0.0003, with an adaptive update mechanism; discount factor 0.99, PPO pruning parameter 0.2. Domain randomization is applied at the beginning of each training cycle: the robot's total mass is randomly perturbed within ±10% (e.g., a baseline mass of 15kg, after perturbing 13.5-16.5kg); the center of mass position is randomly shifted within ±2cm; the ground friction coefficient is randomly between 0.4 and 1.0; Gaussian white noise (mean 0, standard deviation 5% of the measured value) is added to the sensor data. Training was conducted in 8192 parallel simulation environments, with each environment collecting 24 time steps per epoch. The total sample size for a single iteration reached 196,608. Five training epochs were set, and the total sample size was divided into four mini-batches to improve computational efficiency. Data from a certain number of time steps was collected in each iteration to update the network parameters until the reward curve converged.

[0060] It should be noted that the domain randomization parameters need to be set appropriately: excessive perturbation may make training difficult, while insufficient perturbation will fail to cover true mutations. Reinforcement learning algorithms such as SAC or TD3 can be selected, but PPO is more mature. The network structure can be adjusted: more complex tasks may require larger networks or recurrent neural networks (RNNs) to handle the timing. Training efficiency depends on the hardware (such as the number of GPUs); in this embodiment, the training task is deployed on a computing platform equipped with an NVIDIA GeForce RTX 4090 graphics card.

[0061] Furthermore, the domain randomization technique includes randomly perturbing at least one of the robot parameters, environmental parameters, and sensor parameters at the beginning of each training cycle, wherein the robot parameters are the robot's total mass and center of mass position; the environmental parameters are the friction coefficients of the ground and stair surfaces; and the sensor parameters are the noise added to the sensor measurement data.

[0062] The purpose of this approach is to systematically introduce uncertainty, making the policy network robust to model errors, environmental changes, and sensor noise. Mass changes simulate load variations; center-of-mass changes simulate structural deformation; friction variations simulate different materials (such as carpets and ice); and sensor noise simulates real measurement errors. This randomization forces the policy to learn conservative and adaptive behavior.

[0063] In this embodiment, randomization is specifically implemented as follows: Before the start of each training episode, parameters are randomly sampled. The total mass of the robot is sampled from a uniform distribution U(13.5, 16.5) kg; the center of mass position is offset by U(-0.02, 0.02) meters in the x, y, and z directions respectively; the friction coefficient is sampled from U(0.4, 1.0); the sensor noise is additive Gaussian noise, where the standard deviation of the IMU angular velocity observation is set to 0.05 rad / s, the standard deviation of the linear acceleration is 0.1 m / s², and the standard deviation of the encoder noise is 0.01 rad. Randomization is applied independently. During training, the randomization intensity is gradually increased to stabilize learning.

[0064] It should be noted that domain randomization is a key technique in Sim-to-Real transfer, but not the only method; other methods such as system identification or adaptive control can also be combined. The parameter range should be based on real robot calibration: for example, by experimentally measuring the range of quality errors. Randomization can also include visual variations (such as texture), but this embodiment focuses on physical parameters.

[0065] S40. The converged policy network is deployed to the computing platform of the real robot. The computing platform generates control commands based on real-time sensor data and controls the real robot to perform the climbing action through the control commands.

[0066] The purpose of this step is to deploy the policy network trained in the simulation onto a real robot to achieve autonomous stair climbing control. Deployment involves model conversion, integration into the robot's software stack, and ensuring real-time performance. The policy network infers and outputs control commands in real time based on real sensor data, forming a closed-loop control system. This step verifies the effectiveness of the simulation training and realizes the practical application of the method.

[0067] In this embodiment, the deployment process is as follows: First, the trained policy network (PyTorch or TensorFlow model) is converted to an optimized format, such as ONNX or TensorRT, to reduce inference latency. Then, it is deployed to the computing platform of the real robot (such as NVIDIA Jetson Orin NX). The robot software architecture includes: sensor drive (reading IMU, encoder), data preprocessing (normalization), policy inference module (running neural network), and actuator control (sending motor commands). The control loop runs at 100Hz: sensor data is read every 10ms, preprocessed into the same state vector as the simulation, input to the network, outputting actions, and sent to the motors after safety checks. In the example, the robot successfully climbed 10 steps, adapting to changes in height and slippage.

[0068] It should be noted that fine-tuning may be required during deployment: for example, the strategy may perform poorly due to simulation errors, requiring additional calibration or online adaptation.

[0069] Furthermore, the step of deploying the converged policy network to the computing platform of the real robot, wherein the computing platform generates control commands based on real-time sensor data, and controls the real robot to perform the stair-climbing action through the control commands, further includes: Real-time monitoring of the robot's posture angles; When the attitude angle exceeds the safety threshold, an emergency stop or protective action is triggered.

[0070] The purpose of this limitation is to add a safety monitoring mechanism to prevent the robot from being damaged or causing danger in unexpected situations (such as strategy failure or sudden environmental changes). Attitude angles are a key indicator of balance; exceeding a threshold may indicate a fall. An emergency stop immediately cuts off power, and a protective action (such as crouching) attempts to restore balance. This improves the system's reliability.

[0071] In this embodiment, safety monitoring runs in a separate thread. The base pitch and roll angles measured by the IMU are monitored in real time. Safety thresholds are set at ±35 degrees for pitch and ±25 degrees for roll (based on robot stability analysis). If the thresholds are exceeded, an emergency stop is triggered: all motor torques are set to 0, and the robot brakes. Alternatively, a protective action is triggered: the policy network outputs a predefined action (such as adjusting the center of gravity), or the system switches to a backup controller. The thresholds can be adjusted according to the scenario; for example, the thresholds are more stringent during high-speed movement.

[0072] It should be noted that the security mechanism should not interfere with normal control; therefore, the threshold needs to be carefully set. The monitoring frequency should be higher than the control frequency (e.g., 100Hz). Protective actions can be based on reinforcement learning training, but this embodiment uses simple rules. Furthermore, a hardware watchdog can be added to enhance security.

[0073] Furthermore, such as Figure 2 As shown, based on the above-described reinforcement learning-based control method for upright wheeled robots climbing stairs, this invention also provides a reinforcement learning-based control system for upright wheeled robots climbing stairs. This reinforcement learning-based control system is used to implement the reinforcement learning-based control method for upright wheeled robots climbing stairs as described above. The reinforcement learning-based control system for upright wheeled robots climbing stairs includes: The simulation environment construction module 51 is used to build a parameterized staircase scene and load the robot model in the physical simulation platform to obtain the simulation environment; The decision process construction module 52 is used to construct a Markov decision process, wherein the Markov decision process includes a state space, an action space, and a reward function; The policy network training module 53 is used to introduce domain randomization technology into the simulation environment according to the Markov decision process, and to iteratively train the policy network using a reinforcement learning algorithm. The policy network deployment module 54 is used to deploy the trained and converged policy network to the computing platform of the real robot. The computing platform generates control commands based on real-time sensor data and controls the real robot to perform the climbing action through the control commands.

[0074] Furthermore, such as Figure 3 As shown, based on the above-mentioned reinforcement learning-based method and system for controlling the climbing stairs of an upright wheeled robot, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 3 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0075] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a reinforcement learning-based stair-climbing control program 40 for a vertical wheeled robot. This reinforcement learning-based stair-climbing control program 40 can be executed by the processor 10, thereby implementing the reinforcement learning-based stair-climbing control method for a vertical wheeled robot in this application.

[0076] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the reinforcement learning-based stair climbing control method for upright wheeled robots.

[0077] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0078] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a reinforcement learning-based control program for a vertical wheeled robot climbing stairs, and when the reinforcement learning-based control program for a vertical wheeled robot climbing stairs is executed by a processor, it implements the steps of the reinforcement learning-based control method for a vertical wheeled robot climbing stairs as described above.

[0079] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0080] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0081] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for controlling a vertical wheeled robot climbing stairs based on reinforcement learning, characterized in that, The reinforcement learning-based control method for climbing stairs in a vertical wheeled robot includes the following steps: A parametric staircase scene was created in a physical simulation platform, and a robot model was loaded to obtain the simulation environment. Construct a Markov decision process, wherein the Markov decision process includes a state space, an action space, and a reward function; Based on the Markov decision process, domain randomization is introduced into the simulation environment, and reinforcement learning algorithm is used to iteratively train the policy network. The converged policy network is deployed to the computing platform of a real robot. The computing platform generates control commands based on real-time sensor data and controls the real robot to perform the climbing action through the control commands.

2. The method for controlling a vertical wheeled robot climbing stairs based on reinforcement learning according to claim 1, characterized in that, The height and depth of the steps in the staircase scene are randomly generated within a preset range.

3. The method for controlling a vertical wheeled robot climbing stairs based on reinforcement learning according to claim 1, characterized in that, The state space includes a policy observation vector and an evaluation observation vector. The policy observation vector includes robot body state information, which is used for real-time control of the real robot. The robot body state information includes the base attitude, base angular velocity and linear acceleration calculated by the inertial measurement unit, the joint angle and joint velocity measured by the encoder, and the base linear velocity obtained by state estimation. The evaluation observation vector includes privileged environment perception information, which is used for policy network training in a simulation environment. The privileged environment perception information includes a local terrain height map of the robot's feet and surrounding area.

4. The reinforcement learning-based stair-climbing control method for an upright wheeled robot according to claim 1, characterized in that, The motion space is configured to output motor control commands for the robot. These motor control commands include: 4D absolute position commands, 2D relative position commands, and 2D angular velocity commands. The 4D absolute position commands, the 2D relative position commands, and the 2D angular velocity commands work together to execute a composite stair-climbing motion, specifically: The 4D absolute position commands correspond to the hip and knee joints of the robot's left and right legs, respectively, and are used to adjust the robot's posture support relative to the step. The two-dimensional relative position commands correspond to the triangular star-shaped wheel frame flipping mechanism of the robot's left and right legs, respectively. They are configured to output angle increments to drive the star-shaped wheel frame to dynamically adapt to the edge of the step, thereby realizing the continuous flipping of the star-shaped wheel frame and stair climbing. The 2D angular velocity commands correspond to the wheel drive joints of the robot's left and right legs, respectively, and are used to provide continuous propulsion for the robot's climbing process.

5. The method for controlling a vertical wheeled robot climbing stairs based on reinforcement learning according to claim 1, characterized in that, The reward function is a composite reward function, and the expression of the composite reward function is: ； in, For a composite reward function, Progress rewards are based on the speed of progress. To maintain a balanced survival reward, To address the energy consumption penalty for motor output torque, To address the issue of smoothing out sudden changes in action commands, , , and These are the weighting coefficients for each sub-item.

6. The method for controlling a vertical wheeled robot climbing stairs based on reinforcement learning according to claim 1, characterized in that, The domain randomization technique includes randomly perturbing at least one of the robot parameters, environmental parameters, and sensor parameters at the beginning of each training cycle, wherein the robot parameters are the robot's total mass and center of mass position; the environmental parameters are the friction coefficients of the ground and stair surfaces; and the sensor parameters are the noise added to the sensor measurement data.

7. The method for controlling a vertical wheeled robot climbing stairs based on reinforcement learning according to claim 1, characterized in that, The process involves deploying the converged policy network to the computing platform of a real robot. The computing platform generates control commands based on real-time sensor data and uses these commands to control the real robot to perform a stair-climbing action. The process further includes: Real-time monitoring of the robot's posture angles; When the attitude angle exceeds the safety threshold, an emergency stop or protective action is triggered.

8. A reinforcement learning-based control system for a vertical wheeled robot climbing stairs, characterized in that, The reinforcement learning-based stair-climbing control system for an upright wheeled robot is used to implement the reinforcement learning-based stair-climbing control method for an upright wheeled robot as described in any one of claims 1-7. The reinforcement learning-based stair-climbing control system for an upright wheeled robot includes: The simulation environment construction module is used to create a parametric staircase scene and load the robot model in the physical simulation platform to obtain the simulation environment; A decision process construction module is used to construct a Markov decision process, wherein the Markov decision process includes a state space, an action space, and a reward function; The policy network training module is used to introduce domain randomization technology into the simulation environment according to the Markov decision process, and to iteratively train the policy network using a reinforcement learning algorithm. The policy network deployment module is used to deploy the trained and converged policy network to the computing platform of the real robot. The computing platform generates control commands based on real-time sensor data and controls the real robot to perform the climbing action through the control commands.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a reinforcement learning-based stair climbing control program for an upright wheeled robot stored in the memory and executable on the processor. When the reinforcement learning-based stair climbing control program for an upright wheeled robot is executed by the processor, it implements the steps of the reinforcement learning-based stair climbing control method for an upright wheeled robot as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a reinforcement learning-based control program for a vertical wheeled robot climbing stairs, which, when executed by a processor, implements the steps of the reinforcement learning-based control method for a vertical wheeled robot climbing stairs as described in any one of claims 1-7.

Images