Obstacle crossing control method and system for a biped robot based on explicit state estimation
By co-optimizing explicit state estimation and policy network, the problems of poor adaptability of traditional control methods in complex terrain and uninterpretable state estimation in reinforcement learning are solved, realizing stable and efficient movement of bipedal robots in complex environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANDONG UNIV
- Filing Date
- 2026-02-25
- Publication Date
- 2026-06-05
AI Technical Summary
Traditional model-based control methods are difficult to adapt to complex terrains, and the state estimation of reinforcement learning methods is uninterpretable, resulting in poor motion stability and low efficiency of bipedal robots.
A control method based on explicit state estimation is adopted. Through the coordinated operation of the state estimator and the policy network, combined with the proximal policy optimization algorithm, explicit privileged information is obtained and the control policy is optimized, thereby improving motion stability and efficiency.
It enables smooth and reliable movement of bipedal robots in complex environments, improves the scalability and reliability of the control strategy, and increases the success rate of obstacle crossing.
Smart Images
Figure CN122151849A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of robot control, and particularly relates to a method and system for obstacle crossing control of biwheeled legged robots based on explicit state estimation. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] Robot motion control is one of the core areas of robotics technology, especially in applications in unstructured and complex environments, where extremely high requirements are placed on the autonomy, adaptability, and robustness of robot motion control. Bicycle-legged robots, as a composite mobile platform combining the efficiency of wheeled mobility with the obstacle-crossing capabilities of legged movement, have shown great potential in scenarios such as field inspection, disaster relief, and special operations. Therefore, achieving stable and reliable motion control of bicycle-legged robots in complex terrain has become a key technical challenge.
[0004] Currently, in the field of robot motion control, traditional control methods are dominated by model-based control such as model predictive control, linear quadratic regulators, proportional-integral-derivative control, and sliding mode control. These methods rely on accurate robot dynamics and kinematic modeling, requiring pre-defined key physical characteristics such as mass distribution, inertial parameters, and joint constraints to generate control commands. However, bipedal robots, as typical wheel-leg coupled, nonlinear, and underactuated systems, face challenges when their motion scenarios involve complex terrain (such as steps, protruding platforms, and rough surfaces), dynamic changes such as abrupt changes in terrain height and differences in surface characteristics. Traditional model-based control struggles to cover the dynamic characteristics of all scenarios, making it prone to policy adaptation errors.
[0005] Meanwhile, the performance of model-based control is highly dependent on the matching degree between the model and the actual system. When there are external disturbances or model errors, the control accuracy will decrease significantly. In high-dimensional control tasks, these methods are limited by the limitations of observation information, making it difficult to accurately estimate the robot's current state and thus unable to achieve precise posture adjustment. Especially when bipedal robots perform actions such as biomimetic jumps and obstacle crossing in complex terrains, they are prone to mechanical shocks or recovery failures due to abrupt movements and sudden changes in joint load.
[0006] Furthermore, in recent years, reinforcement learning-based control methods have demonstrated significant advantages. They eliminate the need for manually building complex models and can autonomously learn control strategies to adapt to diverse scenarios through trial and error with the environment, exhibiting good generalization ability and robustness in complex terrain motion. However, existing reinforcement learning-based control methods for bipedal robots still have significant shortcomings. Many solutions rely on end-to-end policy networks, implicitly containing the perception and estimation of critical environmental state information within the black box of the neural network, existing as uninterpretable latent vectors. This leads to two major problems: the results of this implicit estimation are difficult to effectively integrate and coordinate with other functional modules in the system that require explicit state input, limiting the scalability and reliability of the control strategy; for tasks such as multimodal cooperative motion of bipedal robots, which require extremely high accuracy of state information, the implicitly estimated latent vectors are insufficient to meet the needs of complex motion coordination, easily leading to problems such as unsmooth motion mode switching, misjudgment of obstacle crossing decision timing, and low obstacle crossing success rate. Essentially, existing methods fail to effectively combine the decision-making advantages of reinforcement learning with interpretable and integrable environmental perception capabilities.
[0007] It is evident that traditional model-based control techniques suffer from poor scene adaptability and insufficient robustness. Existing reinforcement learning solutions also suffer from uninterpretable state estimation and weak module compatibility, resulting in poor motion stability and low efficiency of bipedal robots in complex environments. Summary of the Invention
[0008] To overcome the shortcomings of the prior art, this invention provides a method and system for obstacle-crossing control of biwheeled robots based on explicit state estimation. By coordinating the designed state estimator, policy network, and value network, and using the optimized control of the proximal policy optimization algorithm, the biwheeled robot is driven to perform obstacle-crossing actions, thereby improving the motion stability and efficiency of the biwheeled robot in complex environments.
[0009] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of the present invention provides an obstacle-crossing control method for a biwheeled legged robot based on explicit state estimation, comprising: Acquire the historical motion state observation sequence and the current motion state observation information of the biwheeled legged robot; The historical motion state observation sequence is input into the state estimator to obtain the explicit privileged information estimate; The explicit privileged information estimate is fused with the motion state observation information at the current moment to obtain the policy network input observation space; The policy network is input into the observation space and then into the pre-trained policy network to obtain the action instructions at the current time. The proportional-derivative controller converts motion commands into joint torque control commands, driving the biwheeled legged robot to perform obstacle-crossing actions. The policy network is trained based on the output of the value network, and is optimized and trained using a proximal policy optimization algorithm.
[0010] As one implementation method, the motion state observation information includes at least the base angular velocity, projected gravity, joint position, joint velocity, motion command and control command of the previous control cycle.
[0011] As one implementation method, the historical motion state observation sequence is input into the state estimator, which employs a multilayer perceptron, a temporal convolutional network, or a recurrent neural network structure to estimate terrain-related privileged information through supervised learning and extract terrain-related display privileged information estimates.
[0012] As one implementation, the explicit privileged information estimate includes at least the base linear velocity estimate, the distance between the foot and the ground estimate, and the contact probability estimate between the foot and the ground estimate.
[0013] As one implementation method, based on the output of the value network, the policy network is optimized and trained using a proximal policy optimization algorithm. The specific process is as follows: The real privileged state information of the acquired simulation environment and the motion state observation information at the current moment are input into the value network to obtain the state value estimate; Construct a reward function based on the results of the obstacle-crossing action; Based on state value estimation and reward function, a proximal policy optimization algorithm is used to generate policy gradients, which are then backpropagated to the policy network to iteratively optimize the policy network parameters.
[0014] As one implementation method, the reward function formula is: ; in, This represents the weighted sum of all rewards. This represents the weighted reward for maintaining the feasibility of the motion through positional constraints, velocity constraints, and various smoothing penalties. This indicates that the weighted approach ensures the rationality of the bipedal robot's motion posture by constraining the base angle, the distance between its two legs, the number of wheels in contact with the ground, and the collision location with the ground. This represents the reward function for the weighted tracking terminal commands. This represents the weighted reward function that the bipedal robot can overcome when encountering a high obstacle.
[0015] As one implementation method, the weighted reward function for the bipedal robot when encountering a high obstacle is expressed as follows: ; in, It is a reward function that encourages the bipedal robot to lift its feet when encountering high obstacles. , To determine the matrix for determining whether the duration of foot lifting of a bipedal robot is within a suitable range, It is a matrix used to determine when a robot encounters a height obstacle. yes Inverse of; It is a reward function that encourages bipedal robots to lift their legs to a target height when encountering obstacles. , It is an exponential function of the sum of the differences between the actual leg lift height and the target value. and These are matrices used to determine whether the legs of a bipedal robot are subjected to forces in the x and y directions. and These are the weighting coefficients.
[0016] In one implementation method, a proportional-derivative controller converts motion commands into joint torque control commands. The control method used is position and velocity control, and the formula is as follows: ; ; in, , These are the joint torques issued to the leg joints and wheel joints, respectively. , These are the control coefficients of the PD controller; , For position control and speed control, the motion scaling factor is used. , These are the leg joint movements and wheel joint movements output by the policy network, respectively. , The default leg joint position and the current leg joint position are set respectively. , These are the joint velocities of the legs and wheels, respectively.
[0017] A second aspect of the present invention provides an obstacle-crossing control system for a bipedal robot based on explicit state estimation, comprising: The data acquisition module is used to acquire the historical motion state observation sequence and the current motion state observation information of the bipedal robot. The state estimation module is used to input the historical motion state observation sequence into the state estimator to obtain the explicit privileged information estimate; The policy execution module is used to fuse the explicit privileged information estimate with the motion state observation information at the current moment to obtain the policy network input observation space; and input the policy network input observation space into the pre-trained policy network to obtain the action command at the current moment. The control output module is used to convert motion commands into joint torques via a proportional-derivative controller, driving the bipedal robot to perform obstacle-crossing actions.
[0018] As one implementation, a training module is also included for parallel and independent training of the policy network and the state estimator, wherein the training module includes: The supervised learning unit is used to optimize the state estimator through supervised learning based on the estimated value of explicit privileged information. The reinforcement learning unit is used to optimize and train the policy network based on the output of the value network through a proximal policy optimization algorithm.
[0019] The above one or more technical solutions have the following beneficial effects: In this embodiment, an independent explicit state estimator is designed. Based on historical observation sequences, it directly outputs explicit privileged information with clear physical meaning, including base linear velocity, foot-to-ground distance, and foot-to-ground contact probability, through supervised learning. This allows the privileged information to be called by other control modules, improving the scalability and reliability of the control strategy. At the same time, it retains the ability of reinforcement learning to adapt to complex scenarios, effectively solving the problems of ambiguous state perception and insufficient control precision of bipedal robots in complex terrain.
[0020] In this embodiment, the value network continuously optimizes the accuracy of state value estimation based on the input of real privilege information and motion state information of the current frame, providing a reliable reward signal basis for the parameter update of the policy network. Furthermore, the policy network is iteratively optimized through the near-end policy optimization algorithm in reinforcement learning, thereby achieving a synergistic improvement in policy output performance and value assessment accuracy.
[0021] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0023] Figure 1 This is a schematic diagram of the obstacle-crossing control method for a biwheeled legged robot based on explicit state estimation according to Embodiment 1 of the present invention. Figure 2This is a schematic diagram of a bipedal robot continuously traversing obstacles of varying heights according to Embodiment 1 of the present invention; Among them, Figure (a) shows obstacle detection preparation; Figure (b) shows raising the standing height; Figure (c) shows stepping on one wheel foot; Figure (d) shows landing on one wheel foot; Figure (e) shows crossing on the other wheel foot; and Figure (f) shows cushioning and recovery. Figure 3 This is a schematic diagram illustrating the process of a bipedal robot crossing a ditch according to Embodiment 1 of the present invention; Among them, Figure (a) shows obstacle and ravine detection and status assessment; Figure (b) shows the preparation for pushing off the ground and accumulating power; Figure (c) shows the take-off and crossing stage; Figure (d) shows the precise landing of the front leg; Figure (e) shows the landing of the other hind leg; and Figure (f) shows the posture adjustment and reset. Figure 4 This is a schematic diagram illustrating the process of the bipedal robot of Embodiment 1 of the present invention moving on a rough 20° slope; Among them, Figure (a) shows the slope detection and initial alignment; Figure (b) shows the attitude pre-adjustment; Figure (c) shows the stable uphill movement; Figure (d) shows the slope attitude calibration; Figure (e) shows the adaptation to approach the top of the slope; and Figure (f) shows the smooth reset. Detailed Implementation
[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.
[0025] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.
[0026] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.
[0027] Example 1 This embodiment discloses an obstacle-crossing control method for a biwheeled legged robot based on explicit state estimation.
[0028] To more clearly illustrate this embodiment, the obstacle-crossing control implementation process of the biwheeled legged robot based on explicit state estimation can be specifically described as follows: An obstacle-crossing control method for biwheeled legs based on explicit state estimation includes: S1. Obtain the historical motion state observation sequence and the current motion state observation information of the bipedal robot; S2. Input the historical motion state observation sequence into the state estimator to obtain the explicit privileged information estimate; S3. The explicit privileged information estimate is fused with the motion state observation information at the current moment to obtain the policy network input observation space; The policy network is input into the observation space to obtain the action command at the current time. S4. The motion commands are converted into joint torques by a proportional-derivative controller, which drives the bi-wheeled legged robot to perform obstacle-crossing actions. The policy network is trained based on the output of the value network, and is optimized and trained using a proximal policy optimization algorithm.
[0029] like Figure 1 As shown, in step S1, the historical motion state observation sequence of the bipedal robot and the motion state observation information at the current moment are obtained.
[0030] S101. Before acquiring data, simulation modeling and environment setup should be performed.
[0031] (1) Build a dynamic model of the bipedal robot in the simulation environment and initialize the model.
[0032] The model observation space integrates key information such as the robot's overall dynamics, balance state, joint motion information, control action records, and target commands. Furthermore, considering the characteristics of the two-wheeled leg configuration, the observation dimensions related to joint motion and control actions are expanded to accurately match the dynamic characteristics of wheel-leg coupling, providing a reliable foundation for state mapping and strategy training during subsequent obstacle crossing.
[0033] (2) Construct a virtual training scene that includes terrain with high obstacles.
[0034] The virtual scene needs to cover the diverse characteristics of obstacles to ensure the generalization ability of the trained strategy: the bipedal robot is trained with domain randomization on obstacle height, side wall surface roughness, top geometry, side wall and ground inclination angle and ground friction coefficient to avoid overfitting of the strategy caused by training in a single scene.
[0035] S102. Obtain the historical motion state observation sequence and the current motion state observation information of the bipedal robot. (1) Set the system control frequency to 50Hz, that is, the time of each frame is 0.02s. Within each control cycle t, the motion state observation information at the current moment is obtained through the body sensor of the two-wheeled robot.
[0036] The motion state observation information includes at least the base angular velocity, projected gravity, joint position, joint velocity, motion command and control command of the previous control cycle.
[0037] The bipedal robot can obtain proprioceptive information from the inertial measurement unit (IMU) and motor encoders, as well as action output from the policy network and command information output from external terminals. Specifically: 1) Obtain the base angular velocity and gravity direction vector through the inertial measurement unit (IMU).
[0038] 2) Read the joint angle and joint angular velocity from the encoder of each joint motor.
[0039] 3) Read the action instructions and control instructions from the previous control cycle (time t-1) from memory.
[0040] That is, the motion state observation information was obtained. Information such as base angular velocity, gravity projection in the robot's body coordinate system, joint positions excluding wheel joints, all joint velocities, and given speed and base height commands.
[0041] (2) The historical motion state observation sequence contains observation information from multiple consecutive control cycles.
[0042] If the current frame motion state information is set for: ; The historical motion state observation sequence is as follows: ; in, This indicates the rotational speed of the machine body about its three coordinate axes; This represents the direction vector of gravitational acceleration in the robot's body coordinate system, used to estimate the robot's attitude; , These represent the robot's joint angle and joint angular velocity, respectively. Indicates the target joint position output in the previous control cycle; These are command signals, including the desired forward speed, turning speed, and aircraft altitude; Indicates the first Motion state information of the frame.
[0043] like Figure 1 As shown, in step S2, the historical motion state observation sequence is input into the state estimator to obtain the explicit privileged information estimate.
[0044] Since much key information is not directly available, a state estimator is needed to directly estimate observable state variables. For example, obstacle height is indirectly estimated as the distance from the terrain to the foot and the probability of foot contact.
[0045] (1) Design a state estimator.
[0046] The state estimator employs a multilayer perceptron, a temporal convolutional network, or a recurrent neural network structure.
[0047] In this example, a multilayer perceptron with a hidden layer structure of 256×128 is used. The training method is supervised learning. The terrain-related privileged information is estimated through supervised learning, and the estimated values of terrain-related display privileged information are extracted.
[0048] (2) Obtain privilege information.
[0049] Privileged information consists of observable posture data and terrain information of bipedal robots from the training environment, which are difficult to obtain directly during actual deployment.
[0050] In this example, privilege information Including base linear velocity Distance from the ground to your feet and the probability of contact between the two wheel feet and the ground. The formula is: ; in, These represent the linear velocities of the current frame's base along the x, y, and z directions in the world coordinate system, respectively. These represent the distances from the ground to the left and right feet, respectively. Let x, y, and z represent the probabilities of the left and right feet of the bipedal robot contacting the ground in the x, y, and z directions, respectively.
[0051] (3) Obtain the explicit privileged information estimate through the state estimator.
[0052] The explicit privileged information estimates include at least the base linear velocity estimate, the foot-to-ground distance estimate, and the foot-to-ground contact probability estimate.
[0053] The explicit privileged information estimate is calculated using the following formula: ; in, They are respectively for The estimated angular velocity of the base; They are respectively for The estimated distance between the feet and the ground; They are respectively for The estimated probability of foot contact with the ground.
[0054] After the above steps, based on the robot's historical motion state information (including posture sensor data, joint position and velocity data, policy output data, terminal command data, etc. from the past multiple frames), terrain-related privileged information is accurately estimated through supervised learning. The core parameters related to the terrain are extracted from the black box model of reinforcement learning, and the privileged information is explicitly output. This allows the control module to directly call parameters with clear physical meaning that reflect environmental information, providing an interpretable basis for policy optimization.
[0055] (4) Construct the loss function of the supervised learning regression algorithm and optimize the state estimator.
[0056] The loss function of supervised learning regression algorithms can be any of the following without restriction: mean squared error (MSE), root mean squared error (RMSE), or smooth L1 loss.
[0057] In this example, mean squared error is chosen as the loss function, and the formula is: ; in , 。
[0058] like Figure 1 As shown, in step S3, the explicit privileged information estimate is fused with the motion state observation information at the current moment to obtain the policy network input observation space; The policy network is input into the observation space and then into the pre-trained policy network to obtain the action instructions at the current time. (1) The explicit privileged information estimate is fused with the motion state observation information at the current moment to obtain the policy network input observation space.
[0059] The privileged estimate obtained by fitting the state estimator Status information of the current frame The observation space is merged to form a strategy network.
[0060] (2) Input the policy network into the observation space and obtain the action command at the current time.
[0061] Both the policy network and the value network are structured as multilayer perceptrons, each with three hidden layers.
[0062] The policy network is input into the observation space to obtain the motion of the bipedal robot. .
[0063] Then, by using the scaling factor of the motion space, the abstract motion is mapped to the physical joint space, thereby improving the stability of the training process.
[0064] like Figure 1 As shown, in step S4, the motion command is converted into joint torque control command by the proportional-derivative controller, driving the bipedal robot to perform obstacle-crossing actions.
[0065] In this embodiment, the PD controller converts the strategy output into joint torques and sends them to the bipedal robot. The control method used is position and speed control, and the formula is as follows: ; ; in, , These are the joint torques issued to the leg joints and wheel joints, respectively. , These are the control coefficients of the PD controller; , For position control and speed control, the motion scaling factor is used. , These are the leg joint movements and wheel joint movements output by the policy network, respectively. , The default leg joint position and the current leg joint position are set respectively. , These are the joint velocities of the legs and wheels, respectively.
[0066] When a height obstacle is detected ahead, the action sequence output by the policy network triggers the following obstacle-crossing behavior: Preparation stage: The height of the base is raised appropriately and the center of gravity is adjusted.
[0067] Front leg crossing: Quickly raise the wheel leg closest to the obstacle and step forward to cross.
[0068] Follow-up with the rear leg: After the front wheel foot lands and stabilizes, drive the other wheel foot to repeat the "lift-step" action.
[0069] Landing cushioning: After both wheels have crossed the obstacle, the leg joints slowly contract to cushion the impact.
[0070] Posture recovery: Restore the target standing height and continue executing movement commands.
[0071] like Figure 1 As shown, it also includes the training and optimization of the policy network. Based on the output of the value network, the policy network is optimized and trained using a proximal policy optimization algorithm.
[0072] In this example, the policy optimization algorithm used is Proximal Policy Optimization (PPO). While the policy network is being trained, the value network incorporates the motion state information of the current frame. and real privileged information obtained from the simulation environment The neural network calculates and outputs an evaluation of the state value function used to feed back the policy network.
[0073] The training process for the policy network is as follows: (1) Input the real privileged state information of the simulation environment and the motion state observation information at the current moment into the value network to obtain the state value estimate.
[0074] The input information of the value network differs from that of the policy network. It is based on real privileged information that can be directly obtained in the simulation environment, combined with the motion state observation information of the current frame, to evaluate the value of the current motion state and output the state value estimation result.
[0075] The value network continuously optimizes the accuracy of state value estimation based on the input of real privilege information and motion state information of the current frame, providing a reliable reward signal basis for the parameter update of the policy network, and ultimately achieving a synergistic improvement in state estimation accuracy, policy output performance and value assessment accuracy.
[0076] (2) Construct a reward function based on the results of the obstacle crossing action.
[0077] Based on the motion target, a reward function is designed for the bipedal robot to complete the task safely, stably, and with low power consumption. A guiding term is designed in the reward function to enable the bipedal robot to stably cross high obstacles.
[0078] Based on the motion target of the bipedal robot, a reward function and its weights are defined. The formula for the reward function is as follows: ; in, This represents the weighted sum of all rewards. This represents the weighted reward for maintaining the feasibility of the motion through positional constraints, velocity constraints, and various smoothing penalties. This indicates that the weighted approach ensures the rationality of the bipedal robot's motion posture by constraining the base angle, the distance between its two legs, the number of wheels in contact with the ground, and the collision location with the ground. This represents the reward function for the weighted tracking terminal commands. This represents the weighted reward function that the bipedal robot can overcome when encountering a high obstacle.
[0079] The weighted reward function for the bipedal robot when encountering high obstacles. This involves guiding the bipedal robot to overcome obstacles from two aspects: speed detection and force probability analysis. The formula is: ; in, It is a reward function that encourages the bipedal robot to lift its feet when encountering high obstacles. , To determine the matrix for determining whether the duration of foot lifting of a bipedal robot is within a suitable range, It is a matrix used to determine when a robot encounters a height obstacle. yes Inverse of; It is a reward function that encourages bipedal robots to lift their legs to a target height when encountering obstacles. , It is an exponential function of the sum of the differences between the actual leg lift height and the target value. and These are matrices used to determine whether the legs of a bipedal robot are subjected to forces in the x and y directions. and These are the weighting coefficients.
[0080] A reward function that encourages bipedal robots to lift their feet when encountering high obstacles. By detecting the deviation threshold between the base linear velocity and the command linear velocity, the robot can accurately capture the motion stagnation state caused by high obstacles, and use this as a trigger condition to encourage the robot to lift its foot and take a step. Another layer of guidance logic is constructed from the force-sensory interaction dimension: by detecting the force on the wheeled legs in the x and y directions, the constraints of obstacles on the robot's physical motion space are directly perceived, and then this is used as a basis to encourage the bipedal robot to lift its feet to the target height. At a deeper level, the core detection quantities of the two reward functions (base linear velocity deviation and foot force probability) are both included in the privileged information category of the state estimator output. This means that the policy network does not need to painstakingly dig out implicit correlations from the raw sensor data, but directly obtains the refined key features of obstacle interaction. Thus, in the value learning process, a mapping relationship from obstacle signals to action responses is quickly established, and the judgment of the timing of lifting the foot is transformed from a fuzzy exploration based on experience and trial and error to a precise decision based on explicit environmental features, ultimately improving the adaptive response capability to height obstacles.
[0081] (3) Based on state value estimation and reward function, the policy gradient is generated by the near-end policy optimization algorithm and back-propagated to the policy network to iteratively optimize the policy network parameters.
[0082] In step S4, the proportional-derivative controller converts motion commands into joint torque control commands, drives joint motors to generate motion, and changes the robot's real-time state. These states then serve as inputs to the reward function to recalculate the reward, thereby inversely optimizing the policy network so that its output torque commands are continuously adjusted to be closer to the motion target.
[0083] The policy network, based on the fused state information, iteratively updates using a reinforcement learning algorithm to optimize the output accuracy of the control policy. Based on the actions output by the policy network and the state value function output by the value network, as well as the real interaction feedback obtained from trajectory sampling, the PPO algorithm quantifies the action value using a dominance function and constrains the update magnitude with a shearing mechanism. Ultimately, this achieves stable iterations of the policy network towards higher rewards and greater stability.
[0084] The policy gradient is generated using a proximal policy optimization algorithm and backpropagated to the policy network. The policy network parameters are then iteratively optimized. The specific process is as follows: 1) Based on the current policy network, output action instructions according to the robot's real-time status to drive the robot to interact with the environment and collect interaction trajectory data including status, action, immediate reward and next state.
[0085] 2) Based on the interaction trajectory data and combined with state value estimation, the advantage function of the action at each time step is calculated using the generalized advantage estimation algorithm.
[0086] Quantify the quality of the action compared to the average level to clarify the direction for action optimization.
[0087] 3) Calculate the action selection probability ratio of the old and new policy networks for the same action, and construct a policy optimization objective function with pruning constraints based on the probability ratio and the advantage function.
[0088] This measures the magnitude of policy updates, and a pruning mechanism is used to limit the range of policy updates to prevent excessively large update magnitudes from causing training instability.
[0089] 4) With the goal of maximizing the policy optimization objective function, update the parameters of the policy network through gradient optimization methods.
[0090] This makes the network more inclined to output high-dominance actions and suppress low-dominance actions.
[0091] 5) Repeat steps 1) to 4) until the performance of the policy network meets the preset convergence condition, and obtain the trained policy network.
[0092] This process is repeated until the optimized action instructions are obtained.
[0093] After the above steps, the optimal policy network is obtained, which in turn enables more effective action commands, improving the motion stability and efficiency of the bipedal robot in complex environments.
[0094] (4) Use the near-end policy optimization algorithm to generate policy gradients, and back-propagate them to the value network to simultaneously optimize the value network parameters.
[0095] After the policy gradient is solved using the PPO algorithm, the generated policy gradient signal is fed back to the value network through this path. This is used to iteratively optimize the parameters of the value network, improve the accuracy of its state value assessment, and realize bidirectional information flow between state value assessment and policy gradient generation. This further improves the effectiveness of action commands and enhances the motion stability and efficiency of the bipedal robot in complex environments.
[0096] Guided by a reward function, and combined with a reinforcement learning algorithm featuring a state estimator, a policy network is derived based on privileged estimation information and the motion state information of the current frame. The policy network outputs the following action policy for the bipedal robot: When the robot detects an obstacle of height, it performs a crossing action. The crossing action includes the robot standing at a higher height, quickly raising the wheel leg closest to the obstacle and stepping forward. After that wheel leg lands, it drives the other wheel leg to repeat the "raise-step" action. After both wheels have crossed the obstacle, the leg joints slowly contract, causing the base to move downward to buffer and restore the target standing height.
[0097] like Figure 1 As shown, it also includes training the state estimator, using real privileged information obtained in the simulation environment as labels, and based on supervised learning, optimizing the parameters of the state estimator synchronously by minimizing the deviation between the estimated value of explicit privileged information and the real privileged information.
[0098] like Figure 2 As shown, when the bipedal robot faces a high obstacle, a driving obstacle-crossing strategy is obtained through steps S1-S4 to drive the bipedal robot to cross the obstacle.
[0099] When encountering a high obstacle, the robot performs a crossing action guided by privileged estimation information and a reward function to overcome situations where its linear velocity lags significantly behind the target linear velocity and its feet are subjected to horizontal lateral forces. Specifically, after the robot with linear velocity approaches the high obstacle, it quickly raises the wheel closer to the obstacle as its standing height increases. After that wheel lands, it drives the other wheel to repeat the "raise-step" action. Once both wheels have crossed the obstacle, the leg joints slowly contract, causing the base to cushion the impact and return to the target standing height.
[0100] in, Figure 2 (a) In preparation for obstacle detection, the robot determines that there is a height obstacle in front of it by using privileged information such as foot distance and contact probability output by the state estimator, enters the crossing preparation state, and keeps the base level and the wheels and feet stably on the ground; Figure 2 (b) In order to increase the standing height and leave room for crossing, the robot's leg joints extend, the overall standing height increases, and at the same time, it maintains its posture balance and waits for the stepping command; Figure 2(c) Stepping on one side of the wheel, quickly raising the wheel closest to the obstacle and extending it forward above the obstacle to prepare for landing and support, while the other wheel remains on the ground to stabilize the fuselage; Figure 2 (d) Landing on one side of the wheel, the lifted wheel accurately lands on the top of the obstacle or on the ground on the other side, forming a new support point, paving the way for the other wheel to cross; Figure 2 (e) The other wheel foot crosses over, driving the uncrossed wheel foot to repeat the "lift-step" action to cross over the obstacle. At this time, both wheel feet are on the other side of the obstacle or in a stable support state. Figure 2 (f) To buffer the reset, the leg joints slowly contract, causing the base to move downwards to buffer the impact and avoid mechanical shock, eventually restoring the target standing height, completing a full hurdle and preparing for subsequent movements.
[0101] like Figure 3 As shown, the simulation demonstrates the process of a bipedal robot traversing a ditch in complex terrain, following steps S1-S4. Figure 3 (a) For ravine detection and state assessment, the robot analyzes historical motion data (joint velocity, posture sensor information, etc.) through a state estimator, infers that there is a height obstacle in front, determines that it needs to perform a crossing action, and maintains the stability of the initial posture; Figure 3 (b) In preparation for pushing off the ground, the robot adjusts the angle of its leg joints, pushes off the ground with its hind legs to generate power, and lifts its front legs slightly to store kinetic energy for take-off and leap. At the same time, the robot evaluates the rationality of its posture in real time through the value network. Figure 3 (c) During the take-off and crossing phase, both legs extend synchronously, the robot is airborne as a whole, and crosses the ditch. The state estimator continuously updates privileged information such as the base linear velocity and the distance between the feet and the ground to ensure the balance of the airborne posture. Figure 3 (d) To ensure accurate landing of the forelegs, the forelegs first land on the stable area on the other side of the ditch, forming a support point. At this time, the leg joints flexibly cushion the impact of landing. Figure 3 (e) The rear leg follows the landing, retracts from the air position and quickly crosses the ditch, landing behind the front leg. The two wheels form a stable support, completing the core movement of crossing the ditch. Figure 3 (f) is the attitude adjustment reset. The robot adjusts the joint torque through the PD controller, corrects the horizontal attitude of the base, retracts the leg joints to the default position, restores the target standing height, and prepares for subsequent movements.
[0102] like Figure 4 The simulation shows the process of a bipedal robot moving on a 20° rough slope. Among them, Figure 4 (a) For slope detection and initial alignment, the robot analyzes historical motion data (attitude sensor, joint speed, etc.) through the state estimator and concludes that a larger torque output is needed to keep the wheel joints from deviating from control, thus entering the uphill state, with the wheel feet stably touching the ground, and entering the uphill preparation state; Figure 4 (b) For attitude pre-adjustment, the leg joints extend and adapt according to the slope angle, the base tilts slightly forward to counteract the gravity component of the slope, and the wheel joints adjust the initial speed to reserve power for uphill. The state estimator updates the privileged information related to the ground friction coefficient in real time. Figure 4 (c) To ensure stable uphill movement, the wheels and feet work together to exert force, the leg joints maintain posture balance, and the wheel joints output power that adapts to the rough ground to avoid slipping; the value network evaluates the motion state through real privilege information, and the reward function constrains the stability of the base level and wheel speed. Figure 4 (d) Slope attitude calibration: due to the rough ground causing slight attitude deviation, the PD controller quickly adjusts the leg joint torque, corrects the base angle, and updates the wheel-foot contact probability in real time to ensure that the two wheels always stably fit the slope. Figure 4 (e) To adapt to the approaching top of the slope, as the slope is about to end, the state estimator predicts the terrain change, and the strategy network adjusts the wheel speed and leg joint angle to reduce the uphill power output, prepare for a smooth transition, and avoid over-rushing the slope. Figure 4 (f) To achieve a smooth reset, the robot arrives at the flat area at the top of the slope, retracts its leg joints to the default position, restores the base to the horizontal target posture, and switches the wheel joints to the flat ground movement mode, completing the full adaptation of the slope scene and preparing for subsequent movements.
[0103] Example 2 The purpose of this embodiment is to provide an obstacle-crossing control system for a bipedal robot based on explicit state estimation, including: The data acquisition module is used to acquire the historical motion state observation sequence and the current motion state observation information of the bipedal robot. The state estimation module is used to input the historical motion state observation sequence into the state estimator to obtain the explicit privileged information estimate; The policy execution module is used to fuse the explicit privileged information estimate with the motion state observation information at the current moment to obtain the policy network input observation space; and input the policy network input observation space into the pre-trained policy network to obtain the action command at the current moment. The control output module is used to convert motion commands into joint torques via a proportional-derivative controller, driving the bipedal robot to perform obstacle-crossing actions.
[0104] The method steps in Embodiment 1 are implemented based on the provided obstacle-crossing control system for a bipedal robot with explicit state estimation.
[0105] It also includes a training module for co-training the policy network and the state estimator, wherein the training module includes: The supervised learning unit is used to optimize the state estimator through supervised learning based on the estimated values of explicit privileged information.
[0106] The reinforcement learning unit is used to optimize and train the policy network based on the output of the value network through a proximal policy optimization algorithm.
[0107] Example 3 The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.
[0108] Example 4 The purpose of this embodiment is to provide a computer-readable storage medium.
[0109] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.
[0110] Example 5 The purpose of this embodiment is to provide a computer program product containing instructions that, when run on a computer, causes the computer to perform the methods and functions involved in any of the embodiments described above.
[0111] The steps and methods involved in the apparatus of the above embodiments correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.
[0112] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.
[0113] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.
Claims
1. A method for obstacle-crossing control of a biwheeled legged robot based on explicit state estimation, characterized in that, include: Acquire the historical motion state observation sequence and the current motion state observation information of the biwheeled legged robot; The historical motion state observation sequence is input into the state estimator to obtain the explicit privileged information estimate; The explicit privileged information estimate is fused with the motion state observation information at the current moment to obtain the policy network input observation space; The policy network is input into the observation space and then into the pre-trained policy network to obtain the action instructions at the current time. The proportional-derivative controller converts motion commands into joint torque control commands, driving the biwheeled legged robot to perform obstacle-crossing actions. The policy network is trained based on the output of the value network, and is optimized and trained using a proximal policy optimization algorithm.
2. The obstacle-crossing control method for a bipedal robot based on explicit state estimation as described in claim 1, characterized in that, The motion state observation information includes at least the base angular velocity, projected gravity, joint position, joint velocity, motion command and control command of the previous control cycle.
3. The obstacle-crossing control method for a bipedal robot based on explicit state estimation as described in claim 1, characterized in that, The historical motion state observation sequence is input into the state estimator, which uses a multilayer perceptron, temporal convolutional network, or recurrent neural network structure to estimate terrain-related privileged information through supervised learning and extract the estimated values of explicit privileged information related to the terrain.
4. The obstacle-crossing control method for a bipedal robot based on explicit state estimation as described in claim 3, characterized in that, The explicit privileged information estimates include at least the base linear velocity estimate, the foot-to-ground distance estimate, and the foot-to-ground contact probability estimate.
5. The obstacle-crossing control method for a bipedal robot based on explicit state estimation as described in claim 1, characterized in that, Based on the output of the value network, the policy network is optimized and trained using a proximal policy optimization algorithm. The specific process is as follows: The real privileged state information of the acquired simulation environment and the motion state observation information at the current moment are input into the value network to obtain the state value estimate; Construct a reward function based on the results of the obstacle-crossing action; Based on state value estimation and reward function, a proximal policy optimization algorithm is used to generate policy gradients, which are then backpropagated to the policy network to iteratively optimize the policy network parameters.
6. The obstacle-crossing control method for a bipedal robot based on explicit state estimation as described in claim 1, characterized in that, The reward function formula is: ； in, This represents the weighted sum of all rewards. This represents the weighted reward for maintaining the feasibility of the motion through positional constraints, velocity constraints, and various smoothing penalties. This indicates that the weighted approach ensures the rationality of the bipedal robot's motion posture by constraining the base angle, the distance between its two legs, the number of wheels in contact with the ground, and the collision location with the ground. This represents the reward function for the weighted tracking terminal commands. This represents the weighted reward function that the bipedal robot can overcome when encountering a high obstacle.
7. The obstacle-crossing control method for a bipedal robot based on explicit state estimation as described in claim 1, characterized in that, The weighted reward function for the bipedal robot when encountering a high obstacle is given by the following formula: ； in, It is a reward function that encourages the bipedal robot to lift its feet when encountering high obstacles. , To determine the matrix for determining whether the duration of foot lifting of a bipedal robot is within a suitable range, It is a matrix used to determine when a robot encounters a height obstacle. yes Inverse of; It is a reward function that encourages bipedal robots to lift their legs to a target height when encountering obstacles. , It is an exponential function of the sum of the differences between the actual leg lift height and the target value. and These are matrices used to determine whether the legs of a bipedal robot are subjected to forces in the x and y directions. and These are the weighting coefficients.
8. The obstacle-crossing control method for a bipedal robot based on explicit state estimation as described in claim 1, comprising: The motion command is converted into a joint torque control command through a proportional-derivative controller. The control method used is position and velocity control, and the formula is as follows: ； ； in, , These are the joint torques issued to the leg joints and wheel joints, respectively. , These are the control coefficients of the PD controller; , For position control and speed control, the motion scaling factor is used. , These are the leg joint movements and wheel joint movements output by the policy network, respectively. , The default leg joint position and the current leg joint position are set respectively. , These are the joint velocities of the legs and wheels, respectively.
9. A bipedal robot obstacle-crossing control system based on explicit state estimation, characterized in that, include: The data acquisition module is used to acquire the historical motion state observation sequence and the current motion state observation information of the bipedal robot. The state estimation module is used to input the historical motion state observation sequence into the state estimator to obtain the explicit privileged information estimate; The policy execution module is used to fuse the explicit privileged information estimate with the motion state observation information at the current moment to obtain the policy network input observation space; and input the policy network input observation space into the pre-trained policy network to obtain the action command at the current moment. The control output module is used to convert motion commands into joint torques via a proportional-derivative controller, driving the bipedal robot to perform obstacle-crossing actions.
10. The obstacle-crossing control system for a bipedal robot based on explicit state estimation as described in claim 9, characterized in that, It also includes a training module for co-training the policy network and the state estimator, wherein the training module includes: The supervised learning unit is used to optimize the state estimator through supervised learning based on the estimated value of explicit privileged information. The reinforcement learning unit is used to optimize and train the policy network based on the output of the value network through a proximal policy optimization algorithm.