A phase-driven multi-critic humanoid robot sit-to-stand transition control model
Patent Information
- Application Number
- CN202610847903.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-12
- Publication Date
- 2026-08-04
Smart Images
Figure CN122500719A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of humanoid robot motion control technology, specifically to a stage-driven multi-Critic humanoid robot sit-to-stand switching control model. Background Technology
[0002] With the rapid development of humanoid robot technology, humanoid robots are gradually expanding from structured industrial environments to scenarios such as home services, elderly care, medical assistance, and complex human-computer interactions. In these application scenarios, sitting down and standing up are among the most basic and frequent movement behaviors of humanoid robots, which not only directly affect the robot's autonomous movement capabilities but also determine the robot's interaction stability and safety in complex environments.
[0003] However, compared to traditional walking tasks, the humanoid robot's sitting-to-standing transition involves a large-scale shift in the center of gravity, changes in body posture, and complex changes in the contact relationship between the robot and the environment, making it a typical high-dimensional nonlinear complex contact control problem. Especially during the robot's contact with the seat and its return to standing, the robot needs to simultaneously satisfy multiple control objectives, including posture stability, smooth contact, continuous motion, and dynamic safety. These objectives often conflict significantly. For example, quickly completing the sitting action can easily lead to increased contact impact, while overemphasizing contact stability can reduce motion efficiency. Therefore, traditional single-objective control methods are insufficient to address the multiple control requirements in complex sitting-to-standing tasks.
[0004] Most existing methods for controlling humanoid robots to sit or stand employ trajectory planning, finite state machines, or single-value network reinforcement learning structures. Trajectory planning methods typically rely on manually designed reference trajectories, resulting in weak generalization ability and difficulty adapting to different seat heights and complex contact environments. Finite state machine methods require extensive manual design of switching rules, leading to insufficient robustness of the control model. Traditional single-criteria reinforcement learning methods generally map all rewards to a single value network for optimization, which can easily lead to interference between different reward objectives in complex contact tasks, resulting in unstable value estimation, training oscillations, and difficulties in policy convergence.
[0005] Furthermore, during the sitting and standing process of a high-center-of-gravity humanoid robot, phenomena such as pelvic shift, instability of the center of gravity, and sudden changes in speed at the moment of contact are prone to occur before contact, further increasing the difficulty of control. Especially during the descent of the robot from a standing to a sitting posture, traditional symmetrical potential field or simple target point tracking methods can easily cause the robot to tilt excessively forward, thus creating a risk of falling.
[0006] Therefore, how to construct a humanoid robot sitting-standing transition control model that can adapt to complex contact conditions, reduce multi-objective optimization conflicts, improve contact stability, and take into account motion continuity has become an important problem that urgently needs to be solved in the field of humanoid robot motion control. Summary of the Invention
[0007] To address the shortcomings of existing technologies, this invention provides a stage-driven multi-Critic humanoid robot sit-stand switching control model, which solves the problems mentioned in the background section.
[0008] To achieve the above objectives, the present invention specifically adopts the following technical solution: A stage-driven control model for the sit-to-stand transition of a multi-Critic humanoid robot includes the following steps: Step 1: Construct a humanoid robot sitting and standing control environment and collect information on robot pelvic height, joint angle, joint speed, foot contact status, and seat contact status. Step 2: Establish a phase-driven control architecture, dividing the robot sitting and standing process into a guided descent phase, a stable sitting phase, and a dynamic standing phase, and switching between phases according to the robot's motion state. Step 3: Construct a functionally decoupled Multi-Critic value assessment network, including task evaluation, action style evaluation, and constraint evaluation; Step 4: Use the stage reward activation mechanism and reinforcement learning PPO algorithm to complete the control strategy update training and realize the sitting and standing control of the humanoid robot.
[0009] Furthermore, this invention uses Unitree Robotics' G1 humanoid robot as the experimental subject. The robot possesses a dual-leg multi-degree-of-freedom joint structure and human-like movement capabilities, enabling it to perform complex movements such as standing, sitting, and rising. The robot's lower limb joints employ a motor-driven structure and possess joint position, velocity, and torque feedback capabilities, providing robot body state information for reinforcement learning control.
[0010] Furthermore, this invention employs the Isaac Gym parallel physics simulation platform to construct the robot training environment, simultaneously running multiple robot environments via GPU parallel acceleration to improve reinforcement learning training efficiency. The simulation environment includes a robot model, a ground contact model, and a seat model, where seat height, contact stiffness, and friction coefficient can be randomly varied to enhance the adaptability of the control strategy under different environmental conditions. Furthermore, this invention constructs the robot sitting / standing task as a Markov decision process: ; Where M represents the Markov decision process corresponding to the robot's sit-stand task, S represents the robot's state space, A represents the action space, P represents the state transition probability function, and R represents the reward function. This represents the discount factor. The robot adjusts the discount factor based on the current state at each discrete control time step. Output Action The environment is updated to the next state based on the robot's actions. and return the corresponding instant reward. Furthermore, the robot state space includes robot joint states, pelvic states, and contact state information, and its state space is defined as follows: ; in, This indicates the robot's joint angle information. This indicates the robot's joint velocity information. This indicates the robot's pelvic height information. This indicates the contact status of the robot's feet and the contact status of the seat.
[0011] Furthermore, the motion space adopts the residual motion form of the target joint position. At each discrete control time step t, the reinforcement learning policy network outputs the robot joint target position residual: ; The target joint position of the robot is defined as: ; in, This represents the target joint position residual output by the policy network. Indicates the robot's reference joint position. This represents the target joint position of the robot. By employing residual motion, the motion search space complexity of the reinforcement learning strategy is reduced, thus improving training stability.
[0012] Furthermore, the hierarchical cascaded control module adopts a control structure combining a high-level reinforcement learning strategy and a low-level PD controller. The low-level PD controller is used to track the target joint position in real time, and its driving torque is defined as: ; in, This represents the driving torque of the robot joints, and q represents the current actual joint position. and These represent the target joint angular velocity and the current actual joint angular velocity, respectively. Represents the proportional gain coefficient. This represents the damping gain coefficient. The proportional term is used to drive the robot joints to quickly approach the target position, while the differential term is used to suppress velocity oscillations and high-frequency jitter during joint movement, thereby improving control stability under complex contact conditions. Compared to reinforcement learning control methods that directly output joint torque, this invention employs a target position cascaded control structure, which can effectively suppress large-amplitude torque oscillations caused by random exploration in the early stages of training and improve the smoothness of robot motion control.
[0013] Furthermore, the stage-driven control module divides the sitting-to-standing process into a guided descent stage, a stable seating stage, and a dynamic standing stage based on the robot's motion state. The guided descent stage controls the robot's center of gravity to move backward and gradually approach the seat area; the stable seating stage reduces collision impact during contact between the robot and the seat; and the dynamic standing stage restores the robot to a stable standing posture. The different stages are dynamically switched based on the robot's pelvic height, pelvic speed, and seat contact state.
[0014] Furthermore, addressing the issue that traditional symmetric field methods easily lead to anterior displacement of the robot's pelvis, this invention proposes an asymmetric rear gravitational potential field to guide the robot's descent. The center position of both feet is defined as: ; in, The center of the robot's bipedal support area. Indicates the position of the left foot. This indicates the position of the right foot. The posterior displacement of the pelvis is defined as: ; in, Indicates the position of the pelvis in the front-to-back direction. This represents the maximum allowed backward movement distance. Further, an asymmetric backward potential field reward is constructed: ; in, denoted as the potential field attenuation coefficient. Compared to the traditional method of directly descending to the target position, this invention prioritizes constraining the robot's pelvis to move backward, enabling the robot to form a more stable backward movement trend during the descent phase, thereby reducing the problem of robot contact-induced forward tilting instability.
[0015] Furthermore, this invention constructs a functionally decoupled Multi-Critic reinforcement learning structure oriented towards multi-objective optimization constraints. Since the sit-to-stand task involves multiple control objectives simultaneously, such as target tracking, motion smoothing, and contact safety, traditional single-Critic structures struggle to stably estimate different types of rewards concurrently. Therefore, this invention divides the reward function into three categories: task reward, motion style reward, and constraint reward, and constructs independent value networks for each category to estimate long-term rewards. Specifically, task rewards guide the robot to complete the sit-to-stand objective, including pelvic height tracking rewards, stage completion rewards, and target position rewards; motion style rewards improve the naturalness of robot movements, including motion smoothing rewards, body sway suppression rewards, and motion continuity rewards; and constraint rewards improve robot control safety, including joint torque constraints, joint velocity constraints, and contact point constraints.
[0016] Furthermore, the Multi-Critic reinforcement learning structure employs independent value networks to estimate the long-term value of different rewards, and its overall advantage function is expressed as: ; in, This represents the advantage function estimate of the j-th evaluation network at each discrete control time step t, used to evaluate the current state. With action Contributions in long-term cumulative rewards; This represents the weight coefficient of the corresponding evaluation network, used to adjust the degree of influence of different evaluation objectives in the strategy optimization process; These represent task evaluation, action style evaluation, and constraint evaluation, respectively. Compared to traditional single-crit reinforcement learning structures, this invention reduces optimization conflicts between different reward objectives through functional decoupling, thereby improving the stability of value estimation and policy convergence efficiency in complex contact tasks.
[0017] Furthermore, this invention proposes a dynamic switching of stage rewards for robot motion states. Since the robot corresponds to different control objectives at different motion stages, the control model dynamically activates the corresponding reward function based on the robot's real-time state. The total reward function is expressed as: ; Where r represents the robot's total reward. Indicates task reward; Indicates a reward for action style; This indicates a constraint on rewards.
[0018] During the guided descent phase, the control model primarily activates rewards for pelvic displacement and target height tracking; during the stable seating phase, it primarily activates rewards for contact velocity constraints and collision suppression; and during the dynamic standing phase, it primarily activates rewards for standing stability and posture recovery. By dynamically switching rewards, the reinforcement learning strategy focuses on the corresponding control objective at different stages, thereby reducing ineffective exploration during training.
[0019] Furthermore, this invention employs the PPO reinforcement learning algorithm to update the control policy, and its optimization objective function is defined as: ; in, This represents the PPO pruning objective function. This represents the ratio of the probabilities of the new and old strategies. This represents the estimated value of the advantage function. This represents the cutting factor.
[0020] By limiting the policy update magnitude through the PPO pruning mechanism, the stability during training can be improved.
[0021] Furthermore, in the early stages of training, the present invention employs a low level of contact difficulty and fixed seat parameters to enable the robot to learn basic sitting and standing movements first. As training progresses, the range of seat height variation, random disturbances of contact friction, and random disturbances of initial posture are gradually increased to improve the adaptability of the control strategy to complex environments.
[0022] Furthermore, this invention employs ablation experiments to verify the effectiveness of different modules. The experiments removed stage-driven mechanisms, the Multi-Critic structure, and stage reward switching, and compared the results with the complete model. Specifically, removing stage-driven mechanisms made it difficult for the robot to achieve a stable center-of-gravity transfer process, easily leading to premature contact and forward tilting instability. Removing the Multi-Critic structure resulted in significant optimization conflicts between different reward objectives, increasing oscillations during training. Removing stage reward switching caused numerous invalid explorations during training, significantly slowing down the training convergence speed.
[0023] Compared with existing technologies, this invention provides a stage-driven multi-Critic humanoid robot sit-stand switching control model, which has the following beneficial effects: This invention constructs a phase-driven control system based on physical phase divisions, dividing the robot's sitting-to-standing process into a guided descent phase, a stable seating phase, and a dynamic standing phase. It dynamically switches the corresponding control target based on the robot's real-time motion state, enabling the robot to focus on different control tasks at different motion stages. Compared to traditional unified control methods, this invention effectively reduces control confusion under complex contact conditions and improves the continuity and stability of the robot's movements during the sitting-to-standing transition. It also improves the state guidance method using an asymmetric rearward gravitational potential field by prioritizing the robot's pelvic posterior displacement trend, guiding the robot to form a stable backward movement before contacting the seat, thus reducing the forward tilting instability problem during the descent phase. Compared to traditional direct target point tracking methods, this invention achieves autonomous sitting-to-standing control of the robot without relying on manual motion trajectories or motion capture data, improving the environmental adaptability of the control strategy.
[0024] This invention constructs a functionally decoupled Multi-Critic reinforcement learning structure, dividing the reward function into task reward, action style reward, and constraint reward, and employing independent value networks for long-term value estimation for each. Compared to traditional single-Critic reinforcement learning structures, this invention effectively reduces optimization conflicts between different reward objectives, improves the stability of value estimation and policy convergence efficiency in complex contact tasks, thereby reducing oscillations during training.
[0025] This invention proposes a dynamic switching of stage rewards for robot motion states. The corresponding reward function is dynamically activated based on the robot's current motion stage, allowing the robot to focus on the corresponding control objective at different stages. Specifically, during the guided descent stage, the focus is on shifting the center of gravity backward to approach the target; during the stable sitting stage, the focus is on contact impact suppression; and during the dynamic standing stage, the focus is on standing stability and posture recovery. Compared to traditional fixed reward methods, this invention effectively reduces ineffective exploration during training, improving reinforcement learning training efficiency and task success rate.
[0026] This invention employs a hierarchical cascaded control structure combining a high-level reinforcement learning strategy with a low-level PD controller. The high-level strategy generates the target joint position, while the low-level PD controller handles stable joint tracking. Compared to control methods that directly output joint torque, this invention effectively reduces the complexity of the reinforcement learning action search space, minimizes high-frequency torque oscillations caused by random exploration in the early stages of training, and improves the smoothness of motion, contact stability, and overall control robustness during the robot's sit-to-stand transition. Attached Figure Description
[0027] Figure 1 This is a flowchart of a stage-driven multi-Critic humanoid robot sit-to-stand switching control model according to the present invention. Figure 2This is a general framework diagram of a stage-driven multi-Critic humanoid robot sit-stand switching control model of the present invention. Figure 3 This is a schematic diagram illustrating the stage division of a stage-driven multi-Critic humanoid robot sit-stand switching control model according to the present invention. Figure 4 This is a schematic diagram illustrating the success rate of an experimental multi-Critic humanoid robot sitting-standing transition control model driven by the present invention. The experimental seat models include: office chair, recliner, folding chair, bar stool, stool, and sofa. Figure 5 This is a schematic diagram comparing the stage-driven multi-Critic humanoid robot sit-to-stand switching control model of the present invention with other methods; Figure 6 This is a schematic diagram showing the ablation experiment comparison of a stage-driven Multi-Critic humanoid robot sit-stand switching control model according to the present invention. Figure 7 This is a schematic diagram of an experimental multi-Critic humanoid robot sitting-standing switching control model driven by the present invention. The experimental seats, from top to bottom, are: folding chair, bar stool, office chair, recliner, stool, and sofa. Detailed Implementation
[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example
[0029] like Figure 1-7 As shown in the figure, an embodiment of the present invention proposes a stage-driven multi-Critic humanoid robot sit-stand switching control model, which includes the following steps: Step 1: Construct a humanoid robot sitting / standing control environment and collect status information; The experiment used the Unitree Robotics G1 humanoid robot as the experimental subject. The robot has a bipedal multi-degree-of-freedom drive structure and humanoid movement capabilities, enabling it to perform standing, sitting, and standing movements under complex contact conditions. The robot's lower limb joints are driven by motors, and the system can acquire real-time status information such as joint angles, joint velocities, and joint drive torques.
[0030] To improve parallel training efficiency, this embodiment constructs a robot reinforcement learning training environment based on Isaac Gym and uses GPU parallel physics simulation to run multiple robot instances simultaneously. The simulation environment includes a robot model, a planar ground, and a seat model. The seat height, friction coefficient, and initial robot posture are all randomized to improve the generalization ability of the control strategy in complex environments.
[0031] During the training phase, the PPO reinforcement learning algorithm was used to train the robot's sit-stand control strategy. Both the policy network and the value network adopted a multi-layer neural network structure. During training, the control frequency was set to 50Hz to ensure the robot's control stability under complex contact conditions.
[0032] First, we construct a sit-stand reinforcement learning environment for the robot and describe the robot's sit-stand task as a Markov decision process: ; Where M represents the Markov decision process corresponding to the robot's sit-stand task, S represents the robot's state space, A represents the action space, P represents the state transition probability function, and R represents the reward function. This represents the discount factor. The robot adjusts the discount factor based on the current state at each discrete control time step. Output Action The environment is updated to the next state based on the robot's actions. and return the corresponding instant reward. The robot's state space is defined as follows: ; in, This indicates the robot's joint angle information. This indicates the robot's joint velocity information. This indicates the robot's pelvic height information. This represents the robot's foot contact state and seat contact state information. The reinforcement learning strategy optimizes the robot's sitting-to-standing transition control strategy by maximizing long-term cumulative rewards. Its objective function is expressed as: ; in, Representation strategy Long-term cumulative reward expectation, Indicates the discount factor. This represents the immediate reward corresponding to time step t, where T represents the total time steps of the task. By maximizing the long-term cumulative reward, the robot's sitting-standing switching control strategy is optimized.
[0033] Under complex contact conditions, the robot's state space further includes pelvic pose information, body angular velocity information, and historical motion information to improve the reinforcement learning policy's ability to perceive the robot's dynamic state. The robot's motion space adopts the residual motion form of the target joint position. At each discrete control time step t, the reinforcement learning policy network outputs the robot's joint target position residual: ; in, This represents the target joint position residual; the robot target joint position is defined as: ; in, This indicates the robot's nominal initial posture or the reference joint position at the previous moment. This indicates the target joint position of the robot. Compared to methods that directly output the robot joint driving torque, this embodiment uses the target position residual method, which can effectively reduce the complexity of the reinforcement learning action search space and reduce the control oscillation problem caused by random exploration in the early stage of training.
[0034] For the robot control method, the underlying controller uses proportional-derivative control to complete the robot joint tracking control, and its driving torque is expressed as: ; in, This represents the driving torque of the robot joints, and q represents the current actual joint position. and These represent the target joint angular velocity and the current actual joint angular velocity, respectively. Represents the proportional gain coefficient. This represents the damping gain coefficient. The proportional term is used to drive the robot joints to quickly approach the target position, while the differential term is used to suppress velocity oscillations and high-frequency jitter during robot motion, thereby improving control stability under complex contact conditions.
[0035] Step 2: Establish a phase-driven control architecture, dividing the robot's sitting and standing process into a guided descent phase, a stable sitting phase, and a dynamic standing phase; Because the humanoid robot exhibits distinct phased dynamic characteristics during the sitting and standing process, this embodiment employs a phase-driven control architecture to model the robot's sitting and standing task. This embodiment divides the robot's sitting and standing process into a guided descent phase, a stable seating phase, and a dynamic standing phase. The guided descent phase primarily controls the robot's pelvis to move backward and gradually approach the seat area; the stable seating phase primarily reduces the contact impact between the robot and the seat; and the dynamic standing phase primarily restores the robot to a stable standing posture.
[0036] In this embodiment, the phase switching is dynamically determined based on the robot's pelvic height, vertical pelvic velocity, and seat contact state. During the descent phase of the humanoid robot, traditional symmetrical potential field control methods can easily cause the robot's pelvis to shift forward, resulting in instability due to forward tilting before contact. To address this issue, this embodiment proposes an asymmetric rear-mounted gravitational potential field to guide the robot's descent. The center position of both feet is defined as: ; in, The center of the robot's bipedal support area. Indicates the position of the left foot. Indicates the position of the right foot; the posterior displacement of the pelvis is defined as: ; in, Indicates the position of the pelvis in the front-to-back direction. This represents the maximum allowed backward movement distance; further, an asymmetric backward potential field reward is constructed: ; in, The potential field attenuation coefficient is denoted as . This embodiment prioritizes constraining the robot's pelvis's backward movement trend, ensuring the robot first shifts its center of gravity backward during the descent phase before gradually approaching the seat area, thus reducing the robot's forward tilting instability before contact. Once the robot's pelvis approaches the target height and establishes contact with the seat, the control model enters the stable seating phase. Because sudden velocity changes and collision impacts are prone to occur during contact establishment, this embodiment further introduces a contact velocity constraint mechanism. By limiting the rate of change of the robot's pelvis's vertical velocity, it reduces the high-frequency oscillation problem at the moment of contact.
[0037] Once the robot achieves a stable sitting posture, the control model enters the dynamic standing phase. In this phase, the main control objectives include restoring standing height, maintaining body balance, and ensuring stable foot support. The control model guides the robot to gradually regain a stable standing posture by activating standing stability and posture recovery rewards.
[0038] Step 3: Construct a functionally decoupled Multi-Critic value assessment network, including task evaluation, action style evaluation, and constraint evaluation; Since robot sitting and standing tasks involve multiple optimization objectives such as target tracking, motion smoothing, and contact stability, traditional single-crit reinforcement learning structures are prone to mutual interference between different rewards, which leads to increased oscillations during training and difficulty in policy convergence.
[0039] To address the aforementioned issues, this embodiment proposes a functionally decoupled Multi-Critic reinforcement learning structure. This embodiment divides the reward function into three categories: task reward, action style reward, and constraint reward, and uses independent value networks for long-term reward estimation for each. Specifically, the task reward guides the robot to achieve the sitting / standing objective, including pelvic height tracking rewards, target position rewards, and stage completion rewards; the action style reward improves the naturalness of the robot's movements, including action smoothing rewards, body sway suppression rewards, and motion continuity rewards; and the constraint reward prevents the robot from exhibiting significant forward tilting at the moment of contact, including joint torque constraints, joint velocity constraints, and contact stability constraints. The Multi-Critic value network employs an independent value function structure: ; Where V represents the evaluation network of the reward module, This represents the task evaluation network. This indicates a movement style evaluation network. This represents a constrained evaluation network, where each Critic is only responsible for evaluating the long-term returns of its corresponding reward module: ; in, Indicates the discount factor. Let represent the reward corresponding to reward i, and T represent the total time steps of the task. Compared to estimating the long-term value of corresponding rewards separately using different value networks, its overall advantage function is expressed as: ; in, This represents the advantage function estimate of the j-th evaluation network at each discrete control time step t, used to evaluate the current state. With action Contributions in long-term cumulative rewards; This represents the weight coefficient of the corresponding evaluation network, used to adjust the degree of influence of different evaluation objectives in the strategy optimization process; These represent task evaluation, action style evaluation, and constraint evaluation, respectively. Compared to the traditional single-criteria structure, this embodiment reduces optimization conflicts between different reward objectives through functional decoupling, enabling different rewards to be stably estimated in independent value spaces, thereby improving training stability and policy convergence efficiency in complex contact tasks.
[0040] Step 4: Use the stage reward activation mechanism and reinforcement learning PPO algorithm to complete the control policy update training and realize the sitting and standing control of the humanoid robot; Since the robot corresponds to different control objectives at different motion stages, this embodiment further proposes a dynamic switching mechanism for stage rewards of robot motion states. In this embodiment, the total reward function is defined as: ; Where r represents the robot's total reward. Indicates task reward; Indicates a reward for action style; The system represents a constraint-based reward mechanism. During the guided descent phase, the control model primarily activates rewards for pelvic displacement and target height tracking; during the stable seating phase, it primarily activates rewards for contact velocity constraints and collision suppression; and during the dynamic standing phase, it primarily activates rewards for standing stability and posture recovery. This dynamic reward switching mechanism allows the reinforcement learning strategy to focus on the corresponding control objective at different stages, thereby reducing ineffective exploration during training and improving training efficiency.
[0041] The PPO reinforcement learning algorithm is used to update the robot control policy. Its optimization objective function is expressed as: ; in, This represents the PPO pruning objective function. This represents the ratio of the probabilities of the new and old strategies. This represents the estimated value of the advantage function. This represents the cutting factor.
[0042] By limiting the policy update magnitude through the PPO pruning mechanism, the stability during training is improved. During training, this embodiment further employs an environment randomization training method, including randomization of seat height, friction coefficient, and robot initial posture, to enhance the control strategy's adaptability to complex environments.
[0043] Figure 4 Experimental results show that the stage-driven mechanism, functional decoupling Multi-Critic structure, and stage reward dynamic switching mechanism proposed in this embodiment can effectively improve the training stability, motion continuity, and contact control stability of humanoid robots in the sit-to-stand transition task.
[0044] Figure 6This diagram illustrates the ablation experiment, used to verify the effectiveness of different modules. The experiment removed the stage-driven mechanism, the Multi-Critic structure, and the dynamic stage reward switching mechanism, and compared them with the complete model. Experimental results show that: after removing the stage-driven mechanism, the robot struggles to achieve a stable backward shift of its center of gravity, easily exhibiting premature contact and forward tilting instability; after removing the Multi-Critic structure, significant optimization conflicts arise between different reward objectives, increasing oscillations during training; and after removing the dynamic stage reward switching mechanism, the robot exhibits numerous ineffective explorations during training, leading to a significant decrease in training convergence speed.
[0045] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A stage-driven multi-Critic humanoid robot sit-stand switching control model, characterized in that, Includes the following steps: Step 1: Construct a humanoid robot sitting and standing control environment and collect information on robot pelvic height, joint angle, joint speed, foot contact status, and seat contact status. Step 2: Establish a phase-driven control architecture, dividing the robot sitting and standing process into a guided descent phase, a stable sitting phase, and a dynamic standing phase, and switching between phases according to the robot's motion state. Step 3: Construct a functionally decoupled Multi-Critic value assessment network, including task evaluation, action style evaluation, and constraint evaluation; Step 4: Use the stage reward activation mechanism and reinforcement learning PPO algorithm to complete the control strategy update training and realize the sitting and standing control of the humanoid robot.
2. The stage-driven multi-Critic humanoid robot sit-stand switching control model according to claim 1, characterized in that: In step 1, the control environment is built using the experimental platform Isaac Gym, and reinforcement learning training is completed in parallel using GPUs. The robot model adopts the Unitree G1 humanoid robot model. The robot has a dual-leg multi-degree-of-freedom joint structure and humanoid movement ability. The maximum joint torque is 120 N.m, and it can complete complex movement behaviors such as standing, sitting and standing up. The robot's lower limb joints adopt a motor-driven structure and have the ability to provide feedback on joint position, speed and torque, providing robot body state information for reinforcement learning control; Based on this, the Isaac Gym parallel physics simulation platform was used to build a robot training environment. By using GPU parallel acceleration, the synchronous operation of multiple robot environments was achieved, thereby improving the efficiency of reinforcement learning training. The simulation environment includes a robot model, a ground contact model, and a seat model. The seat height and contact stiffness vary with the model to improve the adaptability of the control strategy under different environmental conditions. Dynamic randomization is introduced during training, including factors such as mass disturbance, friction coefficient disturbance, and control time delay, to enhance the robustness of the strategy in complex contact environments.
3. The stage-driven multi-Critic humanoid robot sit-stand switching control model according to claim 1, characterized in that: In step 2, the control architecture is constructed as a Markov decision process, which is typically defined as a quintuple: ； Where M represents the Markov decision process corresponding to the robot's sit-to-stand task, S represents the state space, which contains all possible motion states of the robot during the sit-to-stand transition; A represents the action space, which represents the joint control actions that the robot can perform in the current state; P represents the state transition probability function, which describes the probability distribution of the robot transitioning from the current state to the next state after performing an action; and R represents the reward function, which is used to evaluate the control effect corresponding to the robot's current action. The discount factor measures the importance of future rewards to the optimization of the current strategy; the robot adjusts its strategy according to the current state at each discrete control time step t. The corresponding action is output through the policy network. The environment is updated to the next state based on the action, and the corresponding immediate reward is returned. The robot's state space is defined as follows: ； in, This indicates the robot's joint angle information. This indicates the robot's joint velocity information. This indicates the robot's pelvic height information. This represents the robot's foot contact state and seat contact state information; the optimization objective of the reinforcement learning strategy is expressed as: ； in, Representation strategy Long-term cumulative reward expectation, Indicates the discount factor. This represents the immediate reward corresponding to time step t, where T represents the total time steps of the task. By maximizing the long-term cumulative reward, the robot's sitting-standing switching control strategy is optimized.
4. The stage-driven multi-Critic humanoid robot sit-stand switching control model according to claim 1, characterized in that: The stage-driven control architecture in step 2 switches stages based on the robot's pelvic height, pelvic vertical velocity, and contact state. Specifically: the guided descent stage is used to guide the robot to form a stable backward descent action; the stable seating stage is used to reduce the contact impact between the robot and the seat; and the dynamic standing stage is used to restore the robot to a stable standing state. During the overall sitting-to-standing transition, the corresponding motion states at different stages are quite different. If all reward functions are kept running synchronously throughout the training process, the strategy is prone to interference between different objectives, leading to ineffective exploration and reduced training stability. Therefore, it is necessary to dynamically activate the reward module for the corresponding stage based on the robot's current motion state. The robot's phase transition is determined based on pelvic height, pelvic vertical velocity, foot contact status, and seat contact status information. During the guided descent phase, the robot is guided to form a stable descent motion using an asymmetric rearward gravitational potential field to guide pelvic posterior displacement, target height tracking, and posture stability-related rewards. After establishing stable contact with the seat, the robot switches to the stable seating phase, activating contact velocity constraints and posture stability rewards to reduce collision impact and maintain body stability. During the dynamic standing phase, rewards related to standing height recovery, motion smoothing, and lower limb stability are activated to restore the robot to a stable standing posture. The reward activation weights for different phases are defined as follows: ； in, This represents the reward at time t. This indicates the current stage the robot is in. This indicates the activation weight of the reward corresponding to the stage. This represents the reward corresponding to reward i; it enables the policy to focus on the corresponding control objective at different stages, thereby reducing ineffective optimization during training and improving training stability in complex contact tasks; at the same time, different Critics only estimate the value of the currently activated reward, thereby further reducing the mutual influence between different objectives.
5. The stage-driven multi-Critic humanoid robot sit-stand switching control model according to claim 1, characterized in that: In step 3, task evaluation is used to assess the completion of the task objective, action style evaluation is used to assess action continuity and motion smoothness, and constraint evaluation is used to assess contact stability and posture safety. Traditional reinforcement learning methods typically use a single value network to uniformly evaluate all rewards. However, in complex contact tasks involving multiple heterogeneous rewards, the long-term reward characteristics of different rewards vary significantly, making it difficult for a single value network to accurately estimate all reward items simultaneously. Therefore, it is necessary to construct independent value networks for different reward modules. ； Where V represents the evaluation network of the reward module, This represents the task evaluation network. This indicates a movement style evaluation network. This represents a constrained evaluation network, where each Critic is only responsible for evaluating the long-term returns of its corresponding reward module: ； in, Indicates the discount factor. Let i represent the reward corresponding to reward i, and T represent the total time steps of the task. Compared with the single Critic structure, it can reduce the mutual influence between different rewards, make the value estimation more stable, and thus improve the training efficiency and policy stability in complex sit-and-stand tasks.
6. The stage-driven multi-Critic humanoid robot sit-stand switching control model according to claim 1, characterized in that: The Multi-Critic value assessment network in step 3 adopts an independent value function structure, and its overall advantage function is expressed as: ； in, This represents the advantage function estimate of the j-th evaluation network at each discrete control time step t, used to evaluate the current state. With action Contributions in long-term cumulative rewards; This represents the weight coefficient of the corresponding evaluation network, used to adjust the degree of influence of different evaluation objectives in the strategy optimization process; These represent task evaluation, action style evaluation, and constraint evaluation, respectively.
7. The stage-driven multi-Critic humanoid robot sit-stand switching control model according to claim 1, characterized in that: The stage reward activation mechanism in step 4 dynamically adjusts the reward function weights based on the robot's current motion stage, and its total reward function is expressed as: ； Where r represents the robot's total reward. This indicates a task reward, used to guide the robot to complete the sitting-to-standing conversion goal; It represents a reward for movement style, used to constrain movement continuity and smoothness; This represents a constraint reward, used to constrain the robot's contact stability and posture safety.
8. The stage-driven multi-Critic humanoid robot sit-stand switching control model according to claim 7, characterized in that: The task rewards include: Task rewards are primarily used to drive the robot to complete sitting and standing tasks, including: pelvic posterior displacement guided rewards. Pelvic height tracking reward Target standing height reward Rewards for achieving phase goals ;in, The corresponding asymmetric rear gravitational potential field in the guided descent phase is used to constrain the robot to form a stable backward movement posture during the descent; the task reward mainly determines the overall motion trend of the robot and is the core driving force for completing the sit-to-stand conversion task. Motion style rewards are primarily used to improve the quality and continuity of robot movements, including: trunk posture stability rewards. Smooth motion reward Joint coordination reward Body swing inhibition reward This part of the reward does not directly determine whether the task is completed, but is used to reduce obvious oscillations, forward tilting and discontinuous movements, thereby improving the stability and naturalness of the robot's movement. Constraint rewards are primarily used to ensure safety and physical stability during training, including contact speed constraint rewards. Joint torque constraint reward Joint speed constraint reward Stable foot contact bonus ;in, The joint torque constraint reward and joint velocity constraint reward are used to reduce the impact of collision when the robot comes into contact with the seat; they are used to limit the control output amplitude and reduce the problem of high frequency oscillation; the constraint reward can effectively improve the training stability in complex contact tasks.
9. The stage-driven multi-Critic humanoid robot sit-stand switching control model according to claim 1, characterized in that: The reinforcement learning PPO algorithm in step 4 uses the following objective function to complete the policy update: ； in, This represents the PPO pruning objective function. This represents the ratio of the probabilities of the new and old strategies. This represents the estimated value of the advantage function. This represents the cutting factor.
10. The stage-driven multi-Critic humanoid robot sit-stand switching control model according to claim 1, characterized in that: It also includes an asymmetric rearward gravitational potential field to constrain the robot's pelvis's tendency to shift backward relative to the bipedal support area, thereby reducing the forward tilting instability problem during the robot's sitting process. The goal of the guided descent phase is to enable the robot to gradually approach the seat while maintaining overall stability. Traditional artificial potential field methods usually use a symmetric gravitational field with Euclidean distance from the target point. The core idea is to drive the control model to continuously approach the target position. However, in the sitting-standing task of a high-center-of-gravity humanoid robot, simply reducing the distance between the robot and the seat can easily lead to excessive forward shift of the pelvis during descent, resulting in instability or even a fall before contact. To solve this problem, an improved asymmetric rearward gravitational potential field is proposed for the sitting-standing transition task. Unlike the traditional APF which directly constrains the robot's approach to the target position, this potential field prioritizes constraining the backward shift of the pelvis relative to the bipedal support area, so that the robot first completes the backward shift of its center of gravity during descent and then gradually approaches the seat area. The center position of both feet is defined as: ； in, The center of the robot's bipedal support area. Indicates the position of the left foot. Indicates the position of the right foot; the posterior displacement of the pelvis is defined as: ； in, Indicates the position of the pelvis in the front-to-back direction. This represents the maximum allowed backward movement distance; further, an asymmetric backward potential field reward is constructed: ； in, The potential field attenuation coefficient is used; instead of directly driving the robot to move towards the center of the seat, it prioritizes constraining the relative positional relationship between the pelvis and the support area, thereby forming a more stable descent posture; compared with the traditional symmetrical potential field, this asymmetrical structure can effectively reduce the phenomenon of the center of gravity shifting forward before contact and reduce the probability of instability during the descent phase; in addition, the pelvic height constraint is introduced at the same time during this phase to guide the robot to form a stable descent trajectory before contacting the seat, thereby reducing the sudden speed change and high-frequency oscillation phenomenon at the moment of contact; The robot control model adopts a hierarchical cascaded control architecture, where a high-level reinforcement learning policy is responsible for generating the robot's motion target, and a low-level PD controller is responsible for joint drive control. At each discrete control time step t, the high-level policy network outputs the residual positions of each joint of the robot. ； in, This represents the target joint position residual; the robot target joint position is defined as: ； in, This indicates the robot's nominal initial posture or the reference joint position at the previous moment. This indicates the position of the robot's target joints; the underlying physical control layer uses a proportional-derivative (PD) controller to track the target joint positions in real time and calculate the driving torque of each joint of the robot. ； in, This represents the driving torque of the robot joints, and q represents the current actual joint position. and These represent the target joint angular velocity and the current actual joint angular velocity, respectively. Represents the proportional gain coefficient. This represents the damping gain coefficient; the proportional term is used to drive the robot joint to quickly approach the target position, and the differential term is used to suppress velocity oscillations and high-frequency jitters during joint movement, thereby improving control stability under complex contact conditions. Regarding training stability and convergence, due to the strong randomness in the early stages of reinforcement learning, directly outputting torque can easily cause high-frequency oscillations and abnormally large-angle torques, leading to rapid divergence of the control model's kinetic energy and training collapse. Introducing a low-level PD controller essentially creates a physical barrier between the policy output and the actual dynamics; through reasonable settings... and The system incorporates preset virtual stiffness and damping, which not only provides passive stability boundaries for the robot's joint movements but also effectively filters out high-frequency control noise, enabling the policy network to focus on learning more macroscopic sit-stand transition gaits and improving the algorithm's convergence efficiency. Compared to control methods that directly output joint torque, the cascaded control architecture of the target joint position can effectively reduce the motion search space complexity of the reinforcement learning strategy, reduce the oscillation problem caused by high-frequency torque output, and improve the motion smoothness, contact stability and robustness of the entire sitting and standing control process, including sitting down, touching and standing up.