Autonomous recovery control strategy network training method and device for robot with lower limb assembly for movement based on reinforcement learning, electronic equipment, robot and computer readable storage medium

Abstract

This application provides a method, apparatus, electronic device, robot, and computer-readable storage medium for training an autonomous recovery control policy network for a robot with lower limb components for mobility, based on reinforcement learning. The method places the robot in a random initial non-standing posture in the training environment, acquires state observation information, and inputs it into the recovery control policy network to obtain joint control variables. A reward function comprising multiple reward components is constructed based on the state observation information, and the network parameters are updated based on this reward function. This enables the trained recovery control policy network to drive the robot from a non-standing posture to a target standing state and maintain stability. This approach helps suppress incomplete recovery, non-convergence of motion after recovery, and states where the robot's posture is upright but its height is insufficient.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a method for training an autonomous recovery control strategy network for a robot with a lower limb assembly for mobility based on reinforcement learning. Background Technology

[0002] Robots with lower limb components for mobility typically achieve body support, movement, and posture adjustment through leg joint components, wheel-leg composite components, or other lower limb actuators. When such robots operate in complex terrain environments, they may deviate from their normal standing state due to factors such as ground undulations, external collisions, control errors, actuator response delays, and their own motion impacts, and may even enter non-standing postures such as tipping, rolling over, tilting, inverting, or abnormal joint postures.

[0003] When the robot is in a non-standing posture, it needs to re-establish support through the lower limb components and related control systems, and gradually restore the body to a state suitable for continuing to perform tasks. Compared with conventional motion control, autonomous recovery control involves factors such as body posture, body height, joint status, ground contact, actuator output, and motion stability during the recovery process, making the control process more complex.

[0004] In related technologies, robot recovery control can be achieved using preset action sequences, manually designed control logic, trajectory planning methods, or control strategies based on training. Preset action sequences and manually designed control logic typically rely on specific initial postures and contact states, offering limited adaptability to random tipping postures, different ground contact conditions, and changes in initial joint states. Trajectory planning methods may require significant prior constraints and online computational resources when facing complex contact transitions and large posture changes. While control strategies based on training can improve adaptability to various states, training results are often affected by factors such as the training environment, state observations, evaluation objectives, constraints, and differences between simulation and actual deployment.

[0005] Therefore, for robots with lower limb components for movement, there is still room for improvement in how to train an autonomous recovery control strategy network that can recover from a non-standing posture to a target standing state. Summary of the Invention

[0006] This application provides a method, apparatus, electronic device, robot, and computer-readable storage medium for training an autonomous recovery control strategy network for a robot with a lower limb assembly for mobility based on reinforcement learning, so as to improve the training effect of the robot recovering from a non-standing posture to a target standing state.

[0007] To achieve the above objectives, in one embodiment, the training method includes: placing a robot with a lower limb assembly for movement in a training environment in a random initial non-standing posture; acquiring the robot's state observation information and inputting it into a recovery control strategy network to obtain joint control quantities for driving the robot to perform recovery actions; constructing a reward function based on the state observation information, and updating the network parameters of the recovery control strategy network based on the reward function. The reward function includes a first reward component, a second reward component, and a third reward component. The first reward component is used to suppress states where the robot only meets some of the recovery objectives in height recovery, posture recovery, and joint configuration recovery; the second reward component is used to suppress states where the robot remains in a non-converged state after reaching the target standing state; and the third reward component is used to suppress states where the robot's body posture is close to the target upright posture but the body height has not reached the corresponding height requirement. Thus, the updated recovery control strategy network can output joint control quantities to drive the robot from a non-standing posture to the target standing state and maintain stability.

[0008] In one implementation, the first reward component is determined based on the synergistic coupling result of the height recovery evaluation, the posture recovery evaluation, and the joint configuration recovery evaluation; the second reward component is determined based on the coupling result of the recovery evaluation and the degree of stillness; and the third reward component is obtained by gating the posture recovery evaluation based on the height gating factor. These three reward components can together constitute a combined reward function, or they can be used as independent reward constructs in different training schemes.

[0009] In one implementation, a training method based on collaborative recovery rewards is provided. This method determines multiple recovery evaluation quantities to characterize the degree of height recovery, posture recovery, and joint configuration recovery based on body height information, body posture information, and leg joint state information, respectively. These multiple recovery evaluation quantities are then synergistically coupled to obtain a collaborative recovery reward, which is suppressed when any recovery evaluation quantity deviates from its corresponding recovery target. This reduces the probability that the robot only meets some of the recovery targets in height, posture, or joint configuration.

[0010] In one implementation, the multiple recovery evaluation metrics include height score, posture score, and joint configuration score. The height score, posture score, and joint configuration score can all be continuous scores that increase as the corresponding deviation decreases. The synergistic recovery reward can be determined based on the product of the height score, posture score, and joint configuration score, such that a decrease in any one score results in a decrease in the overall reward.

[0011] In one implementation, a training method based on stability maintenance rewards is provided. This method determines a recovery evaluation quantity characterizing the robot's approach to a target standing state based on recovery state information, determines a degree of stillness based on motion state information, and couples the recovery evaluation quantity and the degree of stillness to obtain a stability maintenance reward, such that the stability maintenance reward decreases as either the recovery evaluation quantity or the degree of stillness decreases. This enables the robot not only to reach the target standing state but also to reduce its motion state and maintain stability after reaching it.

[0012] In one implementation, the degree of stillness is determined based on at least one of leg joint velocity, body angular velocity, and body linear velocity, and increases as the leg joint velocity, body angular velocity, or body linear velocity decreases. The stability maintenance reward can be determined based on the product of the recovery evaluation quantity and the degree of stillness. Furthermore, when the robot continuously meets the target standing state conditions for a preset duration, recovery can be determined to be successful, and the current training round can be terminated early.

[0013] In one implementation, a training method based on height-gated posture rewards is provided. This method determines a posture recovery evaluation quantity based on robot posture information, determines a height gating quantity based on robot height information, and uses the height gating quantity to gating the posture recovery evaluation quantity to obtain a height-gated posture reward. When the robot's posture is close to the target upright posture but the robot height does not reach the corresponding height requirement, the height-gated posture reward is suppressed. This reduces the probability of the robot forming a posture that is close to upright but still close to the ground.

[0014] In one implementation, the aircraft attitude information includes the projected gravity direction, and the attitude recovery evaluation value is determined based on the deviation between the projected gravity direction and the target upright direction. The altitude gating value is determined based on the difference between the aircraft height and the target height range, and increases as the aircraft height approaches the target height range. The altitude-gated attitude reward can be determined based on the product of the attitude recovery evaluation value and the altitude gating value.

[0015] In one implementation, the random initial posture in a non-standing position is obtained by randomizing at least one of the following: body roll angle, body pitch angle, initial joint position, and initial joint velocity; when randomizing the initial joint position, the randomized initial joint position is restricted within the joint soft limit. During training, at least one of the following parameters can also be randomized: ground friction parameters, collision recovery coefficient, body mass, leg component mass and / or inertia, actuator stiffness, actuator damping, and actuator control delay.

[0016] In one implementation, the joint control quantities include leg joint control quantities and foot joint control quantities. During recovery training, the motion scaling factor of the foot joint control quantities can be set to zero, or the foot joints can be restricted from being used as active recovery drivers, so that the recovery movements are mainly generated by the leg joints in the lower limb assembly.

[0017] In one implementation, during the training phase, a teacher latent variable is generated by a teacher encoder based on privileged physical information, and a student latent variable is generated by a student encoder based on historical ontology observations. The student latent variable is supervised based on the teacher latent variable. A velocity estimator is used to generate an estimate of the organism's linear velocity based on historical ontology observations, and the velocity estimator is trained using the real organism linear velocity available during the training phase.

[0018] In one implementation, the reward function further includes at least one of an inverted state penalty, a basic regularization term, a contact impact constraint term, and a state proximity constraint term. The inverted state penalty is applied when it is determined that the robot is in an inverted state based on the robot's posture information; the contact impact constraint term is not calculated during a preset control period at the beginning of the training round; and the state proximity constraint term is enhanced as the robot approaches the target standing state.

[0019] In one implementation, at least one of the following can be mirrored left and right: current ontology observation, historical ontology observation, leg action, contact force, foot speed, control command, privileged observation, and terrain height scan data. The recovery control strategy network is then updated based on the action consistency loss between the original state and the mirrored state.

[0020] This application also provides a training device. The training device includes an environment and observation module, an action generation module, a reward construction module, and a network update module. The environment and observation module is used to put the robot in a random initial posture (non-standing posture) in the training environment and acquire state observation information; the action generation module is used to input the state observation information into the recovery control policy network and obtain joint control variables; the reward construction module is used to construct a reward function, a cooperative recovery reward, a stability maintenance reward, or a height-gated posture reward; the network update module is used to update the network parameters of the recovery control policy network based on the reward function, cooperative recovery reward, stability maintenance reward, or height-gated posture reward.

[0021] This application also provides an electronic device including a processor and a memory, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements any of the training methods described above. This application also provides a robot with a lower limb assembly for movement, the robot including the lower limb assembly, a controller, and a memory, wherein the memory stores a recovery control strategy network trained using any of the training methods described above, the controller is used to acquire state observation information of the robot, and input the state observation information into the recovery control strategy network to output joint control quantities for driving the lower limb assembly to perform recovery actions. This application also provides a computer-readable storage medium.

[0022] This application has at least the following beneficial effects: training the recovery control policy network by using a reward function containing a first reward component, a second reward component, and a third reward component helps the robot recover from a non-standing posture to the target standing state and maintain stability; the synergistic coupling of multiple recovery evaluation quantities helps suppress states that only partially meet the recovery target; the coupling of recovery evaluation quantities with the degree of stillness helps suppress states where the robot remains in a state of non-convergence after reaching the target standing state; and gating the posture recovery evaluation quantities with height gating helps suppress states where the robot's posture is close to the target upright posture but the robot's height does not reach the corresponding height requirement. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a method for training an autonomous recovery control strategy network for a robot with a lower limb assembly for mobility based on reinforcement learning, as provided in an embodiment of this application. Figure 2 A flowchart illustrating the training method based on collaborative recovery rewards provided in this application embodiment; Figure 3 A flowchart illustrating the training method based on stability retention rewards provided in this application embodiment; Figure 4 A flowchart of a training method based on height-gated posture reward provided for embodiments of this application; Figure 5 Flowchart of a preferred embodiment of the training method provided in this application; Figure 6 Functional block diagram of the autonomous recovery control strategy network training device provided in the embodiments of this application; Figure 7A structural block diagram of the electronic device provided in the embodiments of this application; Figure 8 A structural block diagram of the robot provided in the embodiments of this application; Figure 9 This is a schematic diagram of the training process data flow provided in an embodiment of this application; Figure 10 This is a schematic diagram illustrating the variation patterns of the recovery evaluation quantity and the height gating quantity in an embodiment of this application. Figure 11 A block diagram illustrating the reward function provided in the embodiments of this application; Figure 12 This is a schematic diagram illustrating the definition of the three recovery evaluation quantities provided in the embodiments of this application.

[0025] Figure 13 This is a schematic diagram of a random recovery training environment provided in an embodiment of this application; Figure 14 This is a schematic diagram of the recovery success rate variation curve provided in the embodiments of this application; Figure 15 This is a schematic diagram of the attitude recovery index change curve provided in the embodiments of this application; Figure 16 This is a schematic diagram of the joint configuration recovery index change curve provided in the embodiments of this application; Figure 17 This is a schematic diagram of the body height reward change curve provided in the embodiments of this application; Figure 18 This is a schematic diagram of the attitude recovery reward change curve provided in the embodiments of this application; Figure 19 This is a schematic diagram of the joint target reward change curve provided in the embodiments of this application; Figure 20 This is a schematic diagram of the collaborative recovery reward change curve provided in the embodiments of this application; Figure 21 A schematic diagram of the continuous states of a dynamic standing process provided in an embodiment of this application; Figure 22 This is a schematic diagram of an abnormal standing up provided in an embodiment of this application.

[0026] Explanation of reference numerals in the attached figures Detailed Implementation The embodiments of this application will be further described below with reference to the accompanying drawings. It should be understood that the following embodiments are used to illustrate the implementation of this application and are not intended to limit the scope of protection of this application. Unless otherwise specified, the steps, modules, observation information, reward components, and constraints in the various embodiments can be combined with each other.

[0027] The robot in this application refers to a robot with lower limb components for movement. The lower limb components can be leg joint components, wheel-leg composite components, or other lower limb execution components that can both support the body and participate in movement or posture adjustment. When the robot enters a non-standing posture, it can exhibit states such as lying on its side, lying prone, lying supine, inverted, semi-supported, or abnormal joint posture. The target standing state can include the body height reaching the target height requirement, the body posture approaching the target upright posture, the leg joint configuration approaching the target standing configuration, and a reduced motion state. The foot can be any of the following: a ball of feet, a wheel, or a point foot; generally referring to the robot's lower limb movement and ground contact mechanism.

[0028] In a specific scenario, a robot's recovery from a non-standing posture to a target standing state is not a simple rolling over action. The robot typically needs to first change its orientation, then find a support relationship between its lower limb components and the ground, subsequently raise its body, adjust the leg joint configuration, and finally reduce the movement of the joints and the body. Each step in this process can fail; therefore, this application trains a recovery control policy network that outputs joint control quantities based on state observation information within continuous control cycles.

[0029] like Figure 1 As shown, in one embodiment, the autonomous recovery control policy network training method includes steps 101 to 104. This process constitutes a basic closed loop of one round of recovery training: the training environment first generates a random initial posture in a non-standing posture, the recovery control policy network outputs joint control quantities based on state observation information, the training environment obtains a new state after executing the joint control quantities, and then constructs a reward function and updates the network parameters based on the recovery effect.

[0030] In step 101, the robot, equipped with a lower limb assembly for movement, is placed in a random initial non-standing posture in the training environment, and state observation information of the robot is acquired. The training environment can be a simulation training environment. The state observation information may include at least one of the following: body height information, body posture information, leg joint state information, motion state information, current body observation, and historical body observation.

[0031] like Figure 13 As shown, in one embodiment, multiple robots can be deployed simultaneously in the training environment, with each robot in a different non-standing posture. The multiple robots can each correspond to different body orientations, support contact states, and initial joint configurations. This setup allows the recovery control strategy network to repeatedly encounter various falling postures and abnormal postures during training, rather than simply memorizing a single fixed standing action.

[0032] Figure 21The diagram further illustrates multiple consecutive states during a dynamic standing process. Six screens, arranged chronologically, show the robot gradually adjusting its posture from a non-standing position, swinging and supporting its lower limbs, raising its body, and retracting its leg joints until it approaches the target standing position. This illustration demonstrates that the joint control variables output by the recovery control strategy network act on the robot over a continuous control cycle, rather than providing a single static judgment at a single moment.

[0033] In some implementations, the random initial attitude for a non-standing posture can be obtained by randomizing at least one of the following: body roll angle, body pitch angle, initial joint position, and initial joint velocity. When randomizing the initial joint position, the randomized initial joint position is restricted within the joint soft limit to avoid generating an initial state that is unexecutable or prone to causing simulation anomalies. During training, at least one of the following can also be randomized: ground friction parameters, collision recovery coefficient, body mass, leg component mass and / or inertia, actuator stiffness, actuator damping, and actuator control delay.

[0034] In some implementations, the roll and pitch angles can be randomly sampled over a wide range, allowing the robot to begin training from side-lying, prone, supine, fully inverted, or intermediate postures. Random offsets can be added to each joint based on its default position, and the initial joint velocities can be randomly sampled. In this way, the recovery control policy network faces a wide range of initial state distributions, and the training objective more closely approximates the falls or abnormal postures that a real robot might encounter.

[0035] In step 102, the state observation information is input into the recovery control policy network to obtain the joint control quantities used to drive the robot to perform recovery actions. The joint control quantities may include leg joint control quantities and foot joint control quantities, or may only include the joint control quantities involved in the recovery action. In one example, the joint control quantities may be joint position control quantities.

[0036] In some implementations, the motion scaling factor of the foot joint control quantities can be set to zero during recovery training, or the foot joints can be restricted from being used as active recovery drivers. Periodic external force disturbances and upward assistive forces can also be disabled, allowing the robot to primarily rely on the leg joints in the lower limb assembly to complete turning, supporting, lifting, and posture correction. This reduces the reliance on wheel rolling, external force pushing, or assistive forces for recovery.

[0037] In step 103, a reward function for training the recovery control policy network is constructed based on the state observation information. The reward function includes a first reward component, a second reward component, and a third reward component. The first reward component is used to suppress the robot from only partially satisfying the recovery objectives of height recovery, posture recovery, and joint configuration recovery; the second reward component is used to suppress the situation where the robot reaches the target standing state but its motion has not decreased; the third reward component is used to suppress the situation where the robot's body posture is close to the target upright posture but the body height has not yet reached the corresponding height requirement.

[0038] In one implementation, the first reward component can be obtained by co-coupling the height recovery evaluation, the posture recovery evaluation, and the joint configuration recovery evaluation; the second reward component can be obtained by coupling the recovery evaluation and the degree of stillness; and the third reward component can be obtained by gating the posture recovery evaluation with a height-gated value. In other words, the reward function is concerned not only with whether the robot has lifted up, but also with whether the posture is correct, whether the joints are stable, and whether it is still wobbling after reaching the target state.

[0039] In step 104, the network parameters of the recovery control strategy network are updated based on the reward function. The network parameter updates can be based on state observation information, joint control quantities, reward function values, state transition results, and termination judgment results collected during training rounds. The updated recovery control strategy network is used to continue outputting joint control quantities based on the new state observation information, thereby gradually forming the control capability to recover from a non-standing posture to the target standing state and maintain stability.

[0040] like Figure 2 As shown, in one embodiment, the training method based on collaborative recovery rewards includes steps 201 to 205. The key point of this embodiment is to bind height recovery, posture recovery, and joint configuration recovery into the same reward logic, so that the robot cannot obtain a higher reward by relying solely on the improvement of one metric.

[0041] In step 201, the robot is placed in a random initial posture (non-standing) and state observation information is acquired. This state observation information includes body height information, body posture information, and leg joint state information. Body height information can represent the height of the robot relative to the ground or support surface; body posture information can be characterized by the projected gravity direction in the body coordinate system, body tilt angle, body angular velocity, or a combination thereof; leg joint state information can include leg joint position, leg joint velocity, deviation of the leg joint from the default position, or a combination thereof.

[0042] In step 202, the state observation information is input into the recovery control strategy network to obtain the joint control quantities used to drive the robot to perform recovery actions. The training environment advances the robot's state based on the joint control quantities, enabling the robot to transition from its current non-standing posture to the posture and contact state of the next control cycle.

[0043] In step 203, multiple recovery evaluation quantities are determined based on the aircraft height information, aircraft posture information, and leg joint state information. These multiple recovery evaluation quantities may include a height recovery evaluation quantity, a posture recovery evaluation quantity, and a joint configuration recovery evaluation quantity. The height recovery evaluation quantity characterizes the degree to which the aircraft height approaches the target height range; the posture recovery evaluation quantity characterizes the degree to which the aircraft posture approaches the target upright posture; and the joint configuration recovery evaluation quantity characterizes the degree to which the leg joint state approaches the target standing configuration.

[0044] like Figure 12 As shown, the three recovery evaluation metrics can be determined by their respective deviations. The deviation between the body height and the target height range is used to determine the height score; the deviation between the projected gravity direction and the target's upright direction is used to determine the posture score; and the deviation between the leg joint state and the target's standing configuration is used to determine the joint configuration score. All of these scores can be continuous scores that increase as the corresponding deviation decreases.

[0045] In reward construction, the organism's altitude reward can be used as an altitude recovery evaluation metric or altitude score; an example of its training record can be found here. Figure 17 The reward increases as the robot's height approaches the target height range, guiding the robot to raise its body during the recovery process. If the robot merely changes its orientation but remains close to the ground, the height reward will not provide sufficient encouragement.

[0046] In reward construction, pose recovery reward can be used as a pose recovery evaluation metric or pose score; an example of its training record can be found here. Figure 18 This reward can be determined based on the deviation between the projected gravity direction and the target's upright direction, and is used to guide the aircraft's attitude to recover towards the target's upright attitude.

[0047] In the reward construction, the joint target reward can be used as a joint configuration recovery evaluation metric or a joint configuration score; an example of its training record can be found in [link to training record]. Figure 19 This reward can be determined based on the deviation between the leg joint state and the target standing configuration, and is used to guide the leg joint from a random initial configuration to gradually approach the target standing configuration.

[0048] In step 204, multiple recovery evaluation quantities are synergistically coupled to obtain a synergistic recovery reward, which is suppressed as any recovery evaluation quantity deviates from the corresponding recovery target. In one embodiment, the synergistic recovery reward can be determined based on the product of height score, posture score, and joint configuration score. This product uses the scores from height, posture, and joint configuration together to evaluate the recovery state.

[0049] Figures 17 to 20 The recorded changes in body height reward, posture recovery reward, joint target reward, and synergistic recovery reward during training are shown separately.

[0050] Figure 17 The diagram illustrates the recorded changes in body height reward during training. This curve reflects the changes in training records as training progresses and does not represent a reward function relationship with body height or height deviation as the x-axis. In reward construction, body height reward characterizes the degree to which the body height approaches the target height range, and increases as the body height approaches the target height range; when the body height is significantly below the target height range, it is difficult to obtain a high body height reward even if the robot has adjusted its orientation. This reward is used to guide the recovery control policy network to learn actions to lift the body, reducing incomplete recovery states that are close to the ground.

[0051] Figure 18 The diagram shows the recorded changes in posture recovery reward during training, but does not represent the reward function relationship with body posture or posture deviation as the x-axis. In the reward construction, the posture recovery reward is used to characterize the degree to which the body posture approaches the target upright posture, and it increases as the deviation between the projected gravity direction and the target upright direction decreases. This reward is used to guide the recovery control policy network to learn actions of flipping and correcting the body posture, enabling the robot to gradually recover from non-standing postures such as lying on its side, prone, supine, or inverted towards the target upright posture.

[0052] Figure 19 The diagram shows the recorded changes in joint target reward during training, but does not represent the reward function relationship with leg joint state or configurational deviation as the x-axis. In the reward construction, the joint target reward is used to characterize the degree to which the leg joint state approaches the target standing configuration, and increases as the deviation between the leg joint state and the target standing configuration decreases. This reward is used to guide the recovery control policy network to learn to adjust the leg joint from a random initial configuration to a target standing configuration suitable for supporting the body, reducing the state where the body height and posture are close to the target but the leg joint is still not close to the target configuration.

[0053] Figure 20The diagram illustrates the recorded changes in the collaborative recovery reward during training. In reward construction, the robot height reward, posture recovery reward, and joint target reward corresponding to the same robot state are co-coupled, allowing multiple recovery targets to jointly influence the collaborative recovery reward. In one implementation, the collaborative recovery reward can be determined based on the product of these three rewards. This product relationship refers to the three evaluation metrics used in reward construction, rather than... Figures 17 to 19 The training record values ​​displayed separately are directly multiplied together.

[0054] For example, when the robot's body posture is close to the target upright posture but the body height has not yet reached the target height range, the posture recovery reward can be high, but the body height reward is low, so the cooperative recovery reward is still suppressed; when the body height is close to the target height range but the body posture is still tilted, the body height reward can be high, but the posture recovery reward is low, and the cooperative recovery reward is also suppressed; when both the body height and body posture are close to the target, but the leg joint state has not yet approached the target standing configuration, the joint target reward is low, and the cooperative recovery reward is also difficult to reach a high level.

[0055] As can be seen from the coupling relationship of the recovery evaluation quantities above, the collaborative recovery reward is not simply a combination of individual rewards, but rather a system where the three recovery objectives mutually constrain each other. If the robot's posture is close to upright but its height is insufficient, the height score will limit the collaborative recovery reward; if the robot's height reaches the target range but its posture is tilted, the posture score will limit the collaborative recovery reward; if the height and posture are good but the leg joint configuration is unstable, the joint configuration score will also continue to limit the collaborative recovery reward.

[0056] In this way, the recovery control policy network needs to simultaneously improve body height, body posture, and leg joint configuration during training to obtain a higher cooperative recovery reward. Therefore, action strategies that only achieve a single recovery objective or only a partial recovery objective can be suppressed, enabling the robot to recover more stably from a non-standing posture to the target standing state.

[0057] Figure 22 This is a schematic diagram of an abnormal standing up provided in an embodiment of this application. In the diagram, the robot body is in an inverted posture and has not returned to the target standing state.

[0058] In step 205, the network parameters of the recovery control strategy network are updated based on the collaborative recovery reward, so that the joint control quantities output by the updated recovery control strategy network drive the robot to recover from a non-standing posture to the target standing state. Through this training method, the recovery control strategy network is more likely to learn a complete standing action, rather than forming an action that only satisfies local indicators.

[0059] like Figure 3As shown, in one embodiment, the training method based on stability retention rewards includes steps 301 to 306. This embodiment focuses on the latter half of the recovery action: the robot not only needs to reach the vicinity of the target standing state, but also gradually reduce its movement after reaching it to maintain stability.

[0060] In step 301, the robot is placed in a random initial pose (not standing) and state observation information is acquired. This state observation information includes restored state information and motion state information. The restored state information may include at least two of the following: body height information, body posture information, and leg joint state information; the motion state information may include leg joint velocity, body angular velocity, body linear velocity, or other information that can characterize the degree of robot motion.

[0061] In step 302, the state observation information is input into the recovery control strategy network to obtain the joint control variables used to drive the robot to perform recovery actions. These joint control variables are used to change the robot's posture, contact state, and joint configuration within the current control cycle.

[0062] In step 303, a recovery evaluation quantity is determined based on the recovery state information to characterize the robot's approach to the target standing state. The recovery evaluation quantity can be determined based on at least two of the following: body height, body posture, and leg joint configuration; it can also be a collaborative recovery reward or a combination of multiple recovery evaluation quantities.

[0063] In step 304, the degree of stillness is determined based on the motion state information. The degree of stillness characterizes the degree of motion inhibition of the robot during the recovery process. The degree of stillness can be determined based on at least one of leg joint velocity, body angular velocity, and body linear velocity, and can increase as the leg joint velocity, body angular velocity, or body linear velocity decreases. In simple terms, the closer the robot is to a standing and stable state, the higher its degree of stillness.

[0064] In step 305, the recovery evaluation value and the degree of stillness are coupled to obtain a stability retention reward, which decreases as either the recovery evaluation value or the degree of stillness decreases. In one embodiment, the stability retention reward can be determined based on the product of the recovery evaluation value and the degree of stillness. Thus, even if the robot briefly moves near the target standing posture, it cannot obtain a high stability retention reward if it still has a large joint velocity or body angular velocity.

[0065] In step 306, the network parameters of the recovery control strategy network are updated based on the stability retention reward, so that the joint control quantity output by the updated recovery control strategy network drives the robot to recover from the non-standing posture to the target standing state and maintain stability. Figure 14 The recovery success rate curve shown can be used to observe changes in overall recovery ability during training; Figure 15 and Figure 16 The posture recovery index and joint configuration recovery index shown can be used to observe the training effect from the two perspectives of body posture and joint configuration, respectively.

[0066] In some implementations, a successful recovery can be determined when the robot continuously meets the target standing state conditions for a preset duration. The target standing state conditions may include the robot's height being within a target height range, the robot's tilt angle being less than a tilt angle threshold, and the error between the leg joints and the target standing configuration being less than a configuration error threshold. After determining successful recovery, the current training round can be terminated early, and the bootstrapping process can be handled using a time truncation method to reduce the possibility of mistakenly interpreting a successful premature termination as a failure.

[0067] In some implementations, joint velocity penalties or joint torque penalties can be increased based on how closely the joint position approximates the target standing configuration, body angular velocity penalties can be increased based on how closely the body posture approximates the target upright posture, and / or body linear velocity penalties can be increased based on how closely the body height approximates the target height range. Thus, the robot can retain necessary motion capabilities in the early stages of recovery, while gradually increasing damping and stability in the later stages.

[0068] like Figure 4 As shown, in one embodiment, the training method based on height-gated posture rewards includes steps 401 to 405. This embodiment focuses on a common local state: the robot body is close to upright, but the whole body is still close to the ground and has not truly stood up.

[0069] In step 401, the robot is placed in a random initial pose that is not standing, and state observation information is acquired. This state observation information includes the robot's height information and its pose information.

[0070] In step 402, the state observation information is input into the recovery control strategy network to obtain the joint control quantities used to drive the robot to perform recovery actions.

[0071] In step 403, an attitude recovery evaluation quantity is determined based on the robot's attitude information. The attitude recovery evaluation quantity characterizes the degree to which the robot's attitude approximates the target upright posture. In one embodiment, the robot attitude information includes the projected gravity direction, and the attitude recovery evaluation quantity can be determined based on the deviation between the projected gravity direction and the target upright direction.

[0072] In step 404, a height gating value is determined based on the aircraft height information, and the attitude recovery evaluation value is gating-processed using the height gating value to obtain a height-gated attitude reward. The height gating value can be determined based on the difference between the aircraft height and the target height range, and increases as the aircraft height approaches the target height range. Figure 10The diagram illustrates the relationship between the recovery evaluation quantity and the height gating quantity as the state approaches the target. The curves in the figure are only used to illustrate the monotonic trend and do not limit the specific function form.

[0073] In step 404, when the robot's posture is close to the target upright posture but its height does not meet the corresponding height requirement, the height-gated posture reward is suppressed. When the robot rises to near the target height, the height gating amount increases, and the suppression of the posture recovery evaluation amount decreases. In this way, the robot cannot obtain a high posture-related reward simply by righting itself while remaining close to the ground.

[0074] In step 405, the network parameters of the recovery control strategy network are updated based on the height-gated posture reward, so that the joint control quantity output by the updated recovery control strategy network drives the robot to recover from the non-standing posture to the target standing state.

[0075] In some implementations, whether the robot is in a fully inverted state can be determined based on the projected gravity direction; when the robot is in a fully inverted state, an inverted state penalty is applied to prevent the robot from maintaining a fully inverted state. The target height score is determined based on the deviation between the robot's height and the target height range, and increases as this deviation decreases. The height-gated attitude reward can also be used together with the target height score to update the recovery control policy network, enabling the robot to simultaneously lift its body during attitude recovery.

[0076] like Figure 5 As shown, in a preferred embodiment, the training process may further include steps 501 to 509. This process describes randomization, observation, action output, latent variable supervision, velocity estimation, reward construction, left-right mirror transformation, network parameter update, and success determination in the same training round.

[0077] In step 501, the initial posture and training parameters are randomized. Specifically, at least one of the robot's roll angle, pitch angle, initial joint position, and initial joint velocity can be randomized; when randomizing the initial joint position, the randomized initial joint position is restricted within the joint's soft limit. During training, at least one of the following parameters can also be randomized: ground friction parameters, collision recovery coefficient, body mass, leg component mass and / or inertia, actuator stiffness, actuator damping, and actuator control delay. Through randomization, the training environment can cover more non-standing states and dynamic conditions.

[0078] In step 502, current and historical volumetric observations are acquired. Current volumetric observations may include at least one of the following: body angular velocity, projected gravity direction in the body coordinate system, relative position of non-foot joints, foot odometry position increment, joint velocity, and the previous control action. Historical volumetric observations may include volumetric observations from multiple consecutive control cycles. Historical volumetric observations can be used to estimate body linear velocities not directly provided in the current observations, and can also be used to infer implicit dynamic characteristics such as mass, friction, collisions, and attitude changes.

[0079] In step 503, the state observation information is input into the recovery control strategy network to obtain joint control quantities. In one embodiment, the joint control quantities include leg joint control quantities and foot joint control quantities; during recovery training, the motion scaling factor of the foot joint control quantities can be set to zero, or the foot joints can be restricted from being used as active recovery drivers. Periodic external force perturbations and upward auxiliary forces can also be turned off during training, so that the recovery movements are mainly generated by the leg joints in the lower limb components.

[0080] In step 504, teacher latent variable supervision, student latent variable supervision, and velocity estimator training are performed. Privileged physical information can be obtained during the training phase. Privileged physical information may include at least one of the following: actual linear velocity of the organism, actual joint torque, joint acceleration, foot contact force, foot linear velocity, changes in organism mass, contact material parameters, and terrain height scan data. The teacher encoder generates teacher latent variables based on the privileged physical information, and the student encoder generates student latent variables based on historical ontological observations, and supervises the student latent variables based on the teacher latent variables.

[0081] In some implementations, the teacher encoder and student encoder can output latent variables of the same dimension. The student latent variables can be fitted to the teacher latent variables using mean squared error loss. Thus, physical information visible during the training phase but not readily available during the deployment phase can be distilled into latent states inferred during the deployment phase through historical ontology observations. In multiple parallel training environments, one group can use the teacher latent variables to generate actions, while another group uses the student latent variables to generate actions, and both groups can work together to update a shared recovery control policy network.

[0082] In some implementations, the velocity estimator generates estimates of the body's linear velocity based on historical ontological observations and uses these estimates as input to the recovery control policy network. The velocity estimator is trained using real body linear velocities available during the training phase as a supervised target. The velocity estimator and the student encoder can operate in parallel; the former more explicitly estimates the body linear velocity, while the latter learns more abstract latent dynamic variables.

[0083] In step 505, a reward function is constructed. The reward function may include at least one of the following: a first reward component, a second reward component, a third reward component, an inverted state penalty, a basic regularization term, a contact impact constraint term, and a state proximity constraint term. Figure 11 The diagram shows the composition of the reward function, where the first reward component, the second reward component, and the third reward component all participate in the training objective, and additional reward terms can be connected to the reward function as needed for training.

[0084] Basic regularization terms can be used to constrain at least one of the following: the degree to which joint position approaches a soft limit, joint torque, joint acceleration, joint velocity, body roll rate, body pitch rate, rate of change of adjacent movements, second-order smoothness of movement, and joint power. Joint power can be determined based on joint torque and joint velocity.

[0085] The contact impact constraint can include a penalty for non-foot contact force and a positive reward for low contact force. The penalty for non-foot contact force increases with increasing non-foot contact force, while the positive reward for low contact force increases with decreasing non-foot contact force. During the initial preset control period of the training round, the penalty for non-foot contact force or contact impact statistics can be skipped to reduce the interference of initial collisions on policy learning.

[0086] In some implementations, additional penalties may be imposed on leg joint speeds exceeding a set threshold. Within the threshold, the robot is allowed to complete rollovers, braces, and lifts at appropriate speeds; exceeding the threshold increases the penalty to suppress violent swinging of the leg joints during recovery.

[0087] The state proximity constraint can be strengthened as the robot approaches the target standing state. For example, the closer the joint position is to the target standing configuration, the stronger the joint velocity penalty or joint torque penalty; the closer the body posture is to the target upright posture, the stronger the body angular velocity penalty; and the closer the body height is to the target height range, the stronger the body linear velocity penalty. Thus, the robot retains the necessary motion capabilities in the early recovery phase and gradually increases damping and stability in the later recovery phase.

[0088] In step 506, the relevant data undergoes a left-right mirror transformation, and the recovery control policy network is updated based on the action consistency loss between the original state and the mirror state. Data eligible for left-right mirror transformation includes at least one of the following: current prophecy observations, historical prophecy observations, leg movements, contact forces, foot velocities, control commands, privileged observations, and terrain height scan data. Utilizing the symmetry of the left and right lower limb structures can reduce the policy's preference for a particular side of the rising posture.

[0089] In step 507, the network parameters of the recovery control policy network are updated based on the reward function. The update of network parameters can simultaneously utilize the original samples, mirror samples, teacher latent variable supervision, real organism linear velocity supervision, reward function, and constraint terms.

[0090] In step 508, it is determined whether the robot meets the target standing state conditions. The target standing state conditions may include the robot's height being within the target height range, the robot's tilt angle being less than a tilt angle threshold, and the error between the leg joints and the target standing configuration being less than a configuration error threshold, and the above conditions being maintained for a preset duration.

[0091] In step 509, if the target standing state condition is met for a preset duration, recovery is determined to be successful, the current training round ends early, and the value bootstrapping is processed according to the time truncation method; if the target standing state condition is not met, the state observation, joint control output, reward function construction, and network parameter update process can continue to be executed.

[0092] like Figure 6 As shown in the illustration, this application embodiment also provides an autonomous recovery control strategy network training device 30. The training device 30 includes an environment and observation module 301, an action generation module 302, a reward construction module 303, and a network update module 304. The environment and observation module 301 is used to position the robot in a random initial non-standing posture in the training environment and acquire state observation information. The action generation module 302 is used to input the state observation information into the recovery control strategy network to obtain joint control quantities used to drive the robot to perform recovery actions. The reward construction module 303 is used to construct training rewards based on the state observation information. The training reward can be any one of the above-mentioned reward function, cooperative recovery reward, stability maintenance reward, or height-gated posture reward. The network update module 304 is used to update the network parameters of the recovery control strategy network based on the constructed training rewards.

[0093] like Figure 7 As shown in the illustration, this application also provides an electronic device 40, which includes a processor 401 and a memory 402. The memory 402 stores a computer program, which, when executed by the processor 401, implements any of the training methods described above. The electronic device 40 can be a training server, a control computing device, an edge computing device, or other devices with data processing capabilities.

[0094] like Figure 8As shown in the illustration, this application also provides a robot 50, which includes a lower limb assembly 501, a controller 502, and a memory 503. The memory 503 stores a recovery control strategy network trained using any of the training methods described above. The controller 502 acquires the robot's state observation information and inputs this information into the recovery control strategy network to output joint control quantities for driving the lower limb assembly 501 to perform recovery actions. The lower limb assembly 501 may include a leg joint assembly or a wheel-leg composite assembly.

[0095] like Figure 9 As shown, this embodiment also provides a training process data stream. State observation information 601 includes current ontology observations and historical ontology observations; joint control quantities 602 include leg joint control quantities or wheel joint control quantities, used to drive the lower limb assembly 501; reward function 603 includes a first reward component, a second reward component, and a third reward component, etc. State observation information 601 is input to the recovery control strategy network 60, and the recovery control strategy network 60 outputs joint control quantities 602. A reward function 603 is constructed based on the state observation information, and the network parameters of the recovery control strategy network 60 are updated based on the reward function 603. During the training phase, the teacher encoder 604 generates teacher latent variables based on privileged physical information, the student encoder 605 generates student latent variables based on historical ontology observations, and simultaneously, the velocity estimator 606 generates an estimate of the body linear velocity based on historical ontology observations. The teacher latent variables can supervise the student latent variables, and the actual body linear velocity during the training phase can supervise the velocity estimator 606. During the deployment phase, only the action output parts of the student encoder, velocity estimator, and recovery control strategy network can be used, without requiring the input of the privileged physical information from the training phase.

[0096] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-described training methods.

[0097] The above descriptions are merely optional embodiments of this application and are not intended to limit this application. Those skilled in the art can combine, substitute, or adjust the features in each embodiment without departing from the concept of this application, and such combinations, substitutions, or adjustments should all fall within the protection scope of this application.

Claims

1. A method for training an autonomous recovery control policy network for a robot with a lower limb assembly for mobility based on reinforcement learning, characterized in that, include: A robot with a lower limb assembly for movement is placed in a random initial non-standing posture in a training environment, and state observation information of the robot is acquired. The state observation information is input into the recovery control strategy network to obtain the joint control quantities used to drive the robot to perform recovery actions; A reward function for training the recovery control policy network is constructed based on the state observation information. The reward function includes a first reward component, a second reward component, and a third reward component. The first reward component is used to suppress the state in which the robot only meets some of the recovery targets of height recovery, posture recovery and joint configuration recovery; The second reward component is used to suppress the robot from remaining in a non-converged state after reaching the target standing position; The third reward component is used to suppress the state where the robot's body posture is close to the target upright posture but the body height does not reach the corresponding height requirement; The network parameters of the recovery control strategy network are updated based on the reward function, so that the joint control quantity output by the updated recovery control strategy network drives the robot to recover from the non-standing posture to the target standing state and maintain stability.

2. The method according to claim 1, characterized in that, The first reward component is determined based on the synergistic coupling result of the height recovery evaluation quantity, the posture recovery evaluation quantity, and the joint configuration recovery evaluation quantity; the second reward component is determined based on the coupling result of the recovery evaluation quantity and the degree of stillness; and the third reward component is obtained by gating the posture recovery evaluation quantity based on the height gating quantity.

3. The method according to claim 1 or 2, characterized in that, The state observation information includes current body observation and historical body observation. The current body observation includes at least one of the following: body angular velocity, projected gravity direction in the body coordinate system, relative position of non-foot joints, foot odometer position increment, joint velocity, and the previous control action. The historical body observation includes body observations from multiple consecutive control cycles.

4. The method according to claim 1, characterized in that, The random initial posture of the non-standing posture is obtained by randomizing at least one of the body roll angle, body pitch angle, joint initial position and joint initial velocity; when the joint initial position is randomized, the randomized joint initial position is restricted within the joint soft limit; during the training process, at least one of the following is also randomized: ground friction parameters, collision recovery coefficient, body mass, leg component mass and / or inertia, actuator stiffness, actuator damping and actuator control delay.

5. The method according to claim 1, characterized in that, The joint control quantities include leg joint control quantities and foot joint control quantities. During the recovery training process, the motion scaling factor of the foot joint control quantities is set to zero or the foot joints are restricted from being used as active recovery drivers, so that the recovery movements are mainly generated by the leg joints in the lower limb components.

6. The method according to claim 1, characterized in that, During the training phase, teacher latent variables are generated by the teacher encoder based on privileged physical information, and student latent variables are generated by the student encoder based on historical ontology observations. The student latent variables are supervised based on the teacher latent variables. The velocity estimator generates the estimated linear velocity of the organism based on historical ontology observations, and the velocity estimator is trained under the supervision of the real linear velocity of the organism that is available during the training phase.

7. The method according to claim 1, characterized in that, The reward function further includes at least one of the following: inverted state penalty, basic regularization term, contact impact constraint term, and state proximity constraint term; wherein, the inverted state penalty is applied when it is determined that the robot is in an inverted state based on the robot's posture information, the contact impact constraint term is not calculated within the preset control period at the beginning of the training round, and the state proximity constraint term is enhanced as the robot approaches the target standing state.

8. A method for training an autonomous recovery control policy network for a robot with a lower limb assembly for mobility based on reinforcement learning, characterized in that, include: A robot with a lower limb assembly for movement is placed in a random initial non-standing posture in a training environment, and state observation information of the robot is acquired, including body height information, body posture information, and leg joint state information. The state observation information is input into the recovery control strategy network to obtain the joint control quantities used to drive the robot to perform recovery actions; Based on the body height information, the body posture information, and the leg joint state information, multiple recovery evaluation quantities are determined to characterize the degree of height recovery, posture recovery, and joint configuration recovery. By synergistically coupling the multiple recovery evaluation quantities, a synergistic recovery reward is obtained, such that the synergistic recovery reward is suppressed as any of the recovery evaluation quantities deviates from the corresponding recovery target; The network parameters of the recovery control strategy network are updated based on the collaborative recovery reward, so that the joint control quantity output by the updated recovery control strategy network drives the robot to recover from the non-standing posture to the target standing state.

9. The method according to claim 8, characterized in that, The multiple recovery evaluation metrics include height score, posture score, and joint configuration score. The height score, posture score, and joint configuration score are all continuous scores that increase as the corresponding deviation decreases. Specifically, the height score is determined based on the deviation between the body height and the target height range, the posture score is determined based on the deviation between the projected gravity direction and the target upright direction, and the joint configuration score is determined based on the deviation between the leg joint state and the target standing configuration.

10. The method according to claim 9, characterized in that, The collaborative recovery reward is determined based on the product of the height score, the posture score, and the joint configuration score, and is used to suppress at least one of the following states: the body posture meets the upright requirement but the body height does not reach the target height; the body height reaches the target height but the body posture does not meet the upright requirement; the body height and body posture meet the requirements but the leg joint configuration does not reach the target standing configuration.

11. The method according to claim 9, characterized in that, The target standing configuration includes the target configuration of the left and right leg joints, and the deviation between the leg joint state and the target standing configuration includes the leg joint angle error or the leg joint configuration mean square error; the joint control amount includes the position control amount of each leg joint.

12. A method for training an autonomous recovery control policy network for a robot with a lower limb assembly for mobility based on reinforcement learning, characterized in that, include: A robot with a lower limb assembly for movement is placed in a random initial non-standing posture in a training environment, and state observation information of the robot is acquired, including recovery state information and motion state information. The state observation information is input into the recovery control strategy network to obtain the joint control quantities used to drive the robot to perform recovery actions; Based on the recovery status information, a recovery evaluation quantity is determined to characterize the robot's approach to the target standing state; The degree of stillness is determined based on the motion state information, and the degree of stillness is used to characterize the degree of motion inhibition of the robot during the recovery process; The recovery evaluation quantity and the stillness degree are coupled to obtain the stability retention reward, which is reduced as either the recovery evaluation quantity or the stillness degree decreases; The network parameters of the recovery control strategy network are updated based on the stability retention reward, so that the joint control quantity output by the updated recovery control strategy network drives the robot to recover from the non-standing posture to the target standing state and maintain stability.

13. The method according to claim 12, characterized in that, The recovery status information includes at least two of the following: body height information, body posture information, and leg joint status information; the degree of stillness is determined based on at least one of leg joint velocity, body angular velocity, and body linear velocity, and increases as the leg joint velocity, body angular velocity, or body linear velocity decreases; the stability retention reward is determined based on the product of the recovery evaluation quantity and the degree of stillness.

14. The method according to claim 12 or 13, characterized in that, When the robot continuously meets the target standing state conditions for a preset time, it is determined that the recovery is successful. The target standing state conditions include the robot height being within the target height range, the robot tilt angle being less than the tilt angle threshold, and the error between the leg joints and the target standing configuration being less than the configuration error threshold. After the recovery is determined to be successful, the current training round is terminated early, and the value bootstrapping is processed according to the time truncation method.

15. The method according to claim 12, characterized in that, Increase joint velocity penalty or joint torque penalty based on the degree to which the joint position is close to the target standing configuration; increase body angular velocity penalty based on the degree to which the body posture is close to the target upright posture; and / or increase body linear velocity penalty based on the degree to which the body height is close to the target height range.

16. A method for training an autonomous recovery control policy network for a robot with a lower limb assembly for mobility based on reinforcement learning, characterized in that, include: A robot with a lower limb assembly for movement is placed in a random initial non-standing posture in a training environment, and state observation information of the robot is acquired, including body height information and body posture information. The state observation information is input into the recovery control strategy network to obtain the joint control quantities used to drive the robot to perform recovery actions; The posture recovery evaluation value is determined based on the body posture information. The posture recovery evaluation value is used to characterize the degree to which the robot's body posture is close to the target upright posture. The height gating amount is determined based on the body height information, and the height gating amount is used to perform gating processing on the posture recovery evaluation amount to obtain a height-gated posture reward. This results in the suppression of the height-gated posture reward when the robot's body posture is close to the target upright posture but the body height does not reach the corresponding height requirement. The network parameters of the recovery control strategy network are updated based on the height-gated posture reward, so that the joint control quantity output by the updated recovery control strategy network drives the robot to recover from the non-standing posture to the target standing state.

17. The method according to claim 16, characterized in that, The aircraft attitude information includes the projected gravity direction, and the attitude recovery evaluation value is determined based on the deviation between the projected gravity direction and the target upright direction; the height threshold value is determined based on the difference between the aircraft height and the target height range, and increases as the aircraft height approaches the target height range.

18. The method according to claim 16 or 17, characterized in that, The height-gated attitude reward is determined based on the product of the attitude recovery evaluation and the height gating value, and is used to suppress the state in which the robot body attitude approaches the target upright attitude but is still close to the ground.

19. The method according to claim 16, characterized in that, When the robot is determined to be in a completely inverted state based on the direction of projected gravity. An inverted state penalty is applied; a target height score is determined based on the deviation between the body height and the target height range, and the target height score increases as the deviation decreases; the height-gated attitude reward and the target height score are used together to update the recovery control strategy network, so that the recovery control strategy network simultaneously promotes body lifting during the body attitude recovery process.

20. The method according to claim 1, 8, 12 or 16, characterized in that, The lower limb assembly for mobility includes a leg joint assembly or a wheel-leg composite assembly.

21. The method according to claim 1, 8, 12 or 16, characterized in that, The system performs a left-right mirror transformation on at least one of the following data: current body observation, historical body observation, leg action, contact force, foot speed, control command, privileged observation, and terrain height scan data. The recovery control strategy network is then updated based on the action consistency loss between the original state and the mirrored state.

22. A training device for an autonomous recovery control strategy network of a robot with a lower limb assembly for mobility based on reinforcement learning, characterized in that, include: The environment and observation module is used to put a robot with a lower limb assembly for movement into a random initial non-standing posture in the training environment and to acquire state observation information. The motion generation module is used to input the state observation information into the recovery control strategy network to obtain the joint control quantities used to drive the robot to perform recovery actions; A reward construction module is used to construct a training reward based on the state observation information, wherein the training reward is any one of the following: the reward function of claim 1, the collaborative recovery reward of claim 8, the stability maintenance reward of claim 12, or the height-gated attitude reward of claim 16. The network update module is used to update the network parameters of the recovery control policy network based on the training rewards.

23. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the method of any one of claims 1 to 21.

24. A robot having a lower limb assembly for mobility, characterized in that, include: The lower limb assembly, controller, and memory, wherein the memory stores a recovery control strategy network trained by the method of any one of claims 1 to 21, and the controller is used to acquire state observation information of the robot and input the state observation information into the recovery control strategy network to output joint control quantities for driving the lower limb assembly to perform recovery actions.

25. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 21.

Images