Robot fall self-recovery method and apparatus

Abstract

The application discloses a robot falling self-recovery method and device, and relates to the technical field of robots. The method comprises the following steps: detecting the current posture of the robot; in the case that the current posture meets preset conditions, determining the current parameter sequence of the robot according to the current posture, and obtaining the preset parameter sequence of the robot; inputting the current parameter sequence and the preset parameter sequence into a falling recovery model to output a self-recovery strategy of the robot, wherein the falling recovery model is obtained through pre-training by means of imitation learning and reinforcement learning; and executing the self-recovery strategy to make the robot recover to a standing posture, so that the problem that the existing traditional robot falling recovery method has obvious limitations is solved, and the autonomous, stable, robust and human-like falling recovery capability of the humanoid robot is improved.

Description

Robot fall self-recovery method and device Technical Field

[0001] This application relates to the field of robotics, and more specifically, to a method and apparatus for self-recovery from a robot falling. Background Technology

[0002] Humanoid robots, due to their human-like structure, are better able to adapt to complex and unstructured human production environments compared to wheeled and quadrupedal robots. They possess greater environmental adaptability and mobility, making them suitable for performing diverse tasks and thus have broad application prospects in industrial and domestic settings. However, falls are an unavoidable problem when humanoid robots perform tasks in complex environments. Developing autonomous, stable, robust, and human-like fall recovery capabilities is one of the key technologies for enabling humanoid robots to operate reliably in real-world scenarios.

[0003] Traditional methods for robot fall recovery can be broadly categorized into two types: model-driven traditional control methods and data-driven pure reinforcement learning methods. However, both types have significant limitations in terms of adaptability and naturalness of movement. Early humanoid robot control methods relied heavily on model-based trajectory planning. By establishing kinematic and dynamic models of the robot, multi-stage standing trajectories were designed, such as a sequence of actions from a fixed lying posture to a fixed squatting (or kneeling) posture and then to standing. This type of method requires extremely high precision in the robot model, necessitating tedious and repetitive manual parameter tuning of the model and controller. Furthermore, it struggles to cope with uncertainties in real-world environments, such as external disturbances, changes in ground friction, and actuator errors. In addition, model-based traditional control methods require extensive and repetitive parameter adjustments for specific scenarios, lacking adaptability and generalization capabilities for different starting postures (such as lying on one's side or prone) and different scenarios (such as slopes or walls), thus limiting their applicability in real-world human scenarios. Subsequently, with the increasing maturity of Deep Reinforcement Learning (DRL) technology, researchers began to attempt to use DRL methods to learn fall recovery strategies for humanoid robots. These methods do not rely on precise robot models, but rather learn adaptive and robust control strategies through continuous interaction and trial and error between the robot and its environment in a simulation setting. This is achieved by adjusting and perturbing the environmental and robot parameters (i.e., through continuous interaction and trial and error in a simulation environment with both environmental and robot parameter perturbations). However, the convergence performance and control effectiveness of these strategies still depend on the design of the reward function. It is difficult to use simple reward functions to enable the strategy to learn natural, anthropomorphic movement patterns, lacking the coordination exhibited in human standing processes, and prone to problems such as violent movements, excessive torque, and unnatural movement styles. Summary of the Invention

[0004] The main objective of this application is to provide a method and apparatus for self-recovery from robot falls, in order to solve the problem that there are obvious limitations in existing traditional robot fall recovery methods, and to improve the autonomous, stable, robust and human-like fall recovery capabilities of humanoid robots.

[0005] To achieve the above objectives, according to one aspect of the embodiments of this application, a robot fall self-recovery method is proposed, comprising: detecting the current posture of the robot; if the current posture meets preset conditions, determining the current parameter sequence of the robot based on the current posture, and obtaining the preset parameter sequence of the robot; inputting the current parameter sequence and the preset parameter sequence into a fall recovery model, and outputting a self-recovery strategy for the robot, wherein the fall recovery model is obtained through pre-training by imitation learning and reinforcement learning; and executing the self-recovery strategy to enable the robot to recover its standing posture.

[0006] According to another aspect of the embodiments of this application, a robot fall recovery device is also provided, comprising: a detection unit for detecting the current posture of the robot; a determination unit for determining the current parameter sequence of the robot based on the current posture and obtaining the preset parameter sequence of the robot when the current posture meets preset conditions; an input unit for inputting the current parameter sequence and the preset parameter sequence into a fall recovery model and outputting a self-recovery strategy for the robot, wherein the fall recovery model is obtained through pre-training by imitation learning and reinforcement learning; and an execution unit for executing the self-recovery strategy to enable the robot to restore its standing posture.

[0007] Optionally, the above input unit includes: a historical action subunit, used to determine the robot's historical execution actions based on the current parameter sequence and the preset parameter sequence; a splicing parameter subunit, used to splice the current parameter sequence, the preset parameter sequence and the historical execution actions to obtain the current splicing parameters; and a strategy output subunit, used to input the splicing parameters into the fall recovery model and output a self-recovery strategy.

[0008] Optionally, the above-mentioned robot fall self-recovery device further includes: a posture information acquisition unit for acquiring the gravity projection during the robot's operation; and a posture confirmation unit for comparing the gravity projection with a preset projection, and confirming that the robot has recovered its standing posture if the comparison result meets the preset conditions.

[0009] Optionally, the input unit further includes: a sequence acquisition subunit, used to acquire a robot simulation model and multiple non-standing postures, and to acquire human motion data sequences and robot initial parameter sequences corresponding to each non-standing posture; a sequence preprocessing subunit, used to preprocess the multiple human motion data sequences to obtain multiple reference fall recovery trajectories; an imitation learning subunit, used to iteratively train the initial fall recovery model through imitation learning based on the robot simulation model, multiple robot initial parameter sequences, and multiple reference fall recovery trajectories to obtain a reference fall recovery model; and a reinforcement learning subunit, used to iteratively train the reference fall recovery model through reinforcement learning based on the robot simulation model, multiple robot initial parameter sequences, and multiple reference fall recovery trajectories to obtain a fall recovery model.

[0010] Optionally, the above sequence preprocessing subunit includes: a motion data subunit, used to acquire multiple human motion data sequences corresponding to the human data model, wherein each human motion data sequence corresponds to a fall recovery mode; a first trajectory subunit, used to map the multiple human motion data sequences to the robot simulation model to obtain multiple first reference recovery trajectories; and a reference trajectory subunit, used to integrate the multiple first reference recovery trajectories as tracking targets into the first simulation environment where the robot simulation model is located to obtain multiple reference fall recovery trajectories.

[0011] Optionally, the aforementioned imitation learning subunit includes: a first sequence subunit, used to acquire first robot motion data sequences corresponding to multiple reference fall recovery trajectories, and multiple second robot motion data sequences generated by controlling the robot simulation model to perform a standing process according to the multiple reference fall recovery trajectories; a parameter combination subunit, used to pair and combine multiple first robot motion data sequences, multiple second robot motion data sequences, and multiple robot initial parameter sequences according to multiple non-standing postures to obtain multiple parameter sequence combinations; and a first model subunit, used to iteratively train the initial fall recovery model according to the multiple parameter sequence combinations to obtain a reference fall recovery model.

[0012] Optionally, the first model subunit includes: a first splicing subunit, used to distribute multiple parameter sequence combinations to multiple threads for model training, and splice the parameters in the parameter sequence combination corresponding to each single thread to obtain a first splicing result corresponding to each single thread; a first action subunit, used to input the first splicing results into the policy subnetwork in the initial fall recovery model and output the first expected execution action of the robot simulation model; a first value subunit, used to obtain multiple preset tracking parameters corresponding to the robot simulation model, splice each first splicing result and multiple preset tracking parameters to obtain multiple second splicing results, and input the multiple second splicing results into the value subnetwork in the initial fall recovery model to output a first value estimate; and a first iteration subunit, used to iteratively train the initial fall recovery model using the PPO algorithm based on the first expected execution action and the first value estimate, with the first comprehensive reward as the reward signal, to obtain a reference fall recovery model.

[0013] Optionally, the reinforcement learning subunit includes: a second environment subunit, used to introduce multiple domain randomizations into the simulation definition parameters corresponding to the first simulation environment to obtain an updated second simulation environment; a second action subunit, used to add random noise to each first splicing result based on the second simulation environment to obtain multiple third splicing results, and input the third splicing results into the policy subnetwork in the reference fall recovery model to output the second expected execution action of the robot simulation model; a second value subunit, used to splice each third splicing result with multiple preset tracking parameters to obtain multiple fourth splicing results, and input the multiple fourth splicing results into the value subnetwork in the reference fall recovery model to output the second value estimate; and a second iteration subunit, used to iteratively train the reference fall recovery model using the PPO algorithm based on the second expected execution action and the second value estimate, with the second comprehensive reward as the reward signal, to obtain the fall recovery model.

[0014] Optionally, the first comprehensive reward includes: global tracking reward, local tracking reward and constraint reward; the second comprehensive reward includes: global tracking reward, local tracking reward, constraint reward and preset reward.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: at least one processor, and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the above-described robot fall self-recovery method.

[0016] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0017] In this application, the above-mentioned robot fall recovery method and device solve the obvious limitations of existing traditional robot fall recovery methods, and improve the autonomous, stable, robust and human-like fall recovery capabilities of humanoid robots. Attached Figure Description

[0018] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application. In the drawings:

[0019] Figure 1 is a schematic diagram of the hardware environment for an optional robot fall self-recovery method provided in this application;

[0020] Figure 2 is a flowchart of an optional robot fall self-recovery method provided in this application;

[0021] Figure 3 is a flowchart of another optional robot fall self-recovery method provided in this application;

[0022] Figure 4 is a schematic diagram of an optional robot fall recovery device provided in this application;

[0023] Figure 5 is a schematic diagram of an optional electronic device provided in this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in any order other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. Without conflict, the embodiments and features in the embodiments of this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] To address the significant limitations of existing traditional robot fall recovery methods (such as adaptability, robustness, and anthropomorphic movement style in humanoid robot fall recovery), and to improve the autonomous, stable, robust, and anthropomorphic fall recovery capabilities of humanoid robots, this application provides a humanoid robot fall recovery strategy method and corresponding device based on imitation learning and reinforcement learning, namely, a robot fall self-recovery method and device. This application first uses imitation learning to learn prior action patterns from human motion data, enabling the strategy to learn an anthropomorphic fall recovery method under ideal conditions. Then, reinforcement learning is used to train a highly robust and environmentally adaptable strategy based on the anthropomorphic movement prior strategy, improving its anti-interference ability and motion performance. This enables the humanoid robot to autonomously and stably complete fall recovery tasks in complex environments, while reducing a series of problems caused by unnatural movement styles, thus endowing the robot with basic motion capabilities for working in complex environments. As an optional embodiment, the above-described robot fall self-recovery method can be applied, but is not limited to, to the robot fall self-recovery system consisting of terminal device 102 and server 104 as shown in FIG1. ​​As shown in FIG1, terminal device 102 is connected to server 104 through network 110. The network 110 may include, but is not limited to, wired networks and wireless networks. The wired network includes, but is not limited to, local area networks, metropolitan area networks, and wide area networks. The wireless network includes, but is not limited to, Bluetooth, WIFI, and other networks that enable wireless communication. The terminal device 102 may include, but is not limited to, at least one of the following: mobile phones (such as Android phones, iOS phones, etc.), laptops, tablets, handheld computers, MIDs (Mobile Internet Devices), desktop computers, smart TVs, in-vehicle devices, etc.

[0027] The aforementioned terminal device 102 is also equipped with a display 106, a processor 108, and a memory 112. The display 106 can be used to display the robot's fall recovery process, etc. The processor 108 can be used to process the collected data, and the memory 112 can be used to store relevant data. It is understood that when the terminal device 102 receives a robot fall recovery request, it sends the request to the server 104 via the network 110. The server 104 responds to the request and implements the specific robot fall recovery process.

[0028] The aforementioned server 104 can be a single server, a server cluster consisting of multiple servers, or a cloud server. The aforementioned server 104 includes a database 114 and a processing engine 116. The database 114 can be used to store data, models, etc., and the processing engine 116 is used to process the aforementioned data.

[0029] According to one aspect of the present invention, the above-described robot fall self-recovery system may further perform the following steps: First, the terminal device 102 executes S102 as shown in FIG1; then, the server 104 executes S104 to S110 as shown in FIG1, thereby realizing robot fall self-recovery.

[0030] In the above embodiments of the present invention, the robot fall recovery method described above solves the problem of obvious limitations of existing traditional robot fall recovery methods, and improves the autonomous, stable, robust and human-like fall recovery capabilities of humanoid robots.

[0031] As an optional implementation, please refer to Figure 2, which shows a flowchart of a robot fall self-recovery method according to an embodiment of this application. The method includes at least one of the following steps (S202 to S208):

[0032] S202, detect the robot's current posture; S204, if the current posture meets the preset conditions, determine the robot's current parameter sequence based on the current posture, and obtain the robot's preset parameter sequence; S206, input the current parameter sequence and the preset parameter sequence into the fall recovery model, and output the robot's self-recovery strategy, wherein the fall recovery model is obtained through pre-training by imitation learning and reinforcement learning; S208, execute the self-recovery strategy to enable the robot to restore its standing posture.

[0033] Specifically, in step S202, the current state of the robot (whether it is in a fall state) can be detected by using an IMU (Inertial Measurement Unit) installed near the base link (hip) to obtain the current gravity projection vector of the humanoid robot. This gravity projection vector reflects the robot's acceleration in the x, y, and z directions. In the default standing state, the z-axis component of the gravity projection vector is close to the gravitational acceleration (-9.81 m / s²); when the robot falls or rolls over, the value of this z-axis component will be greater than 0 m / s². During actual deployment, the system runs multiple processes: one process is responsible for the robot's motion output (e.g., controlling normal walking, performing special actions, or realizing fall recovery), and another independent monitoring process is dedicated to monitoring the robot's state variables and automatically selecting and switching motion output strategies accordingly. The working logic of this monitoring process is as follows: 1. Fall detection and recovery: When the z-axis component of the gravity projection vector is detected to be greater than 0 m / s², it is determined that the robot is in a fall state. Subsequently, the system switches the motion output strategy to the self-recovery strategy until the robot completes the fall recovery action. 2. Standing Determination and Strategy Switching: When the system detects that the z-axis component of the gravity projection vector is close to the gravitational acceleration (-9.81 m / s²) for more than 1 second, it determines that the robot has reached a stable standing state. Subsequently, the system will switch to other corresponding action output strategies.

[0034] In step S204 above, the current parameter sequence refers to the robot parameter sequence corresponding to the robot's current fallen state, and the preset parameter sequence refers to the robot parameter sequence in the ideal state. In step S206 above, the robot's self-recovery strategy is obtained through the fall recovery model. In step S208, executing the self-recovery strategy includes generating torque from the underlying PD controller to control the robot to complete the corresponding action. The self-recovery scheme in S206 above (the action executed from the robot's posture at the current timestamp to the robot's posture at the next timestamp, where the current timestamp and the next timestamp are two adjacent timestamps in the robot's fall recovery and standing process). The input to the PD controller is the self-recovery strategy, and the output is torque.

[0035] The above-described embodiments of this application, employing the robot fall recovery method, solve the problem of significant limitations in existing traditional robot fall recovery methods, and improve the autonomous, stable, robust, and human-like fall recovery capabilities of humanoid robots.

[0036] As an optional implementation, the above-described method inputs the current parameter sequence and a preset parameter sequence into the fall recovery model and outputs the robot's self-recovery strategy, including:

[0037] S1, determine the robot's historical execution actions based on the current parameter sequence and the preset parameter sequence;

[0038] S2, concatenate the current parameter sequence, the preset parameter sequence, and the historical execution actions to obtain the current concatenated parameters;

[0039] S3 inputs the splicing parameters into the fall recovery model and outputs a self-recovery strategy.

[0040] It should be noted that in step S1 above, the historical execution actions refer to a series of specific behaviors or actions performed by the robot before or during the fall. These actions are operations already completed by the robot, and their data is recorded and can be used to analyze how the robot entered the current fall state. Step S1 can be understood, but is not limited to, in the case of multiple robots, determining the target robot based on the current parameter sequence and a preset parameter sequence, and then further determining the historical execution actions of the target robot after determining the target robot. These historical execution actions can be recorded in real time during the robot's recovery process. Historical execution actions are operations already completed by the robot. If the robot has no historical execution actions (i.e., the robot is in the initial fall state and has not performed any recovery actions), the historical execution actions are the current joint positions. That is, when the robot switches its fall recovery strategy, this application will manually reset the historical execution actions to the current joint positions. In this application, "action" refers to the robot's own parameter information (such as joint angles) under the corresponding action, as well as the robot's motion parameters (such as joint angular velocity) under the current action. In step S2 above, the splicing operation can be implemented through a preset function. As shown in Figure 3, the splicing parameters are composed of three types of information: the current parameter sequence, the preset parameter sequence, and the historical executed actions. They contain complete information about the robot's "current state" (current parameter sequence), "ideal state" (preset parameter sequence), and "past behavior" (historical executed actions). In step S3, the splicing parameters provide the fall recovery model with complete and rich information, enabling the fall recovery model to output a more accurate self-recovery strategy.

[0041] For example, historical execution actions include, but are not limited to: movement actions (walking fast, crossing obstacles, etc.), manipulation actions (lifting heavy objects, throwing objects, etc.), posture adjustment actions (squatting, standing on one leg), etc. The historical execution actions are the expected and actually executed joint angles. Historical execution actions can indicate why the robot fell. By splicing operations, the isolated "current state" (current parameter sequence) can be combined with the "ideal state" (preset parameter sequence) and "past behavior" (historical execution actions), thereby implicitly determining that the robot is now falling backward (current parameter sequence) due to the historical execution operation of "walking fast", and has not achieved the ideal state of walking fast (preset parameter sequence). The imbalance and fall may be related to the low ground friction. The generated self-recovery strategy should pay attention to small and slow adjustments to avoid secondary falls.

[0042] As an optional implementation, after executing the above self-recovery strategy, the method further includes:

[0043] S210, acquire the gravity projection during the robot's operation; S212, compare the gravity projection with the preset projection, and if the comparison result meets the preset conditions, confirm that the robot has returned to its standing posture.

[0044] The above operation process can be understood, but is not limited to, the process of the robot executing a self-recovery strategy to stand up. In addition, after executing the above self-recovery strategy, the robot's recovery of its standing posture can be confirmed through the following S1-S3 processes:

[0045] S1, obtain the preset reference parameter sequence set, and the target parameter sequence after the robot executes the self-recovery strategy;

[0046] S2, perform parameter sequence matching between the target parameter sequence and the preset reference parameter sequence set to obtain the matching result;

[0047] S3, if the matching result meets the preset standing conditions, confirm that the robot has resumed its standing posture.

[0048] In other words, after executing the above self-recovery strategy, the robot's recovery of its standing posture can be confirmed by either the gravity projection ratio (S210-S212 above) or parameter sequence matching (S1-S3 above). Both the parameter sequence matching and the preset standing conditions can be pre-set according to actual needs, and this application does not impose any limitations on them.

[0049] As an optional implementation, the above-mentioned pre-training through imitation learning and reinforcement learning includes:

[0050] S1, acquire the robot simulation model and multiple non-standing postures, and acquire the human motion data sequence and robot initial parameter sequence corresponding to each non-standing posture based on the multiple non-standing postures;

[0051] S2, preprocess multiple human motion data sequences to obtain multiple reference fall recovery trajectories;

[0052] S3. Based on the robot simulation model, multiple robot initial parameter sequences, and multiple reference fall recovery trajectories, the initial fall recovery model is iteratively trained through imitation learning to obtain the reference fall recovery model.

[0053] S4. Based on the robot simulation model, multiple robot initial parameter sequences, and multiple reference fall recovery trajectories, the reference fall recovery model is iteratively trained through reinforcement learning to obtain the fall recovery model.

[0054] It should be noted that the above methods of obtaining robot simulation models include pre-building robot simulation models, obtaining existing robot simulation models, etc., and robot simulation models can also be obtained through robot model files. The model obtained in this application can be understood, but is not limited to, as a model after being imported into the simulation environment (i.e., a model combining the robot model and the simulation environment). Commonly used robot model files refer to standardized files used to digitally describe the robot's structure, physical properties, sensor configuration, and behavioral rules. They do not directly contain 3D assemblies (such as STL and OBJ files), but will be associated with these resources through introduction or embedding to ultimately form a complete virtual robot model required for simulation or control. Specifically, these include URDF, XML, and USD description files.

[0055] The robot simulation model includes the dimensions, mass, moment of inertia, simulated collision volume, motion type, angle range, and velocity and torque limits of each joint of the humanoid robot. After obtaining the robot simulation model, it needs to be imported into a simulation environment. Commonly used simulation environments include Isaac Gym, Isaac Sim, Mujuco, and Pybullet. When importing the robot into the simulation environment, it is necessary to verify the correctness of the simulation model and the accuracy of the import (therefore, this part can be understood, but is not limited to, as obtaining the data required for training and simulation deployment from the simulation environment).

[0056] Methods for obtaining non-standing posture include, but are not limited to, descriptive text of non-standing posture, images representing non-standing posture, sensor data, etc.

[0057] The aforementioned human motion data sequence refers to the sequence of human movements as they rise from different postures, including key point location information. Methods for obtaining human motion data sequences include: motion data acquisition: using motion capture equipment (optical motion capture equipment, inertial motion capture equipment, etc.) to collect human motion data sequences of rising from different postures (supine, prone, lateral, etc.) in different ways (simultaneous exertion of both legs, alternating exertion of both legs, etc.) (the principle is to record and analyze the complex motion trajectory of the human body in three-dimensional space by capturing the three-dimensional spatial coordinate changes of reflective markers fixed to the surface of the human skin); and direct acquisition from human motion databases: obtaining human motion data sequences from publicly available human motion databases (such as AMASS, LAFAN1, etc.), which include predefined information such as the location sequence of key human points. In addition, human motion data sequences can be used to construct human data models, thereby enabling the visualization and analysis of human motion data sequences. Human data models are advanced parametric models used in computer graphics and computer vision for 3D human modeling. They provide an efficient and flexible way to represent the human body by statistically learning changes in human shape and posture. Common human data models include SMPL, SMPL+H, and SMPL-X.

[0058] The aforementioned robot initial parameter sequence refers to the initial posture parameters corresponding to the robot's actual initial non-standing posture in the simulation environment. These parameters include initial motion parameters and initial body parameters. The initial motion parameters refer to the overall motion state parameters of the robot when it begins to perform the recovery action, indicating the robot's kinematic characteristics, not the specific shape of its body parts. The initial body parameters refer to the specific positions and postures of the robot's various links when it begins the recovery action, indicating the robot's "posture" at that moment. The motion parameters during the robot's recovery action are related to each joint of the robot, while the robot body parameters are related to each link of the robot. The aforementioned robot initial parameter sequence can be used, but is not limited to, to determine whether the robot has fallen. Once the robot determines that it has fallen, it executes the fall recovery scheme of this application (i.e., the technical solution for the process from fall recovery to standing).

[0059] In addition, it can also obtain historical training data of the robot in the simulation environment, such as the sequence of ontological state observations over the past T time points. Topographic data Robot-environment interaction data (Including contact forces of each link, etc.).

[0060] The above steps S1-S4 can be understood as follows: step S1 provides complete data preparation for subsequent operations; step S2 transforms the original human motion data sequence into high-quality multiple reference fall recovery trajectories; and steps S3 and S4, through imitation learning and reinforcement learning respectively, enable the final fall recovery model to learn quickly and have better performance.

[0061] As an optional implementation, the above preprocessing of multiple human motion data sequences yields multiple reference fall recovery trajectories, including:

[0062] S2-1, Obtain multiple human motion data sequences corresponding to the human data model, wherein each human motion data sequence corresponds to a fall recovery mode;

[0063] S2-2, multiple human motion data sequences are mapped to the robot simulation model to obtain multiple first reference recovery trajectories;

[0064] S2-3, multiple first reference recovery trajectories are integrated into the first simulation environment where the robot simulation model is located as tracking targets, to obtain multiple reference fall recovery trajectories.

[0065] In S2-1 above, each human motion data sequence corresponds to a fall recovery mode. A fall recovery mode can be understood, but is not limited to, as the way to transition from an initial non-standing posture to a final standing posture (for example, two fall recovery modes may have the same initial non-standing posture and final standing posture, but different intermediate recovery (standing up) processes), i.e., a way to stand up from that non-standing posture. In S2-2 above, motion redirection technology can be used to map the human motion data sequence to a robot simulation model to obtain a first reference recovery trajectory. This first reference recovery trajectory includes the positions of each joint of the robot and the robot's central link (hip). Regarding position and orientation, the aforementioned reference recovery trajectory includes the positions and velocities of each robot joint, predefined link positions and orientations, link linear velocities, and angular velocities. While the first reference recovery trajectory can reproduce the fall recovery process, its feasibility has not been verified by physical constraints, and issues such as clipping and collisions may occur. Therefore, through steps S2-3, this dynamic and trackable first reference recovery trajectory is integrated into the first simulation environment as a tracking target. In this first simulation environment, the robot model tracks the first reference recovery trajectory and, under the constraints of physical rules, autonomously adjusts and generates a smooth and dynamically reasonable reference fall recovery trajectory. The optimized reference fall recovery trajectory fully considers the dynamic characteristics and physical feasibility of the robot model, thereby ensuring the executability of the reference fall recovery trajectory on the physical robot and effectively avoiding risks such as clipping, collisions, self-interference, and exceeding control command limits.

[0066] As an optional implementation, the above-mentioned reference fall recovery model is obtained by iteratively training the initial fall recovery model through imitation learning based on the robot simulation model, multiple robot initial parameter sequences, and multiple reference fall recovery trajectories, including:

[0067] S3-1, Obtain the first robot motion data sequence corresponding to multiple reference fall recovery trajectories, and the second robot motion data sequence generated by the robot simulation model according to the multiple reference fall recovery trajectories to perform the standing process;

[0068] S3-2, Based on multiple non-standing postures, multiple first robot motion data sequences, multiple second robot motion data sequences, and multiple robot initial parameter sequences are paired and combined to obtain multiple parameter sequence combinations;

[0069] S3-3, the initial fall recovery model is iteratively trained based on multiple parameter sequence combinations to obtain a reference fall recovery model.

[0070] In step S3-1 above, the first robot motion data sequence refers to the parameter information of the robot corresponding to the aforementioned reference fall recovery trajectory, and the second robot motion data sequence refers to the actual joint information generated when the robot simulation model executes the reference fall recovery trajectory. The aforementioned robot simulation model executing the standing process according to multiple reference fall recovery trajectories refers to the process of the robot simulation model standing up in the simulation environment.

[0071] In the multiple parameter sequence combinations obtained in step S3-2 above, each parameter sequence combination corresponds to a non-standing posture.

[0072] In step S3-2 above, iterative training can employ policy-based reinforcement learning methods (such as PPO), value-based reinforcement learning methods (such as Q-learning), or a combination of both (such as Actor-Critic).

[0073] Specifically, the first robot motion data sequence represents the ideal, standard recovery action pattern, providing expert demonstration data for the initial fall recovery model. The second robot motion data sequence represents the actual recovery action pattern of the initial fall recovery model when executing the ideal command, which may contain deviations from the ideal recovery action pattern. Pairing and combining multiple first robot motion data sequences, multiple second robot motion data sequences, and multiple robot initial parameter sequences can be understood as associating three types of data from the same recovery attempt based on different initial non-standing postures of the robot (lying flat, prone, etc.): namely, the "first robot motion data sequence" as a demonstration, the "second robot motion data sequence" reflecting the actual simulation response, and the "robot initial parameter sequence" describing the initial state (such as posture, angle, etc.), respectively, to form multiple parameter sequence combinations. Finally, a reference fall recovery model can be obtained through iterative training based on the above parameter sequence combinations. Each reference sequence combination includes a second robot motion data sequence and an initial parameter sequence, and each robot corresponds to one reference sequence combination.

[0074] As an optional implementation, the above-mentioned initial fall recovery model is iteratively trained based on multiple parameter sequence combinations to obtain a reference fall recovery model, including:

[0075] S3-3-1: Multiple parameter sequence combinations are assigned to multiple threads for model training. The parameters in the parameter sequence combination corresponding to each single thread are concatenated to obtain the first concatenation result corresponding to each single thread.

[0076] S3-3-2, input the first splicing result into the strategy sub-network in the initial fall recovery model, and output the first expected action of the robot simulation model;

[0077] S3-3-3: Obtain multiple preset tracking parameters corresponding to the robot simulation model, and concatenate each first concatenation result with the multiple preset tracking parameters to obtain multiple second concatenation results. Input the multiple second concatenation results into the value subnetwork in the initial fall recovery model and output the first value estimate.

[0078] S3-3-4, Based on the first expected action and the first value estimate, and using the first comprehensive reward as the reward signal, the initial fall recovery model is iteratively trained using the PPO algorithm to obtain the reference fall recovery model.

[0079] It should be noted that when multiple parameter sequence combinations are assigned to multiple threads for model training, each thread corresponds to one parameter sequence combination. The single thread mentioned above refers to any single thread within the multi-threaded model. The policy sub-network can be understood as the "decision-making" part of the model. After receiving the first concatenation result, it decides what the robot should do (the first expected action). Preset tracking parameters refer to a set of fixed parameters pre-defined within the model, used to define evaluation criteria, such as the desired standing posture and center of gravity position. They provide additional, more global reference information to help evaluate the quality of the current state. The value sub-network can be understood as the "evaluation" part of the model. The value sub-network receives the "second concatenation result" (composed of the "first concatenation result" and the "preset tracking parameters") and evaluates how much reward can be obtained in the long run by executing the current policy in the current state. The obtained evaluation value is the "first value estimate." The first comprehensive reward is a comprehensive reward signal that combines the effect of the "first expected action" (action reward) and the feedback of the "first value estimate" (state value). This allows the model to obtain higher cumulative rewards in the future, i.e., learn more effective fall recovery actions. Iterative training using the PPO (Proximal Policy Optimization) algorithm can efficiently and stably optimize the "initial fall recovery model". By utilizing the "first comprehensive reward" as a guiding signal, it can simultaneously improve the model's decision-making ability (policy subnetwork) and state evaluation ability (value subnetwork), ultimately resulting in a high-performance "reference fall recovery model".

[0080] Take imitation learning through an actor-critic network architecture as an example:

[0081] The Actor-Critic network architecture comprises an Actor network and a Critic network. The Actor network, acting as the policy sub-network, is responsible for generating joint movements based on the current state of the robot simulation model. This network employs position control, and its output is the target angle value that each joint of the robot needs to reach in the next moment. This approach defines joint movements as changes in angle and does not involve posture control. The Critic network, acting as the value sub-network, is responsible for evaluating the value of the current state of the robot simulation model. It aims to predict the expected reward obtained after performing actions in this state, providing guidance for the policy optimization of the Actor network. Both the Actor and Critic networks can be implemented using multi-layer MLP networks, with the number of hidden layers including [512, 512, 256, 128]. For example, at the current moment, the joint angles of the robot's reference trajectory... Joint angular velocities of the robot reference trajectory Gravity projection of robot base linkage Angular velocity of the robot base link The actual joint angles of the robot The actual joint angular velocity of the robot Last action performed By combining the above state variables as input to the Actor network, the Actor network will output the first expected action to be executed. :

[0082]

[0083] in

[0084] All data input into the Actor network is also input into the Critic network. At the same time, preset tracking parameters are input, including the linear velocity of the robot base link. The pose of the predefined link relative to the anchor point (base link) in the reference trajectory. , The predefined pose of the link relative to the anchor point (base link) during the robot's actual movement. , The true values ​​of the above state variables (without random noise) are concatenated and used as input to the Critic network. The model then calculates the state-value function to evaluate the value of the current state.

[0085]

[0086] in, .

[0087] The value of the current state The evaluation assesses the total expected reward (i.e., cumulative reward) that can be obtained in the future, starting from the current state and following the current strategy. The specific reward value of the current action is calculated by imitating the learning reward function.

[0088] As an optional implementation, the above-mentioned method involves iteratively training a reference fall recovery model using reinforcement learning based on a robot simulation model, multiple robot initial parameter sequences, and multiple reference fall recovery trajectories to obtain a fall recovery model, including:

[0089] S4-1, Multiple domain randomizations are introduced into the simulation definition parameters corresponding to the first simulation environment to obtain the updated second simulation environment;

[0090] S4-2, Based on the second simulation environment, random noise is added to each first splicing result to obtain multiple third splicing results, and the third splicing results are input into the policy sub-network in the reference fall recovery model to output the second expected execution action of the robot simulation model;

[0091] S4-3, each third stitching result is stitched together with multiple preset tracking parameters to obtain multiple fourth stitching results, and the multiple fourth stitching results are input into the value subnetwork in the reference fall recovery model to output the second value estimate;

[0092] S4-4, based on the second expected action and the second value estimate, and using the second comprehensive reward as the reward signal, the reference fall recovery model is iteratively trained using the PPO algorithm to obtain the fall recovery model.

[0093] In step S4-1 above, multiple domain randomization includes internal and external randomization: Internal randomization introduces random perturbations to the intrinsic attribute parameters of the robot's body model, aiming to simulate changes in the inherent physical characteristics of a real robot due to manufacturing tolerances, wear, calibration errors, etc. Typical randomization parameters include: robot mass (simulating small deviations in actual weight), robot initial posture (simulating posture differences during each start-up or reset), joint friction (simulating dynamic frictional force changes caused by different levels of lubrication in joints), and control delay (simulating delays caused by signal transmission and processing). External randomization introduces random perturbations to the attributes of the robot's external environment and interacting objects, aiming to simulate the diversity and unpredictable interference of the external environment in the real world. Typical randomization parameters include: ground friction, terrain changes, and random thrust. In step S4-2 above, random noise is used to simulate errors that occur in sensors under deployment conditions. In step S4-3 above, the reinforcement learning training process is carried out in the updated second simulation environment, which helps to improve the robustness and generalization ability of the fall recovery model. In step S4-4 above, the fall recovery model can be understood, but is not limited to, as the policy subnetwork in the target fall recovery model obtained after training.

[0094] It should be noted that during reinforcement learning, the training difficulty can be gradually increased. For example, training can begin on flat ground and then gradually introduce complex terrain such as slopes. In other words, steps S4-1 to S4-4 above use the PPO reinforcement learning algorithm to fine-tune the policy sub-network (Actor network) and the value sub-network (Critic network). The policy trained in this stage can improve its resistance to interference, smoothness of movement, and rationality in real-world environments while maximizing the preservation of anthropomorphic action patterns. After model training is complete, the trained policy sub-network is converted for use in model inference and real-device deployment (e.g., converting the Actor network into an ONNX file); the value sub-network is only used during the training phase to assist the policy sub-network in improving performance and is not used for real-device deployment.

[0095] As an optional implementation, the first comprehensive reward includes: global tracking reward, local tracking reward and constraint reward; the second comprehensive reward includes: global tracking reward, local tracking reward, constraint reward and preset reward.

[0096] The specific locations targeted by global tracking and local tracking can be predefined according to training requirements. Global tracking rewards are used to evaluate the tracking accuracy of the robot as a whole relative to a reference motion trajectory: using a core anchor point of the robot (usually set as the "waist" or body center) as a reference, the error between the actual pose of the robot's anchor point and the expected pose at the corresponding timestamp in the reference motion trajectory is calculated, thus ensuring that the robot's macroscopic motion in global space remains consistent with the expected trajectory. Local tracking rewards are used to evaluate the posture tracking accuracy of various local parts of the robot's body relative to the anchor point: using the posture of predefined parts of the robot (such as limbs and joints) relative to the anchor point (waist) as a reference, the error between the actual relative pose of these local parts and the relative pose defined in the reference motion is calculated, thus further ensuring the standard of detailed movements of various parts of the robot's body. Constraint rewards are used to apply a negative reward when the robot's motion exceeds preset physical limits, thereby ensuring that the robot's movements are within the capabilities of its physical structure and actuators. The preset rewards include regularization rewards and other rewards. Regularization rewards are calculated based on joint velocity, joint torque, and motion smoothness, and are used to encourage the strategy to generate smooth, jitter-free movements. Other rewards are used to limit excessive foot contact force or excessive torque in certain joints. In summary, both global tracking rewards and local tracking rewards fall under the category of tracking rewards. Used to preserve anthropomorphic action patterns, constraint rewards, regularization rewards, and other rewards all belong to action adaptation rewards. This is used to ensure the rationality of the generated actions and improve the robot's motion performance. The second comprehensive reward is a weighted sum of the tracking reward and the action adaptation reward: Among them, the initial tracking reward weight The initial movement adaptation reward is relatively large and gradually decreases with the number of training rounds; It is relatively small, and gradually increases with the number of training rounds.

[0097] It should be noted that in the model training process of this application, the calculation stages of the first comprehensive reward and the second comprehensive reward can adopt the reward calculation stages in the existing Actor-Critic model training process. However, the specific rewards calculated are different. The global tracking reward in this application can be understood as the tracking reward obtained by tracking a preset position in the robot simulation model (a pre-set global tracking position that can indicate the whole robot simulation model, such as a certain position on the waist of the robot simulation model). The local tracking reward can be understood as the tracking reward obtained by tracking other local preset positions in the robot simulation model (such as a certain position on the ankle of the robot simulation model, a certain position on the wrist of the robot simulation model, or a certain joint of the robot model). The local preset positions and preset number of local tracking positions can be determined according to user needs, and this application does not limit them.

[0098] The above-described embodiments of this application solve the obvious limitations of existing traditional robot fall recovery methods and improve the autonomous, stable, robust and human-like fall recovery capabilities of humanoid robots.

[0099] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0100] According to another aspect of the present invention, a robot fall recovery device for implementing the above-described robot fall recovery method is also provided. As shown in FIG4, the device includes: a detection unit 402 for detecting the current posture of the robot; a determination unit 404 for determining the current parameter sequence of the robot based on the current posture when the current posture meets preset conditions, and obtaining the preset parameter sequence of the robot; an input unit 406 for inputting the current parameter sequence and the preset parameter sequence into a fall recovery model and outputting a self-recovery strategy for the robot, wherein the fall recovery model is obtained through pre-training by imitation learning and reinforcement learning; and an execution unit 408 for executing the self-recovery strategy to enable the robot to restore its standing posture.

[0101] The specific methods of execution of each unit in the above device embodiments have been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0102] According to another aspect of the present invention, an electronic device for implementing a robot fall self-recovery method is also provided. This electronic device may be a terminal device or a server as shown in FIG. 5. This embodiment uses the electronic device as a terminal device as an example. As shown in FIG. 4, the electronic device includes: at least one processor 504; and a memory 502 communicatively connected to the at least one processor 504; wherein the memory 502 stores a computer program executable by the at least one processor 504, the computer program being executed by the at least one processor 504 to cause the at least one processor 504 to perform the steps in any of the above method embodiments. The electronic device may be located in at least one of a plurality of network devices in a computer network. The processor may be configured to execute the robot fall self-recovery method via the computer program.

[0103] Optionally, those skilled in the art will understand that the structure shown in Figure 5 is merely illustrative, and the electronic device may also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, or a mobile internet device (MID), etc. Figure 5 does not limit the structure of the aforementioned electronic device.

[0104] The memory 502 can be used to store software programs and modules, such as the program instructions / modules corresponding to the robot fall self-recovery method and device in this embodiment of the invention. The processor 504 executes various functional applications and data processing by running the software programs and modules stored in the memory 502, thereby realizing the robot fall self-recovery method described above. The memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 502 may further include memory remotely located relative to the processor 504, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. As an example, as shown in FIG5, the memory 502 may include, but is not limited to, the units in the robot fall self-recovery device described above. In addition, it may include, but is not limited to, other module units in the robot fall self-recovery device described above, which will not be described in detail in this example.

[0105] Optionally, the transmission device 506 described above is used to receive or send data via a network. Specific examples of the network described above may include wired networks and wireless networks. In one example, the transmission device 406 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. Furthermore, the electronic device described above also includes a display 508 and a connection bus 510 for connecting the various module components within the electronic device.

[0106] Obviously, those skilled in the art should understand that the various units or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0107] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for self-recovery from a robot fall, characterized in that, include: Detect the robot's current posture; When the current posture meets preset conditions, the current parameter sequence of the robot is determined based on the current posture, and the preset parameter sequence of the robot is obtained; the current parameter sequence and the preset parameter sequence are input into a fall recovery model, and the self-recovery strategy of the robot is output, wherein the fall recovery model is obtained through pre-training via imitation learning and reinforcement learning; the self-recovery strategy is executed to enable the robot to recover to a standing posture; the pre-training via imitation learning and reinforcement learning includes: obtaining a robot simulation model and multiple non-standing postures, and obtaining human motion data sequences and robot initial parameter sequences corresponding to each non-standing posture; preprocessing the multiple human motion data sequences to obtain multiple reference fall recovery trajectories; iteratively training the initial fall recovery model through imitation learning based on the robot simulation model, the multiple robot initial parameter sequences, and the multiple reference fall recovery trajectories to obtain a reference fall recovery model; iteratively training the reference fall recovery model through reinforcement learning based on the robot simulation model, the multiple robot initial parameter sequences, and the multiple reference fall recovery trajectories to obtain the fall recovery model.

2. The method according to claim 1, characterized in that, The process involves inputting the current parameter sequence and the preset parameter sequence into a fall recovery model and outputting the robot's self-recovery strategy, including: determining the robot's historical actions based on the current parameter sequence and the preset parameter sequence; concatenating the current parameter sequence, the preset parameter sequence, and the historical actions to obtain current concatenated parameters; and inputting the current concatenated parameters into the fall recovery model to output the self-recovery strategy.

3. The method according to claim 1, characterized in that, After executing the self-recovery strategy, the method further includes: acquiring the gravity projection during the robot's operation; comparing the gravity projection with a preset projection; and confirming that the robot has resumed its standing posture if the comparison result meets the preset conditions.

4. The method according to claim 1, characterized in that, Preprocessing multiple human motion data sequences to obtain multiple reference fall recovery trajectories includes: acquiring multiple human motion data sequences corresponding to a human data model, wherein each human motion data sequence corresponds to a fall recovery mode; mapping the multiple human motion data sequences to the robot simulation model to obtain multiple first reference recovery trajectories; and integrating the multiple first reference recovery trajectories as tracking targets into the first simulation environment where the robot simulation model is located to obtain multiple reference fall recovery trajectories.

5. The method according to claim 4, characterized in that, Based on the robot simulation model, multiple robot initial parameter sequences, and multiple reference fall recovery trajectories, the initial fall recovery model is iteratively trained through imitation learning to obtain a reference fall recovery model. This includes: acquiring first robot motion data sequences corresponding to the multiple reference fall recovery trajectories, and controlling the robot simulation model to execute a standing process according to the multiple reference fall recovery trajectories to generate multiple second robot motion data sequences; pairing and combining the multiple first robot motion data sequences, the multiple second robot motion data sequences, and the multiple robot initial parameter sequences according to multiple non-standing postures to obtain multiple parameter sequence combinations; and iteratively training the initial fall recovery model based on the multiple parameter sequence combinations to obtain the reference fall recovery model.

6. The method according to claim 5, characterized in that, The reference fall recovery model is obtained by iteratively training the initial fall recovery model based on multiple parameter sequence combinations, including: assigning the multiple parameter sequence combinations to multiple threads for model training; concatenating the parameters in the parameter sequence combination corresponding to each single thread to obtain a first concatenation result for each single thread; inputting the first concatenation result into the policy subnetwork of the initial fall recovery model to output the first expected action of the robot simulation model; obtaining multiple preset tracking parameters corresponding to the robot simulation model; concatenating each of the first concatenation results and the multiple preset tracking parameters to obtain multiple second concatenation results; inputting the multiple second concatenation results into the value subnetwork of the initial fall recovery model to output a first value estimate; and iteratively training the initial fall recovery model using the PPO algorithm based on the first expected action and the first value estimate, with a first comprehensive reward as the reward signal, to obtain the reference fall recovery model.

7. The method according to claim 6, characterized in that, Based on the robot simulation model, multiple robot initial parameter sequences, and multiple reference fall recovery trajectories, the reference fall recovery model is iteratively trained using reinforcement learning to obtain the fall recovery model. This includes: introducing multiple domain randomizations into the simulation definition parameters corresponding to the first simulation environment to obtain an updated second simulation environment; adding random noise to each of the first splicing results based on the second simulation environment to obtain multiple third splicing results, and inputting each of the third splicing results into the policy subnetwork of the reference fall recovery model to output the second expected action of the robot simulation model; splicing each of the third splicing results with multiple preset tracking parameters to obtain multiple fourth splicing results, and inputting each of the fourth splicing results into the value subnetwork of the reference fall recovery model to output a second value estimate; and iteratively training the reference fall recovery model using the PPO algorithm based on the second expected action and the second value estimate, with a second comprehensive reward as the reward signal, to obtain the fall recovery model.

8. The method according to claim 7, characterized in that, The first comprehensive reward includes: global tracking reward, local tracking reward, and constraint reward; the second comprehensive reward includes: the global tracking reward, the local tracking reward, the constraint reward, and the preset reward.

9. A robot fall recovery device, characterized in that, include: The detection unit is used to detect the robot's current posture; The determining unit is configured to determine the current parameter sequence of the robot based on the current posture when the current posture meets the preset conditions, and to obtain the preset parameter sequence of the robot. An input unit is used to input the current parameter sequence and the preset parameter sequence into a fall recovery model and output the robot's self-recovery strategy, wherein the fall recovery model is obtained through pre-training via imitation learning and reinforcement learning; an execution unit is used to execute the self-recovery strategy to enable the robot to recover to a standing posture; the pre-training via imitation learning and reinforcement learning includes: acquiring a robot simulation model and multiple non-standing postures, and acquiring human motion data sequences and robot initial parameter sequences corresponding to each of the multiple non-standing postures; preprocessing the multiple human motion data sequences to obtain multiple reference fall recovery trajectories; iteratively training the initial fall recovery model through imitation learning based on the robot simulation model, the multiple robot initial parameter sequences, and the multiple reference fall recovery trajectories to obtain a reference fall recovery model; iteratively training the reference fall recovery model through reinforcement learning based on the robot simulation model, the multiple robot initial parameter sequences, and the multiple reference fall recovery trajectories to obtain the fall recovery model.

Images