Hierarchical learning framework for whole-body model predictive control of humanoid robot

Abstract

The invention discloses a hierarchical learning framework for whole-body model prediction control of a humanoid robot. The hierarchical learning framework comprises the steps of defining an initial state and a task target of the robot, initializing a control strategy of each layer to construct an initial kinematics model, collecting a motion data set, and identifying model parameters from data. Afterwards, through reinforcement learning based on the model, the dynamic model is continuously updated and optimized so as to reduce the difference between the model and an actual system; finally, a whole-body model prediction control method is used for calculating a control strategy in real time through a dynamic model optimized through reinforcement learning, a hierarchical learning framework is innovatively developed, and it is ensured that the system can achieve real-time control under the condition that the central control system is updated slowly through the multi-layer design; the efficient real-time response capability is still kept, the flexibility and the dynamic performance of the robot are remarkably improved, and the humanoid robot can generate multi-contact and dynamic behaviors even if the low-frequency strategy of whole body model prediction control is updated.

Description

Technical Field

[0001] The present invention belongs to the field of robot control, and in particular relates to a hierarchical learning framework for full-body model predictive control of a humanoid robot. Background Art

[0002] Humanoid robots are robots that mimic humans in appearance and behavior. They are usually composed of a head, torso, limbs, etc., and are usually designed to mimic human movement, perception, and cognitive abilities. Humanoid robots are not only as close to humans as possible in appearance, but also mimic human abilities in behavior and task execution, enabling them to interact with humans in complex and changing environments, and even replace or assist humans in completing specific tasks. With the development of science and technology, humanoid robots are increasingly being used in scientific research, medical care, public services, and other fields, bringing great convenience to people's lives.

[0003] Model predictive control is an optimization-based control method that uses the system's dynamic model to predict the state in the future and optimizes the control input in real time based on these predictions to achieve the optimal control effect. It can handle complex constraints and has flexibility and real-time performance, so it is widely used in industrial process control, autonomous driving, robot control and other fields.

[0004] Model predictive control is also an active research direction in the field of humanoid robots. Among them, a model predictive control method that considers the whole-body dynamics of the robot is called whole-body model predictive control. It achieves comprehensive control of the robot's whole-body movements by modeling the kinematics and dynamics of the whole body of the humanoid robot and generates a variety of movements. Compared with traditional local control methods, the whole-body model predictive control method can better maintain balance, optimize motion paths, and respond to external disturbances, but its optimization calculation takes a long time, which poses a challenge to the motion generation of humanoid robots with high real-time requirements. Therefore, it is necessary to propose a method that allows humanoid robots to generate multi-contact and dynamic behaviors even under the low-frequency strategy update of whole-body model predictive control. Summary of the invention

[0005] The technical problem to be solved by the present invention is: in view of the common shortcomings of existing whole-body model predictive control methods, such as long calculation time and slow strategy update, a hierarchical learning framework is proposed. The hierarchical learning framework is a three-layer hierarchical learning framework. In the upper layer, a precise dynamic model is learned and the control strategy is calculated through a model-based reinforcement learning algorithm and stochastic whole-body model predictive control; additional strategies are learned in the middle and lower layers to generate high-frequency control inputs.

[0006] To solve the above technical problems, the technical solution adopted by the present invention is: a hierarchical learning framework for full-body model predictive control of a humanoid robot, comprising the following steps:

[0007] Step S1: initialization;

[0008] Define the robot's initial state and mission objectives, and initialize each layer of control strategy.

[0009] Step S2: upper layer calculation;

[0010] Construct an initial kinematic model, collect motion data sets, and identify model parameters from the data. Then, through model-based reinforcement learning, continuously update and optimize the dynamic model to reduce the difference between the model and the actual system. Finally, use the whole-body model predictive control method to use the dynamic model optimized by reinforcement learning to calculate the control strategy in real time, send the control strategy to the underlying controller, generate and collect new motion data, add the collected data to the motion data set, and relearn and optimize the dynamic model. Repeat this process until the maximum number of iterations is reached.

[0011] The control strategy of the upper layer output can be expressed by the following equation:

[0012]

[0013] The constraints are:

[0014] x k+1 =f(x k ,u k )+σF(x k ,u k ) k ;

[0015] g(x k ,u k )≤0,k=t,...t+N-2;

[0016] The state of the upper layer output can be expressed by the following equation:

[0017] x k+1 =f(x k ,u k )+σF(x k ,u k ) k ;

[0018] in:

[0019] is the control strategy output by the top level,

[0020] To optimize the operation of the target, it means to find the expected value of the objective function E[J(X k ,U k )]The minimum control input sequence U k ,

[0021] E[J(X k ,U k )] is the expected value of the objective function, J(X k ,U k ) is the objective function representing the transition from state X k and control input U k The starting performance indicators (such as energy consumption, time, etc.), the expected value E[·] is used to deal with the randomness and uncertainty of the system.

[0022] x k is the current state of the system;

[0023] u k is the current control input of the system;

[0024] f(·) is the deterministic dynamic model of the system, which is used to describe the state evolution law of the system under the current state and control input, and determines the main part of the state at the next moment.

[0025] F(·) is an uncertain mapping function, which is used to describe the influence of random noise on the state change of the system under the current state and control input.

[0026] σ is the noise amplitude;

[0027] w k is the system random noise;

[0028] g(·) is a constraint function, which represents the constraints that the current state of the system and the control input must satisfy.

[0029] Step S3: middle-level calculation;

[0030] The middle layer of the framework iteratively learns by imitating the upper layer control strategy, and gradually approaches the upper layer input and state by adjusting the middle layer input and state until the middle layer control strategy completely imitates the upper layer control strategy. At the same time, during the learning process, the learning rate and forgetting rate are used to adjust the update pace. Finally, the middle layer control strategy is passed to the low-level controller to calculate the next control input.

[0031] The control strategy of the middle-level output can be expressed by the following equation:

[0032]

[0033] The state of the middle layer output is expressed by the following equation:

[0034]

[0035] in:

[0036] i is the number of iterations;

[0037] Control strategy for upper layer output;

[0038] Control strategy for middle-level output;

[0039] is the current state of the upper layer;

[0040] is the current status of the middle layer;

[0041] is the initial state of the middle layer;

[0042] α is the learning rate;

[0043] μ is the forgetting rate.

[0044] Step S4: lower layer calculation;

[0045] The lower layer of the framework is used to generate the original motion pattern, including dynamic motion primitives and stretch reflex mechanisms. In this method, the original motion pattern is represented by a time series, including the position and direction of the base rod, the linear velocity and angular velocity of the base rod, the joint angle and the joint angular velocity. The base rod position and joint angle of each degree of freedom are learned as dynamic motion primitive trajectories, while the base rod direction is learned as a dynamic motion primitive trajectory based on quaternions. At the same time, when the robot cannot generate motion, the expected position and direction will be iteratively adjusted according to the state. In addition, a stretch reflex mechanism is designed. When the joint angle approaches the limit, the reflex is activated and moves the angle in the opposite direction. Finally, the lower-level control strategy combines the dynamic motion primitive trajectory and the stretch reflex input and sends it to the low-level controller for the calculation of the control input.

[0046] The control strategy of the lower layer output can be expressed by the following equation:

[0047]

[0048] in:

[0049] Control strategy for lower-level output;

[0050] is the trajectory of dynamic motion primitives;

[0051] Input for stretch reflection.

[0052] Step S5: integrating applications;

[0053] The bottom-level controller integrates the control strategies of the upper, middle and lower layers and converts them into specific control instructions to control the humanoid robot to generate corresponding actions.

[0054] The formula is:

[0055]

[0056] in:

[0057] x k is the current state of the system;

[0058] u k is the current control input of the system;

[0059] Control strategy for upper layer output;

[0060] Control strategy for middle-level output;

[0061] Control strategy for lower-level output;

[0062] is the current state of the upper layer;

[0063] is the current status of the middle layer;

[0064] For upper layer gain;

[0065] K k 2 For mid-level gain.

[0066] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0067] The present invention innovatively develops a hierarchical learning framework, which ensures through a multi-level design that the system can maintain efficient real-time response capabilities even when the central control system is updated relatively slowly, significantly improving the flexibility and dynamic performance of the robot, and enabling the humanoid robot to generate multi-contact, dynamic behaviors even under the low-frequency strategy update of the whole-body model predictive control.

[0068] The present invention innovatively combines model-based reinforcement learning and whole-body model predictive control to achieve long-term optimization of the system and real-time control strategy generation, fully utilizing the complementarity of long-term learning and short-term feedback to improve the adaptability, stability and control accuracy of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0070] Figure 1 It is a flowchart of the hierarchical learning framework;

[0071] Figure 2 This is the schematic diagram of the robot timing control. DETAILED DESCRIPTION

[0072] The technical solution of the present invention will be described clearly and completely in conjunction with the embodiments below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The embodiments of the present invention and all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] Example 1

[0074] See also Figure 1 As shown, the present invention provides a hierarchical learning framework for full-body model predictive control of a humanoid robot, comprising the following steps:

[0075] Step 1: Define the robot's initial state and mission objectives, and initialize each layer of control strategy.

[0076] Step 2: Build an initial kinematic model, collect motion data sets, and identify model parameters from the data. After that, the dynamic model is continuously updated and optimized through model-based reinforcement learning to reduce the difference between the model and the actual system. Finally, the whole-body model predictive control method is used to calculate the control strategy in real time using the dynamic model optimized by reinforcement learning, and the control strategy is sent to the underlying controller to generate and collect new motion data. The collected data is added to the motion data set to relearn and optimize the dynamic model. Repeat this process until the maximum number of iterations is reached.

[0077] The control strategy of the upper layer output can be expressed by the following equation:

[0078]

[0079] The constraints are:

[0080] x k+1 =f(x k ,u k )+σF(x k ,u k ) k ;

[0081] g(x k ,u k )≤0,k=t,...t+N-2;

[0082] The state of the upper layer output can be expressed by the following equation:

[0083] x k+1 =f(x k ,u k )+σF(x k ,u k ) k ;

[0084] in:

[0085] is the control strategy output by the top level,

[0086] is the control strategy output by the top level,

[0087] To optimize the operation of the target, it means to find the expected value of the objective function E[J(X k ,U k )]The minimum control input sequence U k ,

[0088] E[J(X k ,U k )] is the expected value of the objective function, J(X k ,U k ) is the objective function representing the transition from state X k and control input U k The starting performance indicators (such as energy consumption, time, etc.), the expected value E[·] is used to deal with the randomness and uncertainty of the system.

[0089] x k is the current state of the system;

[0090] u k is the current control input of the system;

[0091] f(·) is the deterministic dynamic model of the system, which is used to describe the state evolution law of the system under the current state and control input, and determines the main part of the state at the next moment.

[0092] F(·) is an uncertain mapping function, which is used to describe the influence of random noise on the state change of the system under the current state and control input.

[0093] σ is the noise amplitude;

[0094] w k is the system random noise;

[0095] g(·) is a constraint function, which represents the constraints that the current state of the system and the control input must satisfy.

[0096] Step 3: The middle layer of the framework iteratively learns by imitating the upper layer control strategy, and gradually approaches the upper layer input and state by adjusting the middle layer input and state until the middle layer control strategy completely imitates the upper layer control strategy. At the same time, during the learning process, the learning rate and forgetting rate are used to adjust the update pace. Finally, the middle layer control strategy is passed to the low-level controller to calculate the next control input.

[0097] The control strategy of the middle-level output can be expressed by the following equation:

[0098]

[0099] The state of the middle layer output is expressed by the following equation:

[0100]

[0101] in:

[0102] i is the number of iterations;

[0103] Control strategy for upper layer output;

[0104] Control strategy for middle-level output;

[0105] is the current state of the upper layer;

[0106] is the current status of the middle layer;

[0107] is the initial state of the middle layer;

[0108] α is the learning rate;

[0109] μ is the forgetting rate.

[0110] Step 4: The lower layer of the framework is used to generate the original motion pattern, including dynamic motion primitives and stretch reflex mechanisms. In this method, the original motion pattern is represented by a time series, including the position and direction of the base rod, the linear velocity and angular velocity of the base rod, the joint angle, and the joint angular velocity. The base rod position and joint angle of each degree of freedom are learned as dynamic motion primitive trajectories, while the base rod direction is learned as a dynamic motion primitive trajectory based on quaternions. At the same time, when the robot cannot generate motion, the expected position and direction will be iteratively adjusted according to the state. In addition, a stretch reflex mechanism is designed. When the joint angle approaches the limit, the reflex is activated and moves the angle in the opposite direction. Finally, the lower-level control strategy combines the dynamic motion primitive trajectory and the stretch reflex input and sends it to the low-level controller for calculation of the control input.

[0111] The control strategy of the lower layer output can be expressed by the following equation:

[0112]

[0113] in:

[0114] Control strategy for lower-level output;

[0115] is the trajectory of dynamic motion primitives;

[0116] Input for stretch reflection.

[0117] Step 5: The bottom-level controller integrates the control strategies of the upper, middle and lower layers and converts them into specific control instructions to control the humanoid robot to generate corresponding actions.

[0118] The formula is:

[0119]

[0120] in:

[0121] x k is the current state of the system;

[0122] u k is the current control input of the system;

[0123] Control strategy for upper layer output;

[0124] Control strategy for middle-level output;

[0125] Control strategy for lower-level output;

[0126] is the current state of the upper layer;

[0127] is the current status of the middle layer;

[0128] For upper layer gain;

[0129] For mid-level gain.

[0130] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A hierarchical learning framework for full-body model predictive control of humanoid robots, characterized by: The following steps are involved: Step S1: initialization; Define the robot's initial state and mission objectives, and initialize each layer of control strategy. Step S2: upper layer calculation; Construct an initial kinematic model, collect motion data sets, and identify model parameters from the data. Then, through model-based reinforcement learning, continuously update and optimize the dynamic model to reduce the difference between the model and the actual system. Finally, use the whole-body model predictive control method to use the dynamic model optimized by reinforcement learning to calculate the control strategy in real time, send the control strategy to the underlying controller, generate and collect new motion data, add the collected data to the motion data set, and relearn and optimize the dynamic model. Repeat this process until the maximum number of iterations is reached. The control strategy of the upper layer output can be expressed by the following equation: The constraints are: x k+1 =f(x k ,u k )+σF(x k ,u k )w k ; g(x k ,u k )≤0,k=t,...t+N-2; The state of the upper layer output can be expressed by the following equation: x k+1 =f(x k ,u k )+σF(x k ,u k )w k ; in: is the control strategy output by the top level, To optimize the operation of the target, it means to find the expected value of the objective function E[J(X k ,U k )]The minimum control input sequence U k , E[J(X k ,U k )] is the expected value of the objective function, J(X k ,U k ) is the objective function representing the transition from state X k and control input U k The starting performance indicators (such as energy consumption, time, etc.), the expected value E[·] is used to deal with the randomness and uncertainty of the system. x k is the current state of the system; u k is the current control input of the system; f(·) is the deterministic dynamic model of the system, which is used to describe the state evolution law of the system under the current state and control input, and determines the main part of the state at the next moment. F(·) is an uncertain mapping function, which is used to describe the influence of random noise on the state change of the system under the current state and control input. σ is the noise amplitude; w k is the system random noise; g(·) is a constraint function, which represents the constraints that the current state of the system and the control input must satisfy. Step S3: middle-level calculation; The middle layer of the framework iteratively learns by imitating the upper layer control strategy, and gradually approaches the upper layer input and state by adjusting the middle layer input and state until the middle layer control strategy completely imitates the upper layer control strategy. At the same time, during the learning process, the learning rate and forgetting rate are used to adjust the update pace. Finally, the middle layer control strategy is passed to the low-level controller to calculate the next control input. The control strategy of the middle-level output can be expressed by the following equation: The state of the middle layer output is expressed by the following equation: in: i is the number of iterations; Control strategy for upper layer output; Control strategy for middle-level output; is the current state of the upper layer; is the current status of the middle layer; is the initial state of the middle layer; α is the learning rate; μ is the forgetting rate. Step S4: lower layer calculation; The lower layer of the framework is used to generate the original motion pattern, including dynamic motion primitives and stretch reflex mechanisms. In this method, the original motion pattern is represented by a time series, including the position and direction of the base rod, the linear velocity and angular velocity of the base rod, the joint angle and the joint angular velocity. The base rod position and joint angle of each degree of freedom are learned as dynamic motion primitive trajectories, while the base rod direction is learned as a dynamic motion primitive trajectory based on quaternions. At the same time, when the robot cannot generate motion, the expected position and direction will be iteratively adjusted according to the state. In addition, a stretch reflex mechanism is designed. When the joint angle approaches the limit, the reflex is activated and moves the angle in the opposite direction. Finally, the lower-level control strategy combines the dynamic motion primitive trajectory and the stretch reflex input and sends it to the low-level controller for the calculation of the control input. The control strategy of the lower layer output can be expressed by the following equation: in: Control strategy for lower-level output; is the trajectory of dynamic motion primitives; Input for stretch reflection. Step S5: integrating applications; The bottom-level controller integrates the control strategies of the upper, middle and lower layers and converts them into specific control instructions to control the humanoid robot to generate corresponding actions. The formula is: in: x k is the current state of the system; u k is the current control input of the system; Control strategy for upper layer output; Control strategy for middle-level output; Control strategy for lower-level output; is the current state of the upper layer; is the current status of the middle layer; For upper layer gain; For mid-level gain.

Images