Hybrid balance control method and device for double-wheel foot type robot based on reinforcement learning optimization

Abstract

The invention discloses a hybrid balance control method and device of a double-wheel foot type robot based on reinforcement learning optimization, and relates to the field of machine control, and the method comprises the steps: based on a mechanical structure of the robot, defining a coordinate system, and respectively establishing a forward kinematics model and an inverse kinematics model; simplifying the two-wheel foot type robot into a second-order inverted pendulum model, and establishing a balance controller according to the target moving distance; establishing an attitude controller based on an inverse kinematics model according to the target height and the target attitude; and training a reinforcement learning strategy, and performing coupling optimization on joints of the whole body through real-time online compensation control torque. According to the method, the balance performance of the robot can be improved, and the robustness and generalization of the controller are improved.

Description

Technical Field

[0001] The present invention relates to the field of machine control, and in particular to a hybrid balance control method and device of a two-wheeled legged robot based on reinforcement learning optimization. Background Art

[0002] The two-wheeled legged robot cleverly combines the advantages of high-speed mobility of wheels and the obstacle-crossing ability of legs, making it show excellent mobility and adaptability in complex and changing environments, and has broad application prospects in logistics, search and rescue, surveying and other fields. However, as an under-actuated, nonlinear and unstable system, the balance control of the two-wheeled legged robot has always been a difficult point in research. At the same time, good balance performance is not only a prerequisite for realizing complex movements of the robot, but also the key to its adaptation to complex terrain and completion of diversified tasks. Traditional model-based controllers have many limitations, and their performance is severely restricted by the accuracy of the model. When the robot joints have a high degree of freedom, the controller design becomes particularly complex. In view of the limitations of traditional controllers, deep reinforcement learning provides a data-driven approach. It learns the end-to-end mapping from perception to action by directly interacting with the environment, without the need for an accurate system model, and can better cope with complex and dynamic robot control tasks. However, the two-wheeled legged robot itself is unstable, which means that the direct application of standard reinforcement learning algorithms requires a large amount of data for training. The robot is easy to deviate from the expected state in the early stage of exploration, and it is difficult to find an effective control strategy. In order to solve the above problems, a new balancing control scheme is urgently needed, which can reduce the design difficulty of the controller and improve the balancing performance. Summary of the invention

[0003] In view of the above technical problems, the present invention proposes a hybrid balance control method and device for a two-wheeled legged robot based on reinforcement learning optimization, and the technical solution adopted is: According to a first aspect of the present invention, a hybrid balance control method of a two-wheeled legged robot based on reinforcement learning optimization is provided, the method comprising the following steps: S100, based on the mechanical structure of the two-wheeled legged robot, define a coordinate system, and derive its forward kinematics and inverse kinematics models in sequence to describe the basic physical characteristics of the two-wheeled legged robot; S200, simplify the two-wheeled legged robot into a second-order inverted pendulum model, establish a model-based balance controller according to the target moving distance and target moving speed, and obtain the control torque of the wheel hub motor; S300, based on the forward and inverse kinematics model, according to the target center of gravity height and the target posture, establish a model-based posture controller to obtain the control torque of the forward swing motor and the knee flexion motor; S400, the balance controller in S200 and the attitude controller in S300 are combined as the basic controller based on the model to initially ensure the stability of the system motion; S500, based on the S400 controller, trains a reinforcement learning strategy to optimize the coupling of all joint motors through real-time online compensation of control torque to improve balance performance; The described total joint motors include a wheel hub motor, a front swing motor and a knee flexion motor.

[0004] Furthermore, the coordinate system definition and the process of establishing the forward kinematics and inverse kinematics models in step S100 of the present invention meet the following conditions: First, derive the homogeneous transformation matrix from the base coordinate system {B} to each joint coordinate system ; Then derive the homogeneous transformation matrix from the source coordinate system {O} to the base coordinate system {B} , and based on this, the homogeneous transformation matrix from the control coordinate system {C} to the base coordinate system {B} is derived ; According to the continuous multiplication rule, the homogeneous transformation matrix from the control coordinate system {C} to each joint coordinate system is obtained ,With this transformation, control instructions are applied under the control system {C} to achieve precise control of each joint of the robot; In the above derivation, the base coordinate system {B} is fixed on the head of the robot's floating base, the origin of {B} coincides with the center of the inertial measurement unit, and the directions of its x, y, and z axes change with the rotation of the head, which is the starting point for connecting the coordinate systems of each joint; the origin of the control coordinate system {C} is located at the midpoint of the line connecting the two wheels, the x axis points to the robot's forward direction, the z axis is vertically upward, and the positive direction of the y axis is obtained according to the right-hand rule; the origin of the source coordinate system {O} coincides with the base coordinate system {B}, and the positive directions of its x, y, and z axes are consistent with the control coordinate system {C}, the left front swing motor coordinate system {1L}, the right front swing motor coordinate system {1R}, the left knee flexion motor coordinate system {2L}, the right knee flexion motor coordinate system {2R}, the left hub motor coordinate system {3L}, and the right hub motor coordinate system {3R} are the coordinate systems of the joint motors of the two-wheeled robot; Define the state vector , which includes the rotation angle of each joint motor , and the rolling angle of the floating base , pitch angle , and the yaw angle , based on the conversion relationship between coordinate systems, the center of gravity position of the robot relative to the control coordinate system is obtained , whose expression is ,in For the The position of the center of mass of the connecting rod in the control coordinate system {C}, Representative The mass of the connecting rod, further, derives the Jacobian matrix of the joint position from the center of gravity position: .

[0005] Furthermore, the second-order inverted pendulum model establishment and balance controller design described in step S200 of the present invention meet the following conditions:

[0006] in ,in is the wheel mass, is the mass of the floating base, is the wheel radius, is the displacement, for The first derivative of for The second derivative of represents the length of the pendulum, corresponding to the height of the robot, is the forward tilt angle of the vehicle body, for The first derivative of for The second derivative of is the wheel control torque, defining the current equilibrium state of the system as , the current equilibrium state and expected state The deviation is used as the feedback signal and multiplied by the gain matrix After that, the optimal wheel hub motor control torque input is obtained. : .

[0007] Furthermore, the posture controller design in step S300 of the present invention satisfies the following conditions:

[0008] in, represents the target center of gravity height tracking task, represents the target attitude angle task, Indicates that the wheel lateral slip is 0, represents the joint angle vector, represents the posture vector, is the corresponding Jacobian matrix. By inverse solution, the target speeds of the front swing motor and the knee flexion motor are obtained from the task objectives:

[0009] The corner mark Indicates the target value.

[0010] Furthermore, the model-based basic controller in step S400 of the present invention satisfies the following conditions: The balance controller that outputs the control torque of the hub motor and the posture controller that outputs the control torque of the front swing motor and the knee flexion motor are combined as the basic controller based on the model to initially ensure the stability of the system operation.

[0011] Furthermore, the reinforcement learning optimization strategy described in step S500 of the present invention satisfies the following conditions: The reinforcement learning optimization policy state s at time t is defined as ,in is the robot state vector, defined as , is the desired equilibrium state and desired posture vector, defined as In particular, to enhance the robustness of the strategy in unknown terrains, Inaccurate expected pitch angles intentionally removed information, is the motion error and history vector, defined as , where the error According to the expected vector Calculate the components in is the control torque and torque history vector, defined as ,in represents the control torque of the basic controller, represents the optimal compensation torque of the reinforcement learning strategy, It represents the total control torque finally applied to each joint.

[0012] According to a second aspect of the present invention, there is provided a hybrid balance control device for a two-wheeled legged robot based on reinforcement learning optimization, the device comprising: The dynamic model building module is used to define the coordinate system according to the mechanical structure of the two-wheeled foot robot and derive its forward kinematics and inverse kinematics models; The model-based balance controller module is used to simplify the two-wheeled legged robot into a second-order inverted pendulum model. According to the target moving distance and target moving speed, a model-based balance controller is established to obtain the control torque of the wheel hub motor. A model-based attitude controller module is used to establish a model-based attitude controller based on the forward and inverse kinematics models according to the target center of gravity height and target attitude, and obtain the control torque of the forward swing motor and the knee flexion motor; The basic controller module is used to combine the balance controller and the attitude controller as a model-based basic controller to initially ensure the stability of the system motion; Reinforcement learning optimization module, which trains reinforcement learning strategies, optimizes the coupling of all joint motors through real-time online compensation of control torque, and improves balance performance; The described total joint motors include a wheel hub motor, a front swing motor and a knee flexion motor.

[0013] The present invention has at least the following beneficial effects: The embodiments of the present invention provide a hybrid balance control method and device for a two-wheeled legged robot based on reinforcement learning optimization, which fully combines the advantages of traditional controllers and reinforcement learning. On the one hand, the addition of traditional controllers can stabilize the training process and improve the sample efficiency of learning; on the other hand, reinforcement learning can optimize the traditional controller online, thereby improving the control accuracy and enhancing the robustness and generalization ability of the system. The reinforcement learning strategy directly compensates the control torque. The framework has good applicability and scalability for generalized two-wheeled legged robots and different traditional controllers.

[0014] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 A flowchart of a hybrid balance control method for a two-wheeled legged robot based on reinforcement learning optimization provided by an embodiment of the present invention; Figure 2 A diagram showing the mechanical connection structure and coordinate system definition of a two-wheeled, legged robot provided in an embodiment of the present invention; Figure 3 A schematic diagram of a simplified second-order inverted pendulum structure of a two-wheeled legged robot provided in an embodiment of the present invention; Figure 4 A curve diagram of the execution state of a balancing task of a two-wheeled legged robot provided in an embodiment of the present invention; Figure 5 This is a curve diagram of the position and forward tilt angle changes of the two-wheeled legged robot under external force interference provided by the example of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the accompanying drawings of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0018] The embodiment of the present invention provides a hybrid balance control method of a two-wheeled legged robot based on reinforcement learning optimization, such as Figure 1 As shown, the method comprises the following steps: S100, based on the mechanical structure of the two-wheeled legged robot, define a coordinate system, and derive its forward kinematics and inverse kinematics models in sequence to describe the basic physical characteristics of the two-wheeled legged robot.

[0019] In an embodiment of the present invention, the structure of the two-wheeled legged robot may be an existing structure, including a base, two hip joints, left and right legs, and onboard hardware devices, etc., wherein each hip joint is provided with three joint motors, one motor is used to control the leg to swing forward and backward, one motor is used to control the leg to swing left and right, and one motor is used to control the angle of the knee joint. A hub motor and a wheel are installed at the foot end of each leg, and the hub motor is used to control the movement of the wheel. The onboard hardware devices may include an IMU and an onboard computing unit, etc.

[0020] In the present invention, Figure 2 The front view of the mechanical structure of the two-wheeled legged robot and the distribution of the coordinate system are shown. In the present invention, the optimization of the motor for left and right swing is not considered. At the same time, the specific positions of the front swing motor, the knee flexion motor and the hub motor have been marked in the figure. The coordinate system is defined as follows: the base coordinate system {B} is fixed on the head of the robot's floating base, the origin of {B} coincides with the center of the inertial measurement unit, and the directions of its x, y, and z axes change with the rotation of the head. It is the starting point for connecting the coordinate systems of each joint; the origin of the control coordinate system {C} is located at the midpoint of the line connecting the two wheels, the x-axis points to the robot's forward direction, the z-axis is vertically upward, and the positive direction of the y-axis is obtained according to the right-hand rule; the origin of the source coordinate system {O} coincides with the base coordinate system {B}, and the positive directions of its x, y, and z axes are consistent with the control coordinate system {C}. The origin of the inertial system {I} coincides with the base coordinate system {B}, and the positive directions of its x, y, and z axes are consistent with the world coordinate system. The left front swing motor coordinate system {1L}, the right front swing motor coordinate system {1R}, the left knee flexion motor coordinate system {2L}, the right knee flexion motor coordinate system {2R}, the left wheel hub motor coordinate system {3L}, and the right wheel hub motor coordinate system {3R} are the joint motor coordinate systems of the two-wheeled foot robot. The rotation angles of the joint motors are .

[0021] In the present invention, the derivation process of forward kinematics is as follows: First, the homogeneous transformation matrix from the base coordinate system {B} to each joint coordinate system is derived: ; Then derive the homogeneous transformation matrix from the source coordinate system {O} to the base coordinate system {B} , and based on this, the homogeneous transformation matrix from the control coordinate system {C} to the base coordinate system {B} is derived ;According to the continuous multiplication rule, the homogeneous transformation matrix from the control coordinate system {C} to each joint coordinate system is obtained By means of this transformation, control instructions can be applied under the control system {C} to achieve precise control of each joint of the robot.

[0022] In the present invention, the robot's center of gravity position is derived as follows: Define the state vector , including the rotation angle of each joint motor and the roll angle of the floating base , pitch angle , and the yaw angle Based on the conversion relationship between coordinate systems, the center of gravity position of the robot relative to the control coordinate system can be obtained , whose expression is ,in For the The position of the center of mass of the connecting rod in the control coordinate system {C}, Representative The mass of the connecting rod. Further, the Jacobian matrix of the joint position can be derived from the center of gravity position: .

[0023] S200, the two-wheeled legged robot is simplified into a second-order inverted pendulum model, and a model-based balance controller is established according to the target moving distance and target moving speed to obtain the control torque of the hub motor.

[0024] In the present invention, the structure of the two-wheeled foot robot is simplified to a second-order inverted pendulum. Figure 3 As shown. According to the Lagrangian dynamics equation, the second-order dynamics model of the system is established as:

[0025] in . is the wheel mass, is the mass of the floating base, is the wheel radius, is the displacement, for The first derivative of for The second derivative of represents the length of the pendulum, corresponding to the height of the robot. is the forward tilt angle of the vehicle body, for The first derivative of for The second derivative of It is the wheel control torque.

[0026] In the present invention, a linear quadratic regulator (LQR) is selected as the basic balancing controller. It is an optimized linear feedback controller that minimizes a quadratic performance index so that the system can balance the system state and the cost of the control input while achieving the desired state. Specifically, by adjusting the weight matrices Q and R, solving the Riccati equation to obtain an optimal feedback gain matrix The system equilibrium state vector is defined as: , the current equilibrium state and expected state The deviation is used as the feedback signal and multiplied by the gain matrix After that, the optimal wheel hub motor control torque input is obtained. : At the same time, the yaw angle γ adjustment of the vehicle body is integrated into the LQR. Optimal parameters The value of needs to be adjusted specifically for different systems.

[0027] S300, based on the forward and inverse kinematics model, according to the target center of gravity height and the target posture, a model-based posture controller is established to obtain the control torque of the forward swing motor and the knee flexion motor.

[0028] In the example of the present invention, in order to achieve dynamic control of the floating base posture of the robot, three specific control objectives are set: Denotes the centroid height tracking task, with The base roll and pitch adjustment tasks are represented by It indicates the task of ensuring that the wheels do not slip sideways. Definition represents the joint angle vector, Represents the attitude vector, and organizes the control task into the following expression:

[0029] By inverse solution, the target speeds of the forward swing motor and the knee flexion motor can be obtained from the task objectives:

[0030] in, is the corresponding Jacobian matrix, with the subscript represents the target value, and + represents the inverse of the matrix.

[0031] In the example of the present invention, a Proportional Differential (PD) controller is selected to realize the dynamic control of the floating base posture of the robot. The PD controller measures the deviation between the actual output of the system and the set value, that is, the current error e(t) of the system and its rate of change de(t) / dt, and calculates the control effect of the proportional term and the differential term according to the proportional gain Kp and the differential gain Kd, and adds the two to obtain the final control torque, which acts on the system to reduce the error and make the system output track the set value. In the present invention, different PD control parameters are used for the front swing motor and the knee flexion motor to obtain the corresponding control torque. Kp and Kd are adjustable parameters.

[0032] S400 combines the balance controller in S200 with the attitude controller in S300 as a model-based basic controller to initially ensure the stability of the system motion.

[0033] In the example of the present invention, the balance controller that outputs the control torque of the wheel hub motor is integrated with the posture controller that outputs the control torque of the front swing motor and the knee flexion motor to obtain a model-based basic controller. This controller can initially ensure the stability of the system movement and provide a stable initial state for subsequent reinforcement learning training.

[0034] S500, based on the S400 controller, trains reinforcement learning strategies to optimize the coupling of all joint motors (including wheel hub motors, front swing motors and knee flexion motors) through real-time online compensation of control torque to improve balance performance.

[0035] In the present invention, the state space of the reinforcement learning optimization strategy is defined as follows: the state s at time t is defined as .in is the robot state vector, defined as . is the desired equilibrium state and desired posture vector, defined as In particular, to enhance the robustness of the strategy in unknown terrains, Inaccurate expected pitch angles intentionally removed Information. It aims to inspire the agent to learn from other state information so that it can make accurate decisions even when facing terrain with unknown angles. is the motion error and history vector, defined as , where the error According to the expected vector Calculate the components in . is the control torque and torque history vector, defined as ,in represents the control torque of the basic controller, represents the optimal compensation torque of the reinforcement learning strategy, It represents the total control torque finally applied to each joint.

[0036] In the present invention, the reinforcement learning optimization strategy directly outputs the 6-dimensional compensation torque , in the control torque Based on this, the coupling optimization is performed on the 6 motors in the whole body. The total control torque applied to the motor is . The Soft Actor-Critic (SAC) algorithm is used to train the policy network, which is a reinforcement learning method based on random strategies. By introducing the concept of entropy, SAC encourages the agent to explore more state-action pairs, thereby avoiding falling into local optimal solutions. The random exploration strategy can effectively balance exploration and utilization, improve learning efficiency, and ultimately obtain a better strategy. At the same time, the reinforcement learning optimization strategy in the present invention interacts with the basic controller only through torque information, and the design of the state space and action space is independent of the specific structure inside the basic controller. This modular design enables the method to be flexibly integrated into other model-based control frameworks and has good versatility.

[0037] In the example of the present invention, the reward function is designed as : Part 1 The first part represents the balance reward and posture reward, which optimizes the terminal task performance; the second part represents the joint restriction reward, which ensures the robot's motion safety. In the design of this reward function, only the maximum restriction is taken for the state of each joint, and the joint state is not used to guide task optimization. By measuring the terminal status and the target The deviation is used to guide the optimization performance, expressed as Item 1 Measuring the state of equilibrium performance, encouraging the robot to reduce tracking errors and encouraging the robot's floating base to be as stable as possible; the second Encourage the robot to maintain the desired center of gravity height ; Item 3 Measuring the robot’s posture performance . The order of all error terms measured in corresponds to the error vector Each item in the function is used To quantify. The weights of each item in need to be adjusted specifically for the task. In particular, in order to make the strategy better adapt to unknown terrain, the present invention does not set a specific expected value for the vehicle body forward tilt angle, but enhances the stability of the system by limiting its change speed. Specifically, it is hoped that the change of the forward tilt angle tends to be gentle to avoid system instability caused by drastic changes.

[0038] In the examples of the present invention, compared with the restrictions on the movement of a single joint, the present invention adopts a more global reward function design, that is, directly evaluating the performance of the robot based on the completion of the terminal task. The strategy update is guided according to the actual balance performance and posture performance of the robot, so that it can make decisions from the perspective of the overall task, rather than just focusing on a certain local action or joint motion optimization. This design gives the robot greater freedom of exploration and can prompt it to learn better action strategies that comprehensively consider global goals. In addition, the reward design based on the terminal task can also avoid the local optimal problem that may exist in the traditional method, and improve the robot's adaptability in diverse and dynamic environments.

[0039] In the example of the present invention, in order to improve the robustness of the model to environmental uncertainty and model parameter changes, a threshold randomization strategy is introduced in the training process, including randomization of physical parameters, randomization of sensor data, and randomization of environmental parameters.

[0040] In the example of the present invention, the policy network adopts a two-layer fully connected structure, with 1024 and 512 neurons respectively, and the activation function is tanh. The network input is a 15-dimensional state vector, and the output is a 6-dimensional action vector. The SAC algorithm is used for training, and the maximum number of iterations is set to 50,000 steps. The policy network is updated every 1,000 steps, and the batch size is 512. The learning rate is 1e-3, the discount factor is 0.99, the entropy regularization coefficient is initialized to 0.1, and the reward scaling factor is 10. The algorithm is trained on a computer equipped with a 12-core CPU and an NVIDIA 3060 GPU. The specific structure of the network and the training hyperparameters need to be specifically adjusted.

[0041] In the example of the present invention, two sets of training tasks are set to verify the effectiveness of the method. Group 1: The height remains unchanged, and the moving targets between 1 and 4 meters are randomly set, and three slopes with different slopes of 0, 5 and 10 degrees are considered. Group 2: The moving targets between 1 and 4 meters on the flat ground are randomly set, and the center of gravity height change targets between 0.05 and 0.15 meters are randomly generated. The specific parameters in the task can be adjusted.

[0042] The embodiment of the present invention provides a hybrid balance control method for a two-wheeled legged robot based on reinforcement learning optimization. In an actual application scenario, the two-wheeled legged robot moves straight on flat ground while maintaining the center of gravity height and posture stability. The motion curve is shown in the figure Figure 4 As shown. Figure 4It can be seen that the robot can balance various tasks and achieve the goal stably. The comparison method is before reinforcement learning optimization, which only has the effect of basic controller control. On the basis of maintaining the stability of the forward tilt angle and attitude angle, the tracking accuracy of the method of the present invention in position and height is significantly improved. In particular, the position of the method of the present invention almost coincides with the expected trajectory. In addition, tests were conducted to apply instantaneous external forces of 10N, 20N, and 30N at 12s, 20s, and 32s respectively. The test results are as follows Figure 5 As shown in the figure, whether in motion or at rest, the proposed method can quickly recover to the target trajectory after being disturbed by the reverse external force, and effectively suppress the oscillation of the pitch angle. The comparison method shows obvious oscillation in both the pitch angle and position, and it is difficult to restore stability in a short time.

[0043] The embodiment of the present invention provides a hybrid balance control method for a two-wheeled legged robot based on reinforcement learning optimization, which fully combines the advantages of traditional controllers and reinforcement learning. The addition of traditional controllers can stabilize the training process and improve the sample efficiency of learning. At the same time, reinforcement learning can optimize the traditional controller online, thereby improving the control accuracy and enhancing the robustness and generalization ability of the system. The reinforcement learning strategy directly compensates the control torque. The framework has good applicability and scalability for generalized two-wheeled legged robots and different traditional controllers.

[0044] Based on the same inventive concept, an embodiment of the present invention further provides a hybrid balance control device for a two-wheeled legged robot based on reinforcement learning optimization, the device comprising: The dynamic model building module is used to define the coordinate system according to the mechanical structure of the two-wheeled foot robot and derive its forward kinematics and inverse kinematics models; The model-based balance controller module is used to simplify the two-wheeled legged robot into a second-order inverted pendulum model. According to the target moving distance and target moving speed, a model-based balance controller is established to obtain the control torque of the wheel hub motor. A model-based attitude controller module is used to establish a model-based attitude controller based on the forward and inverse kinematics models according to the target center of gravity height and target attitude, and obtain the control torque of the forward swing motor and the knee flexion motor; The basic controller module is used to combine the balance controller and the attitude controller as a model-based basic controller to initially ensure the stability of the system motion; The reinforcement learning optimization module trains reinforcement learning strategies, optimizes the coupling of all joint motors (including wheel hub motors, front swing motors and knee flexion motors) through real-time online compensation of control torque, and improves balance performance.

[0045] The device can be used to perform Figure 1Therefore, for the functions that can be realized by each functional module of the device, please refer to Figure 1 The description of the illustrated embodiment is not repeated in detail.

[0046] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and the present invention is not limited here.

[0047] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A hybrid balance control method for a two-wheeled legged robot based on reinforcement learning optimization, characterized in that: The method comprises the following steps: S100, based on the mechanical structure of the two-wheeled legged robot, define a coordinate system, and derive its forward kinematics and inverse kinematics models in sequence to describe the basic physical characteristics of the two-wheeled legged robot; S200, simplify the two-wheeled legged robot into a second-order inverted pendulum model, establish a model-based balance controller according to the target moving distance and target moving speed, and obtain the control torque of the wheel hub motor; S300, based on the forward and inverse kinematics model, according to the target center of gravity height and the target posture, establish a model-based posture controller to obtain the control torque of the forward swing motor and the knee flexion motor; S400 combines the balance controller in S200 with the attitude controller in S300 as a model-based basic controller to initially ensure the stability of the system motion; S500, based on the S400 controller, trains a reinforcement learning strategy to optimize the coupling of all joint motors through real-time online compensation of control torque to improve balance performance; The described total joint motors include a wheel hub motor, a front swing motor and a knee flexion motor.

2. The method according to claim 1, characterized in that The coordinate system definition and the process of establishing the forward kinematics and inverse kinematics models in step S100 meet the following conditions: First, derive the homogeneous transformation matrix from the base coordinate system {B} to each joint coordinate system ; Then derive the homogeneous transformation matrix from the source coordinate system {O} to the base coordinate system {B} , and based on this, the homogeneous transformation matrix from the control coordinate system {C} to the base coordinate system {B} is derived ; According to the continuous multiplication rule, the homogeneous transformation matrix from the control coordinate system {C} to each joint coordinate system is obtained ,With this transformation, control instructions are applied under the control system {C} to achieve precise control of each joint of the robot; In the above derivation, the base coordinate system {B} is fixed on the head of the robot's floating base, the origin of {B} coincides with the center of the inertial measurement unit, and the directions of its x, y, and z axes change with the rotation of the head, which is the starting point for connecting the coordinate systems of each joint; the origin of the control coordinate system {C} is located at the midpoint of the line connecting the two wheels, the x axis points to the robot's forward direction, the z axis is vertically upward, and the positive direction of the y axis is obtained according to the right-hand rule; the origin of the source coordinate system {O} coincides with the base coordinate system {B}, and the positive directions of its x, y, and z axes are consistent with the control coordinate system {C}, the left front swing motor coordinate system {1L}, the right front swing motor coordinate system {1R}, the left knee flexion motor coordinate system {2L}, the right knee flexion motor coordinate system {2R}, the left hub motor coordinate system {3L}, and the right hub motor coordinate system {3R} are the coordinate systems of the joint motors of the two-wheeled robot; Define the state vector , which includes the rotation angle of each joint motor , and the rolling angle of the floating base , pitch angle , and the yaw angle , based on the conversion relationship between coordinate systems, the center of gravity position of the robot relative to the control coordinate system is obtained , whose expression is ,in For the The position of the center of mass of the connecting rod in the control coordinate system {C}, Representative The mass of the connecting rod, further, derives the Jacobian matrix of the joint position from the center of gravity position: 。 3. The method according to claim 1, characterized in that The second-order inverted pendulum model establishment and balance controller design described in step S200 meet the following conditions: in ,in is the wheel mass, is the mass of the floating base, is the wheel radius, is the displacement, for The first derivative of for The second derivative of represents the length of the pendulum, corresponding to the height of the robot, is the forward tilt angle of the vehicle body, for The first derivative of for The second derivative of is the wheel control torque, defining the current equilibrium state of the system as , the current equilibrium state and expected state The deviation is used as the feedback signal and multiplied by the gain matrix After that, the optimal wheel hub motor control torque input is obtained. : .

4. The method according to claim 1, characterized in that: The posture controller design in step S300 satisfies the following conditions: in, represents the target center of gravity height tracking task, represents the target attitude angle task, Indicates that the wheel lateral slip is 0, represents the joint angle vector, represents the posture vector, is the corresponding Jacobian matrix. By inverse solution, the target speeds of the front swing motor and the knee flexion motor are obtained from the task objectives: The corner mark Indicates the target value.

5. The method according to claim 1, characterized in that The model-based basic controller in step S400 satisfies the following conditions: The balance controller that outputs the control torque of the hub motor and the posture controller that outputs the control torque of the front swing motor and the knee flexion motor are combined as the basic controller based on the model to initially ensure the stability of the system operation.

6. The method according to claim 1, characterized in that The reinforcement learning optimization strategy described in step S500 satisfies the following conditions: The reinforcement learning optimization policy state s at time t is defined as ,in is the robot state vector, defined as , is the desired equilibrium state and desired posture vector, defined as In order to enhance the robustness of the strategy in unknown terrain, Inaccurate expected pitch angles intentionally removed information, is the motion error and history vector, defined as , where the error According to the expected vector Calculate the components in is the control torque and torque history vector, defined as ,in represents the control torque of the basic controller, represents the optimal compensation torque of the reinforcement learning strategy, It represents the total control torque finally applied to each joint.

7. The method according to claim 1, characterized in that The reinforcement learning optimization strategy described in step S500 satisfies the following conditions: Reinforcement learning optimization strategy directly outputs 6-dimensional compensation torque , in the control torque Based on this, the coupling optimization is performed on the 6 motors in the whole body, and the total control torque applied to the motor is .

8. The method according to claim 1, characterized in that The reinforcement learning optimization strategy described in step S500 satisfies the following conditions: The reward function is designed as : Part 1 It represents the balance reward and posture reward, and optimizes the terminal task performance; The second part represents the joint restriction reward, which ensures the robot's motion safety. In the design of this reward function, only the maximum restriction is taken for the state of each joint, and the joint state is not used to guide task optimization; By measuring the terminal status and the target The deviation is used to guide the optimization performance, expressed as , the first item Measuring the state of equilibrium performance, encouraging the robot to reduce tracking errors and encouraging the robot's floating base to be as stable as possible; the second Encourage the robot to maintain the desired center of gravity height ; Item 3 Measuring the robot’s posture performance , The order of all error terms measured in corresponds to the error vector Each item in the function is used To quantify.

9. A hybrid balance control device for a two-wheeled legged robot based on reinforcement learning optimization that implements any one of the methods of claims 1-8, characterized in that: The device comprises: The dynamic model building module is used to define the coordinate system according to the mechanical structure of the two-wheeled foot robot and derive its forward kinematics and inverse kinematics models; The model-based balance controller module is used to simplify the two-wheeled legged robot into a second-order inverted pendulum model. According to the target moving distance and target moving speed, a model-based balance controller is established to obtain the control torque of the wheel hub motor. A model-based attitude controller module is used to establish a model-based attitude controller based on the forward and inverse kinematics models according to the target center of gravity height and target attitude, and obtain the control torque of the forward swing motor and the knee flexion motor; The basic controller module is used to combine the balance controller and the attitude controller as a model-based basic controller to initially ensure the stability of the system motion; Reinforcement learning optimization module, which trains reinforcement learning strategies, optimizes the coupling of all joint motors through real-time online compensation of control torque, and improves balance performance; The described total joint motors include a wheel hub motor, a front swing motor and a knee flexion motor.

Images