Biped robot motion control method, system and electronic device

Abstract

The application discloses a biped robot motion control method, system and electronic equipment. The method comprises the following steps: obtaining a strategy network through auxiliary reward training; the strategy network generates a control signal according to a received control instruction and a current state of the biped robot; a control component controls the motion of the biped robot based on the control signal; wherein the auxiliary reward is used to feed back the training action of the biped robot and optimize the strategy network in the training; the strategy network obtains the current state based on the control instruction; and the control component drives the rotation of the joint of the biped robot to make the biped robot tend to the state required by the control instruction. The control method of the biped robot provided by the application controls the rotation of the joint of the biped robot through the pre-trained strategy network of the auxiliary reward, thereby improving the fluency of the motion of the biped robot and improving the consistency of the trunk and limbs of the biped robot.

Description

Technical Field

[0001] This application relates to the field of control technology for bipedal robots, and in particular to a method, system and electronic device for motion control of bipedal robots. Background Technology

[0002] Bipedal robots are upright-legged robots designed to mimic the human form. Unlike wheeled robots, the whole-body dynamics models of bipedal robots are typically high-dimensional and nonlinear, making data optimization difficult unless the models are simplified. However, motion control based on simplified models is heavily dependent on the environment, requiring continuous optimization based on the environment, which poses a significant challenge to dynamically changing application scenarios. Furthermore, the movements of bipedal robots trained in existing technologies are not smoothly connected, and the consistency between the torso and limbs is poor. Summary of the Invention

[0003] This application provides a method, system, and electronic device for controlling the motion of a bipedal robot, in order to solve the technical problems of stiff movements and poor consistency between the torso and limbs in existing bipedal robots.

[0004] To address the aforementioned technical problems, embodiments of this application disclose the following technical solutions: Firstly, a motion control method for a bipedal robot is provided, the method comprising: The policy network is obtained through auxiliary reward training; The strategy network generates control signals based on the received control commands and the current state of the bipedal robot; The control component controls the movement of the bipedal robot based on the control signal; The auxiliary reward is used to provide feedback on the training actions of the bipedal robot and optimize the policy network during training; the policy network obtains the current state based on the control command; the control component drives the joints of the bipedal robot to rotate so that the bipedal robot tends from the current state to the state required by the control command.

[0005] In conjunction with the first aspect, the control signal includes a joint angle, and the control component outputs a joint torque based on the joint angle to control the joint to rotate to a specified position.

[0006] In conjunction with the first aspect, the auxiliary reward includes at least one of a power consumption reward function, a consistency reward function, and a gait reward function; The power consumption reward function is used to output a power consumption reward value based on the power consumption of the joint during rotation. The consistency reward function is used to output a consistency reward value based on the level of body consistency during the bipedal robot's movement. The gait reward function is used to output a gait reward value based on the distance between the feet and knees of the bipedal robot.

[0007] In conjunction with the first aspect, the body consistency includes the state jointly maintained by the torso of the bipedal robot and at least one of its feet, knees, and hips.

[0008] In conjunction with the first aspect, the power consumption reward function includes the following formula: ; In the formula, Indicates power consumption value. Indicates joint torque. Indicates joint velocity; The consistency reward function includes the following formula: ; In the formula, This represents the consistency reward value. Parameters representing the torso Parameters representing the feet, Parameters representing the knee, Parameters representing the buttocks; The gait reward function includes the following formula: ; ; In the formula, and Both represent distance. and These represent the distances between the left and right feet and the knees, respectively. and This indicates the preset spacing hyperparameter. and These represent the actual sole height and the reference sole height, respectively.

[0009] In conjunction with the first aspect, the step of the policy network generating control signals based on the received control commands and the current state of the bipedal robot further includes: Obtain the desired joint position from the control command; The strategy network obtains the current joint position of the joint based on the desired joint position; The strategy network obtains the joint angle to be rotated by the joint based on the desired joint position and the current joint position; The joint angle is the control signal.

[0010] In conjunction with the first aspect, the training method for the policy network includes: Simulation training and real machine training are performed, and an initial-order policy network is obtained after the simulation training is completed. The initial strategy network is input into the bipedal robot and connected to the control component. In the real machine training, the initial policy network is combined with the bipedal robot after training to obtain the policy network and evaluation network.

[0011] In conjunction with the first aspect, the simulation training method includes: An initial policy network, an initial evaluation network, and a simulated bipedal robot are constructed in a simulation platform. The first observation data of the simulated bipedal robot is input into the initial policy network, and the initial policy network outputs the initial action value. The second observation data of the simulated bipedal robot and the initial motion value are input into the initial evaluation network, and the initial evaluation network outputs the initial evaluation value. The initial evaluation value is added to the input of the initial policy network in the next iteration; Based on the initial evaluation value, the initial policy network optimizes the initial action value; The input to the initial policy network also includes control instructions, and the second observation data includes the first observation data; After the simulation training is completed, the initial policy network and the evaluation network are obtained.

[0012] In conjunction with the first aspect, the method for real-device training includes: The acquired third observation data of the bipedal robot is input into the initial policy network; The initial policy network outputs initial action values ​​based on the third observation data and the control command; The evaluation network outputs a preliminary evaluation value based on the preliminary action value, and the preliminary evaluation value is added to the input of the preliminary policy network in the next iteration. Based on the initial evaluation value, the initial policy network optimizes the initial action value; The trained policy network is obtained through continuous optimization.

[0013] In conjunction with the first aspect, prior to the step of inputting the initial policy network into the bipedal robot and connecting it to the control component, the method further includes: Collect the ground truth data of the bipedal robot; The true data is divided into a training set, a validation set, and a test set; The initial policy network, after training, is retrained using the ground truth data in the training set, and after training, it is validated and tested using the ground truth data in the validation set and the test set, respectively. Once the test is passed, the initial policy network can be obtained.

[0014] In conjunction with the first aspect, the first observation data includes one or more of the following related to the simulated bipedal robot: joint position, joint velocity, gravity vector, angular velocity, linear velocity, linear velocity command, and historical motion information.

[0015] In conjunction with the first aspect, the second observation data also includes terrain data, which includes height measurements around the simulated bipedal robot.

[0016] In conjunction with the first aspect, the control component includes a PD controller, the joint includes a motor, and the PD controller is electrically connected to the motor.

[0017] Secondly, this application provides a bipedal robot motion control system, the system comprising: The training module is used to obtain the policy network through auxiliary reward training; An output module is provided for the policy network to generate control signals based on the received control commands and the current state of the bipedal robot. A control module enables the control components to control the movement of the bipedal robot based on the control signals; The auxiliary reward is used to provide feedback on the training actions of the bipedal robot and optimize the policy network during training; the policy network obtains the current state based on the control command; the control component drives the joints of the bipedal robot to rotate so that the bipedal robot tends from the current state to the state required by the control command.

[0018] Thirdly, this application provides an electronic device, including a memory and a processor; the memory is used to store a computer program; the processor is used to implement the bipedal robot motion control method as described in any one of the first aspects when the computer program is executed.

[0019] One of the above technical solutions has the following advantages or beneficial effects: Compared with existing technologies, this application provides a bipedal robot motion control method, which includes: obtaining a policy network through auxiliary reward training; the policy network generating control signals based on received control commands and the current state of the bipedal robot; a control component controlling the bipedal robot's motion based on the control signals; wherein, the auxiliary reward is used to provide feedback on the bipedal robot's training actions and optimize the policy network during training; the policy network obtaining the current state based on the control commands; and the control component driving the joints of the bipedal robot to rotate so that the bipedal robot tends from the current state to the state required by the control commands. The bipedal robot control method provided in this application controls the joint rotation of the bipedal robot through a policy network pre-trained with auxiliary rewards, thereby improving the smoothness of the bipedal robot's movement and enhancing the consistency between the bipedal robot's torso and limbs. Attached Figure Description

[0020] The technical solution and other beneficial effects of this application will become apparent from the following detailed description of specific embodiments in conjunction with the accompanying drawings.

[0021] Figure 1 A flowchart illustrating the method provided in the embodiments of this application; Figure 2 A flowchart illustrating the simulation training method provided in this application embodiment; Figure 3 A schematic flowchart illustrating the real-machine training method provided in this application embodiment; Figure 4 A flowchart illustrating the training process of the evaluation network and measurement network provided in embodiments of this application; Figure 5 This is a schematic diagram of system module connections provided in an embodiment of this application; Figure 6 A schematic diagram of a real bipedal robot provided for an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. In the description of this application, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0023] Those skilled in the art will note that existing motion control methods for legged robots mainly include model-based hierarchical control, trajectory optimization using simplified models, and trajectory tracking using feedback linearization control methods. Many common control methods exist, including simplified model linear inverted pendulums, spring inverted pendulums, capture points, the Zero Moment Point (ZMP) model used by ASIMO, MPC used by ATLAS, Whole-body control (WBC) used by Mini Cheetah, the Hybrid Zero Motion (HZD) model used by Jessy Grizzle's group, reinforcement learning methods and nonlinear MPC methods used by ETH Zurich. Other fundamental control methods for legged robots include inverse kinematics, QP, PD, and LQR. In legged robot control, user control commands and environmental information serve as inputs to the robot controller, which then outputs torque to control the robot. Motion control is essentially an optimization problem; the goal is to find the most efficient motor torque when the loss function is minimized.

[0024] Existing model-based motion control is essentially an optimization problem, and the whole-body dynamics models of bipedal robots are typically high-dimensional and nonlinear. Unless the model is simplified, these optimization problems are difficult to solve. However, motion control based on simplified models is heavily dependent on the environment and requires continuous optimization based on the environment, which poses a significant challenge to dynamically changing application scenarios.

[0025] Because model-based control methods have limitations, mathematical models cannot be used to model many situations. This application utilizes reinforcement learning to solve the motion control problem of a legged robot, obtaining network parameters through iterative learning using a simulator to acquire a large number of samples. The robot's state information serves as the network input, and the network output is the torque of the motor in the simulator.

[0026] To address the stable walking and interference resistance capabilities of bipedal humanoid robots in complex real-world environments, this paper presents a motion control method and system for bipedal robots, leveraging advancements in artificial intelligence technology. By utilizing the bipedal robot's own sensory information, stable walking and interference resistance are achieved in complex environments without relying on external sensors.

[0027] The specific implementation methods of this application are illustrated below through examples: It should be noted that the bipedal robot mentioned in this application is 1.5 meters tall and weighs 45 kilograms. Its mechanical structure consists of a torso, two arms, and two legs. Each leg contains five joints: three for the hip, one for the knee, and one for the ankle. Each arm contains four joints: three shoulder joints and one elbow joint. The torso has one joint for yaw and rotational movements. The entire bipedal robot has 19 degrees of freedom (e.g., ...). Figure 6 (As shown). Each joint of the bipedal robot is equipped with an encoder, which can acquire the joint position in real time. By sensing the drive current, the joint torque value can be estimated in real time. The main body is equipped with an IMU to obtain real-time information on the body's center of mass movement.

[0028] like Figure 1 As shown in the figure, this application provides a motion control method for a bipedal robot, the method including: S1: Obtain the policy network through auxiliary reward training.

[0029] Specifically, in this embodiment, auxiliary rewards are used to provide feedback on the training actions of the bipedal robot and optimize the policy network during training. Auxiliary rewards help the bipedal robot learn smooth gait and facilitate the transfer from a good simulated environment to a real environment. Rewards include variables such as joint torque rewards, z-axis velocity rewards, power consumption rewards, control smoothness rewards, and consistency rewards for trunk and joint positions. Among these, joint torque rewards are used to output joint torque reward values ​​based on the rotation of the joint torques; the joint torque reward has the following functional formula: ; In the above formula, This represents the joint torque bonus value. Indicates joint torque; The z-axis speed reward is used to output a z-axis speed reward value based on the consistency between the bipedal robot's speed on the z-axis and the control commands; the z-axis speed reward has the following function formula: ; In the above formula, r z This represents the z-axis velocity value. This represents the velocity along the z-axis in the body coordinate system. The power consumption reward function outputs a power consumption reward value based on the power consumption of the joint during rotation; the power consumption reward function has the following formula: ; In the formula, Indicates power consumption value. Indicates joint torque. Indicates joint velocity; The control smoothing reward is used to output a control smoothing reward value based on the smoothness of control between the limbs or joints of a bipedal robot; the control smoothing reward has the following function formula: ; In the above formula, r smooth This indicates that the reward value is controlled and smoothed. a t Indicates the action value. a t-1 Indicates historical action information; The consistency reward function outputs a consistency reward value based on the level of body consistency during the bipedal robot's movement. Body consistency includes the state maintained jointly by one or more of the robot's torso, legs, knees, and hips. The consistency reward function has the following formula: ; In the above formula, This represents the consistency reward value. Parameters representing the torso Parameters representing the feet, Parameters representing the knee, Parameters representing the buttocks.

[0030] In this embodiment, the auxiliary reward also includes a gait reward function, which outputs a gait reward value based on the distance between the bipedal robot's feet and knees. The gait reward function allows the bipedal robot to imitate and learn human gait, thereby increasing the stability and aesthetics of the bipedal robot's walking. The gait reward function includes: ; ; In the formula, and Both represent distance. and These represent the distances between the left and right feet and the knees, respectively. and This indicates the preset spacing hyperparameter. and These represent the actual sole height and the reference sole height, respectively.

[0031] In this embodiment, the training of the policy network also includes task rewards. The learning objective of the bipedal robot is to walk stably according to speed commands; therefore, the task reward is defined as a speed tracking reward, including linear velocity and angular velocity. The corresponding task reward function is defined as: ; ; in, and These represent the actual linear velocity and commanded linear velocity along the x-axis and y-axis of the bipedal robot's base coordinate system, respectively. and These represent the actual angular velocity and the command angular velocity, respectively.

[0032] S:2: The policy network generates control signals based on the received control commands and the current state of the bipedal robot.

[0033] The specific method includes: obtaining the desired joint position from the control command; the policy network obtaining the current joint position based on the desired joint position; the policy network obtaining the joint angle to be rotated based on the desired joint position and the current joint position; the joint angle is the control signal. The control signal includes the joint angle, and the control component outputs joint torque based on the joint angle to control the joint to rotate to the specified position. The control component includes a PD controller, and the joint includes a motor. The PD controller is electrically connected to the motor. With 19 joints, there are 19 motors, and all 19 motors are controlled by the PD controller. The policy network determines the joint angle that the motor needs to rotate based on the control command and the bipedal robot's current state. The PD controller controls the motor to rotate to the corresponding position according to the required rotation angle, thereby achieving drive control of the bipedal robot's joint rotation. It can be understood that by jointly controlling the 19 joints, the bipedal robot can perform different actions and postures.

[0034] S3: The control component controls the bipedal robot's movement based on control signals. The policy network is obtained through pre-training and acquires the current state based on control commands. The control component drives the bipedal robot's joints to rotate, causing the robot to move from the current state towards the state required by the control commands. Specifically, the policy network training methods include simulation training and real-machine training. Simulation training uses a computer simulator to create a scene and model identical to the existing environment, resulting in an initial policy network. This initial policy network is then input into a real bipedal robot in the real environment and matched with the control component. After matching, real-machine training is performed using a real bipedal robot in a displayed scene. Combining the initial policy network obtained from the simulation environment with the trained bipedal robot yields the final policy network and evaluation network.

[0035] like Figure 2 and Figure 4 As shown in the embodiments of this application, the main methods of simulation training include: A1: An initial policy network, an initial evaluation network, and a simulated bipedal robot are constructed within the simulation platform. Specifically, the simulated bipedal robot is built based on a real-world bipedal robot. The relevant parameters of the simulated bipedal robot, such as height, weight, mechanical structure, and number of joints, are identical to those of the real robot. The initial policy network serves as a prototype of the policy network, used to analyze relevant data of the simulated bipedal robot in the simulation environment based on control commands, and then make decisions to control the robot's movement. The initial evaluation network is used to evaluate the actions made by the policy network in the simulation environment, thereby optimizing subsequent actions of the initial policy network and training a better policy network.

[0036] A2: Input the first observation data of the simulated bipedal robot into the initial policy network, and the initial policy network outputs initial motion values. Specifically, the initial policy network outputs the initial motion values ​​of the simulated bipedal robot based on the first observation data and control commands. The initial motion values ​​refer to the joint angles controlling the 19 joints of the bipedal robot. The first observation data is defined in the state space of the initial policy network, and the first observation data includes the joint positions related to the simulated bipedal robot. Joint velocity Gravity vector describing the fuselage attitude g Angular velocity of the fuselage on the x, y, and z axes Linear velocity relative to the x-axis and y-axis of the fuselage coordinate system v Linear velocity command v cmd Historical action information a t-1 One or more of the following. Among them, joint position... Joint velocity and historical action information a t-1 All data are from 19 joints, and the joint velocities are... Joint position The first derivative. And the gravity vector. g angular velocity linear velocity v Linear velocity command v cmd These represent parameters in three directions: the z-axis is the direction the bipedal robot is pointing vertically upwards, the x-axis is the direction the head is facing, and the y-axis is the line connecting the two shoulders. The x-axis, y-axis, and z-axis are perpendicular to each other.

[0037] A3: The second observation data and initial action values ​​of the simulated bipedal robot are input into the initial evaluation network, which outputs an initial evaluation value. Specifically, the initial evaluation value is used to assess the value of the initial action values ​​made by the initial strategy network, i.e., the degree to which the initial strategy network controls the bipedal robot to complete tasks based on control commands. The higher the completion rate, the higher the initial evaluation value; the lower the completion rate, the lower the initial evaluation value. The evaluation network uses time difference error to continuously iterate the initial strategy network, thereby reducing the error between the initial output value and the actual value. In addition to containing all the parameters in the first observation data, the second observation data also includes multiple discrete height measurements around the bipedal robot. These discrete height measurements are randomly collected within a 1m×1m square area centered on the bipedal robot. The collection method includes sampling once every 0.1m within the square area, thus obtaining at least 121 discrete height measurements within the square area. It should be noted that in other embodiments of this application, the collected area can be larger than 1m×1m or smaller than 1m×1m, and the sampling distance can be greater than 0.1m or less than 0.1m, which can be selected according to the actual situation.

[0038] A4: The initial evaluation value is added to the input of the next initial policy network. Specifically, inputting the initial evaluation value output by the initial evaluation network into the input of the initial policy network can improve the output accuracy of the initial policy network.

[0039] A5: The initial policy network optimizes the initial motion values ​​based on the initial evaluation values. The input to the initial policy network also includes control commands. Specifically, to improve the accuracy of the initial motion values ​​output by the initial policy network, initial evaluation values ​​are input into the network, allowing it to optimize the output motion values ​​based on the scores. When the initial evaluation score is higher than a certain value, the initial policy network can maintain the previous output, thus ensuring stability. Training is complete when the initial policy network achieves certain standards for the initial motion values ​​output by all joints and actions. Control commands include user-input speed or angular velocity. The initial policy network controls the corresponding joints of the simulated bipedal robot to rotate according to these speed or angular velocity commands, thereby controlling the robot's movement.

[0040] A6: After simulation training, a basic policy network and an evaluation network are obtained. Specifically, after training in the simulation environment, since the data is obtained from the simulation environment, which does not yet possess the various disturbance data that simulate the real environment, what is obtained is only a preliminary basic policy network and a mature evaluation network.

[0041] like Figure 3 and Figure 4 As shown in the embodiments of this application, the method for real-device training includes: B1: Input the acquired third observation data of the bipedal robot into the primary policy network. Specifically, the third observation data has the same content as the first observation data, including joint positions. q Joint velocity Gravity vector g angular velocity linear velocity v Linear velocity command v cmd Historical action information a t-1 The difference between the first and third observation data lies in the fact that the first observation data represents the relevant data of the simulated bipedal robot, while the second observation data represents the relevant data of the real bipedal robot. Due to various constraints between reality and simulation, although the content is the same, the values ​​are not equivalent. It should be noted that the third observation data input to the initial-order policy network also includes the estimation data of the real bipedal robot made by the initial-order policy network through state estimation. After adding the estimation data to the third observation data, it is input into the initial-order policy network, which then controls the real bipedal robot to perform corresponding actions based on the third observation data and control commands.

[0042] B2: The initial policy network outputs initial motion values ​​based on the third observation data and control commands. Specifically, the initial policy network outputs the actual initial motion values ​​of the bipedal robot based on the third observation data and the input control commands. The actual bipedal robot then controls its joints to rotate according to the initial motion values, thereby performing the corresponding actions.

[0043] B3: The evaluation network outputs an initial evaluation value based on the initial action values, which is then added to the input of the initial policy network for the next iteration. Specifically, the evaluation network outputs an initial evaluation value based on the control effect of the actual bipedal robot's actions on the initial action values ​​of the initial policy network; if the actual bipedal robot's actions are accurate, the initial evaluation value score is high; if the actual bipedal robot's actions are not standard, the initial evaluation value score is low.

[0044] B4: Based on the initial evaluation value, the initial policy network optimizes the initial action value. Specifically, the initial policy network uses the initial evaluation value as a reference or input to optimize the initial action value for the next iteration.

[0045] B5: Obtain the trained policy network through continuous optimization. The trained policy network is finally obtained through continuous optimization of the initial policy network and evaluation network in steps B1 to B4.

[0046] In this embodiment, before inputting the initial policy network into the bipedal robot and connecting it with the control components, the method further includes: collecting ground truth data from the bipedal robot; dividing the ground truth data into training, validation, and test sets; retraining the initial policy network after training using the ground truth data from the training set, and validating and testing it using the ground truth data from the validation and test sets respectively after training; the initial policy network is obtained after passing the tests. It should be noted that, to reduce the gap between the simulation environment and the real-world bipedal robot algorithm deployment effect, this application collects real ground truth data from bipedal robots, forming a dataset of at least 900,000 ground truth data points. 70% of the ground truth data in the dataset is used as the training set, 20% as the validation set, and 10% as the test set. The actuator network is trained using historical motion information of joint position tracking errors, joint speeds, and torque commands to construct a mapping relationship between expected torque and actual torque. The torque estimated by the trained actuator network is then mixed with the torque calculated by the PD controller in the simulation environment for further training. This leads to the acquisition of a trained PD controller, which is then deployed onto a real bipedal robot to improve the matching strength with the policy network.

[0047] In this embodiment, only one MLP (Multilayer Perceptron) forms the evaluation network during the bipedal robot model training process. The evaluation network directly projects the network input to the state-value function or action-value function. The actual and hidden layer nodes of the evaluation network are adjusted and assigned different tasks. The input to the policy network is the proprioception data collected by the real bipedal robot, i.e., the first observation data; then the first observation data is input into the RNN module. The RNN module extracts the historical action information created by the real bipedal robot and then sends the historical action information to another evaluation network. The RNN module combines the proprioception data and the historical action information to calculate the output to the PD controller system, thereby realizing the control of the real bipedal robot's actions. This policy network can be implemented through reinforcement learning. To improve the performance of the bipedal robot model, the minimum loss function is designed as follows: ; in This represents the parameters in the policy network. This represents the loss function in the reinforcement learning process.

[0048] like Figure 6 As shown, based on the same inventive concept, this application also provides a bipedal robot motion control system, the system comprising: The training module is used to obtain the policy network through auxiliary reward training; The output module is used by the policy network to generate control signals based on the received control commands and the current state of the bipedal robot. The control module enables the control components to control the bipedal robot's movement based on control signals. The auxiliary reward is used to provide feedback on the training actions of the bipedal robot and optimize the policy network during training; the policy network obtains the current state based on the control command; the control component drives the joint rotation of the bipedal robot so that the bipedal robot tends from the current state to the state required by the control command.

[0049] Based on the same inventive concept, embodiments of this application also provide an electronic device, including a memory and a processor; the memory is used to store a computer program; the processor is used to implement the bipedal robot motion control method provided in any of the above embodiments when executing the computer program.

[0050] The foregoing has provided a detailed description of a bipedal robot motion control method, system, and electronic device according to the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the technical solutions and core ideas of this application. Those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A motion control method for a bipedal robot, characterized in that, The method includes: The policy network is obtained through auxiliary reward training; The strategy network generates control signals based on the received control commands and the current state of the bipedal robot; The control component controls the movement of the bipedal robot based on the control signal; The auxiliary reward is used to provide feedback on the training actions of the bipedal robot and optimize the policy network during training; the policy network obtains the current state based on the control command; the control component drives the joints of the bipedal robot to rotate so that the bipedal robot tends from the current state to the state required by the control command; The auxiliary reward includes at least one of a power consumption reward function, a consistency reward function, and a gait reward function; the power consumption reward function is used to output a power consumption reward value based on the power consumption of the joint during rotation; the consistency reward function is used to output a consistency reward value based on the body consistency of the bipedal robot during movement; and the gait reward function is used to output a gait reward value based on the distance between the foot and knee of the bipedal robot.

2. The bipedal robot motion control method as described in claim 1, characterized in that, The control signal includes a joint angle, and the control component outputs a joint torque based on the joint angle to control the joint to rotate to a specified position.

3. The bipedal robot motion control method as described in claim 1, characterized in that, The body consistency refers to the state maintained jointly by the torso and at least one of the feet, knees, and hips of the bipedal robot.

4. The bipedal robot motion control method as described in claim 1, characterized in that, The power consumption reward function includes the following formula: ； In the formula, Indicates power consumption value. Indicates joint torque. Indicates joint velocity; The consistency reward function includes the following formula: ； In the formula, This represents the consistency reward value. Parameters representing the torso Parameters representing the feet, Parameters representing the knee, Parameters representing the buttocks; The gait reward function includes the following formula: ； In the formula, and Both represent distance. and These represent the distances between the left and right feet and the knees, respectively. and This indicates the preset spacing hyperparameter. and These represent the actual sole height and the reference sole height, respectively.

5. The bipedal robot motion control method as described in claim 1, characterized in that, The step of generating control signals by the policy network based on the received control commands and the current state of the bipedal robot further includes: Obtain the desired joint position from the control command; The strategy network obtains the current joint position of the joint based on the desired joint position; The strategy network obtains the joint angle to be rotated by the joint based on the desired joint position and the current joint position; The joint angle is the control signal.

6. The bipedal robot motion control method as described in claim 1, characterized in that, The training method for the policy network includes: Simulation training and real machine training are performed, and an initial-order policy network is obtained after the simulation training is completed. The initial strategy network is input into the bipedal robot and connected to the control component. In the real machine training, the initial policy network is combined with the bipedal robot after training to obtain the policy network and evaluation network.

7. The bipedal robot motion control method as described in claim 6, characterized in that, The simulation training method includes: An initial policy network, an initial evaluation network, and a simulated bipedal robot are constructed in a simulation platform. The first observation data of the simulated bipedal robot is input into the initial policy network, and the initial policy network outputs the initial action value. The second observation data of the simulated bipedal robot and the initial motion value are input into the initial evaluation network, and the initial evaluation network outputs the initial evaluation value. The initial evaluation value is added to the input of the initial policy network in the next iteration; Based on the initial evaluation value, the initial policy network optimizes the initial action value; The input to the initial policy network also includes control instructions, and the second observation data includes the first observation data; After the simulation training is completed, the initial policy network and the evaluation network are obtained.

8. The bipedal robot motion control method as described in claim 6, characterized in that, The method for real-device training includes: The acquired third observation data of the bipedal robot is input into the initial policy network; The initial policy network outputs initial action values ​​based on the third observation data and the control command; The evaluation network outputs a preliminary evaluation value based on the preliminary action value, and the preliminary evaluation value is added to the input of the preliminary policy network in the next iteration. Based on the initial evaluation value, the initial policy network optimizes the initial action value; The trained policy network is obtained through continuous optimization.

9. The bipedal robot motion control method as described in claim 7, characterized in that, Before the step of inputting the initial policy network into the bipedal robot and connecting it to the control component, the method further includes: Collect the ground truth data of the bipedal robot; The true data is divided into a training set, a validation set, and a test set; The initial policy network, after training, is retrained using the ground truth data in the training set, and after training, it is validated and tested using the ground truth data in the validation set and the test set, respectively. Once the test is passed, the initial policy network can be obtained.

10. The bipedal robot motion control method as described in claim 7, characterized in that, The first observation data includes one or more of the following related to the simulated bipedal robot: joint position, joint velocity, gravity vector, angular velocity, linear velocity, linear velocity command, and historical motion information.

11. The bipedal robot motion control method as described in claim 7, characterized in that, The second observation data also includes terrain data, which includes height measurements around the simulated bipedal robot.

12. The bipedal robot motion control method as described in claim 1, characterized in that, The control component includes a PD controller, the joint includes a motor, and the PD controller is electrically connected to the motor.

13. A motion control system for a bipedal robot, characterized in that, The system includes: The training module is used to obtain the policy network through auxiliary reward training; An output module is provided for the policy network to generate control signals based on the received control commands and the current state of the bipedal robot. The control module enables the control components to control the movement of the bipedal robot based on the control signals; The auxiliary reward is used to provide feedback on the training actions of the bipedal robot and optimize the policy network during training; the policy network obtains the current state based on the control command; the control component drives the joints of the bipedal robot to rotate so that the bipedal robot tends from the current state to the state required by the control command; The auxiliary reward includes at least one of a power consumption reward function, a consistency reward function, and a gait reward function; the power consumption reward function is used to output a power consumption reward value based on the power consumption of the joint during rotation; the consistency reward function is used to output a consistency reward value based on the body consistency of the bipedal robot during movement; and the gait reward function is used to output a gait reward value based on the distance between the foot and knee of the bipedal robot.

14. An electronic device, characterized in that: It includes a memory and a processor; the memory is used to store a computer program; the processor is used to implement the bipedal robot motion control method as described in any one of claims 1-12 when the computer program is executed.

Images