Motion control method and device, model training method and device, and electronic device

Abstract

This disclosure relates to a motion control method and apparatus, a model training method and apparatus, and an electronic device. The method includes: acquiring observation data of the current frame collected by at least one sensor of a robot; inputting observation data of at least one frame prior to the current frame into a state estimation network to obtain state estimation data of the current frame output by the state estimation network, wherein the state estimation data is used to characterize the state of the robot itself and / or the environment in which the robot is located; inputting the observation data of the current frame and the state estimation data of the current frame into a policy network to obtain control commands output by the policy network; and controlling the robot to move based on the control commands.

Description

Technical Field

[0001] This disclosure relates to the field of robotics, specifically to a motion control method and apparatus, a model training method and apparatus, and electronic equipment. Background Technology

[0002] In recent years, robotics technology has continuously developed, becoming increasingly intelligent and automated, with improvements in the richness, stability, and flexibility of its movements. Robots can be controlled not only using classical control methods such as PID control, linear control, state-space control, and model predictive control, but also using learning-based control methods such as reinforcement learning and imitation learning. All these control methods rely on the robot collecting information about itself and its environment; in other words, the richness of the information collected directly affects the precision and accuracy of robot control.

[0003] However, in related technologies, the types and number of robot sensors limit the amount of information that robots can collect for robot control. Summary of the Invention

[0004] To overcome the problems existing in the related technologies, the present disclosure provides a motion control method and device, a model training method and device, and an electronic device to solve the defects in the related technologies.

[0005] According to a first aspect of the present disclosure, a motion control method is provided, the method comprising:

[0006] Acquire observation data of the current frame from at least one of the robot's sensors;

[0007] The observation data of at least one frame prior to the current frame is input into the state estimation network to obtain the state estimation data of the current frame output by the state estimation network, wherein the state estimation data is used to characterize the state of the robot itself and / or the environment in which the robot is located.

[0008] The observation data and state estimation data of the current frame are input into the policy network to obtain the control command output by the policy network.

[0009] The robot is controlled to move based on the control commands.

[0010] In some embodiments of this disclosure, the observation data includes at least one of the following: fuselage angular velocity, fuselage attitude, joint angle, joint angular velocity, fuselage linear velocity command, and control command from the previous frame.

[0011] In some embodiments of this disclosure, the state estimation data includes fuselage linear velocity and ground environment information, wherein the ground environment information is characterized by feature vectors.

[0012] In some embodiments of this disclosure, the state estimation network is pre-trained to convergence in the following manner:

[0013] Simulation observation data from at least one frame prior to the current frame is input into the state estimation network to obtain the state estimation data for the current frame output by the state estimation network. The simulation observation data is the observation data collected by the simulation robot in the simulation environment based on at least one sensor.

[0014] The state estimation data of the current frame is input into the state decoding network to obtain the estimated observation data of the current frame output by the state decoding network;

[0015] A first network loss is determined based on the state estimation data and the simulated state data of the current frame. A second network loss is determined based on the estimated observation data and the simulated observation data of the current frame. The network parameters of the state estimation network and the state decoding network are adjusted based on the first network loss and the second network loss until convergence.

[0016] In some embodiments of this disclosure, the state estimation data includes fuselage linear velocity and ground environment information; the simulation state data includes fuselage linear velocity and ground environment information.

[0017] Determining the first network loss based on the state estimation data of the current frame and the simulation state data of the current frame includes:

[0018] The first sub-network loss is determined based on the fuselage linear velocity in the state estimation data and the fuselage linear velocity in the simulation state data;

[0019] The loss of the second sub-network is determined based on the conditional probability distribution of the ground environment information in the state estimation data and the prior probability distribution of the simulation environment.

[0020] The first network loss is determined based on the first sub-network loss and the second sub-network loss.

[0021] In some embodiments of this disclosure, determining the first sub-network loss based on the fuselage linear velocity in the state estimation data and the fuselage linear velocity in the simulation state data includes:

[0022] The mean square error between the fuselage linear velocity in the state estimation data and the fuselage linear velocity in the simulation state data is determined as the first sub-network loss.

[0023] In some embodiments of this disclosure, determining the second sub-network loss based on the conditional probability distribution of ground environment information in the state estimation data and the prior probability distribution of the simulation environment includes:

[0024] The KL divergence between the conditional probability distribution of the ground environment information in the state estimation data and the prior probability distribution of the simulation environment is determined as the second sub-network loss.

[0025] In some embodiments of this disclosure, determining the second network loss based on the estimated observation data and the simulated observation data of the current frame includes:

[0026] The mean square error between the estimated observation data and the simulated observation data of the current frame is determined as the second network loss.

[0027] In some embodiments of this disclosure, the policy network is pre-trained to convergence based on the simulation observation data and the state estimation data of the current frame.

[0028] According to a second aspect of the present disclosure, a model training method is provided, the method comprising:

[0029] Simulation observation data from at least one frame prior to the current frame is input into the state estimation network to obtain the state estimation data for the current frame output by the state estimation network. The simulation observation data is the observation data collected by the simulation robot in the simulation environment based on at least one sensor.

[0030] The state estimation data of the current frame is input into the state decoding network to obtain the estimated observation data of the current frame output by the state decoding network;

[0031] A first network loss is determined based on the state estimation data and the simulated state data of the current frame. A second network loss is determined based on the estimated observation data and the simulated observation data of the current frame. The network parameters of the state estimation network and the state decoding network are adjusted based on the first network loss and the second network loss until convergence.

[0032] In some embodiments of this disclosure, the method further includes:

[0033] The policy network is trained based on the simulation observation data and the state estimation data of the current frame until it converges.

[0034] According to a third aspect of the present disclosure, a motion control device is provided, the device comprising:

[0035] The acquisition module is used to acquire observation data of the current frame collected by at least one sensor of the robot;

[0036] An estimation module is used to input observation data from at least one frame prior to the current frame into a state estimation network to obtain state estimation data for the current frame output by the state estimation network, wherein the state estimation data is used to characterize the state of the robot itself and / or the environment in which the robot is located.

[0037] The instruction module is used to input the observation data and state estimation data of the current frame into the policy network to obtain the control instructions output by the policy network.

[0038] The control module is used to control the robot to move based on the control commands.

[0039] In some embodiments of this disclosure, the observation data includes at least one of the following: fuselage angular velocity, fuselage attitude, joint angle, joint angular velocity, fuselage linear velocity command, and control command from the previous frame.

[0040] In some embodiments of this disclosure, the state estimation data includes fuselage linear velocity and ground environment information, wherein the ground environment information is characterized by feature vectors.

[0041] In some embodiments of this disclosure, the state estimation network is pre-trained to convergence in the following manner:

[0042] Simulation observation data from at least one frame prior to the current frame is input into the state estimation network to obtain the state estimation data for the current frame output by the state estimation network. The simulation observation data is the observation data collected by the simulation robot in the simulation environment based on at least one sensor.

[0043] The state estimation data of the current frame is input into the state decoding network to obtain the estimated observation data of the current frame output by the state decoding network;

[0044] A first network loss is determined based on the state estimation data and the simulated state data of the current frame. A second network loss is determined based on the estimated observation data and the simulated observation data of the current frame. The network parameters of the state estimation network and the state decoding network are adjusted based on the first network loss and the second network loss until convergence.

[0045] In some embodiments of this disclosure, the state estimation data includes fuselage linear velocity and ground environment information; the simulation state data includes fuselage linear velocity and ground environment information.

[0046] Determining the first network loss based on the state estimation data of the current frame and the simulation state data of the current frame includes:

[0047] The first sub-network loss is determined based on the fuselage linear velocity in the state estimation data and the fuselage linear velocity in the simulation state data;

[0048] The loss of the second sub-network is determined based on the conditional probability distribution of the ground environment information in the state estimation data and the prior probability distribution of the simulation environment.

[0049] The first network loss is determined based on the first sub-network loss and the second sub-network loss.

[0050] In some embodiments of this disclosure, determining the first sub-network loss based on the fuselage linear velocity in the state estimation data and the fuselage linear velocity in the simulation state data includes:

[0051] The mean square error between the fuselage linear velocity in the state estimation data and the fuselage linear velocity in the simulation state data is determined as the first sub-network loss.

[0052] In some embodiments of this disclosure, determining the second sub-network loss based on the conditional probability distribution of ground environment information in the state estimation data and the prior probability distribution of the simulation environment includes:

[0053] The KL divergence between the conditional probability distribution of the ground environment information in the state estimation data and the prior probability distribution of the simulation environment is determined as the second sub-network loss.

[0054] In some embodiments of this disclosure, determining the second network loss based on the estimated observation data and the simulated observation data of the current frame includes:

[0055] The mean square error between the estimated observation data and the simulated observation data of the current frame is determined as the second network loss.

[0056] In some embodiments of this disclosure, the policy network is pre-trained to convergence based on the simulation observation data and the state estimation data of the current frame.

[0057] According to a fourth aspect of the present disclosure, a model training apparatus is provided, the apparatus comprising:

[0058] The simulation estimation module is used to input the simulation observation data of at least one frame before the current frame into the state estimation network to obtain the state estimation data of the current frame output by the state estimation network. The simulation observation data is the observation data collected by the simulation robot in the simulation environment based on at least one sensor.

[0059] The decoding module is used to input the state estimation data of the current frame into the state decoding network to obtain the estimated observation data of the current frame output by the state decoding network;

[0060] The parameter tuning module is used to determine a first network loss based on the state estimation data and the simulation state data of the current frame, determine a second network loss based on the estimated observation data and the simulation observation data of the current frame, and adjust the network parameters of the state estimation network and the state decoding network based on the first network loss and the second network loss until convergence.

[0061] In some embodiments of this disclosure, the apparatus further includes a synchronous training module for:

[0062] The policy network is trained based on the simulation observation data and the state estimation data of the current frame until it converges.

[0063] According to a fifth aspect of the present disclosure, an electronic device is provided, the electronic device including a memory and a processor, the memory being used to store computer instructions executable on the processor, and the processor being used to implement the motion control method provided in any embodiment of the first aspect or the model training method described in any embodiment of the second aspect when executing the computer instructions.

[0064] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the program is executed by a processor, it implements the motion control method provided in any embodiment of the first aspect or the model training method described in any embodiment of the second aspect.

[0065] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0066] The motion control method provided in this disclosure, after acquiring observation data of the current frame collected by at least one sensor of the robot, can input observation data of at least one frame prior to the current frame into a state estimation network to obtain state estimation data of the current frame output by the state estimation network. Then, the observation data and state estimation data of the current frame are input into a policy network to obtain control commands output by the policy network. Finally, the robot can be controlled to move based on these control commands. Since the information input to the policy network includes not only observation data directly collected by the robot's sensors but also state estimation data predicted based on historical observation data to characterize the robot itself and / or the environment, this method achieves high control precision and accuracy. Attached Figure Description

[0067] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0068] Figure 1This is a schematic diagram of the structure of a humanoid robot shown in an exemplary embodiment of the present disclosure;

[0069] Figure 2 This is a flowchart illustrating a motion control method according to an exemplary embodiment of this disclosure;

[0070] Figure 3 This is a flowchart illustrating the training process of a state estimation network according to an exemplary embodiment of this disclosure;

[0071] Figure 4 This is a training logic diagram of a state estimation network illustrated in an exemplary embodiment of this disclosure;

[0072] Figure 5 This is a flowchart illustrating a model training method according to an exemplary embodiment of this disclosure;

[0073] Figure 6 This is a schematic diagram of the structure of a motion control device shown in an exemplary embodiment of the present disclosure;

[0074] Figure 7 This is a schematic diagram of the structure of a model training apparatus shown in an exemplary embodiment of the present disclosure;

[0075] Figure 8 This is a structural block diagram of an electronic device illustrated in an exemplary embodiment of the present disclosure. Detailed Implementation

[0076] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0077] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0078] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0079] In recent years, robotics technology has continuously developed, becoming increasingly intelligent and automated, with improvements in the richness, stability, and flexibility of its movements. Robots can be controlled not only using classical control methods such as PID control, linear control, state-space control, and model predictive control, but also using learning-based control methods such as reinforcement learning and imitation learning. All these control methods rely on the robot collecting information about itself and its environment; in other words, the richness of the information collected directly affects the precision and accuracy of robot control.

[0080] However, in related technologies, the types and number of robot sensors limit the amount of information that robots can collect for robot control.

[0081] Based on this, in a first aspect, at least one embodiment of this disclosure provides a motion control method that can be applied to robots, such as bipedal robots (humanoid robots) and quadrupedal robots (robot dogs).

[0082] Next, let's take a humanoid robot as an example, combined with... Figure 1 A simplified description of the robot's structure is provided below. This humanoid robot comprises a waist, two upper limbs, and two lower limbs. The waist, serving as the body, has three degrees of freedom (DOF) in the pitch, roll, and yaw directions. The shoulders of the upper limbs have three DDFs in the pitch, roll, and yaw directions, the elbows have one DDF in the pitch direction, and the wrists may have two or three DDFs. The hips of the lower limbs have three DDFs in the pitch, roll, and yaw directions, the knees have one DDF in the pitch direction, and the ankles have two DDFs in the pitch and roll directions. Roll, pitch, and yaw represent rotational directions around the X, Y, and Z axes, respectively. The body is equipped with IMU sensors to acquire the robot's posture, specifically its angles and angular velocities in the pitch, roll, and yaw directions. Each joint in the upper and lower limbs has an encoder to acquire the joint's angle and angular velocity.

[0083] Please refer to the appendix. Figure 2The diagram illustrates the flow of the motion control method, including steps S201 to S204.

[0084] In step S201, observation data of the current frame collected by at least one sensor of the robot is acquired.

[0085] The observation data refers to data representing the robot's own or the environment's state that the robot can directly acquire through sensors. For example, the observation data includes at least one of the following: body angular velocity, body posture, joint angles, joint angular velocities, body linear velocity commands, and control commands from the previous frame. Specifically, body angular velocity and body posture can be acquired by the robot's IMU sensors; joint angles and joint angular velocities can be acquired by encoders on each joint of the robot; the body linear velocity command is the desired body linear velocity input by the user or automatically generated by the robot based on the current task; and the control commands from the previous frame can be stored by the robot's controller, meaning the robot saves the control commands generated in each frame for later use.

[0086] For example, the observation data o of the current frame t As shown in the following formula:

[0087]

[0088] Where, ω t g t c t θ t , a t-1 These are respectively the commands for fuselage angular velocity, fuselage attitude, joint angle, joint angular velocity, fuselage linear velocity, and control commands from the previous frame.

[0089] Preferably, this step can be performed in each control frame of the robot.

[0090] In step S202, the observation data of at least one frame prior to the current frame is input into the state estimation network to obtain the state estimation data of the current frame output by the state estimation network, wherein the state estimation data is used to characterize the state of the robot itself and / or the environment in which the robot is located.

[0091] The state estimation network is pre-trained to convergence, enabling it to predict the state estimation data for the current frame. This state estimation data is data that the robot's sensors cannot directly collect; the state it represents is necessary for the robot's motion control. Predicting this data through the state estimation network compensates for the robot's weakness in directly acquiring data, enriching the information and data used for motion control. The state estimation network can be pre-built based on a multilayer perceptron.

[0092] Among them, historical observation data, that is, observation data of at least one frame before the current frame, was acquired and saved when it became the current frame.

[0093] Preferably, this step inputs the observation data of the one frame or multiple consecutive frames closest to the current frame into the state estimation network to obtain the control command for the current frame. For example, in the current frame t, historical observation data is input into the state estimation network. As shown in the following formula:

[0094]

[0095] In the above formula, o t-1 o t-2 ... o t-H+1 H represents the observation data of consecutive frames closest to the current frame, where H is an integer greater than 0.

[0096] Preferably, the state estimation data includes fuselage linear velocity and ground environment information, wherein the ground environment information is represented by a feature vector.

[0097] The training process of the state estimation network will be described in detail below, and will not be repeated here.

[0098] In step S203, the observation data and state estimation data of the current frame are input into the policy network to obtain the control command output by the policy network.

[0099] The policy network is pre-trained to convergence, enabling it to output control commands based on the robot's own and / or the environment's state. This policy network can be a policy network within a reinforcement learning algorithm; its specific training process will be detailed below and will not be elaborated upon here.

[0100] Preferably, the control commands may include the output torque of each joint of the robot, etc.

[0101] In step S204, the robot is controlled to move based on the control command.

[0102] For example, the robot is controlled to execute the control command output in step S203, so that the robot moves to reach or approach the body linear velocity command of the current frame.

[0103] The motion control method provided in this disclosure, after acquiring observation data of the current frame collected by at least one sensor of the robot, can input observation data of at least one frame prior to the current frame into a state estimation network to obtain state estimation data of the current frame output by the state estimation network. Then, the observation data and state estimation data of the current frame are input into a policy network to obtain control commands output by the policy network. Finally, the robot can be controlled to move based on these control commands. Since the information input to the policy network includes not only observation data directly collected by the robot's sensors but also state estimation data predicted based on historical observation data to characterize the robot itself and / or the environment, this method achieves high control precision and accuracy.

[0104] This method, based on a state estimation network and observation data, enables robot state estimation. It can effectively estimate the robot's linear velocity and ground environment information, thereby enriching the types of parameters applicable to robot motion control. Compared to traditional filtering algorithms, this method is more efficient and direct in robot state estimation. Its state estimation includes feature vectors characterizing ground environment information, which can provide a complete and comprehensive implicit estimate of the ground environment information, unlike traditional methods that can only estimate slope and obstacles. This improves the robot's environmental adaptability during motion.

[0105] In some embodiments of this disclosure, the state estimation network is pre-defined according to the appendix. Figure 3 Train to convergence as shown:

[0106] In step S301, the simulation observation data of at least one frame prior to the current frame is input into the state estimation network to obtain the state estimation data of the current frame output by the state estimation network. The simulation observation data is the observation data collected by the simulation robot in the simulation environment based on at least one sensor.

[0107] In controlling the state estimation network, a simulation environment can be constructed, and a robot can be constructed in the simulation environment. This allows the simulated robot to move in the simulation environment in the same way that a real robot moves in the real environment. The simulation environment can obtain some privileged information that is not available in the real environment, such as the linear velocity of the fuselage and the prior probability distribution of the simulation environment (ground environment information).

[0108] A simulated robot can acquire observational data in a simulated environment through at least one sensor, denoted as simulated observational data. This data is consistent with the types of observational data a robot can acquire in a real environment, and includes, for example, at least one of the following: body angular velocity, body posture, joint angles, joint angular velocities, body linear velocity commands, and control commands from the previous frame. Specifically, body angular velocity and body posture can be acquired by the simulated robot's IMU sensors; joint angles and joint angular velocities can be acquired by encoders on each joint of the simulated robot; the body linear velocity command is the desired body linear velocity input by the user or automatically generated by the simulated robot based on the current task; and the control commands from the previous frame can be stored by the simulated robot's controller, meaning the simulated robot saves the control commands generated in each frame for later use. In other words, a simulated robot can simulate the movement of a robot in a real environment within a simulated environment.

[0109] The state estimation network in this step has not yet been trained to convergence. The state estimation data is data that cannot be directly collected by the sensors of the simulated robot, and the state it represents is required for robot motion control. Preferably, the state estimation data includes the body linear velocity and ground environment information, wherein the ground environment information is represented by feature vectors.

[0110] Preferably, this step inputs the simulation observation data of the frame or multiple consecutive frames closest to the current frame into the state estimation network to obtain the control command for the current frame.

[0111] In step S302, the state estimation data of the current frame is input into the state decoding network to obtain the estimated observation data of the current frame output by the state decoding network.

[0112] The state decoding network can be pre-built based on a multilayer perceptron.

[0113] Here, the estimated observation data refers to the simulated observation data predicted by the state decoding network, while the true value of the simulated observation data can be obtained based on the sensors of the simulated robot.

[0114] In step S302, a first network loss is determined based on the state estimation data of the current frame and the simulated state data of the current frame, a second network loss is determined based on the estimated observation data of the current frame and the simulated observation data of the current frame, and the network parameters of the state estimation network and the state decoding network are adjusted based on the first network loss and the second network loss until convergence.

[0115] For example, the state estimation data includes fuselage linear velocity and ground environment information; the simulation state data includes fuselage linear velocity and ground environment information.

[0116] Based on this example, the first network loss can be determined in the following manner:

[0117] First, the first sub-network loss is determined based on the fuselage linear velocity in the state estimation data and the fuselage linear velocity in the simulated state data. For example, the mean square error between the fuselage linear velocity in the state estimation data and the fuselage linear velocity in the simulated state data is determined as the first sub-network loss; that is, the first sub-network loss L1 is as follows:

[0118]

[0119] In the above formula, v represents the fuselage linear velocity in the simulation state data. t Here, represents the fuselage linear velocity in the state estimation data, and MSE represents the mean square error.

[0120] Next, based on the conditional probability distribution of the ground environment information in the state estimation data and the prior probability distribution of the simulation environment, the second sub-network loss is determined. For example, the KL divergence between the conditional probability distribution of the ground environment information in the state estimation data and the prior probability distribution of the simulation environment is determined as the second sub-network loss; that is, the second sub-network loss L2 is as follows:

[0121]

[0122] In the above formula, This indicates that in historical simulation observation data In the case of ground environment information z in state estimation data t The conditional probability distribution, p(z) t Let D be the prior probability distribution of the simulation environment. KL Let KL divergence be denoted as KL divergence.

[0123] KL divergence, also known as relative entropy, is a method in information theory used to measure the difference between two probability distributions. It quantifies the degree to which one probability distribution (P) differs from another probability distribution (Q).

[0124] Finally, the first network loss is determined based on the first sub-network loss and the second sub-network loss. For example, the first sub-network loss and the second sub-network loss are weighted and summed according to their respective weights, and the result is used as the first network loss.

[0125] Based on this example, the second network loss can be determined as follows: the mean square error between the estimated observation data and the simulated observation data of the current frame is used as the second network loss; that is, the second network loss L3 is shown in the following formula:

[0126]

[0127] In the above formula, For the simulation observation data of the current frame, o t This is the estimated observation data for the current frame.

[0128] For example, in this step, the network parameters of the state estimation network and the state decoding network can be adjusted until convergence as follows: the first sub-network loss and the second sub-network loss are weighted and summed according to their respective weights, and the result is used as the total network loss. The network parameters of the state estimation network and the state decoding network are then adjusted based on the total network loss until convergence. For example, the gradient of the total network loss at each network parameter of the state estimation network is determined, and the network parameters are adjusted in the direction of gradient descent; the gradient of the total network loss at each network parameter of the state decoding network is determined, and the network parameters are adjusted in the direction of gradient descent.

[0129] It should be understood that this embodiment can perform batch training based on the network loss obtained from multiple frames of data to improve training efficiency and the utilization of training data.

[0130] Please refer to the appendix. Figure 4 In this example, the state estimation network and state decoding network can be incorporated as sub-units into the robot's motion control model, exhibiting strong generalization and adaptability. This not only improves the robot's control precision and accuracy during motion control but also allows for synchronous training with the motion control model's policy network. In other words, based on this embodiment, the policy network can be pre-trained to convergence using simulation observation data and state estimation data from the current frame. The policy network can be trained synchronously with the state estimation network, and the policy network also incorporates the state estimation network into its input during the training phase, ensuring consistency between training and application processes, thereby guaranteeing training efficiency and application effectiveness.

[0131] For example, if the policy network is the Actor in the Actor-Critic (AC) architecture of a reinforcement learning algorithm, then the policy network can be trained to convergence in the following manner:

[0132] First, the simulation observation data and state estimation data of the current frame are input into the policy network to obtain the control commands output by the policy network.

[0133] Next, based on the control commands, the simulated robot is controlled to move in the simulation environment, and the simulation observation data and state estimation data of the next frame are obtained.

[0134] Next, the simulation observation data of the current frame, the state estimation data of the current frame, the control command of the current frame, the simulation observation data of the next frame, and the state estimation data of the next frame are input into the reward network (i.e., the Critic in the AC architecture above) to obtain the reward value output by the evaluation network.

[0135] Finally, the network parameters of the policy network and the reward network are adjusted based on the reward value.

[0136] It should be understood that this embodiment can perform batch training based on the network loss obtained from multiple frames of data to improve training efficiency and the utilization of training data.

[0137] Secondly, at least one embodiment of this disclosure provides a model training method, the method including... Figure 5 Steps S501 to S503 are shown.

[0138] In step S501, the simulation observation data of at least one frame before the current frame is input into the state estimation network to obtain the state estimation data of the current frame output by the state estimation network. The simulation observation data is the observation data collected by the simulation robot in the simulation environment based on at least one sensor.

[0139] In step S502, the state estimation data of the current frame is input into the state decoding network to obtain the estimated observation data of the current frame output by the state decoding network;

[0140] In step S503, a first network loss is determined based on the state estimation data of the current frame and the simulated state data of the current frame, a second network loss is determined based on the estimated observation data of the current frame and the simulated observation data of the current frame, and the network parameters of the state estimation network and the state decoding network are adjusted based on the first network loss and the second network loss until convergence.

[0141] For example, the method further includes:

[0142] The policy network is trained based on the simulation observation data and the state estimation data of the current frame until it converges.

[0143] More details about the above steps have been described in detail in the first part about the training process of the state estimation network and the policy network, and will not be repeated here.

[0144] According to a third aspect of the embodiments of this disclosure, a motion control device is provided; please refer to the appendix. Figure 6 The device includes:

[0145] The acquisition module 601 is used to acquire observation data of the current frame collected by at least one sensor of the robot;

[0146] The estimation module 602 is used to input the observation data of at least one frame before the current frame into the state estimation network to obtain the state estimation data of the current frame output by the state estimation network, wherein the state estimation data is used to characterize the state of the robot itself and / or the environment in which the robot is located.

[0147] The instruction module 603 is used to input the observation data and state estimation data of the current frame into the policy network to obtain the control instructions output by the policy network.

[0148] The control module 604 is used to control the robot to move based on the control commands.

[0149] In some embodiments of this disclosure, the observation data includes at least one of the following: fuselage angular velocity, fuselage attitude, joint angle, joint angular velocity, fuselage linear velocity command, and control command from the previous frame.

[0150] In some embodiments of this disclosure, the state estimation data includes fuselage linear velocity and ground environment information, wherein the ground environment information is characterized by feature vectors.

[0151] In some embodiments of this disclosure, the state estimation network is pre-trained to convergence in the following manner:

[0152] Simulation observation data from at least one frame prior to the current frame is input into the state estimation network to obtain the state estimation data for the current frame output by the state estimation network. The simulation observation data is the observation data collected by the simulation robot in the simulation environment based on at least one sensor.

[0153] The state estimation data of the current frame is input into the state decoding network to obtain the estimated observation data of the current frame output by the state decoding network;

[0154] A first network loss is determined based on the state estimation data and the simulated state data of the current frame. A second network loss is determined based on the estimated observation data and the simulated observation data of the current frame. The network parameters of the state estimation network and the state decoding network are adjusted based on the first network loss and the second network loss until convergence.

[0155] In some embodiments of this disclosure, the state estimation data includes fuselage linear velocity and ground environment information; the simulation state data includes fuselage linear velocity and ground environment information.

[0156] Determining the first network loss based on the state estimation data of the current frame and the simulation state data of the current frame includes:

[0157] The first sub-network loss is determined based on the fuselage linear velocity in the state estimation data and the fuselage linear velocity in the simulation state data;

[0158] The loss of the second sub-network is determined based on the conditional probability distribution of the ground environment information in the state estimation data and the prior probability distribution of the simulation environment.

[0159] The first network loss is determined based on the first sub-network loss and the second sub-network loss.

[0160] In some embodiments of this disclosure, determining the first sub-network loss based on the fuselage linear velocity in the state estimation data and the fuselage linear velocity in the simulation state data includes:

[0161] The mean square error between the fuselage linear velocity in the state estimation data and the fuselage linear velocity in the simulation state data is determined as the first sub-network loss.

[0162] In some embodiments of this disclosure, determining the second sub-network loss based on the conditional probability distribution of ground environment information in the state estimation data and the prior probability distribution of the simulation environment includes:

[0163] The KL divergence between the conditional probability distribution of the ground environment information in the state estimation data and the prior probability distribution of the simulation environment is determined as the second sub-network loss.

[0164] In some embodiments of this disclosure, determining the second network loss based on the estimated observation data and the simulated observation data of the current frame includes:

[0165] The mean square error between the estimated observation data and the simulated observation data of the current frame is determined as the second network loss.

[0166] In some embodiments of this disclosure, the policy network is pre-trained to convergence based on the simulation observation data and the state estimation data of the current frame.

[0167] According to a fourth aspect of the embodiments of this disclosure, a model training apparatus is provided; please refer to the appendix. Figure 7 The device includes:

[0168] The simulation estimation module 701 is used to input the simulation observation data of at least one frame before the current frame into the state estimation network to obtain the state estimation data of the current frame output by the state estimation network, wherein the simulation observation data is the observation data collected by the simulation robot in the simulation environment based on at least one sensor.

[0169] The decoding module 702 is used to input the state estimation data of the current frame into the state decoding network to obtain the estimated observation data of the current frame output by the state decoding network;

[0170] The parameter tuning module 703 is used to determine a first network loss based on the state estimation data and the simulation state data of the current frame, determine a second network loss based on the estimated observation data and the simulation observation data of the current frame, and adjust the network parameters of the state estimation network and the state decoding network based on the first network loss and the second network loss until convergence.

[0171] In some embodiments of this disclosure, the apparatus further includes a synchronous training module for:

[0172] The policy network is trained based on the simulation observation data and the state estimation data of the current frame until it converges.

[0173] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the motion control method provided in any embodiment of the first aspect or the model training method described in any embodiment of the second aspect.

[0174] According to a sixth aspect of the present disclosure, an electronic device is provided, as shown in the appendix. Figure 8 As shown, the electronic device includes a memory and a processor. The memory is used to store computer instructions that can be executed on the processor. The processor is used to implement the motion control method provided in any embodiment of the first aspect or the model training method described in any embodiment of the second aspect when executing the computer instructions.

[0175] The electronic device can be a robot, such as a legged robot (e.g., a humanoid robot, a quadruped robot), a wheeled robot, etc.

[0176] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the motion control method provided in any embodiment of the first aspect or the model training method described in any embodiment of the second aspect.

[0177] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0178] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A motion control method, characterized in that, The method includes: Acquire observation data of the current frame from at least one of the robot's sensors; The observation data of at least one frame prior to the current frame is input into the state estimation network to obtain the state estimation data of the current frame output by the state estimation network, wherein the state estimation data is used to characterize the state of the robot itself and / or the environment in which the robot is located. The observation data and state estimation data of the current frame are input into the policy network to obtain the control command output by the policy network. The robot is controlled to move based on the control commands.

2. The motion control method according to claim 1, characterized in that, The observation data includes at least one of the following: fuselage angular velocity, fuselage attitude, joint angles, joint angular velocities, fuselage linear velocity commands, and control commands from the previous frame.

3. The motion control method according to claim 1, characterized in that, The state estimation data includes fuselage linear velocity and ground environment information, wherein the ground environment information is represented by feature vectors.

4. The motion control method according to claim 1, characterized in that, The state estimation network is pre-trained to convergence in the following manner: Simulation observation data from at least one frame prior to the current frame is input into the state estimation network to obtain the state estimation data for the current frame output by the state estimation network. The simulation observation data is the observation data collected by the simulation robot in the simulation environment based on at least one sensor. The state estimation data of the current frame is input into the state decoding network to obtain the estimated observation data of the current frame output by the state decoding network; A first network loss is determined based on the state estimation data and the simulated state data of the current frame. A second network loss is determined based on the estimated observation data and the simulated observation data of the current frame. The network parameters of the state estimation network and the state decoding network are adjusted based on the first network loss and the second network loss until convergence.

5. The motion control method according to claim 4, characterized in that, The state estimation data includes fuselage linear velocity and ground environment information; the simulation state data includes fuselage linear velocity and ground environment information. The determination of the first network loss based on the state estimation data of the current frame and the simulation state data of the current frame includes: The first sub-network loss is determined based on the fuselage linear velocity in the state estimation data and the fuselage linear velocity in the simulation state data; Based on the conditional probability distribution of the ground environment information in the state estimation data and the prior probability distribution of the simulation environment, the loss of the second sub-network is determined. The first network loss is determined based on the first sub-network loss and the second sub-network loss.

6. The motion control method according to claim 5, characterized in that, The determination of the first sub-network loss based on the fuselage linear velocity in the state estimation data and the fuselage linear velocity in the simulation state data includes: The mean square error between the fuselage linear velocity in the state estimation data and the fuselage linear velocity in the simulation state data is determined as the first sub-network loss.

7. The motion control method according to claim 5, characterized in that, The determination of the second sub-network loss based on the conditional probability distribution of ground environment information in the state estimation data and the prior probability distribution of the simulation environment includes: The KL divergence between the conditional probability distribution of the ground environment information in the state estimation data and the prior probability distribution of the simulation environment is determined as the second sub-network loss.

8. The motion control method according to claim 4, characterized in that, The determination of the second network loss based on the estimated observation data and the simulated observation data of the current frame includes: The mean square error between the estimated observation data and the simulated observation data of the current frame is determined as the second network loss.

9. The motion control method according to claim 4, characterized in that, The policy network is pre-trained to converge based on the simulation observation data and the state estimation data of the current frame.

10. A model training method, characterized in that, The method includes: Simulation observation data from at least one frame prior to the current frame is input into the state estimation network to obtain the state estimation data for the current frame output by the state estimation network. The simulation observation data is the observation data collected by the simulation robot in the simulation environment based on at least one sensor. The state estimation data of the current frame is input into the state decoding network to obtain the estimated observation data of the current frame output by the state decoding network; A first network loss is determined based on the state estimation data and the simulated state data of the current frame. A second network loss is determined based on the estimated observation data and the simulated observation data of the current frame. The network parameters of the state estimation network and the state decoding network are adjusted based on the first network loss and the second network loss until convergence.

11. The model training method according to claim 10, characterized in that, The method further includes: The policy network is trained based on the simulation observation data and the state estimation data of the current frame until it converges.

12. A motion control device, characterized in that, The device includes: The acquisition module is used to acquire observation data of the current frame collected by at least one sensor of the robot; An estimation module is used to input observation data from at least one frame prior to the current frame into a state estimation network to obtain state estimation data for the current frame output by the state estimation network, wherein the state estimation data is used to characterize the state of the robot itself and / or the environment in which the robot is located. The instruction module is used to input the observation data and state estimation data of the current frame into the policy network to obtain the control instructions output by the policy network. The control module is used to control the robot to move based on the control commands.

13. A model training device, characterized in that, The device includes: The simulation estimation module is used to input the simulation observation data of at least one frame before the current frame into the state estimation network to obtain the state estimation data of the current frame output by the state estimation network. The simulation observation data is the observation data collected by the simulation robot in the simulation environment based on at least one sensor. The decoding module is used to input the state estimation data of the current frame into the state decoding network to obtain the estimated observation data of the current frame output by the state decoding network; The parameter tuning module is used to determine a first network loss based on the state estimation data and the simulation state data of the current frame, determine a second network loss based on the estimated observation data and the simulation observation data of the current frame, and adjust the network parameters of the state estimation network and the state decoding network based on the first network loss and the second network loss until convergence.

14. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory is used to store computer instructions that can be executed on the processor. The processor is used to implement the motion control method of any one of claims 1 to 9 or the model training method of any one of claims 10 to 11 when executing the computer instructions.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of any one of claims 1 to 11.

Images