Multi-gait controller generation method, controller and quadruped robot

Abstract

The invention provides a multi-gait controller generation method, a controller and a quadruped robot, and relates to the technical field related to robot control, in the multi-gait controller generation method, a two-stage training framework is designed based on deep reinforcement learning, the controller is generated through the two-stage training framework, and the quadruped robot is controlled through the controller. The control mode can be divided into automatic posture control and manual posture control, so that when kinematics and dynamics models of the quadruped robot are not needed, the quadruped robot can be controlled to stably move in various gaits only by using an IMU and an encoder of a joint motor as body sensors, and in addition, the stability of the quadruped robot is improved. It is guaranteed that the quadruped robot can pass under the complex terrain and stably move in various gaits.

Description

Technical Field

[0001] The present invention relates to the technical field related to robot control, and in particular to a multi-gait controller generation method, a controller and a quadruped robot. Background Art

[0002] Currently, motion control algorithms for quadruped robots can be divided into two categories: traditional motion control algorithms based on kinematic and dynamic models, and intelligent motion control algorithms based on deep reinforcement learning. These algorithms share the common characteristic of requiring the construction of kinematic and dynamic models for quadruped robots, and their performance relies on the accuracy of these models. As a complex machine with a high degree of freedom, the complete kinematic and dynamic models for quadruped robots are often quite complex. To meet control frequency requirements, these control algorithms often use simplified models to increase computational speed. This characteristic significantly reduces the motion performance of quadruped robots controlled using traditional control algorithms when the models are inaccurate or when encountering external disturbances that have not yet been modeled, thereby reducing the quadruped robot's ability to adapt to complex environments. Summary of the Invention

[0003] The present invention aims to provide a multi-gait controller generation method, controller, and quadruped robot. In the multi-gait controller generation method, a two-stage training framework is designed based on deep reinforcement learning. The controller generated through the two-stage training framework eliminates the need for kinematic and dynamic models of the quadruped robot during controller implementation, and can be controlled to stably move in multiple gaits using only body sensors. Based on this, the generated controller can be divided into automatic posture control and manual posture control. Depending on the complexity of the environment, the operator can select control mode for the quadruped robot, ensuring that the quadruped robot can navigate complex terrain and stably move in multiple gaits.

[0004] In order to solve the above technical problems, the present invention adopts the following solutions:

[0005] A multi-gait controller generation method, comprising:

[0006] S1. Build a simulation environment. When the quadruped robot's movements interact with the simulation environment, determine the quadruped robot's ontological historical observation, ontological observation, and privileged observation at the current moment through the simulation environment.

[0007] The ontology historical observation is composed of ontology observations at multiple consecutive moments. The ontology observations are information obtained through the ontology sensors carried by the quadruped robot and the instructions of the remote control. The privileged observations include the ontology observations and information obtained through the simulation environment.

[0008] S2. Enter the first stage of training. Input the historical observations of the subject into the environmental feature estimator. Through training, the environmental feature estimator estimates the potential characteristics of the environment and the linear velocity of the robot body. The estimation results and the subject observations are input into the policy network. Through training, the action output by the policy network is obtained. At the same time, the privileged observations are input into the value network. Through training, the state value output by the value network is obtained. When the environmental feature estimator, policy network and value network are updated, the state value guides the update of the policy network.

[0009] S3, determine whether the number of updates reaches the preset rounds. If so, interrupt the first stage training, save the trained parameters of the environment feature estimator, policy network and value network, and go to step S4;

[0010] S4. Enter the second stage of training, load the parameters trained in the environment feature estimator, policy network, and value network in the first stage of training, train the terrain-command estimator based on the privileged observation to output the first command, and the first command covers the information of the entity observation input in the policy network; at the same time, train the history-command estimator based on the entity observation history to output the second command, and fit the second command to the first command through the mean square error;

[0011] S5. Determine whether the mean square error between the first instruction and the second instruction is less than a threshold. If so, end the second stage training, save the trained parameters of the policy network, history-instruction estimator, and environment feature estimator, and go to step S6.

[0012] S6. Based on the parameters of the policy network, history-command estimator, and environment feature estimator, an end-to-end controller is constructed from the body sensor and remote control information to the joint motor control commands.

[0013] A further preferred technical solution is: the proprioceptor sensor refers to the IMU and the encoder of the joint motor, and the proprioceptor observation includes the Euler angle and angular velocity measured by the IMU, the joint angle and joint angular velocity measured by the encoder, and the remote control instruction;

[0014] The information obtained through the simulation environment in the privileged observation includes the terrain information of the position of the quadruped robot in the simulation environment, the linear velocity of the robot body, and the external interference force on the robot body.

[0015] A further preferred technical solution is: the environmental feature estimator is an encoder of a variational autoencoder. In the first stage of training, the historical observations of the entity are input into the encoder, and the estimated environmental potential features and the linear velocity of the robot body are sampled from its output using a reparameterization method. The estimated environmental potential features and the linear velocity of the robot body are then input into the policy network together with the entity observations.

[0016] A further preferred technical solution is: in the first stage, the action output by the strategy network is a control instruction for driving the joint motor to rotate, and the control instruction calculates the torque of the joint according to the desired position of the joint motor to drive,

[0017] A further preferred technical solution is: the process of obtaining the terrain information of the position of the quadruped robot in the simulation environment is: in the simulation environment, the height information of each point is sampled by a dot matrix centered on the origin of the quadruped robot's body coordinate system, and the sampling points are equally spaced in the x-axis and y-axis directions. The height information of each point is expressed as the difference between the terrain height at the coordinate position of the point and the current body height of the quadruped robot. The vector composed of all points is called terrain information.

[0018] A further preferred technical solution is: in S2, when the environment feature estimator, the policy network and the value network are updated, the process of guiding the policy network update by the state value is:

[0019] When the quadruped robot's action reaches a preset number of steps, the policy network and value network are updated according to the update method of the proximal policy optimization algorithm, and the value network guides the policy network to update through the estimated state value, while the environmental feature estimator is updated every time the policy network and value network are updated.

[0020] A further preferred technical solution is: the environmental feature estimator is an encoder of a variational autoencoder, and the encoder is equipped with a decoder, which is used to assist in updating the encoder. When the encoder is updated, the environmental potential features are used as the input of the decoder, so that the decoder reconstructs the estimated ontological observations at the next moment. Then, the real ontological observations at the next moment are determined through the simulation environment, and the mean square error between the estimated ontological observations and the real ontological observations is calculated. The loss function of the variational autoencoder is added to serve as the loss function for updating the encoder and decoder, and finally the optimizer is used to update the parameters of the environmental feature estimator and decoder through the gradient descent method.

[0021] A further preferred technical solution is: in S4, the process of training the terrain-command estimator to output the first command based on the privileged observation specifically comprises: extracting the remote control command and terrain information of the quadruped robot's position in the simulation environment obtained through the simulation environment from the privileged observation, using the remote control command and the terrain information as inputs to the terrain-command estimator, so that the terrain-command estimator outputs the first command;

[0022] In S4, the process of training the history-command estimator to output the second command based on the subject observation is specifically as follows: removing the remote control command from the subject observation history, using the removed subject observation history as the input of the history-command estimator, and causing the history-command estimator to output the second command;

[0023] The first instruction and the second instruction are both used to estimate the current action of the quadruped robot.

[0024] A controller is obtained using a multi-gait controller generation method. The controller controls a quadruped robot in two modes: automatic posture control and manual posture control. When the controller switches to automatic posture control, a history-command estimator operates and outputs a second instruction based on the body observation history. The body observation input into the policy network is overwritten by the second instruction, so that the policy network drives the joint motor to rotate according to the second instruction. When the controller switches to manual posture control, the history-command estimator does not operate, and the instruction input into the policy network comes from the body observation and is given by a remote control, so that the policy network drives the joint motor to rotate according to the instruction given by the remote control.

[0025] A quadruped robot is deployed with a controller obtained by using the multi-gait controller generation method described above. The structure of the quadruped robot consists of a torso, hip, thigh, calf, foot end and joint motors. The hip is connected to the torso via a hip joint, the thigh is connected to the hip via a thigh key, the calf is connected to the thigh via a calf joint, and the foot end is rigidly connected to the calf. The hip and thigh joints are respectively driven by their corresponding joint motors. The calf motor is moved up to be coaxial with the thigh motor, and the calf joint is driven by a connecting rod. The foot end, calf, thigh, hip and corresponding joint motors form a leg. The two legs of the quadruped robot on the same side are configured identically, and the two legs on opposite sides are mirror-symmetrical. A joint encoder is integrated on each key motor of the quadruped robot.

[0026] Beneficial effects of the present invention:

[0027] The present invention provides a method for generating a multi-gait controller, which adopts a deep reinforcement learning method and only uses the IMU and joint motor encoders on the quadruped robot as sensors to design a two-stage training framework. First, the training data in the two-stage training framework only uses the IMU and joint motor encoders on the quadruped robot as sensor data. Compared with the existing technology that uses sensors such as cameras and radars on the quadruped robot, it reduces external influences and facilitates actual machine deployment. Then, the two-stage training framework is composed of a first-stage training and a second-stage training with an associated relationship. Specifically, the first-stage training is conducted through an asymmetric actor-critic training framework. To train an environmental feature estimator, so that the robot can indirectly obtain environmental information using only proprioceptive information, and at the same time add gait-related instructions and rewards, so that the trained policy network can output corresponding actions, control the robot to stably pass through complex terrain with multiple gaits, and can change gait, adjust body height, leg lift height, body pitch angle, roll angle and other parameters according to the operator's instructions; in the second stage of training, a command estimator is designed, and the current optimal command is inferred through the command estimator based on part of the proprioceptive historical information of the quadruped robot, thereby automatically controlling some command parameters in the policy network trained in the first stage, further improving the ability of the quadruped robot to traverse complex terrain.

[0028] On this basis, the corresponding controller can be generated based on the two-stage training framework. During the generation, only three networks with fixed parameters, namely the policy network, the environmental feature estimator and the history-command estimator, are used, and only inference is performed. It is relatively convenient to build an end-to-end controller from sensor and remote control information to joint motor control commands. Moreover, through the influence between the policy network in the first stage training and the command estimator in the second stage training, the controller has two control modes, namely automatic posture control and manual posture control. The switching of control modes can reduce the operating difficulty of the operator, so that the operator can choose how to control the quadruped robot according to the complexity of the environment, ensuring that the quadruped robot can pass through complex terrain and move stably with a variety of gaits. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of the flow of the multi-gait controller generation method in Example 1 of the present invention.

[0030] Figure 2 Schematic diagram of the training process of the two-stage training framework in Example 1 of the present invention.

[0031] Figure 3 Schematic diagram of the training process of the environmental feature estimator in Example 1 of the present invention.

[0032] Figure 4This is a schematic diagram of the controller control process in Example 2 of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0034] Unless otherwise specifically stated, the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0035] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0036] Additionally, descriptions of well-known structures, functions, and configurations may be omitted for clarity and conciseness. Those skilled in the art will recognize that various changes and modifications can be made to the examples described herein without departing from the spirit and scope of the present disclosure.

[0037] Technologies, methods and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods and equipment should be considered part of the authorization specification.

[0038] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0039] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments:

[0040] Example 1

[0041] In this embodiment, a multi-gait controller generation method is provided, and the multi-gait controller generation method is applied to a controller carried by a quadruped robot. The multi-gait controller generation method adopts deep reinforcement learning. Compared with the existing technology, the present invention does not require the kinematic and dynamic models of the quadruped robot in the controller generation process based on deep reinforcement learning. There is no need to perform complex modeling, and the model can be smoothly deployed on the controller. Among them, the deep reinforcement learning refers to a proximal strategy optimization algorithm, which is a learning method that continuously trains and optimizes the strategy of the intelligent agent through "trial and error" during the interaction between the intelligent agent and the environment.

[0042] On this basis, the present invention also takes into account that the cameras, radars and other sensors carried on the quadruped robots in the prior art have a low update frequency and are easily affected by external conditions such as light. It chooses to use only the IMU and joint motor encoders carried on the quadruped robot as sensors, that is, the body sensors, which can reduce external influences and facilitate actual machine deployment. Specifically, the Euler angle and angular velocity can be measured by the IMU, and the joint angle and joint angular velocity can be measured by the encoder of the joint motor. In order to enable the present invention to cope with various complex environments through these data, a two-stage training framework is proposed. The two-stage training framework can not only enable the quadruped robot to understand gait-related instructions when coping with different environments to switch different gaits, but also enable the quadruped robot to automatically select and execute corresponding instructions through the instruction estimator. In this way, the operator can choose to manually or automatically control the quadruped robot to walk in various gaits according to the complexity of the environment.

[0043] like Figure 1 As shown, a multi-gait controller generation method includes the following steps:

[0044] S1. Build a simulation environment. When the quadruped robot's movements interact with the simulation environment, determine the quadruped robot's ontological historical observation, ontological observation, and privileged observation at the current moment through the simulation environment.

[0045] The ontology historical observation is composed of ontology observations at multiple consecutive moments. The ontology observations are information obtained through the ontology sensors carried by the quadruped robot and the instructions of the remote control. The privileged observations include the ontology observations and information obtained through the simulation environment.

[0046] S2. Enter the first stage of training. Input the historical observations of the subject into the environmental feature estimator. Through training, the environmental feature estimator estimates the potential characteristics of the environment and the linear velocity of the robot body. The estimation results and the subject observations are input into the policy network. Through training, the action output by the policy network is obtained. At the same time, the privileged observations are input into the value network. Through training, the state value output by the value network is obtained. When the environmental feature estimator, policy network and value network are updated, the state value guides the update of the policy network.

[0047] S3: Determine whether the number of updates has reached the preset number of rounds. If so, interrupt the first phase of training, save the trained parameters of the environment feature estimator, policy network, and value network, and go to step S4. The purpose is to ensure that the environment feature estimator, policy network, and value network are trained for a sufficient number of rounds in the first phase of training to ensure that the robot can correctly follow the remote control instructions after training.

[0048] S4. Enter the second stage of training, load the parameters trained in the environment feature estimator, policy network, and value network in the first stage of training, train the terrain-command estimator based on the privileged observation to output the first command, and the first command covers the information of the entity observation input in the policy network; at the same time, train the history-command estimator based on the entity observation history to output the second command, and fit the second command to the first command through the mean square error;

[0049] S5. Determine whether the mean square error between the first instruction and the second instruction is less than a threshold. If so, end the second stage training, save the trained parameters of the policy network, history-instruction estimator, and environment feature estimator, and go to step S6.

[0050] S6. Based on the parameters of the policy network, history-command estimator, and environment feature estimator, an end-to-end controller is constructed from the body sensor and remote control information to the joint motor control commands.

[0051] Specifically, in this embodiment, the body observation refers to information obtained solely based on body sensors such as IMU and encoders of joint motors and instructions from the remote control; the body historical observation refers to a composition of body observations at multiple consecutive moments; the privileged observation refers to information that, in addition to the body observation, also includes terrain information, external force interference information, and other information that can only be obtained in the placement environment and cannot be obtained in reality; the action refers to the action generated by the policy network for interacting with the environment. In addition, the policy network uses generated actions to interact with the environment; the value network is used to guide the update of the policy network; the environmental feature estimator is a network for estimating the potential features of the environment based on the body observation history; the instruction estimator includes two networks: a terrain-instruction estimator and a history-instruction estimator. Among them, the terrain-instruction estimator refers to a network that gives instructions based on the terrain information in the privileged observation, and the history-instruction estimator refers to a network that gives instructions based on the body observation history, and approximates the terrain-instruction estimator through regression.

[0052] As can be seen, the present invention proposes a method for generating a multi-gait controller based on a two-stage training framework. The two-stage training framework includes first-stage training and second-stage training. The first-stage training only trains the policy network, the medium network, and the environmental feature estimator, while the second-stage training mainly trains the command estimator, including the history-command estimator and the terrain-command estimator. Furthermore, the first-stage and second-stage training are related. During the second-stage training, the output of the command estimator can overlay the information of the input entity observations of the policy network, allowing the command estimator to influence the actions output by the policy network. This influence relationship links the first-stage and second-stage training. Furthermore, after completing the first-stage training, the quadruped robot has acquired a certain degree of walking ability, can adapt to complex terrain, and can switch gaits according to commands. After completing the second-stage training, the quadruped robot can automatically change parameters such as body posture and leg lift height when navigating complex terrain such as stairs and slopes, thereby improving its ability to navigate these terrains.

[0053] like Figure 2 As shown in the figure, at the beginning of the first phase of training, the simulation environment provides feedback on the robot's body observation history, body observations, and privileged observations. The value network inputs the privileged observations and outputs their estimated state values, which guide policy updates when updating network parameters. The environmental feature estimator inputs the body observation history and outputs estimates of the environment's latent features and the robot's linear velocity. Subsequently, the body observations, the estimated latent features of the environment, and the estimated linear velocity of the robot are combined and used as input to the policy network. The policy network outputs an action, which, after interacting with the simulation environment, then provides feedback on the body observation history, body observations, and privileged observations for the next moment, and this cycle repeats.

[0054] like Figure 2 As shown, at the start of the second phase of training, the trained network parameters of the policy network, value network, and environmental feature estimator from the first phase are loaded, and further parameter updates of these three networks are prevented. Based on the first phase of training, the second phase of training feeds the history-command estimator with the ontological observations and the terrain-command estimator with a portion of the privileged observations. Both estimators output a predicted command. The command output by the terrain-command estimator overwrites the posture control command content of the information input to the policy network, while the command output by the history-command estimator is regressed to the command output by the terrain-command estimator using a mean squared error loss. After the second phase of training is completed, the network parameters of the policy network, environmental feature estimator, and history-command estimator are saved for deployment.

[0055] A further preferred technical solution is: the environmental feature estimator is an encoder of a variational autoencoder, which uses probability distribution to describe the latent space, greatly improving the generalization ability of the variational autoencoder. The main task of using the variational autoencoder is to extract features from the input sample data, thereby mapping the input data to a low-dimensional latent space, and then reconstruct the input data using the extracted latent features. After sufficient training, the variational autoencoder can extract data features to achieve the function of data compression; it can also randomly sample in the latent feature space and reconstruct sample data through features to achieve the function of data generation. It can be seen that in this embodiment, in the first stage of training, the historical observations of the entity are input into the encoder, and the estimated environmental latent features and the linear velocity of the robot body are sampled from its output using the reparameterization method. The estimated environmental latent features and the linear velocity of the robot body are then input into the policy network along with the entity observations.

[0056] A further preferred technical solution is: the environmental feature estimator is an encoder of a variational autoencoder, and the encoder is equipped with a decoder, which is used to assist in updating the encoder. When the encoder is updated, the environmental potential features are used as the input of the decoder, so that the decoder reconstructs the estimated ontological observations at the next moment. Then, the real ontological observations at the next moment are determined through the simulation environment, and the mean square error between the estimated ontological observations and the real ontological observations is calculated. The loss function of the variational autoencoder is added to serve as the loss function for updating the encoder and decoder, and finally the optimizer is used to update the parameters of the environmental feature estimator and decoder through the gradient descent method.

[0057] As can be seen, the environmental feature estimator is equivalent to the encoder of the variational autoencoder, while its decoder only assists in updating the environmental feature estimator and does not participate in updating the policy network. Specifically, the environmental feature estimator's input layer has a dimension of 348, which is the dimension of the six-time-step body observation history; the output layer has a dimension of 38, where the first six dimensions represent the explicitly estimated mean and variance of the robot's linear velocity, and the last 32 dimensions represent the implicitly estimated mean and variance of the environment's latent features. The environmental feature estimator has two hidden layers with dimensions of 128 and 64, respectively, and uses the ELU activation function. The decoder's input layer has a dimension of 19, with the first three dimensions representing samples of the explicitly estimated robot linear velocity and the last 16 dimensions representing samples of the implicitly estimated environment's latent features. The decoder's output layer has a dimension of 58, which is the dimension of the body observation. The decoder also has two hidden layers with dimensions of 64 and 128, respectively, and also uses the ELU activation function.

[0058] The environmental feature estimator training process is as follows Figure 3 As shown in the figure is the ontological observation history at the current moment, O t+1It is the real ontological observation at the next moment of the simulation environment feedback. It is the estimated ontological observation at the next moment based on the ontological observation history at the current moment; is the linear velocity of the robot body estimated by the environmental feature estimator, V is the real linear velocity of the simulation environment feedback at the current moment; z is the potential feature of the environment estimated by the environmental feature estimator.

[0059] Specifically, the environment feature estimator is updated every time the policy network and the value network are updated. When updating, the ontology observation history is input to the environment feature estimator. The estimated robot body linear velocity is sampled from its output using the reparameterization method And the environmental latent feature z. The environmental latent feature z is then used as the input of the decoder to enable the decoder to reconstruct the ontological observation at the next moment Then use the real value V fed back from the simulation environment and the estimated value The mean squared error is calculated and added to the loss function of the variational autoencoder as the loss function for updating the environmental feature estimator and decoder. Finally, the optimizer is used to update the network parameters of the environmental feature estimator and decoder through gradient descent.

[0060] A further preferred technical solution is: inputting the estimation results and the ontology observation into the policy network, obtaining the action output by the policy network through training, and the action is used to interact with the environment, and designing the action space. Specifically, when the quadruped robot has n joint motors, the output of the policy network needs to include the control instructions of the n joint motors. Therefore, the action space is designed to be n dimensions, each dimension represents a key expected position. In the simulation environment, the torque of each key can be calculated according to the expected position to drive the key.

[0061] A further preferred technical solution is: input privileged observations into the value network, obtain the state value output by the value network through training, and design the state space. In this embodiment, the value network can obtain privileged observations, and the policy network can only obtain ontological observations. Two observation spaces, ontological observations and privileged observations, are designed for the state space. Since the dimensions of each frame of data in the observation space are different, the numerical differences are large. At the same time, considering the importance of each type of data in the observation space, a weight coefficient needs to be multiplied before each type of data.

[0062] The proprioceptive observation only uses the quadruped robot's IMU and joint motor encoders as proprioceptive sensors. In addition to the sensors, the remote control's commands are also used as part of the proprioceptive observation. The remote control's commands include:

[0063] (1) Issue speed commands to control the robot to move forward, backward, left and right, and rotate in place.

[0064] (2) Issue gait instructions to control the robot to switch gaits or change parameters such as gait frequency.

[0065] (3) Issue posture commands to enable the robot to change parameters such as pitch angle, roll angle, leg lift height, and body height during walking.

[0066] Among them, the remote control commands have the same dimension as the number of joint motors.

[0067] For ontological observation, since the reward function often shows periodic changes when training gait-related tasks, this often affects the agent's judgment of the optimal strategy and makes it unable to find a fixed pattern. Therefore, the state provided to the strategy must contain information about the periodic reward function. Specifically, the ontological observation contains a timing reference variable so that the policy network receives the timing reference variable of the phase of the current reward cycle.

[0068] The privileged observation is only used during simulation training. Therefore, it can include some information that cannot be obtained in reality on the basis of the body observation, improve the perception ability of the intelligent body in the simulation, and thus improve the training efficiency and training effect. Specifically, in order to enable the quadruped robot to fully understand the surrounding terrain, the surrounding terrain height information is added to the privileged observation, and a 17*11 dot matrix centered on the origin of the robot's body coordinate system is used to sample the height information of each point. The sampling points are spaced 0.1m apart in the x and y directions, and the height information of each point is expressed as the difference between the terrain height at the coordinate position of the point and the current body height of the robot. The vector composed of these 187 numbers is called terrain information. In addition, the linear velocity of the robot body and the external interference force on the body are also added to the privileged observation.

[0069] In summary, in this embodiment, a state space and action space are designed using deep reinforcement learning. The state space defines the information the agent can perceive in the environment, while the action space determines how the agent interacts with the environment. Based on this, a reward function is also designed. The reward function is divided into a movement command following reward, a posture command following reward, a gait reward, and an auxiliary movement reward. The movement command following reward is a positive reward, while the other three are negative penalties.

[0070] The movement instruction following reward is used to encourage the robot to move according to the movement instruction of the remote control. The closer the current linear speed of the robot body is to the linear speed instruction of the remote control, the higher the reward.

[0071] The gait reward is used to regulate the robot's gait. First, it penalizes the contact force between the foot end of the leg in the swing phase and the ground, and penalizes the linear velocity of the foot end of the leg in the support phase; secondly, it penalizes the behavior of the foot end deviating from the nominal foot end position in the horizontal direction.

[0072] The posture command following reward is used to penalize the robot for deviations from posture-related commands. The closer the robot's current height, roll, and pitch are to the desired height, roll, and pitch given by the remote controller, the smaller the penalty. The closer the foot-end height of the robot's swinging leg is to the nominal foot-end height, the smaller the penalty.

[0073] The auxiliary motion reward is used to punish the robot's unstable and unsmooth motion and excessive power consumption of the joints; and to punish the self-collision between various parts and the collision between parts other than the foot end and the ground.

[0074] Based on the above reward functions, the total reward obtained from a single-step interaction between the robot and the environment can be calculated. To balance the contribution of each reward to the total reward and facilitate fine-tuning of the reward during subsequent training, the reward function in this embodiment has a large number of penalty terms, so the single-step total reward is designed to be a non-negative value. Although the calculated single-step total reward decreases with more negative penalties, the total reward is always positive.

[0075] A further preferred technical solution is: in S2, when the environment feature estimator, the policy network and the value network are updated, the process of guiding the policy network update by the state value is:

[0076] When the quadruped robot's action reaches a preset number of steps, the policy network and value network are updated according to the update method of the proximal policy optimization algorithm, and the value network guides the policy network to update through the estimated state value, while the environmental feature estimator is updated every time the policy network and value network are updated.

[0077] A further preferred technical solution is: in S4, the process of training the terrain-command estimator to output the first command based on the privileged observation specifically comprises: extracting the remote control command and terrain information of the quadruped robot's position in the simulation environment obtained through the simulation environment from the privileged observation, using the remote control command and the terrain information as inputs to the terrain-command estimator, so that the terrain-command estimator outputs the first command;

[0078] In S4, the process of training the history-command estimator to output the second command based on the subject observation is specifically as follows: removing the remote control command from the subject observation history, using the removed subject observation history as the input of the history-command estimator, and causing the history-command estimator to output the second command;

[0079] The first instruction and the second instruction are both used to estimate the current action of the quadruped robot.

[0080] For the second phase of training, a command estimator is designed. This command estimator is used to automatically select posture commands when facing complex terrain. Because the current quadruped robot is equipped with n joint motors, in order to control the joint motors, the controller generated by deep reinforcement learning needs to give n commands. The commands correspond to the joint motors, and the policy network outputs actions. This greatly increases the difficulty of operation for the operator. For example, when facing terrain such as stairs, slopes, rocks, and steps, the operator first needs to adjust the body pitch and roll angle commands according to the slope of the terrain, and then adjust the leg lift height command and the body height command according to the height of the obstacles in the terrain. In addition, the operator also needs to control the speed command. In this case, the increased difficulty of operation for the operator may reduce the robot's ability to pass.

[0081] Therefore, to avoid this situation, the present invention sets up a second stage of training based on the first stage of training. The second stage of training is associated with the first stage of training through the policy network. In the second stage of training, a command estimator is designed. The command estimator is composed of two neural networks, namely a history-command estimator and a terrain-command estimator. The history-command estimator generates an estimated value of the posture command based on the body's observation history, and the terrain-command estimator generates an estimated value of the posture command based on the terrain information. Through training, the history-command estimator is regressed into the terrain-command estimator, and the estimated value output by the history-command estimator is overwritten with the content of the posture control command in the information input to the policy network, that is, information overlay. The overwritten information enters the policy network, and the policy network automatically gives the robot posture commands to assist the robot in navigating the environment.

[0082] The terrain-command estimator takes as input the robot's velocity command from the remote controller and terrain height information from privileged observations. The history-command estimator takes as input the robot's observation history after removing the remote controller's commands. The outputs of both networks are estimates of the current posture to be adopted, including body height, leg lift height, body pitch angle, and body roll angle. As training progresses, the posture commands generated by both networks are continuously optimized, approaching the optimal posture for the current environment.

[0083] Specifically, the terrain-command estimator has an input layer dimension of 190, consisting of 187-dimensional terrain information plus 3-dimensional speed commands. The terrain-command estimator has three hidden layers, with dimensions of 128, 64, and 32, respectively, and uses the ELU activation function. The history-command estimator has an input dimension of 324, consisting of the dimensions of the six-step history of the proprietor's observations after removing the four-dimensional remote control commands from the proprietor's observations at each moment. The hidden layer structure of the history-command estimator is the same as that of the terrain-command estimator. The output layers of both networks have a dimension of 4, representing the four-dimensional attitude commands. Because the estimation of attitude commands is explicit, and the attitude commands themselves have a restricted range of values, this paper adds a Tanh activation layer after the output layers of both networks to map the output values ​​of both networks to (-1, 1).

[0084] At the beginning of the second phase of training, the weight of the posture command following reward needs to be reset to 0. Otherwise, the command estimator can obtain the highest posture command following reward by simply outputting the robot's current body height, leg lift height, pitch angle, and roll angle, rather than selecting the optimal posture based on the current environment. This prevents the final trained command estimator from becoming ineffective.

[0085] Example 2

[0086] On the basis of Example 1, in this embodiment, a controller is generated using a multi-gait controller generation method provided in Example 1. After the first stage training and the second stage training are completed, an end-to-end controller is constructed from the body sensor and remote control information to the joint motor control instructions based on the parameters of the policy network, the history-command estimator, and the environmental feature estimator. In addition, the parameters of these three networks are fixed, and only the posture of the quadruped robot at the next moment is inferred through the three networks. The value network, the decoder of the environmental feature estimator, and the terrain-command estimator in the two-stage training framework are only used to assist in the training of the above three networks. Therefore, they do not need to be used during deployment, which makes it more convenient for operators to carry out actual machine deployment.

[0087] Among them, the goal of the first stage training is: the environmental feature estimator can correctly estimate the potential characteristics of the environment and the linear velocity of the fuselage, and the policy network can give actions that are consistent with the guidance of the reward function based on the observation information; the goal of the second stage training is to enable the terrain-command estimator to give reasonable posture commands, so that the robot can pass through different complex environments, and at the same time, the history-command estimator can give the same posture commands as the terrain-command estimator based on the proprioception history.

[0088] like Figure 4As shown, the proprioceptive observation and proprioceptive observation history are given by the encoders of the remote control, IMU and joint motor. Based on the two-stage training framework, the controller's control mode for the quadruped robot is divided into automatic posture control and manual posture control. The operator can choose the control mode according to the complexity of the environment. When facing a complex environment, you can choose to switch to the automatic posture instruction mode, and use the history-instruction estimator to give posture instructions to cover the posture instructions given by the remote control, so that the robot can automatically adjust its posture to cope with the current environment. When switching to automatic posture control, the history-instruction estimator works and outputs a second instruction based on the proprioceptive observation history. The proprioceptive observation input into the policy network is covered by the second instruction, so that the policy network drives the joint motor to rotate according to the second instruction; when switching to manual posture control, the history-instruction estimator does not work, and the instruction input to the policy network comes from the proprioceptive observation, which is given by the remote control, so that the action output by the policy network is used as the control instruction of the joint motor to control the rotation of the joint motor.

[0089] Furthermore, in real-world scenarios, the transitions between the quadruped robot's modes—from power-up to walking, to standby, to stopping, and finally to power-off—are logically linked. For example, switching directly from prone mode to walking mode could damage the motor due to excessive position error. Furthermore, to handle emergencies, a power-off protection mode is also required. If the quadruped robot experiences unexpected motion, switching to power-off protection mode can minimize damage to the robot.

[0090] In summary, the controller is generated based on a two-stage training framework, which enables two control modes. When the environment is complex, it can switch to an automatic posture command mode. The controller can estimate the robot through the history-command estimator, overlay the estimated value with the body observation, and feed the estimated value and some body observation into the policy network. The policy network automatically issues the robot posture command to assist the robot in navigating the environment. In addition, the environmental feature estimator based on the variational autoencoder implicitly estimates the potential characteristics of the environment and the linear velocity of the robot body, thus enabling the controller to have a certain degree of perception of the current environment.

[0091] Example 3

[0092] On the basis of Example 1, in this example, a quadruped robot is proposed, on which a controller obtained by using the multi-gait controller generation method is deployed, the structure of the quadruped robot consists of a trunk, hip, thigh, calf, foot end and joint motors, the hip is connected to the trunk through the hip joint, the thigh is connected to the hip through the thigh key, the calf is connected to the thigh through the calf joint, the foot end is rigidly connected to the calf, the hip and thigh joints are respectively driven by their corresponding joint motors, the calf motor is moved up to be coaxial with the thigh motor, and the calf joint is driven by a connecting rod; the foot end, calf, thigh, hip and corresponding joint motors form a leg, the two legs of the quadruped robot on the same side are arranged identically, and the two legs on the opposite sides are mirror-symmetrical; a joint encoder is integrated on each key motor of the quadruped robot.

[0093] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Based on the technical essence of the present invention and within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement of the above embodiment shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for generating a multi-gait controller, characterized in that: include: S1. Build a simulation environment. When the quadruped robot's movements interact with the simulation environment, determine the quadruped robot's ontological historical observation, ontological observation, and privileged observation at the current moment through the simulation environment. The ontology historical observation is composed of ontology observations at multiple consecutive moments. The ontology observations are information obtained through the ontology sensors carried by the quadruped robot and the instructions of the remote control. The privileged observations include the ontology observations and information obtained through the simulation environment. S2. Enter the first stage of training. Input the historical observations of the subject into the environmental feature estimator. Through training, the environmental feature estimator estimates the potential characteristics of the environment and the linear velocity of the robot body. The estimation results and the subject observations are input into the policy network. Through training, the action output by the policy network is obtained. At the same time, the privileged observations are input into the value network. Through training, the state value output by the value network is obtained. When the environmental feature estimator, policy network and value network are updated, the state value guides the update of the policy network. S3, determine whether the number of updates reaches the preset rounds. If so, interrupt the first stage training, save the trained parameters of the environment feature estimator, policy network and value network, and go to step S4; S4. Enter the second stage of training, load the parameters trained in the environment feature estimator, policy network, and value network in the first stage of training, train the terrain-command estimator based on the privileged observation to output the first command, and the first command covers the information of the entity observation input in the policy network; at the same time, train the history-command estimator based on the entity observation history to output the second command, and fit the second command to the first command through the mean square error; S5. Determine whether the mean square error between the first instruction and the second instruction is less than a threshold. If so, end the second stage training, save the trained parameters of the policy network, history-instruction estimator, and environment feature estimator, and go to step S6. S6. Based on the parameters of the policy network, history-command estimator, and environment feature estimator, an end-to-end controller is constructed from the body sensor and remote control information to the joint motor control commands.

2. A multi-gait controller generation method according to claim 1, characterized in that: The proprioceptive sensors refer to the encoders of the IMU and joint motors, and the proprioceptive observations include the Euler angles and angular velocities measured by the IMU, the joint angles and joint angular velocities measured by the encoders, and the instructions of the remote control; The information obtained through the simulation environment in the privileged observation includes the terrain information of the position of the quadruped robot in the simulation environment, the linear velocity of the robot body, and the external interference force on the robot body.

3. The method for generating a multi-gait controller according to claim 1, wherein: The environmental feature estimator is an encoder of a variational autoencoder. In the first stage of training, the historical observations of the entity are input into the encoder, and the estimated environmental potential features and the linear velocity of the robot body are sampled from its output using a reparameterization method. The estimated environmental potential features and the linear velocity of the robot body are then input into the policy network along with the entity observations.

4. A method for generating a multi-gait controller according to claim 2, characterized in that: In the first stage, the action output by the strategy network is a control instruction for driving the joint motor to rotate. The control instruction calculates the torque of the joint according to the expected position of the joint motor to drive.

5. The method for generating a multi-gait controller according to claim 2, wherein: The process of obtaining the terrain information of the position of the quadruped robot in the simulation environment is as follows: in the simulation environment, the height information of each point is sampled from a dot matrix centered at the origin of the quadruped robot's body coordinate system, and the sampling points are equally spaced in the x-axis and y-axis directions. The height information of each point is expressed as the difference between the terrain height at the coordinate position of the point and the current body height of the quadruped robot. The vector composed of all points is called terrain information.

6. A method for generating a multi-gait controller according to claim 1, characterized in that: In S2, the process of guiding the policy network update through the state value when the environment feature estimator, policy network and value network are updated is: When the quadruped robot's action reaches a preset number of steps, the policy network and value network are updated according to the update method of the proximal policy optimization algorithm, and the value network guides the policy network to update through the estimated state value, while the environmental feature estimator is updated every time the policy network and value network are updated.

7. A method for generating a multi-gait controller according to claim 6, characterized in that: The environmental feature estimator is an encoder of a variational autoencoder, which is equipped with a decoder. The decoder is used to assist in updating the encoder. When the encoder is updated, the potential features of the environment are used as the input of the decoder, so that the decoder reconstructs the estimated ontological observations at the next moment. Then, the real ontological observations at the next moment are determined through the simulation environment, and the mean square error between the estimated ontological observations and the real ontological observations is calculated. The loss function of the variational autoencoder is added to it and used as the loss function for updating the encoder and decoder. Finally, the optimizer is used to update the parameters of the environmental feature estimator and decoder through the gradient descent method.

8. The method for generating a multi-gait controller according to claim 1, wherein: In S4, the process of training the terrain-command estimator to output the first command based on the privileged observation is specifically as follows: extracting the remote control command and terrain information of the quadruped robot's position in the simulation environment obtained through the simulation environment from the privileged observation, using the remote control command and the terrain information as inputs to the terrain-command estimator, so that the terrain-command estimator outputs the first command; In S4, the process of training the history-command estimator to output the second command based on the subject observation is specifically as follows: removing the remote control command from the subject observation history, using the removed subject observation history as the input of the history-command estimator, and causing the history-command estimator to output the second command; The first instruction and the second instruction are both used to estimate the current action of the quadruped robot.

9. A controller, characterized in that: The controller is obtained by using a multi-gait controller generation method as described in any one of claims 1-8. The controller's control mode for the quadruped robot is divided into automatic posture control and manual posture control. When switching to automatic posture control, the history-instruction estimator works and outputs a second instruction based on the body observation history. The body observation input into the policy network is overwritten by the second instruction, so that the policy network drives the joint motor to rotate according to the second instruction; when switching to manual posture control, the history-instruction estimator does not work, and the instruction input into the policy network comes from the body observation and is given by the remote control, so that the policy network drives the joint motor to rotate according to the instruction given by the remote control.

10. A quadruped robot, characterized in that: The quadruped robot is deployed with a controller obtained by a multi-gait controller generation method as described in any one of claims 1-8. The structure of the quadruped robot consists of a torso, hip, thigh, calf, foot end and joint motors. The hip is connected to the torso through a hip joint, the thigh is connected to the hip through a thigh key, the calf is connected to the thigh through a calf joint, and the foot end is rigidly connected to the calf. The hip and thigh joints are respectively driven by their corresponding joint motors. The calf motor is moved up to be coaxial with the thigh motor, and the calf joint is driven by a connecting rod; the foot end, calf, thigh, hip and corresponding joint motors form a leg, the two legs of the quadruped robot on the same side are set in the same way, and the two legs on the opposite sides are mirror-symmetrical; a joint encoder is integrated on each key motor of the quadruped robot.

Images