Method for training motion controller of robot animal

Abstract

The present invention discloses a method for training a motion controller of a robot animal, comprising: obtaining motion data, environmental parameters and interference parameters of a reference animal, generating a plurality of primitive distributions and a first primitive weight by a strategy network according to the motion data, selecting the current state of the reference animal from the motion data, setting the adaptation state of the reference animal, generating a second primitive weight by the strategy network at least according to the current state, the adaptation state and the primitive distribution, wherein the adaptation state is the state passed by the reference animal in the current state after responding to the environmental parameters and the interference parameters, and training the strategy network by a discriminator according to the comparison result passed by the first primitive weight and the second primitive weight.

Description

Technical Field

[0001] The invention relates to the field of control, and in particular to a method for making a controller for controlling the movement of a virtual animal suitable for controlling a physical machine animal. Background Art

[0002] Controlling robotic quadrupeds to move naturally is still quite challenging. Traditional methods require extensive experience in robotics domain knowledge and complex parameter tuning. Constant drops during experiments can permanently damage the hardware and make the entire control system unstable. Therefore, training in a physical simulator seems to be the most reasonable approach, as it is fast, safe, and cheap.

[0003] Recently, Deep Reinforcement Learning (DRL) has shown great progress in solving locomotion problems. Compared to model-based approaches that require prior knowledge of real physical dynamics, deep reinforcement learning can learn a variety of skills through trial and error, reducing the need for human involvement. Many optimization techniques used to develop simulated character controllers employ deep reinforcement learning.

[0004] However, in the case of virtual robotic quadrupeds, behaviors learned through deep reinforcement learning still do not look as natural as those designed manually. Part of the challenge stems from the difficulty of specifying reward functions for natural movements, especially in the absence of biomechanical models and goals that can be used to achieve natural simulated movements. On the other hand, policies trained by letting virtual robotic quadrupeds perform real movements at the synthetic or simulated level often do not transfer well to the real world. Summary of the invention

[0005] In view of this, the present invention provides a method for training a motion controller of a robot animal, thereby allowing the controller for controlling the motion of a virtual animal to be applicable to a robot animal in the real world.

[0006] A method for training a motion controller of a robot animal proposed in accordance with one embodiment of the present invention includes: obtaining motion data, environmental parameters and interference parameters of a reference animal; using a strategy network to generate multiple primitive distributions and first primitive weights based on the motion data; selecting the current state of the reference animal from the motion data and setting the adaptation state of the reference animal; using the strategy network to generate second primitive weights based on at least the current state, the adaptation state, and the primitive distribution, wherein the adaptation state is the state passed by the reference animal in the current state after responding to the environmental parameters and the interference parameters; and using a discriminator to train the strategy network based on the comparison result of the first primitive weight and the second primitive weight.

[0007] In summary, the method for training a motion controller for a robotic animal proposed in the present invention trains a controller with robustness by adding augmented parameters applicable to actuator models, mechanical limitations, sensor interference, environmental changes, inference engine delays, and microcontroller delays during the training process. This controller is not only suitable for virtual animals in character animation, but can also directly integrate physical robotic animals and use high-level instructions to control the movement of physical robotic animals.

[0008] The above description of the disclosed contents of the present invention and the following description of the embodiments are used to demonstrate and explain the spirit and principle of the present invention, and to provide further explanation of the claims of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a schematic diagram of a robot animal;

[0010] Figure 2 is a flow chart of a method for training a motion controller of a robot according to an embodiment of the present invention;

[0011] Figure 3 yes Figure 1 A detailed flow chart after step S1; and

[0012] Figure 4 Schematic diagram of the policy network for augmenting the height graph.

[0013] Wherein, the reference numerals are:

[0014] J1~J20 joints

[0015] Steps S1~S6, S11~S13

[0016] H Height Map DETAILED DESCRIPTION

[0017] The detailed features and advantages of the present invention are described in detail in the following embodiments, and the contents are sufficient to enable any person skilled in the art to understand the technical content of the present invention and implement it accordingly, and according to the contents disclosed in this specification, the claims and the drawings of the specification, any person skilled in the art can easily understand the relevant purposes and advantages of the present invention. The following examples further illustrate the viewpoints of the present invention in detail, but do not limit the scope of the present invention in any viewpoint.

[0018] The present invention provides a method for training a motion controller of a robot. In general, for a controller that controls the motion of a virtual animal based on control parameters, the present invention augments the input parameters and output parameters in the training process on the basis of training the controller, thereby obtaining a robust controller that can be directly applied to a physical quadruped robot, and the controller can also be used in a virtual world.

[0019] Figure 1 is a schematic diagram of a robot animal. The robot animal includes multiple joints, such as Figure 1 shown. Figure 1 The quadruped robot shown has 20 joints, each of which can be equipped with a motor. The controller trained by the present invention is used to provide a rotational momentum to each motor, so that the motor at that location generates torque, thereby driving the robot to move.

[0020] Figure 2 A flow chart of an embodiment of the present invention is shown, including steps S1 to S5. Step S1 is "obtaining motion data, environmental parameters and interference parameters of a reference animal". The reference animal is a real version of a virtual animal, such as a dog. The motion data includes multiple state information of the reference animal over a period of time (e.g., 10 seconds), each state information includes the following measurement data: position, speed, rotation amount and angular velocity. Except for the rotation amount represented by 4-dimensional quaternions, the other three are represented by 3-dimensional vectors.

[0021] In one embodiment of the present invention, the motion data obtained in step S1 includes the current state and the historical state. After the motion data of the "current state" is obtained, the step of storing the current state in an input queue is further included. In other words, one embodiment of the present invention maintains an input queue that stores all state information of the reference animal within a period of time. In this way, during training, in addition to obtaining the current information of the reference animal, the state information of the reference animal a period of time ago can also be obtained.

[0022] Figure 3 is a detailed flow chart of an embodiment of step S1. Figure 3 In the method, step S11 to S13 are further included after step S1.

[0023] Step S11 is "obtaining the environmental parameters and interference parameters of the reference animal". Step S12 is "obtaining the current state of the reference animal and storing it in the input queue". Step S13 is "obtaining the historical state of the reference animal from the input queue".

[0024] In detail, steps S11 to S13 take into account the delay of the inference engine and the microcontroller. In another embodiment of the present invention, in addition to storing the state information of the previous reference animal in the input queue, the historical information of the PD controller is also stored. During training, the policy network receives the historical state queried from the input queue according to a given inference delay, and the PD controller also receives the delayed control information from the input queue according to a given PD delay time. For example, the inference delay and the PD delay are set to 15 milliseconds and 2 milliseconds respectively, and the control frequency of the policy network is increased from 30 Hertz (Hz) to 200 Hz, thereby achieving better system stability and agility. Therefore, in step S1, the current state, the historical state 15 milliseconds ago, the PD control parameters 2 milliseconds ago, and the environmental parameters and interference parameters can be received. The current state and historical state of the above-mentioned motion data can also be extended to the current and historical information of environmental parameters and interference parameters, and the present invention is not limited to this.

[0025] In one embodiment of the present invention, the environmental parameters include information such as the speed and direction of the reference animal in a specific environment. For example, consider a scene where the reference animal walks in a straight line. Because the straight road in the environment has potholes, the reference animal changes its forward direction from 12 o'clock to 9 o'clock at the tth second, and its forward speed changes from 2 meters per second to 1.5 meters per second. The numbers in the above examples are examples of environmental parameters, but these numbers and the example actions of the reference animal to avoid potholes are not used to limit the present invention. In practice, in order to enable the robot animal or virtual animal to adapt to various scenes, the number of training samples of the environmental parameters can be increased.

[0026] In one embodiment of the present invention, one implementation of "obtaining motion data" is to set multiple sensors on the body of an animal in the real world to collect motion data. Another implementation of step S11 is to obtain motion data through physics engine simulation. The present invention does not specifically limit the method of obtaining motion data in step S11.

[0027] In another embodiment of the present invention, the environmental parameters include a heightmap of the environment in which the reference animal is located. In view of the difference between the real environment and the simulated environment, the present invention takes into account the special situation of terrain changes through this environmental parameter. Generally speaking, the reference movement of the reference animal is sampled on flat terrain. The present invention takes into account irregular environmental factors and uses heightmap input to expand the policy network. For hilly environments, the heightmap is sampled on a 2D uniform grid around the character, and the size of the heightmap is 32*32. For stair environments, the heightmap is sampled in a one-dimensional form at the landing point of the robot animal, and the size of the heightmap is 100. The stair environment considered above is a single direction. If the stairs are of any form, the heightmap can also be in a 2D form. The present invention does not particularly limit the form of heightmaps to be used in various scenes.

[0028] Figure 4 Schematic diagram of the strategy network for augmenting the height map. The present invention uses 16 8x8 filters, 32 4×4 filters, and three convolution layers of 32 4×4 filters to process the height map, and then expands the feature map through a fully connected layer with 64 units, and then connects to fully connected layers with 1024 and 512 units to process the output state. The activation function uses the rectified linear unit function (ReLU) to excite all hidden layers.

[0029] In order to speed up the training, an embodiment of the present invention adopts a progressive learning mechanism, also known as a warm-start strategy, in which the policy network is initialized with a trained model for a flat ground environment without a height map, and then a height map is added for further training.

[0030] In yet another embodiment of the present invention, the environmental parameters include ground surfaces having various friction coefficients.

[0031] In one embodiment of the present invention, the interference parameter includes Gaussian noise applied to the proportional-derivative controller, which is an amplified parameter applied to the input end. In detail, each joint of the robot has a motor and a proportional-derivative (PD) controller. The actuator model of the robot is driven by the position control of the PD controller. In practice, as the current increases, the torque gain (torque) may reach saturation. This is because the linear "torque-current" relationship is only applicable to ideal motors. In order to eliminate the difference in actuator models between virtual animals and robot animals, the present invention multiplies the Gaussian noise by the torque output by the PD controller to simulate the impact of changes in motor power. Through the above-mentioned amplified interference parameters, even if the intensity of the movement is changed, the robot animal may still achieve movements that are close enough to the virtual animal.

[0032] In another embodiment, the interference parameters include the magnitude of the external force applied to a joint, the direction of the force, and the duration. For example, the interference parameters for the back joint J3 of the robot are 50 to 100 Newtons (N), 9 o'clock, and last for 0.5 seconds. These example values ​​can be randomly generated or manually specified, and the present invention is not limited to this. In the real world, the state of the robot is measured by sensors. Considering the mechanical limitations of the joints of the robot and the noise influence of the sensors, it is necessary to amplify the input of the "real" sensor into the controller trained by the present invention. The robustness of a well-trained robot can be measured by simulating the recovery behavior of the robot in response to external force interference in the environment.

[0033] Please refer to Figure 2 . Step S2 is "the policy network generates multiple primitive distributions and first primitive weights based on motion data". In step S2, an embodiment of the present invention adopts the policy network described in the following document, "Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. 2019. MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies. In NeurIPS".

[0034] The policy network in the above document includes two modules: a primitive network and a gating network. The primitive network is used to generate a primitive distribution according to motion data, and the gating network is used to generate a plurality of first primitive weights according to the current state and the target state of the reference animal. In one embodiment, the current state and the reference state are motion data at a specific time point.

[0035] Primitive distribution, or primitive influence, is the basic unit of action. A Gaussian distribution formed by combining multiple primitive distributions and their corresponding first primitive weights in a multiplicative form can make the robot animal present an action such as a target state or a current state.

[0036] Step S3 is "selecting the current state of the reference animal from the motion data and setting the adaptation state of the reference animal". In detail, in the time range included in the motion data, the motion data at a specific time point is designated as the current state. The adaptation state is the state that the reference animal in the current state passes through after responding to environmental parameters and interference parameters. For example, the reference animal has a change in forward speed because it has to dodge falling objects in the environment. The adaptation state is, for example, a sequence reflecting the above-mentioned speed changes.

[0037] Step S4 is "the policy network generates the second primitive weight based on at least the current state, the adaptation state, and the primitive distribution". In detail, step S2 uses the gating network in the policy network to generate the first primitive weight. This step S4 also uses the gating network in the policy network to generate the second primitive weight. The difference is that in step S4, the parameters used to train the gating network further include the adaptation state obtained based on the environmental parameters.

[0038] Step S5 is "the discriminator trains the strategy network based on the comparison result of the first primitive weight and the second primitive weight". In one embodiment of the present invention, the discriminator in the Generative Adversarial Network architecture is used to compare whether the first and second primitive weights are the same. In another embodiment of the present invention, L1 distance or L2 distance and a maximum distance threshold can be used as the discriminator. If the similarity between the first and second primitive weights is within a specified range, the currently generated second primitive weight is retained. Otherwise, return to step S4 and the gating network in the strategy network generates another second primitive weight again.

[0039] After executing the process of steps S1 to S5, the trained policy network can be used as a controller to control the movement of the robot. In one embodiment of the present invention, after step S5, step S6 is further included, and step S6 is "generating an action distribution based on the primitive distribution and the second primitive weight, and the action distribution includes the output momentum of each joint." Step S6 actually issues a command to the robot to perform a specified action through the trained controller.

[0040] In summary, the method for training a motion controller for a robot animal proposed in the present invention trains a controller with robustness by adding augmented parameters applicable to actuator models, mechanical limitations, sensor interference, environmental changes, inference engine delays, and microcontroller delays during the training process. This controller is not only applicable to virtual animals in character animation, but can also directly integrate physical robot animals and use high-level instructions to control the movement of physical robot animals. In the process of training a controller that controls the movement of a virtual animal based on control parameters, the present invention achieves that the controller used to control the movement of a virtual animal can also be used to control the movement of a physical robot animal by augmenting the input or output parameters used for training.

Claims

1. A method for training a motion controller of a robot, applicable to the robot having multiple joints, characterized in that: The method for training a motion controller of a robot comprises: Obtaining movement data, an environmental parameter and a disturbance parameter of a reference animal; Using a strategy network to generate a plurality of primitive distributions and a first primitive weight according to the motion data; Selecting a current state of the reference animal from the motion data and setting an adaptation state of the reference animal; generating a second primitive weight by the strategy network at least according to the current state, the adaptation state, and the primitive distributions, wherein the adaptation state is a state that the reference animal in the current state passes through after responding to the environmental parameter and the interference parameter; and The policy network is trained by a discriminator according to the comparison result of the first primitive weight and the second primitive weight.

2. The method for training a motion controller of a robot according to claim 1, characterized in that: Each of the joints has a motor and a proportional-differential controller, and the disturbance parameter includes Gaussian noise applied to the proportional-differential controller.

3. The method for training a motion controller of a robot according to claim 1, characterized in that: The interference parameter includes an external force applied to one of the joints and a force direction.

4. The method for training a motion controller of a robot according to claim 1, characterized in that: The environment parameters include a height map.

5. The method for training a motion controller of a robot according to claim 1, characterized in that: After the policy network is trained by the discriminator according to the comparison result of the first primitive weight and the second primitive weight, it further includes: generating an action distribution according to the primitive distributions and the second primitive weights, and the action distribution includes the output momentum of the joints.

6. The method for training a motion controller of a robot according to claim 1, characterized in that: The motion data includes the current state and a historical state, the historical state is stored in an input queue; and after obtaining the motion data of the reference animal, the environmental parameter and the interference parameter, further includes: Store the current state to the input queue.

Images