Method for training virtual animals to move based on control parameters

Abstract

A method for training a virtual animal to move based on control parameters includes an imitation learning stage and a control adaptation stage. The imitation learning stage includes: obtaining the first and second momentums, current and target states of a reference animal associated with the virtual animal when performing an action, a primitive network analyzing the first and second momentums to generate a plurality of primitive distributions, and training a first gating network to generate a first primitive weight based on the current state and the primitive distribution to convert from the current state to the target state. The control adaptation stage includes: obtaining a control parameter group, training a second gating network to generate a second primitive weight based on the current state and the primitive distribution to convert from the current state to a combination of the current state and the control parameter group, and a discriminator generating a discrimination result based on the first and second primitive weights, and updating the second gating network.

Description

Technical Field

[0001] The invention relates to action synthesis and character animation, in particular to a method for training a virtual animal to move based on control parameters. Background Art

[0002] Over the past few decades, the quality of character animation in animation, video games, and digital special effects has greatly improved due to new tools and techniques developed by researchers in the field. Among various types of characters, quadrupeds are particularly different in style, rhythm, and gait patterns, making them difficult to animate. For real-time applications such as video games, the characters need to react dynamically to the environment, which further complicates the problem.

[0003] Motion graphs are used to synthesize new animations from motion capture data, where nodes represent actions defined from the motion capture data and edges represent transitions between actions. However, the process of annotating nodes and their transitions is long and cumbersome, and a large amount of motion capture data is required to support different gait patterns. Motion graphs become large and complex in dynamic environments, given the interactions between the agent and its surroundings. Even so, motion graphs are still not applicable to scenarios that have never been seen before.

[0004] Kinematic controllers allow users to control the subject to produce the desired motion. Kinematic controllers are designed to mimic motion datasets, and when the subject needs to interact with the environment in a dynamic environment that cannot be seen, such as a scene where a quadruped walks on a smooth and undulating boat, the subject generated by the kinematic controller cannot respond naturally. It is obviously impractical to collect or manually design sufficient reference motion datasets to train motion controllers. Although physics-based controllers can effectively model complex phenomena, because physical simulations enable the subject to produce meaningful reactions to external disturbances without collecting or animating these reactions. However, physical constraints such as gravity, friction, and collisions still have many difficulties in designing physics-based controllers. Summary of the invention

[0005] In view of this, the present invention proposes a method for training a virtual animal to move based on control parameters, thereby solving the problems of the above-mentioned traditional approach.

[0006] According to an embodiment of the present invention, a method for training a virtual animal to move based on control parameters includes an imitation learning stage and a control adaptation stage. The imitation learning stage includes: obtaining a first momentum, a second momentum, a current state, and a target state; analyzing the first momentum and the second momentum with a primitive network to generate a plurality of primitive distributions; and training a first gated network to generate a first primitive weight according to the current state and the primitive distributions to convert from the current state to the target state; wherein the first momentum is a reference animal performing a first type of action, the second momentum is the reference animal performing a second type of action, the reference animal is associated with the virtual animal, and the current state and the target state are two sampled actions of the reference animal that are continuous in time. The control adaptation stage includes: obtaining a control parameter set; training a second gating network to generate a second primitive weight according to the current state and the primitive distributions to convert from the current state to a combination of the current state and the control parameter set; and using a discriminator to generate a discrimination result according to the first primitive weight and the second primitive weight; and updating the second gating network according to the discrimination result; wherein the discrimination result is used to retain the second primitive weight or to use the second gating network to generate another second primitive weight according to the current state and the primitive distributions to convert from the current state to a combination of the current state and the control parameter set.

[0007] In summary, the virtual quadruped established according to the method of training virtual animals based on the motion of control parameters proposed in the present invention can naturally respond to high-order control instructions in a dynamic physical environment. In the imitation learning stage, the present invention captures the natural motion perceived in the animation clip; in the control adaptation stage, the high-order instructions are mapped to the motion distribution corresponding to the animation using a generative adversarial network; in the fine-tuning stage, the controller is further fine-tuned through deep reinforcement learning so that it can recover from external interference and produce smooth and natural movements. A navigation module can be attached to the controller of the virtual animal established according to the present invention, so that the virtual animal can autonomously perform tasks such as getting out of a maze. Overall, the method proposed in the present invention involves motion synthesis tasks in a dynamic environment, and has the two major advantages of natural motion images and high-order command control.

[0008] The above description of the contents of the present disclosure and the following description of the embodiments are used to demonstrate and explain the spirit and principle of the present invention, and to provide a further explanation of the scope of the patent application of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a schematic diagram of a virtual animal;

[0010] Figure 2 is a flow chart of an embodiment of the present invention;

[0011] Figure 3It is a detailed flow chart of the imitation learning stage S1;

[0012] Figure 4 is a detailed flow chart of the control adaptation phase S2; and

[0013] Figure 5 It is a detailed flow chart of the fine-tuning stage S3.

[0014] The reference numerals are as follows:

[0015] J1~J20…Joints

[0016] S1~S3…stage

[0017] S11~S14, S21~S24, S31~S34...steps DETAILED DESCRIPTION

[0018] The detailed features and characteristics of the present invention are described in detail in the following embodiments, and the contents are sufficient to enable any person skilled in the art to understand the technical content of the present invention and implement it accordingly, and according to the contents disclosed in this specification, the scope of the patent application and the drawings, any person skilled in the art can easily understand the relevant concepts and characteristics of the present invention. The following examples are to further illustrate the viewpoints of the present invention in detail, but are not intended to limit the scope of the present invention in any way.

[0019] The present invention provides a method for training a virtual animal to move based on control parameters, wherein the virtual animal includes a plurality of joints, such as Figure 1 shown. Figure 1 A four-legged virtual animal with 20 joints J1-J20 is shown, each joint can be equipped with a virtual motor. The present invention is used to generate a controller for controlling the movement of the virtual animal, the controller provides a rotational momentum to each virtual motor, so that the virtual motor at that location generates torque, thereby driving the virtual animal to move based on the control parameters.

[0020] Figure 2 The flowchart of an embodiment of the present invention includes three main stages: an imitation learning stage S1, a control adaptation stage S2 and a fine-tuning stage S3. Figure 3 It is a detailed flowchart of the imitation learning stage S1.

[0021] Step S11 is "obtaining a first momentum, a second momentum, a current state, and a target state". In one embodiment, each of the first momentum and the second momentum includes the following measurement data: position, speed, rotation, and angular velocity. Except for the rotation which is represented by a 4-dimensional quaternion, the other three are represented by a 3-dimensional vector. Step S11 is used to obtain momentum data and state data of a reference animal during movement. The reference animal is a real version of a virtual animal, such as a dog. One implementation of step S11 is to set a plurality of sensors on the body of an animal in the real world to collect measurement data. Another implementation of step S11 is as follows: momentum data and state data are obtained through physics engine simulation. The present invention does not particularly limit the manner in which momentum data and state data are obtained in step S11.

[0022] Regarding the first and second momentums, in detail, for each joint of the reference animal, data of the first momentum is collected when the reference animal performs a first type of action, and data of the second momentum is collected when the reference animal performs a second type of action. The first type of action and the second type of action last for an interval time, for example, 10 seconds. In short, the first and second momentums correspond to two types of actions of the same joint of the reference animal. An example of the first and second types of actions are: walking and running. Another example is: trotting at a speed of 1.5 meters per second and cantering at a speed of 3 meters per second. The momentum obtained in step S11 includes at least two types of actions, such as the first momentum and the second momentum, but the present invention does not limit the upper limit of the number of types of actions. In other words, the third, fourth... momentum of the reference animal can be obtained according to demand.

[0023] Regarding the current state and the target state, in detail, the current state and the target state represent two sampling postures of the reference animal that are continuous in time. In terms of data structure, each of the current state and the target state is the same as the first momentum or the second momentum. In other words, each of the current state and the target state also includes data such as position, speed, rotation amount and angular velocity; and the difference is that the current state and the target state only include measurement data sampled at two consecutive time points, while the first momentum and the second momentum each include measurement data for a period of time. In one embodiment, the current state and the target state can be selected from the first or second momentum. For example: the first momentum includes a trotting action of 10 seconds in length, then the current state and the target state can be the measurement data at the 3rd and 4th seconds.

[0024] Step S12 is "the primitive network analyzes the first momentum and the second momentum to generate multiple primitive distributions", and step S13 is "training the first gating network to generate the first primitive weight according to the current state and the primitive distribution", thereby converting from the current state to the target state. The primitive network and the gating network are two modules of a policy network. For the implementation of this policy network, please refer to this document: "Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. 2019. MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies. In NeurIPS" The present invention does not describe the implementation methods of "the primitive network generates the primitive distribution" and "the first gating network generates the first primitive weight".

[0025] Primitive distribution, or primitive influence, is the basic unit of action. In one embodiment, the primitive distribution generated by the primitive network P is φ1...φ k Each primitive distribution φ i The model consists of a Gaussian distribution with a state-depedent action mean μ i (s t ) and a diagonal covariance matrix Σ i , as shown in the following mode 1. The present invention particularly adopts a fixed diagonal covariance matrix, thereby avoiding premature convergence (prematrue convergence) caused by modifying Σ during the training process of the primitive network.

[0026] φ i =N(μ i (s t ),Σ i ),i=1,2...k (Formula 1)

[0027] The combination of one or more primitive distributions can make the virtual animal present a specified action. Step S13 is to generate one or more primitive weights for combining the one or more primitive distributions using a first gating network, and the weights are shown in the following method 2.

[0028] w=G low (s t ,c low ) (Formula 2)

[0029] Where w is the first primitive weight, w∈R k , k is the number of primitive distributions. G low is the first gating network, s t is the current state, c low is the target state, Contains joint-level information such as position, velocity, rotation, and angular velocity. t represents the state at the tth second, c low Represents the status at t+1 second and t+2 second.

[0030] Step S14 is "generating an action distribution according to the primitive distribution and the first primitive weight". The first primitive weight and the primitive distribution can be combined into a composite distribution in a multiplication form, as shown in Formula 3. The composite distribution shown in Formula 3 is also a Gaussian distribution. This action distribution can be used as a control instruction to control the joints of the virtual animal.

[0031]

[0032] Where Z(s t ,c t ) is the normalization function, c t is the control objective, c t =c low According to formula 3, the next action distribution a of the virtual animal can be sampled. t+1 .

[0033] In one embodiment of the present invention, the imitation learning stage S1 further includes three reward functions of posture, speed and center of mass, as shown in Equation 4, Equation 5 and Equation 6.

[0034]

[0035]

[0036]

[0037] The posture reward function R of formula 4 P By calculating the actual orientation q of the jth joint of the virtual animal j The quaternion difference θ between the target orientation and the target state is used to encourage the controller to match the target state.

[0038] The speed reward function R of Formula 5 V Calculate the difference in joint velocities, and They represent the current angular velocity and target angular velocity of the j-th joint of the virtual animal respectively.

[0039] The center of mass reward function R of formula 6 com The current center of mass p used to avoid the virtual animal c Deviation from target center of mass Provide discouragement.

[0040] In one embodiment of the present invention, it is characterized by using the reward function R of the contact point C Replace the reward function of the end effector as shown in Equation 7.

[0041]

[0042] in, Represents the logical exclusive-OR (XOR) operation, p e is the Boolean contact state of the end effector of the virtual animal, and e belongs to one of "left front, right front, left rear, right rear". The reward function R of Formula 7 C It is used to prevent the gait pattern of the virtual animal from deviating from the target state and helps solve the problem of foot sliding. For example, when only the left front end effector touches the ground, it can be marked as p = [1, 0, 0, 0]. λc in Formula 7 represents a hyperparameter that controls the slope of the exponential function. In one embodiment of the present invention, λc = 5 can produce the best results. The final form of the reward function is shown in Formula 8.

[0043] R=0.65R p +0.1R v +0.1R com +0.15R c (Formula 8)

[0044] In order to reduce the complexity of training, the present invention calculates the target state of each control target separately. Therefore, the physics-based controller obtained in the imitation learning stage of the present invention can imitate the given target state by learning the corresponding primitive distribution and primitive weight, and generate natural movement to execute the gait style described by the target state.

[0045] In one embodiment, step S14 may be omitted and the control adaptation phase S2 may be directly entered.

[0046] Figure 4 is a detailed flow chart of the control adaptation stage S2, which includes steps S21 to S24. Step S21 is "obtaining a control parameter set". The control parameter set can be obtained by the target state C low It is deduced that the control parameter group includes the speed and heading of the virtual animal. high =(σ,Δθ), where σ represents the target speed of the virtual animal, and Δθ represents the angle difference between the current direction and the target direction. For example, if the virtual animal is specified to move forward at 1 meter per second and 90 degrees counterclockwise, the control parameter group is c high =(1,0.5π).

[0047] Step S22 is "training the second gating network to generate second primitive weights according to the current state and primitive distribution". In detail, when the target state is replaced by the control parameter group, the second gating network needs to learn the mapping relationship between the high-order instructions and the primitive weights, thereby converting from the current state to the combination of the current state and the control parameter group. It should be noted that traditional error functions such as L1 distance or L2 distance will only retain low-order distributions, and cannot guarantee that samples are obtained from the correct action distribution. Therefore, the present invention uses the generator in the Generative Adversarial Network as the second gating network in this step S22. Given an action distribution w from the real world real ~G low (s t ,c low ) The first primitive weight w extracted real , the second gating network G in step S22 high Produced from w fake ~G high (s t ,c high ) The second primitive weight w extracted fake .

[0048] Step S23 is "the discriminator generates a discrimination result according to the first primitive weight and the second primitive weight". In one embodiment of the present invention, the second gating network and the discriminator are both modules in the generative adversarial network architecture. The discriminator D maximizes the adversarial loss function L defined by the following method 9: adv To produce a discrimination result.

[0049]

[0050] L rec =||w fake -wreal ||1 (Formula 10)

[0051] L G =λ adv *L adv +λ rec *L rec (Formula 11)

[0052] The reconstruction error (reconstruction loss) L of formula 10 rec The absolute value distance between the first primitive weight and the second primitive weight is calculated. In one embodiment of the present invention, the second gating network G is trained by minimizing the objective function defined by equation 11. high . Through the loss function L adv , the discriminator D can determine whether the generated second primitive weights are retainable (real) or not retainable (fake), so that the second gating network learns the real data distribution. rec The value of is 100, and λ adv The value of is 1. Jointly minimize L rec and L adv The two loss functions can make the second primitive weights generated by the second gating network more similar to the actual first primitive weights generated in the imitation learning stage S1.

[0053] Step S24 is "updating the second gating network according to the discrimination result". If the discriminator determines that the second primitive weight is available, the second primitive weight is retained. Otherwise, the second gating network generates another second primitive weight according to the current state and primitive distribution, and then the discriminator determines whether this other second primitive weight can make the virtual animal transform from the current state to the combination of the current state and the control parameter group. The method of generating another second primitive weight is, for example, generated in a random manner, which is not limited by the present invention.

[0054] After step S24 is completed, another action distribution can be combined based on one or more primitive distributions and the second primitive weights generated by the updated second gating network to control each joint of the virtual animal. This other action distribution is similar to the action distribution generated in step S14, except that this other action distribution further has criteria obtained based on the generative adversarial network. The composition method of the other action distribution is the same as that described in step S14.

[0055] In one embodiment of the present invention, only the imitation learning stage S1 and the control adaptation stage S2 are completed to realize a control adapter for a virtual animal that can move based on high-level instructions. In another embodiment of the present invention, a fine-tuning stage S3 is further included after the control adaptation stage S2. Figure 5This is a detailed flow chart of the fine-tuning stage S3. The fine-tuning stage S3 includes Figure 5 Steps S31 to S34.

[0056] Step S31 is "obtaining an environmental parameter group". Similar to the aforementioned control parameter group, the environmental parameter group also includes the speed and direction of the virtual animal. The difference is that the environmental parameter group reflects a large amount of scene information. In detail, in the control adaptation stage S2, the control parameter group only reflects a small part of the environmental scene. In order to enable the virtual animal to return to a normal state in an unseen scene, it is necessary to further fine-tune the second gating network established in the control adaptation stage S2.

[0057] Step S32 is "training the second gated network to generate the third primitive weight according to the current state, primitive distribution and reward function group", thereby converting from the current state to the adaptation state. The adaptation state is a combination of the current state and the environmental parameter group, and the virtual animal has the adaptation state in response to the environmental parameter group. The reward function group speed reward function and direction reward function described in step S32 are shown in the following formula 12 and formula 13 respectively.

[0058] R spd =exp[-λ spd (σ-||v||) 2 ] (Formula 12)

[0059]

[0060] Speed ​​reward function R spd Calculate the L2 distance between the target speed σ and the current speed ||v||. spd Setting the value to 0.8 gives the best results.

[0061] Direction reward function R head Calculate target direction The cosine similarity between the vector v projected onto the action plane and the current direction of the virtual animal, where Represents the measure of the target direction. The cosine similarity is normalized between 0 and 1.

[0062] Step S33 is "the discriminator generates another discrimination result based on the third primitive weight and the first primitive weight". Step S33 is basically the same as step S23 of the control adaptation stage S2, and will not be repeated here. Step S34 is "at least updating the second gating network based on another discrimination result". In one embodiment of the present invention, the second gating network can be updated only based on another discrimination result. In another embodiment of the present invention, updating the second gating network at least based on another discrimination result includes: updating the second gating network based on another discrimination result and a regularization function. One of the key points of step S34 is: when updating the second gating network, it is prohibited to modify the parameters of each primitive distribution. The second key point of step S34 is: in order to ensure that the controller does not deviate from the action distribution learned in the control adaptation stage S2, the regularization function L shown in the following method 14 is added. reg .

[0063]

[0064] in represents the parameters of the lth fully connected layer of the gated network trained with the generative adversarial network, α l represents the parameters of the lth fully connected layer of the currently trained gating network, and L represents the total number of layers of each gating network. The present invention applies this regularization function to the parameter space rather than the output of the layer, because applying the regularization function to the output of the layer results in additional penalties for real unknown scenarios. When the second gating network is updated in the control adaptation stage S2 and the fine-tuning stage S3, the regularization function L is used. reg , a balance between the two can be achieved without over-modifying the second gating network trained in the adaptation stage S2 in the later performed fine-tuning stage S3.

[0065] After step S34 is completed, another action distribution can be combined according to one or more primitive distributions and the second primitive weights generated by the updated second gating network to control each joint of the virtual animal. This another action distribution is similar to the action distributions generated in steps S14 and S24, except that this another action distribution has criteria obtained based on the generative adversarial network and deep reinforcement learning. The composition of the another action distribution is the same as that described in step S14.

[0066] In summary, the virtual quadruped established according to the method of training virtual animals based on the motion of control parameters proposed in the present invention can naturally respond to high-order control instructions in a dynamic physical environment. In the imitation learning stage, the present invention captures the natural motion perceived in the animation clip; in the control adaptation stage, the high-order instructions are mapped to the motion distribution corresponding to the animation using a generative adversarial network; in the fine-tuning stage, the controller is further fine-tuned through deep reinforcement learning so that it can recover from external interference and produce smooth and natural movements. A navigation module can be attached to the controller of the virtual animal established according to the present invention, so that the virtual animal can autonomously perform tasks such as getting out of a maze. Overall, the method proposed in the present invention involves motion synthesis tasks in a dynamic environment, and has the two major advantages of natural motion images and high-order command control.

Claims

1. A method for training a virtual animal to move based on control parameters, characterized in that: The virtual animal includes a plurality of joints, and the method includes: The first stage of imitation learning includes: Obtaining a first momentum, a second momentum, a current state, and a target state; analyzing the first momentum and the second momentum using a primitive network to generate a plurality of primitive distributions; and A first gating network is trained to generate a first primitive weight according to the current state and the plurality of primitive distributions, so as to transform from the current state to the target state; wherein The first momentum is a first type of action performed by a reference animal, the second momentum is a second type of action performed by the reference animal, the reference animal is associated with the virtual animal, the current state and the target state are two temporally continuous sampled actions of the reference animal; and A control adaptation phase, including: Obtaining a control parameter group; Training a second gating network to generate a second primitive weight according to the current state and the plurality of primitive distributions, so as to transform from the current state to a combination of the current state and the control parameter set; and Using a discriminator to generate a discriminant result according to the first primitive weight and the second primitive weight; and The second gating network is updated according to the discrimination result; wherein The determination result is used to retain the second primitive weight or to generate another second primitive weight by the second gating network according to the current state and the plurality of primitive distributions, so as to convert from the current state to a combination of the current state and the control parameter group.

2. The method for training a virtual animal to move based on control parameters as claimed in claim 1, characterized in that: A fine-tuning phase is further included after the control adaptation phase, and the fine-tuning phase includes: Obtain an environmental parameter group; Training the second gating network to generate a third primitive weight according to the current state, the plurality of primitive distributions and a reward function set to transform from the current state to an adapted state; generating another discrimination result by the discriminator according to the third primitive weight and the first primitive weight; and updating the second gating network at least according to the other determination result; wherein The adaptation state is a combination of the current state and the environmental parameter group, and the virtual creature has the adaptation state in response to the environmental parameter group.

3. The method for training a virtual animal to move based on control parameters as claimed in claim 1, characterized in that: The imitation learning stage further comprises: A motion distribution is generated according to the plurality of primitive distributions and the first primitive weights, and the motion distribution includes output momenta of the plurality of joints.

4. The method for training a virtual animal to move based on control parameters as claimed in claim 1, characterized in that: The control parameter set is derived from the target state, the control parameter set includes a speed and a direction of the virtual animal, and the second gating network and the discriminator belong to a generative adversarial network.

5. The method for training a virtual animal to move based on control parameters as claimed in claim 2, characterized in that: The environmental parameter group includes a speed and a direction of the virtual animal, and the reward function group includes a speed reward function and a direction reward function.

6. The method for training a virtual animal to move based on control parameters as claimed in claim 2, characterized in that: Updating the second gating network at least according to the other discrimination result includes: updating the second gating network according to the other discrimination result and a normalization function.

7. The method for training a virtual animal to move based on control parameters as claimed in claim 1, characterized in that: When updating the second gating network, modifying parameters of each of the plurality of primitive distributions is prohibited.

Images