A text-driven method for generating digital human motion

Abstract

The present invention discloses a text-driven digital human action generation method. When constructing a training data set, the various body parts of the human body are decoupled, and a corresponding text description unit is generated for each body part. Different texts describing the same action are constructed by randomly selecting text description units, thereby alleviating the influence of the local action combination bias existing in previous data sets, and helping the model to better associate the text with its corresponding local body action; the action encoder and action decoder in the human action vector quantization autoencoder both adopt the structure of a graph convolutional network, which more effectively characterizes the connection relationship of the human skeleton. Human motion data should actually be a topological graph structure rather than grid data. Therefore, compared with convolutional neural networks, graph convolutional neural networks more effectively utilize the topological structure of the human body to extract the intrinsic connection between joints. The modal conversion model benefits from the use of a local semantic reinforcement loss function during training.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a text-driven digital human action generation method. Background Art

[0002] 3D digital humans have a wide range of applications in games and movies. In recent years, using voice and text to drive digital humans to complete related actions has become a hot topic in the field of digital human action generation. Users can use a text description of human body movements to drive digital humans to complete corresponding actions. The full-body movements of digital humans include local movements of multiple parts of the human body, which are specifically expressed in the rotation angles of various joints of the human body. The text usually contains a description of the full-body movements of the digital human and detailed descriptions of various parts of the human body. Using text to drive digital humans to complete corresponding actions provides a new idea for human-computer interaction.

[0003] The current mainstream text-driven action generation methods can be divided into action generation based on shared latent space and action generation based on diffusion model according to the model.

[0004] However, there are two problems with the existing methods:

[0005] 1) The whole-body integrated motion encoding it uses cannot effectively preserve the local details of the motion;

[0006] 2) The existing datasets have obvious local action combination bias, that is, some local body movements appear too frequently in the dataset, but they have no correlation in the text. This makes the actions generated by the model not match the text in details. Summary of the invention

[0007] The present invention aims at the problems existing in the prior art and provides a method for generating digital human actions based on text drive, which can generate digital human actions matching the detailed description of the text.

[0008] The specific technical solution adopted by the present invention is:

[0009] The present invention discloses a method for generating digital human actions based on text drive, comprising:

[0010] Training phase:

[0011] Obtain the digital human's action data and corresponding text description;

[0012] Based on the motion data of digital humans, human motion vector quantization autoencoder is trained;

[0013] Based on the digital human's motion data, the corresponding text description, and the human motion vector quantization autoencoder, the modal conversion model is trained using the local semantic enhancement loss function.

[0014] Reasoning stage:

[0015] The text description of the desired digital human action is used as the input of the modal conversion model to obtain the probability distribution of the action encoding of each body part of the digital human;

[0016] The motion of the digital human is generated by sampling from the probability distribution of the motion encoding of each body part using the human motion vector quantization autoencoder.

[0017] As a further improvement, the method of obtaining the digital human's motion data and the corresponding text description in the present invention specifically includes:

[0018] Generate corresponding text description units according to the movements of each body part of the digital human;

[0019] Different descriptions of the same action data are generated by randomly selecting some text description units and changing the splicing order.

[0020] As a further improvement, the human motion vector quantization autoencoder described in the present invention is used to construct a motion coding library for various parts of the human body, and generate digital human motion according to the motion coding; the human motion vector quantization autoencoder consists of three parts: a motion encoder, a quantization module, and a motion decoder;

[0021] The action encoder is a graph convolutional neural network, whose input is a joint graph. The nodes of the joint graph correspond to the joint points of the human body, storing the axis-angle representation of the rotation of the human joints. The edges of the joint graph correspond to the bones connecting the joint points of the human body. The node features in the output joint graph are the extracted joint motion features.

[0022] The graph convolutional neural network consists of multiple layers of graph convolutional layers, and its specific operation is expressed as:

[0023] ;

[0024] in Represents input features middle The eigenvalues ​​of the nodes, express Node weights, output features Represents output features middle The eigenvalue of the node;

[0025] is the set of nodes that need to be traversed, given by the spatial distance threshold Sure:

[0026] ;

[0027] Where: Representation Node and nodes The distance between

[0028] Finally, the joints are divided into six groups according to the body parts: head, left hand, right hand, torso, left leg, and right leg. The motion features of the joints in each group are concatenated and then the motion features of the body parts are obtained using the fully connected layer.

[0029] The quantization module includes six action coding libraries corresponding to the head, left hand, right hand, torso, left leg, and right leg of the human body, converting action features into action codes;

[0030] The action decoder is a graph neural network structure, whose input is a joint graph. The nodes of the joint graph correspond to the joints of the human body, storing the action codes of the corresponding parts of the human joints. The edges of the joint graph correspond to the bones connecting the joints of the human body. The node features in the output joint graph are the axis angle representations of the joint rotation.

[0031] As a further improvement, the modal conversion model described in the present invention is used to convert text into the probability of encoding in the action encoding library of each part of the human body. The modal conversion model is a neural network with a transformer structure, the input is text, and the output is the probability distribution of six action codes, which respectively represent the probability of each action code in the action encoding library of the human head, left hand, right hand, torso, left leg, and right leg of the human action vector quantization autoencoder;

[0032] The local semantic enhancement loss function is a body part weighted loss function, which is expressed as:

[0033] ;

[0034] in represents the mode conversion model prediction The group text corresponds to The action codes of each body part are grouped into the corresponding part action code library. The probability of a group of codes, Represents the weight of the body part. If and only if the action description text involves the corresponding body part and the corresponding action code matches the action description text, the weight is 1, otherwise it is 0; represents the number of samples, Indicates the number of action codes in the action code library.

[0035] As a further improvement, the method of sampling from the probability distribution of the motion codes of each body part and using the human motion vector quantization autoencoder to generate the motion of the digital human includes:

[0036] Sampling from the probability distribution of the action codes of each body part to obtain the action codes of each body part;

[0037] Assign action codes to nodes of the joint graph according to body parts;

[0038] The joint graph and the motion decoder of the human motion vector quantization autoencoder are used to generate the axis-angle representation of joint rotations.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1) In the present invention, the various body parts of the human body are decoupled when constructing a training data set, and a corresponding text description unit is generated for each body part. Different texts describing the same action are constructed by randomly selecting text description units. That is, the text description of the action in the data set does not include the action description of all body parts of the whole body. This processing method can alleviate the impact of the local action combination bias existing in previous data sets, and help the model better associate the text with its corresponding local body action.

[0041] 2) In the present invention, both the motion encoder and the motion decoder in the human motion vector quantization autoencoder adopt the structure of graph convolutional network, which can more effectively represent the connection relationship of the human skeleton. Human motion data should actually be a topological graph structure rather than grid data. Therefore, compared with convolutional neural networks, graph convolutional neural networks can more effectively utilize the topological structure of the human body and extract the intrinsic connection between joints.

[0042] 3) In the present invention, the human body motion vector quantization autoencoder explicitly divides the human body parts into six parts: head, left hand, right hand, torso, left leg, and right leg. It can effectively extract and retain more motion detail information, and decouple the whole body motion of the human body into local motions of each body part, thereby improving the accuracy of associating text and motion.

[0043] 4) In the present invention, the modality conversion model benefits from the use of a local semantics-enhanced loss function during training, which focuses on the body parts mentioned in the text when calculating the loss. The local semantics-enhanced loss function can help the model understand the text requirements more effectively and establish the correct connection between the action description of the local body parts in the text and the corresponding joints, thereby generating digital human actions that better match the given text. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of the training phase of the algorithm for generating digital human actions using texts in the present invention;

[0045] Figure 2 It is a flow chart of the reasoning phase of the algorithm for generating digital human actions using texts in the present invention;

[0046] Figure 3 It is a schematic diagram of the present invention generating digital human actions based on text. DETAILED DESCRIPTION

[0047] The technical solution of the present invention is further described below by combining the accompanying drawings of the specification through specific embodiments:

[0048] The purpose of the present invention is to solve the problem that the existing method of generating digital human actions using text is difficult to generate actions that match the text details, and propose a digital human action generation method based on text drive;

[0049] The present invention includes two modules: a human motion vector quantization autoencoder and a modal conversion model, and the process can be divided into two stages: training and reasoning. Figure 1 is a flow chart of the training phase of the algorithm for generating digital human actions using text in the present invention; wherein the algorithm training process is as follows Figure 1 As shown, corresponding to steps one to three, Figure 2 It is a flow chart of the reasoning phase of the algorithm for generating digital human actions using texts in the present invention;

[0050] The reasoning stage Figure 2 As shown, it includes step four and step five.

[0051] The specific implementation method of the present invention is as follows:

[0052] Step 1: Obtain the motion data of the digital human, design rules for converting local motions of each body part into text units, and design different rules for the head, elbows, hands, torso, knees, and feet.

[0053] The head rule determines whether the face is facing left or right based on the deflection angle of the head, and determines the state of lowering or raising the head based on the pitch angle of the head; the deflection angle and pitch angle of the head correspond to the z-axis and y-axis components of the Euler angle of the neck joint rotation, respectively. If the deflection angle is greater than 10°, the face is judged to be facing left, less than -10°, the face is judged to be facing right, and between -10° and 10°, the face is ahead. A pitch angle greater than 30° is judged to be lowering the head, less than -30° is to be raising the head, and between these angles, the head is kept level.

[0054] The elbow and knee rules can be divided into full bending, right-angle bending, partial bending and no bending according to the bending degree of the elbow and knee, corresponding to bending angles greater than 120°, 60° to 120°, 30° to 60° and less than 30° respectively. The bending angle is the modulus length corresponding to the rotation of the knee joint expressed in the form of axis angle.

[0055] The hand rule is a quantitative rule based on relative position relationship. It searches for the body part closest to the hand based on the distance. If the distance is less than a given threshold, the hand is judged to be placed next to the corresponding body part. Otherwise, the hand is judged to be in the corresponding direction of the corresponding body part based on the spatial position relationship, for example, the left hand is behind the hip. When judging the spatial position relationship between the hand and other body parts, first find the body part closest to the hand, then based on the relative position relationship between the center of the hand and the center of the corresponding body part, fill in the corresponding body part and the relative position relationship into "The hand is in...of..." to generate the corresponding text description.

[0056] The torso rule determines the body orientation and the forward and backward tilt of the torso. The body orientation is determined by the chest orientation and is divided into eight directions: front, left front, left side, left rear, rear, right rear, right side, and right front. The forward and backward tilt of the torso is determined by the difference in the pitch angle between the chest orientation and the abdomen orientation. The chest orientation is the normal orientation corresponding to the plane composed of the left shoulder, right shoulder and chest joints. The abdomen orientation is the orientation of the x direction in the local coordinate system of the abdomen joint in the world coordinate system. The z-axis component of the Euler angle of the chest orientation is between -22.5° and 22.5°, corresponding to the front, 22.5° to 67.5° to the left front, 67.5° to 112.5° to the left side, 112.5° to 157.5° to the left rear, 157.5° to 202.5° to the rear, 202.5° to 247.5° to the right rear, 247.5° to 292.5° to the right side, and 292.5° to 337.5° to the right front. The difference between the y-axis components of the Euler angles of the chest orientation and the abdomen orientation is between -20° and 20°, corresponding to the torso keeping straight, less than -20° is backward, and greater than 20° is forward.

[0057] The foot rule includes the distance quantification rule to determine whether the feet are in contact with the ground and the relative position rule of the feet and other parts of the body; the distance quantification rule is the distance from the foot to the ground. If the distance is less than 5cm, the foot is in contact with the ground, otherwise the foot is off the ground. The relative position rule of the feet and other parts of the body is the relative position relationship between the foot and the hip in the x direction. If the x coordinate of the foot center is less than the hip center, the foot is extended forward, otherwise the foot is extended backward.

[0058] After using the rules to obtain the text description units of each body part, multiple text description units are obtained by randomly selecting some body parts, and a coherent action description is generated using connectives and a pre-trained large language model. By selecting different body parts and different connectives, different texts describing the same action can be generated to achieve data enhancement; according to the rules of head, elbow, hand, torso, knee and foot, the descriptions of the head, left hand, right hand, torso, left leg and right leg are obtained respectively, where the description of the left hand includes the description of the left hand and elbow, the description of the right hand includes the description of the right hand and elbow, the description of the left leg includes the description of the left knee and left foot, and the description of the right leg includes the description of the right knee and right foot; when the body parts are randomly selected, 6 numbers between 0 and 1 are randomly generated, corresponding to the six body parts respectively. If the number is less than 0.3, the corresponding body part is not used to generate the final action description. Then the text descriptions of the selected body parts are randomly sorted by body parts, and the sentences are connected with "and", "while" and "as well as". The prompt words "Help me improve the following human body description in terms of language coherence" are used to prompt the language model to output the improved text description.

[0059] Step 2: Based on the motion data of the digital human, train the human motion vector quantization autoencoder.

[0060] The human motion vector quantization autoencoder consists of three parts: motion encoder, quantization module, and motion decoder;

[0061] The action encoder is a graph convolutional neural network, whose input is a joint graph. The nodes of the joint graph correspond to the joint points of the human body, and store the axis-angle representation of the rotation of the human joints. The edges of the joint graph correspond to the bones connected between the joint points of the human body. The node features in the output joint graph are the extracted joint motion features. The joint graph is a graph structure data that represents the connection relationship of human joints and contains joint rotation information. It is composed of nodes and edges. The edges correspond to the bones connected between the joints, and the nodes store the joint rotation information. It is the specific form of the action data in step 1.

[0062] The graph convolutional neural network consists of multiple layers of graph convolutional layers, and its specific operation can be expressed as:

[0063]

[0064] in Represents input features middle The eigenvalues ​​of the nodes, express Node weights, output features Represents output features middle The eigenvalue of the node;

[0065] is the set of nodes that need to be traversed, given by the spatial distance threshold Sure:

[0066]

[0067] Where: Representation Node and nodes The distance between.

[0068] Finally, the joints are divided into six groups according to the body parts: head, left hand, right hand, torso, left leg, and right leg. The motion features of the joints contained in each group are concatenated and then the fully connected layer is used to obtain the motion features of the body parts.

[0069] The quantization module includes six action coding libraries corresponding to the head, left hand, right hand, torso, left leg, and right leg of the human body, which converts action features into action codes. The output of the quantization module is the action code with the shortest distance to the action feature in Euclidean space;

[0070] The action decoder is a graph neural network structure, whose input is a joint graph. The nodes of the joint graph correspond to the joints of the human body, storing the action codes of the corresponding parts of the human joints. The edges of the joint graph correspond to the bones connecting the joints of the human body. The node features in the output joint graph are the axis angle representation of the joint rotation.

[0071] During training, the motion encoder, quantization module and motion decoder are trained together, and the loss function of the training process is defined as:

[0072]

[0073] in The input digital human action, is the action encoder, is the action decoder, The action code in the quantization module action code library that is closest to the action coder output in Euclidean space, represents the cut-off gradient, is the weight coefficient of the first term in the loss. The first term supervises the action encoder so that the action features encoded by it are as close as possible to the existing action codes in the action code library of the quantization module. The second term is used to update the action codes in the six body parts action code library in the quantization module to make them close to the action features extracted by the action encoder and reduce the quantization error. The third term calculates the error between the action generated by the action decoder using the action code and the original action.

[0074] The entire model can be trained end-to-end via back-propagation.

[0075] Step 3: Based on the digital human’s motion data, the corresponding text description, and the human motion vector quantization autoencoder, the modal conversion model is trained using the local semantic enhancement loss function. The modal conversion model is a transformer-structured neural network whose input is the text description of the action, which is encoded into a text vector using the pre-trained CLIP text encoder. In the forward process of the transformer, the cross-attention mechanism is used to complete the fusion of the motion modality and the text modality, where the motion feature is used as the query and the text vector is used as the key and value. Finally, the fully connected layer and the softmax function are used to generate the probability of the action coding in the action coding library of each body part.

[0076] During training, the loss function is defined as:

[0077]

[0078] in represents the mode conversion model prediction The group text corresponds to The action codes of each body part are grouped into the corresponding part action code library. The probability of a group of codes. Represents the weight of the body part. If and only if the action description text involves the corresponding body part and the corresponding action code matches the action description text, the weight is 1, otherwise it is 0. represents the number of samples, Indicates the number of action codes in each action code library.

[0079] Step 4: Use the text description of the desired digital human action as the input of the modal conversion model to obtain the probability distribution of the action encoding of each body part of the digital human. The specific process includes:

[0080] The text description of the desired digital human action is encoded into a text vector using the pre-trained CLIP text encoder. The text vector is used as the key and value of the cross-attention mechanism in the forward process of the modality conversion model. Finally, the fully connected layer and softmax function are used to generate the probability of the action coding in the action coding library of each body part.

[0081] Step 5: Use the greedy sampling strategy to sample from the probability distribution of the motion encoding of each body part output by the modal conversion model, and construct the input joint graph of the human motion vector quantization autoencoder. Specifically, for each body part, select the motion encoding with the highest probability as the sampling result. In the constructed joint graph, the nodes correspond to the joints of the human body, the edges represent the bones connected between the joints, and the features of the joints are the motion encodings of the body parts to which the joints belong. The joint graph is used as the input of the human motion vector quantization autoencoder, and the encoder is used to obtain a human joint graph whose joint features are the rotation angles of the joints.

[0082] Figure 3 is a schematic diagram of the present invention generating digital human actions based on text, Figure 3 (1) is a text-driven digital human action using "A person kneels on his right leg. His torso is bent forward and his back is arched. His arms are slightly bent and his hands are touching the ground. His left thigh is horizontal. His left knee is at a right angle and located in front of his torso, and his left foot is on the ground. His left calf is vertical. His right knee is bent and his right foot is touching the ground behind his torso. His head is looking down and forward." Figure 3 Middle (2) is the digital human action generated using the text "The person is in a forward leaning posture with the right leg stretched out and the toes on the ground, and the other leg bent with the knee towards the chest. The arms are slightly bent and placed vertically on both sides of the left leg. The torso is tilted forward and the back is arched. The head is in line with the spine and looking forward." Figure 3 (3) is the digital human action generated using the text "He runs forward with his left leg forward, knee bent and foot flat on the ground. His left elbow is slightly bent and his left hand is placed on the left side of his body. His right knee is bent nearly 90 degrees and his right calf is flat. His right foot is located behind his torso. His right arm is bent and his hand is placed on the right side of his left knee. His torso is bent forward on his left leg and his back is arched." Figure 3 Middle (4) is the use of the text "The person is running with his upper body straight. His right elbow is bent at a right angle and his right hand is close to his right hip. His left knee is bent, close to a 90-degree angle, and his left calf is flat. His left foot is located behind his torso. His right knee is bent forward, close to a 90-degree angle, and his foot is flat on the ground. His left arm is bent at a right angle and extended forward, with his left forearm kept horizontal. His head is facing forward." to drive the generated digital human movements.

[0083] The above is not a limitation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several changes, modifications, additions or substitutions can be made without departing from the essential scope of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A text-driven digital human action generation method, characterized in that: include: Training phase: Obtain the digital human's action data and corresponding text description; Based on the motion data of digital humans, human motion vector quantization autoencoder is trained; Based on the digital human's motion data, the corresponding text description, and the human motion vector quantization autoencoder, the modal conversion model is trained using the local semantic enhancement loss function. Reasoning phase: The text description of the desired digital human action is used as the input of the modal conversion model to obtain the probability distribution of the action encoding of each body part of the digital human; Sampling from the probability distribution of motion encoding of each body part, and using human motion vector quantization autoencoder to generate the motion of the digital human; The human motion vector quantization autoencoder is used to construct a motion coding library for various parts of the human body, and generate digital human motion according to the motion coding; the human motion vector quantization autoencoder is composed of three parts: a motion encoder, a quantization module, and a motion decoder; The action encoder is a graph convolutional neural network, whose input is a joint graph, the nodes of the joint graph correspond to the joint points of the human body, and the axis angle representation of the rotation of the human joint is stored. The edges of the joint graph correspond to the bones connected between the joint points of the human body, and the node features in the output joint graph are the extracted joint motion features; The graph convolutional neural network consists of multiple layers of graph convolutional layers, and its specific operation is expressed as follows: f out (x)=∑ v∈B(x) f in (v)w(v); where f in (v) represents the input feature f in The feature value of node v, w(v) represents the weight of node v, and the output feature f out (x) represents the output feature f out The eigenvalue of the x node in the B(x) is the set of nodes that need to be traversed, which is determined by the given spatial distance threshold D: B(x) = {v|d(v,x)≤D}; Where: d(v, x) represents the distance between node v and node x; Finally, the joints are divided into six groups according to the body parts: head, left hand, right hand, torso, left leg, and right leg. The motion features of the joints in each group are concatenated and then the motion features of the body parts are obtained using the fully connected layer. The quantization module includes six action coding libraries corresponding to the head, left hand, right hand, trunk, left leg and right leg of the human body, and converts the action features into action codes; The action decoder is a graph neural network structure, whose input is a joint graph. The nodes of the joint graph correspond to the joints of the human body, and the action codes of the corresponding parts of the human joints are stored. The edges of the joint graph correspond to the bones connected between the joints of the human body. The node features in the output joint graph are the axis angle representations of the joint rotations. The modal conversion model is used to convert text into the probability of encoding in the action encoding library of each part of the human body. The modal conversion model is a neural network with a transformer structure, with text as input and six action encoding probability distributions as output, which respectively represent the probability of each action encoding in the action encoding library of the human head, left hand, right hand, torso, left leg and right leg of the human action vector quantization autoencoder; The local semantic enhancement loss function is a body part weighted loss function, which is expressed as: Where P i，j，b represents the probability that the modal conversion model predicts that the action encoding of the bth group of body parts corresponding to the i-th group of text is the j-th group of encoding in the corresponding body part action encoding library, m i，j，b Represents the weight of the body part. If and only if the action description text involves the corresponding body part and the corresponding action code matches the action description text, the weight is 1, otherwise it is 0. N represents the number of samples, and M represents the number of action codes in the action code library.

2. The text-driven digital human motion generation method according to claim 1, characterized in that: The obtaining of the digital human's motion data and the corresponding text description specifically includes: Generate corresponding text description units according to the movements of each body part of the digital human; Different descriptions of the same action data are generated by randomly selecting some text description units and changing the splicing order.

3. The text-driven digital human motion generation method according to claim 1, characterized in that: Sampling from the probability distribution of the motion encoding of each body part, using the human motion vector quantization autoencoder to generate the motion of the digital human, including: Sampling from the probability distribution of the action codes of each body part to obtain the action codes of each body part; assigning the action codes to each node of the joint graph according to the body part; The joint graph and the motion decoder of the human motion vector quantization autoencoder are used to generate the axis-angle representation of joint rotations.

Images