Human body action generation method based on diffusion model and fine-grained text description

Abstract

The invention discloses a human body action generation method based on a diffusion model and fine-grained text description. The method comprises the following steps: converting an original text description into fine-grained descriptions of different body parts by utilizing a large language model; guiding a diffusion model based on Transform by using fine-grained description, and generating a human body action conforming to the description; in the diffusion model, coding the overall description into a global token, coding the description of different body parts into a plurality of local tokens, and jointly guiding the generation process; through the combination of a global token and a local token, the model pays attention to global and detailed information at the same time, and an action which better conforms to description is generated. According to the method, zero-sample human body actions exceeding the range of an original data set can be generated, and the method has good generalization ability.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing and computer vision, and in particular to a method for generating human motion based on a diffusion model and fine-grained text description. Background Art

[0002] Human motion generation is an important research field in computer vision and computer graphics, which aims to simulate and generate realistic human motion through computers. With the development of technologies such as virtual reality, augmented reality, and movie special effects, the demand for high-quality human motion continues to increase. In recent years, the task of human motion generation has become more and more challenging, and quite a number of innovative methods and technologies have emerged to meet this challenge. Among them, deep generative models play an important role in the field of human motion generation, including generative adversarial networks, variational autoencoders, and diffusion models.

[0003] However, existing human motion generation methods still face some challenges, especially the difficulty in zero-shot generation. Specifically, existing methods do not work well when generating complex motions outside the distribution range of the training data. In addition, the limited scale and diversity of human motion datasets make it more challenging to generate complex motions outside the distribution range of the dataset. Therefore, in order to enhance the generalization ability of the model and achieve zero-shot generation, it is necessary to propose a new solution. Summary of the invention

[0004] The purpose of the present invention is to provide a method for generating human motion based on a diffusion model and fine-grained text description. When only a limited motion capture dataset is available, by utilizing fine-grained descriptions of different body parts, the model can better learn the essence of motion and thus generate motion beyond the distribution range of the original dataset.

[0005] The technical solution to achieve the purpose of the present invention is: a method for generating human body movements based on a diffusion model and fine-grained text description, comprising the following steps:

[0006] Step 1: Use a large language model to perform fine-grained paraphrasing of the original text describing human actions to obtain fine-grained text descriptions of various parts of the body;

[0007] Step 2: Use the pre-trained text encoder to encode the fine-grained text description to generate a global token and multiple local tokens;

[0008] Step 3: In the diffusion model, the global token and multiple local tokens are input into the Transformer-based diffusion model to guide the model to denoise the noisy human motion;

[0009] Step 4: During model training, human body geometry loss is introduced, including position loss, foot contact loss, and speed loss, to generate more natural human body movements;

[0010] Step 5: After training the model, remove the noise through the back diffusion process to obtain the final clean motion.

[0011] Furthermore, the large language model described in step one is a ChatGPT-3.5 or ChatGPT-4 model; the prompt used is: "Convert the given action description into an action description of each body part, expressed in only one paragraph; the available body parts include ['arm', 'leg', 'torso', 'neck', 'hip', 'waist']".

[0012] Furthermore, the pre-trained text encoder in step 2 is a CLIP model, which encodes the entire fine-grained text description into a global token GL, and the encoding formula is: GL = CLIP{text}, where text represents the overall fine-grained text description; the text descriptions of different body parts are encoded into local tokens PT respectively. 1:k , the coding formula is: PT i =CLIP{part_text_i}, where part_text_i represents the text description of the i-th body part, i=1...k, and k represents the number of divided body parts.

[0013] Furthermore, the model in step 3 needs to denoise the human action sequence, and the data is represented as: Where n is the length of the motion sequence, t is the time step of the diffusion model, and the action representation of each frame is x i Including the joint rotation and position of the human skeleton; the denoising generation network G adopts the basic Transformer encoder architecture, and the model outputs a clean human action sequence Then add noise t-1 times to get the human action sequence that is only denoised once Where c is the text condition (GL, PT 1:k ); the training loss function is:

[0014]

[0015] in, represents a set of The real action sequence sampled in , t~[1,T] represents a time point randomly sampled from time steps 1~T, and T represents the total time steps of the diffusion model.

[0016] Furthermore, the position loss used in step 4 is:

[0017]

[0018] The foot contact loss is:

[0019]

[0020] The speed loss is:

[0021]

[0022] in, represents the pose of the i-th frame of the real action sequence, represents the pose of the i-th frame of the generated action sequence, FK() represents the forward kinematics function that converts joint rotation into joint position, and f u ∈{0,1} represents the binary foot contact mask.

[0023] Furthermore, in step five, a diffusion step size of 1000 is used. Under the guidance of the text condition, the model denoises the Gaussian noise t times and obtains a human action sequence that conforms to the text description.

[0024] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the computer program is loaded into the processor.

[0025] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0026] A computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0027] Compared with the prior art, the present invention has the following significant advantages: (1) The present invention uses a divide-and-conquer strategy to split the entire action into actions of different body parts, and associates them with related actions in the training data set, which helps the generative model to better learn the essence of the action; (2) The present invention uses a large language model to convert short and vague text descriptions into fine-grained descriptions of different body parts, thereby expanding the data set and improving the generalization ability of the model in the zero-sample case. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a complete technical solution flow chart of the present invention.

[0029] Figure 2 This is a model architecture diagram of the present invention.

[0030] Figure 3 It is a diagram showing the effects of the present invention. DETAILED DESCRIPTION

[0031] Currently, there are the following problems with existing text generation action methods:

[0032] 1) Existing datasets are limited in size and lack diversity, making it challenging to directly generate motions beyond the data distribution and the generalization ability of motion generation models is insufficient.

[0033] 2) Existing methods do not have a deep understanding that complex motion is composed of simple motions of various parts of the body, resulting in poor performance when faced with unprecedented complex motion descriptions.

[0034] In view of the above problems, the present invention proposes to adopt a diffusion model and fine-grained text description to solve the above problems.

[0035] A method for generating human actions based on a diffusion model and fine-grained text description comprises the following steps:

[0036] Step 1: Use a large language model to perform fine-grained paraphrasing of the original text describing human body movements to obtain fine-grained text descriptions of various parts of the body;

[0037] Step 2: Encode the fine-grained text description using a pre-trained text encoder to generate a global token and multiple local tokens;

[0038] Step 3, in the diffusion model, the global token and multiple local tokens are input into the Transformer-based diffusion model to guide the model to denoise the noisy human motion;

[0039] Step 4: During the model training process, human body geometry loss is introduced, including position loss, foot contact loss, and speed loss, to generate more natural human motions.

[0040] Step 5: After training the model, the noise is eliminated through the back diffusion process to obtain the final clean motion.

[0041] First, when there is only a limited motion capture dataset, there is a way to associate textual descriptions of motion outside the dataset with motion within the dataset. Although the motion of the whole body may be out of the distribution of the dataset, the motion of specific body parts may be represented in the dataset. Therefore, a divide-and-conquer strategy can be adopted to associate the motion of different body parts with the motion of the corresponding parts in the dataset by re-annotating the fine-grained descriptions of these body parts.

[0042] For example, a vague description "a man walks very depressed". can be restated as "his arms hang heavily at his sides. His legs move slowly, with short, weak steps..." In this vague unseen action description, the leg movement may appear in "a man walks slowly and aimlessly", which is included in the dataset. The arm movement may appear in "his arms hang heavily at his sides", which is included in the dataset. On the one hand, adopting fine-grained descriptions enables the model to understand in detail how unseen movements are performed. On the other hand, re-annotating the actions of different body parts allows the model to better learn the essence of the actions. Using fine-grained text descriptions, our goal is to improve the generalization ability of the model.

[0043] Then, based on these fine-grained descriptions, we design a fine-grained human motion diffusion model. Specifically, we replace the original simple and ambiguous text with fine-grained descriptions of individual body parts generated by a large language model to guide the Transformer-based diffusion model.

[0044] The various steps of the present invention are described in detail below with reference to the accompanying drawings.

[0045] like Figure 1 As shown in FIG, the method for generating human actions based on the diffusion model and fine-grained text description includes the following steps:

[0046] Step 1: Use a large language model to perform fine-grained paraphrasing of the original text describing human actions to obtain fine-grained text descriptions of various body parts.

[0047] The large language model used is OpenAI's ChatGPT-3.5. ChatGPT is a dialogue model based on a large language model that can participate in natural conversations and generate corresponding responses. ChatGPT's answers are usually directly affected by the information and expressions provided in the prompt. If the prompt provides a clear and detailed question or description, ChatGPT can usually provide relevant and accurate answers. However, if the prompt is too simple, ambiguous or unclear, ChatGPT may produce unexpected responses or unclear expressions. For this task, the present invention carefully designs an effective prompt through experimental verification. The prompt used is: "Convert the given action description into an action description of each body part, expressed in only one paragraph. Available body parts include ['arm', 'leg', 'torso', 'neck', 'hip', 'waist'], and the following are some examples: [Q...A...]. Question: []". [Q...A...] is four examples of manually designed Q&A pairs. The following is one of the four examples: Question: A person kicks something or someone with his left leg. Answer: His left leg extends with force as he kicks something or someone. His arms are held steady at his sides. His torso is slightly twisted. His hip and waist muscles are contracted. His neck remains neutral.

[0048] With prompts, ChatGPT-3.5 is used to perform fine-grained paraphrasing of the original text descriptions in the HumanML3D and KIT datasets, generating 44,970 fine-grained descriptions for HumanML3D and 6,353 fine-grained descriptions for KIT.

[0049] The key to the first step is to use the powerful reasoning ability of the large language model to convert the original simple text description into a more specific and detailed description of each part of the body to provide richer information and provide better conditions for the subsequent generation model.

[0050] Step 2: Encode the fine-grained text description using a pre-trained text encoder to generate a global token and multiple local tokens.

[0051] The fine-grained text description is encoded using a pre-trained text encoder CLIP, which is a multimodal pre-trained model that can map images and text into the same semantic space. The entire fine-grained text description is input into the CLIP text encoder and encoded into a global token GL. This global token can capture the semantic information of the entire text description, thereby providing the model with an overall understanding of the entire text description. The text related to different body parts in the fine-grained text description is input into the CLIP text encoder separately and encoded into multiple local tokens. These local tokens can capture the semantic information related to each body part, thereby providing the model with an understanding of different body parts.

[0052] Step 3: In the diffusion model, the global token and multiple local tokens are input into the Transformer-based diffusion model to guide the model to denoise the noisy human motion.

[0053] like Figure 2 , showing the model architecture diagram of the present invention.

[0054] The diffusion model consists of a forward process and a backward process, both of which follow a Markov chain. The forward process involves adding noise. The input is the original motion from the data distribution. The output is the action with Gaussian noise added t times When enough noise is added, the action Can be close to Gaussian distribution The reverse process aims to reduce Gaussian noise In the denoising process, in the diffusion step t, part of the noise is eliminated, and a motion image with less noise is obtained. This step is repeated iteratively until the noise is completely eliminated, generating a clean motion

[0055] The present invention adopts a simple Transformer encoder architecture to implement the network G. Different from the above-mentioned traditional diffusion model, the present invention directly predicts the clean motion Instead of predicting the noise added at each time step, the input to G is obtained by adding the noise t times to the original motion The noisy motion Noisy Movement Together with the text condition tokens GL, PT 1:k It is input into the Transformer encoder together with the time step t, and outputs the clean motion One reason for directly predicting the clean motion at each time step of the diffusion model is to incorporate the human geometry loss during the training of the network to make the generated human motion more natural. For each sampling step t, from T to 1, our model predicts the clean motion Then add the noise back again After T iterations, the final clean motion is obtained This form of diffusion model has been widely adopted.

[0056] Step 4: During model training, human body geometry losses are introduced, including position loss, foot contact loss, and speed loss, to generate more natural human motions.

[0057] The position loss adopted is:

[0058]

[0059] The role of position loss is to ensure that the positions of the key points generated by the model are as close as possible to the key points in the real data. This helps the model learn the correct limb positions and generate actions that conform to human kinematics.

[0060] The foot contact loss is:

[0061]

[0062] The purpose of foot contact loss is to ensure that the moment the foot contacts the ground during walking, running, and other actions complies with the laws of physics and prevents the foot from sliding. This helps the model learn the correct way for the foot to contact the ground during dynamic motion and avoids generating unreasonable suspension or sliding states.

[0063] The speed loss is:

[0064]

[0065] The purpose of speed loss is to ensure the temporal coherence of the motion generated by the model. It helps the model generate smooth and physically correct motion trajectories, avoiding abrupt motions or unnatural speed changes.

[0066] Among them, FK() represents the forward kinematics function that converts joint rotation into joint position, f i ∈{0,1} represents the binary foot contact mask.

[0067] Step 5: After training the model, remove the noise through the back diffusion process to obtain the final clean motion.

[0068] like Figure 3As shown, an example of the action sequence results corresponding to the input text generated by the present invention using a trained diffusion model to denoise the randomly sampled Gaussian noise is shown. The order of the timeline is represented from right to left, showing multiple frames of a person walking forward in two postures. By retelling the input text description in a fine-grained manner and describing the details of the actions of different body parts, the present invention can more accurately understand and generate actions outside the distribution of the data set. For example, for the description of "a person walking happily", the action generated by the present invention is that the arms swing freely, the legs stride forward vigorously, and the neck is upright and comfortable. For the description of "a person walking frustratedly", the action generated is that the arms are heavy and drooping, the legs move slowly and weakly in small steps, and the neck is drooping.

[0069] The present invention verifies the indicators on the HumanML3D dataset, HuMMan dataset, and kungfu dataset. The HuMMan dataset and the kungfu dataset are evaluated under the condition of zero samples. As shown in Table 1, Table 2, and Table 3, compared with recent methods, the present method has better generalization.

[0070] Table 1 shows the quantitative results on the HumanML3D dataset. The model is trained on the HumanML3D training set and then evaluated on the HumanML3D test set.

[0071] Table 1

[0072]

[0073] Table 2 shows the quantitative results on the HuMMan dataset. In the zero-shot setting, the model is trained on the HumanML3D training set and then directly evaluated on the HuMMan dataset.

[0074] Table 2

[0075]

[0076] Table 3 shows the quantitative results on the kungfu dataset. In the zero-shot setting, the model is trained on the HumanML3D training set and then directly evaluated on the kungfu dataset.

[0077] Table 3

[0078]

[0079] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for generating human actions based on a diffusion model and fine-grained text description, characterized in that: The following steps are involved: Step 1: Use a large language model to perform fine-grained paraphrasing of the original text describing human body movements to obtain fine-grained text descriptions of various parts of the body; Step 2: Use the pre-trained text encoder to encode the fine-grained text description to generate a global token and multiple local tokens; Step 3: In the diffusion model, the global token and multiple local tokens are input into the Transformer-based diffusion model to guide the model to denoise the noisy human motion; Step 4: During model training, human body geometry loss is introduced, including position loss, foot contact loss, and speed loss, to generate more natural human body movements; Step 5: After training the model, remove the noise through the back diffusion process to obtain the final clean motion.

2. The method for generating human motion based on diffusion model and fine-grained text description according to claim 1, characterized in that: The large language model described in step 1 is a ChatGPT-3.5 or ChatGPT-4 model; the prompt used is: "Convert the given action description into an action description of each body part, expressed in only one paragraph; the available body parts include ['arm', 'leg', 'torso', 'neck', 'hip', 'waist']".

3. The method for generating human motion based on diffusion model and fine-grained text description according to claim 1, characterized in that: The pre-trained text encoder in step 2 is the CLIP model, which encodes the entire fine-grained text description into a global token GL. The encoding formula is: GL = CLIP{text}, where text represents the overall fine-grained text description; the text descriptions of different body parts are encoded into local tokens PT respectively. 1:k , the coding formula is: PT i =CLIP{part_text_i}, where part_text_i represents the text description of the i-th body part, i=1...k, and k represents the number of divided body parts.

4. The method for generating human motion based on diffusion model and fine-grained text description according to claim 1, characterized in that: In step 3, the model needs to denoise the human action sequence, and the data is represented as: Where n is the length of the motion sequence, t is the time step of the diffusion model, and the action representation of each frame is x i Including the joint rotation and position of the human skeleton; the denoising generation network G adopts the basic Transformer encoder architecture, and the model outputs a clean human action sequence Then add noise t-1 times to get the human action sequence that is only denoised once Where c is the text condition (GL, PT 1:k ); the training loss function is: in, represents a set of The real action sequence sampled in , t~[1,T] represents a time point randomly sampled from time steps 1~T, and T represents the total time steps of the diffusion model.

5. The method for generating human motion based on diffusion model and fine-grained text description according to claim 1, characterized in that: The position loss used in step 4 is: The foot contact loss is: The speed loss is: in, represents the pose of the i-th frame of the real action sequence, represents the pose of the i-th frame of the generated action sequence, FK() represents the forward kinematics function that converts joint rotation into joint position, and f i ∈{0,1} represents the binary foot contact mask.

6. The method for generating human motion based on diffusion model and fine-grained text description according to claim 1, characterized in that: In step 5, a diffusion step size of 1000 is used. Under the guidance of the text condition, the model denoises the Gaussian noise t times and obtains a human action sequence that conforms to the text description.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

Images