A text-conditioned human action generation method based on discrete diffusion model
By using the action quantization variational autoencoder of the self-attention layer and the lightweight discrete diffusion model in the action generation method, the problem of difficult to take into account in the prior art generation accuracy and diversity is solved, and the action generation effect of high precision, high diversity and low parameter quantity is achieved.
Patent Information
- Application Number
- CN202410782924.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-18
- Publication Date
- 2025-03-07
- Estimated Expiration
- 2044-06-18
AI Technical Summary
The existing action generation method based on text conditions is difficult to balance between generation accuracy and diversity, and the model parameters are large and the reasoning time is long.
Using a motion quantized variational autoencoder based on the self-attention layer and a lightweight discrete diffusion model, a high-precision and high-diversity 3D key point motion sequence is generated through pre-training and distillation technology.
Improve the accuracy and diversity of action generation, reduce model parameters and inference time, and improve the usability of the model in actual deployment.
Smart Images
Figure CN118644895B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of artificial intelligence and relates to a text-conditional human motion generation method based on a discrete diffusion model. Background Art
[0002] With the development of humanoid robots, virtual human technology, VR and AR technology, the demand for key point motion generation in various fields is increasing. This increase in demand not only comes from the game and entertainment industry, but also extends to education, medical care, security and other fields. In various fields, the key point motion generation task of the human body plays a vital role. The basic purpose of the motion generation task is to generate a human key point motion sequence with visual coherence and physical rationality constraints. The input conditions of traditional motion generation tasks are usually separate category labels, such as running, standing, etc. For natural language texts such as "a person lies down and then stands up" and its corresponding key point motion sequence, it is difficult to map the data of two different modalities to each other. The authenticity of the motions generated by the existing methods needs to be improved, and the existing models usually have a large number of parameters and a long reasoning time.
[0003] At present, the action generation methods based on text conditions are mainly divided into two categories: one is to use quantized variational autoencoders and autoregressive generative models (Transformer) to generate actions in discrete state space; the other is to use a diffusion model to directly add noise and denoise the continuous motion noise to gradually restore the action. Among them, the former method generates actions with high accuracy and good quality, but because the training data set text and actions are in a one-to-one form, the diversity of generated actions is poor; while the latter supplements the data set in the process of gradually adding noise, which improves the diversity of generated actions. However, due to the randomness of noise in the continuous state space, the accuracy of generated actions is poor. At the same time, although the two methods improve the quality of generated actions compared with traditional methods, the parameters required by the model are larger and the reasoning time is longer. Therefore, it is urgent to propose a more effective method for generating key point actions under the condition of natural language text input, so as to provide a more effective method for generating robot and virtual human actions according to natural language instructions in the future, and a more realistic method for generating actions. Summary of the invention
[0004] The purpose of the present invention is to provide a text-conditional human motion generation method based on a discrete diffusion model, aiming to solve the problems of poor generation accuracy of existing models, large number of model parameters, and long model inference speed in the current environment.
[0005] The present invention provides a text-conditioned human action generation method based on a discrete diffusion model, comprising:
[0006] Step 1: Obtain a human 3D key point dataset;
[0007] Step 2: Use the human 3D key point dataset to pre-train the motion quantization variational autoencoder, and use the trained motion quantization variational autoencoder to encode the motion latent variable;
[0008] Step 3: Use the human 3D key point dataset and action latent variables to train the lightweight discrete diffusion model;
[0009] Step 4: Input the given text condition into the text encoder to generate a text feature vector, input the text feature vector into the trained lightweight discrete diffusion model to obtain the action latent variable, input the action latent variable into the trained action quantized variational autoencoder, and decode to obtain the 3D key point motion sequence information corresponding to the given text condition.
[0010] The text-conditioned human action generation method based on the discrete diffusion model of the present invention has at least the following beneficial effects:
[0011] Compared with the prior art, the present invention adopts a motion quantization variational autoencoder based on a self-attention layer to better learn the distribution characteristics of 3D key point motion sequence information; a discrete diffusion model is used to generate 3D key point motion sequence information under text conditional input, which improves the model accuracy while improving the diversity of motion sequence information generated by the model; the denoising process allows the model to better learn the intrinsic relationship between action latent variables under different confusion levels, so that the model can restore noisy action latent variable sequences with different confusion levels to their corresponding action latent variable sequences under given text conditions; the present invention adopts a heavy parameter distillation loss for distilling discrete diffusion models, which retains the generation accuracy of the model as much as possible while distilling the model, and further reduces the model parameters and inference time. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 It is a flow chart of a text-conditioned human action generation method based on a discrete diffusion model of the present invention;
[0013] Figure 2 It is the structural diagram of the action quantized variational autoencoder;
[0014] Figure 3 It is a schematic diagram of the denoising and denoising process of the lightweight discrete diffusion model;
[0015] Figure 4 It is a schematic diagram of the denoising network structure. DETAILED DESCRIPTION
[0016] like Figure 1 As shown, a text-conditioned human action generation method based on a discrete diffusion model of the present invention comprises:
[0017] Step 1: Obtain a human 3D key point dataset;
[0018] In specific implementation, the human body 3D key point dataset can be obtained by the following methods:
[0019] (1) Input the human action video into the pre-trained 3D posture estimation network to extract the 3D key point action sequence information of the corresponding action; manually annotate the 3D key point action sequence information, mark the text conditions corresponding to the action sequence, and obtain the 3D key point dataset;
[0020] (2) Use public datasets for network human pose estimation: including the Human3.6M dataset, the MPI-INF-3DHP dataset, or the CMU Panoptic dataset; manually annotate the dataset, mark the text conditions corresponding to its action sequences, and obtain a 3D key point dataset;
[0021] (3) Use network text-conditioned human motion synthesis datasets as 3D key point datasets: including the HumanML3D dataset or the KIT Motion-Language dataset.
[0022] Step 2: Use the human 3D key point dataset to pre-train the motion quantization variational autoencoder, and use the trained motion quantization variational autoencoder to encode the motion latent variable;
[0023] The motion quantized variational autoencoder includes a motion latent variable encoder, a motion latent variable decoder and a motion latent variable codebook. The motion latent variable encoder is responsible for encoding continuous 3D key point motion sequence information into motion latent variables; the motion latent variable decoder is responsible for restoring motion latent variables into 3D key point motion sequence information; and the motion latent variable codebook is responsible for collecting motion latent variables corresponding to the input 3D key point motion sequence information.
[0024] like Figure 2 As shown, the action latent variable encoder includes: a first convolution layer, multiple first feature extraction layers and a second convolution layer; the first feature extraction layer includes a third convolution layer, a first ReLU activation layer, a downsampling layer and a first self-attention layer. The action latent variable decoder includes: a fourth convolution layer, multiple second feature extraction layers and a fifth convolution layer; the second feature extraction layer includes a second self-attention layer, a sixth convolution layer, a second ReLU activation layer and an upsampling layer.
[0025] The step 2 is specifically as follows:
[0026] Step 2.1: Substitute the 3D key point action sequence information P in the human 3D key point dataset into 1 ,p 2,…,p n} Obtain the encoding vector set through the action latent variable encoder encoding
[0027]
[0028] Step 2.2: In the action latent variable codebook Find and encode vector sets in The encoding vector l in i The action latent variable of the closest distance
[0029]
[0030] in, represents the i-th encoding vector, n L =n / s, s is the downsampling rate, which is determined by the convolution step size in the downsampling layer and the number of downsampling layers, d L is the dimension of the encoding vector; the action latent variable set is represented as
[0031] Step 2.3: Input the action latent variable into the action latent variable decoder to obtain the restored 3D key point action sequence information.
[0032] Step 2.4: Using the gradient pass-through method in the back-propagation process, the loss function of the action quantized variational autoencoder is:
[0033]
[0034] In the formula, To reconstruct the position loss, is the reconstruction speed loss, α is the reconstruction speed loss weight, sg[] is the gradient stop operation, and β is the partial loss weight of the action latent variable decoder;
[0035]
[0036] in, The feature vector representing the i-th 3D key point action sequence information includes the position coordinates, velocity and acceleration of each key point, and d represents the dimension of the feature vector; the restored 3D key point action sequence information is expressed as The feature vector representing the restored action sequence information of the i-th 3D key point.
[0037] Step 2.5: Update the network parameters of the motion quantized variational autoencoder according to the loss function of the motion quantized variational autoencoder until the network parameters converge or the maximum number of iterations is reached.
[0038] Step 3: Using the human 3D key point data set and action latent variables to train the lightweight discrete diffusion model, the step 3 is specifically as follows:
[0039] Step 3.1: Encode the 3D key point action sequence information into action latent variables through the trained action quantization variational autoencoder to obtain each action latent variable The corresponding action latent variable index sequence in the action latent variable codebook x i ∈{1,2,...,n c},n c is the total number of action latent variables in the action latent variable codebook.
[0040] Step 3.2: For the noise state, define an additional noise variable M = n c +1, given the total number of noise adding steps T and the noise adding probability at each moment;
[0041] Define the noise probability β at time t t Expressed as: action latent variable index sequence X at time t-1 t-1 All indices x in which the noise is not present i,t-1 The probability q(x) of converting to noise variable M at time t i,t =M|x i,t-1 ≠M)=β t , the action latent variable index sequence of the noise-added state at time t after processing is x i,t ∈{1,2,...,n c ,n c +1}; After the noise addition process with the number of steps T is completed, the action latent variable index sequence at time t = 0 Transformed into noise state X T ={M,M,…,M}.
[0042] Step 3.3: Input the text condition into the pre-trained CLIP text encoder to generate a text feature vector.
[0043] Step 3.4: Through the teacher denoising network, according to the text feature vector generated in step 3.3 and the action latent variable index X of the noisy state obtained in step 3.2 t Predict the action latent variable index sequence corresponding to the text feature vector and obtain the predicted action latent variable index like Figure 3 The figure shows a schematic diagram of the noise addition and denoising process.
[0044] Step 3.5: In each batch of training data, update the parameters of the teacher denoising network by stochastic gradient descent according to the loss function of the following teacher denoising network until the loss function value of the teacher denoising network is small;
[0045]
[0046] In the formula, Represents about x i,0 and The cross entropy loss function corresponding to the distribution, [x i,t ]=M is marked as all noise state variables at time t, i∈[0,n L ] is the position index of the current variable at the current moment in the action latent variable index sequence.
[0047] Step 3.6: Get the trained teacher denoising network through the method in step 3.5 After that, the student denoising network with fewer parameters will be obtained through network distillation. The student denoising network has the same architecture as the teacher denoising network but has fewer network layers. Some teacher denoising network parameters are used to initialize the student network parameters. Through the student denoising network, the text feature vector generated in step 3.3 and the action latent variable index X of the noise state obtained in step 3.2 are used. t Predict the action latent variable index sequence corresponding to the text feature vector, obtain the predicted action latent variable index, and update the parameters θ of the student denoising network according to the following student denoising network loss function 1 ;
[0048]
[0049] In the formula, is the true label loss function, α 1 is the weight of the true label loss ratio; is the output of the student denoising network, is the output of the teacher denoising network, τ is the distillation temperature coefficient, for The corresponding soft label category distribution, for The jth item in Its soft label category is defined as:
[0050]
[0051] for The corresponding soft label category distribution, for The jth item in Its soft label category is defined as:
[0052]
[0053] Step 3.7: Complete the training of the lightweight discrete diffusion model.
[0054] In specific implementation, the teacher denoising network and the student denoising network both include an encoding layer, multiple self-attention layers and a linear mapping layer; the student denoising network has the same architecture as the teacher denoising network but has fewer network layers. Figure 4 shown.
[0055] Step 4: Input the given text condition into the text encoder to generate a text feature vector, input the text feature vector into the trained lightweight discrete diffusion model to obtain the action latent variable, input the action latent variable into the trained action quantized variational autoencoder, and decode to obtain the 3D key point motion sequence information corresponding to the given text condition.
[0056] The present invention adopts a motion quantization variational autoencoder based on a self-attention layer to enhance the learning ability of the network. Since the training of the motion quantization variational autoencoder is performed on a large number of datasets that do not require annotation and the datasets are relatively easy to obtain, the method adopted can achieve better results under the premise of collecting datasets at low cost.
[0057] The present invention adopts a lightweight discrete diffusion model to generate corresponding action latent variables. The diffusion process in the discrete state space can enable the model to inherit the learning results of the action quantization variational autoencoder to ensure the high accuracy of the generated actions. At the same time, the noise variables introduced in the diffusion process can ensure the high diversity of the actions generated by the model.
[0058] The present invention adopts shared parameter layers and network distillation to reduce model parameters and inference time, reduces the delay time of the model in actual deployment, and improves the availability of the model.
[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the concept of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A text-conditioned human action generation method based on a discrete diffusion model, characterized in that: include: Step 1: Obtain a human 3D key point dataset; Step 2: Use the human 3D key point dataset to pre-train the motion quantization variational autoencoder, and use the trained motion quantization variational autoencoder to encode the motion latent variable; Step 3: Use the human 3D key point dataset and action latent variables to train the lightweight discrete diffusion model; Step 4: Input the given text condition into the text encoder to generate a text feature vector, input the text feature vector into the trained lightweight discrete diffusion model to obtain the action latent variable, input the action latent variable into the trained action quantized variational autoencoder, and decode to obtain the 3D key point motion sequence information corresponding to the given text condition; The step 3 is specifically as follows: Step 3.1: Encode the 3D key point action sequence information into action latent variables through the trained action quantization variational autoencoder to obtain each action latent variable The corresponding action latent variable index sequence in the action latent variable codebook x i ∈{1,2,...,n c },n c is the total number of action latent variables in the action latent variable codebook; Step 3.2: For the noise state, define an additional noise variable M = n c +1, given the total number of noise adding steps T and the noise adding probability at each moment; Define the noise probability β at time t t Expressed as: action latent variable index sequence X at time t-1 t-1 All indices x in which the noise is not present i,t-1 The probability q(x) of converting to noise variable M at time t i,t =M|x i,t-1 ≠M)=β t , the action latent variable index sequence of the noise-added state at time t after processing is x i,t ∈{1,2,...,n c ,n c +1}; After the noise addition process with the number of steps T is completed, the action latent variable index sequence at time t = 0 Transformed into noise state X T ={M,M,…,M}; Step 3.3: Input the text condition into the pre-trained CLIP text encoder to generate a text feature vector; Step 3.4: Through the teacher denoising network, according to the text feature vector generated in step 3.3 and the action latent variable index X of the noisy state obtained in step 3.2 t Predict the action latent variable index sequence corresponding to the text feature vector and obtain the predicted action latent variable index Step 3.5: In each batch of training data, update the parameters of the teacher denoising network by stochastic gradient descent according to the loss function of the following teacher denoising network until the loss function value of the teacher denoising network is small. The loss function of the teacher denoising network is: In the formula, Represents about x i,0 and The cross entropy loss function corresponding to the distribution, [x i,t ]=M is marked as all noise state variables at time t, i∈[0,n L ] is the position index of the current variable at the current moment in the action latent variable index sequence; Step 3.6: Get the trained teacher denoising network through the method in step 3.5 Finally, a student denoising network with fewer parameters is obtained through network distillation. The student denoising network has the same architecture as the teacher denoising network but has fewer network layers. Some teacher denoising network parameters are used to initialize the student model parameters. Through the student denoising network, the text feature vector generated in step 3.3 and the action latent variable index X of the noise state obtained in step 3.2 are used. t Predict the action latent variable index sequence corresponding to the text feature vector, obtain the predicted action latent variable index, and freeze the teacher denoising network parameters and update the student denoising network parameters θ1 according to the following student denoising network loss function. The student denoising network loss function is: In the formula, That is the true label loss function, α1 is the weight of the true label loss ratio; is the output of the student denoising network, is the output of the teacher denoising network, τ is the distillation temperature coefficient, for The corresponding soft label category distribution, for The jth item in Its soft label category is defined as: for The corresponding soft label category distribution, for The jth item in Its soft label category is defined as: Step 3.7: Complete the training of the lightweight discrete diffusion model.
2. The text-conditioned human action generation method based on discrete diffusion model as claimed in claim 1, characterized in that: In step 1, a human 3D key point dataset is obtained by: (1) Input the human action video into the pre-trained 3D posture estimation network to extract the 3D key point action sequence information of the corresponding action; manually annotate the 3D key point action sequence information, mark the text conditions corresponding to the action sequence, and obtain the 3D key point dataset; (2) Use public datasets for network human pose estimation: including the Human3.6M dataset, the MPI-INF-3DHP dataset, or the CMU Panoptic dataset; manually annotate the dataset, mark the text conditions corresponding to its action sequences, and obtain a 3D key point dataset; (3) Use network text-conditioned human motion synthesis datasets as 3D key point datasets: including the HumanML3D dataset or the KIT Motion-Language dataset.
3. The text-conditioned human action generation method based on discrete diffusion model as claimed in claim 1, characterized in that: The action quantized variational autoencoder includes an action latent variable encoder, an action latent variable decoder and an action latent variable codebook; the action latent variable encoder is responsible for encoding continuous 3D key point action sequence information into action latent variables; the action latent variable decoder is responsible for restoring the action latent variables into 3D key point action sequence information; The action latent variable codebook is responsible for collecting the action latent variables corresponding to the input 3D key point action sequence information; The action latent variable encoder includes: a first convolution layer, a plurality of first feature extraction layers and a second convolution layer; the first feature extraction layer includes a third convolution layer, a first ReLU activation layer, a downsampling layer and a first self-attention layer; The action latent variable decoder includes: a fourth convolutional layer, multiple second feature extraction layers and a fifth convolutional layer; the second feature extraction layer includes a second self-attention layer, a sixth convolutional layer, a second ReLU activation layer and an upsampling layer.
4. The text-conditioned human action generation method based on discrete diffusion model as claimed in claim 3, characterized in that: The step 2 is specifically as follows: Step 2.1: Substitute the 3D key point action sequence information P = {p1, p2, ..., p n } Obtain the encoding vector set through the action latent variable encoder encoding Step 2.2: In the action latent variable codebook Find and encode vector sets in The encoding vector l in i The action latent variable of the closest distance in, represents the i-th encoding vector, n L =n / s, s is the downsampling rate, which is determined by the convolution step size in the downsampling layer and the number of downsampling layers, d L is the dimension of the encoding vector; the action latent variable set is represented as Step 2.3: Input the action latent variable into the action latent variable decoder to obtain the restored 3D key point action sequence information; Step 2.4: Using the gradient pass-through method in the back-propagation process, the loss function of the action quantized variational autoencoder is: In the formula, To reconstruct the position loss, is the reconstruction speed loss, α is the reconstruction speed loss weight, sg[] is the gradient stop operation, and β is the partial loss weight of the action latent variable decoder; in, The feature vector representing the i-th 3D key point action sequence information includes the position coordinates, velocity and acceleration of each key point, and d represents the dimension of the feature vector; the restored 3D key point action sequence information is expressed as The feature vector representing the restored i-th 3D key point action sequence information; Step 2.5: Update the network parameters of the motion quantized variational autoencoder according to the loss function of the motion quantized variational autoencoder until the network parameters converge or the maximum number of iterations is reached.
5. The text-conditioned human action generation method based on discrete diffusion model as claimed in claim 1, characterized in that: The teacher denoising network and the student denoising network both include an encoding layer, multiple self-attention layers and a linear mapping layer; the student denoising network has the same architecture as the teacher denoising network but has fewer network layers.