Human motion generation method in latent space based on diffusion model

Abstract

The present invention provides a method for generating human motion in a latent space based on a diffusion model, comprising: step 1: encoding a selected text prompt condition into a text vector; step 2: weighted averaging each cross-attention map generated by the text prompt condition to generate a target cross-attention map; step 3: inputting the text vector and random Gaussian noise into a pre-trained motion variational autoencoder to obtain an initial human motion sequence; step 4: performing a diffusion denoising operation on the initial human motion sequence in the dimension of the latent space, calculating the KL divergence loss between the target motion sequence and the actual motion sequence and the mean square error between the target cross-attention map and the actual cross-attention map; step 5: by using the KL divergence loss and the mean square error, gradually updating the predicted noise in the denoising step to minimize the total loss and generate a final human motion sequence. The present invention can reduce the computational overhead and can generate a human motion sequence that meets the input conditions.

Description

Technical Field

[0001] The present invention belongs to the technical field of human motion generation in computer vision, and in particular, relates to a method for generating human motion in a latent space based on a diffusion model. Background Art

[0002] Text-driven human motion generation has recently become an emerging research focus due to the semantic richness and user-friendly nature of natural language descriptions.

[0003] However, due to the large difference in the distribution between the two data modes, text data and motion data, it is a challenging task to infer human motion from text descriptions. To address this challenge, some existing works use autoencoders for motion synthesis and aim to align cross-modal information in a shared embedding space, and then constrain the text encoder and motion encoder into a compatible latent space through KL divergence loss, which drives the basic steps of creating human motion sequences from natural language input. However, due to the large difference in the distribution of text descriptions and motion sequences, forcing the alignment of the two Gaussian distributions is prone to misalignment.

[0004] On the other hand, inspired by the successful attempts of diffusion models in cross-modal image synthesis, some pioneering works have used diffusion models for text-to-action synthesis, significantly improving the fidelity and cross-modal consistency of the generated results. However, these works diffuse on the original motion sequence data, which not only introduces redundant information on the time axis and causes low efficiency, but also the powerful diffusion model may learn the noise in the original data of motion capture, causing artifacts, so that the generated human body movements cannot maintain good consistency with the text prompt conditions.

[0005] It can be seen that how to generate human body movements that are more in line with the input text prompt conditions while reducing computing overhead is a technical problem that currently needs further improvement. Summary of the invention

[0006] To solve the above technical problems, the present invention provides a method for generating human motion in latent space based on a diffusion model, the method comprising the following steps: Step 1: Select text prompt conditions for generating human motion, and encode the text prompt conditions into a text vector.

[0007] Step 2: Extract the cross-attention maps generated by the text prompt conditions, and perform weighted averaging on the cross-attention maps to generate a target cross-attention map.

[0008] Step 3: Input the text vector and random Gaussian noise into a pre-trained motion variational autoencoder, and obtain an initial human motion sequence through the motion variational autoencoder.

[0009] Step 4: Perform a diffusion denoising operation on the initial human motion sequence through a denoiser in the dimension of the latent space, and calculate the KL divergence loss between the target motion sequence obtained in the denoising step and the actual motion sequence, as well as the mean square error between the target cross-attention map and the actual cross-attention map.

[0010] Step 5: Through KL divergence loss and mean square error, the predicted noise is gradually updated in the denoising step to minimize the total loss and generate the final target motion sequence.

[0011] Optionally, in step 1, encoding the text prompt condition into a text vector includes: encoding the text prompt condition into a text vector The pre-trained CLIP Model generator obtained by downloading is used for encoding to obtain the text vector; wherein the text prompt condition is expressed as, ;in, It means a sentence that describes an action in detail; An action tag representing a set of predefined action classes. Indicates a null condition.

[0012] Optionally, in step 2, performing weighted averaging on the cross-attention maps to generate a target cross-attention map comprises: performing weighted averaging on the cross-attention maps by the following formula: ;in, represents the target cross-attention map; represents the cross-attention map generated by the [SOT] token in the text prompt condition, represents the cross-attention map generated by the [EOT] token in the text prompt condition, Represents weighted weight.

[0013] Optionally, in step 3, the pre-trained motion variational autoencoder adopts a long skip connection structure, which is expressed as: ;in, Represents an encoder based on the Transformer architecture, which is used to convert motion sequences Encoded as latent space variables , Represents a decoder based on the Transformer architecture, which is used to transform the latent space variables Decoding into motion sequences .

[0014] Optionally, in step 3, the random Gaussian noise is expressed as , and Represents the dimension, the initial human motion sequence is expressed as , Represents the length of the human motion sequence.

[0015] Optionally, in step 4, the calculation of the KL divergence loss of the denoiser learning is expressed as: ;in, represents KL divergence loss; Represents the true distribution The intermediate state generated after t steps Distribution of Indicates a given and current status When the previous state The true distribution of Indicates that the model passes the current state Generate the previous state The approximate distribution of ; the mean square error between the target cross-attention map and the actual cross-attention map is expressed as: ;in, represents the mean square error between the target cross-attention map and the actual cross-attention map; is the total number of cross attention maps, is the actual cross attention map; the motion sequence in the latent space dimension is expressed as: , and Represents dimension.

[0016] Optionally, in step 5, the mean square error of denoiser learning is expressed as: ; The total loss function is expressed as: ;in, It is a factor that controls the strength of the mean square error intervention between the target cross-attention map and the actual cross-attention map.

[0017] The noise of the forecast is expressed as a linear combination of the conditional forecast and the unconditional forecast: ;in, represents the predicted noise, represents the noise of the conditional prediction, represents the noise of the unconditional prediction; Indicates the empty condition, is a hyperparameter of the bootstrap scale.

[0018] The beneficial effects of the present invention are as follows: the method of the present invention generates human body motion through text prompt conditions, designs a method for generating human body motion in a latent space, does not use a diffusion model to establish a connection between the original action sequence and the input conditions, but uses a diffusion model based on the action sequence latent space to learn the probability mapping from the input conditions to the representative action sequence latent space encoding, and constrains the generated results through the cross-attention map of the text, which can not only reduce the computational overhead to a certain extent, but also generate a human body action sequence that meets the input conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flow chart of a method for generating human motion in latent space based on a diffusion model disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following is a description of the implementation of the present application by specific specific embodiments. People familiar with the technology can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0022] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0023] like Figure 1 As shown, an embodiment of the present invention discloses a method for generating latent space human motion based on a diffusion model, the method comprising the following steps: Step 1: Select text prompt conditions for generating human motion, and encode the text prompt conditions into a text vector.

[0024] Step 2: Extract the cross-attention maps generated by the text prompt conditions, and perform weighted averaging on the cross-attention maps to generate a target cross-attention map.

[0025] Step 3: Input the text vector and random Gaussian noise into a pre-trained motion variational autoencoder, and obtain an initial human motion sequence through the motion variational autoencoder.

[0026] Step 4: Perform a diffusion denoising operation on the initial human motion sequence through a denoiser in the dimension of the latent space, and calculate the KL divergence loss between the target motion sequence obtained in the denoising step and the actual motion sequence, as well as the mean square error between the target cross-attention map and the actual cross-attention map.

[0027] Step 5: Through KL divergence loss and mean square error, the predicted noise is gradually updated in the denoising step to minimize the total loss and generate the final target motion sequence.

[0028] In the embodiment, considering the role of the diffusion model-based text control condition to human motion generation model in understanding the relationship between text conditions and action sequences, the present invention designs a method for generating a human motion sequence from text prompt conditions in the latent space dimension, thereby solving the consistency problem of text prompt conditions and human motion sequence generation. Thus, based on a given text prompt condition, a human motion sequence that is consistent with the text prompt condition can be directly generated.

[0029] The actual motion sequence refers to the serialized representation of human motion in the real world, that is, the motion sequence corresponding to the real human motion, which is usually obtained through motion capture technology. Correspondingly, the actual cross-attention map is the cross-attention map corresponding to the actual motion sequence.

[0030] Optionally, in step 1, encoding the text prompt condition into a text vector includes: encoding the text prompt condition into a text vector The pre-trained CLIP Model generator obtained by downloading is used for encoding to obtain the text vector; wherein the text prompt condition is expressed as, ;in, It means a sentence that describes an action in detail; An action tag representing a set of predefined action classes. Indicates a null condition.

[0031] Optionally, in step 2, performing weighted averaging on the cross-attention maps to generate a target cross-attention map comprises: performing weighted averaging on the cross-attention maps by the following formula: ;in, represents the target cross-attention map; represents the cross-attention map generated by the [SOT] token in the text prompt condition, represents the cross-attention map generated by the [EOT] token in the text prompt condition, Represents weighted weight.

[0032] Optionally, in step 3, the pre-trained motion variational autoencoder adopts a long skip connection structure, which is expressed as: ;in, Represents an encoder based on the Transformer architecture, which is used to convert motion sequences Encoded as latent space variables , Represents a decoder based on the Transformer architecture, which is used to transform the latent space variables Decoding into motion sequences .

[0033] Optionally, in step 3, the random Gaussian noise is expressed as , and Represents the dimension, the initial human motion sequence is expressed as , Represents the length of the human motion sequence.

[0034] Optionally, in step 4, the calculation of the KL divergence loss of the denoiser learning is expressed as: ;in, represents KL divergence loss; Represents the true distribution The intermediate state generated after t steps Distribution of Indicates a given and current status When the previous state The true distribution of Indicates that the model passes the current state Generate the previous state The approximate distribution of ; the mean square error between the target cross-attention map and the actual cross-attention map is expressed as: ;in, represents the mean square error between the target cross-attention map and the actual cross-attention map; is the total number of cross attention maps, is the actual cross attention map; the motion sequence in the latent space dimension is expressed as: , and Represents dimension.

[0035] Optionally, in step 5, the mean square error of denoiser learning is expressed as: ; The total loss function is expressed as: ;in, It is a factor that controls the strength of the mean square error intervention between the target cross-attention map and the actual cross-attention map.

[0036] The noise of the forecast is expressed as a linear combination of the conditional forecast and the unconditional forecast: ;in, represents the predicted noise, represents the noise of the conditional prediction, represents the noise of the unconditional prediction; Indicates the empty condition, is a hyperparameter of the bootstrap scale.

[0037] Compared with the existing methods, the method of the present invention reduces the cost of computing resources, improves the quality of overall human motion generation, and improves the consistency between human motion and text control conditions while realizing the generation of human motion under text control conditions.

[0038] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0039] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that with the evolution of technology, many of the elements described herein may be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A latent space human motion generation method based on a diffusion model, characterized in that: The method comprises the following steps: Step 1: Selecting a text prompt condition for generating a human action, and encoding the text prompt condition into a text vector; Step 2: extracting each cross-attention map generated by the text prompt condition, and performing weighted averaging on each cross-attention map to generate a target cross-attention map; Step 3: input the text vector and random Gaussian noise into a pre-trained motion variational autoencoder, and obtain an initial human motion sequence through the motion variational autoencoder; Step 4: Perform a diffusion denoising operation on the initial human motion sequence through a denoiser in the dimension of the latent space, and calculate the KL divergence loss between the target motion sequence obtained in the denoising step and the actual motion sequence, and the mean square error between the target cross-attention map and the actual cross-attention map; wherein the actual motion sequence refers to the serialized representation of human motion in the real world, that is, the motion sequence corresponding to the real human motion, and correspondingly, the actual cross-attention map is the cross-attention map corresponding to the actual motion sequence; Step 5: Through KL divergence loss and mean square error, the predicted noise is gradually updated in the denoising step to minimize the total loss and generate the final target motion sequence.

2. The method for generating latent space human motion based on a diffusion model according to claim 1, characterized in that: In step 1, encoding the text prompt condition into a text vector includes: The text prompt condition Encode using the downloaded pre-trained CLIP Model generator to obtain the text vector; Wherein, the text prompt condition is expressed as: ;in, , which means a sentence describing an action in detail; An action tag representing a set of predefined action classes. Indicates a null condition.

3. The method for generating latent space human motion based on a diffusion model according to claim 2, characterized in that: In step 2, the weighted average of the cross-attention maps is performed to generate a target cross-attention map, including: The cross attention maps are weighted averaged by the following formula: ;in, represents the target cross-attention map; represents the cross-attention map generated by the [SOT] token in the text prompt condition, represents the cross-attention map generated by the [EOT] token in the text prompt condition, Represents weighted weight.

4. The method for generating latent space human motion based on a diffusion model according to claim 3, characterized in that: In step 3, the pre-trained motion variational autoencoder uses a long skip connection structure, which is expressed as: ;in, Represents an encoder based on the Transformer architecture, which is used to convert motion sequences Encoded as latent space variables , Represents a decoder based on the Transformer architecture, which is used to transform the latent space variables Decoding into motion sequences .

5. The method for generating latent space human motion based on a diffusion model according to claim 4, characterized in that: In step 3, the random Gaussian noise is expressed as , and Represents the dimension, the initial human motion sequence is expressed as , Represents the length of the human motion sequence.

6. The method for generating latent space human motion based on a diffusion model according to claim 5, characterized in that: In step 4, the calculation of the KL divergence loss of denoiser learning is expressed as: ; represents KL divergence loss; Represents the true distribution The intermediate state generated after t steps Distribution of Indicates a given and current status When the previous state The true distribution of Indicates that the model passes the current state Generate the previous state The approximate distribution of The mean square error between the target cross-attention map and the actual cross-attention map is expressed as: ;in, represents the mean square error between the target cross-attention map and the actual cross-attention map; is the total number of cross attention maps, is the actual cross attention map; The motion sequence in the latent space dimension is represented as: , and Represents dimension.

7. The method for generating latent space human motion based on a diffusion model according to claim 6, characterized in that: In step 5, the mean square error of denoiser learning is expressed as: ; The total loss function is expressed as: ;in, is a factor that controls the intervention strength of the mean square error between the target cross-attention map and the actual cross-attention map; The noise of the forecast is expressed as a linear combination of the conditional forecast and the unconditional forecast: ;in, represents the predicted noise, represents the noise of the conditional prediction, represents the noise of the unconditional prediction; Indicates the empty condition, is a hyperparameter of the bootstrap scale.

Images