Method for generating human motion based on text of diffusion model
By using a diffusion model-based text generation method and employing a CLIP encoder and motion data filter to filter key information, the problem of excessive redundant information in text-driven human motion generation is solved, thus improving generation efficiency and accuracy.
Patent Information
- Application Number
- CN202511199069.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-26
- Publication Date
- 2025-11-21
AI Technical Summary
Existing methods for text-driven human motion generation suffer from problems such as excessive redundant information, high computational cost, low denoising efficiency, and inaccurate generation results, and fail to effectively distinguish between key and non-key information.
A diffusion-based approach is adopted, which generates low-dimensional text vectors through CLIP text encoder, performs diffusion and denoising operations on the initial human motion sequence by combining motion data filter, uses a multi-level attention mechanism to filter key information, and optimizes the generation process by combining simple loss, position loss, foot contact loss, velocity loss and LipMLP loss.
It reduces redundant information, improves generation efficiency and quality, enhances the semantic alignment between text and actions, and makes the generated actions more consistent with the text description.
Smart Images
Figure CN120997402A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of human motion generation technology in computer vision, and specifically relates to a method for generating human motion from text based on a diffusion model. Background Technology
[0002] Due to the semantic richness and user-friendly nature of natural language, text-driven human motion generation has recently become an emerging research focus. With the rapid development of human-computer interaction, virtual reality, and animation, text-driven human motion generation, as an efficient and intuitive control method, has attracted widespread attention. Compared to traditional methods based on manual design or parameter control, natural language possesses stronger semantic expressive power and more user-friendly interactive characteristics, which can greatly improve the controllability and flexibility of generated actions.
[0003] However, text and human actions are heterogeneous modal data, with inherent differences in structure and distribution, making the inference of corresponding high-quality action sequences from natural language a challenging task. To address this cross-modal alignment problem, some existing methods employ autoencoder architectures to encode text and motion separately, and then use methods such as KL divergence to align their distributions in a shared latent space. While these methods have some effectiveness, the mandatory assumptions made during the alignment process struggle to accurately capture the complex nonlinear relationships between text and actions, easily leading to embedding space shifts or semantic information loss, thus affecting the accuracy and naturalness of the generated results.
[0004] In recent years, diffusion models have demonstrated powerful modeling capabilities in generation tasks, particularly in image, audio, and cross-modal generation tasks, exhibiting high fidelity and diversity. Inspired by this, some studies have introduced diffusion models into text-driven action generation tasks, achieving human action reconstruction based on textual conditions by progressively adding and denoising noise to action sequences. This strategy has improved the diversity and semantic consistency of generated actions to some extent.
[0005] While diffusion models have improved modeling capabilities, existing methods still have several limitations. First, most methods perform the diffusion process directly on the complete original motion sequence, introducing a large amount of redundant information in the time dimension, resulting in high computational costs and low denoising efficiency. Second, the model may overfit the noise features in the training data during the diffusion process, leading to physical inconsistencies such as foot slippage and posture drift. Current mainstream diffusion methods fail to distinguish between critical and non-critical information in their modeling, resulting in redundant computation and a loss of focus on modeling important action nodes.
[0006] Therefore, how to reduce redundant information, improve modeling efficiency, and enhance text control capabilities and generation quality has become a technical problem that needs to be solved in the current field of text-driven action generation. Summary of the Invention
[0007] In response, the present invention provides a method, electronic device, computer storage medium, and computer program product for generating human motion from text based on a diffusion model, to solve at least one of the aforementioned technical problems.
[0008] In a first aspect, the present invention provides a method for generating human motion from text based on a diffusion model, the method comprising the following steps: Step 1: Input the text used for human motion generation, convert the text into low-dimensional text vectors using the CLIP text encoder, including word text vectors and sentence text vectors, and generate an initial human motion sequence based on the low-dimensional text vectors; Step 2: Input the initial human motion sequence and random Gaussian noise into the diffusion model for diffusion operation; Step 3: Input the noisy human motion sequence after the diffusion operation into the motion data filter for filtering; Step 4: Perform denoising on the filtered human motion sequence to obtain the actual motion sequence, and calculate the simple loss, position loss, foot contact loss, velocity loss and LipMLP loss between the desired target motion sequence and the actual motion sequence. Step 5: Construct the total loss using simple loss, position loss, foot contact loss, velocity loss, and LipMLP loss. In the denoising step, the predicted actual motion sequence is updated successively to minimize the total loss and generate the final human motion sequence.
[0009] Optionally, converting the text into a low-dimensional text vector using the CLIP text encoder includes: The text is encoded using the open-source, pre-trained CLIP encoder to obtain the low-dimensional text vector, which includes word text vectors and sentence text vectors.
[0010] Optionally, the step of inputting the initial human motion sequence and random Gaussian noise into the diffusion model for diffusion operation includes: Random Gaussian noise is represented as , and The dimension is represented as; the initial human motion sequence is represented as , This indicates the length of the initial human motion sequence.
[0011] Optionally, the noisy human motion sequence after the diffusion operation in step three is input into a motion data filter for filtering, including: The motion data filter consists of a motion-phrase cross attention, a motion-sentence cross attention, target feature matrix self-attention, a normalization layer, a Lipschitz MLP layer, and residual links; Motion-phrase cross attention and motion-sentence cross attention are extracted from the noisy human motion sequence after the diffusion operation. The former enables the motion sequence to capture information from the motion sequence and local text, while the latter can capture information from the motion sequence and global text. These are then concatenated together, projected back to the transformer dimension by LipMLP, and stretched to a one-dimensional vector mask after self-attention of the target feature matrix. The filtered information and the original data are joined by residuals and then normalized. The specific calculation for filtering is as follows: Where H is the target matrix, R is the residual obtained after masking; matrix A represents the cross-attention matrix generated by the motion sequence and the global text vector extracted by CILP, matrix B represents the cross-attention matrix generated by the motion sequence and the local text vector extracted by CILP, LipMLP represents the Lipschitz MLP calculation performed on the concatenated matrix A and matrix B, the calculated matrix can capture the global and local features between the input text and motion data; Mask represents stretching and masking the target matrix; Indicates the mask ratio; This represents the learnable parameter matrix in the self-attention mechanism; This represents the dimension of the key vector.
[0012] Optionally, in step four, the calculation of the simple loss learned by the denoiser is expressed as follows: ;in, Indicates simple loss; Indicates the distribution of real data That is, from the conditions The real and clear data below Sampling is performed during the process. Represents a text vector; Indicates time step It is uniformly distributed between 1 and T, where T is the total number of time steps in the diffusion process; Representing true and clear data With model L2 norm loss between predicted clear data; model It is a decoder based on the Transformer architecture; The calculation of the position loss learned by the denoiser is expressed as: ; The calculation of the foot contact loss during denoising is expressed as follows: ; The calculation of the speed loss of the denoiser learning is expressed as: ; in, This represents the forward kinematics function, used to convert the rotation angle of a joint into its position; for each frame... , It is a binary foot contact mask. It represents the number of joints. This foot contact mask is only valid for foot joints and is used to indicate whether the foot is in contact with the ground.
[0013] The LipMLP loss calculation is expressed as follows: ;in, This represents the LipMLP function. Represents the input vector. This represents the Lipschitz constant. It represents the mathematical norm.
[0014] Optionally, in step five, the total loss function for denoiser learning is expressed as: ;in, It is a hyperparameter used to control the weights of the position loss; a larger value indicates a higher weight. The value indicates that the model pays more attention to the accuracy of joint positions; It is a hyperparameter used to control the weight of velocity loss; a larger value indicates a higher risk. The value indicates that the model will pay more attention to the dynamic consistency of the generated actions; It is a hyperparameter used to control the weight of foot loss; a larger one... The value will make the model pay more attention to preventing slippage when the foot contacts the ground; Control the smoothing amplitude.
[0015] A second aspect of the invention provides an electronic device comprising: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, the computer program being executed by the processor to implement the method as described in any of the preceding claims.
[0016] In a third aspect, the present invention provides a computer storage medium storing a computer program executable by a processor, the computer program being executed by the processor to implement the method as described in any of the preceding claims.
[0017] In a fourth aspect, the present invention provides a computer program product comprising a computer program executable by a processor, the computer program being executed by the processor to perform the method as described in any of the preceding claims.
[0018] The beneficial effects of this invention are as follows: A motion data filter was designed to perform diffusion denoising without using the original action sequences in the diffusion model. A multi-level attention mechanism within the motion data filter evaluates and filters noisy action sequences, extracting key information, reducing redundant information, and improving generation quality and efficiency. Furthermore, it enhances the semantic alignment between text and actions, improving the matching degree between text and generated actions. Attached Figure Description
[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a flowchart illustrating a method for generating human motion from text based on a diffusion model, as disclosed in an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. Detailed Implementation
[0021] The following specific embodiments illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0022] Furthermore, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.
[0023] Considering that there is a large amount of non-critical redundant information in human movements, while a small amount of critical information often carries the semantic structure and physical logic of the complete movement; in this regard, the present invention designs a motion data filter, thereby enhancing the matching degree between text prompt conditions and human movement sequence generation and improving generation quality and efficiency.
[0024] like Figure 1 As shown in the figure, this invention discloses a method for generating human motion from text based on a diffusion model, the method comprising the following steps: Step 1: Input the text used for human motion generation, convert the text into low-dimensional text vectors using the CLIP text encoder, including word text vectors and sentence text vectors, and generate an initial human motion sequence based on the low-dimensional text vectors.
[0025] The initial human motion sequence refers to the human motion sequence directly generated from random Gaussian noise using a diffusion model based on the text in this step, which is used to perform diffusion operations in step two below by combining it with Gaussian noise.
[0026] Step 2: Input the initial human motion sequence and random Gaussian noise into the diffusion model for diffusion operation; Step 3: Input the noisy human motion sequence after the diffusion operation into the motion data filter for filtering; In this step, multiple human motion sequences are obtained after motion-word cross-attention, motion-sentence cross-attention, and self-attention calculations in the motion data filter. Specifically: motion-word cross-attention calculation filters motion features related to local text semantics by associating word text vectors (Q) with the diffused noisy motion sequences (K, V); motion-sentence cross-attention calculation filters motion features related to global text semantics by associating sentence text vectors (Q) with the diffused noisy motion sequences (K, V); the obtained motion-word attention matrix and motion-sentence attention matrix are concatenated, and LipMLP is used for feature fusion to obtain a motion matrix containing global and local text information. Self-attention calculation is then used on the obtained motion matrix to further capture the temporal dependencies within the motion sequences, ultimately outputting multiple candidate sequences; key information is filtered by score to reduce redundancy. The results are then residually linked with the noisy human motion sequences.
[0027] Step 4: Perform denoising on the filtered human motion sequence to obtain the actual motion sequence, and calculate the simple loss, position loss, foot contact loss, velocity loss and LipMLP loss between the desired target motion sequence and the actual motion sequence. The actual motion sequence is an intermediate result obtained after diffusion, attention calculation, filtering, and preliminary denoising (corresponding to the calculation formula below). The target motion sequence refers to the ideal motion sequence that is expected to be generated, that is, real and clear motion data that conforms to the text description (corresponding to the calculation formula below). ).
[0028] Step 5: Construct the total loss using simple loss, position loss, foot contact loss, velocity loss, and LipMLP loss. In the denoising step, the predicted actual motion sequence is updated successively to minimize the total loss and generate the final human motion sequence.
[0029] In this embodiment, a motion data filter is used to evaluate the action sequence and filter out a certain proportion of data with lower scores, thus selecting key information, reducing redundant information, and improving generation quality and efficiency. The diffusion denoising device is a decoder based on the Transformer architecture, with 8 decoder layers, a 512-dimensional transformer, a 1024-dimensional feedforward neural network, 4 multi-head attention points, and GELU as the activation function. Multiple losses are also considered to improve the generated human motion sequence in terms of joint position accuracy, dynamic consistency, and foot contact with the ground, enhancing text control and generation quality.
[0030] Optionally, converting the text into a low-dimensional text vector using the CLIP text encoder includes: The text is encoded using the open-source, pre-trained CLIP encoder to obtain the low-dimensional text vector, which includes word text vectors and sentence text vectors.
[0031] Optionally, the step of inputting the initial human motion sequence and random Gaussian noise into the diffusion model for diffusion operation includes: Random Gaussian noise is represented as , and The dimension is represented as; the initial human motion sequence is represented as , This indicates the length of the initial human motion sequence.
[0032] Optionally, step three, which involves inputting the human motion sequence into a motion data filter for filtering, includes: The motion data filter consists of a motion-phrase cross attention, a motion-sentence cross attention, target feature matrix self-attention, a normalization layer, a Lipschitz MLP layer, and residual links; In step one, word text vectors and sentence text vectors are obtained from the input text used for human motion generation. In order for the motion filter to filter out invalid and low-quality information that is irrelevant to the input text, motion-phrase cross attention and motion-sentence cross attention are applied to the motion sequence. The former enables the motion sequence to capture information between the motion sequence and the local text, while the latter can capture information between the motion sequence and the global text. Then, they are concatenated together, projected back to the transformer dimension by LipMLP, and stretched to a one-dimensional vector mask after self-attention of the target feature matrix.
[0033] The filtered information and the original data are joined by residuals and then normalized. The specific calculation for filtering is as follows: Where H is the target matrix, R is the residual obtained after masking; matrix A represents the cross-attention matrix generated by the motion sequence and the global text vector extracted by CILP, matrix B represents the cross-attention matrix generated by the motion sequence and the local text vector extracted by CILP, LipMLP represents the Lipschitz MLP calculation performed on the concatenated matrix, the calculated matrix can capture the global and local features between the input text and motion data, preparing for the subsequent filtering operation. Mask represents stretching and masking the target matrix; Indicates the mask ratio; This represents the learnable parameter matrix in the self-attention mechanism; This represents the dimension of the key vector.
[0034] Optionally, in step four, the calculation of the simple loss learned by the denoiser is expressed as follows: ;in, Indicates simple loss; Indicates the distribution of real data That is, from the conditions The real and clear data below Sampling is performed during the process. Represents a text vector; Indicates time step It is uniformly distributed between 1 and T, where T is the total number of time steps in the diffusion process; Representing true and clear data With model L2 norm loss between predicted clear data; model It is a denoiser for the diffusion model, a decoder based on the Transformer architecture with 8 layers, a 512-dimensional transformer, a 1024-dimensional feedforward neural network, 4 multi-head attention points, and uses Gelu as the activation function.
[0035] Optionally, in step four, the calculation of the location loss learned by the denoiser is expressed as follows: ; The calculation of the foot contact loss during denoising is expressed as follows: ; The calculation of the speed loss of the denoiser learning is expressed as: ; in, This represents the forward kinematics function, used to convert the rotation angle of a joint into its position; for each frame... , It is a binary foot contact mask. It represents the number of joints. This foot contact mask is only valid for foot joints and is used to indicate whether the foot is in contact with the ground.
[0036] The LipMLP loss calculation is expressed as follows: ;in, This represents the LipMLP function. Represents the input vector. This represents the Lipschitz constant. This represents the mathematical norm. Introducing LipMLP loss can improve the stability and robustness of the model.
[0037] Optionally, in step five, the total loss function for denoiser learning is expressed as: ;in, It is a hyperparameter used to control the weights of the position loss; a larger value indicates a higher weight. The value indicates that the model pays more attention to the accuracy of joint positions; It is a hyperparameter used to control the weight of velocity loss; a larger value indicates a higher risk. The value indicates that the model will pay more attention to the dynamic consistency of the generated actions; It is a hyperparameter used to control the weight of foot loss; a larger one... The value will make the model pay more attention to preventing slippage when the foot contacts the ground; Control the smoothing amplitude.
[0038] Compared with existing methods, the method of this invention reduces redundant information, improves modeling efficiency, and enhances text control and generation quality while realizing the generation of human body movements from text.
[0039] like Figure 2 As shown, embodiments of the present invention also provide an electronic device, the electronic device comprising: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, the computer program being executed by the processor to implement the method as described in any of the preceding claims.
[0040] This invention also provides a computer storage medium storing a computer program executable by a processor, the computer program being executed by the processor to implement the method as described in any of the preceding claims.
[0041] This invention also provides a computer program product comprising a computer program executable by a processor to perform the method as described in any of the preceding claims.
[0042] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0043] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for generating human motion from text based on a diffusion model, characterized in that, The method includes the following steps: Step 1: Input the text used for human motion generation, convert the text into low-dimensional text vectors using the CLIP text encoder, including word text vectors and sentence text vectors, and generate an initial human motion sequence based on the low-dimensional text vectors; Step 2: Input the initial human motion sequence and random Gaussian noise into the diffusion model for diffusion operation; Step 3: Input the noisy human motion sequence after the diffusion operation into the motion data filter for filtering; Step 4: Perform denoising on the filtered human motion sequence to obtain the actual motion sequence, and calculate the simple loss, position loss, foot contact loss, velocity loss and LipMLP loss between the desired target motion sequence and the actual motion sequence. Step 5: Construct the total loss using simple loss, position loss, foot contact loss, velocity loss, and LipMLP loss. In the denoising step, the predicted actual motion sequence is updated successively to minimize the total loss and generate the final human motion sequence.
2. The method for generating human motion from text based on a diffusion model according to claim 1, characterized in that: The text is converted into a low-dimensional text vector using the CLIP text encoder, including: The text is encoded using the open-source, pre-trained CLIP encoder to obtain the low-dimensional text vector, which includes word text vectors and sentence text vectors.
3. The method for generating human motion from text based on a diffusion model according to claim 1, characterized in that: The initial human motion sequence and random Gaussian noise are input into the diffusion model for diffusion operations, including: Random Gaussian noise is represented as , and The dimension is represented as; the initial human motion sequence is represented as , This indicates the length of the initial human motion sequence.
4. The method for generating human motion from text based on a diffusion model according to claim 1, characterized in that: The noisy human motion sequence after the diffusion operation in step three is input into a motion data filter for filtering, including: The motion data filter consists of a motion-phrase cross attention, a motion-sentence cross attention, target feature matrix self-attention, a normalization layer, a Lipschitz MLP layer, and residual links; Motion-phrase cross attention and motion-sentence cross attention are extracted from the noisy human motion sequence after the diffusion operation. The former enables the motion sequence to capture information from the motion sequence and local text, while the latter can capture information from the motion sequence and global text. These are then concatenated together, projected back to the transformer dimension by LipMLP, and stretched to a one-dimensional vector mask after self-attention of the target feature matrix. The filtered information and the original data are joined by residuals and then normalized. The specific calculation for filtering is as follows: Where H is the target matrix, R is the residual obtained after masking; matrix A represents the cross-attention matrix generated by the motion sequence and the global text vector extracted by CILP, matrix B represents the cross-attention matrix generated by the motion sequence and the local text vector extracted by CILP, LipMLP represents the Lipschitz MLP calculation performed on the concatenated matrix A and matrix B, the calculated matrix can capture the global and local features between the input text and motion data; Mask represents stretching and masking the target matrix; Indicates the mask ratio; This represents the learnable parameter matrix in the self-attention mechanism; This represents the dimension of the key vector.
5. The method for generating human motion from text based on a diffusion model according to claim 4, characterized in that: In step four, the calculation of the simple loss learned by the denoiser is expressed as follows: ;in, Indicates simple loss; Indicates the distribution of real data That is, from the conditions The real and clear data below Sampling is performed during the process. Represents a text vector; Indicates time step It is uniformly distributed between 1 and T, where T is the total number of time steps in the diffusion process; Representing true and clear data With model L2 norm loss between predicted clear data; model It is a decoder based on the Transformer architecture; The calculation of the position loss learned by the denoiser is expressed as: ; The calculation of the foot contact loss during denoising is expressed as follows: ; The calculation of the speed loss of the denoiser learning is expressed as: ; in, This represents the forward kinematics function, used to convert the rotation angle of a joint into its position; for each frame... , It is a binary foot contact mask. It represents the number of joints. This foot contact mask is only valid for foot joints and is used to indicate whether the foot is in contact with the ground. The LipMLP loss calculation is expressed as follows: ;in, This represents the LipMLP function. Represents the input vector. This represents the Lipschitz constant. It represents the mathematical norm.
6. The method for generating human motion from text based on a diffusion model according to claim 5, characterized in that: In step five, the total loss function for denoiser learning is expressed as: ;in, It is a hyperparameter used to control the weights of the position loss; a larger value indicates a higher weight. The value indicates that the model pays more attention to the accuracy of joint positions; It is a hyperparameter used to control the weight of velocity loss; a larger value indicates a higher risk. The value indicates that the model will pay more attention to the dynamic consistency of the generated actions; It is a hyperparameter used to control the weight of foot loss; a larger one... The value will make the model pay more attention to preventing slippage when the foot contacts the ground; Control the smoothing amplitude.
7. An electronic device, characterized in that: The electronic device includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, the computer program being executed by the processor to implement the method as claimed in any one of claims 1-7.
8. A computer storage medium, characterized in that: The computer storage medium stores a computer program that can be executed by a processor to implement the method as described in any one of claims 1-7.
9. A computer program product, characterized in that: The computer program product includes a computer program executable by a processor to perform the method as described in any one of claims 1-7.