Whole body human motion generation system and method based on autoregressive diffusion transformer

Abstract

The embodiment of the application discloses a full-body human motion generation system and method based on an autoregressive diffusion transformer, the system comprising an autoregressive diffusion transformer, a unified human motion module and a multi-modal encoder, wherein the multi-modal encoder extracts each modal feature in the input data respectively, and projects each modal feature to a unified multi-modal feature space; the unified human motion module splices the projected multi-modal features to form a fixed prefix context input into the diffusion transformer; and the diffusion transformer generates and outputs full-body human motion. The application solves the problems of insufficient motion quality, limited scene generalization, inaccurate multi-modal condition control and difficulty in generating long-time motion with spatiotemporal consistency. The application significantly improves the authenticity and controllability of generated motion, and supports diversified motion generation tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of action generation, and in particular to a full-body human motion generation system and method based on an autoregressive diffusion transformer. BACKGROUND

[0002] The full-body human motion generation technology has a wide application potential in many fields due to its high fidelity and controllable characteristics. The existing technology in the field of full-body human motion generation has obvious shortcomings. Most of them are single task models and cannot consider diversified tasks. Some models pay attention to multiple modalities and can be divided into three types. The first type is to train a model for each modality condition, which leads to the inability of interaction between different modalities. The second type is to use independent training branches to process different inputs, which is prone to modality conflicts between branches. The third type is to directly mix multiple modality conditions for training, but it is difficult for the model to accurately learn the relationship between conditions and motions due to the differences in condition constraints. On the data, the quantity is heavy and the quality is light when integrating multiple data sources, and the visual language annotation is not combined, which leads to the scarcity of text annotation or the serious illusion problem, thereby causing poor generalization from text to motion. In addition, the existing technology can only generate short motions at a time, and it is difficult to generate long-time three-dimensional human motions with consistent style and spatiotemporal dynamics. SUMMARY

[0003] The technical problem to be solved by the embodiments of the present application is to provide a full-body human motion generation system and method based on an autoregressive diffusion transformer, so as to generate long-time three-dimensional human motions with consistent style and spatiotemporal dynamics.

[0004] In order to solve the above technical problems, the embodiments of the present application provide a full-body human motion generation system based on an autoregressive diffusion transformer, which comprises an autoregressive diffusion transformer, a unified human motion module and a multi-modal encoder, wherein,

[0005] The multi-modal encoder extracts the features of each modality in the input data respectively, and projects the features of each modality to a unified multi-modal feature space; the input data includes text, global motion, speech, music and reference motion;

[0006] The unified human motion module concatenates the projected multi-modal features and noise motion features to form a fixed prefix context input into the diffusion transformer;

[0007] The diffusion transformer generates and outputs full-body human motion.

[0008] Correspondingly, the embodiments of the present application also provide a full-body human motion generation method based on an autoregressive diffusion transformer, which comprises:

[0009] The model construction step comprises: constructing a full-body human motion generation model based on an autoregressive diffusion transformer, the model comprising an autoregressive diffusion transformer, a unified human motion module and a multi-modal encoder;

[0010] The model extracts features corresponding to each modality of text, global motion, voice, music, reference motion and noise motion features through the multi-modal encoder.

[0011] The unified human motion module concatenates the projected multi-modal features and noise motion features to form a fixed prefix context input into the diffusion transformer.

[0012] The diffusion transformer generates and outputs full-body human motion.

[0013] The model training step comprises: training the model using a progressive weak-to-strong condition strategy, gradually generating motion from abstract semantic constraints to dense spatiotemporal alignment constraints.

[0014] The motion generation step comprises: inputting Gaussian noise sampled from a Gaussian distribution into the trained full-body human motion generation model to generate corresponding full-body human motion.

[0015] The present application solves the problems of insufficient motion quality, limited scene generalization, inaccurate multi-modal condition control and difficulty in generating long-time motion with spatiotemporal consistency. By introducing reference motion as a special condition and combining diffusion transformer and progressive training strategy, the present application significantly improves the authenticity and controllability of generated motion, while supporting diversified motion generation tasks. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a flowchart of a full-body human motion generation system based on an autoregressive diffusion transformer according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] It should be noted that the embodiments and features in the present application can be combined with each other without conflict, and the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0018] In the embodiments of the present application, if there is a directional indication (such as up, down, left, right, front, back, etc.), it is only used to explain the relative position relationship, motion condition, etc. between components in a certain specific posture (as shown in the drawings), if the specific posture changes, the directional indication will also change accordingly.

[0019] In addition, in the present application, if the description involves "first", "second" and the like, it is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implying the number of the indicated technical features. Therefore, the features defined as "first" and "second" can be explicitly or implicitly included at least one of the features.

[0020] Please refer to Figure 1 The autoregressive diffusion transformer-based whole-body motion generation system of the embodiment of the present application comprises an autoregressive diffusion transformer, a unified body motion module and a multi-modal encoder.

[0021] The system framework of the present application adopts a diffusion transformer (DiT) as a core model, fuses multi-modal conditions as fixed prefix context sequences through a self-attention mechanism, and gradually generates motion from high-level semantic constraints to dense spatiotemporal alignment constraints by adopting a progressive weak-to-strong condition strategy. In addition, since the present application introduces reference motion as a special condition, the quality of the generated motion is significantly improved, and the consistency of the reference motion and the generated motion is realized, thereby supporting autoregressive interactive motion generation and easily expanding to long-time motion generation.

[0022] The multi-modal encoder extracts each modal feature in the input data and projects each modal feature to a unified multi-modal feature space; the input data includes text, global motion, voice, music and reference motion.

[0023] In order to retain rich cross-modal condition information, the present application uses modal-specific encoders (such as a pre-trained T5-XXL for text conditions, a wav encoder for voice conditions, and Librosa for music conditions) to extract modal features. The multi-modal encoder includes a text encoder, a global motion encoder, a voice encoder, a music encoder, a whole-body noise motion encoder, a reference motion encoder, etc. The text encoder adopts a T5 encoder. The reference motion encoder, the whole-body noise motion encoder and the global motion encoder all use multiple linear layers for encoding, and each linear layer encodes a body part (distinguished by an index). The input format is that it can be explicitly known which index corresponds to which body part. The three encoders all encode a single frame of human body and do not do time series encoding. The music encoder uses Librosa to extract intensity (OnsetEnvelope), MFCC (Mel Frequency Cepstrum Coefficient), chroma feature (Chroma CENS), starting point and beat. The voice encoder extracts features through a learnable convolution. The reference motion encoder structure and the whole-body noise motion encoder structure are consistent, and the parameters are different.

[0024] The unified human body motion module splices the projected multi-modal features and noise motion features to form a fixed prefix context input into the diffusion transformer.

[0025] The fusion formula of the unified human body motion module is:

[0026] ,

[0027] Wherein, f represents a modal-specific encoder, h represents a projection layer, and c represents a spliced multi-modal condition representation (fixed arrangement).

[0028] To constrain the physical properties of motion, the present application directly predicts the motion Instead of noise. Therefore, the diffusion target of the present application is defined as follows:

[0029] ;

[0030] Wherein, Indicates the data distribution, and T is the maximum diffusion step number. G Indicates the learned denoising function, and x t Indicates the noise motion of step t , expressed as , wherein each Corresponds to the i-th pose in the motion at step t.

[0031] The diffusion transformer generates and outputs the full-body human motion.

[0032] The present application generates corresponding space-time masks according to the task type of the global space-time controllable task, and realizes multiple global space-time controllable tasks through a global motion condition and a task corresponding space-time mask.

[0033] The present application divides different global space-time controllable tasks into: space-intensive tasks covering multiple joints and space-sparse tasks only focusing on sparse joints; for space-intensive tasks, the global motion condition is decomposed into sparse global motion representation and dense relative representation (space-sparse does not need to be decomposed, space-sparse tasks are generally not a problem, and space-dense problems are relatively large, so dense is decomposed into sparse).

[0034] The present application realizes global space-time controllable generation through the following steps:

[0035] 1. Global motion condition and task-dependent mask strategy:

[0036] The conditions of the global spatiotemporal controllable generation task can all come from a global motion condition (precisely describing the positions of human joints in space in three-dimensional global space x, y, z coordinates), and the global motion condition is then designed with different spatiotemporal masks according to the task type (such as motion prediction, interpolation, completion, joint / trajectory guided synthesis). For motion prediction, the previous motion (such as the first 25% of frames) is visible, and the latter motion (such as the last 75%) is invisible; for motion interpolation, the previous motion (such as the first 25% of frames) and the latter motion (such as the last 25%) are visible, and the middle motion (such as the middle 50%) is invisible; for motion completion, some joints on some frames are invisible, and most of the remaining areas are visible; for joint / trajectory guided synthesis, only a small part of the joints are visible on some frames, and most of the remaining areas are invisible. Then we can realize multi-task training by applying different types of spatiotemporal masks to a global motion condition.

[0037] 2. Global position gradient optimization and global motion representation decomposition:

[0038] In order to make the local motion representation of the model output (local motion representation: as the output of the model, it is a kind of representation designed specifically to reduce the difficulty of the model learning motion features. Specifically, it includes root joint representation, including the height, linear velocity and angular velocity of the root joint; the rest of the joints are represented as position and rotation relative to the root joint. In this way, structured and hierarchical data is provided to the model to help the model learn motion features more efficiently) meet the constraints of the global motion condition, in addition to directly inputting the global motion condition, the local motion representation of the model output is integrated to obtain the global motion representation, and then the difference between the input global motion condition and the output global motion representation is calculated. In the process of diffusing model denoising, the gradient optimization method is used to guide the output results of the model. However, this gradient optimization method often causes unnatural generated motion when involving multiple joints.

[0039] To this end, the present application divides these different global spatiotemporal controllable tasks into: space-intensive tasks (such as motion prediction, interpolation and completion) covering multiple joints, and space-sparse tasks (such as joint / trajectory guided synthesis) only focusing on sparse joints.

[0040] For space-intensive tasks, the present application decomposes the global motion condition into a sparse global motion representation (only sparse joint global representation is retained) and a dense relative representation (dense joints are converted from global representation to local representation). By optimizing the global position of sparse joints while injecting noise motion into the dense relative representation during inference, the mean of the diffusion denoising model is injected in the form of splicing operation. For this, the formulaic representation of global spatio-temporal controllable generation is:

[0041] ;

[0042] ;

[0043] Where c_g is the global motion condition, M_t is the task spatio-temporal mask, M_d is the global motion decomposition mask, S is the noise adding process, M' represents the mask converted to the form of local representation, G is the distance between the calculation condition and the output, and the G function definition of the OmniControl paper can be referred to; μ t μ refers to the mean of the Gaussian distribution (related to the theory of diffusion), and τ is a hyperparameter that controls the strength / amplitude of the gradient.

[0044] As an implementation, the present application adopts a progressive training strategy, gradually introducing fine-grained conditions, with the following specific steps:

[0045] First stage: only use text conditions for training, establish the alignment of motion and abstract semantics, at this time the condition only constrains the motion at the abstract level.

[0046] Second stage: introduce reference motion conditions based on text conditions to enhance the spatio-temporal constraints of the generated motion at the starting point. At this time, it is only necessary to satisfy the continuity of the generated motion at the starting point and the end of the reference motion.

[0047] Third stage: then, introduce global spatio-temporal conditions to enhance the sparsity of the generated motion. At this time, there is only a constraint relationship on some frames and some joints.

[0048] Fourth stage: finally, introduce music and voice to enhance the dense spatio-temporal alignment of the generated motion on all time frames. At this time, there is a constraint relationship on all time frames, and all modal conditions are coexistable. Train under complete multi-modal conditions to ensure high quality of the generated motion and accurate control of multi-modal conditions.

[0049] The present application realizes the ability to combine conditions arbitrarily during inference by assigning probabilities to different conditions during the training process.

[0050] As an implementation, the present application uses data in a preset data set for training, and the preset data set is constructed according to the following steps:

[0051] Motion capture data covering various motion generation tasks are collected;

[0052] The data is unified into SMPL-X format and transformed into a unified coordinate system;

[0053] Unified to a preset fps (such as 30 fps);

[0054] The motion is rendered into a video, and a hierarchical text label is generated by using a visual language model. The present application accurately and automatically generates a hierarchical text label by rendering the motion into a video and using visual language models (VLMs) to combine visual information and text information with background information of the data set itself, such as original text, category, data set introduction, etc. If it is music or voice data, it is directly included in the preset data set.

[0055] The present application is used for full-body human motion generation. The system framework of the present application uses an autoregressive diffusion transformer to support various multi-modal motion generation tasks in a unified sequence-to-sequence manner, including text-to-motion, music-to-dance, speech-to-gesture, human interaction in a scene, object and human interaction, human and human interaction, and global space-time control tasks (such as motion prediction, intermediate frame generation, motion completion, and joint / trajectory guided synthesis), and also supports flexible combinations of these tasks.

[0056] The present application innovatively uses reference motion as a new conditional signal, which significantly improves the consistency, style, and temporal dynamics of the generated content, which is extremely critical for generating realistic animations. To solve the conflict problem of different granularities of multi-modal conditions, the present application introduces a gradual weak-to-strong condition strategy, gradually adds conditions from text, to reference motion, to global motion condition, to music and speech, etc., realizes a learning process from abstract level constraint to strict space-time alignment constraint, and realizes unified multi-task modeling. The global motion condition is further decomposed into sparse global representation and dense relative representation, so as to maintain the naturalness of the motion.

[0057] The application can effectively integrate multiple motion capture data sources (motion capture data refers to the data obtained by using advanced high-precision devices such as wearable sensors, inertial measurement units (IMU) and the like to accurately measure and record human motion. These devices can capture real-time motion information of each part of the human body, including position, angle, speed and the like. The data collected by such devices usually has high quality and can accurately reflect the real motion state of the human body, providing a reliable data basis for subsequent human motion analysis, modeling and generation), covering various tasks, including text-to-motion generation, audio-to-dance generation, speech-to-gesture generation, controllable generation of global control signals to motion, and generation of human interaction in a scene, generation of object and human interaction, generation of human and human interaction, etc. To ensure detailed and consistent text labeling, the application renders the motion sequence into a video and uses advanced natural language processing models to automatically generate structured and hierarchical descriptions to capture low-level actions and high-level semantics.

[0058] The whole-body motion generation method based on the autoregressive diffusion transformer of the embodiment of the application comprises:

[0059] The model construction step is to construct a whole-body motion generation model based on an autoregressive diffusion transformer, wherein the model comprises an autoregressive diffusion transformer, a unified human motion module and a multi-modal encoder.

[0060] The model extracts features corresponding to each modality of text, global motion, speech, music, reference motion and noise motion features through the multi-modal encoder, and projects the features of each modality to a unified multi-modal feature space.

[0061] The unified human motion module concatenates the projected multi-modal features and noise motion features to form a fixed prefix context input into the diffusion transformer.

[0062] The diffusion transformer generates and outputs whole-body motion.

[0063] The model training step is to train the model by using a progressive weak-to-strong condition strategy, and gradually generate motion from abstract semantic constraints to dense spatiotemporal alignment constraints.

[0064] The motion generation step is to input Gaussian noise sampled from a Gaussian distribution into the trained whole-body motion generation model to generate corresponding whole-body motion.

[0065] As an implementation mode, the multi-modal encoder comprises a text encoder, a global motion encoder, a speech encoder, a music encoder, a whole-body noise motion encoder and a reference motion encoder.

[0066] As an implementation mode, in the model training step, a progressive weak-to-strong condition strategy is used.

[0067] First stage: training with text condition only, establishing the alignment between motion and abstract semantics;

[0068] Second stage: introducing reference motion condition on the basis of text condition, enhancing the spatiotemporal constraint of generated motion at the start;

[0069] Third stage: then introducing global spatiotemporal condition, enhancing the spatiotemporal constraint of generated motion sparsely;

[0070] Fourth stage: finally introducing music and voice, enhancing the dense spatiotemporal alignment of generated motion at all time frames.

[0071] As an implementation, the model generates corresponding spatiotemporal masks according to the task type of the global spatiotemporal controllable task, and implements multiple global spatiotemporal controllable tasks through a global motion condition and a task corresponding spatiotemporal mask;

[0072] Different global spatiotemporal controllable tasks are divided into spatially dense tasks covering multiple joints and spatially sparse tasks focusing on sparse joints only;

[0073] For spatially dense tasks, the global motion condition is decomposed into sparse global motion representation and dense relative representation;

[0074] The formulaic representation of global spatiotemporal controllable generation is:

[0075] ;

[0076] ;

[0077] Wherein, c g is the global motion condition, M t is the task spatiotemporal mask, M d is the global motion decomposition mask, S is the noise adding process, M' represents the mask converted to the form of local representation, G is the distance between the calculation condition and the output; μ t is the mean of Gaussian distribution, and τ is a hyperparameter.

[0078] As an implementation, the model training step further includes a training data set construction step before the model training step:

[0079] Collecting motion capture data covering various motion generation tasks;

[0080] Unifying the data into SMPL-X format and transforming to a unified coordinate system;

[0081] Unified to the preset fps;

[0082] Motion is rendered as video and hierarchical text annotations are generated using a visual language model.

[0083] The application supports various conditional inputs such as text, music, voice, etc., and generates a process or model of corresponding three-dimensional human motion according to these conditional constraints. The process realizes multi-source information driven human motion synthesis by fusing information of different modalities to meet the demand for human motion in various scenarios.

[0084] Although embodiments of the present application have been shown and described, it would be appreciated by those skilled in the art that changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An autoregressive diffusion transformer based whole-body human motion generation system, characterized by, The system comprises an autoregressive diffusion transformer, a unified human motion module, and a multi-modal encoder, wherein The multi-modal encoder extracts respective modal features in input data and projects the respective modal features to a unified multi-modal feature space; the input data comprises text, global motion, speech, music, and reference motion; The unified human motion module concatenates the projected multi-modal features and noise motion features to form a fixed prefix context input to the diffusion transformer; The diffusion transformer generates and outputs whole-body human motion; The system adopts a progressive weak-to-strong condition strategy: In the first stage, only text conditions are used for training to establish alignment between motion and abstract semantics; In the second stage, reference motion conditions are introduced on the basis of text conditions to enhance the spatiotemporal constraints of generated motion at the starting point; In the third stage, global spatiotemporal conditions are then introduced to enhance the spatiotemporal constraints of sparse generated motion; In the fourth stage, music and speech are finally introduced to enhance the dense spatiotemporal alignment of generated motion at all time frames.

2. The autoregressive diffuser transformer-based full-body human motion generation system of claim 1, wherein, The multi-modal encoder comprises a text encoder, a global motion encoder, a speech encoder, a music encoder, a whole-body noise motion encoder, and a reference motion encoder.

3. The autoregressive diffuser transformer-based full-body human motion generation system of claim 1, wherein, The system generates corresponding spatiotemporal masks according to the task type of a global spatiotemporal controllable task, and implements multiple global spatiotemporal controllable tasks through a global motion condition and a spatiotemporal mask corresponding to the task; Different global spatiotemporal controllable tasks are divided into spatially dense tasks covering multiple joints and spatially sparse tasks focusing on only sparse joints; For spatially dense tasks, the global motion condition is decomposed into sparse global motion representation and dense relative representation; The formulaic representation of global spatiotemporal controllable generation is: ， ， where c g is the global motion condition, M t is the task spatiotemporal mask, M d is the global motion decomposition mask, S is the noise addition process, M’ represents the mask converted to the form of local representation, G is the distance between the calculation condition and the output; μ t is the mean of the Gaussian distribution, and τ is a hyperparameter.

4. The autoregressive diffuser transformer-based full-body human motion generation system of claim 1, wherein, The system is trained using data in a preset data set, which is constructed according to the following steps: Motion capture data covering multiple motion generation tasks are collected; The data is unified into SMPL-X format and transformed into a unified coordinate system; The data is unified to a preset fps; The motion is rendered into a video, and a visual language model is used to generate hierarchical text annotations.

5. A method for generating a whole-body human motion based on an autoregressive diffusion transformer, characterized by, The system comprises an autoregressive diffusion transformer, a unified human motion module, and a multi-modal encoder, wherein The multi-modal encoder extracts respective modal features in input data and projects the respective modal features to a unified multi-modal feature space; the input data comprises text, global motion, speech, music, and reference motion; The unified human motion module concatenates the projected multi-modal features and noise motion features to form a fixed prefix context input to the diffusion transformer; The diffusion transformer generates and outputs whole-body human motion; The system adopts a progressive weak-to-strong condition strategy: In the first stage, only text conditions are used for training to establish alignment between motion and abstract semantics; In the second stage, reference motion conditions are introduced on the basis of text conditions to enhance the spatiotemporal constraints of generated motion at the starting point; In the third stage, global spatiotemporal conditions are then introduced to enhance the spatiotemporal constraints of sparse generated motion; In the fourth stage, music and speech are finally introduced to enhance the dense spatiotemporal alignment of generated motion at all time frames. The multi-modal encoder comprises a text encoder, a global motion encoder, a speech encoder, a music encoder, a whole-body noise motion encoder, and a reference motion encoder. The system generates corresponding spatiotemporal masks according to the task type of a global spatiotemporal controllable task, and implements multiple global spatiotemporal controllable tasks through a global motion condition and a spatiotemporal mask corresponding to the task; Different global spatiotemporal controllable tasks are divided into spatially dense tasks covering multiple joints and spatially sparse tasks focusing on only sparse joints; For spatially dense tasks, the global motion condition is decomposed into sparse global motion representation and dense relative representation; The formulaic representation of global spatiotemporal controllable generation is: The system is trained using data in a preset data set, which is constructed according to the following steps: Motion capture data covering multiple motion generation tasks are collected; The data is unified into SMPL-X format and transformed into a unified coordinate system; The data is unified to a preset fps; The motion is rendered into a video, and a visual language model is used to generate hierarchical text annotations. The system comprises an autoregressive diffusion transformer, a unified human motion module, and a multi-modal encoder, wherein The multi-modal encoder extracts respective modal features in input data and projects the respective modal features to a unified multi-modal feature space; the input data comprises text, global motion, speech, music, and reference motion; The unified human motion module concatenates the projected multi-modal features and noise motion features to form a fixed prefix context input to the diffusion transformer; The diffusion transformer generates and outputs whole-body human motion; The system adopts a progressive weak-to-strong condition strategy: In the first stage, only text conditions are used for training to establish alignment between motion and abstract semantics; The second stage: on the basis of the text condition, a reference motion condition is introduced to enhance the spatiotemporal constraint of the generated motion at the starting point; The third stage: then, a global spatiotemporal condition is introduced to enhance the spatiotemporal constraint of the generated sparse motion; The fourth stage: finally, music and voice are introduced to enhance the dense spatiotemporal alignment of the generated motion in all time frames.

6. The autoregressive diffuser transformer-based whole-body human motion generation method of claim 5, wherein, The multi-modal encoder comprises a text encoder, a global motion encoder, a voice encoder, a music encoder, a whole-body noise motion encoder, and a reference motion encoder.

7. The autoregressive diffuser transform-based whole-body human motion generation method according to claim 5, wherein, The model generates corresponding spatiotemporal masks according to the task types of the global spatiotemporal controllable tasks, and implements multiple global spatiotemporal controllable tasks through a global motion condition and the spatiotemporal masks corresponding to the tasks; the different global spatiotemporal controllable tasks are divided into a spatial dense task covering multiple joints and a spatial sparse task focusing on only sparse joints; For the spatial dense task, the global motion condition is decomposed into a sparse global motion representation and a dense relative representation; The formulaic representation of the global spatiotemporal controllable generation is: ， ， where c g is the global motion condition, M t is the task spatio-temporal mask, M d is the global motion decomposition mask, S is the noise addition process, M’ represents the mask converted to the form of local representation, G is the distance between the calculation condition and the output; μ t is the mean of the Gaussian distribution, and τ is a hyperparameter.

8. The autoregressive diffuser transform-based whole-body human motion generation method according to claim 5, wherein, The model training step further comprises a training data set construction step: Motion capture data covering multiple motion generation tasks are collected; The data are unified into an SMPL-X format and transformed into a unified coordinate system; The data are unified to a preset fps; The motion is rendered into a video, and a hierarchical text label is generated by using a visual language model.

Images