A three-dimensional human body motion generation method and device based on a parallel multi-granularity transformer

Abstract

This invention discloses a method for generating 3D human motion based on a parallel multi-granularity Transformer, comprising: acquiring a 3D human motion dataset; constructing an initial model, which includes a parallel multi-granularity motion generation module, a multi-granularity motion fusion module, and a motion reconstruction and output module; and training the initial model using the 3D human motion dataset to obtain an image generation model for generating high-fidelity 3D human motion sequences. This invention also provides a 3D human motion generation device. The method provided by this invention can generate high-fidelity, semantically coherent, and detail-rich 3D human motion sequences.

Description

Technical Field

[0001] This invention belongs to the technical field of visual content generation, specifically relating to a method and apparatus for generating three-dimensional human motion based on parallel multi-granularity Transformer. Background Technology

[0002] With the rapid development of deep learning technology, cross-modal generation tasks in computer vision (CV) and natural language processing (NLP) have become a research hotspot in the field of artificial intelligence. Among them, text-driven 3D human motion generation, as a key bridge connecting abstract natural language descriptions and concrete visual dynamic representations, has broad development prospects and application value. The core goal of this technology is to build an intelligent generation model that can automatically synthesize a 3D skeletal sequence that conforms to human biomechanics, features smooth and natural movements, and is highly consistent with the text description semantically, based on any natural language text description input by the user (e.g., "A person walks forward, catches a thrown ball, and then jumps up excitedly").

[0003] In traditional digital content creation (DCC) workflows, high-quality 3D human motion primarily relies on two methods: one is frame-by-frame or key-framing by professional animators using software such as Maya and 3ds Max; the other is capturing motion data from live actors using expensive optical or inertial motion capture (Mocap) systems. The former demands extremely high levels of professional skill from production staff, is time-consuming and labor-intensive, and has a long production cycle, making it difficult to meet the real-time demands of generating massive amounts of 3D content in the metaverse era; while the latter offers higher precision, it involves expensive equipment, limited space, high actor costs, and often requires arduous post-production data cleaning to remove noise and clipping artifacts.

[0004] Patent document CN111311729A discloses a method for reconstructing 3D human pose in natural scenes based on a bidirectional projection network, comprising the following steps: 1. Acquiring data using a camera; 2. Inputting the acquired video and image data into a 2D pose detector to obtain the coordinates of the corresponding 2D human joints; 3. Designing two types of bidirectional projection networks based on whether or not 3D pose data labels are available during the training process; 4. Training the designed network using a deep adversarial learning strategy to minimize the network loss function, and finally obtaining a trained 3D pose generator through iteration; 5. Inputting the output of the 2D pose detector in step 2 into the trained 3D pose generator in step 4.

[0005] Patent document CN119992651A discloses a motion capture optimization method and model training method based on monocular video, including: inputting monocular video into a ground contact detection model to obtain ground contact detection results used to indicate the ground contact probability of a human figure's feet; obtaining first three-dimensional motion data of the human figure based on the monocular video, and then obtaining the trajectory of the human figure's feet; optimizing the foot trajectory based on the ground contact probability, and optimizing the foot trajectory based on the ground contact optimization; reconstructing the first three-dimensional motion data of the human figure using a backpropagation dynamics algorithm based on the foot trajectory optimized by the backpropagation dynamics algorithm to obtain second three-dimensional motion data of the human figure. Summary of the Invention

[0006] The purpose of this invention is to provide a method and apparatus for generating three-dimensional human motion based on parallel multi-granularity Transformer. This method can generate high-fidelity, semantically coherent and detail-rich three-dimensional human motion sequences.

[0007] To achieve the first objective of this invention, the following technical solution is provided, comprising the following steps: Obtain a 3D human motion dataset, which includes continuous 3D motion sequences and corresponding natural language text descriptions; The initial model is constructed, which includes a parallel multi-granularity action generation module, a multi-granularity action fusion module, and an action reconstruction and output module; The parallel multi-granularity action generation module includes L parallel branches with different granularities, and each branch contains generation stage one and generation stage two. The generation stage one extracts text from the input natural language text description to obtain the corresponding text embedding vector, and generates an initial token sequence that is randomly masked based on the text embedding vector using a masking modeling method. The second generation stage is used to supplement the randomly masked token sequence to output the corresponding complete token sequence; The multi-granularity action fusion module is used to map the complete token sequence output by multiple branches back to a continuous latent feature space and perform temporal alignment and fusion to output a latent feature representation; The motion reconstruction and output module generates a corresponding three-dimensional human motion sequence based on the input latent feature representation. The initial model was trained using a 3D human motion dataset to obtain an image generation model for generating high-fidelity 3D human motion sequences.

[0008] This invention reconstructs the text-to-action generation task into a two-dimensional refinement problem, that is, to perform collaborative generation and optimization on both the time axis and the quantization axis. By introducing a parallel multi-granularity hierarchical Transformer, the action sequence is processed in parallel at different time resolutions. Hierarchical residual quantization technology is used to gradually refine the action details, thereby generating a high-fidelity, semantically coherent and detail-rich three-dimensional human action sequence.

[0009] Specifically, the action reconstruction and output module is constructed based on the decoder obtained through training. The training process of the decoder is as follows: A three-dimensional action sequence is input into a one-dimensional convolutional encoder to extract action features, and the action features are mapped into continuous feature vectors in the latent space. Continuous feature vectors are discretized using multi-layer cascaded vector quantizers to obtain quantized feature vectors for all layers. The quantized feature vectors of all layers are summed to obtain the final latent feature representation, and the latent feature representation is input into the decoder to reconstruct the corresponding 3D action sequence; The 3D motion sequences in the 3D human motion dataset are used to train the 1D convolutional encoder, decoder, and accompanying codebook to obtain a decoder for constructing the motion reconstruction and output module.

[0010] Specifically, the motion features include root node angular velocity, root node linear velocity, root node height, local position of all joints relative to the root node, joint local velocity, and redundant feature vectors corresponding to joint rotations using 6D rotation representation.

[0011] Specifically, the training loss function of the decoder includes reconstruction loss, commitment loss, and codebook update components.

[0012] Specifically, during the training process, a code reset and exponential moving average update strategy are adopted.

[0013] Specifically, the formula for summing the quantized feature vectors is as follows: ; in, For the position index of the latent feature sequence, For the first cascaded quantization layer indexes ( (for quantizing the number of layers) For the first Layer in position The quantized feature vector output at that point, For the same position The final quantized feature vector is obtained by summing the quantized feature vectors output by each quantization layer.

[0014] Specifically, the expression for the multi-granularity action fusion module is as follows: ; in, For multi-granularity branching / time-scale indexing, For the number of branches, For the first The latent feature sequence generated by the branch, This refers to the time scale / downsampling factor corresponding to this branch. Indicates by Will Perform time upsampling and align to operators of uniform length. For the first Scale fusion weights, The final fused latent feature representation is obtained by weighted summation of features at each scale after alignment.

[0015] To achieve the second objective of this invention, the following technical solution is provided: a three-dimensional human motion generation device for performing the steps of the above-described three-dimensional human motion generation method based on parallel multi-granularity Transformer.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: By employing a parallel, multi-granular architecture, both long-term and short-term modeling of action sequences is achieved. The coarse-grained module eliminates the "slippage" and "drift" phenomena in long sequence generation, ensuring the accuracy of the global trajectory; while the fine-grained module greatly enriches the detailed expressiveness of the generated actions.

[0017] By combining the efficient discretization of Residual VQ-VAE with the HRR mechanism of PMH-Transformer, high-fidelity motion reconstruction and detail restoration are achieved.

[0018] The action generation task is decoupled from a single "sequence prediction" to a two-dimensional refinement process of "multi-scale time" and "multi-level quantization", enabling the model to generate more expressive and physically consistent three-dimensional action sequences. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a three-dimensional human motion generation method based on parallel multi-granularity Transformer provided in this embodiment. Figure 2 This is a flowchart of the inference stage provided in this embodiment. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0021] like Figure 1 As shown in this embodiment, a three-dimensional human motion generation method based on parallel multi-granularity Transformer is provided, which includes the following steps: Obtain a 3D human motion dataset, which includes continuous 3D motion sequences and corresponding natural language text descriptions; The initial model is constructed, which includes a parallel multi-granularity action generation module, a multi-granularity action fusion module, and an action reconstruction and output module; The parallel multi-granularity action generation module includes L parallel branches with different granularities, and each branch contains generation stage one and generation stage two. The generation stage one extracts text from the input natural language text description to obtain the corresponding text embedding vector, and generates an initial token sequence that is randomly masked based on the text embedding vector using a masking modeling method. The second generation stage is used to supplement the randomly masked token sequence to output the corresponding complete token sequence; The multi-granularity action fusion module is used to map the complete token sequence output by multiple branches back to a continuous latent feature space and perform temporal alignment and fusion to output a latent feature representation; The motion reconstruction and output module generates a corresponding three-dimensional human motion sequence based on the input latent feature representation. The initial model was trained using a 3D human motion dataset to obtain an image generation model for generating high-fidelity 3D human motion sequences.

[0022] More specifically, in this embodiment, the following technical solution is adopted to construct a high-quality three-dimensional human motion dataset: Assume the original dataset provides an action sequence of M, where N is the number of frames. To enhance the model's perception of the physical characteristics of the action, this invention does not directly use joint angles, but instead extracts a set of redundant feature vectors x containing position, velocity, and rotation, with a feature dimension of D (D equals 263 in this embodiment).

[0023] Specific features include: root node angular velocity, root node linear velocity, root node height, local positions of all joints relative to the root node, joint local velocities, and joint rotations represented using 6D rotational notation. Additionally, binary contact tags for four foot keypoints are included to eliminate slippage. All features are normalized to obtain a standardized motion sequence.

[0024] To address the difficulty of generating continuous spatial data, this embodiment introduces and trains a residual vector quantization variational autoencoder, which employs a one-dimensional convolutional neural network combined with residual connections. The input is a standardized action sequence X, and the output is a latent feature sequence Z. The encoder includes downsampling layers with a total downsampling factor of r (set to 4 in this embodiment), meaning that every 4 frames of action are compressed into one latent feature vector. This invention uses V cascaded codebooks, and the quantization process is as follows: For the feature vector z, the nearest neighbor vector is first found in the first layer codebook. The calculation formula is as follows: After V layers of quantization, the original features are represented as a set of V discrete indices. During decoding, the original features are approximated by the sum of the quantization vectors from all layers, calculated as follows: The decoder is the inverse process of the encoder, upsampling features and mapping them back to the action space. The training loss function includes reconstruction loss, commitment loss, and codebook update components.

[0025] The loss calculation formula corresponding to the above process is as follows: .

[0026] In addition, the system receives natural language text descriptions input by the user and uses a pre-trained text encoder of a graph-text multimodal model (such as CLIP-ViT-B / 32) to extract features from the text, resulting in a fixed-length text embedding vector. This vector is rich in semantic information of the text and will be used as a global condition for subsequent generative models.

[0027] For the initial model, a generative architecture with L parallel branches is constructed, each focusing on a specific temporal granularity. Specifically, this includes: ① Multi-granular input construction: For the l-th branch, the action token sequence is downsampled in the temporal dimension according to a preset temporal downsampling factor s. The coarse-grained branch handles short sequences, has a large sensing field, and is responsible for planning the global trajectory; the fine-grained branch handles long sequences, retains all frames, and is responsible for filling in local details. ② Parallel multi-granular hierarchical processing: Each branch contains a PMH-Transformer module, which performs a two-stage generation task. The first stage: initial sequence generation, using the idea of ​​mask modeling, guides the model to predict the first layer of token sequences, which are randomly masked. The second stage: hierarchical residual refinement, based on the base sequence generated in the first stage, the model further predicts the token sequences of subsequent residual layers. This stage utilizes the dependencies between layers to gradually correct quantization errors and add subtle action textures.

[0028] In this embodiment, L is set to three parallel branches: Branch 1 is a coarse-grained branch with a downsampling factor s1 = 4, responsible for capturing the global trajectory. Branch 2 is a medium-grained branch with a downsampling factor s2 = 2, responsible for capturing limb movement transitions. Branch 3 is a fine-grained branch with a downsampling factor s3 = 1, responsible for capturing high-frequency details. Each branch contains two generation stages: Stage 1, Initial Sequence Generation (ISG). This stage only predicts the token sequence of the first layer quantized codebook. The input is a randomly masked token sequence and text conditional embeddings. The model adopts a Transformer Encoder structure, and the output is the token ID of the predicted mask position. Stage 2, Hierarchical Residual Refinement (HRR). This stage predicts the remaining V-1 layer residual token sequences. The input includes the embedding vector of the previously predicted token sequence, the hierarchical indicator embedding vector, and the text conditional embeddings. This stage uses the features of the Base Layer as context and outputs the token prediction results for all remaining layers.

[0029] The token sequences generated by each parallel branch are mapped back to a continuous latent feature space, and then temporally aligned and fused. First, the discrete token sequences generated by each branch are looked up to obtain the corresponding quantized feature vectors. Then, for each branch's feature vector, based on its corresponding downsampling factor, nearest neighbor interpolation or linear interpolation is used to upsample the temporal dimension, restoring it to the time length of the original action sequence. Finally, the upsampled feature vectors from all branches are weighted and summed (e.g., directly added or fused using learnable weight coefficients) to obtain the final fused latent feature representation. This feature integrates the global stability of the coarse-grained branches and the high-frequency details of the fine-grained branches.

[0030] like Figure 2 As shown, in this embodiment, after all branches complete inference, L sets of prediction results are obtained. During the fusion process, each set of discrete token sequences is mapped back to a continuous feature space to obtain a feature sequence Z1. Then, nearest neighbor interpolation is used to stretch the time dimension of feature sequence Z1 by a factor of s1, aligning the feature lengths of all branches. Finally, the feature vectors upsampled from all branches are summed using a weighted average. The calculation formula is as follows: .

[0031] The latent feature representations obtained by fusion are input into the residual vector quantization variational autoencoder, and after passing through deconvolution layers and nonlinear activation layers, a high-fidelity three-dimensional human motion sequence is finally reconstructed.

[0032] In this embodiment, a "Codebook Reset" and "Exponential Moving Average" (EMA) update strategy are employed. To prevent some codewords in the codebook from "dead" due to not being selected, codewords with low usage are periodically detected during training and reset to feature vectors randomly selected from the current batch. At the same time, EMA is used instead of direct gradient descent to update the codebook vectors to ensure the stability of the codebook distribution.

[0033] The PMH-Transformer module employs a "Mixture of Experts" (MoE) design concept. In this embodiment, "expert" does not refer to the sparse activation of network weights, but rather to an explicit division of labor in the structure: coarse-grained branches act as "global trajectory experts," while fine-grained branches act as "local detail experts." This design, through a parallel physical architecture, forces the model to learn features at different levels of abstraction, avoiding feature interference within a single model.

[0034] During the training phase, the Initial Sequence Generation (ISG) phase employs a dynamic mask rate, which varies with the training process in a cosine distribution.

[0035] The total loss function is a weighted average of the ISG loss and the HRR loss. During the inference phase, an iterative decoding strategy is employed. First, a full mask sequence is initialized. In each iteration, the model predicts the probability distribution of all mask positions, retaining only tokens with confidence levels above a threshold, while keeping the remaining positions masked and continuing prediction in the next round. Once the initial sequence is generated, the HRR model is run to fill in the details of all residual layers in one go.

[0036] During inference, a "confidence-based iterative decoding" strategy was employed. Instead of outputting the result all at once, the model iterates through multiple rounds (e.g., 10 rounds). In each round, the model predicts the probability distribution of all masked positions, retaining only the tokens with the highest confidence, and masking the remaining positions for prediction in the next round. This strategy allows the model to utilize already generated, highly deterministic contextual information to assist in generating more uncertain parts, significantly improving the coherence of long sequence generation.

[0037] In the hierarchical residual refinement stage, "hierarchical conditional encoding" is introduced. When predicting the residual of layer v, the model receives not only textual conditions but also an embedding vector indicating the current level v, as well as the accumulated feature sum of the previous v-1 layers. This allows the same Transformer network to be reused for prediction of different residual layers, improving parameter efficiency.

[0038] This example also provides a three-dimensional human motion generation device for performing the steps of the three-dimensional human motion generation method based on parallel multi-granularity Transformer provided in the above embodiments.

[0039] To comprehensively and objectively verify the effectiveness of this invention, this embodiment conducts a comprehensive comparative experiment based on KIT-ML, a standard text-action dataset widely used in the field of human motion generation. The experimental design aims to cover multiple dimensions such as generation quality, semantic alignment, and diversity, and compares them with existing methods.

[0040] This embodiment is based on experiments using the KIT-ML text-action dataset, which was proposed in the paper "The KITmotion-language dataset" (authors M Plappert, C Mandery, T Asfour, published in 2016). The dataset contains 3,911 selected action sequences and 6,278 corresponding text descriptions.

[0041] Existing Method 1: The method in the paper "Executing your commands via motion diffusion inlatent space" (authors Chen, Xin, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen and Gang Yu, published at the 2023 IEEE / CVF conference on computer vision and pattern recognition) performs a diffusion process in the latent space to predict action sequences.

[0042] The second existing method is the one in the paper “Mmm: Generative masked motion model” (authors Pinyoanuntapong, Ekkasit, Pu Wang, Minwoo Lee and Chen Chen, published at the 2024 IEEE / CVF Conference on Computer Vision and Pattern Recognition). This method uses a masking modeling strategy to learn spatiotemporal dependencies by filling in the masked motion fragments.

[0043] The third existing method is the one in the paper “Rethinking Diffusion for Text-Driven Human MotionGeneration: Redundant Representations, Evaluation, and Masked Autoregression” (authors Meng, Zichong, Yiming Xie, Xiaogang Peng, Zeyu Han, and Huaizu Jiang, published at the 2025 IEEE / CVF Conference on Computer Vision and Pattern Recognition), which uses a diffusion model to generate three-dimensional human motion sequences.

[0044] This invention: The method of this embodiment.

[0045] The experiments used Fréchet Inception Distance (FID), R-Precision, and MultiModalDistance (MultiModal Dist) as evaluation metrics. Specifically, this invention follows the standard evaluation protocol established in existing work. For each test case, the action generation process was repeated 20 times, and the average value and 95% confidence interval of these results were calculated, as shown in Table 1.

[0046]

[0047] Furthermore, the terms "upper," "lower," "inner," "outer," "front," and "rear" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0048] Of course, the above description is only a specific embodiment of the present invention and is not intended to limit the scope of the present invention. All equivalent changes or modifications made to the structure, features and principles described in the claims of the present invention should be included in the scope of the claims of the present invention.

[0049] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating 3D human motion based on parallel multi-granularity Transformer, characterized in that, Includes the following steps: Obtain a 3D human motion dataset, which includes continuous 3D motion sequences and corresponding natural language text descriptions; The initial model is constructed, which includes a parallel multi-granularity action generation module, a multi-granularity action fusion module, and an action reconstruction and output module; The parallel multi-granularity action generation module includes L parallel branches with different granularities, and each branch contains generation stage one and generation stage two. The generation stage one extracts text from the input natural language text description to obtain the corresponding text embedding vector, and generates an initial token sequence that is randomly masked based on the text embedding vector using a masking modeling method. The second generation stage is used to supplement the randomly masked token sequence to output the corresponding complete token sequence; The multi-granularity action fusion module is used to map the complete token sequence output by multiple branches back to a continuous latent feature space and perform temporal alignment and fusion to output a latent feature representation; The motion reconstruction and output module generates a corresponding three-dimensional human motion sequence based on the input latent feature representation. The initial model was trained using a 3D human motion dataset to obtain an image generation model for generating high-fidelity 3D human motion sequences.

2. The method for generating three-dimensional human motion based on parallel multi-granularity Transformer according to claim 1, characterized in that, The action reconstruction and output module is constructed based on the decoder obtained through training. The training process of the decoder is as follows: A three-dimensional action sequence is input into a one-dimensional convolutional encoder to extract action features, and the action features are mapped into continuous feature vectors in the latent space. Continuous feature vectors are discretized using multi-layer cascaded vector quantizers to obtain quantized feature vectors for all layers. The quantized feature vectors of all layers are summed to obtain the final latent feature representation, and the latent feature representation is input into the decoder to reconstruct the corresponding 3D action sequence; The 3D motion sequences in the 3D human motion dataset are used to train the 1D convolutional encoder, decoder, and accompanying codebook to obtain a decoder for constructing the motion reconstruction and output module.

3. The method for generating three-dimensional human motion based on parallel multi-granularity Transformer according to claim 2, characterized in that, The motion features include root node angular velocity, root node linear velocity, root node height, local position of all joints relative to the root node, joint local velocity, and redundant feature vectors corresponding to joint rotations using 6D rotation representation.

4. The method for generating three-dimensional human motion based on parallel multi-granularity Transformer according to claim 2, characterized in that, The training loss function of the decoder includes reconstruction loss, commitment loss, and codebook update components.

5. The method for generating three-dimensional human motion based on parallel multi-granularity Transformer according to claim 2, characterized in that, During training, a code reset and exponential moving average update strategy is adopted.

6. The method for generating three-dimensional human motion based on parallel multi-granularity Transformer according to claim 2, characterized in that, The formula for summing the quantized feature vectors is as follows: ;in, For the position index of the latent feature sequence, For the first Each cascaded quantization layer index, To quantify the number of layers, For the first Layer in position The quantized feature vector output at that point, For the same position The final quantized feature vector is obtained by summing the quantized feature vectors output by each quantization layer.

7. The method for generating three-dimensional human motion based on parallel multi-granularity Transformer according to claim 1, characterized in that, The expression for the multi-granularity action fusion module is as follows: ;in, For multi-granularity branching / time-scale indexing, For the number of branches, For the first The latent feature sequence generated by the branch, This refers to the time scale / downsampling factor corresponding to this branch. Indicates by Will Perform time upsampling and align to operators of uniform length. For the first Scale fusion weights, The final fused latent feature representation is obtained by weighted summation of features at each scale after alignment.

8. A three-dimensional human motion generation device, characterized in that, The steps are for performing the three-dimensional human motion generation method based on parallel multi-granularity Transformer as described in any one of claims 1 to 7.

Images