A text-based human action content generation method and system

Abstract

The application discloses a kind of human action content generation method and system based on text, belong to computer graphics and artificial intelligence field.The technical solution points include: the ability of MambaTrans hybrid backbone network fusion Mamba model processing long sequence data and the ability of Transformer model capturing global dependence relationship;Introduce layered adaptive feature enhancement mechanism, including adaptive frame weighting module dynamically allocating frame weight and multi-scale feature fusion module fusing different time scale features;By residual vector quantization autoencoder, the action is decomposed into layered tokens, and the basic token is generated using the mask Transformer, and the residual token is generated layer by layer using MambaTrans.The application realizes higher generation quality, stronger layered modeling capability and faster inference speed, and is suitable for virtual reality, augmented reality, film animation production and game character control.

Description

Technical Field

[0001] This invention belongs to the fields of computer graphics and artificial intelligence, and particularly relates to a method and system for generating text-based human motion content. Background Technology

[0002] With the rapid development of multimodal large model technology, directly generating 3D human motion from text descriptions has become a highly anticipated research topic in computer graphics and artificial intelligence. This technology has significant application value in scenarios such as virtual reality, digital entertainment, and film animation production. Currently, the mainstream technical paradigms are mainly divided into two categories. The first category is based on diffusion models, which can generate fine motion sequences through iterative denoising processes. However, in applications requiring rapid response, such as real-time interactive virtual reality or large-scale animation production, this type of method is often computationally expensive and requires numerous denoising steps, making it difficult to meet practical needs in terms of inference speed. The second category is based on vector quantization, which converts continuous motion data into discrete token sequences, thereby simplifying complex regression tasks into sequence modeling tasks and improving efficiency. In vector quantization methods, residual vector quantization technology significantly improves the quality of generated motion through a hierarchical, coarse-to-fine optimization process.

[0003] However, existing hierarchical optimization strategies based on residual vector quantization suffer from a common technical limitation. They typically employ a single, homogeneous processing architecture to predict residuals across all levels, ignoring the objectively existing differences in information density and task focus between different residual levels. Specifically, when generating action sequences, predicting low-level residuals requires the model to focus on long-term global information to determine the basic trajectory and macroscopic structure of the action, while predicting high-level residuals requires stronger local perception capabilities to capture subtle dynamic changes such as limb swaying. In practical applications, such as generating animations containing complex continuous movements and fine end-effector poses, the homogeneous architecture of existing methods struggles to simultaneously maintain global coherence and the realism of local details, potentially resulting in generated actions lacking physical realism. The challenge in solving this multi-scale feature representation problem lies in designing a generative framework that can dynamically adapt to the needs of different task levels, maintaining efficient computational characteristics while accurately modeling complex action features from global contours to local details. Summary of the Invention

[0004] To address the aforementioned technical issues, this invention proposes a text-based method and system for generating human motion content, achieving higher generation quality, stronger layered modeling capabilities, and faster inference speed. It is applicable to virtual reality, augmented reality, film and animation production, and game character control.

[0005] To achieve the above objectives, this invention provides a text-based method for generating human motion content, comprising:

[0006] Receives natural language text input;

[0007] Generate a basic token sequence representing the macroscopic structure of human body movements based on the text input;

[0008] Using the base token sequence and the text input as conditions, residual token sequences are generated layer by layer using the MambaTrans hybrid backbone network;

[0009] When generating each layer of residual token sequence, a hierarchical adaptive feature enhancement mechanism is applied to perform adaptive frame weighting on the input features and multi-scale feature fusion on the output features.

[0010] Based on the base token sequence and all residual token sequences, the final human action sequence is synthesized.

[0011] Optionally, generating a basic token sequence representing the macroscopic structure of human body movements based on the text input includes: using a mask Transformer model, with the text embedding vector extracted from a pre-trained text encoder as a condition, generating the basic token sequence through iterative mask decoding operations; wherein, the iterative mask decoding operations include initializing the mask sequence, predicting high-confidence tokens and replacing the mask positions, and repeating until the sequence is filled.

[0012] Optionally, the base token sequence and the residual token sequence are obtained by processing continuous human motion data through a residual vector quantization autoencoder; the process of the residual vector quantization autoencoder includes: the encoder maps the motion sequence to a latent representation, the cascaded quantizer quantizes the residuals layer by layer and generates hierarchical discrete tokens, and the decoder aggregates all tokens to reconstruct the motion sequence.

[0013] Optionally, the process of using the MambaTrans hybrid backbone network includes: the network is composed of Mamba blocks and Transformer blocks stacked sequentially; the Mamba blocks use a state-space model and local one-dimensional convolutional branches to process long sequence dependencies and local features in parallel, and the Transformer blocks refine global action consistency through a self-attention mechanism.

[0014] Optionally, the process of generating residual token sequences layer by layer includes: for each level, the model input is composed of a concatenation of cumulative action representation, textual conditional embedding, and hierarchical index embedding; wherein, the cumulative action representation is the sum of the generated base token and the residual token, and the hierarchical index embedding is a learnable vector to distinguish different residual levels.

[0015] Optionally, the adaptive frame weighting process includes: generating a weight matrix of the same length as the input sequence through a multilayer perceptron, normalizing it with the sigmoid function, multiplying it element by element with the input features, and dynamically adjusting the frame weights to adapt to the global or local attention requirements of different residual levels.

[0016] Optionally, the multi-scale feature fusion process includes: using multiple parallel one-dimensional convolutional layers to extract features at different time scales, with the kernel size covering the short to medium term range; and weighting and aggregating the multi-scale feature outputs through an attention mechanism to enhance the coherent expression of action details.

[0017] Optionally, the process of synthesizing the final human motion sequence based on the base token sequence and all residual token sequences includes: adding all level tokens to obtain an aggregate representation, and mapping it to a fixed decoder of the residual vector quantization autoencoder as continuous three-dimensional human motion data.

[0018] On the other hand, to achieve the above objectives, the present invention also provides a text-based human motion content generation system, including:

[0019] The text encoding module is used to receive natural language text input and generate text embedding vectors;

[0020] The motion quantization module is used to represent continuous human motion data as a hierarchical discrete token sequence through a residual vector quantization autoencoder;

[0021] The basic action generation module is used to generate a basic token sequence representing the macroscopic structure of human actions based on the text embedding vector.

[0022] The residual refinement module is used to generate residual token sequences layer by layer using the MambaTrans hybrid backbone network, based on the base token sequence and the text embedding vector, and to apply a hierarchical adaptive feature enhancement mechanism.

[0023] The motion synthesis module is used to synthesize the final human motion sequence based on the base token sequence and all residual token sequences.

[0024] Technical advantages of this invention: This invention discloses a text-based method and system for generating human motion content. Through the synergy of Mamba and Transformer, it can better preserve fine temporal details and optimize the coherence of global actions, significantly improving the physical realism of the generated actions. By introducing adaptive frame weighting and multi-scale feature fusion modules, this invention can dynamically process features according to the characteristics of different residual levels, achieving more precise control over actions from coarse outlines to fine details. Benefiting from the linear computational complexity of the Mamba model, the framework of this invention demonstrates good inference efficiency while improving generation quality. Attached Figure Description

[0025] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0026] Figure 1 This is a flowchart illustrating a text-based method for generating human motion content according to an embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of the structure of a text-based human motion content generation system according to an embodiment of the present invention. Detailed Implementation

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0030] This invention proposes an innovative generative framework based on hierarchical quantization and hybrid sequence modeling. The core idea of ​​this framework is to deconstruct the continuous and high-dimensional action generation task into a series of coarse-to-fine, discrete token prediction problems. First, a residual quantization autoencoder (RVQ-VAE) maps continuous action data into a hierarchical, discrete latent space. Then, a mask-based Transformer network is used to generate "basic tokens" representing the overall structure of the action based on the semantic information of the input text. Finally, a novel Mamba-Transformer hybrid architecture (MambaTrans) is introduced to predict "residual tokens" layer by layer, using the previously generated basic tokens and text as conditions, adding high-frequency details and physical realism to the action. This method aims to generate high-quality 3D human motion sequences that are highly consistent with the semantics of the text description, rich in action details, and exhibit natural temporal coherence.

[0031] This invention can be widely applied to various cutting-edge fields such as virtual reality (VR), augmented reality (AR), film and animation production, game character control, digital human interaction, and human-computer interaction. For example, in film production, animators can quickly generate basic animations directly from text descriptions, greatly improving production efficiency. In virtual reality applications, users can drive the movements of their virtual avatars in real time through voice or text commands, creating a more immersive interactive experience.

[0032] The execution environment for this invention uses a workstation or server with high-end computing resources. Specific configuration recommendations are as follows: Central Processing Unit (CPU): Intel Core i9-13900K or AMD Ryzen 9 7950X, with high clock speed and multi-core processing capabilities to efficiently handle tasks such as data preprocessing and model scheduling. Memory (RAM): 128GB DDR5 or higher, ensuring sufficient memory space when processing large-scale datasets (such as HumanML3D) and training complex deep learning models. Graphics Processing Unit (GPU): NVIDIA GeForce RTX 3090 or NVIDIA A100 TensorCore GPU, with at least 24GB of video memory. The GPU is the core hardware for model training and inference in this invention, and its powerful parallel computing capabilities are key to accelerating deep learning operations. Software Environment: The operating system can be Ubuntu 20.04 LTS or higher. The core development framework uses PyTorch 2.0 or higher, utilizing CUDA and cuDNN libraries for GPU acceleration. The programming language is primarily Python 3.9+. In addition, it relies on several Python libraries related to scientific computing and deep learning, such as NumPy, SciPy, Transformers, and CLIP. Provided that basic computing resources are available, this invention can also be based on a distributed training environment, such as using a cluster of servers equipped with multiple high-performance GPUs, and performing parallel training on multiple machines and GPUs through frameworks such as Horovod or PyTorchDDP (DistributedDataParallel) to further shorten the training cycle of the model on ultra-large-scale datasets.

[0033] like Figure 1 As shown, this embodiment provides a text-based method for generating human motion content, including: receiving natural language text input;

[0034] Generate a basic token sequence representing the macroscopic structure of human body movements based on the text input;

[0035] Using the base token sequence and the text input as conditions, residual token sequences are generated layer by layer using the MambaTrans hybrid backbone network;

[0036] When generating each layer of residual token sequence, a hierarchical adaptive feature enhancement mechanism is applied to perform adaptive frame weighting on the input features and multi-scale feature fusion on the output features.

[0037] Based on the base token sequence and all residual token sequences, the final human action sequence is synthesized.

[0038] Further, data preparation and action representation:

[0039] Construct a dataset suitable for model training, containing paired text descriptions and 3D human motion sequences.

[0040] Data Source Acquisition: Training and evaluation were primarily conducted on two industry-recognized large-scale action-language benchmark datasets: HumanML3D and KIT-ML. HumanML3D: This is a large-scale, highly diverse dataset, integrating 14,616 action sequences from multiple data sources such as AMASS and HumanAct12, accompanied by 44,970 natural language descriptions. The average length of the action sequences is 7.1 seconds, and the frame rate was uniformly downsampled to 20 FPS to suit model processing. The diversity of this dataset provides a solid foundation for the model to learn complex text-action mapping relationships. KIT-ML: Compared to HumanML3D, this dataset is smaller, containing 3,911 actions and 6,278 text annotations. It is mainly used to evaluate the model's generalization ability and learning efficiency with limited data.

[0041] Motion data representation: The captured raw 3D human pose data needs to be converted into a numerical representation that the model can process. This invention employs a comprehensive representation method that includes global human motion and local joint poses. A length of... Action sequence of frames It can be represented as a tensor ,in It is the feature dimension of each frame. This feature vector Specifically, this includes: Root joint information: This includes the root joint's global position (3 dimensions) and global orientation (usually represented using 6D rotation, 6 dimensions) in 3D space. Local joint rotation: The rotation of all other joints except the root joint relative to their parent joint. This is typically represented using quaternions (4 dimensions) or 6D rotation representation (6 dimensions) to avoid gimbal lock issues. Joint velocity: To better model the dynamic characteristics of the motion, the feature vector also includes the position and rotational velocity and acceleration information of each joint. Foot contact state: A binary flag indicating whether the heel and toe of the left and right feet are in contact with the ground (4 dimensions). This is crucial for generating physically realistic walking, running, and other motions. Ultimately, the feature dimension of each frame is the sum of all the above component dimensions.

[0042] Text Data Processing: Natural language text descriptions paired with action sequences need to be encoded into semantic feature vectors. This invention employs a pre-trained, powerful CLIP model text encoder. Text Input: The original text description, such as "a person walks forward, then turns left and stops". Encoding Process: The text is input into the CLIP text encoder, which internally uses a Transformer-based architecture. It maps the text string to a high-dimensional, fixed semantic embedding space, obtaining text feature vectors. ,in This is the embedding dimension of the CLIP model. This semantic feature vector The core semantic information captured in the text description will serve as a key conditional input for the subsequent generative model.

[0043] Furthermore, generating a basic token sequence representing the macroscopic structure of human body movements based on the text input includes: using a mask Transformer model, with the text embedding vector extracted from a pre-trained text encoder as a condition, generating the basic token sequence through iterative mask decoding operations; wherein, the iterative mask decoding operations include initializing the mask sequence, predicting high-confidence tokens and replacing the mask positions, and repeating until the sequence is filled.

[0044] Specifically, the implementation process of this embodiment includes:

[0045] Basic Action Generation: After actions are quantified into hierarchical tokens, the generation task is broken down into two stages: first, basic tokens are generated. Then, a residual token is generated.

[0046] Model Architecture: This step uses a masked Transformer, whose architecture is similar to BERT. This is a bidirectional Transformer model that can simultaneously consider the contextual information of all positions in a sequence.

[0047] Training process: The training process follows the mask generation modeling paradigm. Input: During training, the model input consists of two parts: (1) the “real” base token sequence obtained from RVQ-VAE. (2) CLIP text embedding corresponding to the action Masking operation: In the process of masking... Before being fed into the model, a portion of the tokens are randomly replaced with a special masked token according to a specific scheduling function (e.g., cosine scheduling). Prediction objective: The model's task is to predict the outcome of a given set of unmasked tokens and text conditions. In this case, predict the original values ​​of all masked tokens. Loss function: Use the cross-entropy loss function to measure the difference between the token probability distribution predicted by the model and the true tokens.

[0048] Inference Process: The generation process is iterative and non-autoregressive, making it much faster than traditional autoregressive models. Initialization: Given a text description, a sequence consisting entirely of mask tokens with the same length as the desired action is generated first. Iterative Decoding: The current sequence and text embedding are then performed. The data is fed into a Mask-Transformer. The model predicts a probability distribution of a token for each mask position. Confidence sampling: Based on the predicted probabilities, a subset of mask positions with the highest confidence are selected, and they are replaced with the predicted tokens. Remasking: The remaining mask tokens, as well as the generated tokens with lower confidence, are transformed back into masks. These steps are repeated until all positions are filled with specific tokens, or a preset number of iterations is reached. Through this process, the model first generates the overall outline of the action, then gradually fills in the details, ultimately obtaining a complete basic token sequence. .

[0049] Furthermore, the basic token sequence and the residual token sequence are obtained by processing continuous human motion data through a residual vector quantization autoencoder; the process of the residual vector quantization autoencoder includes: the encoder maps the motion sequence to a latent representation, the cascaded quantizer quantizes the residuals layer by layer and generates hierarchical discrete tokens, and the decoder aggregates all tokens to reconstruct the motion sequence.

[0050] Specifically, the implementation process of this embodiment includes:

[0051] Hierarchical motion quantization: Its goal is to quantize continuous, high-dimensional action sequences. This transforms the sequence into a discrete, hierarchical token sequence. This greatly simplifies the generation task, transforming it from a complex regression problem into a more tractable sequence generation problem, similar to natural language generation. This step employs a residual vector quantization variational autoencoder.

[0052] The RVQ-VAE architecture consists of three main parts: an encoder A decoder and a series of cascaded, containing Individual residual quantizers Quantization layer.

[0053] Encoding and Quantization Process: Initial Encoding: First, the action sequence fed into encoder This encoder typically consists of a series of one-dimensional convolutional layers and non-linear activation functions, used to extract temporal features of actions and map them to a low-dimensional continuous latent representation. .in It is the length of the sequence after downsampling. .

[0054] Layer-by-layer residual quantization: latent representation They were then sent to The decomposition is performed in a quantizer with multiple levels of connectivity. This process is iterative: in the first... Layer (from) arrive ), current residual input Through quantizer Quantization is performed. The quantizer searches for values ​​in its codebook that match... The closest code vector at each time step is used to obtain the discrete token. The input of the next layer It is the current residual Rather than the quantitative results The difference between them (i.e., the residual). This process can be formally represented as:

[0055] ;

[0056] .

[0057] Hierarchical tokens: Through this process, the original sequential actions are broken down into a single token containing... Discrete token set of the layer .in, Known as the base token, it captures low-frequency information about the action, such as the overall movement trajectory and basic posture. Known as residual tokens, they capture higher-frequency details layer by layer, such as subtle limb movements and fine-tuning of posture.

[0058] Decoding process: Decoder The task is to create a hierarchical discrete token set. Reconstructing a continuous sequence of actions Token aggregation: First, all discrete tokens are summed element-wise to obtain the original latent representation. An approximate reconstruction Reconstruction action: The aggregated representation Sent into the decoder (Its structure is usually similar to that of an encoder) (symmetric), ultimately generating the reconstructed action sequence. ,

[0059] .

[0060] The training objective of RVQ-VAE is to minimize the original motion. and reconstruction actions The reconstruction loss (e.g., mean squared error) includes a codebook learning loss term. After training, the encoder and decoder of RVQ-VAE will be fixed for subsequent training of generative models.

[0061] Furthermore, the process of using the MambaTrans hybrid backbone network includes: the network is composed of Mamba blocks and Transformer blocks stacked sequentially; the Mamba blocks use a state-space model and local one-dimensional convolutional branches to process long sequence dependencies and local features in parallel, and the Transformer blocks refine global action consistency through a self-attention mechanism.

[0062] The process of generating residual token sequences layer by layer includes: for each level, the model input is composed of a cumulative action representation, a textual conditional embedding, and a hierarchical index embedding; where the cumulative action representation is the sum of the generated base tokens and residual tokens, and the hierarchical index embedding is a learnable vector to distinguish different residual levels.

[0063] Specifically, the implementation process of this embodiment includes:

[0064] Residual detail refinement: Based on the generation of base tokens representing the macroscopic structure of the action. Then, residual tokens are generated layer by layer. This adds high-fidelity details to the motion.

[0065] Considering the high demands placed on residual details regarding long-range temporal dependencies (such as body coordination) and local dynamic changes (such as fine foot movements), this invention innovatively employs a Mamba-Transformer hybrid architecture. Architecture composition: The model consists of... It is made up of stacked blocks, of which the first one is made up of stacked blocks. The first block is a Mamba block, then... Each block is a standard Transformer block ( ).

[0066] Design Concept: Mamba is an emerging State-Space Model (SSM) that offers linear computational complexity and efficient hardware utilization when processing long sequences. Its selective mechanism allows it to dynamically focus on or ignore information based on the input, making it ideal for capturing long-range temporal continuity in action sequences. The Transformer's self-attention mechanism excels at capturing global relationships between any two positions in a sequence, making it suitable for refining the entire action sequence for global consistency in the final stage. Enhanced Mamba Block Design: To better capture local patterns, the Mamba block in this invention employs a dual-path parallel design. One path is a standard State-Space Model (SSM) branch for modeling long-range dependencies; the other is a purely one-dimensional convolutional branch for extracting local features. The outputs of the two paths are concatenated and fused through a linear layer.

[0067] Layer-by-layer generation process: MambaTrans generates residual tokens in a layer-by-layer, autoregressive manner. In predicting the... Layer residual token hour( From 1 to Input Construction: The model's input consists of three parts concatenated: Cumulative Action Representation: The sum of all previously generated tokens, i.e. This provides the current layer's predictions with a generated, coarse-to-fine action context. Text Conditions: CLIP Text Embedding This ensures that the generation of details always remains consistent with the semantics of the text.

[0068] Hierarchical index embedding: The hierarchical index currently being predicted It will be transformed into a learnable embedding vector. This allows the model to recognize at what level of detail it is working and to adopt different strategies.

[0069] .

[0070] Feature processing: The constructed input The data is fed into the MambaTrans network. It first undergoes efficient temporal feature extraction via the Mamba block, then the Transformer block models and refines global relationships, ultimately outputting the first... Layer residual token The prediction.

[0071] Iterative refinement: This process will be repeated. Each time, a layer of detail is added based on the previous one, until a complete residual token set is generated. .

[0072] Furthermore, the adaptive frame weighting process includes: generating a weight matrix of the same length as the input sequence through a multilayer perceptron, normalizing it with the Sigmoid function, multiplying it element-by-element with the input features, and dynamically adjusting the frame weights to adapt to the global or local attention requirements of different residual levels.

[0073] The process of multi-scale feature fusion includes: using multiple parallel one-dimensional convolutional layers to extract features at different time scales, with the kernel size covering the short to medium term range; and weighting and aggregating the multi-scale feature outputs through an attention mechanism to enhance the coherent expression of action details.

[0074] Specifically, the implementation process of this embodiment includes:

[0075] Hierarchical perceptual feature enhancement: In order to enable the MambaTrans model to better adapt to the prediction needs of residuals at different levels (low-level residuals focus on the global, while high-level residuals focus on the local), this invention introduces two auxiliary modules: adaptive frame weighting and multi-scale feature fusion.

[0076] Adaptive Frame Weighting (AFW): As a preprocessing module of MambaTrans, AFW aims to adjust the weighting of frames based on the currently predicted level. Dynamically adjust input features The importance of each frame. Mechanism: [This is related to the mechanism of...] A weight matrix of the same length as the input sequence is generated using a lightweight multilayer perceptron (MLP) and a sigmoid activation function. Then, this weight matrix is ​​compared with the original input. Perform element-wise multiplication.

[0077] ;

[0078] .

[0079] Effects: When predicting low-level residuals, the weights may be more evenly distributed, allowing the model to focus on the entire sequence; when predicting high-level residuals, the weights may be concentrated on a few key frames, allowing the model to focus on characterizing the instantaneous dynamics of these frames.

[0080] Multi-scale Feature Fusion (MSFF): As a post-processing module of MambaTrans, MSFF aims to refine the features output by MambaTrans, making them contain richer multi-scale temporal information before final token prediction. Mechanism: The output features of MambaTrans are fed in parallel into multiple convolutional layers with different kernel sizes (e.g., ... One-dimensional convolutional layers. Small convolutional kernels are used to capture short-term, instantaneous dynamic changes. Large convolutional kernels are used to capture medium- to long-term motion rhythms. Feature aggregation: The outputs of all parallel convolutions are concatenated and weighted and summed through an attention mechanism, ultimately fusing into a feature representation rich in multi-scale information for token prediction.

[0081] Furthermore, the process of synthesizing the final human motion sequence based on the base token sequence and all residual token sequences includes: adding all level tokens to obtain an aggregate representation, and mapping it to continuous three-dimensional human motion data through a fixed decoder of the residual vector quantization autoencoder.

[0082] Specifically, the implementation process of this embodiment includes:

[0083] Final motion synthesis and training objectives:

[0084] Action synthesis: When all levels of tokens After all the data are generated, they are all added together and then passed through a pre-trained RVQ-VAE decoder. Decoding yields the final high-fidelity 3D human motion sequence. ,

[0085] .

[0086] Overall training objective: The training of the entire generative model is carried out in stages.

[0087] Phase 1: Training RVQ-VAE. As described in Step 2, its loss function mainly includes reconstruction loss and codebook learning loss. Phase 2: Training Mask-Transformer. As described in Step 3, its loss function is the cross-entropy loss of predicting the mask's base tokens. Phase 3: Training MambaTrans. As described in Step 4, its loss function is the sum of the cross-entropy losses of predicting residual tokens layer by layer. In practice, these three phases can be performed independently or jointly fine-tuned end-to-end. The overall loss function is a weighted sum of the losses from each component, and gradient descent is used through optimizers such as AdamW to update the model parameters.

[0088] like Figure 2 As shown, this embodiment also provides a text-based human motion content generation system, including:

[0089] The text encoding module is used to receive natural language text input and generate text embedding vectors;

[0090] The motion quantization module is used to represent continuous human motion data as a hierarchical discrete token sequence through a residual vector quantization autoencoder;

[0091] The basic action generation module is used to generate a basic token sequence representing the macroscopic structure of human actions based on the text embedding vector.

[0092] The residual refinement module is used to generate residual token sequences layer by layer using the MambaTrans hybrid backbone network, based on the base token sequence and the text embedding vector, and to apply a hierarchical adaptive feature enhancement mechanism.

[0093] The motion synthesis module is used to synthesize the final human motion sequence based on the base token sequence and all residual token sequences.

[0094] Specifically, the implementation process of this embodiment includes:

[0095] This invention also provides a text-based human motion content generation system, which is a modular implementation of the above-described method. The system includes: a data preprocessing module, a text encoding module, a motion quantization module, a basic motion generation module, a residual refinement module, a feature enhancement module, a motion synthesis module, and a model training and inference module.

[0096] The data preprocessing module is responsible for loading raw motion data (such as BVH and FBX formats) and text data. It performs standardization on the motion data, including frame rate unification, skeleton alignment, and feature extraction (joint position, rotation, velocity, etc.). It creates data samples for each motion-text pair and performs batch processing for model training. This module can be implemented using a series of Python scripts, utilizing libraries such as bvh-python and numpy to process motion data and manage dataset indexing and loading.

[0097] Text Encoding Module: This module receives natural language text input and encodes it into a high-dimensional semantic feature vector using a pre-trained CLIP model. It directly integrates the pre-trained CLIPTextModel from the Hugging Face transformers library. This module provides a simple interface that takes a text string as input and outputs a fixed-dimensional tensor.

[0098] Action Quantization Module: Functionality: Implements RVQ-VAE. It contains an encoder submodule and a decoder submodule. During training, it encodes batches of continuous action data into hierarchical discrete tokens; during inference, its decoder reconstructs the continuous actions from the generated token sets. It is implemented using a custom `nn.Module` in PyTorch. The encoder and decoder consist of layers such as `nn.Conv1d` and `nn.ReLU`. The core vector quantization part can be modified based on existing open-source implementations to support residual quantization.

[0099] Basic Action Generation Module: Implements Mask-Transformer. It is responsible for generating a sequence of basic action tokens based on the semantic features output by the text encoding module. Implementation: Implement either PyTorch's nn.Transformer or a BERT-style nn.Module. The core of this module is a multi-head self-attention layer and a feedforward network. Additionally, iterative decoding logic must be implemented.

[0100] Residual Refinement Module: Implements the MambaTrans hybrid architecture. It is responsible for generating residual tokens layer by layer on top of the base token. This is a complex nn.Module containing multiple Mamba and Transformer blocks. The implementation of the Mamba block requires a dedicated SSM library, or you can implement a selective scanning mechanism yourself based on the paper. The Transformer block can directly use nn.TransformerEncoderLayer.

[0101] Feature Enhancement Module: Implements AFW and MSFF. This module is tightly coupled with the residual refinement module, dynamically adjusting and enhancing features at their input and output, respectively. AFW can be implemented by a simple nn.Sequential layer (containing nn.Linear and nn.Sigmoid). MSFF consists of multiple parallel nn.Conv1d layers and an attention aggregation layer.

[0102] Action Synthesis Module: During the inference phase, this module is responsible for collecting all generated tokens (base tokens and all residual tokens), adding them together, and calling the decoder in the action quantization module to complete the final action sequence synthesis. It mainly consists of logic control code, calling interfaces of other modules and processing data streams.

[0103] The model training and inference module manages the entire system lifecycle. In training mode, it's responsible for loading data, performing forward propagation, calculating loss, performing backpropagation, and updating model parameters. In inference mode, it receives text input and sequentially calls each generation module, ultimately outputting the generated action file (e.g., BVH format) or directly for real-time rendering. This is a top-level control script that uses PyTorch's training loop logic and integrates the AdamW optimizer and learning rate scheduler. The inference part is a separate function or class that encapsulates the complete generation process.

[0104] This invention discloses a text-based method and system for generating human motion content. Through the synergy of Mamba and Transformer, it better preserves fine temporal details and optimizes the coherence of global motion, significantly improving the physical realism of the generated motion. By introducing adaptive frame weighting and multi-scale feature fusion modules, this invention enables dynamic feature processing tailored to the characteristics of different residual levels, achieving more precise control over motion from coarse outlines to fine details. Benefiting from the linear computational complexity of the Mamba model, the framework of this invention demonstrates good inference efficiency while improving generation quality.

[0105] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A text-based method for generating human motion content, characterized in that, include: Receives natural language text input; Generate a basic token sequence representing the macroscopic structure of human body movements based on the text input; Using the base token sequence and the text input as conditions, residual token sequences are generated layer by layer using the MambaTrans hybrid backbone network; The process of generating residual token sequences layer by layer includes: for each level, the model input is composed of a cumulative action representation, a text conditional embedding, and a hierarchical index embedding; where the cumulative action representation is the sum of the generated base tokens and residual tokens, and the hierarchical index embedding is a learnable vector to distinguish different residual levels. When generating each layer of residual token sequence, a hierarchical adaptive feature enhancement mechanism is applied to perform adaptive frame weighting on the input features and multi-scale feature fusion on the output features. The adaptive frame weighting process includes: generating a weight matrix of the same length as the input sequence through a multilayer perceptron, normalizing it with the Sigmoid function, multiplying it element by element with the input features, and dynamically adjusting the frame weights to adapt to the global or local attention requirements of different residual levels. Based on the base token sequence and all residual token sequences, the final human motion sequence is synthesized, specifically including: adding all level tokens to obtain an aggregate representation, and mapping it to continuous three-dimensional human motion data through a fixed decoder of the residual vector quantization autoencoder.

2. The text-based human motion content generation method as described in claim 1, characterized in that, The generation of a basic token sequence representing the macroscopic structure of human body movements based on the text input includes: using a mask Transformer model, with the text embedding vector extracted from a pre-trained text encoder as a condition, generating the basic token sequence through iterative mask decoding operations; wherein, the iterative mask decoding operations include initializing the mask sequence, predicting high-confidence tokens and replacing the mask positions, and repeating until the sequence is filled.

3. The text-based human motion content generation method as described in claim 1, characterized in that, The basic token sequence and the residual token sequence are obtained by processing continuous human motion data through a residual vector quantization autoencoder; The residual vector quantization autoencoder process includes: the encoder mapping the action sequence to a latent representation, the cascaded quantizer quantizing the residuals layer by layer and generating hierarchical discrete tokens, and the decoder aggregating all tokens to reconstruct the action sequence.

4. The text-based human motion content generation method as described in claim 1, characterized in that, The process of using the MambaTrans hybrid backbone network includes: the network is composed of Mamba blocks and Transformer blocks stacked sequentially; the Mamba blocks use a state-space model and local one-dimensional convolutional branches to process long sequence dependencies and local features in parallel, and the Transformer blocks refine global action consistency through a self-attention mechanism.

5. The text-based human motion content generation method as described in claim 1, characterized in that, The process of multi-scale feature fusion includes: using multiple parallel one-dimensional convolutional layers to extract features at different time scales, with the kernel size covering the short to medium term range; and weighting and aggregating the multi-scale feature outputs through an attention mechanism to enhance the coherent expression of action details.

6. A text-based human motion content generation system, characterized in that, For implementing the text-based human motion content generation method as described in any one of claims 1-5, the system comprises: The text encoding module is used to receive natural language text input and generate text embedding vectors; The motion quantization module is used to represent continuous human motion data as a hierarchical discrete token sequence through a residual vector quantization autoencoder; The basic action generation module is used to generate a basic token sequence representing the macroscopic structure of human actions based on the text embedding vector. The residual refinement module is used to generate residual token sequences layer by layer using the MambaTrans hybrid backbone network, based on the base token sequence and the text embedding vector, and to apply a hierarchical adaptive feature enhancement mechanism. Specifically, for each level, the model input consists of a concatenation of cumulative action representation, text conditional embedding, and hierarchical index embedding; wherein, the cumulative action representation is the sum of the generated base tokens and residual tokens, and the hierarchical index embedding is a learnable vector to distinguish different residual levels; a weight matrix of the same length as the input sequence is generated through a multilayer perceptron, which is normalized by the Sigmoid function and multiplied element-wise with the input features to dynamically adjust the frame weights to adapt to the global or local attention requirements of different residual levels. The motion synthesis module is used to synthesize the final human motion sequence based on the base token sequence and all residual token sequences. Specifically, it includes: adding all level tokens to obtain an aggregate representation, and mapping it to continuous three-dimensional human motion data through a fixed decoder of the residual vector quantization autoencoder.

Images