A text-driven human motion generation method based on hierarchical differential discretization and attention

Abstract

The application discloses a text-driven human motion generation method based on hierarchical difference residual discretization and attention, and has the characteristics that the method comprises the following steps: constructing a motion discrete encoder and decoder based on hierarchical difference residual discretization, and reconstructing the sum of discrete feature vectors of each layer into a motion sequence through the decoder; constructing a global motion token predictor, and predicting root layer motion tokens through a Transformer; constructing an incremental motion token predictor, and predicting incremental layer motion tokens layer by layer under the guidance of a text semantic vector; and summing up the discrete feature vectors corresponding to the predicted base layer and each incremental layer motion token, and inputting the sum into the motion discrete decoder to reconstruct a three-dimensional human motion sequence. Compared with the prior art, the application has high-fidelity and editable motion generation, the visual fidelity and semantic consistency of the generated motion are good, the motion sequence conforms to the motion law, the problems of motion detail loss and text-motion semantic misalignment are effectively solved, and the application has good application prospect and commercial development value.

Description

Technical Field

[0001] This invention relates to the field of 3D human motion generation technology, specifically a text-driven human motion generation method based on hierarchical differential discretization and attention mechanisms. Background Technology

[0002] 3D human motion generation has wide applications in film and television special effects, virtual reality, game development, and military simulation training. Traditional motion creation methods mainly rely on optical or inertial motion capture systems to collect motion data, which is then refined and redirected by professional animators before being integrated into an animation state machine to drive the virtual character. Although this process can guarantee high motion quality, its data acquisition and processing costs are high, requiring specific specialized equipment and human intervention, making it difficult to meet the needs of rapid generation of diverse motions.

[0003] In recent years, the development of deep learning technology has provided new directions for 3D human motion generation. Guo et al., in their paper "Generating Diverse and Natural 3D Human Motions from Text" published in *Computer Vision and Pattern Recognition*, released the large-scale text-action dataset HumanML3D, establishing a two-stage autoregressive generation paradigm. Pinyoanuntapong et al., in their paper "MMM: Generative Masked Motion Model," proposed a text-driven motion generation method based on mask modeling, achieving high-speed, high-fidelity, and editable motion generation through motion tokenization and conditional masking Transformers. However, existing methods still have the following shortcomings: 1) In the process of vector discretization, a single discretization operation will lead to the truncation and loss of high-frequency motion details, resulting in a lack of smoothness and precision in the generated actions; 2) In terms of cross-modal alignment between text and actions, existing methods are not capable of modeling the global semantics of the text, which leads to confusion in the execution order of complex instructions in multiple stages; 3) In the process of mask symbol prediction, there is a lack of effective temporal causal constraints, which leads to inter-frame jitter and pose distortion in the generated action sequences that violate the laws of motion. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a text-driven human action generation method based on hierarchical covariant discretization and attention. This method employs a hierarchical covariant discretization action discrete encoder-decoder to encode a 3D human action sequence into a latent vector sequence. This sequence is then discretized layer by layer by a multi-level covariant discretization module to obtain multi-layer discrete action codes. The decoder then reconstructs the action sequence by summing the discrete feature vectors of each layer. A contrastive language-image pre-trained model is used to extract the global and local text semantic vectors of the text. Combined with a masking strategy and a bidirectional and time-series dependent attention masking mechanism, the root-layer action codes are predicted using a Transformer. The method then uses the discrete feature vectors of each preceding layer to generate the action sequence. Using the sum of embeddings and layer indices as input, incremental layer action codes are predicted layer by layer under the guidance of text semantic vectors. The discrete feature vectors corresponding to the predicted base layer and each incremental layer action code are summed and input into the action discrete decoder to reconstruct the 3D human action sequence. This results in the generated actions lacking smoothness and precision, achieving high-speed, high-fidelity, and editable motion generation. The generated actions have high visual fidelity and semantic consistency, and the action sequences conform to the laws of motion, avoiding inter-frame jitter and posture distortion that violate the laws of motion. This effectively alleviates problems such as loss of motion details and insufficient text-action semantic alignment, and has good application prospects and commercial development value.

[0005] The specific technical solution for achieving the objective of this invention is: a text-driven human action generation method based on hierarchical differential discretization and attention, characterized in that the method specifically includes the following steps:

[0006] Step 1: Construction and Training of the Action Discrete Encoder / Decoder

[0007] Step 1-1: Construct a variational autoencoder based on hierarchical differential discretization as an action discrete encoder-decoder. The encoder encodes a 3D human pose sequence of frame length N into a latent vector sequence of length n, where... This is the downsampling factor.

[0008] Step 1-2: Constructing the... Each discretization layer contains a discrete-level differential discretization module, and each discretization layer contains a module of size [size missing]. The learnable motion feature dictionary, where K is the number of discrete feature vectors and d is the dimension of the discrete feature vectors.

[0009] Steps 1-3: From the 0th level difference Starting with the latent vector output by the encoder, the layer-by-layer covariance is mapped to the nearest discrete feature vector in the motion feature dictionary through nearest neighbor lookup. And calculate the level difference of the next level. .

[0010] Steps 1-4: Sum the discrete feature vector sequences output by all discretization layers as an approximation of the latent vector sequence, input it into the decoder to reconstruct the 3D human action sequence, and use the sum of the action reconstruction loss and the embedding constraint loss of each discretization layer as the training objective to train the action discrete encoder and decoder.

[0011] Step 2: Construction and Training of the Global Action Code Predictor

[0012] Step 2-1: Use the contrastive language-image pre-trained model to process the input text description and generate global text semantic vectors and local text semantic vectors.

[0013] Step 2-2: Mask the root action code of the action discrete codec output by determining the mask rate according to the cosine scheduling function. Replace the selected token with the mask code with a probability of 80%, replace it with a random token with a probability of 10%, and keep it unchanged with a probability of 10%.

[0014] Steps 2-3: Prepend the global text semantic vector to the damaged root layer action code to form an action context feature sequence, which serves as the query input for the Transformer, and the local text semantic vector serves as the key and value input.

[0015] Steps 2-4: Construct a global awareness attention mask to enable bidirectional mutual attention between unmasked symbols, and construct a temporally dependent attention mask so that masked symbols only pay attention to historical masked symbols and all unmasked symbols.

[0016] Steps 2-5: Predict the complete root action code using Transformer.

[0017] Step 3: Construction and training of the incremental action symbol predictor

[0018] Step 3-1: For the h-th level difference token ( ), from the 0th layer of the preceding sequence to the 1st layer The hierarchical discrete feature vectors are mapped to embedding vectors and then summed. This summation is then combined with the global text semantic vector, the local text semantic vector, and the current layer index. Input Transformer and predict action symbols for each incremental layer layer.

[0019] Step 3-2: The Transformer architecture of the incremental action symbol predictor is consistent with that of the global action symbol predictor, but without introducing a masking mechanism.

[0020] Step 4: Sum the discrete feature vectors corresponding to the action codes of the base layer and each incremental layer, and input them into the action discrete decoder to reconstruct the three-dimensional human action sequence, so as to obtain a high-fidelity, editable text-driven human action generation with high visual fidelity and semantic consistency, and whose action sequence conforms to the law of motion.

[0021] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:

[0022] 1) This invention uses hierarchical differential discretization technology to encode human motion features at multiple levels, which effectively reduces information loss in a single discretization process and improves the reconstruction accuracy of high-frequency complex motion details.

[0023] 2) This invention enhances the cross-modal alignment capability between global text semantics and action features by placing the global text semantic vector before the action code sequence and combining it with a cross-attention mechanism, enabling the model to accurately understand and execute multi-stage complex text instructions.

[0024] 3) This invention constructs a dual constraint mechanism of global awareness attention mask and temporal dependent attention mask, which applies temporal causal constraints while preserving global context information, effectively suppressing inter-frame jitter and pose distortion in the mask symbol prediction process.

[0025] 4) This invention adopts a decoupled architecture of a basic-level differential dual-layer token predictor. The global action code predictor is responsible for global timing planning, while the incremental action code predictor is responsible for layer-by-layer detail completion, thus achieving a balance between generation quality and inference efficiency.

[0026] 5) This invention can efficiently generate three-dimensional human motion sequences with high visual fidelity and semantic consistency based on text descriptions, and is superior to existing methods in core indicators such as motion realism and text-action semantic alignment. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the overall architecture of the present invention;

[0028] Figure 2 This is a schematic diagram of a discrete motion encoder module;

[0029] Figure 3 This is a schematic diagram of the hierarchical differential discretization module;

[0030] Figure 4 This is a schematic diagram of the action discrete decoder module;

[0031] Figure 5 This is a schematic diagram of the input processing of the global action symbol predictor in the method of the present invention;

[0032] Figure 6 This is a schematic diagram illustrating an example of an attention mask.

[0033] Figure 7 This is a schematic diagram of the global action symbol predictor network architecture;

[0034] Figure 8This is a diagram illustrating the reasoning process;

[0035] Figure 9 The figure shows the experimental results comparing the generation of this invention with other generation methods. Detailed Implementation

[0036] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] Example 1

[0038] See Figure 1 A text-driven human action generation method based on hierarchical differential discretization and attention includes the following steps:

[0039] Step 1: Construction and Training of the Action Discrete Encoder / Decoder

[0040] 1-1: Construct a variational autoencoder based on hierarchical differential discretization as an action discrete encoder / decoder.

[0041] 1-2: The encoder encodes a 3D human pose sequence of frame length N into a latent vector sequence of length n, where... Let be the downsampling factor, set to 4. Construct a system containing... Each discretization layer contains a learnable motion feature dictionary of size 512×512.

[0042] 1-3: From level 0, the difference Starting with the latent vector output by the encoder, the layer-by-layer covariance is mapped to the nearest discrete feature vector in the motion feature dictionary through nearest neighbor lookup. And calculate the level difference of the next level. .

[0043] 1-4: Sum the discrete feature vector sequences output by all discretization layers as an approximation of the latent vector sequence, and input it into the decoder to reconstruct the three-dimensional human motion sequence.

[0044] Step 2: Construction and Training of the Global Action Code Predictor

[0045] 2-1: The input text description is processed using a contrastive language-image pre-trained model to generate global text semantic vectors and local text semantic vectors.

[0046] 2-2: Masking is performed on the root action symbols output by the action discrete codec, which are determined by the masking rate using a cosine scheduling function.

[0047] 2-3: The global text semantic vector is placed before the damaged root layer action code to form an action context feature sequence, and the local text semantic vector is used for cross attention.

[0048] 2-4: Construct a global awareness attention mask and a temporally dependent attention mask, and predict the complete root action code through Transformer.

[0049] Step 3: Construction and training of the incremental action symbol predictor

[0050] The action code of each incremental layer is predicted layer by layer. Each layer takes the sum of the embedded discrete feature vectors of the previous layers and the layer index as input, and predicts the action code of the current incremental layer under the guidance of the text semantic vector.

[0051] Step 4: Sum the discrete feature vectors corresponding to the action codes of the base layer and each incremental layer, and input them into the motion discrete decoder to reconstruct the three-dimensional human action sequence, thus obtaining text-driven human action generation.

[0052] See Figure 2 The discrete motion encoder will process the three-dimensional human pose sequence. Encoded as a latent vector sequence The pose feature dimension of each frame is D, and the latent vector dimension is d.

[0053] See Figure 3 The hierarchical differential discretization module comprises six discretization layers (H = 5), with each layer containing 512 512-dimensional discrete feature vectors in its motion feature dictionary. For each latent vector output by the encoder, the discrete feature vector is obtained directly from the nearest neighbor lookup in layer 0. and action code The input level difference of each subsequent layer is discretized from the previous layer. Specifically, the input level difference of the h-th layer is... By searching in the motion feature dictionary for... The nearest discrete feature vector is obtained And calculate the level difference of the next level. After all discretization layers, the sum of all discrete feature vectors As the final approximation of the latent vector.

[0054] See Figure 4 The motion discrete decoder will Upsampled and reconstructed into a 3D human motion sequence .

[0055] The goal of the global action symbol predictor is to predict root-level action symbols given a text description, specifically including:

[0056] 1) The input text is processed using a contrastive language-image pre-trained model to generate global text semantic vectors embs (global semantic representation) and local text semantic vectors embw (local word-level representation).

[0057] 2) Root action code elements output by the action discrete codec Masking is performed, and the masking rate is determined by a cosine scheduling function. Calculate τ from a uniform distribution Random sampling.

[0058] 3) The selected token is replaced according to the following rules: 80% probability of being replaced with mask code element [M], 10% probability of being replaced with a token randomly selected from the motion feature dictionary, and 10% probability of keeping the original token unchanged.

[0059] See Figure 5 The global text semantic vector emb s Pre-positioned action code elements at the root level of the damage constitute the action context feature sequence emb seq This serves as the input for the Transformer's query; the local text semantic vector (emb) w Used as the key and value inputs for a Transformer.

[0060] See Figure 6 Construct a globally perceptive attention mask (att) bi and temporal dependent attention mask att cau The global awareness attention mask att bi Allow all tokens to focus on unmasked symbols from both directions; the time-dependent attention mask att cau The masked code at position i is forced to focus only on the first i masked code. Through the combined effect of the dual masks, the masked code can focus on all unmasked code and historical masked code, while the unmasked code only focuses on each other, thus maintaining temporal causal dependencies while preserving global context information.

[0061] See Figure 7 The Transformer encoder consists of 10 network layers, 6 attention heads, a latent vector dimension of 384, and a feedforward linear layer dimension of 1024. The first layer is a cross-attention layer, used to integrate action context feature sequences with local text semantic vectors; the remaining nine layers are self-attention layers. The Transformer output is passed through two linear layers and one normalization layer to obtain the predicted root action code.

[0062] The incremental action symbol predictor is responsible for predicting action symbols for each incremental layer layer by layer to supplement motion details and improve reconstruction accuracy, specifically including:

[0063] 1) For the h-th level difference token ( ), from the 0th layer of the preceding sequence to the 1st layer Hierarchical differential discrete feature vector After mapping each element to an embedding vector, the combined embedding is obtained by summing the elements. This process is then performed on the global text semantic vector (emb). s This is placed before the combined embedding and concatenated with the layer index of the current layer to be predicted. This constitutes the Transformer input sequence; the local text semantic vector emb w As keys and values ​​for cross-attention.

[0064] 2) The Transformer architecture is consistent with the global action code predictor: 10 network layers, 6 attention heads, latent vector dimension 384, feedforward linear layer dimension 1024, the first layer is cross-attention, and the remaining 9 layers are self-attention. The Transformer output is passed through two linear layers and one normalization layer to obtain the first... The prediction result of the hierarchical differential tokens. This process does not introduce a masking mechanism and directly outputs the complete hierarchical differential token sequence.

[0065] See Figure 8 The reasoning in this invention is divided into three stages, and the specific process is as follows:

[0066] The first stage is the generation of root action symbols: The total number of iterations is set to I (set to 10), and initially all action symbols are set as mask symbols. In each iteration: 1) The current token sequence and text semantic vector are input into the global action symbol predictor to predict all tokens; 2) The confidence score for each mask position is calculated; 3) The prediction is performed according to the cosine scheduling function. Calculate the number of tokens that need to be retained in the mask; 4) Select the tokens with the lowest confidence level. One token is remasked, while the remaining tokens remain unmasked. This process is repeated until all tokens are decoded, yielding the complete root action code.

[0067] The second stage is the generation of incremental layer action code: from To begin, the embeddings of the generated base layer and the preceding layer's differential tokens are summed, and then combined with the text semantic vector and layer index. Input incremental action symbol predictor, predict the first action symbol. Hierarchical difference tokens. Progressing layer by layer until... This yields all incremental layer action codes.

[0068] The third stage is motion sequence reconstruction: the motion codes of the base layer and each incremental layer are searched through the motion feature dictionary to obtain the corresponding discrete feature vectors, which are then summed element by element and input into the motion discrete decoder to reconstruct a three-dimensional human motion sequence.

[0069] See Figure 9 The experimental results comparing the generation of this invention with other generation methods are as follows:

[0070] See Figure 9 - (1) Both MDM and MoMask can only restore part of the information in the input text prompts and cannot fully present all the descriptions in the text. For example, such as Figure 9-1 As shown above, the input prompt clearly requires the generation of a "waving" action. Although the sequence generated by the MoMask model completes the forward walking action, it does not include a "waving" action during the movement. In contrast, the sequence generated by the MDM model does a "waving" action at the end of the action sequence, but fails to do so in the first half of the sequence.

[0071] See Figure 9 - (2) The MoMask model can accurately generate the "swimming" motion, but not the "butterfly" motion; while the MDM model only generates the "swimming" posture, but does not generate an effective motion sequence.

[0072] See Figure 9 - (3) In the action sequence generated by the MDM model, the distance of "moving forward" is too short and there is no "stumbling left and right" performance.

[0073] Compared with other generation methods, the proposed method demonstrates better generalization ability when handling various text prompts. For text prompts of varying lengths and complexities, it more effectively reflects detailed descriptions in the generated motion. Although some motion loss still occurs in a few complex text prompts, its overall performance is superior to other models. This indicates that the actions generated by this invention have good coherence and fidelity, proving the effectiveness of the method in text-to-human motion generation tasks.

[0074] The above embodiments are merely illustrative of the present invention and are not intended to limit the scope of the present invention. All equivalent implementations of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A text-driven human motion generation method based on hierarchical differential discretization and attention, characterized in that, The method includes the following steps: a) Construction and training of discrete action codecs a-1: Encode the three-dimensional human motion sequence into a latent vector sequence using an encoder; a-2: Construct the hierarchical differential discretization module and set... The system has several discretization layers, each containing a learnable motion feature dictionary. Starting from the level difference of the 0th layer, the system performs nearest neighbor search on the level difference of the discretization layers to obtain the discrete feature vectors and action codes of each layer. a-3: Sum the discrete feature vector sequences output by all discretization layers as an approximation of the latent vector sequence, and input it into the decoder to reconstruct the three-dimensional human motion sequence; a-4: The action discrete encoder-decoder is trained with the sum of the action reconstruction loss and the embedding constraint loss of each discretization layer as the training objective; b) Construction and training of the global action code predictor b-1: The input text description is processed using a contrastive language-image pre-trained model to generate global and local text semantic vectors; b-2: Mask the root layer action code output by the action discrete codec according to the mask rate determined by the cosine scheduling function to obtain the corrupted root layer action code. b-3: The global text semantic vector is placed before the damaged root layer action code to form an action context feature sequence, which is used as the query input of the Transformer, and the local text semantic vector is used as the key and value input; b-4: Construct a globally perceptive attention mask and a temporally dependent attention mask, enabling bidirectional mutual injection between unmasked symbols, and masked symbols only pay attention to historical masked symbols and all unmasked symbols; b-5: Use an iterative decoding strategy during the inference phase to predict the complete root-level action codewords step by step; c) Construction and training of the incremental action symbol predictor c-1: After embedding the discrete feature vectors of the previous layers, sum them and combine them with the global text semantic vector and the layer index of the current layer to be predicted to construct the input sequence; c-2: Using a Transformer with the same structure as the global action code predictor, predicts each incremental layer action code under the guidance of text semantic vectors; d) Text Action Generation The discrete feature vectors corresponding to the action codes of the base layer and each incremental layer are summed and input into the action discrete decoder to reconstruct the three-dimensional human action sequence, thus obtaining text-driven human action generation.

2. The text-driven human motion generation method based on hierarchical differential discretization and attention as described in claim 1, characterized in that, The hierarchical differential discretization module in step a) specifically includes: 1) Settings There are 1 discretization layer, each discretization layer containing a size of 1. A learnable motion feature dictionary, in which The number of discrete feature vectors. The dimension of the discrete feature vector; 2) From the 0th level difference Starting with the latent vector output by the encoder, in each discretization layer, the layer covariance is mapped to the nearest discrete feature vector in the motion feature dictionary through nearest neighbor search. And calculate the level difference of the next level. ,in .

3. The text-driven human motion generation method based on hierarchical differential discretization and attention as described in claim 1, characterized in that, The training objectives in step a) specifically include: 1) Use the loss function shown in the following formula. Training the discrete action codec: ； Where P represents the actual action sequence; The action sequence reconstructed by the decoder; To stop the gradient operation; α is the embedding constraint weight factor; 2) The motion feature dictionary is optimized through exponential moving average updates and motion feature dictionary reset strategies.

4. The text-driven human motion generation method based on hierarchical differential discretization and attention as described in claim 1, characterized in that, The motion discrete encoder in step a) is specifically configured with the following parameters: encoding a three-dimensional human pose sequence with a frame length of N into a latent vector sequence with a length of n, where N / n is a downsampling factor; The hierarchical differential discretization module contains 6 discretization layers, and each layer's motion feature dictionary contains 512 512-dimensional discrete feature vectors.

5. The text-driven human motion generation method based on hierarchical differential discretization and attention as described in claim 1, characterized in that, The masking process in step b) specifically involves randomly selecting m tokens from the root action code for masking. , where n is the length of the token sequence, and τ is randomly sampled from a uniform distribution U(0,1); the selected token is replaced with a mask symbol with an 80% probability, a random token with a 10% probability, and remains unchanged with a 10% probability.

6. The text-driven human motion generation method based on hierarchical differential discretization and attention as described in claim 1, characterized in that, The Transformer architecture in step b) is specifically composed of a Transformer encoder with 10 network layers, 6 attention heads, a latent vector dimension of 384, and a feedforward linear layer dimension of 1024. The first layer is a cross-attention layer used to integrate the action context feature sequence and the local text semantic vector, and the remaining 9 layers are self-attention layers. The Transformer output is passed through two linear layers and one normalization layer to obtain the root action code.

7. The text-driven human motion generation method based on hierarchical differential discretization and attention as described in claim 1, characterized in that, Step b) employs an iterative decoding strategy during the inference phase, specifically: setting a total number of iterations I, initially setting all action symbols as mask symbols; in each iteration, the global action symbol predictor predicts all tokens, calculates the confidence score for each mask position, and determines the confidence score based on the cosine scheduling function. Calculate the number of tokens that need to be masked, remask the m tokens with the lowest confidence, and keep the remaining tokens as unmasked symbols until all tokens are decoded.

8. The text-driven human motion generation method based on hierarchical differential discretization and attention as described in claim 1, characterized in that, The incremental action symbol predictor's layer-by-layer prediction in step c) specifically involves: for the h-th level differential token ( ), from the 0th layer of the preceding sequence to the 1st layer The discrete feature vectors of each layer are mapped to embedding vectors and then summed. The global text semantic vector, the local text semantic vector, and the current layer index h are then input into the Transformer to predict the incremental layer action codewords at all positions in the h-th layer.

9. The text-driven human motion generation method based on hierarchical differential discretization and attention as described in claim 1, characterized in that, The reconstruction of the 3D human motion sequence in step d) specifically involves: obtaining the corresponding discrete feature vector sequences by searching the motion codes of the base layer and each incremental layer through a motion feature dictionary; and summing the discrete feature vectors of each layer element-wise to obtain the final latent vector representation. The latent space motion vectors are input into the discrete motion decoder and mapped back to the original frame rate 3D human motion sequence. It can be expressed as follows: ； in, The discrete feature vector sequence corresponding to the action code of the h-th layer; For decoders.

Images