Tranform-fused virtual human action parameter enhancement method

Abstract

The invention discloses a virtual human action parameter enhancement method fused with a Transform. Comprising the following steps: collecting an original action data set, including actions in different emotional states, and dividing the action data set into a training set, a verification set and a test set after preprocessing; a virtual human action parameter generation model based on a Transform network is constructed; training the constructed virtual human action parameter generation model by using a training set, and verifying by using a verification set after training; and after the verification is passed, inputting the test set into a virtual human action parameter generation model, and outputting corresponding action parameters. According to the method, the dependence on virtual human action experts is reduced, the labor cost and the time and cost of popularization and training are remarkably reduced, and the economical efficiency of a virtual human action system is improved; through training, the method adapts to gesture habits of different regions and crowds, the adaptability and expression accuracy of virtual human actions are improved, and meanwhile, cross-region standardization is kept.

Description

A method for enhancing the motion parameters of virtual humans by incorporating Transformer Technical Field

[0001] This invention relates to the field of virtual human motion parameter enhancement technology, specifically to a method for enhancing virtual human motion parameters by incorporating Transformer. Background Technology

[0002] Currently, the virtual human's movements utilize a sign language database to ensure that humans can understand the gestures. The movements are executed according to the standards of the National Standard Sign Language Dictionary, which is equivalent to the "Mandarin" of sign language. Furthermore, existing practices also incorporate sports terminology, current affairs news, and commonly used phrases. In addition, existing practices use finger motion capture and motion refinement technology to ensure the completeness of motion data for each finger, specifically as follows: 1. Movement Standardization: Executed according to the standards of the National Standard Sign Language Dictionary, this is the "Mandarin" of sign language, ensuring that the gestures are understandable to hearing-impaired people across the country. It also includes sports terminology, current affairs news, and commonly used phrases; 2. Finger Motion Capture and Motion Refinement Technology: Simply put, motion data for each finger must be available, and the bending angle of each finger must have standard parameters; 3. A sign language expert group is invited to review each sign language movement individually. Those that do not meet the standards are identified and sent for optimization. To this end, the team has built a sign language movement review platform to accelerate the project's efficiency.

[0003] The above approach has some drawbacks in promoting and applying the gestures, which are explained below from three aspects: cost, standardization, and the stiffness of the gestures. Firstly, regarding cost, implementing gestures according to the standards of the National General Movement Dictionary requires a significant investment of human and material resources. Training gesture experts is time-consuming and costly, and establishing a gesture review platform and refining the gestures also require additional funding. Furthermore, to ensure that hearing-impaired individuals across the country can understand the gestures, training and promotion are needed in different regions, which increases the overall cost of promoting the gestures.

[0004] Secondly, adherence to standards is also an issue. Gestures are a highly flexible form of language, and differences may exist between different regions and groups. Even within the same region, the way gestures are expressed can change over time. Therefore, performing gestures according to a specific standard may not fully meet the needs of different regions and groups, leading to difficulties in understanding. Sometimes, certain gestures may not be commonly used or understood in a particular region. In such cases, even performing the gestures according to the standard may not achieve the desired effect.

[0005] Third, adhering to standardized procedures can make gestures appear relatively stiff and rigid. Gestures themselves have rich expressive possibilities and individual variations; each person's gestures may differ slightly. An excessive pursuit of standardization and normalization may limit the natural fluency and expressiveness of gestures. The beauty of gestures lies in their ability to adapt to individual expressive styles and emotions; overly rigid adherence to standards can make verbal communication appear unnatural and inflexible. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for enhancing the motion parameters of virtual humans that integrates Transformer.

[0007] To achieve the above objectives, this invention provides a method for enhancing virtual human motion parameters by incorporating Transformer, comprising: collecting an original motion dataset, including actions under different emotional states, wherein the motion dataset is preprocessed and divided into a training set, a validation set, and a test set; constructing a virtual human motion parameter generation model based on a Transformer network; training the constructed virtual human motion parameter generation model using the training set, and validating it using the validation set after training; and, upon successful validation, inputting the test set into the virtual human motion parameter generation model and outputting the corresponding motion parameters.

[0008] Furthermore, the motion dataset includes gesture signals, speech signals, and auxiliary images. The data source for the gesture signals is 3D skeletal joints, the data source for the speech signals is BERT text embeddings, and the data source for the auxiliary images is RGB-D depth maps.

[0009] Furthermore, the preprocessing is performed as follows: the gesture signal is normalized to [-1,1] and speed features are added; stop word filtering and keyword enhancement are performed on the language signal; background segmentation and keypoint alignment are performed on the auxiliary image.

[0010] Furthermore, the virtual human motion parameter generation model includes an encoding module, a Transformer module, a masking module, and a decoding module connected in sequence. The encoding module receives an input segment of time-domain multimodal signal, and after layer-by-layer feature extraction, global-local feature fusion, and feature optimization, converts it into high-dimensional feature data. The Transformer module includes multiple stacked dual-path Transformer blocks, each including a local Transformer block and a global Transformer block. The local Transformer block is used to process the local features in the input high-dimensional feature data. The local feature information is parallelized to focus on short-range dependencies within a time segment. The global Transformer block is used to fuse the output information of the local Transformer blocks to learn global feature information. The masking module includes multiple activation function layers and multiple convolutional layers. The convolutional layers are used to smooth the fused features output by the Transformer module to eliminate noise in the temporal features. The activation function layers are used to perform weighted filtering of features, retaining key feature information for virtual human action generation and suppressing invalid interference features. The decoding module is used to map the features output by the masking module back to a temporal multimodal signal in the same domain as the input.

[0011] Furthermore, the encoding module includes an upsampling layer and multiple encoding layers. The upsampling layer is used to receive a segment of input time-domain multimodal signal. The encoding layers are encoding layers that use virtual human motion parameters. Each encoding layer includes a first convolutional layer, a ReLU function activation layer, a second convolutional layer, and a GLU function activation layer.

[0012] Furthermore, both the local Transformer block and the global Transformer block include a multi-head attention mechanism module and a feedforward network module; the multi-head attention mechanism module integrates h self-attention mechanisms, each self-attention mechanism focusing on the same query Q, key K, and value V; each self-attention mechanism is only responsible for one subspace of the final output sequence, and the output sequences are independent of each other; the feedforward network module includes a GRU layer, a ReLU function activation layer, and a linear transformation layer.

[0013] Furthermore, the gesture signal is tokenized based on word embedding technology.

[0014] Beneficial effects: By utilizing the Transformer model, this invention reduces reliance on virtual human motion experts, significantly lowering labor costs. It also reduces the time and cost of promotion and training, improving the economic efficiency of the virtual human motion system. This invention can be personalized according to the needs of different regions and individuals. By training to adapt to the gesture habits of different regions and groups, it improves the adaptability and expressive accuracy of virtual human motion, while maintaining cross-regional standardization. Attached Figure Description

[0015] Figure 1 is a flowchart illustrating the method for enhancing the motion parameters of a virtual human by incorporating Transformer according to an embodiment of the present invention; Figure 2 is a lexical representation of an embodiment of the present invention. Detailed Implementation

[0016] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0017] As shown in Figure 1, this embodiment of the invention provides a method for enhancing the motion parameters of a virtual human by incorporating Transformer, including: collecting a raw motion dataset, including actions under different emotional states; the motion dataset, after preprocessing, is divided into a training set, a validation set, and a test set. It should be noted that the aforementioned motion dataset includes clean multimodal datasets and noisy multimodal datasets, specifically including gesture signals, speech signals, and auxiliary images. The data source for the gesture signals is 3D skeletal joints, the data source for the speech signals is BERT text embeddings, and the data source for the auxiliary images is RGB-D depth maps. The sampling frequency and preprocessing method of the aforementioned motion data are detailed in Table 1.

[0018] Table 1: Action Dataset A virtual human motion parameter generation model based on a Transformer network is constructed. Specifically, the virtual human motion parameter generation model includes an encoding module, a Transformer module, a masking module, and a decoding module connected in sequence.

[0019] The encoding module receives a segment of temporal multimodal signal (including multi-source temporal data related to virtual human movements such as joint angles, velocities, and accelerations), and converts it into high-dimensional feature identifiers after layer-by-layer feature extraction, global-local feature fusion, and feature optimization. Specifically, the encoding module includes an upsampling layer and multiple encoding layers. The encoding layers use virtual human movement parameters. The upsampling layer receives the input segment of temporal multimodal signal. Each encoding layer includes a first convolutional layer, a ReLU activation layer, a second convolutional layer, and a GLU activation layer. The first convolutional layer has a kernel size of K, a stride of S, and 2 output channels. i-1 *H, where i represents the layer number and H represents the controllable number of channels. The second convolutional layer has a kernel size of 1, a stride of 1, and 2 output channels. i *H is a one-dimensional convolutional layer; the GLU function activation layer converts the number of channels to 2. i-1 *H.

[0020] The Transformer module comprises multiple stacked dual-path Transformer blocks, each consisting of a local Transformer block and a global Transformer block. The local Transformer block parallelizes the local features of the input, focusing on short-range dependencies within temporal segments. The global Transformer block fuses the outputs of the local Transformer blocks to learn global features. Specifically, both the local and global Transformer blocks include a multi-head attention mechanism module and a feedforward network module. The multi-head attention mechanism module integrates h self-attention mechanisms, each focusing on the same query Q, key K, and value V. Each self-attention mechanism is responsible for only one subspace of the final output sequence, and the output sequences are independent of each other. The feedforward network module includes a GRU layer, a ReLU activation layer, and a linear transformation layer. Specific parameters are shown in Table 1.

[0021] Table 2: Transformer Module Parameters The masking module includes multiple activation function layers and multiple convolutional layers. The convolutional layers are used to smooth the fused features output by the Transformer module to eliminate noise in the temporal features. The activation function layers are used to perform weighted filtering of features, retain key feature information for virtual human action generation, and suppress invalid interference features.

[0022] The decoding module maps the features output by the masking module back to the time-domain multimodal signal in the same domain as the input. Specifically, the structure of the decoding module is the reverse of the encoding module, including multiple decoding layers and downsampling layers. Each decoding layer includes a convolutional kernel with a size of 1, a stride of 1, and 2 output channels. i-1 *H is a one-dimensional convolutional layer that converts the number of channels to 2. i-1 *H is a GLU activation layer with a kernel size of K, a stride of S, and 2 output channels. i-2 *H is a one-dimensional transposed convolutional layer. It adopts a stacked structure of "transposed convolution + activation" to achieve a gradual reduction in feature dimension and reconstruction of temporal signals.

[0023] The virtual human motion parameter generation model is trained using the training set and then validated using the validation set.

[0024] After successful verification, the test set is input into the virtual human motion parameter generation model, which outputs the corresponding motion parameters.

[0025] See Table 3: Reasoning Test Additionally, tokenizing gestures using word embedding is an effective method that transforms gestures into a form that computers can understand and process. The process of tokenizing gestures using word embedding is briefly described below.

[0026] Word embedding is a technique that maps words to a real-valued vector space, encoding the semantic information of words into vector representations. In the tokenization of gesture actions, each gesture can be treated as a word, and a unique identifier, or token, can be assigned to each gesture. Then, word embedding is used to map each gesture's token to a real-valued vector representation, thus transforming the gesture into a computer-processable form.

[0027] First, a vocabulary of gestures needs to be constructed, listing all possible gestures and assigning a unique identifier to each gesture. Gestures can be categorized and labeled based on the knowledge of sign language experts and gesture recognition technology to construct the vocabulary, as shown in Figure 2. Vocabulary construction includes: ① Atomic action units: K-Means clustering (K=200) to extract basic gestures; ② Combination token rules: temporal logic generation (e.g., "raise hand → clench fist" → encoded as T_115); ③ Dynamic expansion: incremental learning is triggered when the cosine similarity of a new action is <0.7.

[0028] Next, a word embedding model is used to train the vocabulary of gesture actions. Common word embedding models include Word2Vec, GloVe, and BERT. These models can learn the semantic relationships and contextual information between gesture actions and map the tokens of gesture actions to a high-dimensional vector space. This invention uses an embedding training strategy, including: ① Joint training: end-to-end optimization of the embedding layer and the Transformer, with the loss function including: MSE reconstruction loss (weight 0.7) and Triplet contrastive loss (weight 0.3); ② Temporal augmentation: Sinusoidal positional encoding + learnable parameters.

[0029] During training, contextual information from gestures can be used to optimize the word embedding model. For example, a sliding window can be used to obtain the contextual gestures for each gesture, and these contextual gestures can be used as input to train the word embedding model. This allows the word embedding model to better learn the correlation and fluency between gestures.

[0030] After training, the trained word embedding model can be used to convert gesture tokens into real-valued vector representations. This allows gestures to be input into the computer as vectors for processing and analysis. This vector representation can play a crucial role in tasks such as gesture recognition, gesture translation, and gesture generation.

[0031] The above description is merely a preferred embodiment of the present invention. It should be noted that for those skilled in the art, other parts not specifically described are existing technology or common knowledge. Several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for enhancing virtual human motion parameters by integrating Transformer, characterized in that, include: Collect raw action datasets, including actions under different emotional states. After preprocessing, the action datasets are divided into training, validation, and test sets. Construct a virtual human action parameter generation model based on a Transformer network. Train the constructed virtual human action parameter generation model using the training set, and validate it using the validation set after training. After successful validation, input the test set into the virtual human action parameter generation model and output the corresponding action parameters.

2. The method for enhancing virtual human motion parameters by incorporating Transformer according to claim 1, characterized in that, The motion dataset includes gesture signals, speech signals, and auxiliary images. The data source for the gesture signals is 3D skeletal joints, the data source for the speech signals is BERT text embedding, and the data source for the auxiliary images is RGB-D depth maps.

3. The method for enhancing virtual human motion parameters by incorporating Transformer according to claim 2, characterized in that, The preprocessing method is as follows: the gesture signal is normalized to [-1,1] and speed features are added; the language signal is filtered for stop words and enhanced with keywords; the auxiliary image is segmented for background and aligned with key points.

4. The method for enhancing virtual human motion parameters by incorporating Transformer according to claim 1, characterized in that, The virtual human motion parameter generation model includes an encoding module, a Transformer module, a masking module, and a decoding module connected in sequence. The encoding module receives an input time-domain multimodal signal, performs layer-by-layer feature extraction, global-local feature fusion, and feature optimization, and then converts it into high-dimensional feature data. The Transformer module includes multiple stacked dual-path Transformer blocks, each consisting of a local Transformer block and a global Transformer block. The local Transformer block is used to process local features in the input high-dimensional feature data. The feature information is parallelized to focus on short-range dependencies within temporal segments. The global Transformer block is used to fuse the output information of local Transformer blocks to learn global feature information. The masking module includes multiple activation function layers and multiple convolutional layers. The convolutional layers are used to smooth the fused features output by the Transformer module to eliminate noise in the temporal features. The activation function layers are used to perform weighted filtering of features, retaining key feature information for virtual human action generation and suppressing invalid interference features. The decoding module is used to map the features output by the masking module back to the temporal multimodal signal in the same domain as the input.

5. The method for enhancing virtual human motion parameters by incorporating Transformer according to claim 4, characterized in that, The encoding module includes an upsampling layer and multiple encoding layers. The upsampling layer is used to receive a segment of time-domain multimodal signal. The encoding layers are encoding layers that use virtual human motion parameters. Each encoding layer includes a first convolutional layer, a ReLU function activation layer, a second convolutional layer, and a GLU function activation layer.

6. The method for enhancing virtual human motion parameters by incorporating Transformer according to claim 4, characterized in that, Both the local Transformer block and the global Transformer block include a multi-head attention mechanism module and a feedforward network module. The multi-head attention mechanism module integrates h self-attention mechanisms, each of which focuses on the same query Q, key K, and value V. Each self-attention mechanism is only responsible for one subspace of the final output sequence, and the output sequences are independent of each other. The feedforward network module includes a GRU layer, a ReLU function activation layer, and a linear transformation layer.

7. The method for enhancing virtual human motion parameters by incorporating Transformer according to claim 2, characterized in that, The gesture signals are tokenized based on word embedding technology.

Images