Diversity-enhanced collaborative voice action generation system based on visual style characteristics

Abstract

The invention discloses a diversity enhancement collaborative voice action generation system based on visual style features. The system comprises a feature extraction diversity enhancement module which inputs a reference video into a 3D human body posture estimation network to obtain styleclips; by introducing additional styleclips, more feature information can be obtained, so that the diversity of actions is increased. And the style encoder diversity compensation module is used for effectively extracting deep learning representation of styleclips by utilizing a transform style encoder with an additional attention mechanism pooling layer. And finally, an action predictor diversity driving module adopts a cross attention mechanism to fuse the MFCC and the style code, and influence is carried out in the process of generating the action through cross condition autoregression, so that the generated action is adjusted to increase diversity. According to the method, the diversity of the generated actions is remarkably improved while the naturalness of the actions is kept.

Description

Technical Field

[0001] The present invention belongs to the field of speech-driven action generation, and in particular relates to a diversity-enhanced collaborative speech action generation system based on visual style features. Background Art

[0002] Head, hand and body movements are an integral part of human communication. There are complex and regular connections between them, which can enhance the expressiveness of language, convey emotions and attitudes, and coordinate the flow of conversations. With the widespread application of virtual characters and robots in education, entertainment, medical care and other fields, as well as their importance in information dissemination, how to make them generate natural, appropriate and diverse movements based on speech has become an important and challenging research problem. This problem involves multiple disciplines, such as computer vision, natural language processing, human-computer interaction, etc., and is of great significance in a wide range of application scenarios such as information dissemination, public opinion management and social interaction, such as virtual hosts, intelligent assistants, social robots, etc.

[0003] Existing methods for voice-driven head, hand, and body posture generation can be mainly divided into three categories: rule-based methods, statistical model-based methods, and deep learning-based methods. Among them, deep learning-based technology is currently the most mainstream method. Although current research has achieved certain results in generating the fluency and naturalness of human movements, the data sets in existing technologies mainly contain collaborative voice-action videos of a limited number of people with a relatively fixed style, resulting in a relatively single pattern learned by the model during training. This lack of diverse motion performance limits the adaptability of virtual characters in complex information dissemination scenarios, especially when they need to flexibly respond to changing situations and emotional transmission. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a diversity-enhanced collaborative voice action generation system based on visual style features. The method is jointly executed by a feature extraction diversity enhancement module, a style encoder diversity compensation module and an action predictor diversity driving module. First, the feature extraction diversity enhancement module not only extracts feature information from the voice, but also obtains more information by introducing a custom additional influence named StyleCLIPS; in the style encoder diversity compensation module, a transformer style encoder is designed, and an additive self-attention mechanism pooling layer is used to encode style clips into a deep learning representation called a style code, which allows styleclips to have a better impact on subsequent action generation; the action predictor diversity driving module is used to generate diverse collaborative voice actions. Specifically, the obtained style code and Mel frequency cepstral coefficient MFCC are fused using a cross-attention mechanism, and injected into a cross-conditional autoregressive module, and modulated when generating human body movements, so that the generated movements have higher diversity.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows: A diversity-enhanced collaborative speech action generation system based on visual style features, the system comprises a feature extraction diversity enhancement module, a style encoder diversity compensation module, and an action predictor diversity driving module, wherein: The feature extraction diversity enhancement module is used to extract and enhance features of input audio information and reference video information; The style encoder diversity compensation module is used to represent the extracted and enhanced reference video features as deep feature style codes through a transformer style encoder with an additional self-attention pooling layer; The action predictor diversity driving module is used to fuse the deep feature style code and the audio effective information through a cross-attention mechanism, and output a first result after modulating it in a temporal autoregressive network; it is also used to modulate the encoding result of the audio effective information using identity information and output a second result, and input the first result and the second result into a decoder to obtain the final action.

[0006] Furthermore, the feature extraction diversity enhancement module is used to extract and enhance features of the input audio information and reference video information, including: Voice Audio As input, generate a full-body action sequence , where N is the total number of frames corresponding to the speech audio A, Represents the Nth frame of input speech audio, represents the corresponding generated N-th full-body action, and the overall goal is expressed as: , Where G={ } represents the initial action sequence, ‖‖ represents the Euclidean norm, and argmin represents the parameter value that makes the function reach the minimum value; Use and represents the hand and body movements of the i-th frame, and the hand movement segment sequence is expressed as , the body action segment sequence is represented as ; Use temporal convolutional networks to train sequences of hand and body action clips and Encode to obtain hand and body motion feature sequences and ; The hand motion features and body motion features of the i-th frame and Mapped to their nearest codebook elements , To quantify: , , Where m represents the mth vector in the codebook, and They represent the hand action codebook elements and body action codebook elements of the quantized i-th frame respectively, Q represents the action quantization process, and The codebooks corresponding to hand and body movements respectively; The decoder converts the codebook corresponding to hand and body movements Mapping back to motion space as an action sequence: , , represents the generated hand action sequence, represents the generated body action sequence, D represents the decoding process, and E represents the action encoding process; Optimize the encoder, decoder, and codebook through training: , in, represents the reconstruction loss, sg represents the gradient stopping operation, represents the weight factor, Indicates commitment loss, represents the loss function of the vector quantized variational autoencoder; z represents the full-body motion codebook, and g represents the motion encoding feature.

[0007] Furthermore, the style encoder diversity compensation module is used to represent the extracted and enhanced reference video features as a deep feature style code by adding a transformer style encoder with a self-attention pooling layer, including: The whole body motion sequence is input into the transformer style encoder, the feature dimension is adjusted to the dimension and format expected by the style encoder through the linear layer, and the position information is added to the input whole body motion sequence using position encoding to obtain the style code sequence; The input style code sequence is segmented by using a feedforward neural network in an additional self-attention mechanism pooling layer, and each segmented region is given a weighted attention weight. After modeling the temporal correlation between the tags, all the style vectors are multiplied by the attention weights and added using the self-attention mechanism pooling layer to obtain the final style code. , , in, is a trainable parameter. represents the style code sequence obtained by the transformer style encoder, represents the dimension of each style vector, and the superscript T represents transposition. Represents the normalization function.

[0008] Furthermore, the action predictor diversity driving module is specifically used to perform the following steps: Combined with the cross attention layer, the final style code s is used to modulate the Mel frequency cepstral coefficient features extracted based on audio information ;include, Mel frequency cepstral coefficient features and the final style code s are respectively related to the projection matrix and Multiply, where Represents the Mel frequency cepstral coefficient feature The query projection matrix is ​​used to generate the query matrix , Represents the key projection matrix for the final style code s, used to generate the key matrix ; The final style code s is also related to the projection matrix Multiply, Represents the value projection matrix for the final style code s, which is used to generate the value matrix : , , , , in, is the dimension of the key-value set, and also represents the dimension of the query set. F represents the modal fusion feature, Attention represents the attention mechanism, and softmax represents the normalization process in the attention mechanism. The modal fusion feature F is input into the action matching network based on the temporal autoregressive model to generate a series of codebook vector indexes and , represents the hand action codebook vector index, Represents the body action codebook vector index, including: Temporal autoregressive models leverage past hand movements and past body movements Predict current hand movements and current body movements , and also uses the current body movements Predict current hand movements : , Indicates input audio , modality fusion feature F, generate body action codebook vector index and the hand motion codebook vector index , Represents the use of past body motion codebook vector index , the past hand motion codebook vector index , the past audio frame , and modality fusion feature F to predict the current body action codebook vector index , Represents the use of past body motion codebook vector index , the past hand motion codebook vector index , the past audio frame , and modality fusion feature F to predict the current hand action Codebook vector index; Generated codebook vector index for hand and body movements and Used to extract the and Retrieve the corresponding quantized motion element from the decoder, and input the retrieved quantized motion element into the decoder as the first result, Modulate the wav2vec2.0 encoding result using the identity information to obtain a second result; The second result is input into the decoder for final synthesis with the first result to generate the final hand movement and body movements .

[0009] The beneficial effects of the present invention are: The feature extraction diversity enhancement module designed by the present invention inputs the video into the 3D human key point estimation network to obtain styleclips, and obtains complex action styles and detail information by introducing additional styleclips, ensuring that the generated human actions have high diversity; the style encoder diversity compensation module, based on the transformer style encoder with an additional self-attention mechanism pooling layer, can effectively obtain the deep learning representation of styleclips and thus affect the model; the action predictor diversity driving module uses the cross-attention mechanism to fuse MFCC features and style codes, and modulates them in the cross-conditional autoregressive human action generation process, thereby generating more diverse and natural actions. The present invention can generate a series of real, voice-synchronized, high-quality 3D human actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a schematic diagram of the collaborative voice action generation system based on the diversity enhancement of visual style features of the present invention; Figure 2 A comparison of the styleclips visualization obtained after the video passes through the 3D human key point estimation network and the original video clip. DETAILED DESCRIPTION

[0011] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0012] like Figure 1As shown in the figure, it is a schematic diagram of the collaborative voice action generation system based on the diversity enhancement of visual style features of the present invention. In order to improve the diversity of the generated human body movements, the present invention proposes to add a customized additional influence StyleCLIPS, and use the cross attention mechanism to fuse the audio features and style codes, and modulate them in the process of human body movement generation. In the feature extraction diversity enhancement module (VEFE), wav2vec2.0 and MFCC are used to extract important information from the input voice, and styleclips are extracted from the reference video to remove useless interference information. In the style encoder diversity compensation module (VCSE), styleclips are represented as a deep feature style code through a transformer style encoder with an additional self-attention pooling layer, so as to facilitate subsequent integration into the model. In the action predictor diversity driving module (VDGP), the deep feature style code and MFCC are fused by the cross attention mechanism and modulated in the time autoregressive network, and the wav2vec2.0 encoding result is modulated by the identity information, which makes the generated action more diverse and natural. Finally, all features are decoded to obtain the final human body movement. During the training phase, the present invention also increases the diversity of actions by quantizing actions.

[0013] The various parts of the present invention are described in detail below.

[0014] The overall concept of the present invention is to increase the diversity of the generated 3D hand and body movements by adding an additional influence to modulate the generated 3D human body movements. As input, a full-body action sequence is generated accordingly . Where N is the total number of frames corresponding to the voice audio A, Represents the Nth frame of input speech audio, represents the Nth human action generated accordingly, that is, each speech audio frame corresponds to each generated human action. The system will also consider other speech patterns, such as reference video V and speaker identity I, to enhance the diversity of human actions.

[0015] According to the above statement, the overall goal is expressed as: , Where G={ } represents the initial action sequence, ‖‖ represents the Euclidean norm, and argmin represents the parameter value that makes the function reach the minimum value.

[0016] The embodiment of the present invention uses a human body model including 300-dimensional body shape, 156-dimensional body motion (including 3-dimensional jaw motion, 63-dimensional body motion and 90-dimensional hand motion), 3-dimensional camera motion, 3-dimensional translation and 100-dimensional expression, respectively. and represents the hand and body movements of the i-th frame, and the hand movement segment sequence is represented as , the body action segment sequence is represented as , where N represents the number of frames.

[0017] like Figure 1 As shown in Figure 2, we design VQ-VAEs for combining hand and body actions. In the training phase, given a sequence of hand and body action clips, and , where N is the number of frames. First, a temporal convolutional network is used to train the sequence of hand and body action clips and Encode and obtain hand and body motion feature sequences respectively and , where the number of action frames generated after downsampling by the encoder is , d is the time downsampling rate, C is the channel dimension of the feature, this process can be expressed as , g represents the action encoding feature, and E represents the action encoding process. In order to learn the codebook corresponding to the hand and body actions and , by combining the hand motion features and body motion features of the i-th frame and Mapped to their nearest codebook elements , To quantify: , , Where m represents the mth vector in the codebook, and They represent the hand motion codebook elements and body motion codebook elements of the i-th frame obtained by quantization, Q represents the motion quantization process, and argmin represents the variable that minimizes the function; The decoder converts the codebook Mapped back to the motion space as an action sequence. The whole process can be described as: , , represents the generated hand action sequence, represents the generated body action sequence, and D represents the decoding process.

[0018] Therefore, the encoder, decoder and codebook can be optimized through training: , in, is the reconstruction loss, sg represents the gradient stopping operation, is the commitment loss with weight factor β, Indicates commitment loss, represents the loss function of the VQ-VAE vector quantization variational autoencoder; z represents the full-body motion codebook.

[0019] To address the limited diversity of actions generated from speech, existing methods usually rely on extracting features from speech. However, due to the low amount of information in speech itself, the generated actions often lack diversity. To address this problem, the present invention introduces a diversity-enhanced feature extraction module. This module not only extracts information from speech, but also obtains an additional influence from the reference video.

[0020] Since facial expressions are highly correlated with speech signals, the audio encoder It is expected to extract high-level speech features from the input audio. Specifically, an encoder based on the latest self-supervised pre-trained speech model Wav2Vec 2.0

[45] is used to extract rich phoneme information. Audio Encoder The input audio is encoded into latent features through a multi-layer convolutional neural network. These features are then partially masked and a 768-dimensional speech feature representation is constructed using a Transformer network. Then, the audio encoder Adding a linear projection layer reduces the feature dimension to 256 dimensions.

[0021] It is observed that the audio contains some information irrelevant to hand and body movements, such as phonemes, which will affect the output hand and body postures. In order to remove this information, the Mel frequency cepstral coefficient feature is used To represent the audio signal characteristics, that is .

[0022] Since human motion diversity is a dynamic movement pattern of the head, hands and body, it is unrelated to factors such as clothing, hairstyle, lighting, etc. of the characters in the reference video. In order to effectively avoid irrelevant interference information, the reference video is converted into sequential motion parameters, i.e., the initial motion sequence. , and called it styleclips. Specifically, for the input reference video, DECA, PIXIE and PyMAF-X are first used to initialize the parameters of facial expressions and jaws, body and hands respectively, and then the obtained initialized parameters are input into the optimization module. The optimization module combines the human body contours from DeepLab V3, facial landmarks from MediaPipe and facial shapes from MICA for contour constraints to ensure that the rendered SMPL-X body is within the human body mask. At the same time, the photometric loss between the rendered face and the input image is used to better capture facial details. Finally, three-dimensional overall human mesh data with synchronized speech is generated. This method further expands the ability of full-body reconstruction and achieves more realistic human reconstruction by optimizing posture and expression parameters.

[0023] like Figure 1 As shown in Figure 1, after extracting styleclips from the reference video, simply encoding the styleclips may not fully capture their complex features, often resulting in the loss of key information. To solve this problem, a style encoder diversity compensation module is introduced, which can encode and extract style features more comprehensively.

[0024] Style Encoder The styleclips are converted into a deep learning representation called style code. A transformer-based style encoder with additional self-attention pooling layers is designed to model dynamic hand and body pose patterns.

[0025] Taking 3D SMPL-X action parameters as input, the input data is passed through a linear layer to adjust the feature dimension to the dimension and format expected by the style encoder. Then, the position information is added to the input data using position encoding. The style encoder takes the processed sequential 3D SMPL-X action parameters as input tokens.

[0026] Since the action style in an action sequence can be identified by a few typical frames, and the padding part does not contain useful information, it should be ignored during calculation. If a padding mask exists, it should be passed in to handle variable-length sequences.

[0027] Specifically, a feedforward neural network in an additional self-attention pooling layer is used to perform weighted attention weights on each region after segmenting the input pose sequence, representing the frame-level contribution of each region to the overall style of the sequence. After modeling the temporal correlation between markers, the self-attention pooling layer is used to multiply all the style vectors by the attention weights and add them together to obtain the final style code. , , in, is a trainable parameter. is the style code sequence obtained by the style encoder, is the dimension of each style vector.

[0028] Traditionally, researchers directly use regression methods to generate actions based on audio features. However, in this method, since the regression network can only capture the basic rhythm and amplitude changes in the audio, the generated actions have obvious deficiencies in diversity and naturalness. To solve this problem, an action predictor diversity-driven module is introduced. This module uses a cross-attention mechanism to enable features of different modalities to complement and interact at a high level. The fused features are then input into a temporal autoregressive network, and the codebook vector is used for action prediction. In this way, not only can the basic rhythm and amplitude changes in speech be captured, but also the complex action style and details in the reference video can be incorporated into the generated action sequence.

[0029] Use the cross attention layer to modulate the MFCC Mel-frequency cepstral coefficient features through the style code s Among them, the key (Key) and value (Value) are constructed by the style code s, and the query (Query) is constructed by the MFCC feature Specifically, for two input modalities and s, which are respectively related to the projection matrix and Multiply, For modal The query projection matrix is ​​used to generate the query matrix , is the key projection matrix for style code s, used to generate the key matrix . At the same time, the style code s is Multiply, is the value projection matrix for style code s, used to generate the value matrix , , , , , in, is the dimension of the key-value set (also the dimension of the query set). The last formula is used to calculate the attention score matrix (Attention Score) between the two modalities. F represents the modal fusion feature, Attention represents the attention mechanism, and Softmax represents the normalization process in the attention mechanism. This matrix represents the degree of association between the two modal information sequences under a certain mapping. Attention score matrix With value matrix The dot product of embodies the mode as a style code Towards modal potential adaptation.

[0030] After obtaining the fused feature F, the fused feature F is input into the action matching network based on the temporal autoregressive model. A series of codebook vector indexes are generated based on the temporal autoregressive model. and , represents the hand action codebook vector index, and represents the index of the body action codebook vector. The temporal autoregressive model uses mutual information to model gestures and body postures. It does this by leveraging past hand movements and past body movements To predict the current hand movement and current body movements At the same time, the current body movement is also used To predict the current hand movement , the specific formula is as follows:

[0031] Indicates input audio , modality fusion feature F, generate body action codebook vector index and the hand motion codebook vector index , Represents the use of past body motion codebook vector index , the past hand motion codebook vector index , the past audio frame , and fusion feature F to predict the current body action codebook vector index , Represents the use of past body motion codebook vector index , the past hand motion codebook vector index , the past audio frame , and fusion feature F to predict the current hand action Codebook vector index.

[0032] Next, the generated codebook vector index and Used to extract the and The corresponding quantized motion elements are retrieved from the codebook. These codebooks are learned through the motion quantization process during the training phase. Finally, the retrieved quantized motion elements are input into the decoder as the first result, and the wav2vec2.0 encoding result is modulated using the identity information to obtain the second result; the second result is input into the decoder for final synthesis with the first result to generate the final hand motion. and body movements .

[0033] To evaluate and compare existing audio-driven action generation methods, we introduce the SHOW dataset, a high-quality synchronized speech-action dataset. The dataset contains aligned speech audio and 3D full-body meshes of four speakers, which are reconstructed from SMPLX parameters from video clips with a maximum duration of 10 seconds, a frame rate of 30 frames / second, and synchronized audio with a sampling rate of 22K. The dataset is randomly shuffled and divided into training, validation, and test sets in a ratio of 8:1:1.

[0034] The method proposed in the present invention is compared with the methods in the prior art. As shown in Table 1, the new framework proposed in the present invention is superior to other methods in the indicators related to evaluating actions: Variation, Fletcher motion distance (FGD) and rhythm consistency score (BC). In particular, in terms of diversity indicators, it far exceeds all the comparison methods, which shows that the new framework proposed in the present invention can generate more diverse actions in the action space. At the same time, the new framework proposed in the present invention is also significantly superior to the method of the benchmark model in Fletcher motion distance (FGD), which shows that VarGES has strong distribution matching ability, and the actions of the synchronized speech synthesized by the new framework proposed in the present invention are more realistic than other methods. In addition, the score of the new framework proposed in the present invention in the rhythm consistency score (BC) is closest to the true value, which shows that the body movements generated by the new framework proposed in the present invention are better aligned with the input audio.

[0035] Table 1

[0036] In addition, the present invention visualizes the obtained styleclips and compares them with the characters in the original video, so as to demonstrate the reliability of the extracted styleclips in qualitative evaluation. Figure 2The visualization results are shown, and the generated 3D mesh has a high degree of overlap with the characters in the original video in appearance. The position and posture of the key points are accurately reflected in the 3D mesh, which is highly consistent with the character's morphology in the original video. In addition, it can be observed that the 3D mesh generated by the styleclips of the present invention successfully retains the details of the characters, such as facial expressions and gestures. It can also be observed that the generated 3D mesh can naturally and smoothly display different actions. The above aspects fully confirm the accuracy and reliability of the reconstruction results of the present invention.

[0037] In summary, the present invention provides a novel styleclips-based framework for improving the diversity of generated speech-synchronized 3D human motions, aiming to better enhance the diversity of generated motions. In VarGes, the feature extraction diversity enhancement module, the style encoder diversity compensation module, and the motion predictor diversity driving module are used to jointly promote the diverse expression of audio-driven human motions. Experimental results show that the proposed method significantly improves the diversity of generated motions while maintaining their naturalness. This progress in audio-driven human motion generation opens up new avenues for applications with more diverse and expressive characters in generated videos.

[0038] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A diversity-enhanced collaborative speech action generation system based on visual style features, characterized in that: The system includes a feature extraction diversity enhancement module, a style encoder diversity compensation module, and an action predictor diversity driving module, wherein: The feature extraction diversity enhancement module is used to extract and enhance features of input audio information and reference video information; The style encoder diversity compensation module is used to represent the extracted and enhanced reference video features as deep feature style codes through a transformer style encoder with an additional self-attention pooling layer; The action predictor diversity driving module is used to fuse the deep feature style code and the audio effective information through a cross-attention mechanism, and output a first result after modulating it in a temporal autoregressive network; it is also used to modulate the encoding result of the audio effective information using identity information and output a second result, and input the first result and the second result into a decoder to obtain the final action.

2. According to claim 1, a diversity-enhanced collaborative voice action generation system based on visual style features is characterized in that: The feature extraction diversity enhancement module is used to extract and enhance features of input audio information and reference video information, including: Voice Audio As input, generate a full-body action sequence , where N is the total number of frames corresponding to the speech audio A, Represents the Nth frame of input speech audio, represents the corresponding generated N-th full-body action, and the overall goal is expressed as: ， Where G = { } represents the initial action sequence, ‖‖ represents the Euclidean norm, and argmin represents the parameter value that makes the function reach the minimum value; Use and represents the hand and body movements of the i-th frame, and the hand movement segment sequence is expressed as , the body action segment sequence is represented as ; Use temporal convolutional networks to process sequences of hand and body action clips and Encode to obtain hand and body motion feature sequences and ; The hand motion features and body motion features of the i-th frame and Mapped to their nearest codebook elements , To quantify: ， ， Where m represents the mth vector in the codebook, and They represent the hand action codebook elements and body action codebook elements of the quantized i-th frame respectively, Q represents the action quantization process, and The codebooks corresponding to hand and body movements respectively; The decoder converts the codebook corresponding to hand and body movements Mapping back to motion space as an action sequence: ， ， represents the generated hand action sequence, represents the generated body action sequence, D represents the decoding process, and E represents the action encoding process; Optimize the encoder, decoder, and codebook through training: ， in, represents the reconstruction loss, sg represents the gradient stopping operation, represents the weight factor, Indicates commitment loss, represents the loss function of the vector quantized variational autoencoder; z represents the full-body motion codebook, and g represents the motion encoding feature.

3. The diversity-enhanced collaborative voice action generation system based on visual style features according to claim 1, characterized in that: The style encoder diversity compensation module is used to represent the extracted and enhanced reference video features as deep feature style codes through a transformer style encoder with an additional self-attention pooling layer, including: The whole body motion sequence is input into the transformer style encoder, the feature dimension is adjusted to the dimension and format expected by the style encoder through the linear layer, and the position information is added to the input whole body motion sequence using position encoding to obtain the style code sequence; The input style code sequence is segmented by using a feedforward neural network in an additional self-attention mechanism pooling layer, and each segmented region is given a weighted attention weight. After modeling the temporal correlation between the tags, all the style vectors are multiplied by the attention weights and added using the self-attention mechanism pooling layer to obtain the final style code. , ， in, is a trainable parameter. represents the style code sequence obtained by the transformer style encoder, represents the dimension of each style vector, and the superscript T represents transposition. Represents the normalization function.

4. The diversity-enhanced collaborative voice action generation system based on visual style features according to claim 1, characterized in that: The action predictor diversity driving module is specifically used to perform the following steps: Combined with the cross attention layer, the final style code s is used to modulate the Mel frequency cepstral coefficient features extracted based on audio information ; include, Mel frequency cepstral coefficient features and the final style code s are respectively related to the projection matrix and Multiply, where Represents the Mel frequency cepstral coefficient feature The query projection matrix is ​​used to generate the query matrix , Represents the key projection matrix for the final style code s, used to generate the key matrix ; The final style code s is also related to the projection matrix Multiply, Represents the value projection matrix for the final style code s, which is used to generate the value matrix : ， ， ， ， in, is the dimension of the key-value set, and also represents the dimension of the query set. F represents the modal fusion feature, Attention represents the attention mechanism, and softmax represents the normalization process in the attention mechanism. The modal fusion feature F is input into the action matching network based on the temporal autoregressive model to generate a series of codebook vector indexes and , represents the hand action codebook vector index, Represents the body action codebook vector index, including: Temporal autoregressive models leverage past hand movements and past body movements Predict current hand movements and current body movements , but also using the current body movements Predict current hand movements : ， Indicates input audio , modality fusion feature F, generate body action codebook vector index and the hand motion codebook vector index , Represents the use of past body motion codebook vector index , the past hand motion codebook vector index , the past audio frame , and modality fusion feature F to predict the current body action codebook vector index , Represents the use of past body motion codebook vector index , the past hand motion codebook vector index , the past audio frame , and modality fusion feature F to predict the current hand action Codebook vector index; Generated codebook vector index for hand and body movements and Used to extract the and Retrieving the corresponding quantized motion element from the decoder, and inputting the retrieved quantized motion element as a first result into a decoder; Modulate the wav2vec2.0 encoding result using the identity information to obtain a second result; The second result is input into the decoder for final synthesis with the first result to generate the final hand movement and body movements .

Images