Diversity-enhanced collaborative speech-action generation system based on visual style features

Abstract

The present invention discloses a diversity-enhanced collaborative voice action generation system based on visual style features, the system comprising: a feature extraction diversity enhancement module inputting a reference video into a 3D human pose estimation network to obtain styleclips. By introducing additional styleclips, more feature information can be obtained, thereby increasing the diversity of actions. The style encoder diversity compensation module utilizes a transformer style encoder with an additional attention mechanism pooling layer to effectively extract deep learning representations of styleclips. Finally, the action predictor diversity driving module adopts a cross-attention mechanism to fuse MFCC and style codes, and influence the cross-conditional autoregressive action generation process, thereby adjusting the generated actions to increase diversity. The method significantly improves the diversity of generated actions while maintaining the naturalness of the actions.

Description

Technical Field

[0001] The present invention belongs to the field of voice-driven action generation, and in particular relates to a collaborative voice action generation system based on diversity enhancement of visual style features. Background Art

[0002] Head, hand, and body movements are an integral part of human communication. Complex and regular relationships exist between them, enhancing the expressiveness of language, conveying emotions and attitudes, and coordinating the flow of conversation. With the widespread application of virtual characters and robots in education, entertainment, healthcare, and other fields, and their importance in information dissemination, how to enable them to generate natural, appropriate, and diverse movements based on speech has become an important and challenging research problem. This problem involves multiple disciplines, such as computer vision, natural language processing, and human-computer interaction, and has significant application in a wide range of scenarios, including information dissemination, public opinion management, and social interaction, such as virtual hosts, intelligent assistants, and social robots.

[0003] Existing methods for generating voice-driven head, hand, and body gestures fall into three main categories: rule-based, statistical model-based, and deep learning-based. Deep learning-based techniques are currently the most mainstream. While current research has achieved some success in generating smooth and natural human movements, existing datasets primarily consist of videos of a limited number of people performing coordinated voice movements in a relatively fixed style, resulting in relatively uniform patterns learned during model training. This lack of diverse motion performance limits the adaptability of virtual characters in complex information dissemination scenarios, particularly when they require flexible response to changing situations and emotional expression. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides a diversity-enhanced collaborative voice action generation system based on visual style features. The method is jointly executed by a feature extraction diversity enhancement module, a style encoder diversity compensation module and an action predictor diversity driving module. First, the feature extraction diversity enhancement module not only extracts feature information from the voice, but also obtains more information by introducing a custom additional influence called StyleCLIPS; in the style encoder diversity compensation module, a transformer style encoder is designed, and an additive self-attention mechanism pooling layer is used to encode style clips into a deep learning representation called style code, which allows styleclips to have a better impact on subsequent action generation; the action predictor diversity driving module is used to generate diverse collaborative voice actions. Specifically, the obtained style code and Mel-frequency cepstral coefficient MFCC are fused using a cross-attention mechanism and injected into a cross-conditional autoregressive module for modulation when generating human body movements, so that the generated movements have higher diversity.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A diversity-enhanced collaborative speech-action generation system based on visual style features, comprising a feature extraction diversity enhancement module, a style encoder diversity compensation module, and an action predictor diversity driving module, wherein:

[0007] The feature extraction diversity enhancement module is used to extract and enhance features of input audio information and reference video information;

[0008] The style encoder diversity compensation module is used to represent the extracted and enhanced reference video features as deep feature style codes through a transformer style encoder with an additional self-attention pooling layer;

[0009] The action predictor diversity driving module is used to fuse the deep feature style code and the audio valid information through the cross-attention mechanism, and output a first result after modulation in the temporal autoregressive network; it is also used to modulate the encoding result of the audio valid information using the identity information and output a second result, and input the first result and the second result into the decoder to obtain the final action.

[0010] Furthermore, the feature extraction diversity enhancement module is used to extract and enhance features of the input audio information and reference video information, including:

[0011] Voice audio As input, generate a full-body action sequence , where N is the total number of frames corresponding to the voice audio A, Represents the Nth frame of input speech audio, represents the corresponding generated N-th full-body action, and the overall goal is expressed as:

[0012] ,

[0013] Where G={ } represents the initial action sequence, ‖‖ represents the Euclidean norm, and argmin represents the parameter value that makes the function reach the minimum value;

[0014] Use and represents the hand and body movements of the i-th frame, and the hand movement segment sequence is expressed as , the body action segment sequence is represented as ; Use temporal convolutional networks to process sequences of hand and body action clips and Encode to obtain hand and body movement feature sequences and ;

[0015] The hand motion features and body motion features of the i-th frame and Mapped to their respective nearest codebook elements , To quantify:

[0016] ,

[0017] ,

[0018] Where m represents the mth vector in the codebook, and They represent the hand motion codebook elements and body motion codebook elements of the i-th frame obtained by quantization, Q represents the motion quantization process, and The codebooks corresponding to hand and body movements respectively;

[0019] The decoder converts the codebook corresponding to hand and body movements Mapping back to motion space as an action sequence:

[0020] ,

[0021] ,

[0022] represents the generated hand action sequence, represents the generated body action sequence, D represents the decoding process, and E represents the action encoding process;

[0023] Optimize the encoder, decoder, and codebook through training:

[0024] ,

[0025] in, represents the reconstruction loss, sg represents the gradient stopping operation, represents the weight factor, Indicates commitment loss, represents the loss function of the vector quantized variational autoencoder; z represents the full-body motion codebook, and g represents the motion encoding feature.

[0026] Furthermore, the style encoder diversity compensation module is used to represent the extracted and enhanced reference video features as a deep feature style code through a transformer style encoder with an additional self-attention pooling layer, including:

[0027] The whole-body motion sequence is input into the transformer style encoder, and the feature dimension is adjusted to the dimension and format expected by the style encoder through the linear layer. The positional encoding is used to add the position information to the input whole-body motion sequence to obtain the style code sequence;

[0028] The input style code sequence is segmented by using a feedforward neural network in an additional self-attention mechanism pooling layer, and each segmented region is given a weighted attention weight. After modeling the temporal correlation between the tags, all the style vectors are multiplied by the attention weights and added together using the self-attention mechanism pooling layer to obtain the final style code. ,

[0029] ,

[0030] in, It represents the trainable parameters, represents the style code sequence obtained by the transformer style encoder, Represents the dimension of each style vector, and the superscript T represents transposition. Represents the normalization function.

[0031] Furthermore, the action predictor diversity driving module is specifically configured to perform the following steps:

[0032] Combined with the cross attention layer, the final style code s is used to modulate the Mel frequency cepstral coefficient features extracted based on audio information ;include,

[0033] Mel-frequency cepstral coefficient features and the final style code s are respectively related to the projection matrix and Multiply, where Represents the Mel frequency cepstral coefficient feature The query projection matrix is used to generate the query matrix , Represents the key projection matrix for the final style code s, used to generate the key matrix ;

[0034] The final style code s is also related to the projection matrix Multiply, Represents the value projection matrix for the final style code s, which is used to generate the value matrix :

[0035] ,

[0036] ,

[0037] ,

[0038] ,

[0039] in, is the dimension of the key-value set, and also the dimension of the query set. F represents the modal fusion feature, Attention represents the attention mechanism, and softmax represents the normalization process in the attention mechanism.

[0040] Input the modal fusion feature F into the action matching network based on the temporal autoregressive model to generate a series of codebook vector indexes and , represents the hand action codebook vector index, Represents the body action codebook vector index, including:

[0041] Temporal autoregressive models exploit past hand movements and past body movements Predict current hand movements and current body movements , and also uses the current body movements Predict current hand movements :

[0042] ,

[0043] Indicates input audio , modal fusion feature F, generate body action codebook vector index and hand motion codebook vector index , Represents the use of past body motion codebook vector index , the past hand motion codebook vector index , past audio frames , and modality fusion feature F to predict the current body action codebook vector index , Represents the use of past body motion codebook vector index , the past hand motion codebook vector index , past audio frames , and modality fusion feature F to predict the current hand action Codebook vector index;

[0044] Generated hand and body motion codebook vector index and Used to extract data from the codebook and Retrieve the corresponding quantized motion element from the decoder, and input the retrieved quantized motion element into the decoder as the first result.

[0045] Modulating the wav2vec2.0 encoding result using the identity information to obtain a second result;

[0046] The second result is input into the decoder and finally synthesized with the first result to generate the final hand movement and body movements .

[0047] The beneficial effects of the present invention are:

[0048] The feature extraction diversity enhancement module designed by the present invention inputs the video into the 3D human key point estimation network to obtain styleclips. By introducing additional styleclips, complex action styles and detailed information are obtained, ensuring that the generated human actions have high diversity. The style encoder diversity compensation module, based on the transformer style encoder with an additional self-attention mechanism pooling layer, can effectively obtain the deep learning representation of styleclips and thus influence the model. The action predictor diversity driving module uses a cross-attention mechanism to fuse MFCC features and style codes, and modulates them during the cross-conditional autoregressive human action generation process, thereby generating more diverse and natural actions. The present invention can generate a series of realistic, speech-synchronized, high-quality 3D human actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1This is a schematic diagram of a collaborative voice action generation system based on the diversity enhancement of visual style features according to the present invention;

[0050] Figure 2 A comparison of the styleclips visualization obtained by the video after it passes through the 3D human key point estimation network and the original video clip. DETAILED DESCRIPTION

[0051] The present invention will be further described below with reference to the accompanying drawings and examples.

[0052] like Figure 1 Figure 1 shows the schematic diagram of the collaborative voice action generation system based on enhanced diversity of visual style features. To improve the diversity of generated human actions, the present invention proposes adding a custom additional influence, StyleCLIPS, and using a cross-attention mechanism to fuse audio features and style codes, modulating them during the human action generation process. In the Feature Extraction Diversity Enhancement module (VEFE), Wav2vec2.0 and MFCCs are used to extract important information from the input speech, and styleclips are extracted from the reference video to remove useless interference information. In the Style Encoder Diversity Compensation module (VCSE), a Transformer style encoder with an additional self-attention pooling layer is used to represent the styleclips as a deep feature style code, facilitating subsequent model integration. In the Action Predictor Diversity Driving module (VDGP), the deep feature style code and MFCCs are fused using a cross-attention mechanism and modulated in a temporal autoregressive network. Identity information is also used to modulate the Wav2vec2.0 encoding results, making the generated actions more diverse and natural. Finally, all features are decoded to obtain the final human action. During the training phase, the present invention also increases the diversity of actions by quantizing actions.

[0053] The various parts of the present invention are described in detail below.

[0054] The overall concept of the present invention is to increase the diversity of the generated 3D hand and body movements by adding an additional effect to modulate the generated 3D human body movements. As input, a full-body action sequence is generated accordingly Where N is the total number of frames corresponding to the voice audio A. Represents the Nth frame of input speech audio, represents the Nth human action generated, i.e., each speech audio frame corresponds to each generated human action. The system also considers other speech patterns, such as reference video V and speaker identity I, to enhance the diversity of human actions.

[0055] According to the above statement, the overall goal is expressed as:

[0056] ,

[0057] Where G={ } represents the initial action sequence, ‖‖ represents the Euclidean norm, and argmin represents the parameter value that minimizes the function.

[0058] The embodiment of the present invention uses a human body model including 300-dimensional body shape, 156-dimensional body motion (including 3-dimensional jaw motion, 63-dimensional body motion and 90-dimensional hand motion), 3-dimensional camera motion, 3-dimensional translation and 100-dimensional expression, respectively. and represents the hand and body motions of the i-th frame, and the hand motion segment sequence is represented as , the body action segment sequence is represented as , where N represents the number of frames.

[0059] like Figure 1 As shown, we design VQ-VAEs for hand and body action combinations. During the training phase, given a sequence of hand and body action segments, and , where N represents the number of frames. First, a temporal convolutional network is used to train the hand and body action clip sequences. and Encode and obtain hand and body motion feature sequences respectively and , where the number of action frames generated after encoder downsampling is , d is the time downsampling rate, C is the channel dimension of the feature, this process can be expressed as , g represents the action encoding feature, and E represents the action encoding process. In order to learn the codebook corresponding to the hand and body movements and , by combining the hand motion features and body motion features of the i-th frame and Mapped to their respective nearest codebook elements , To quantify:

[0060] ,

[0061] ,

[0062] Where m represents the mth vector in the codebook, and They represent the hand motion codebook elements and body motion codebook elements of the i-th frame obtained by quantization, Q represents the motion quantization process, and argmin represents the variable that minimizes the function;

[0063] The decoder converts the codebook Mapping back to the motion space as an action sequence. The whole process can be expressed as:

[0064] ,

[0065] ,

[0066] represents the generated hand action sequence, represents the generated body action sequence, and D represents the decoding process.

[0067] Therefore, the encoder, decoder, and codebook can be optimized through training:

[0068] ,

[0069] in, is the reconstruction loss, sg represents the gradient stopping operation, is the commitment loss with weight factor β, Indicates commitment loss, represents the loss function of the VQ-VAE vector quantization variational autoencoder; z represents the full-body motion codebook.

[0070] To address the limited diversity of actions generated from speech, existing methods typically rely on feature extraction from speech. However, due to the limited information inherent in speech, the generated actions often lack diversity. To address this issue, the present invention introduces a diversity-enhancing feature extraction module. This module not only extracts information from speech but also gains additional influence from reference videos.

[0071] Since facial expressions are highly correlated with speech signals, audio encoders It is expected to extract high-level speech features from the input audio. Specifically, an encoder based on the latest self-supervised pre-trained speech model Wav2Vec 2.0

[45] is used to extract rich phoneme information. Audio Encoder The input audio is encoded into latent features through a multi-layer convolutional neural network. These features are then partially masked and a 768-dimensional speech feature representation is constructed using a Transformer network. Adding a linear projection layer reduces the feature dimension to 256 dimensions.

[0072] It is observed that the audio contains some information that is irrelevant to hand and body movements, such as phonemes, which will affect the output hand and body postures. In order to remove this information, the Mel frequency cepstral coefficient feature is used. To represent the audio signal characteristics, that is .

[0073] Since human motion diversity is a dynamic movement pattern of the head, hands and body, it is unrelated to factors such as clothing, hairstyle, and lighting of the characters in the reference video. In order to effectively avoid irrelevant interference information, the reference video is converted into sequential motion parameters, i.e., the initial motion sequence. , and called it styleclips. Specifically, for the input reference video, DECA, PIXIE and PyMAF-X are first used to initialize the parameters of facial expressions and jaw, body and hands respectively, and then the obtained initialized parameters are input into the optimization module. The optimization module combines the human body contours from DeepLab V3, facial landmarks from MediaPipe and facial shapes from MICA for contour constraints to ensure that the rendered SMPL-X body is within the human body mask. At the same time, the photometric loss between the rendered face and the input image is used to better capture facial details. Finally, three-dimensional whole body mesh data with synchronized speech is generated. This method further expands the ability of full-body reconstruction and achieves more realistic human body reconstruction by optimizing posture and expression parameters.

[0074] like Figure 1 As shown in Figure 2, after extracting styleclips from the reference video, simply encoding the styleclips may not fully capture their complex features, often resulting in the loss of key information. To address this issue, a style encoder diversity compensation module is introduced, which can more comprehensively encode and extract style features.

[0075] Style Encoder We convert styleclips into a deep learning representation called a style code and design a transformer-based style encoder with additional self-attention pooling layers to model dynamic hand and body pose patterns.

[0076] Taking 3D SMPL-X action parameters as input, the input data is resized to the dimensions and format expected by the style encoder through a linear layer. Next, positional encoding is used to add positional information to the input data. The style encoder uses the processed sequential 3D SMPL-X action parameters as input tokens.

[0077] Since the action style in an action sequence can be identified by a few typical frames, and the padding part does not contain useful information, it should be ignored during calculation. If a padding mask exists, it should be passed in to handle variable-length sequences.

[0078] Specifically, a feedforward neural network in an additional self-attention pooling layer is used to perform weighted attention on each region after segmenting the input pose sequence, representing the frame-level contribution of each region to the overall style of the sequence. After modeling the temporal correlation between markers, the self-attention pooling layer is used to multiply all the style vectors obtained by the attention weights and add them together to obtain the final style code. ,

[0079] ,

[0080] in, is a trainable parameter. is the style code sequence obtained by the style encoder, is the dimension of each style vector.

[0081] Traditionally, researchers have used regression methods to generate actions directly from audio features. However, because the regression network can only capture the basic rhythm and amplitude variations in the audio, the generated actions lack diversity and naturalness. To address this issue, a diversity-driven action predictor module was introduced. This module uses a cross-attention mechanism to enable features from different modalities to complement and interact at a high level. The fused features are then input into a temporal autoregressive network, and the codebook vector is used for action prediction. In this way, not only can the basic rhythm and amplitude variations in speech be captured, but the complex action styles and details of the reference video can also be incorporated into the generated action sequence.

[0082] Use the cross attention layer to modulate the MFCC Mel-frequency cepstral coefficient features through the style code s Among them, the key (Key) and value (Value) are constructed by the style code s, and the query (Query) is constructed by the MFCC feature Specifically, for two input modalities and s, which are respectively related to the projection matrix and Multiply, For modal The query projection matrix is used to generate the query matrix , is the key projection matrix for the style code s, used to generate the key matrix . At the same time, the style code s is Multiply, is the value projection matrix for the style code s, used to generate the value matrix ,

[0083] ,

[0084] ,

[0085] ,

[0086] ,

[0087] in, is the dimension of the key-value set (also the dimension of the query set). The last formula is used to calculate the attention score matrix (Attention Score) between the two modalities. F represents the modal fusion feature, Attention represents the attention mechanism, and Softmax represents the normalization process in the attention mechanism. This matrix represents the degree of correlation between the two modal information sequences under a certain mapping. Attention score matrix and value matrix The dot product of embodies the mode as a style code Towards modal potential adaptation.

[0088] After obtaining the fusion feature F, the fusion feature F is input into the action matching network based on the temporal autoregressive model. A series of codebook vector indexes are generated based on the temporal autoregressive model. and , represents the hand motion codebook vector index, and Represents the body action codebook vector index. The temporal autoregressive model uses mutual information to model gestures and body postures. It does this by utilizing past hand movements and past body movements To predict the current hand movement and current body movements At the same time, the current body movement is also used To predict the current hand movement , the specific formula is as follows:

[0089]

[0090] Indicates input audio , modal fusion feature F, generate body action codebook vector index and hand motion codebook vector index , Represents the use of past body motion codebook vector index , the past hand motion codebook vector index , past audio frames , and fusion feature F to predict the current body action codebook vector index , Represents the use of past body motion codebook vector index , the past hand motion codebook vector index , past audio frames , and fusion feature F to predict the current hand movement Codebook vector index.

[0091] Next, the generated codebook vector index and Used to extract data from the codebook and The corresponding quantized motion elements are retrieved from the codebook. These codebooks are learned during the training phase through the motion quantization process. Finally, the retrieved quantized motion elements are input into the decoder as the first result, and the wav2vec2.0 encoding result is modulated using the identity information to obtain the second result; the second result is input into the decoder and finally synthesized with the first result to generate the final hand movement. and body movements .

[0092] To evaluate and compare existing audio-driven action generation methods, we introduce the SHOW dataset, a high-quality synchronized speech-action dataset. This dataset contains aligned speech audio and 3D full-body meshes of four speakers reconstructed using SMPLX parameters from video clips with a maximum duration of 10 seconds and a frame rate of 30 frames per second. The synchronized audio is sampled at 22K. The dataset is randomly shuffled and split into training, validation, and test sets in an 8:1:1 ratio.

[0093] Comparing the proposed method with existing methods, as shown in Table 1, the proposed framework outperforms other methods in metrics related to motion evaluation: Variation, Fletcher's Gyration Distance (FGD), and Cadence Consistency (BC). In particular, the proposed framework significantly surpasses all other compared methods in terms of Variation, demonstrating that the proposed framework is capable of generating a greater diversity of motions in the motion space. Furthermore, the proposed framework significantly outperforms the baseline model in terms of FGD, demonstrating VarGES's strong distribution matching capabilities. The proposed framework produces more realistic motions in the synchronized speech than other methods. Furthermore, the proposed framework achieves the closest score to the true value in Cadence Consistency (BC), indicating that the generated body motions are better aligned with the input audio.

[0094] Table 1

[0095]

[0096] In addition, the present invention visualizes the obtained styleclips and compares them with the characters in the original video, so as to demonstrate the reliability of the extracted styleclips in qualitative evaluation. Figure 2 The visualization results show that the generated 3D mesh has a high degree of overlap in appearance with the characters in the original video. The positions and postures of key points are accurately reflected in the 3D mesh, highly consistent with the characters in the original video. In addition, it can be observed that the 3D mesh generated by the styleclips of the present invention successfully preserves the details of the characters, such as facial expressions and gestures. It can also be observed that the generated 3D mesh can naturally and smoothly display different actions. The above aspects fully confirm the accuracy and reliability of the reconstruction results of the present invention.

[0097] In summary, the present invention provides a novel framework for improving the diversity of generated speech-synchronized 3D human motion based on styleclips, aiming to better enhance the diversity of generated motions. In VarGes, the feature extraction diversity enhancement module, the style encoder diversity compensation module, and the motion predictor diversity driving module are jointly utilized to promote the diverse expression of audio-driven human motions. Experimental results show that the proposed method significantly improves the diversity of generated motions while maintaining their naturalness. This progress in audio-driven human motion generation opens up new avenues for applications where the characters in the generated videos are more diverse and expressive.

[0098] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A diversity-enhanced collaborative voice action generation system based on visual style features, characterized by: The system includes a feature extraction diversity enhancement module, a style encoder diversity compensation module, and an action predictor diversity driving module, wherein: The feature extraction diversity enhancement module is used to extract and enhance features from the input audio information and reference video information. For the input audio information, based on the latest self-supervised pre-trained speech model Wav2Vec 2.0 encoder, it extracts rich phoneme information and uses Mel-frequency cepstral coefficient features to represent audio signal features. For the input reference video information, it converts it into sequential action parameters, i.e., initial action sequences, which are called styleclips. The style encoder diversity compensation module is used to represent the styleclips as deep feature style codes through a transformer style encoder with an additional self-attention pooling layer; The action predictor diversity driving module is used to use the cross-attention mechanism to modulate the Mel-frequency cepstral coefficient features extracted based on audio information using the style code, fuse the deep feature style code and the Mel-frequency cepstral coefficient features, and output a first result after modulating them in a temporal autoregressive network; it is also used to use the identity information to modulate the encoding result of the phoneme information extracted by the Wav2Vec 2.0 encoder and output a second result, and input the first result and the second result into the decoder to obtain the final action.

2. The diversity-enhanced collaborative voice action generation system based on visual style features according to claim 1 is characterized in that: The feature extraction diversity enhancement module is used to extract and enhance features of input audio information and reference video information, including: Voice audio As input, generate a full-body action sequence , where N is the total number of frames corresponding to the voice audio A, Represents the Nth frame of input speech audio, represents the corresponding generated N-th full-body action, and the overall goal is expressed as: ， Where G={ } represents the initial action sequence, ‖‖ represents the Euclidean norm, and argmin represents the parameter value that makes the function reach the minimum value; Use and represents the hand and body movements of the i-th frame, and the hand movement segment sequence is expressed as , the body action segment sequence is represented as ; Use temporal convolutional networks to process sequences of hand and body action clips and Encode to obtain hand and body movement feature sequences and ; The hand motion features and body motion features of the i-th frame and Mapped to their respective nearest codebook elements , To quantify: ， ， Where m represents the mth vector in the codebook, and They represent the hand motion codebook elements and body motion codebook elements of the i-th frame obtained by quantization, Q represents the motion quantization process, and The codebooks corresponding to hand and body movements respectively; The decoder converts the codebook corresponding to hand and body movements Mapping back to motion space as an action sequence: ， ， represents the generated hand action sequence, represents the generated body action sequence, D represents the decoding process, and E represents the action encoding process; Optimize the encoder, decoder, and codebook through training: ， in, represents the reconstruction loss, sg represents the gradient stopping operation, represents the weight factor, Indicates commitment loss, represents the loss function of the vector quantized variational autoencoder; z represents the full-body motion codebook, and g represents the motion encoding feature.

3. The diversity-enhanced collaborative voice action generation system based on visual style features according to claim 1 is characterized in that: The style encoder diversity compensation module is used to represent the extracted and enhanced reference video features as a deep feature style code through a transformer style encoder with an additional self-attention pooling layer, including: The whole-body motion sequence is input into the transformer style encoder, and the feature dimension is adjusted to the dimension and format expected by the style encoder through the linear layer. The positional encoding is used to add the position information to the input whole-body motion sequence to obtain the style code sequence; The input style code sequence is segmented by using a feedforward neural network in an additional self-attention mechanism pooling layer, and each segmented region is given a weighted attention weight. After modeling the temporal correlation between the tags, all the style vectors are multiplied by the attention weights and added together using the self-attention mechanism pooling layer to obtain the final style code. , ， in, It represents the trainable parameters, represents the style code sequence obtained by the transformer style encoder, Represents the dimension of each style vector, and the superscript T represents transposition. Represents the normalization function.

4. The diversity-enhanced collaborative voice action generation system based on visual style features according to claim 1 is characterized in that: The action predictor diversity driving module is specifically configured to perform the following steps: Combined with the cross attention layer, the final style code s is used to modulate the Mel frequency cepstral coefficient features extracted based on audio information ; include, Mel-frequency cepstral coefficient features and the final style code s are respectively related to the projection matrix and Multiply, where Represents the Mel frequency cepstral coefficient feature The query projection matrix is used to generate the query matrix , Represents the key projection matrix for the final style code s, used to generate the key matrix ; The final style code s is also related to the projection matrix Multiply, Represents the value projection matrix for the final style code s, which is used to generate the value matrix : ， ， ， ， in, is the dimension of the key-value set, and also represents the dimension of the query set. F represents the modal fusion feature, Attention represents the attention mechanism, and softmax represents the normalization process in the attention mechanism. Input the modal fusion feature F into the action matching network based on the temporal autoregressive model to generate a series of codebook vector indexes and , represents the hand action codebook vector index, Represents the body action codebook vector index, including: Temporal autoregressive models leverage past hand movements and past body movements Predict current hand movements and current body movements , and also uses the current body movements Predict current hand movements : ， Indicates input audio , modal fusion feature F, generate body action codebook vector index and hand motion codebook vector index , Represents the use of past body motion codebook vector index , the past hand motion codebook vector index , past audio frames , and modality fusion feature F to predict the current body action codebook vector index , Represents the use of past body motion codebook vector index , the past hand motion codebook vector index , past audio frames , and modality fusion feature F to predict the current hand action Codebook vector index; Generated hand and body motion codebook vector index and Used to extract data from the codebook and Retrieving the corresponding quantized motion element from the decoder, and inputting the retrieved quantized motion element as a first result into the decoder; Modulating the wav2vec2.0 encoding result using the identity information to obtain a second result; The second result is input into the decoder and finally synthesized with the first result to generate the final hand movement and body movements .

Images