A three-dimensional face geometry animation method

Abstract

The application provides a three-dimensional face geometry animation method, comprising the following steps: S1, receiving multi-modal input, encoding through a corresponding encoder to obtain features corresponding to each mode; S2, for the features corresponding to each mode, encoding through an encoder to obtain high-level representation features corresponding to each mode; S3, using a multi-modal fusion method based on an attention mechanism to fuse the high-level representation features corresponding to each mode to obtain multi-modal information; S4, for the multi-modal information, decoding through a decoder based on a Transformer to obtain corresponding three-dimensional face geometry animation. The method of the application generates face animation, solves the problem that the existing three-dimensional face geometry animation generation method cannot fully utilize semantic information and the obtained lip animation effect is poor, and obtains ideal animation.

Description

Technical Field

[0001] This invention relates to the fields of image processing, speech processing, and text processing, and in particular to a three-dimensional face geometric animation method. Background Technology

[0002] With the development of virtual reality technology, 3D virtual interaction technology has received increasing attention. Among these technologies, drivable virtual faces, as a direct means of interaction, have always been a popular research area. Previous research has mostly focused on 2D face animation, with relatively little research on 3D face animation. 2D face modeling methods cannot model 3D facial information to achieve animation in 3D space.

[0003] In existing technologies, 3D face animation is mainly based on voice-driven methods. By inputting a specified voice, a 3D face is driven to obtain the corresponding animation. However, this method does not make full use of the semantic information of the text corresponding to the voice, and the resulting lip-sync animation is prone to being overly smoothed, resulting in poor effects. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a three-dimensional face geometric animation method, which solves the problems that existing three-dimensional face animation generation methods cannot fully utilize semantic information and produce poor lip-sync animation effects.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] This invention provides a three-dimensional facial geometric animation method, comprising the following steps:

[0007] S1: Receive multimodal input, encode it through the corresponding encoder, and obtain the features corresponding to each modality;

[0008] S2: For each modality, the features are encoded by an encoder to obtain the high-level representation features corresponding to each modality;

[0009] S3: Use an attention-based multimodal fusion method to fuse the high-level representation features corresponding to each modality to obtain multimodal information;

[0010] S4: The multimodal information is decoded by a decoder to obtain the corresponding three-dimensional face geometric animation.

[0011] In some embodiments, the present invention further includes the following technical features:

[0012] The multimodal input includes three modalities: audio, text, and phonemes. The encoder is a Transformer-based encoder, and the decoder is a Transformer-based decoder.

[0013] In step S1, the discrete text, phonemes, and audio are transformed into features by the encoder. The audio is encoded using a pre-trained wav2vec network, which consists of multiple temporal encoding networks. Using pre-trained wav2vec can utilize pre-trained knowledge to encode the audio into more accurate semantic information. The text is encoded using a pre-trained BERT to obtain text features. BERT is a Transformer network architecture. The phonemes are encoded using Pre-Net to obtain phoneme features. Pre-Net contains multiple convolutional layers and linear layers.

[0014] In step S2, the Transformer-based encoder includes a multi-head self-attention mechanism and a feedforward network.

[0015] The Transformer encoder consists of three encoder layers, each composed of two sub-layer connection structures. The first sub-layer connection structure includes a multi-head self-attention sub-layer, a normalization layer, and a residual connection. The self-attention sub-layer utilizes a self-attention mechanism. The computation is performed using input vectors Q, K, and V; multiple heads mean multiple computations. The second sub-layer connection structure includes a fully connected sub-layer, a normalization layer, and a residual connection.

[0016] In step S3, the attention-based multimodal fusion method includes an attention mechanism and a fully connected layer, specifically comprising the following steps:

[0017] S3-1: Obtain the attention feature Ft between audio and text, and obtain the attention feature Fp between audio and phonemes;

[0018] S3-2: The attention feature Ft, the attention feature Fp, and the audio feature Fa are connected through a fully connected layer network to obtain the multimodal feature F. fusion As multimodal information.

[0019] In step S3-1, the attention features are obtained as follows:

[0020]

[0021] Where Q is the query vector, K is the content vector, V is the value vector, Softmax is the activation function that performs a non-linear transformation on the input, and d is the dimension of the K vector. When obtaining the attention feature Ft between audio and text, the audio feature is used as Q, and the text feature is used as K and V. When obtaining the attention feature Fp between audio and phonemes, the audio feature is used as Q, and the phoneme feature is used as K and V. The attention feature Ft, the attention feature Fp, and the audio feature Fa are connected through a fully connected layer network to obtain the multimodal feature F. fusionAs multimodal information.

[0022] The Transformer-based decoder includes a multi-layer multi-head attention mechanism and a feedforward network.

[0023] Multimodal information is input into a Transformer-based decoder to output the corresponding 3D facial geometry animation.

[0024] The decoder consists of three stacked decoder layers. Each decoder layer comprises three sub-layer connection structures. The first sub-layer connection structure includes a multi-head self-attention sub-layer, a normalization layer, and a residual connection. The multi-head self-attention sub-layer utilizes a self-attention mechanism. The calculation is performed, and the inputs Q, K, and V are all input vectors. Multi-head means multiple calculations, and here the calculation is performed twice. The second sub-layer connection structure includes a multi-head attention sub-layer, a normalization layer, and a residual connection. The third sub-layer connection structure includes a fully connected sub-layer, a normalization layer, and a residual connection.

[0025] The present invention also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described multimodal-based three-dimensional face geometric animation method.

[0026] The beneficial effects of this invention are:

[0027] The multimodal 3D face geometric animation method provided by this invention encodes each multimodal input using a corresponding encoder to obtain corresponding features, and further obtains corresponding high-level representation features for modeling. When receiving phoneme information, it can accurately model the lip shape corresponding to the face; when receiving text information, it can include text semantic information to achieve emotion-independent representation, thus achieving more accurate representation in face animation.

[0028] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the method flow in an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of the overall network architecture in an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of the multimodal fusion layer architecture in an embodiment of the present invention. Detailed Implementation

[0032] To make the technical solutions and advantages of the present invention clearer, the technical solutions of the embodiments of the present invention will be fully described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0033] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0034] In existing technologies, 3D face animation is mainly based on voice-driven methods. By inputting a specified voice, a 3D face is driven to obtain the corresponding animation. However, this method does not make full use of the semantic information of the text corresponding to the voice. At the same time, the lip-sync animation obtained by using regression training is prone to over-smoothing, resulting in poor performance.

[0035] To solve the above problems, such as Figure 2 In some embodiments, the proposed three-dimensional face geometric animation method of the present invention involves inputting audio and text, using existing tools to convert the text into a phoneme sequence, using pre-trained wav2vec to encode the audio to obtain audio features Fa (Faudio), using pre-trained BERT to encode the text to obtain text features Ft (Ftext), using Pre-net to encode the phoneme sequence to obtain phoneme features Fp (Fphone), inputting the above phoneme features and audio features into a Transformer encoder to obtain high-level representations of phonemes and text, inputting the above high-level representations and text features into a multimodal fusion layer to obtain multimodal information, and then inputting it into a Transformer decoder to obtain three-dimensional face geometric animation.

[0036] In some specific implementations, such as Figure 1 The steps of this invention are as follows:

[0037] S1: Receive multimodal input, encode it through the corresponding encoder, and obtain the features corresponding to each modality;

[0038] S2: For each modality, the features are encoded by an encoder to obtain the high-level representation features corresponding to each modality;

[0039] In some embodiments, the encoder is a Transformer-based encoder;

[0040] S3: Use an attention-based multimodal fusion method to fuse the high-level representation features corresponding to each modality to obtain multimodal information;

[0041] S4: The multimodal information is decoded by a Transformer-based decoder to obtain the corresponding 3D face geometric animation.

[0042] The multimodal input includes three modalities: audio, text, and phonemes.

[0043] In step S1, the discrete text, phonemes, and audio are transformed into features by the encoder. The audio is then encoded using a pre-trained wav2vec network, which consists of multiple temporal encoding networks. Using pre-trained wav2vec allows for the encoding of audio into more accurate semantic information using pre-trained knowledge. Here, the audio is encoded into 256-dimensional features. The text is then encoded using a pre-trained BERT network (pre-training refers to training the network on large-scale data to obtain the corresponding weights. After pre-training, the network has a higher level of semantic understanding and can better encode the input text, audio, and phonemes) to obtain 256-dimensional text features. BERT is a Transformer network architecture, and pre-trained BERT can better represent the text. Finally, the phonemes are encoded using a Pre-Net to obtain 256-dimensional phoneme features, where the Pre-Net contains multiple convolutional layers and linear layers.

[0044] In step S2, the Transformer-based encoder includes a multi-head self-attention mechanism and a feedforward network.

[0045] The Transformer encoder consists of three encoder layers, each composed of two sub-layer connection structures. The first sub-layer connection structure includes a multi-head self-attention sub-layer, a normalization layer, and a residual connection. The self-attention sub-layer utilizes a self-attention mechanism. The calculation is performed, with Q, K, and V being input vectors. Multiple heads mean multiple computations; here, attention is calculated twice. The second sub-layer connection structure includes a fully connected sub-layer, a normalization layer, and a residual connection. Step S2 yields the corresponding further phoneme and audio features.

[0046] In step S3, the attention-based multimodal fusion method employs a multimodal fusion network, which includes an attention mechanism (attention feature Ft) and a fully connected layer, specifically comprising the following steps:

[0047] S3-1: Obtain the attention feature Ft between audio and text, and obtain the attention feature Fp between audio and phonemes;

[0048] S3-2: The attention feature Ft, the attention feature Fp, and the audio feature Fa are connected through a fully connected layer network to obtain the multimodal feature F. fusion As multimodal information.

[0049] In step S3-1, the attention features are obtained as follows:

[0050]

[0051] like Figure 3 In this context, Q is the query vector, K is the content vector, V is the value vector, Softmax is the activation function that performs a non-linear transformation on the input, and d is the dimension of the K vector. When calculating the attention between audio and text, the audio feature Ft (i.e., Attention(Q,K,V)) is used as Q, and the text features are used as K and V, resulting in a 256-dimensional feature vector. Similarly, when calculating the attention between audio and phonemes, the audio feature Fp is used as Q, and the phoneme features are used as K and V, resulting in a 256-dimensional feature vector. These two attention features Fp and Ft, along with the audio feature Fa, are input into a fully connected network to obtain the final multimodal feature F. fusion , where F fusion It is a 256-dimensional feature vector.

[0052] The Transformer-based decoder includes a multi-layer multi-head attention mechanism and a feedforward network.

[0053] Multimodal information is input into a Transformer-based decoder to output the corresponding 3D facial geometry animation.

[0054] The decoder consists of three stacked decoder layers. Each decoder layer comprises three sub-layer connection structures. The first sub-layer connection structure includes a multi-head self-attention sub-layer, a normalization layer, and a residual connection. The multi-head self-attention sub-layer utilizes a self-attention mechanism. The calculation is performed, and the inputs Q, K, and V are all input vectors. Multi-head means multiple calculations, and here it is calculated twice. The second sub-layer connection structure includes a multi-head attention sub-layer, a normalization layer, and a residual connection. The third sub-layer connection structure includes a fully connected sub-layer, a normalization layer, and a residual connection.

[0055] In some embodiments, the present invention also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described multimodal-based three-dimensional face geometric animation method.

[0056] The embodiments of the present invention have the following advantages:

[0057] Phoneme-based modeling can accurately model the lip movements corresponding to human faces, while text-based modeling can incorporate textual semantic information to achieve emotion-independent representations, enabling more accurate representations in facial animations.

[0058] Based on multimodal information from text, audio, and phonemes, a 3D face model can be driven to more accurately represent the corresponding facial lip movements.

[0059] The attention-based multimodal fusion layer (multimodal fusion network) fully integrates multimodal information.

[0060] In the description of this specification, references to terms such as "an embodiment" and "example" refer to specific features, structures, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms are not necessarily intended to refer to corresponding embodiments or examples in a suitable manner.

[0061] It must be pointed out that the above description of the embodiments is not intended to limit the invention but only to help understand the core idea of ​​the invention. For those skilled in the art, any improvements to the invention and equivalent alternatives made to the invention without departing from the principle of the invention are also within the scope of protection of the claims of the invention.

Claims

1. A three-dimensional facial geometric animation method, characterized in that, Includes the following steps: S1: Receive multimodal input, encode it using the corresponding encoder, and obtain the features corresponding to each modality; wherein, the multimodal input includes three modalities: audio, text, and phonemes; use pre-trained wav2vec to encode audio to obtain audio features Fa, use pre-trained BERT to encode text to obtain text features Ft, and use Pre-net to encode phoneme sequences to obtain phoneme features Fp; pre-training refers to training wav2vec and BERT on large-scale data to obtain the weights corresponding to wav2vec and BERT. After pre-training, it has a higher level of semantic understanding ability and can better encode the input text, audio, and phonemes; S2: Input the phoneme features and audio features into the Transformer-based encoder for encoding to obtain high-level representation features of phonemes and audio; the Transformer-based encoder includes a multi-head self-attention mechanism and a feedforward network; S3: Using an attention-based multimodal fusion method, the high-level representation features of the phonemes and audio, along with text features, are input into the multimodal fusion layer to obtain multimodal information, specifically including: S3-1: Obtain the attention feature Ft between audio and text, and obtain the attention feature Fp between audio and phonemes; S3-2: Connect the attention feature Ft and the attention feature Fp, along with the audio feature Fa, through a fully connected layer network. Input the resulting two attention features Fp and Ft, along with the audio feature Fa, into the fully connected network to obtain the final multimodal features. , as multimodal information; S4: Input the multimodal information into a Transformer-based decoder for decoding and output the corresponding 3D facial geometric animation; among them, the phoneme-based modeling method can accurately model the lip shape corresponding to the face, and the text-based modeling method contains text semantic information to achieve a representation that is independent of emotion, thus achieving a more accurate representation in facial animation.

2. The three-dimensional face geometric animation method according to claim 1, characterized in that, In step S1, the audio is encoded using a pre-trained wav2vec network, which consists of multiple temporal coding networks. Using pre-trained wav2vec can utilize pre-trained knowledge to encode the audio into more accurate semantic information. The text is encoded using a pre-trained BERT network to obtain text features. BERT is a Transformer network architecture. The phonemes are encoded using Pre-net to obtain phoneme features. Pre-net contains multiple convolutional layers and linear layers.

3. The three-dimensional face geometric animation method according to claim 1, characterized in that, In step S3-1, the attention features are obtained as follows: Where Q is the query vector, K is the content vector, V is the value vector, Softmax is the activation function that performs a non-linear transformation on the input, and d is the dimension of the K vector; this is used to obtain attention features between audio and text. At that time, audio features are used as Q, and text features are used as K and V; attention features between audio and phonemes are obtained. At that time, the audio feature is represented as Q, and the phoneme features are represented as K and V. The attention feature Ft and the attention feature Fp are connected to the audio feature Fa through a fully connected layer network to obtain the multimodal feature. As multimodal information.

4. The three-dimensional face geometric animation method according to claim 1, characterized in that, The Transformer-based decoder includes a multi-layer multi-head attention mechanism and a feedforward network.

5. The three-dimensional face geometric animation method according to claim 1, characterized in that, The decoder consists of three stacked decoder layers. Each decoder layer comprises three sub-layer connection structures. The first sub-layer connection structure includes a multi-head self-attention sub-layer, a normalization layer, and a residual connection. The multi-head self-attention sub-layer utilizes a self-attention mechanism. The calculation is performed, and the inputs Q, K, and V are all input vectors. Multi-head means multiple calculations. The second sub-layer connection structure includes a multi-head attention sub-layer, a normalization layer, and a residual connection. The third sub-layer connection structure includes a fully connected sub-layer, a normalization layer, and a residual connection.

6. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the three-dimensional face geometric animation method according to any one of claims 1-5.

Images