Single-sample speaker generation method and system based on multi-modal three-dimensional plastic model

Abstract

The disclosure provides a single-sample speaker generation method and system based on a multi-modal three-dimensional plastic model, relating to the technical field of speaker generation, acquiring audio and a reference image, inputting into a multi-modal 3DMM parameter prediction network model, first extracting audio features and text features, and performing context feature enhancement on the audio features and the text features to obtain audio representations and semantic representations injected with context information; obtaining initial head pose embedding parameters and an initial expression description, predicting the head pose parameters of the next moment using the initial head pose embedding parameters; predicting the expression parameters of the next moment using the initial expression description, fusing and embedding the predicted head pose parameters and expression parameters into the reference image, and synthesizing a multi-frame image into a head video sequence of the speaker through a video synthesis network.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of speaker generation, in particular to a single-sample speaker generation method and system based on a multi-modal three-dimensional plastic model. BACKGROUND

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute the prior art.

[0003] The single-sample speaker generation method aims to synthesize a speaker video based on a single face image and a corresponding audio track, ensuring that the speaker's head movements in the video are natural, the lips are synchronized with the audio, and the video images have high fidelity.

[0004] As research in this field continues to advance, its practical application scope covers various fields, such as remote video communication, virtual anchors, e-learning frameworks, etc. In addition, in the process of exploring the metaverse, the discovered ecosystem is growing, and the reality relevance of this technology is becoming more and more obvious, and the required technology is also becoming higher and higher. In the emerging field of speaker generation, contemporary research mainly focuses on two key dimensions: the authenticity of head posture and the accuracy of lip synchronization. Recent research has attempted to use various techniques such as multi-layer perceptron (MLP), recurrent neural mechanisms, and creating audio-lip databases to improve the fidelity of these two dimensions. Although significant progress has been made in speaker generation technology after these studies, there are still two problems:

[0005] (1) Head posture prediction: There is some correlation between speech details and head movement. For example, in English, the amplitude or pitch of the sound has some correlation with the head movement dynamics (usually observed that emphasizing a word is accompanied by nodding and the raised pitch is consistent with the raised head posture). However, relying solely on audio cues can lead to oversimplification of head posture prediction, and existing methods still cannot capture the subtle relationship between word semantics and head posture dynamics, which is crucial for achieving natural and lifelike head movement.

[0006] (2) Detail texture authenticity: Although current lip synchronization technology in the dataset has made considerable progress, the technology for generating complex details still cannot achieve a breakthrough, which directly leads to a decrease in the predicted natural authenticity and the rendered realism. SUMMARY

[0007] The present disclosure proposes a single-sample speaker generation method and system based on a multi-modal three-dimensional plastic model to solve the above problems. The multi-modal three-dimensional plastic model (3D Morphable Model, 3DMM) parameter prediction and high-fidelity video synthesis method are used to ensure that the audio prompt is effectively converted into relevant facial expressions, improve the quality of the synthesized video, generate oral cavity detail texture (such as teeth) as perfectly as possible, and ensure that the generated texture is seamlessly synchronized with the corresponding audio stream.

[0008] According to some embodiments, the present disclosure adopts the following technical solutions:

[0009] The single-sample speaker generation method based on a multi-modal three-dimensional plastic model comprises:

[0010] The audio data and the reference image are obtained, the audio data is input into a multi-modal 3DMM parameter prediction network model, the audio features and the text features are first extracted, and the context feature enhancement is performed on the audio features and the text features to obtain the audio representation and the semantic representation injected with the context information;

[0011] The initial head pose embedding parameters and the initial expression description are obtained using the reference image, the head pose parameters at the next moment are predicted using the initial head pose embedding parameters and the audio representation and the semantic representation at the next moment, and the expression parameters at the next moment are predicted using the initial expression description and the audio representation at the next moment;

[0012] The predicted head pose parameters and expression parameters are embedded into the reference image, and the multi-frame image is synthesized into a head video sequence of the speaker by a video synthesis network.

[0013] According to some embodiments, the present disclosure adopts the following technical solutions:

[0014] The single-sample speaker generation system based on a multi-modal three-dimensional plastic model comprises:

[0015] The data acquisition module acquires the audio data and the reference image, inputs the audio data into a multi-modal 3DMM parameter prediction network model, first extracts the audio features and the text features, and performs context feature enhancement on the audio features and the text features to obtain the audio representation and the semantic representation injected with the context information;

[0016] The prediction module is configured to obtain the initial head pose embedding parameters and the initial expression description using the reference image, predict the head pose parameters at the next moment using the initial head pose embedding parameters and the audio representation and the semantic representation at the next moment, and predict the expression parameters at the next moment using the initial expression description and the audio representation at the next moment;

[0017] The synthesis module is configured to fuse the predicted head pose parameters and expression parameters into a reference image, and synthesize a head video sequence of the speaker through a video synthesis network.

[0018] Compared with the prior art, the present disclosure has the following beneficial effects:

[0019] The present disclosure provides a single-sample speaker generation method based on a multi-modal three-dimensional plastic model, which uniquely focuses on the complexity of semantic-aware head pose prediction and the subtle differences of texture-rich oral cavity rendering, and has a groundbreaking discovery in exploring and utilizing the complex relationship between word semantics and the dynamic head pose of speaker portrait synthesis.

[0020] The present disclosure innovatively combines an enhanced oral network with a local discriminator component, which plays a key role in the high-fidelity video synthesis stage and enhances the authenticity of the oral region, especially focusing on the texture details of the teeth region.

[0021] The generated speaker video of the present disclosure is significantly superior to other existing methods in terms of visual quality and naturalness of head pose. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which form a part of the present disclosure, are intended to provide further understanding of the present disclosure, and the illustrative embodiments thereof and their descriptions serve to explain the present disclosure, and are not intended to limit the present disclosure.

[0023] Figure 1 The model method architecture diagram of the embodiment of the present disclosure;

[0024] Figure 2 The structure diagram of the multi-modal pose decoder of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The present disclosure will be further described below in conjunction with the accompanying drawings and embodiments.

[0026] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs.

[0027] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, the singular form is intended to include the plural form, unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, they indicate the presence of a feature, step, operation, device, component and / or combination thereof.

[0028] Embodiment 1

[0029] In an embodiment of the present disclosure, a single-sample speaker generation method based on a multi-modal three-dimensional plastic model is provided, comprising:

[0030] Step one: obtain audio data and reference images, input the audio data into a multi-modal 3DMM parameter prediction network model, first extract audio features and text features, and perform context feature enhancement on the audio features and text features to obtain audio representations and semantic representations injected with context information;

[0031] Step two: embed the initial head pose parameters and the initial expression description obtained using the reference images, and use the initial head pose parameters and the next time audio representation and semantic representation to predict the head pose parameters of the next time; use the initial expression description to fuse the next time audio representation to predict the next time expression parameters;

[0032] Step three: embed the predicted head pose parameters and expression parameters into the reference images, and synthesize the speaker's head video sequence through a video synthesis network.

[0033] As an embodiment, the present disclosure proposes a high-fidelity single-sample speaker generation method, which is named as HEAD. In structure, the method is divided into two main modules: 1) multi-modal three-dimensional plastic model (3D Morphable Model, 3DMM) parameter prediction: in the preliminary module, a multi-media collaborative parameter prediction network is introduced, which cooperates with the text semantic mainly to improve the accuracy of head posture prediction and ensure that the audio prompt is effectively converted into the relevant facial expression. 2) high-fidelity video synthesis (High-Fidelity Video Synthesis): from the basic prediction, the subsequent modules all refer to a generative adversarial network (Generative Adversarial Network) enhanced by mouth rendering to improve the quality of the synthesized video, to generate the mouth detailed texture as perfectly as possible, and to ensure that the generated texture is seamlessly synchronized with the corresponding audio stream.

[0034] Specifically, it comprises:

[0035] (1) Problem definition

[0036] The 3D shape of the given face image parameterization is determined by a three-dimensional face reconstruction model, and the shape is recorded as S, which is specifically represented as:

[0037]

[0038] wherein, B represents the average facial shape. id and B exp Principal component analysis is based on the shapes and expressions, respectively, and the parameters are... and These represent a person's identity and facial expression, respectively.

[0039] The three-dimensional-to-two-dimensional projection geometry of this model is captured using a perspective camera model, through the rotation matrix R∈SO(3) and the translation vector. To display the 3D head pose p, p is encapsulated by the set {R, t}. SO(3) is a set containing all rotation operations that maintain the length and angle in three-dimensional space.

[0040] (2) Multimodal 3DMM parameter prediction

[0041] This disclosure introduces a multimodal 3DMM parameter prediction network model to simultaneously utilize the correlation between audio and text heads to derive accurate head pose parameter estimates. Figure 1 As shown, this model mainly consists of three interconnected components: multimodal feature extraction: collecting necessary features from auditory and text inputs; contextual feature enhancement: injecting temporal contextual information into multimodal features; and pose and expression decoding: converting the audio and text representations with injected contextual information into specific head poses and facial expressions. These components are explained in detail below:

[0042] 1) Multimodal feature extraction

[0043] For audio feature extraction, the initial input audio is first segmented into non-overlapping parts, denoted as A. 1:T Where T represents the total number of slices. Each slice A i A 45-dimensional audio feature vector a can be calculated from both. i a i This includes a set of 14-dimensional Mel-frequency cepstral coefficients, 28-dimensional Mel-frequency cepstral coefficients delta, and additional parameters such as energy, loudness, and zero-crossing rate. Then, the continuous audio features are normalized to extract feature s. 1:T ,in, This represents the feature belonging to the i-th audio slice.

[0044] For text feature extraction, the input audio is transcribed into text format using the Aigoer Speech Recognition tool, denoted as. To ensure the synchronization of 3DMM parameter prediction, the text information and audio information sequence length are made consistent by using the Gentle Forced Aligner. Meanwhile, the translated text is converted into text features using the pre-trained FastText word embedding network, where each vector has a dimension of 300.

[0045] 2) Contextual Enhancement

[0046] To absorb the contextual nuances inherent in the sequence of audio features s 1:T and text features w 1:T , this method uses an audio-oriented encoder E a and a text-oriented encoder E w , respectively.

[0047] The architecture of the audio encoder E a consists of a four-layer bidirectional LSTM and a fully connected layer, which is mathematically expressed as:

[0048]

[0049] where, represents the final contextually enriched audio representation.

[0050] The architecture of the text encoder E w consists of a four-layer temporal convolutional network (Temporal Convolutional Network), which is designed to skillfully capture the interwoven extended temporal dependencies in text features. The encoder is mathematically expressed as:

[0051]

[0052] where, represents the contextually sensitive semantic representation derived from the input.

[0053] 3) Head Pose and Facial Expression Decoding

[0054] The initial head pose embedding parameters are obtained using the reference image, and the next time predicted head pose parameters are generated using the initial head pose embedding parameters and the next time audio features and semantic representations. The initial expression description is obtained using the reference image, and the next time expression parameters are predicted in combination with the next time audio features.

[0055] [3-1] Multimodal Pose Decoder D p

[0056] The multimodal pose decoder D pto identify the complex relationship between head pose and word semantic information. First, a three-layer Multilayer Perceptron (MLP) is used to process the primary 3DMM pose parameters p0obtained from audio and text data to generate the initial head pose embedding P0= MLP(p0). p0 (p0).

[0057] The context-sensitive audio representation and semantic representation obtained by the fusion encoder are fused to generate a merged data stream wherein represents a concatenation operation. The decoder D p is constructed by a two-layer Long Short-Term Memory (LSTM) network and can simultaneously process P0and to predict the sequence of head poses;

[0058] To improve the authenticity of the predicted head pose, the head pose embedding and audio information of the previous segment are fused at each calculation stage, and the process is as follows Figure 2 The process is explained by taking the ith iteration as an example. P i-1 , and are combined and sent to the LSTM for processing, and then a three-layer MLP is used to predict the head pose The process can be mathematically expressed as:

[0059]

[0060] wherein H i represents the hidden state of the ith iteration, and the final head pose sequence is obtained

[0061] [3-2] Audio representation decoder D e

[0062] According to the conclusion that there is little correlation between text data and facial expressions, especially mouth movements, the method only uses context-aware audio representation to predict the expression parameters. First, an initial expression E0is obtained by a three-layer MLP After that, the expression description E0and the context-aware audio data are passed to the decoder D e , which is constructed by a two-layer LSTM. The specific steps in the decoder D e are as follows: at each jth step, the previous expression description E j-1 and the current context-rich audio representation The data is combined, and the combined data is used to derive facial expression parameters through another three-layer MLP.

[0063]

[0064] Because emotions have an inherent consistency over a short period of time, predictions for subsequent steps only consider the output of the previous step.

[0065] To achieve a natural and smooth synthesis result, this method restricts the generated parameters to a range close to the true values. The final loss function of the multimodal 3DMM parameter prediction framework is:

[0066]

[0067] Wherein, parameter β t and p t Let λ1 and λ2 represent the facial expression and pose parameters of the t-th real image, respectively. The function μ(·) represents the inter-frame variation between the current frame and its consecutive previous frame. The parameters λ1 and λ2 are predefined balancing parameters.

[0068] (3) High-fidelity video synthesis

[0069] After fusing and embedding the obtained predicted pose and expression parameters into a reference image, a high-fidelity video synthesis network is used to generate a speaking head video. This network structure includes a generator component and a pair of discriminator components, which are described in detail below:

[0070] generator

[0071] The generator has three core components: a twisted network G. w Editing Network G e and mouth enhancement network G m For G respectively w G e and G m The work performed will be described in detail:

[0072] [1-1] Twisted Network G w

[0073] First, to generate the i-th frame image, the intermediate motion representation d is generated through a series of 1D convolutional blocks. i ={β i ,p i} is transformed into a continuously generated latent vector z i Then, this method is based on the obtained latent vector z i Construct a twisted network with a U-Net structure, namely G w This structure consists of five convolutional downsampling layers followed by three residual transposed convolutional layers. wThe goal is to synthesize the flow field w, and use the flow field w to deform the reference image I into the i-th preliminary image, i.e.

[0074] [1-2] Edit network G e

[0075] To further enhance the image, this method employs an editing network G. e Its characteristics include six residual convolutional downsampling blocks accompanied by three transposed convolutional layers. G e The goal is to consider the reference image I and the latent vector z i To optimize Wherein, the i-th potential vector z i Integrating into G via the AdaIN process w and G e middle.

[0076] The mathematical expression for this integration process is:

[0077]

[0078] To enhance G w and G e To optimize the model's generative capabilities, a perceptual loss framework is employed. The mathematical expression for this process is as follows:

[0079]

[0080] in, This represents the desired target image. Let L represent the activation graph of the i-th layer of VGG-19, and let L represent the total number of layers in VGG-19.

[0081] [1-3] Oral reinforcement network G m

[0082] In order to solve The teeth area often appears blurry; this model introduces an oral enhancement network G. m G m The goal is to improve the rendering of tooth details. m The structure is a copy-edit network G e The architecture is similar, but it more significantly integrates context-aware representations. Refined Images And an initial mouth image, denoted as M. 1:T The i-th final image The mathematical representation of the synthesis process is as follows:

[0083]

[0084] where denotes element-wise multiplication, and M i The generation can be traced back to the division of the mouth region on the facial landmark. In the training phase, M i is extracted from the reference image by dlib, and in the inference phase, M i coordinates come from This method adopts the strategy of averaging the labels of the five adjacent regions in the inference phase to offset potential noise interference.

[0085] Discriminator

[0086] To refine and optimize the generated images, this method designs two different discriminators: a global discriminator D global and a local discriminator D local . D glolal provides an overall global view, while D local focuses on complex local details. This dual perspective ensures a comprehensive evaluation of the synthetic images, promoting large-scale accuracy and detail preservation. D global and D local are described in detail as follows:

[0087] [2-1] Global Discriminator D global

[0088] Due to the superiority of PatchGAN in preserving high-frequency information, the structure of D global is designed with a multi-scale PatchGAN framework. This structure is suitable for various resolutions, especially at the full, half, and quarter scales of the image, thus enabling the model to conduct in-depth examination of complex details.

[0089] In a given scenario involving a pair of images, such as or D global generates an N x N matrix, where N represents the number of image patches. The global loss of this matrix is mathematically represented as:

[0090]

[0091] where, is used to quantify the similarity between the synthetic image and its corresponding real image, aiming to minimize the deviation. This formula includes an expectation operation to assess the fidelity of the generated image patches and their difference from the real image patches, thereby optimizing the generation process across multiple scales.

[0092] [2-2] Local Discriminator D local

[0093] Local discriminator D​local The structure is constructed using a VGG-inspired architecture from the GFP-GAN method. For D... local The specific working steps are explained in detail as follows: First, the oral cavity is extracted using region-of-interest (ROI) alignment technology. After extraction, a D-structure is constructed. local The architecture, namely the D inspired by VGG mentioned above. local Architecture. Next, we define the local loss function:

[0094]

[0095] ROI refers to the oral cavity region.

[0096] Subsequently, the loss function is defined for the adversarial part of the bifurcation method in this paper:

[0097]

[0098] Where, λ global and λ local It is a hyperparameter that balances the loss of the two discriminators.

[0099] Ultimately, the fundamental goal of the high-fidelity video synthesis mechanism is to achieve the synergistic fusion of equations (8) and (12). The mathematical representation of this synthesis relationship is as follows:

[0100]

[0101] Among them, coefficients λ3, λ4 and λ5 are used to balance the correlation loss.

[0102] Example 2

[0103] One embodiment of this disclosure provides a single-sample speaker generation system based on a multimodal three-dimensional plastic model, comprising:

[0104] The data acquisition module acquires audio data and reference images, inputs the audio data into the multimodal 3DMM parameter prediction network model, first extracts audio features and text features, and then performs context feature enhancement on the audio features and text features to obtain audio representations and semantic representations injected with context information.

[0105] The prediction module is used to obtain initial head pose embedding parameters and initial facial expression description using reference images, and to predict the head pose parameters of the next time step by fusing the initial head pose embedding parameters, the audio representation of the next time step, and the semantic representation of the next time step; and to predict the facial expression parameters of the next time step by fusing the initial facial expression description with the audio representation of the next time step.

[0106] The synthesis module is configured to fuse the predicted head pose parameters and expression parameters into a reference image, and synthesize a head video sequence of the speaker by using a video synthesis network.

[0107] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and a combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed by the computer or other programmable data processing apparatus provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The functions specified in one or more flows and / or blocks

[0109] Although the specific embodiments of the present disclosure are described above with reference to the accompanying drawings, the present disclosure is not limited to the above-described embodiments, and various modifications or changes can be made to the embodiments without departing from the scope of the present disclosure.

Claims

1. A single-sample speaker generation method based on a multimodal three-dimensional plasticity model, characterized in that, include: Acquire audio data and reference images, input the audio data into the multimodal 3DMM parameter prediction network model, first extract audio features and text features, and then perform context feature enhancement on the audio features and text features to obtain audio representation and semantic representation injected with context information. The contextual feature enhancement of audio and text features is performed using an audio-oriented encoder. and text-oriented encoders ; Audio encoder The architecture consists of a four-layer bidirectional LSTM and a fully connected layer. The encoder is mathematically expressed as follows: in, This represents the final, context-rich audio representation; Text encoder The architecture consists of a four-layer temporal convolutional network, designed to skillfully capture the interwoven extended temporal dependencies in text features. The encoder's mathematical expression is: in, This represents a context-sensitive semantic representation derived from the input; Initial head pose embedding parameters and initial facial expression description are obtained using reference images. The initial head pose embedding parameters, the audio representation and semantic representation of the next time step are fused together to predict the head pose parameters of the next time step. The initial facial expression description is fused together with the audio representation of the next time step to predict the facial expression parameters of the next time step. The predicted head pose and facial expression parameters are fused and embedded into the reference image, and the multi-frame images are synthesized into a video sequence of the speaker's head through a video synthesis network.

2. The single-sample speaker generation method based on a multimodal three-dimensional plastic model as described in claim 1, characterized in that, In the multimodal 3DMM parameter prediction network model, the initial input audio data is segmented into non-overlapping slices, and a multi-dimensional audio feature vector is calculated for each slice. The audio feature vector includes Mel frequency cepstral coefficients, the difference of Mel frequency cepstral coefficients, and parameters such as energy, loudness, and zero-crossing rate. The continuous audio features are normalized to extract the audio features, and then the audio encoder is used to enhance the context features to obtain an audio representation injected with context information.

3. The single-sample speaker generation method based on a multimodal three-dimensional plastic model as described in claim 1, characterized in that, For text data, the input audio is first transcribed into text data, the lengths of the text data and audio data sequences are aligned, text features are extracted from the text data through a word embedding network, and then contextual feature enhancement is performed using a text encoder to obtain a semantic representation injected with contextual information.

4. The single-sample speaker generation method based on a multimodal three-dimensional plastic model as described in claim 1, characterized in that, A 3D facial reconstruction model is used to determine the parameterized 3D shape of a given initial image. A multilayer perceptron is used to process the initial 3DMM pose parameters obtained from audio and text features to generate an initial head pose embedding. Audio features and semantic representations are fused to produce a merged data stream. A two-layer long short-term memory network is used to predict the sequence of head poses by simultaneously processing the merged data stream and the head pose embedding.

5. The single-sample speaker generation method based on a multimodal three-dimensional plastic model as described in claim 4, characterized in that, During the prediction phase, each prediction phase integrates the head pose embedding and merged data streams from the previous moment, and performs multiple iterations to obtain the final head pose embedding parameters.

6. The single-sample speaker generation method based on a multimodal three-dimensional plastic model as described in claim 1, characterized in that, An initial facial expression description is obtained through a three-layer MLP. The initial facial expression description and audio representation are then input into a two-layer long short-term memory network. At each prediction step of the current time step, the facial expression description from the previous time step is combined with the current audio representation. The combined data is then used to derive the facial expression parameters through another three-layer MLP.

7. The single-sample speaker generation method based on a multimodal three-dimensional plastic model as described in claim 1, characterized in that, The video synthesis network includes a generator and a discriminator. The generator includes a warping network, an editing network, and an mouth enhancement network. The warping network includes five convolutional downsampling layers followed by three residual transposed convolutional layers. The goal of the warping network is to synthesize a flow field, using the flow field to deform a reference image into a preliminary image.

8. The single-sample speaker generation method based on a multimodal three-dimensional plastic model as described in claim 7, characterized in that, The editing network consists of a six-layer residual convolutional downsampling module accompanied by three transposed convolutional layers. The goal of the editing network is to optimize the initial image by considering the reference image and the latent vectors, and to integrate the latent vectors into the editing network and the warping network through the AdaIN process.

9. The single-sample speaker generation method based on a multimodal three-dimensional plastic model as described in claim 7, characterized in that, An oral enhancement network is introduced to improve the representation of tooth details. The oral enhancement network replicates the architecture of the editing network. A loss function is defined and synergistically fused with the perceptual loss to achieve the overall goal definition of the video synthesis mechanism.

10. A single-sample speaker generation system based on a multimodal three-dimensional plastic model, characterized in that, include: The data acquisition module acquires audio data and reference images, inputs the audio data into the multimodal 3DMM parameter prediction network model, first extracts audio features and text features, and then performs context feature enhancement on the audio features and text features to obtain audio representations and semantic representations injected with context information. The contextual feature enhancement of audio and text features is performed using an audio-oriented encoder. and text-oriented encoders ; Audio encoder The architecture consists of a four-layer bidirectional LSTM and a fully connected layer. The encoder is mathematically expressed as follows: in, This represents the final, context-rich audio representation; Text encoder The architecture consists of a four-layer temporal convolutional network, designed to skillfully capture the interwoven extended temporal dependencies in text features. The encoder's mathematical expression is: in, This represents a context-sensitive semantic representation derived from the input; The prediction module is used to obtain initial head pose embedding parameters and initial facial expression description using reference images, and to predict the head pose parameters of the next time step by fusing the initial head pose embedding parameters, the audio representation of the next time step, and the semantic representation of the next time step; and to predict the facial expression parameters of the next time step by fusing the initial facial expression description with the audio representation of the next time step. The synthesis module is used to fuse and embed the predicted head pose parameters and facial expression parameters into the reference image, and synthesize the speaker's head video sequence from multiple frames of images through the video synthesis network.

Images