Three-dimensional face synthesis method, device, equipment and storage medium

Abstract

The present invention relates to the field of computer vision, and discloses a three-dimensional face synthesis method, device, equipment and storage medium, the method comprising: extracting facial features from a collected face image and extracting voice features from a collected person's voice; encoding the facial features by a preset image encoder to obtain encoded facial features; encoding the voice features by a preset voice encoder to obtain encoded voice features; inputting the encoded facial features and the encoded voice features into a diffusion model for three-dimensional face synthesis to obtain a three-dimensional face mesh sequence. The present invention encodes the facial features and the voice features by an encoder, and predicts the face movement based on the obtained encoded facial features and the encoded voice features by a diffusion model, thereby realizing three-dimensional face synthesis and obtaining a three-dimensional face mesh sequence matching the face image and the person's voice, so as to be able to quickly and accurately synthesize a three-dimensional face.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a three-dimensional face synthesis method, device, equipment and storage medium. Background Art

[0002] Nowadays, virtual digital human technology is widely used in scenarios such as virtual idols and virtual hosts. At present, virtual digital humans are usually synthesized by real-person driving. In virtual digital humans synthesized by real-person driving, real people need to cooperate with motion capture equipment so that virtual digital humans can interact with the audience in real time.

[0003] However, although the virtual digital human synthesized by real-life driving is more realistic and the character movements are more accurate, its application scenarios are limited due to the complex data processing process and high equipment costs. Therefore, the industry is in urgent need of a method that has strong applicability and can quickly and accurately synthesize three-dimensional human faces.

[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present invention is to provide a three-dimensional face synthesis method, device, equipment and storage medium, aiming to solve the technical problem that the prior art cannot quickly and accurately synthesize three-dimensional faces.

[0006] To achieve the above object, the present invention provides a 3D face synthesis method, which comprises the following steps:

[0007] Extract facial features from the collected face images and extract voice features from the collected person voices;

[0008] Encoding the facial features by using a preset image encoder to obtain encoded facial features;

[0009] Encoding the speech feature by a preset speech encoder to obtain the encoded speech feature;

[0010] The encoded facial features and the encoded speech features are input into a diffusion model for three-dimensional face synthesis to obtain a three-dimensional face mesh sequence. The diffusion model is used to predict face motion based on the encoded facial features and the encoded speech features.

[0011] Optionally, the step of extracting facial features from the collected face image includes:

[0012] Performing three-dimensional face modeling on the collected face image to obtain a three-dimensional face template, wherein the collected face image is a single-view face image;

[0013] Determining morphological information and texture information of a face from the three-dimensional face template, and reconstructing the face of the three-dimensional face template based on the morphological information and the texture information to obtain a reconstructed three-dimensional face template;

[0014] Facial features are extracted from the reconstructed three-dimensional face template.

[0015] Optionally, the step of extracting voice features from the collected character voice includes:

[0016] Performing short-time Fourier transform on the collected character voice to obtain a short-time speech frame corresponding to the collected character voice;

[0017] The time domain signal corresponding to the short-term speech frame is converted into a frequency domain signal through Fourier transform, and the speech feature is extracted according to the energy distribution of the frequency domain signal in the frequency domain.

[0018] Optionally, before the steps of extracting facial features from the collected face image and extracting voice features from the collected person's voice, the step further includes:

[0019] Performing image preprocessing on the original face image, and determining the preprocessed original face image as the collected face image, the image preprocessing including resolution adjustment, image cropping, image alignment and image denoising;

[0020] The original human voice is subjected to speech preprocessing, and the preprocessed original human voice is determined as the collected human voice, wherein the speech preprocessing includes endpoint detection, speech pre-emphasis, speech framing, speech normalization and speech denoising.

[0021] Optionally, the step of encoding the facial features by using a preset image encoder to obtain the encoded facial features includes:

[0022] Mapping the facial features to a latent feature space by a preset image encoder, wherein the preset image encoder is a Transformer-VAE encoder, and the Transformer-VAE encoder is constructed based on a Transformer structure and a variational autoencoder;

[0023] The facial features are encoded in the latent feature space to obtain encoded facial features, and the encoding process includes a compression process, an abstraction process and a decoupling process of the facial features.

[0024] Optionally, the step of encoding the speech feature by a preset speech encoder to obtain the encoded speech feature includes:

[0025] The preset speech encoder includes a speech pre-training model and a CBE module;

[0026] The speech features are converted into a plurality of speech sequences by means of the speech pre-training model, and dependencies between the plurality of speech sequences are captured, wherein the speech pre-training model is constructed based on a recurrent neural network and / or a convolutional neural network;

[0027] The plurality of speech sequences are encoded in the CBE module based on the dependency relationship to obtain encoded speech features.

[0028] Optionally, the step of inputting the encoded facial features and the encoded speech features into a diffusion model to perform three-dimensional face synthesis to obtain a three-dimensional face mesh sequence includes:

[0029] Inputting the encoded facial features and the encoded speech features into a diffusion model for decoding to obtain a predicted motion trajectory of the face;

[0030] The encoded facial features and the encoded voice features are subjected to three-dimensional face synthesis based on the predicted face motion trajectory, and a three-dimensional face mesh sequence is output after the encoded facial features and the encoded voice features achieve lip synchronization.

[0031] In addition, to achieve the above-mentioned purpose, the present invention also proposes a 3D face synthesis device, the 3D face synthesis device comprising:

[0032] A feature extraction module is used to extract facial features from the collected face images and extract voice features from the collected voices of the characters;

[0033] A first encoding module, used for encoding the facial features by using a preset image encoder to obtain encoded facial features;

[0034] A second encoding module, used for encoding the speech feature by a preset speech encoder to obtain the encoded speech feature;

[0035] A face synthesis module is used to input the encoded facial features and the encoded voice features into a diffusion model for three-dimensional face synthesis to obtain a three-dimensional face mesh sequence. The diffusion model is used to predict face movement based on the encoded facial features and the encoded voice features.

[0036] In addition, to achieve the above-mentioned purpose, the present invention also proposes a three-dimensional face synthesis device, which includes: a memory, a processor, and a three-dimensional face synthesis program stored in the memory and executable on the processor, wherein the three-dimensional face synthesis program is configured to implement the steps of the three-dimensional face synthesis method described above.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a three-dimensional face synthesis program is stored. When the three-dimensional face synthesis program is executed by a processor, the steps of the three-dimensional face synthesis method described above are implemented.

[0038] The present invention extracts facial features from a collected face image and extracts voice features from a collected person's voice; encodes the facial features through a preset image encoder to obtain encoded facial features; encodes the voice features through a preset voice encoder to obtain encoded voice features; inputs the encoded facial features and the encoded voice features into a diffusion model for three-dimensional face synthesis to obtain a three-dimensional face mesh sequence, and the diffusion model is used to predict face motion according to the encoded facial features and the encoded voice features. Compared with the traditional three-dimensional face synthesis method, the method of the present invention encodes the facial features and the voice features through an encoder, and predicts face motion based on the encoded facial features and the encoded voice features through a diffusion model, so as to realize three-dimensional face synthesis and obtain a three-dimensional face mesh sequence matching the face image and the person's voice, thereby avoiding the complex data processing flow and high equipment cost in the prior art, and thus being able to quickly and accurately synthesize three-dimensional faces and having strong applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the structure of a 3D face synthesis device in a hardware operating environment according to an embodiment of the present invention;

[0040] Figure 2 It is a flowchart of a first embodiment of a three-dimensional face synthesis method of the present invention;

[0041] Figure 3 Schematic diagram of the overall system architecture of the 3D face synthesis method of the present invention;

[0042] Figure 4 It is a flowchart of a second embodiment of a three-dimensional face synthesis method of the present invention;

[0043] Figure 5 It is a flowchart of a third embodiment of a three-dimensional face synthesis method of the present invention;

[0044] Figure 6 A schematic diagram of the user interface design of the three-dimensional face synthesis method of the present invention;

[0045] Figure 7 It is a structural block diagram of the first embodiment of the three-dimensional face synthesis device of the present invention.

[0046] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0048] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the structure of a 3D face synthesis device in the hardware operating environment involved in the embodiment of the present invention.

[0049] like Figure 1 As shown, the three-dimensional face synthesis device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0050] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the three-dimensional face synthesis device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0051] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a three-dimensional face synthesis program.

[0052] exist Figure 1 In the three-dimensional face synthesis device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the three-dimensional face synthesis device of the present invention can be set in the three-dimensional face synthesis device, and the three-dimensional face synthesis device calls the three-dimensional face synthesis program stored in the memory 1005 through the processor 1001, and executes the three-dimensional face synthesis method provided by the embodiment of the present invention.

[0053] The embodiment of the present invention provides a three-dimensional face synthesis method, referring to Figure 2 , Figure 2 FIG. 4 is a flow chart of a first embodiment of a three-dimensional face synthesis method according to the present invention.

[0054] In this embodiment, the 3D face synthesis method includes the following steps:

[0055] Step S10: extracting facial features from the collected face image and extracting voice features from the collected person's voice.

[0056] It should be noted that the execution subject of the method of this embodiment can be a terminal device with data collection, data processing and program running functions, such as a smart phone, a computer, etc., or an electronic device with the same or similar functions, such as the above-mentioned three-dimensional face synthesis device. The following takes the three-dimensional face synthesis device (hereinafter referred to as the synthesis device) as an example to illustrate this embodiment and the following embodiments.

[0057] It is understandable that the above-mentioned facial image can be in JPEG (Joint Photographic Experts Group) format, or in other image formats such as TIFF (Tagged Image File Format), GIF (Graphics Interchange Format), PNG (Portable Network Graphics), etc., and this embodiment does not impose any limitation on this.

[0058] It should be understood that the above-mentioned character voice can be in AAC (Advanced Audio Coding) format, or in AIFF (Audio Interchange File Format), PCM (Pulse-Code Modulation) format and other audio formats, and this embodiment does not limit this.

[0059] In the specific implementation, the position of the face needs to be detected from the face image first. After the face is detected, the next step is to align the face, which usually involves locating the key points of the face, such as the position of the eyes, nose, mouth, etc. Through the key points, the face image can be normalized so that the key points of different face images correspond to each other, which is convenient for subsequent feature extraction. After face alignment, facial features can be extracted based on the following methods. Methods based on geometric features: extract features by measuring the distance, angle and other geometric properties between the key points of the face. Methods based on deep learning: extract features from face images using deep learning models (such as convolutional neural networks (CNNs). These models are trained with a large amount of data and can learn deep-level feature representations of the face. Methods based on texture features: extract features by analyzing the texture information of the face image, such as using Gabor filters or local binary patterns (LBP).

[0060] Step S20: Encode the facial features using a preset image encoder to obtain encoded facial features.

[0061] In a specific implementation, since the facial features extracted by step S10 are usually a high-dimensional vector, it may be inconvenient for storage and transmission. Therefore, in this embodiment, these features can be encoded to convert them into a more compact and easier to process form. More specifically, there can be many ways of encoding, such as hash encoding, binary encoding, etc., which are not limited in this embodiment. The encoded features should be able to retain the information of the original features as much as possible while reducing the storage space.

[0062] Step S30: Encode the speech feature by using a preset speech encoder to obtain the encoded speech feature.

[0063] It should be understood that the above-mentioned preset speech encoder may include a Conformer model, and the Conformer model may be trained using a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0064] In a specific implementation, similarly, since the speech feature extracted by step S10 is also usually a high-dimensional vector, it may be inconvenient for storage and transmission. Therefore, in order to obtain a more concise, easier to store and compare code, the speech feature can be further encoded in this embodiment to obtain the above-mentioned encoded speech feature. The encoding method can be hash coding, binary coding or other methods suitable for feature compression and representation, which is not limited in this embodiment.

[0065] Step S40: Inputting the encoded facial features and the encoded speech features into a diffusion model to perform three-dimensional face synthesis to obtain a three-dimensional face mesh sequence, wherein the diffusion model is used to predict face motion based on the encoded facial features and the encoded speech features.

[0066] It should be noted that the above diffusion model is a three-dimensional face synthesis model based on the diffusion equation, which can be solved by the finite element method.

[0067] In a specific implementation, since the encoded facial features and the encoded speech features belong to different modalities, before performing 3D face synthesis, the fusion layer or fusion algorithm designed in the diffusion model can be used to enable the encoded facial features and the encoded speech features to interact and integrate in the same dimension, thereby achieving 3D face synthesis and obtaining the above 3D face mesh sequence. Figure 3 , Figure 3 Schematic diagram of the overall system architecture of the 3D face synthesis method of the present invention. Figure 3 In this paper, the 3D face synthesis method can be divided into the business logic layer (including core algorithms and data reading and writing), the user interface layer (including user input, face visualization, face synthesis and face access) and the data storage layer (including 3D face data in npy format files).

[0068] This embodiment extracts facial features from the collected face image and extracts voice features from the collected person's voice; encodes the facial features through a preset image encoder to obtain encoded facial features; encodes the voice features through a preset voice encoder to obtain encoded voice features; inputs the encoded facial features and the encoded voice features into a diffusion model for three-dimensional face synthesis to obtain a three-dimensional face mesh sequence, and the diffusion model is used to predict face motion based on the encoded facial features and the encoded voice features. Compared with the traditional three-dimensional face synthesis method, since the above method of this embodiment encodes the facial features and voice features through an encoder, and predicts face motion based on the obtained encoded facial features and encoded voice features through a diffusion model, three-dimensional face synthesis is achieved and a three-dimensional face mesh sequence matching the face image and the person's voice is obtained, thereby avoiding the complex data processing process and high equipment cost in the prior art, and thus being able to quickly and accurately synthesize three-dimensional faces and having strong applicability.

[0069] refer to Figure 4 , Figure 4 FIG. 4 is a flow chart of a second embodiment of a three-dimensional face synthesis method according to the present invention.

[0070] Based on the above first embodiment, in this embodiment, in order to extract more detailed facial features, the step S10 may include:

[0071] Step S101: performing three-dimensional face modeling on the collected face image to obtain a three-dimensional face template, wherein the collected face image is a single-view face image.

[0072] It should be noted that the above-mentioned 3D face template can be a 3D mesh sequence of a specific person with a neutral expression, and the 3D face template is represented in the form of a FLAME model. Among them, the FLAME model can be based on LBS (Linear BlendSkining) and combined with blendshapes as a representation. The FLAME model can contain 5023 vertices and 4 joints. Specifically, LBS describes how the vertices at the connection should change when relative rotation occurs between parts; the input of Blendshape is the value of each attribute, and the output is how the corresponding face model is deformed, specifically, the offset of 5023 vertices.

[0073] Step S102: determining the morphological information and texture information of the face from the three-dimensional face template, and reconstructing the face of the three-dimensional face template based on the morphological information and the texture information to obtain a reconstructed three-dimensional face template.

[0074] It is understandable that the above-mentioned morphological information may refer to the three-dimensional geometric structure of the face, that is, the shape, size and positional relationship of each part of the face. This includes the three-dimensional morphology of key parts such as the forehead, eyes, nose, mouth, chin, and their relative positions and spatial relationships. The above-mentioned texture information may refer to visual effects such as color, texture, and light and shadow on the surface of the face. Texture information can be added to the three-dimensional model by mapping or material mapping to make it look more realistic. For example, details such as the gloss, pores, wrinkles, and other details of the skin, as well as the color and texture of parts such as the eyes and hair, can all be presented through texture information.

[0075] Step S103: extracting facial features from the reconstructed three-dimensional face template.

[0076] Furthermore, in this embodiment, in order to extract more detailed speech features, the step S10 may further include:

[0077] Step S104: performing short-time Fourier transform on the collected character voice to obtain a short-term voice frame corresponding to the collected character voice.

[0078] In a specific implementation, the entire time domain process corresponding to the above-mentioned character voice can be decomposed into countless small processes of equal length, that is, "windowing" or "framing". Each small process (or frame) is approximately stable, so that Fourier transform can be applied to each small process. The length of the frame is usually selected according to the characteristics of the speech signal, and should generally be less than the period of the shortest period signal in the speech signal. At the same time, in order to maintain the continuity of the speech signal, there is usually a certain overlap between adjacent frames. Then, each frame of the speech signal can be subjected to a short-time Fourier transform to obtain the above-mentioned short-term speech frame.

[0079] Step S105: converting the time domain signal corresponding to the short-term speech frame into a frequency domain signal through Fourier transform, and extracting speech features according to the energy distribution of the frequency domain signal in the frequency domain.

[0080] Based on the first embodiment, in this embodiment, in order to avoid redundant elements in the original face image and the original character voice from bringing unnecessary workload to the three-dimensional face synthesis method of this embodiment, before step S10, the following may also be included:

[0081] Step S1: performing image preprocessing on the original face image, and determining the preprocessed original face image as the collected face image, the image preprocessing includes resolution adjustment, image cropping, image alignment and image denoising.

[0082] Step S2: perform speech preprocessing on the original character speech, and determine the preprocessed original character speech as the collected character speech, the speech preprocessing includes endpoint detection, speech pre-emphasis, speech framing, speech normalization and speech denoising.

[0083] In this embodiment, three-dimensional face modeling is performed on the collected face image to obtain a three-dimensional face template, wherein the collected face image is a single-view face image; the morphological information and texture information of the face are determined from the three-dimensional face template, and the face is reconstructed on the three-dimensional face template based on the morphological information and the texture information to obtain a reconstructed three-dimensional face template; facial features are extracted from the reconstructed three-dimensional face template; short-time Fourier transform is performed on the collected character voice to obtain a short-term voice frame corresponding to the collected character voice; and a short-time Fourier transform is performed on the collected character voice to obtain a short-term voice frame corresponding to the collected character voice. The time domain signal corresponding to the short-term speech frame is converted into a frequency domain signal, and the speech feature is extracted according to the energy distribution of the frequency domain signal in the frequency domain; the original face image is preprocessed, and the preprocessed original face image is determined as the collected face image, and the image preprocessing includes resolution adjustment, image cropping, image alignment and image denoising; the original character voice is preprocessed, and the preprocessed original character voice is determined as the collected character voice, and the voice preprocessing includes endpoint detection, voice pre-emphasis, voice framing, voice normalization and voice denoising. Compared with the traditional three-dimensional face synthesis method, the above method of this embodiment introduces operations such as three-dimensional face modeling, face reconstruction, short-time Fourier transform, Fourier transform, image preprocessing and voice preprocessing, so that more detailed facial features and voice features can be extracted, thereby improving the accuracy of feature extraction.

[0084] refer to Figure 5 , Figure 5 FIG. 4 is a flow chart of a third embodiment of a three-dimensional face synthesis method according to the present invention.

[0085] Based on the above embodiments, in this embodiment, in order to obtain more compact and easier to process facial features, step S20 may include:

[0086] Step S201: Mapping the facial features to a latent feature space through a preset image encoder, wherein the preset image encoder is a Transformer-VAE encoder, and the Transformer-VAE encoder is constructed based on a Transformer structure and a variational autoencoder.

[0087] It should be noted that the above-mentioned latent feature space refers to the low-dimensional representation space of data obtained through learning in machine learning and deep learning. This low-dimensional representation usually captures the key features in the data and can be used to represent the potential structure and changes of the data. The dimension of the latent feature space is low, and it is more capable of expressing the essence of the data than the high dimension of the original data. Such a representation helps to extract important information from the data, reduce redundancy, and retain key change patterns.

[0088] It should be understood that the above Transformer structure is a deep learning model based on the self-attention mechanism, which is mainly used for natural language processing tasks such as machine translation, text classification, question-answering systems, etc. Its structure mainly consists of two parts: an encoder and a decoder. Each part is composed of multiple identical layers stacked together, and each layer contains a multi-head self-attention mechanism and a feedforward neural network. The above variational autoencoder (VAE) is a generative model that combines the ideas of autoencoder and probabilistic inference. It achieves data generation and encoding by learning the potential distribution of data. VAE also consists of an encoder and a decoder. The encoder maps the input data to a potential representation in the latent space, while the decoder maps the potential representation back to the reconstructed data space.

[0089] Step S202: Encoding the facial features in the latent feature space to obtain encoded facial features, wherein the encoding process includes a compression process, an abstraction process and a decoupling process of the facial features.

[0090] Based on the above embodiments, in this embodiment, the preset speech encoder includes a speech pre-training model and a CBE module. In order to obtain a more compact and easier to process speech feature, the step S30 may include:

[0091] Step S301: converting the speech features into a plurality of speech sequences through the speech pre-training model, and capturing dependencies between the plurality of speech sequences, wherein the speech pre-training model is constructed based on a recurrent neural network and / or a convolutional neural network.

[0092] Step S302: encoding the plurality of speech sequences based on the dependency relationship in the CBE module to obtain encoded speech features.

[0093] In a specific implementation, the core of the above CBE (Content-Based Encoding) module is to design a coding strategy that can convert the extracted features into a representation suitable for subsequent processing. The CBE module can use a predefined dictionary or codebook to map feature vectors to entries in the codebook, which can be achieved through technologies such as K-means clustering and vector quantization. In addition, in order to further improve the efficiency and robustness of the coding, the CBE module can use sparse coding technology so that each speech sequence is associated with only a few entries in the dictionary.

[0094] Based on the above embodiments, in this embodiment, in order to make the encoded facial features and the encoded voice features more consistent, the step S40 may include:

[0095] Step S401: input the encoded facial features and the encoded speech features into a diffusion model for decoding to obtain a predicted face motion trajectory.

[0096] Step S402: performing three-dimensional face synthesis on the encoded facial features and the encoded voice features based on the predicted face motion trajectory, and outputting a three-dimensional face mesh sequence after the encoded facial features and the encoded voice features achieve lip synchronization.

[0097] In the specific implementation, you can refer to Figure 6 , Figure 6 The following is a schematic diagram of the user interface design of the three-dimensional face synthesis method of the present invention. Figure 6 As shown, assuming that in a certain 3D face synthesis event, the tester can click the "Voice Selection" button to randomly input a voice, click the "Template Selection" button to select a specific face template, and click the "Synthesis" button to perform the 3D face synthesis task; after waiting for the system core algorithm to finish running, a 3D visualization display is performed; finally, the tester can check whether the 3D face mesh shape and timing playback are reasonable, whether the synthesized face movements are consistent with the corresponding voice clips, and whether the 3D face animation is lip-synced.

[0098] In this embodiment, the facial features are mapped to a latent feature space by a preset image encoder, and the preset image encoder is a Transformer-VAE encoder, and the Transformer-VAE encoder is constructed based on a Transformer structure and a variational autoencoder; the facial features are encoded in the latent feature space to obtain encoded facial features, and the encoding process includes a compression process, an abstraction process and a decoupling process of the facial features; the preset speech encoder includes a speech pre-training model and a CBE module; the speech features are converted into a number of speech sequences by the speech pre-training model, and the dependencies between the several speech sequences are captured, and the speech pre-training model is constructed based on a recurrent neural network and / or a convolutional neural network; the several speech sequences are encoded in the CBE module based on the dependencies to obtain encoded speech features; the encoded facial features and the encoded speech features are input into a diffusion model for decoding to obtain a predicted face motion trajectory; the encoded facial features and the encoded speech features are synthesized into a three-dimensional face based on the predicted face motion trajectory, and a three-dimensional face mesh sequence is output after the encoded facial features and the encoded speech features achieve lip synchronization. Compared with the traditional three-dimensional face synthesis method, the above method of this embodiment encodes facial features and speech features based on the Transformer-VAE encoder, the speech pre-training model and the CBE module, thereby obtaining more compact and easier to process encoded facial features and encoded speech features; at the same time, the encoded facial features and the encoded speech features are decoded through the diffusion model, so that the encoded facial features and the encoded speech features are more consistent, thereby achieving more accurate three-dimensional face synthesis.

[0099] In addition, an embodiment of the present invention further provides a storage medium, on which a 3D face synthesis program is stored. When the 3D face synthesis program is executed by a processor, the steps of the 3D face synthesis method described above are implemented.

[0100] Reference Figure 7 , Figure 7 It is a structural block diagram of the first embodiment of the three-dimensional face synthesis device of the present invention.

[0101] like Figure 7 As shown, the 3D face synthesis device proposed in the embodiment of the present invention includes:

[0102] A feature extraction module 701 is used to extract facial features from the collected face image and extract voice features from the collected person's voice;

[0103] A first encoding module 702, configured to encode the facial features by using a preset image encoder to obtain encoded facial features;

[0104] The second encoding module 703 is used to encode the speech feature by using a preset speech encoder to obtain the encoded speech feature;

[0105] The face synthesis module 704 is used to input the encoded facial features and the encoded voice features into a diffusion model to perform three-dimensional face synthesis to obtain a three-dimensional face mesh sequence. The diffusion model is used to predict face movement based on the encoded facial features and the encoded voice features.

[0106] This embodiment extracts facial features from the collected face image and extracts voice features from the collected person's voice; encodes the facial features through a preset image encoder to obtain encoded facial features; encodes the voice features through a preset voice encoder to obtain encoded voice features; inputs the encoded facial features and the encoded voice features into a diffusion model for three-dimensional face synthesis to obtain a three-dimensional face mesh sequence, and the diffusion model is used to predict face motion based on the encoded facial features and the encoded voice features. Compared with the traditional three-dimensional face synthesis method, since the above method of this embodiment encodes the facial features and voice features through an encoder, and predicts face motion based on the obtained encoded facial features and encoded voice features through a diffusion model, three-dimensional face synthesis is achieved and a three-dimensional face mesh sequence matching the face image and the person's voice is obtained, thereby avoiding the complex data processing process and high equipment cost in the prior art, and thus being able to quickly and accurately synthesize three-dimensional faces and having strong applicability.

[0107] Based on the first embodiment of the three-dimensional face synthesis device of the present invention, a second embodiment of the three-dimensional face synthesis device of the present invention is proposed.

[0108] In this embodiment, the feature extraction module 701 is also used to perform three-dimensional face modeling on the collected face image to obtain a three-dimensional face template, where the collected face image is a single-view face image; determine the face morphological information and texture information from the three-dimensional face template, and reconstruct the face of the three-dimensional face template based on the morphological information and the texture information to obtain a reconstructed three-dimensional face template; and extract facial features from the reconstructed three-dimensional face template.

[0109] Furthermore, the feature extraction module 701 is also used to perform short-time Fourier transform on the collected character voice to obtain a short-time speech frame corresponding to the collected character voice; convert the time domain signal corresponding to the short-time speech frame into a frequency domain signal through Fourier transform, and extract the speech feature according to the energy distribution of the frequency domain signal in the frequency domain.

[0110] Furthermore, the feature extraction module 701 is also used to perform image preprocessing on the original facial image, and determine the preprocessed original facial image as the collected facial image, the image preprocessing includes resolution adjustment, image cropping, image alignment and image denoising; perform speech preprocessing on the original character voice, and determine the preprocessed original character voice as the collected character voice, the speech preprocessing includes endpoint detection, speech pre-emphasis, speech framing, speech normalization and speech denoising.

[0111] Furthermore, the first encoding module 702 is also used to map the facial features to a latent feature space through a preset image encoder, wherein the preset image encoder is a Transformer-VAE encoder, and the Transformer-VAE encoder is constructed based on the Transformer structure and a variational autoencoder; the facial features are encoded in the latent feature space to obtain encoded facial features, and the encoding process includes a compression process, an abstraction process, and a decoupling process of the facial features.

[0112] Furthermore, the preset speech encoder includes a speech pre-training model and a CBE module, and the second encoding module 703 is also used to convert the speech features into a number of speech sequences through the speech pre-training model, and capture the dependencies between the several speech sequences, and the speech pre-training model is constructed based on a recurrent neural network and / or a convolutional neural network; in the CBE module, the several speech sequences are encoded based on the dependencies to obtain the encoded speech features.

[0113] Furthermore, the face synthesis module 704 is also used to input the encoded facial features and the encoded speech features into a diffusion model for decoding to obtain a predicted face motion trajectory; perform three-dimensional face synthesis on the encoded facial features and the encoded speech features based on the predicted face motion trajectory, and output a three-dimensional face mesh sequence after the encoded facial features and the encoded speech features achieve lip synchronization.

[0114] Other embodiments or specific implementations of the three-dimensional face synthesis device of the present invention can refer to the above-mentioned method embodiments, which will not be repeated here.

[0115] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0116] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0118] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A three-dimensional face synthesis method, characterized in that: The method comprises the following steps: Extract facial features from the collected face images and extract voice features from the collected person voices; Encoding the facial features by using a preset image encoder to obtain encoded facial features; Encoding the speech feature by a preset speech encoder to obtain the encoded speech feature; Inputting the encoded facial features and the encoded speech features into a diffusion model to perform three-dimensional face synthesis to obtain a three-dimensional face mesh sequence, wherein the diffusion model is used to predict face motion according to the encoded facial features and the encoded speech features; The step of extracting facial features from the collected face image includes: Performing three-dimensional face modeling on the collected face image to obtain a three-dimensional face template, wherein the collected face image is a single-view face image, and the three-dimensional face template is represented in the form of a FLAME model, wherein the FLAME model is represented based on a linear blended skin and a blended deformation, and the FLAME model comprises 5023 vertices and 4 joints, wherein the linear blended skin is used to depict the changes when relative rotation occurs between parts in the FLAME model, and the blended deformation is used to depict the offset of 5023 vertices in the FLAME model; Determining morphological information and texture information of a face from the three-dimensional face template, and reconstructing the face of the three-dimensional face template based on the morphological information and the texture information to obtain a reconstructed three-dimensional face template; Extracting facial features from the reconstructed three-dimensional face template; The step of extracting voice features from the collected voice of the person includes: Performing short-time Fourier transform on the collected character voice to obtain a short-time speech frame corresponding to the collected character voice; Convert the time domain signal corresponding to the short-term speech frame into a frequency domain signal by Fourier transform, and extract speech features according to the energy distribution of the frequency domain signal in the frequency domain; The step of encoding the facial features by a preset image encoder to obtain the encoded facial features comprises: Mapping the facial features to a latent feature space by a preset image encoder, wherein the preset image encoder is a Transformer-VAE encoder, and the Transformer-VAE encoder is constructed based on a Transformer structure and a variational autoencoder; Encoding the facial features in the latent feature space to obtain encoded facial features, the encoding process including a compression process, an abstraction process and a decoupling process of the facial features; The step of encoding the speech feature by a preset speech encoder to obtain the encoded speech feature comprises: The preset speech encoder includes a speech pre-training model and a CBE module; The speech features are converted into a plurality of speech sequences by means of the speech pre-training model, and dependencies between the plurality of speech sequences are captured, wherein the speech pre-training model is constructed based on a recurrent neural network and / or a convolutional neural network; Encoding the plurality of speech sequences based on the dependency relationship in the CBE module to obtain encoded speech features; The step of inputting the encoded facial features and the encoded speech features into a diffusion model to perform three-dimensional face synthesis to obtain a three-dimensional face mesh sequence comprises: Inputting the encoded facial features and the encoded speech features into a diffusion model for decoding to obtain a predicted motion trajectory of the face; The encoded facial features and the encoded voice features are subjected to three-dimensional face synthesis based on the predicted face motion trajectory, and a three-dimensional face mesh sequence is output after the encoded facial features and the encoded voice features achieve lip synchronization.

2. The three-dimensional face synthesis method according to claim 1, characterized in that: Before the steps of extracting facial features from the collected face image and extracting voice features from the collected character voice, the method further includes: Performing image preprocessing on the original face image, and determining the preprocessed original face image as the collected face image, the image preprocessing including resolution adjustment, image cropping, image alignment and image denoising; The original human voice is subjected to speech preprocessing, and the preprocessed original human voice is determined as the collected human voice, wherein the speech preprocessing includes endpoint detection, speech pre-emphasis, speech framing, speech normalization and speech denoising.

3. A three-dimensional face synthesis device, characterized in that: The three-dimensional face synthesis device comprises: A feature extraction module is used to extract facial features from the collected face images and extract voice features from the collected voices of the characters; A first encoding module, used for encoding the facial features by using a preset image encoder to obtain encoded facial features; A second encoding module, used for encoding the speech feature by a preset speech encoder to obtain the encoded speech feature; A face synthesis module, used for inputting the encoded facial features and the encoded speech features into a diffusion model to perform three-dimensional face synthesis to obtain a three-dimensional face mesh sequence, wherein the diffusion model is used for predicting face motion according to the encoded facial features and the encoded speech features; The step of extracting facial features from the collected face image includes: Performing three-dimensional face modeling on the collected face image to obtain a three-dimensional face template, wherein the collected face image is a single-view face image, and the three-dimensional face template is represented in the form of a FLAME model, wherein the FLAME model is represented based on a linear blended skin and a blended deformation, and the FLAME model comprises 5023 vertices and 4 joints, wherein the linear blended skin is used to depict the changes when relative rotation occurs between parts in the FLAME model, and the blended deformation is used to depict the offset of 5023 vertices in the FLAME model; Determining morphological information and texture information of a face from the three-dimensional face template, and reconstructing the face of the three-dimensional face template based on the morphological information and the texture information to obtain a reconstructed three-dimensional face template; Extracting facial features from the reconstructed three-dimensional face template; The step of extracting voice features from the collected voice of the person includes: Performing short-time Fourier transform on the collected character voice to obtain a short-time speech frame corresponding to the collected character voice; Convert the time domain signal corresponding to the short-term speech frame into a frequency domain signal by Fourier transform, and extract speech features according to the energy distribution of the frequency domain signal in the frequency domain; The step of encoding the facial features by a preset image encoder to obtain the encoded facial features comprises: Mapping the facial features to a latent feature space by a preset image encoder, wherein the preset image encoder is a Transformer-VAE encoder, and the Transformer-VAE encoder is constructed based on a Transformer structure and a variational autoencoder; Encoding the facial features in the latent feature space to obtain encoded facial features, the encoding process including a compression process, an abstraction process and a decoupling process of the facial features; The step of encoding the speech feature by a preset speech encoder to obtain the encoded speech feature comprises: The preset speech encoder includes a speech pre-training model and a CBE module; The speech features are converted into a plurality of speech sequences by means of the speech pre-training model, and dependencies between the plurality of speech sequences are captured, wherein the speech pre-training model is constructed based on a recurrent neural network and / or a convolutional neural network; Encoding the plurality of speech sequences based on the dependency relationship in the CBE module to obtain encoded speech features; The step of inputting the encoded facial features and the encoded speech features into a diffusion model to perform three-dimensional face synthesis to obtain a three-dimensional face mesh sequence comprises: Inputting the encoded facial features and the encoded speech features into a diffusion model for decoding to obtain a predicted motion trajectory of the face; The encoded facial features and the encoded voice features are subjected to three-dimensional face synthesis based on the predicted face motion trajectory, and a three-dimensional face mesh sequence is output after the encoded facial features and the encoded voice features achieve lip synchronization.

4. A three-dimensional face synthesis device, characterized in that: The device comprises: a memory, a processor, and a 3D face synthesis program stored in the memory and executable on the processor, wherein the 3D face synthesis program is configured to implement the steps of the 3D face synthesis method according to any one of claims 1 to 2.

5. A storage medium, characterized in that: The storage medium stores a three-dimensional face synthesis program, which, when executed by the processor, implements the steps of the three-dimensional face synthesis method according to any one of claims 1 to 2.

Images