Speech video synthesis method and system

Abstract

The invention provides a speech video synthesis method and system, and the method comprises the steps: inputting speech audio into a pre-trained audio-motion encoder, extracting a speech feature sequence from the speech audio, and generating a motion sequence of a face according to the speech feature sequence; inputting a single two-dimensional face picture shot from a first visual angle into a pre-trained picture encoder, and extracting three-dimensional identity features of an object contained in the face picture; inputting the three-dimensional identity features and the emotion labels into a pre-trained emotion mapping layer, and fusing to obtain emotion-containing three-dimensional identity features; the action sequence and the emotion-containing three-dimensional identity features are input into a video generation network based on a neural radiation field, a speech video of the object matched with the speech audio at a required second view angle is synthesized through the neural radiation field and camera parameters, and the second view angle can be adjusted within a preset range through the camera parameters.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics technology, specifically to the field of digital human video synthesis technology based on generative artificial intelligence, and more specifically to a speech video synthesis method and system. Background Art

[0002] With the development of digital economy and virtual reality technology, studying digital human video synthesis technology is an important research direction, and the research on three-dimensional digital human synthesis and driving is becoming increasingly important.

[0003] Previous work primarily used 2D data for training and directly synthesized the input image or video. Due to the lack of 3D data and the complexity of modeling and training, it is difficult to directly extend existing 2D methods to the 3D domain. 3D digital human speech video synthesis methods, both domestically and internationally, primarily focus on synthesizing videos of specific faces. Specifically, they train a speech synthesis network for the target person using large amounts of single-person speech video data.

[0004] Existing common audio-driven digital face recognition methods can be categorized by the representation dimension as either 2D image-based or 3D representation-based (primarily based on neural radiation fields). 2D image-based audio-driven solutions lack 3D information, leading to image blur and facial distortion when viewing angles change or when the face moves.

[0005] Existing method 1 (SadTalker) builds on face image translation technology. It uses two variational autoencoders (VAEs) to synthesize facial expression parameters and head pose parameters from audio. This parameter sequence serves as a control condition for the image translation network, achieving audio-driven face video synthesis. However, due to the lack of a geometric foundation provided by a three-dimensional representation, this method suffers from image blurring and other issues when the head pose or viewpoint changes significantly.

[0006] Existing method 2 (StyleTalk) also builds on image translation technology, but introduces a style reference video input and a style encoding network. This network generates a style code, which is used to globally control facial expression synthesis, achieving video synthesis of a specific style. While this approach improves the synthesis effect, it suffers from a similar problem as existing method 1: it cannot synthesize high-quality videos from new perspectives.

[0007] Existing method 3 (Real3D-Portrait) uses a three-plane neural radiance field as a facial representation. It uses a two-dimensional convolutional network and a visual Transformer network to construct a monocular 3D face reconstruction model, synthesizing the input image into a three-plane representation. Audio is also used to synthesize a three-plane deformation field to achieve deformation of the 3D representation. However, due to the lack of emotional control, this method cannot achieve personalized and diverse video synthesis.

[0008] When the inventors were conducting research on reconstructing audio-driven digital faces from a single image, they found that existing technologies generally use technologies that are directly implemented through affine transformation and image deformation in the two-dimensional image domain. These technologies have problems such as the inability to synthesize new perspectives, image blurring due to changes in head posture, and facial deformation.

[0009] It should be noted that this background information is intended solely to introduce relevant information related to the present invention to facilitate understanding of the present invention's technical solution. It does not necessarily constitute prior art. Relevant information submitted and disclosed together with the present invention's solution should not be considered prior art unless there is evidence that the relevant information was disclosed prior to the filing date of the present invention. Summary of the Invention

[0010] Therefore, the purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide a speech video synthesis method and system.

[0011] The purpose of the present invention is achieved through the following technical solutions:

[0012] According to a first aspect of the present invention, a method for synthesizing a speech video is provided, comprising: inputting speech audio into a pre-trained audio-action encoder to extract a speech feature sequence from the speech audio and generate a facial action sequence based on the speech feature sequence; inputting a single two-dimensional face image taken from a first perspective into a pre-trained image encoder to extract three-dimensional identity features of an object contained in the face image; inputting the three-dimensional identity features and emotion labels into a pre-trained emotion mapping layer to fuse them to obtain three-dimensional identity features containing emotions; inputting the action sequence and the three-dimensional identity features containing emotions into a video generation network based on a neural radiation field, and synthesizing a speech video of the object at a desired second perspective that matches the speech audio through the neural radiation field and camera parameters, wherein the second perspective can be adjusted within a predetermined range through the camera parameters. This solution can at least achieve the following beneficial technical effects: the solution decouples identity features and facial movements and can generate three-dimensional identity features, and then independently couples emotional information on the basis of the three-dimensional identity features, thereby controlling the synthesis of speech videos in a more fine-grained manner and improving the synthesis quality and controllability of speech videos; in addition, the solution only requires a single first-perspective two-dimensional face image combined with speech audio to drive the generation of the required second-perspective speech video. Compared with traditional multi-perspective image or video-driven solutions, it effectively reduces the implementation difficulty of synthesized videos and improves user experience.

[0013] Optionally, the pre-trained audio-action encoder is trained in the following manner: obtain an audio-action encoder and a first auxiliary training model, wherein the audio-action encoder includes an audio encoder and an action encoder, and the first auxiliary training model includes a lip reading recognition module and a pre-trained speech recognition model; input the training audio extracted from the training video into the audio encoder to extract a training speech feature sequence, wherein the training video includes facial action frames and audio files of a predetermined person speaking; input the training speech feature sequence into the speech recognition module, extract a potential feature sequence from the training speech feature sequence, and encode the potential feature sequence into a first morpheme sequence; input the training speech feature sequence into the action encoder, and obtain The method comprises the following steps: a) reconstructing a facial action sequence corresponding to the training audio by using a lip reading recognition module, extracting a lip movement geometry sequence from the reconstructed facial sequence, and determining a second morpheme sequence based on the lip movement feature sequence extracted from the lip movement geometry sequence; and, while freezing the parameters of the audio encoder, speech recognition model, and video generation network, updating the parameters of the action encoder and lip reading recognition module based on a first reconstruction loss between the facial action sequence corresponding to the training audio and a 3DMM parameter sequence detected from the training video, a second reconstruction loss between the latent feature sequence and the lip movement feature sequence, and a connection temporal classification loss between the first morpheme sequence and the second morpheme sequence, as determined by a first total loss function. This solution can achieve at least the following beneficial technical effects: in addition to the audio-action encoder, the solution provides a first auxiliary training model so that some pre-trained models that have been stably trained can be used to assist in the supervised training of the audio-action encoder based on the training audio and the first total loss function. Furthermore, during the supervised training process, since the lip reading recognition module can convert the action sequence into a second morpheme sequence corresponding to the first morpheme sequence encoded by the speech recognition module to complete the supervised training, the collection of a large amount of supervised training data can be avoided.

[0014] Optionally, the action encoder includes a cascade of multiple Transformer encoders and an action mapping layer, wherein each Transformer encoder includes: a periodic position encoding layer, which extracts audio codes with position information from input features, wherein the input features of the first Transformer encoder are the output of the audio encoder; a multi-head attention layer, which processes the audio codes input by the previous layer to incorporate contextual information to obtain a feature sequence; a feedforward layer, which processes each feature in the feature sequence input by the previous layer to obtain action hidden features, and the feedforward layer of the last Transformer encoder obtains the final action hidden feature sequence; and an action mapping layer, which is used to map the final action hidden feature sequence to an action sequence. This solution can at least achieve the following beneficial technical effects: the action encoder can extract more matching facial action features based on the input audio through the cascade of multiple Transformer encoders, and ultimately obtain an action sequence for driving the speech video, so as to improve the accuracy of facial actions.

[0015] Optionally, the pre-trained image encoder is trained in the following manner: obtaining an image encoder, which includes a feature pyramid encoding layer for extracting multi-scale features from an input image and an identity mapping layer for extracting three-dimensional identity features from the multi-scale features; obtaining image data of multiple characters from different perspectives and different facial expressions, selecting any two images of the same character in the image data as the original image and the driving image to form a sample pair, thereby forming multiple sample pairs; using multiple sample pairs to train the image encoder, wherein the source image is input into the image encoder to extract the three-dimensional identity features, 3DMM parameters are extracted from the driving image using a preset method, and volume rendering is performed based on the three-dimensional identity features and the motion features and camera parameters in the extracted 3DMM parameters to obtain a predicted driving image; and updating the parameters of the image encoder according to the difference between the driving image in the sample pair and the predicted driving image. This solution can at least achieve the following beneficial technical effects: this solution can cleverly use sample pairs consisting of pictures of the same person from different perspectives and expressions to iteratively train the picture encoder to separate emotional information to extract three-dimensional identity information, use an picture encoder to extract three-dimensional identity features from the source image, use a preset image to extract 3DMM parameters from the driving image, and regenerate the predicted driving image based on the three-dimensional identity features and 3DMM parameters, so that the picture encoder can extract higher-quality three-dimensional identity features that do not contain emotional information.

[0016] Optionally, the pre-trained emotion mapping layer is trained in the following manner: the emotion labels and the three-dimensional identity features extracted from the image frames of the training video using the pre-trained picture encoder are input into the emotion mapping layer to extract the emotion-containing three-dimensional identity features corresponding to the training video; the emotion-containing three-dimensional identity features corresponding to the training video are input into the video generation network to obtain a predicted speech video; the graphic emotion features corresponding to the video frames are extracted from the predicted speech video, and the text emotion features are extracted from the emotion descriptions corresponding to the emotion labels; the parameters of the emotion mapping layer are updated according to the cosine similarity between the graphic emotion features and the text emotion features. This solution can at least achieve the following beneficial technical effects: in this solution, the emotion mapping layer is trained after the picture encoder is trained. After the pre-trained picture encoder has stably extracted the three-dimensional identity features, the emotion mapping layer is trained separately to reduce the interference caused by identity feature learning during joint training, so that the emotion mapping layer can better learn the subtle changes between different emotion labels, thereby improving the performance of the emotion mapping layer.

[0017] Optionally, the pre-trained emotion mapping layer is trained in the following manner: non-zero emotion labels and three-dimensional identity features extracted from image frames of training videos using a pre-trained picture encoder are input into the emotion mapping layer to extract the emotion-containing three-dimensional identity features corresponding to the training video, wherein the non-zero emotional expression represents a certain emotion type other than calm emotion; all-zero emotion labels and three-dimensional identity features extracted from image frames of training videos using a pre-trained picture encoder are input into the emotion mapping layer to extract the three-dimensional identity features of emotional calmness corresponding to the training video, wherein all-zero emotional expressions represent a calm emotion type; the emotion-containing three-dimensional identity features and the emotional calmness three-dimensional identity features corresponding to the training video are respectively input into the video generation network to obtain the predicted first speech video and the second speech video; the graphic emotion features corresponding to the video frames are extracted from the predicted speech video, and the text emotion features are extracted from the emotion description corresponding to the emotion label; the parameters of the emotion mapping layer are updated using a second total loss function, the second total loss function includes a loss function for calculating the consistency of identity features between the first speech video and the second speech video, a loss function for calculating the consistency of color distribution, and an alignment loss function for calculating the cosine similarity between the graphic emotion features and the text emotion features. This scheme can at least achieve the following beneficial technical effects: in addition to the alignment loss of emotional features, this scheme additionally sets the loss of identity feature consistency and color distribution consistency between speech videos with calm emotions and speech videos with non-calm emotions. Since the standard facial features under calm emotions have a stable reference benchmark for identity features and color distribution, the identity information deviation and color distribution deviation after emotion mapping can be reduced based on the loss of identity feature consistency and color distribution consistency between the standard facial features under calm emotions and the facial features under non-calm emotions, thereby further ensuring the performance of the emotion mapping layer.

[0018] According to a second aspect of the present invention, there is provided a speech video synthesis system for implementing the method as described in the first aspect, comprising: a pre-trained audio-action encoder for extracting a speech feature sequence from an input speech audio and generating a facial action sequence based on the speech feature sequence; a pre-trained image encoder for acquiring a single two-dimensional face image taken from a first perspective, and extracting three-dimensional identity features of an object contained in the face image; a pre-trained emotion mapping layer for fusing three-dimensional identity features with emotion labels to obtain three-dimensional identity features containing emotions; a video generation network based on a neural radiation field for acquiring action sequences and three-dimensional identity features containing emotions, and synthesizing a speech video of the object at a desired second perspective that matches the speech audio through the neural radiation field and camera parameters, wherein the second perspective can be adjusted within a predetermined range through the camera parameters.

[0019] According to a third aspect of the present invention, an electronic device is provided, characterized in that it includes: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the method described in the first aspect by executing the executable instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0021] Figure 1 2. A schematic diagram of modules of a speech video synthesis system according to an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a supervised training architecture of an audio-motion encoder according to an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of training a picture encoder according to an embodiment of the present invention;

[0024] Figure 4 2. A schematic diagram of training an emotion mapping layer according to an embodiment of the present invention;

[0025] Figure 5 Schematic diagram showing a comparison between the method according to an embodiment of the present invention and an existing method based on three-dimensional representation;

[0026] Figure 6 2 is a schematic diagram comparing the effects of the method according to an embodiment of the present invention and the existing method based on image deformation. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] As mentioned in the background section, while researching audio-driven digital face reconstruction from a single image, the inventors discovered that existing techniques generally employ affine transformations and image warping directly in the 2D image domain. These techniques suffer from issues such as the inability to synthesize new perspectives, image blurring, and facial distortion caused by changes in head pose. To address this, the inventors developed an audio-driven 3D face speech video synthesis method based on neural radiation fields, aiming to address the inability of existing methods, which use 2D image warping to maintain video quality and geometric stability during perspective changes and head movement. Through research, the inventors concluded that existing techniques for reconstructing audio-driven digital faces from a single 2D image either employ affine transformations in the 2D image domain to distort the image, or extract only 2D identity features to synthesize the video. This results in image blurring and facial distortion caused by a lack of 3D identity information when the perspective changes. Therefore, the present invention employs a 3D face synthesis method based on neural radiation fields, reconstructing a 3D face using 3D identity features extracted from a single image and using audio to drive the synthesis of speech videos. The solution consists of four main parts: First, a three-dimensional face identity encoding is reconstructed from an input image at a specific perspective (referred to as the first perspective for distinction). This generates 3D identity features, fundamentally resolving geometric inconsistencies in situations such as head rotation. Then, an emotion mapping layer is used to optimize the identity encoding, incorporating emotion-specific textures and geometry. Simultaneously, the input audio is encoded to generate motion sequences corresponding to facial motion features. Finally, the motion sequences and a neural radiance field-based video generation network are used to drive the 3D face to synthesize a speech video of the subject from a desired perspective (referred to as the second perspective for distinction) that matches the speech audio. This approach, based on a single face image and speech audio, reduces blur and facial distortion in the synthesized video, resulting in higher-quality speech videos.

[0029] According to one embodiment of the present invention, see Figure 1 , provides a speech video synthesis system, including: a pre-trained audio-action encoder; a picture encoder ; Video generation network, which includes a dynamic generator And renderer. Among them, the pre-trained audio-action encoder is used to extract the speech feature sequence from the input speech audio and generate the facial action sequence according to the speech feature sequence; the pre-trained image encoder , used to obtain a single two-dimensional face image taken from the first person perspective and extract the three-dimensional identity features of the object contained in the face image; pre-trained emotion mapping layer , for fusing 3D identity features and emotion labels to obtain 3D identity features containing emotion; and a video generation network based on neural radiance fields, for obtaining action sequences and 3D identity features containing emotion, and synthesizing a desired second-perspective speech video of the subject that matches the speech audio using the neural radiance fields and camera parameters, wherein the second perspective can be adjusted within a predetermined range using the camera parameters. To facilitate the decoupling of action sequences, 3D identity features, and emotion features, and to better achieve higher-quality speech video generation from monocular 2D images, this embodiment does not employ an end-to-end training approach to train the system. Instead, each model in the system is trained separately, primarily constructing appropriate supervision tasks for the audio-action encoder, image encoder, and emotion mapping layer. This embodiment builds on the dynamic three-plane generator to construct a decoupled dual-encoder structure, which is equivalent to expanding two parallel encoding branches (i.e., one branch includes the audio-action encoder, and the other branch includes the image encoder and emotion mapping layer). This decouples 3D identity features and action information in the generator, thereby better completing monocular reconstruction and audio-driven tasks.

[0030] The audio-action encoder, image encoder, emotion mapping layer, and video generation network are schematically explained below.

[0031] 1. Audio-Motion Encoder

[0032] According to one embodiment of the present invention, see again Figure 1 , audio-motion encoder, which includes an audio encoder and action encoder Among them, the audio encoder Use a pre-trained Wav2Vec model, for example, Wav2Vec 1.0 or Wav2Vec 2.0, preferably Wav2Vec 2.0. Of course, in principle, implementers can also use other models similar to Wav2Vec 2.0, such as the HuBERT model or the WavLM model. Action Encoder It includes N cascaded Transformer encoders, where each Transformer encoder includes: a periodic position encoding layer, which extracts audio codes with position information from input features, where the input features of the first Transformer encoder are the output of the audio encoder; a multi-head attention layer, which processes the audio codes input by the previous layer to incorporate context information to obtain a feature sequence; a feedforward layer, which processes each feature in the feature sequence input by the previous layer to obtain action hidden features, and the feedforward layer of the last Transformer encoder obtains the final action hidden feature sequence; an action mapping layer, which is used to map the final action hidden feature sequence to an action sequence.

[0033] Schematically, the first branch of the dual encoder includes an audio-action encoder. The schematic supervised training architecture of the audio-action encoder is as follows Figure 2 As shown in Figure 1. The audio-action encoder consists of a network body of an action encoder composed of a multi-layer Transformer encoder and an audio encoder implemented using the audio pre-trained network Wav2Vec 2.0. This branch first uses the audio pre-trained network Wav2Vec 2.0 as the audio encoder during the processing. The audio input signal is preliminarily processed to extract a speech feature sequence containing contextual information. This example then constructs an action encoder based on a multi-layer Transformer encoder to further integrate contextual information and convert audio features into action sequences. Schematically, in each layer of the Transformer encoder, the input feature sequence first undergoes periodic positional encoding to add positional information relative to the context, resulting in an audio code with positional information. The audio code with positional information is then integrated into the contextual information through a multi-head attention layer to obtain a contextual feature sequence. Finally, the contextual feature sequence is processed through a feedforward layer to process each feature in the sequence, resulting in action hidden features. All action hidden features form an action hidden feature sequence. After multiple encoding processes, the action hidden feature sequence obtains the final action hidden feature sequence. The final action hidden feature sequence is then processed by the action mapping layer and mapped into the action feature space to obtain an action sequence, where each element in the action sequence is an action feature for one frame.

[0034] According to one embodiment of the present invention, the audio-action encoder is trained in the following manner: an audio-action encoder and a first auxiliary training model are obtained, wherein the audio-action encoder includes an audio encoder and an action encoder, and the first auxiliary training model includes a lip reading recognition module and a pre-trained speech recognition model, wherein the audio encoder is a pre-trained Wav2Vec model; the training audio extracted from the training video is input into the audio encoder to extract a training speech feature sequence, wherein the training video includes facial action frames and audio files of a predetermined person speaking; the training speech feature sequence is input into the speech recognition module, a potential feature sequence is extracted from the training speech feature sequence, and the potential feature sequence is encoded into a first morpheme sequence; the training speech feature sequence is input into the action encoder to obtain a facial action sequence corresponding to the training audio; the lip reading recognition module reconstructs the face according to the facial parameters, extracts a lip movement geometry sequence from the reconstructed face sequence, and determines a second morpheme sequence based on the lip movement feature sequence extracted from the lip movement geometry sequence; when the parameters of the audio encoder, the speech recognition model and the video generation network are frozen, the first reconstruction loss ( Figure 2 (not shown), the second reconstruction loss between the latent feature sequence and the lip movement feature sequence and the connection temporal classification loss between the first morpheme sequence and the second morpheme sequence To update the parameters of the motion encoder and lip recognition module.

[0035] Schematically, the audio-motion encoder uses a multimodal alignment training approach to achieve audio-synchronized motion sequence synthesis. This involves mutually supervised training through the correspondence between audio, text, and lip movement geometry sequences. Schematically, based on the existing audio encoder and motion encoder, this example introduces a pre-trained audio encoder and constructs a Transformer-based lip reading recognition module with the same structure as the supervisory network. The audio encoder can be fine-tuned on an audio recognition dataset based on a pre-trained Wav2Vec2.0 model. An additional Wav2Vec mapping layer converts the speech feature sequence into a latent feature sequence, which is then encoded into a first morpheme sequence using a morpheme encoder. The lip reading recognition module uses the reconstructed face sequence geometry as input, extracts lip geometry through segmentation, and obtains a lip movement geometry sequence. The lip movement mapping network maps the lip movement geometry sequence into a feature space with the same dimensionality as the audio features, resulting in a lip movement feature sequence. The lip reading recognition network (a multi-layer Transformer network) then further maps this feature sequence into a morpheme space, resulting in a second morpheme sequence (a probability distribution of the morpheme sequence). Morphemes are the basic units of speech recognition. The morpheme space refers to the vector space composed of morpheme features constructed by the Wav2Vec2.0 pre-trained speech recognition network. This method predicts the probability of occurrence of different morphemes at each position and samples the final morpheme sequence based on this probability. The probability of occurrence of each morpheme at each position is the probability distribution sequence mentioned in the article. Based on this, this example uses the reconstruction loss function and the first total loss function constructed by the Connectionist Temporal Classification loss (CTC loss) for supervised training. This first total loss function achieves multimodal alignment and completes action sequence synthesis synchronized with the input audio.

[0036] 2. Image Encoder

[0037] According to one embodiment of the present invention, another branch of the dual encoder includes a picture encoder, which includes a feature pyramid encoding layer for extracting multi-scale features from an input image and an identity mapping layer for extracting three-dimensional identity features from the multi-scale features. Schematically, the picture encoder is based on the feature pyramid extractor, and the structure of the picture encoder is as follows: Figure 1As shown in the lower left corner, it includes a feature pyramid encoder and an identity mapping layer. The input to the image encoder is a face image that has been cropped and aligned according to the FFHQ dataset preprocessing method. This image is downsampled multiple times and then fed into the feature pyramid encoder. Facial feature vectors are extracted from different resolutions and concatenated. The concatenated vectors are then mapped to a preset identity feature space by the identity mapping layer, resulting in a three-dimensional identity feature.

[0038] According to one embodiment of the present invention, a pre-trained image encoder is trained in the following manner: obtaining an image encoder, which includes a feature pyramid encoding layer for extracting multi-scale features from an input image and an identity mapping layer for extracting three-dimensional identity features from the multi-scale features; obtaining image data of multiple characters from different perspectives and different facial expressions, selecting any two images of the same character in the image data as the original image and the driving image to form a sample pair, thereby forming multiple sample pairs; using the multiple sample pairs to train the image encoder, wherein the source image is input into the image encoder to extract the three-dimensional identity features, 3DMM parameters are extracted from the driving image using a preset method, and volume rendering is performed based on the three-dimensional identity features and the motion features and camera parameters in the extracted 3DMM parameters to obtain a predicted driving image; and updating the parameters of the image encoder according to the difference between the driving image in the sample pair and the predicted driving image.

[0039] Schematically, the image encoder uses cross-contrast learning to build single-person multi-view and multi-action image data using a pre-trained dynamic generator, providing a good data foundation for decoupling action sequences from 3D identity features. For example, see Figure 3 , using a dynamic generator and a renderer (not shown) based on the three-dimensional identity features and different action features (such as action features or ) and camera parameters (such as camera parameters or ) Generate multiple image data of the same person from different perspectives, different expressions and actions, and select any two images from them as source images during training and driver images , construct a sample pair; the source image is used as the input of the image encoder, the encoding extracts the three-dimensional identity features of the source image, and the DECA method is used to extract the action features of the driving image in the sample pair and camera parameters (such as camera perspective, etc.), used to drive the 3D face model constructed by the 3D identity features of the source image, and render the predicted driving image to minimize the difference between the driving image in the sample pair and the predicted driving image As an optimization goal, the image encoder is trained to extract accurate and action-decoupled 3D identity features using monocular image input for subsequent 3D face reconstruction.

[0040] It should be understood that using a pre-trained dynamic generator to construct multiple image data of a single person from different perspectives and with different facial expressions and gestures is relatively convenient, allowing for rapid acquisition of multiple sample pairs for training. However, implementers may also employ other methods to obtain sample pairs, such as capturing images of the same person from different perspectives and with different facial expressions and gestures as required, thereby constructing multiple image data of a single person from different perspectives and with different facial expressions and gestures.

[0041] 3. Emotion Mapping Layer

[0042] See also Figure 4 , emotion mapping layer It is used to label emotions A module that incorporates three-dimensional identity features. The emotion mapping layer can be a linear mapping layer. The space of emotion labels includes, for example, calm, happy, angry, sad, disgusted, surprised, and / or afraid. In this case, a 6-bit one-hot code can be used to represent different emotion types, for example: 000000 represents calm, 000001 represents happy, 000010 represents angry, 000100 represents sad, 001000 represents disgusted, 010000 represents surprised, and 100000 represents afraid. It should be understood that the above is only an example, and other emotion types can be added or other representation methods can be used according to the needs of the implementer.

[0043] According to an optional embodiment of the present invention, a pre-trained emotion mapping layer is trained as follows: emotion labels and three-dimensional identity features extracted from image frames of a training video using a pre-trained image encoder are input into the emotion mapping layer to extract emotion-containing three-dimensional identity features corresponding to the training video; the emotion-containing three-dimensional identity features corresponding to the training video are input into a video generation network to obtain a predicted speech video; graphical emotion features corresponding to the video frames are extracted from the predicted speech video, and textual emotion features are extracted from the emotion descriptions corresponding to the emotion labels; and the parameters of the emotion mapping layer are updated based on the cosine similarity between the graphical emotion features and the textual emotion features. Preferably, the three-dimensional identity features obtained by the image encoder are embedded with emotion information through an emotion mapping layer composed of a multi-layer perceptron. The emotion mapping layer uses one-hot encoding of emotion as a control condition and projects the encoded three-dimensional identity features onto the emotion-containing three-dimensional identity features in the same identity feature space, so that the texture and geometry of the three-dimensional face conform to a specific emotional state. The emotion-containing three-dimensional identity features and the three-dimensional identity features have the same feature dimension. Illustratively, after the training of other modules is completed, the emotion mapping layer is trained separately using the CLIP image-text alignment network as a supervisory term. For example, for the randomly sampled identity feature encoding, an arbitrary emotion label is superimposed on it and mapped to obtain the emotion-identity feature. It is hoped that the three-dimensional face synthesized by this feature can not only keep the original identity feature unchanged, but also have the texture and geometric state that can show the emotion. Therefore, the CLIP encoder is used to align the rendered image with the corresponding emotion expression text, and the cosine similarity of the CLIP-encoded implicit vectors of the two is calculated. With minimizing the cosine similarity as the optimization goal, the parameters of the emotion mapping layer are updated. Among them, on the one hand, the CLIP image encoder is used to extract the first emotion feature from the rendered image, and on the other hand, the CLIP text encoder is used to extract the first emotion feature from the emotion label. (One-hot encoding) corresponding emotion description The second emotion feature is extracted from the text description and cosine similarity is calculated based on the first and second emotion features. Furthermore, when updating the parameters of the emotion mapping layer (flame symbol, indicating an update), the parameters of the dynamic generator in the video generation network are frozen (snowflake symbol, indicating a freeze), allowing the emotion mapping layer to focus on learning the differences between different emotions.

[0044] In order to improve the performance of the emotion mapping layer, the previous embodiment may be improved. According to another optional embodiment of the present invention, the pre-trained emotion mapping layer is trained in the following manner: non-zero emotion labels and three-dimensional identity features extracted from image frames of a training video using a pre-trained image encoder are input into the emotion mapping layer to extract the emotion-containing three-dimensional identity features corresponding to the training video, wherein the non-zero emotional expression represents a certain emotion type other than calm emotion; all-zero emotion labels and three-dimensional identity features extracted from image frames of a training video using a pre-trained image encoder are input into the emotion mapping layer to extract the three-dimensional identity features of emotional calmness corresponding to the training video, wherein the all-zero emotional expressions represent a calm emotion type; the emotion-containing three-dimensional identity features and the emotional calmness three-dimensional identity features corresponding to the training video are respectively input into the video generation network to obtain the predicted first speech video and the second speech video; the graphic emotion features corresponding to the video frames are extracted from the predicted speech video, and the text emotion features are extracted from the emotion description corresponding to the emotion label; the parameters of the emotion mapping layer are updated using a second total loss function, the second total loss function including a loss function for calculating the identity feature consistency between the first speech video and the second speech video, a loss function for calculating the color distribution consistency, and an alignment loss function for calculating the cosine similarity between the graphic emotion features and the text emotion features. For illustration, the aforementioned cosine similarity is used as one of the optimization objectives. Next, identity feature consistency and color distribution consistency loss functions are introduced to constrain the original face rendering and the emotion-enhanced face rendering, ensuring that the facial identity remains stable before and after mapping, thereby improving the performance of the emotion mapping layer.

[0045] 4. Video Generation Network

[0046] According to one embodiment of the present invention, a video generation network includes a dynamic generator and a renderer. The dynamic generator is used to generate a neural radiance field corresponding to each frame based on each action feature (per-frame action feature) and the three-dimensional identity feature containing emotion in the action sequence. The renderer is used to extract the second-perspective color information and volume information from the neural radiance field corresponding to each frame based on the user's camera parameter c to render and generate video frames. The generated multiple video frames are then synthesized into a speech video of the subject. It should be understood that implementers can fuse the speech video with the speech audio to obtain a speech video with sound.

[0047] According to one embodiment of the present invention, a method for synthesizing a speech video is provided, comprising: inputting speech audio into a pre-trained audio-action encoder to extract a speech feature sequence from the speech audio and generating a facial motion sequence based on the speech feature sequence; inputting a single two-dimensional face image captured from a first perspective into a pre-trained image encoder to extract three-dimensional identity features of the subject contained in the face image; inputting the three-dimensional identity features and emotion labels into a pre-trained emotion mapping layer to fuse them to obtain a three-dimensional identity feature containing emotion; inputting the motion sequence and the three-dimensional identity feature containing emotion into a video generation network based on a neural radiance field, and synthesizing a speech video of the subject from a desired second perspective that matches the speech audio using the neural radiance field and camera parameters, wherein the second perspective can be adjusted within a predetermined range based on the camera parameters. Based on user needs, the second perspective can be adjusted within a certain angle range, thereby generating a speech video of the subject from the same or different angle as the first perspective. The video generation network renders each video frame separately based on each motion feature, the three-dimensional identity feature containing emotion, and the camera parameters in the motion sequence, and then sequentially merges the video frames into a video. Therefore, for different elements in the action sequence, you can also customize the corresponding emotion label for each frame, so that different emotional changes can be shown in the synthesized video as needed.

[0048] In order to verify the effects of the embodiments of the present invention, the inventors also conducted comparative experiments.

[0049] In order to reasonably evaluate the robustness and quality of the model, this experiment uses the following multiple indicators:

[0050] Peak Signal-to-Noise Ratio (PSNR) measures the degree of distortion by calculating the error between the original image and the distorted image. The larger the value, the better.

[0051] The Structure Similarity Index Measure (SSIM) is a perceptual model that measures the degree of distortion of an image. The larger the value, the better.

[0052] The FID (Fréchet Inception Distance) indicator is a metric used to evaluate the quality of images generated by a generative model. The smaller the value, the better.

[0053] Mouth Landmark Distance (M-LMD): The mouth landmark distance metric measures the accuracy of mouth movement by detecting the positions of mouth landmarks in real and synthetic videos and calculating the difference between them. A smaller value means more accurate mouth movement.

[0054] The Sync-Net Confidence (Sync) metric uses the Sync-Net network to map the audio sequence and mouth video sequence into a shared implicit feature space. The Sync-Net network then measures the synchronization between the two feature sequences. A higher Sync confidence value indicates better results.

[0055] Emotional accuracy index, the emotion accuracy index uses the emotion detection model to classify the emotions of the video results and calculate the accuracy. The higher the accuracy, the more accurate the method is in controlling the emotions of the synthesized results.

[0056] Based on multiple indicators, the method of the present invention was compared with a single image audio-driven face speech video method under the same data set. The experimental results show that the method of the present invention has certain improvements over the comparison method in most examples.

[0057] This experiment mainly tests and compares the test set part (including four people with IDs M003, M030, W009 and W015) in the MEAD dataset (the MEAD dataset is a commonly used dataset for speech video synthesis tasks, which contains single-person speech videos with different emotions and multiple perspectives. This experiment adopts the training and test set division method commonly used in previous work). At the same time, some face images outside the dataset were selected for testing to demonstrate the robustness of the method of this invention.

[0058] On the MEAD dataset test set, the first frame of a frontal video was used as input and the corresponding audio was used as a driver. The quality of the synthesized videos was tested at 60° frontal and side angles. The results of the comparison of numerical metrics for speech videos are shown in Table 1, and the results of the comparison of numerical metrics for emotion editing are shown in Table 2. As can be seen, the proposed method shows significant performance improvements over the comparison methods in most metrics. The synthesis effect from the new perspective is significantly better than that of existing methods, and the emotion accuracy is also significantly higher.

[0059] Table 1

[0060]

[0061] Table 2

[0062]

[0063] For visual comparison, please refer to Figure 5 and Figure 6, comparing methods based on 3D representation (OTAvatar, Next3D) with methods based on image deformation (PC-AVS, SadTalker, StyleTalk). As can be seen, the proposed method can synthesize relatively clearer, more stable, and more emotionally accurate spoken-word videos at any viewing angle within a 120° frontal angle.

[0064] In general, some embodiments of the present invention can achieve at least one of the following beneficial effects:

[0065] (1) A dual-encoder decoupling framework based on a three-plane neural radiation field generator is proposed, which can achieve decoupled control of the three-dimensional identity features and facial movements of the face generator;

[0066] (2) A 3D face monocular reconstruction and emotion editing technology based on a feature pyramid encoder is proposed. It can reconstruct a 3D face using a single image and synthesize facial textures and geometry with specific emotions, and perform emotion control in a controllable and parameterized manner through emotion labels.

[0067] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0068] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0069] A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. Computer-readable storage media may include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove having instructions stored thereon, and any suitable combination thereof.

[0070] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A speech video synthesis method, characterized in that: include: Input the speech audio into a pre-trained audio-action encoder to extract a speech feature sequence from the speech audio and generate a facial action sequence based on the speech feature sequence; Input a single 2D face image taken from a first-person perspective into a pre-trained image encoder to extract the 3D identity features of the object contained in the face image; Input the 3D identity features and emotion labels into the pre-trained emotion mapping layer and fuse them to obtain the 3D identity features containing emotions. The action sequence and the three-dimensional identity features containing emotions are input into a video generation network based on the neural radiation field, and the speech video of the object with the required second perspective that matches the speech audio is synthesized through the neural radiation field and camera parameters, wherein the second perspective can be adjusted within a predetermined range through the camera parameters.

2. The method according to claim 1, characterized in that The pre-trained audio-motion encoder is trained as follows: Obtaining an audio-motion encoder and a first auxiliary training model, wherein the audio-motion encoder includes an audio encoder and a motion encoder, and the first auxiliary training model includes a lip reading recognition module and a pre-trained speech recognition model; Inputting the training audio extracted from the training video into an audio encoder to extract a training speech feature sequence, wherein the training video includes facial action frames and an audio file of a predetermined person speaking; Inputting the training speech feature sequence into the speech recognition module, extracting a potential feature sequence from the training speech feature sequence, and encoding the potential feature sequence into a first morpheme sequence; Input the training speech feature sequence into the action encoder to obtain the facial action sequence corresponding to the training audio; The lip reading recognition module reconstructs the face according to the facial parameters, extracts the lip movement geometric sequence from the reconstructed face sequence, and determines the second morpheme sequence based on the lip movement feature sequence extracted from the lip movement geometric sequence; While freezing the parameters of the audio encoder, speech recognition model, and video generation network, the parameters of the action encoder and lip reading recognition module are updated according to the first reconstruction loss between the facial action sequence corresponding to the training audio and the 3DMM parameter sequence detected from the training video, the second reconstruction loss between the latent feature sequence and the lip movement feature sequence, and the connection temporal classification loss between the first morpheme sequence and the second morpheme sequence determined by the first total loss function.

3. The method according to claim 2, characterized in that The action encoder includes a plurality of cascaded Transformer encoders and an action mapping layer, wherein each Transformer encoder includes: The periodic position encoding layer extracts audio codes with position information from the input features, where the input features of the first Transformer encoder are the output of the audio encoder; The multi-head attention layer processes the audio encoding input from the previous layer to incorporate contextual information and obtain a feature sequence; The feedforward layer processes each feature in the feature sequence input by the previous layer to obtain the action hidden feature. The feedforward layer of the last Transformer encoder obtains the final action hidden feature sequence; The action mapping layer is used to map the final action hidden feature sequence into an action sequence.

4. The method according to claim 2, characterized in that The pre-trained image encoder is trained in the following manner: Obtain an image encoder comprising a feature pyramid encoding layer for extracting multi-scale features from an input image and an identity mapping layer for extracting three-dimensional identity features from the multi-scale features; Acquire multiple image data of one or more people with different viewing angles and different facial expressions, select any two images of the same person in the image data as the original image and the driving image to form a sample pair, and form multiple sample pairs; Training an image encoder using multiple sample pairs, wherein a source image is input into the image encoder to extract 3D identity features, 3DMM parameters are extracted from the driving image using a preset method, and volume rendering is performed based on the 3D identity features and motion features and camera parameters in the extracted 3DMM parameters to obtain a predicted driving image; The parameters of the picture encoder are updated according to the difference between the driving image and the predicted driving image in the sample pair.

5. The method according to claim 4, characterized in that The pre-trained emotion mapping layer is trained in the following manner: The emotion labels and the 3D identity features extracted from the image frames of the training video using a pre-trained image encoder are input into the emotion mapping layer to extract the 3D identity features containing emotions corresponding to the training video; The three-dimensional identity features containing emotions corresponding to the training video are input into the video generation network to obtain the predicted speech video; Extracting graphic emotion features corresponding to video frames from the predicted speech video, and extracting text emotion features from the emotion descriptions corresponding to the emotion labels; The parameters of the emotion mapping layer are updated according to the cosine similarity between the graphic emotion features and the text emotion features.

6. The method according to claim 4, characterized in that The pre-trained emotion mapping layer is trained in the following manner: The non-zero emotion labels and the 3D identity features extracted from the image frames of the training video using a pre-trained image encoder are input into the emotion mapping layer to extract the 3D identity features containing emotions corresponding to the training video. Among them, the non-zero emotion expression represents an emotion type other than calm emotion. The emotion labels with all zeros and the 3D identity features extracted from the image frames of the training video using a pre-trained image encoder are input into the emotion mapping layer to extract the 3D identity features of the calm emotion corresponding to the training video. The emotional expressions with all zeros represent the calm emotion type. Inputting the emotional three-dimensional identity features and the emotionally calm three-dimensional identity features corresponding to the training video into the video generation network respectively to obtain the predicted first speech video and second speech video; Extracting graphic emotion features corresponding to video frames from the predicted speech video, and extracting text emotion features from the emotion descriptions corresponding to the emotion labels; The parameters of the emotion mapping layer are updated using a second total loss function, which includes a loss function for calculating the consistency of identity features between the first speech video and the second speech video, a loss function for calculating the consistency of color distribution, and an alignment loss function for calculating the cosine similarity between the graphic emotion features and the text emotion features.

7. A speech video synthesis system for implementing the method according to any one of claims 1 to 6, characterized in that: include: A pre-trained audio-action encoder for extracting a speech feature sequence from an input speech audio and generating a facial action sequence based on the speech feature sequence; A pre-trained image encoder is used to take a single 2D face image taken from a first-person perspective and extract the 3D identity features of the object contained in the face image; A pre-trained emotion mapping layer is used to fuse the 3D identity features and emotion labels to obtain the 3D identity features containing emotions; A video generation network based on neural radiation fields is used to obtain action sequences and three-dimensional identity features containing emotions, and to synthesize a speech video of the object with a required second perspective that matches the speech audio through the neural radiation field and camera parameters, wherein the second perspective can be adjusted within a predetermined range through the camera parameters.

8. A computer program product comprising a computer program / instruction, which implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: one or more processors; as well as a memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 7 by executing the executable instructions.

Images