A voice-driven facial animation generation method and system

Abstract

The application provides a speech-driven face animation generation method and system, carries out face key point extraction and standardization, takes the lip as the main reference, corrects the geometric position of the face key point by using the position relationship of the eyes and the lips, predicts the lip key point from the audio features, includes audio feature extraction, data preprocessing, Audio2MKP modeling and training and parameter optimization, Audio2MKP is a model for realizing the mapping from the speech to the lip key point, generates the reference image based on the lip key point, includes mask image generation, face region division, FTGAN modeling and training, and parameter optimization, FTGAN is a model for realizing the conversion of the face mask image into the face reference image, on the basis of the reference image, uses the audio features to guide the generation of the face animation, includes A2FGAN modeling and training, parameter optimization, and face animation synthesis, and A2FGAN is a model for realizing the face animation with the lip sound synchronization effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence virtual face applications, and in particular relates to a voice-driven facial animation generation method and system. Background Art

[0002] Voice-driven facial animation generation aims to generate smooth, natural, and lip-synced facial animations based on input voice information and facial imagery. It has broad application prospects in a wide range of fields, including virtual anchors, virtual customer service, online education, film special effects, and gaming entertainment. High-quality, highly realistic, lip-synced facial animations can enhance user identification and experience.

[0003] In recent years, with the continuous development of deep learning technology, various convolutional neural networks, generative adversarial networks and other models have been proposed and widely used, which has given rise to new research directions in facial animation generation technology. The learning mechanism is used to enable the trained facial model to have good lip shape performance.

[0004] Current technical means can be mainly summarized into two types, namely, facial animation generation based on intermediate features and end-to-end voice-synchronized facial animation generation.

[0005] (1) Facial animation generation based on intermediate features The lip key points are obtained from the audio features and used as intermediate features to guide the facial image to obtain lip-synchronized facial animation. Suwajanakorn et al. (2017) used deep learning methods to obtain lip movement information from speech as intermediate features, and then used traditional computer vision processing methods to synthesize facial textures and generate facial animations. Although their method can generate facial animations with strong synchronization, the edge areas of the synthesized lip textures are blurred and blocked, and the image frame quality needs to be improved. In addition, this method is studied for specific objects, and the model does not have the ability to generalize to the target object. Kumar et al. (2017) proposed an ObamaNet model based on the research work of Suwajanakorn et al. It uses the predicted lip movement information to mask the facial image, and then converts the masked image into a facial animation image frame through the image conversion model Pix2Pix. Although this method uses a trainable neural network module instead of the traditional computer vision method, it can improve the efficiency of facial animation generation to a certain extent, but the lip edges of the generated facial image are still relatively blurred, and the generalization ability to the target object is also very limited.

[0006] (2) End-to-end speech-synchronized facial animation generation directly inputs audio features and facial images into the same neural network module for training to generate facial animation. Vougioukas et al. (2018) used a generative adversarial network to establish a mapping from audio to facial animation. The generative adversarial network model they proposed includes a generator and three discriminators. Among them, the generator is a codec that receives audio and facial images as input and generates facial animation image frames; the discriminator consists of a frame discriminator, a sequence discriminator, and a synchronization discriminator, which is used to guide facial reconstruction, smoothness between image frames, and lip synchronization. Although this method can obtain facial animation with good lip synchronization and has certain generalization capabilities, when using only one facial image to generate facial animation, the character still lacks natural lip movement and lacks realism. Vougioukas et al. (2020) subsequently added character blinking movements to the animation, but blinking alone is still not enough to reflect the realism of facial animation. Chung et al. (2016) proposed the lip synchronization discriminant network SyncNet. It determines the similarity between audio and facial images in a common parameter space and calculates the cross-entropy loss between audio and facial image features in this common parameter space to reflect the lip synchronization effect. To address the shortcomings of the facial animations generated by Vougioukas et al. (2018), which lack natural facial movement, Prajwal et al., building on the work of Chung et al., further proposed the generative adversarial network-based LipGAN (2019) and its improved model Wav2Lip (2020). Unlike Vougioukas et al., which used a single facial image, the Wav2Lip model takes a sequence of image frames as input. While retaining the facial movement information of the original frame sequence, it uses audio features to guide changes in lip movement to generate facial animations. This model can generate facial animations with natural head movement and good lip synchronization, but the generated facial image frames are still unclear and the image quality needs to be improved.

[0007] A review of the current state of research in speech-driven facial animation reveals several key challenges, including limited lip synchronization, improved frame quality, weak generalization of facial models, and a lack of natural head movement in facial animations. Managing lip shape and facial movement while preserving local details to generate realistic lip-synced faces remains a significant technical challenge. Summary of the Invention

[0008] To address the challenges of existing technologies, this paper proposes a novel speech-driven facial animation generation scheme. Based on a given audio and facial image or video, a facial animation with the human figure as the speaker and synchronized with the audio is generated. The generated facial animation has good performance, meeting the following requirements: (1) good lip synchronization; (2) high-quality facial animation image frames; (3) the facial animation generation model has generalization capabilities; and (4) the generated facial animation has natural head movement.

[0009] To achieve this goal, the present invention provides a voice-driven facial animation generation method, which includes automatically generating facial animation through the following steps:

[0010] Step S1, extracting and standardizing facial key points, wherein the facial key point standardization process includes using the lips as a primary reference and utilizing the positional relationship between the eyes and lips to perform geometric position correction on the facial key points;

[0011] Step S2, predicting lip key points from audio features, including audio feature extraction, data preprocessing, modeling and training of the lip key point prediction model Audio2MKP, and parameter optimization of the lip key point prediction model Audio2MKP; the lip key point prediction model Audio2MKP is a model that realizes the mapping from speech to lip key points;

[0012] Step S3, generating a reference image based on lip key points, including mask image generation, face region segmentation, face transformation generative adversarial network (FTGAN) model building and training, and face transformation generative adversarial network (FTGAN) model parameter optimization; the face transformation generative adversarial network (FTGAN) model is a model for converting a face mask image into a face reference image;

[0013] Step S4, based on the reference image obtained in step S3 and using the audio features obtained in step S2 to guide the generation of facial animation, including speech-to-face generative adversarial network A2FGAN modeling and training, speech-to-face generative adversarial network A2FGAN parameter optimization, and facial animation synthesis; the speech-to-face generative adversarial network A2FGAN is a model for achieving facial animation with lip synchronization effect.

[0014] Moreover, when extracting facial key points, the key facial parts of the face image are first located, including eyebrows, eyes, nose, lips and facial contours, and the basic facial key points are determined.

[0015] Moreover, in step S2, the lip key point prediction model Audio2MKP is implemented based on the convolutional neural network to obtain accurate lip key point coordinates from the voice information. The implementation method includes the following steps:

[0016] 1) Audio2MKP modeling, including implementing a lip keypoint prediction model Audio2MKP based on a convolutional neural network to achieve mapping from speech to lip keypoints; the lip keypoint prediction model Audio2MKP includes several convolutional layers and several fully connected layers connected in sequence, and residual connections from the previous convolutional layer are added to some convolutional layers to form residual blocks;

[0017] 2) Audio2MKP training, which includes receiving input audio features, training the lip key point prediction model Audio2MKP, and optimizing the model parameters through backpropagation to obtain accurate predicted lip key points.

[0018] Furthermore, when the mask image is generated in step S3, the size of the mask area is determined by extending the lip area outward.

[0019] Moreover, when performing facial area division in step S3, the facial image is divided into different areas according to the importance from high to low, including the lip area, the face area and the background area; at the same time, a weight is set for each area, the weight of the lip area is greater than that of the face area, and the weight of the face area is greater than that of the background area.

[0020] Moreover, in step S3, a face conversion generative adversarial network model FTGAN is established based on the architecture improvement of the generative adversarial network to convert the face mask image into a face reference image. The implementation method includes the following steps:

[0021] 1) FTGAN modeling, including converting a mask image obtained by predicting lip key points into a face reference image based on the face conversion generative adversarial network model FTGAN; the face conversion generative adversarial network model FTGAN is composed of a generator network and a discriminator network, the generator network receives an input mask image and generates an output face reference image, wherein the STN module, the face encoder module, the CBAM module and the face encoder module are connected in sequence; the discriminator network includes a frame discriminator module, the reference image generated by the generator network and the corresponding true value image are input to the frame discriminator module together, and the mean square error between the generated label and the true value label is calculated to evaluate the quality of the generated image;

[0022] 2) FTGAN training: receiving the input mask image, training the face transformation generative adversarial network model FTGAN, optimizing the model parameters through back propagation, and obtaining a high-quality face reference image.

[0023] Moreover, in step S4, based on the architecture improvement of the generative adversarial network, a speech-to-face generative adversarial network A2FGAN is proposed to obtain the lip synchronization effect of facial animation. The implementation method includes the following steps:

[0024] 1) A2FGAN modeling, including using speech information to guide facial animation generation based on a reference image based on a speech-to-face generative adversarial network (A2FGAN); the speech-to-face generative adversarial network (A2FGAN) includes a generator network and two discriminator networks. The generator network consists of a face encoder / decoder module, an STN module, and a CBAM module. The two discriminator networks consist of a frame discriminator module and a lip synchronization discriminator module, respectively.

[0025] 2) A2FGAN training: receiving input audio features and reference images, training the speech-to-face generative adversarial network (A2FGAN), optimizing model parameters through backpropagation, and obtaining lip-synced facial animation image frames.

[0026] Moreover, when training the discriminator of A2FGAN, the lip synchronization discriminator module is first trained separately. Then, based on the trained lip synchronization facial animation images, the frame discriminator module is trained, and finally the facial animation image frames with lip synchronization and maintaining the true value image effect are obtained.

[0027] On the other hand, the present invention also provides a voice-driven facial animation generation system for implementing the voice-driven facial animation generation method described above.

[0028] Furthermore, the invention comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the voice-driven facial animation generation method as described above.

[0029] The present invention has the following characteristics and beneficial effects:

[0030] 1. To accurately obtain lip keypoint coordinates from speech information, we propose a lip keypoint prediction model, Audio2MKP, based on a convolutional neural network (CNN). This model addresses the computational burden and limited automation of earlier image processing methods for obtaining lip keypoints. Furthermore, compared to methods based on long short-term memory networks (LSTMs), the error between the predicted lip keypoints and the true values ​​is smaller, resulting in more accurate lip keypoint prediction using this model.

[0031] 2. To improve the quality of animated facial image frames, a face transformation generative adversarial network (FTGAN) is proposed to convert a face mask image into a face reference image. Considering the uncertainty of the face's size and position relative to the image, a combination of the STN module and the CBAM module is introduced in the FTGAN generator to improve the quality of the reference image generation without adding any additional overhead. The FTGAN discriminator utilizes a multi-scale discriminator (PatchGAN) to better guide the generator in outputting the reference image. Furthermore, considering the varying importance of different regions within a facial image, an attention mechanism module is introduced, and an image reconstruction loss function is designed based on facial region division. Therefore, the designed model can maximize the generation of high-quality face reference images.

[0032] 3. We propose a speech-to-face generative adversarial network (A2FGAN). Leveraging the SyncNet network's strengths in lip sync, we incorporate a lip sync discriminator into the A2FGAN discriminator. Furthermore, this model uses speech information to guide facial animation generation based on a reference image, further improving lip sync performance.

[0033] 4. To address the problem of lacking natural head movement in the generated facial animation when a facial image is provided as a single image, we propose a solution based on intermediate face transitions, using the facial pose transfer model Few_Shot_Vid2Vid, to transfer the facial animation of the intermediate character to the human image. This solution ensures natural head movement in facial animations generated from a single image, resulting in facial animations with generalizability.

[0034] In summary, the voice-driven facial animation generation solution proposed in this paper can automatically generate facial animations with high-quality image frames, good lip synchronization, and natural head movements. Furthermore, the facial animation generation model has generalizability. It has broad application prospects in a wide range of fields, including virtual anchors, virtual customer service, online education, film special effects, and gaming entertainment, and holds significant market value. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of voice-driven facial animation generation according to an embodiment of the present invention;

[0036] Figure 2 is a schematic diagram of key points of a face according to an embodiment of the present invention, wherein Figure 2 Part (a) is a schematic diagram of the key parts of the face. Figure 2 (b) Part of the diagram is a schematic diagram of facial key points;

[0037] Figure 3 The lip key point prediction model Audio2MKP according to an embodiment of the present invention;

[0038] Figure 4 Schematic diagram of several inappropriate mask area processing methods according to an embodiment of the present invention, wherein Figure 4 Part (a) is a schematic diagram covering the lower half of the entire face image. Figure 4 Part (b) is a schematic diagram of the lower half of the face area. Figure 4 Part (c) is a schematic diagram of the lip area only;

[0039] Figure 5 Schematic diagram of error in predicting lip key points according to an embodiment of the present invention, wherein Figure 5 Part (a) is a schematic diagram of the masked area of ​​the original image. Figure 5 Part (b) is a schematic diagram showing the error between the actual and predicted values;

[0040] Figure 6 is a schematic diagram of mask area selection according to an embodiment of the present invention, wherein Figure 6 Part (a) is a schematic diagram of the original image. Figure 6 Part (b) is a schematic diagram of mask area selection;

[0041] Figure 7 is a schematic diagram of the mask area segmentation process according to an embodiment of the present invention, wherein Figure 7 Part (a) is a schematic diagram of the original image. Figure 7 Part (b) is a schematic diagram of the masked area after segmentation;

[0042] Figure 8 2 is a schematic diagram of face area division according to an embodiment of the present invention;

[0043] Figure 9 It is a face transformation generative adversarial network FTGAN according to an embodiment of the present invention;

[0044] Figure 10 It is the STN module network structure of the embodiment of the present invention;

[0045] Figure 11 is the network structure of the CBAM module according to an embodiment of the present invention;

[0046] Figure 12 It is the speech-to-face generative adversarial network A2FGAN of an embodiment of the present invention;

[0047] Figure 13 This is a lip synchronization discrimination network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0049] The present invention generates a facial animation based on a given audio and facial image or video, with the facial image as the speaking subject and synchronized with the audio. The method includes extracting facial key points and their standardization, predicting lip key points from audio features, generating a reference image based on the lip key points, and generating facial animation based on the audio features and the reference image. The present invention provides a complete set of facial animation production steps, and the generated facial animation meets the following requirements: (1) good lip synchronization effect; (2) high-quality facial animation image frames; (3) the facial animation generation model has generalization capabilities; and (4) the generated facial animation has natural head movement.

[0050] like Figure 1 As shown, the voice-driven facial animation generation method provided by the embodiment of the present invention includes the following steps: Step 1: Extraction of facial key points and their standardization

[0051] Extracting facial feature points is the foundation of facial animation. The accuracy of these feature point locations directly impacts subsequent lip region processing and facial animation. This step extracts key facial points from the input facial image or video and normalizes them. The examples further address steps 1.1 and 1.2.

[0052] Step 1.1 Face key point extraction

[0053] The subsequent training of the lip key point prediction model requires obtaining the lip key point data corresponding to the audio. Therefore, the key facial parts of the face image are first located, including eyebrows, eyes, nose, lips, and facial contours, and the basic facial key points are determined. Figure 2 The extracted facial key points are shown in the figure. Some face recognition tools can be used in the specific implementation, such as the face recognition open source library Dlib, etc. Figure 2 The key parts of the face of the face image shown in (a) are located and 68 key points of the face are extracted, of which 20 are lip key points (such as Figure 2 (b)). The first key point of a face is generally the point on the upper left of the facial contour. Figure 2 As shown in (b), each key point is numbered in turn, and the first key point of the lip is numbered 49.

[0054] Step 1.2 Normalization of facial key points

[0055] The position, size, and rotation angle of the face relative to the entire image vary within the facial image frame. Therefore, the facial key points obtained in step 1.1 are normalized. Using the lips as the primary reference and the positional relationship between the eyes and lips, the facial key points are geometrically corrected to eliminate the effects of tilt angle deviations within the entire image and ensure that the face's position remains unchanged.

[0056] Step 2: Predict lip keypoints from audio features

[0057] This step extracts audio features from the input raw speech, vectorizes the data after data preprocessing, and accurately predicts the coordinates of the lip key points based on the audio features by building and training a lip key point prediction model. The embodiments further provide specific steps 2.1 to 2.4.

[0058] Step 2.1 Audio feature extraction

[0059] Extract the audio features of the speech from the original speech. In specific implementation, some speech recognition systems can be used to extract audio features, such as DeepSpeech. DeepSpeech is an open source speech recognition system based on a deep learning framework that can achieve end-to-end automatic speech recognition functions. The embodiment of the present invention preferably adopts the DeepSpeech speech recognition system to perform fast Fourier transform on the input speech and convert the time domain signal into a frequency domain signal; at the same time, the feature vector of each audio segment is obtained in the two-dimensional space composed of the time domain and the frequency domain; and the obtained feature vector is windowed to obtain the audio feature. The extracted audio features are 29-dimensional. In the embodiment of the present invention, the audio features extracted by DeepSpeech are defined as DSAudio_features for subsequent description.

[0060] It should be noted that current deep learning technology can already automatically extract audio features from raw audio. Therefore, the embodiment of the present invention preferably extracts DSAudio_features audio features instead of extracting traditional MFCC feature parameters.

[0061] Step 2.2 Data preprocessing

[0062] The data preprocessing processes the audio data and the video data respectively, and the embodiments further propose to specifically involve steps 2.2.1 and 2.2.2.

[0063] Step 2.2.1 Audio data processing

[0064] During specific implementation, multimedia video processing tools such as FFmpeg can be used to pre-process the original speech and convert them into monophonic speech data with a sampling rate of 16,000 Hz. The present invention further proposes that, on the basis of obtaining the DSAudio_features audio features, the dimension of one audio feature is added to meet the requirements of the convolution layer operation of the subsequent convolutional neural network for the data format. After data preprocessing, the speech data per second can be represented as one audio feature vector. The preferred unified format is (number of audio frames, dimension, window size of windowing processing, audio feature vector size), for example (n, 1, 16, 29), where n is the number of audio frames.

[0065] Step 2.2.2 Video data processing

[0066] In specific implementations, multimedia video processing tools such as FFmpeg can be used to extract facial image frames at a rate of 25 frames per second. Based on the facial key points extracted in step 1, principal component analysis (PCA) is used to reduce the dimensionality of the 20 lip key points in each facial frame. These 20 lip key points are represented by a certain number of PCA principal components. After data preprocessing, each second of video data can be represented as a lip key point feature vector in the uniform format of (number of video frames, number of PCA principal components), for example, (25, 8).

[0067] Step 2.3: Modeling and training the Audio2MKP lip key point prediction model

[0068] This embodiment of the present invention proposes building a convolutional neural network-based lip key point prediction model (Audio to Mouth Key Points, Audio2MKP) to map speech to lip key points. This model receives the input DSAudio_features audio features and repeatedly trains the Audio2MKP model in conjunction with the subsequent step 2.4, ultimately achieving accurate prediction of lip key points. This involves steps 2.3.1 and 2.3.2.

[0069] Step 2.3.1 Audio2MKP Modeling

[0070] Early image processing methods for obtaining lip key points required a large computational workload and were not very automated. Methods based on long short-term memory (LSTM) networks also provided low accuracy in obtaining lip key points. Convolutional neural networks (CNNs) are a type of neural network specifically designed to process data with a grid-like structure. They can capture local properties of images and perform well in image recognition. This paper designs the Audio2MKP model based on CNNs.

[0071] The network structure of the Audio2MKP model is as follows Figure 3 As shown in the figure, it includes several convolutional layers and several fully connected layers connected in sequence. Residual connections from the previous convolutional layer are added to some convolutional layers to form residual blocks.

[0072] In the embodiment, 12 convolutional layers (conv, sequentially denoted as conv-1 to conv-12) and 2 fully connected layers (linear, sequentially denoted as linear-1 and linear-2) are preferably included to learn the mapping of DSAudio_features audio features to lip key points.

[0073] In this model, the convolutional layers use a uniform 3×3×3 convolution kernel. Each convolutional layer contains a vector convolution operation, and the feature map of the audio features is activated using the ReLU function before entering the next convolutional layer. This function is defined by Equation (2).

[0074] Relu(x)=max(0,x) (2)

[0075] Among them, x is the feature map of audio features.

[0076] Some convolutional layers (such as conv-2, conv-3, conv-5, conv-6, conv-8, conv-9, and conv-12 in the figure) are connected with the residual connection from the previous convolutional layer to form a residual block (see Figure 3 The 7 residual blocks formed by the dotted arrows in the middle) are used to reduce the problem of training errors as the depth of the network increases. Each residual block adds the input to the output of the convolution layer and then activates it. Take the connection between conv-1 and conv-2 as an example. The input of conv-2 (that is, the output of conv-1 after activation) is directly added to the convolution result of conv-2, and then activated. The same applies to other similar situations. Each audio feature vector of shape (1,16,29) is processed by 12 convolutional layers to obtain a feature map of shape (256,1,1). The feature map is further reduced in dimensionality by reshaping to obtain a one-dimensional column vector containing 256 eigenvalues.

[0077] The convolutional layer is connected to the fully connected layer via Reshape. Fully connected layer 1 (linear-1) extracts a column vector containing 64 eigenvalues, performs BatchNorm normalization, and passes through fully connected layer 2 (linear-2) to obtain 8 eigenvalues ​​representing the lip key points. PCA restoration and inverse normalization are then performed to obtain the predicted coordinates of the 20 lip key points. Here, the sklearn.inverse_transform() method in Python can be used to restore the data during PCA restoration. Inverse normalization multiplies the two-dimensional matrix restored by PCA by the binormal recorded during normalization, and then adds the lip center coordinates to obtain the inverse-normalized coordinates of the lip key points.

[0078] Step 2.3.2 Audio2MKP training

[0079] like Figure 3 As shown, the DSAudio_features audio features are input into the Audio2MKP model, and the Audio2MKP model is trained in combination with step 2.4 to obtain the predicted lip key point feature vector representation.

[0080] Step 2.4 Audio2MKP model parameter optimization

[0081] The root mean square error (RMSE) between the predicted lip keypoint feature representation and the true lip keypoint feature representation is calculated to evaluate the accuracy of the predicted lip keypoints, expressed as Loss RMSE , calculated by formula (1).

[0082]

[0083] Among them, n represents the number of lip key points, y i 、 They represent the true value lip key point coordinates corresponding to the i-th frame face image and the predicted lip key point coordinates respectively.

[0084] The model training in step 2.3 is an iterative process. The error is calculated in step 2.4, and the model parameters are updated through backpropagation to finally obtain a well-trained Audio2MKP model.

[0085] Step 3: Generate reference image based on lip key points

[0086] This step uses the predicted lip key points obtained in step 2 to process the lip area of ​​the face image to generate a mask image, and then establishes and trains a face conversion generative adversarial network model to obtain a high-quality face reference image. The embodiment further proposes specific steps 3.1 to 3.4.

[0087] Step 3.1 Mask image generation

[0088] Using the predicted lip key points as a reference, a suitable area in the facial image is masked to conceal the original lip area. The lip contour is then outlined based on the predicted lip key points to form a mask image. In this method, partial or full masking of the selected image is performed to control the image processing area. The specific image to be masked is defined as the mask image. The effectiveness of the masking process on the facial image directly affects the performance of the FTGAN model in step 3.3. This involves steps 3.1.1 and 3.1.2.

[0089] Step 3.1.1 Mask area selection

[0090] Generating a mask image prepares for the next steps of FTGAN model training and reference image generation. This requires selecting an appropriate mask region. If the mask region is too large, the model's attention will be dispersed, hindering lip reconstruction. If the mask region is too small, discrepancies between the predicted and true lip keypoints will result in distortion of the generated reference image. Figure 4 Several inappropriate mask area processing methods are given.

[0091] Figure 4 (a) The mask covers the entire lower half of the face image. This method simply masks the entire lower half of the image, which will lose many facial details and does not help the model learn the mapping from the masked area to the face. Figure 4 (b) The mask area covers the lower half of the face. This method will lose facial details in the lower half of the face, such as the chin. Figure 4 (c) The masked area only covers the lip area. Although this method can preserve facial details to the greatest extent, the lip key points predicted by the model training will have certain errors, which will directly affect the authenticity of the reference image.

[0092] Figure 5 Schematic diagram showing errors in the predicted lip key points. Figure 5 (a) and Figure 5In (b), the area enclosed by the light gray line represents the actual lip area of ​​the given face, while the area enclosed by the dark gray line represents the lip area predicted based on the audio. If the predicted lip keypoints contain errors, masking the image using the predicted lip area will not fully cover the lip area of ​​the original face image, indicating the light gray line outside the black mask area.

[0093] Therefore, when selecting the mask area, it is recommended to consider the following two principles:

[0094] (1) The mask area covers as little facial area of ​​the given face image as possible except for the lip area;

[0095] (2) The mask area can accommodate a certain degree of error in predicting lip key points.

[0096] The present invention adopts the method of extending the lip area outward to determine the size of the mask area. Figure 6 As shown, Figure 6 (a) The original image is based on the rectangular area where the lip key points are located, and extends to the left and right in the width direction. The width of the lip area and its extension in the height direction up and down The height of the lip area. At the same time, the first key point of the lip (i.e. Figure 2 The feature point numbered 49 in (b) is used as the reference to shift and align the predicted lip key points to further reduce the impact of the error caused by the predicted lip key points, and the following is obtained: Figure 6 (b) The masked area shown.

[0097] In specific implementation, the preferred method is to use the Dlib tool to extract 20 lip key points of the face image through step 1.1, enclose these key points into the lip area, and form a rectangular area, and then extend outward from the area as the center. times to form a mask area.

[0098] Step 3.1.2 Mask region segmentation

[0099] Although the appropriate mask region was selected in step 3.1.1, it contains different components. To allow the model to subsequently train on these components, the selected mask region is segmented into lips, tongue, teeth, and others. Each of these segmented components is then filled in to create a masked image. Figure 7 Schematic diagram of mask area segmentation processing. Figure 7 (b) for Figure 7(a) shows the result of mask region segmentation on the original image. The area surrounded by solid lines represents the lips, the white filled area represents the teeth and tongue, and the black area represents the rest of the mask region.

[0100] Step 3.2 Face area segmentation

[0101] When converting a facial image, it is necessary to retain the non-masked area for the reference image, that is, to map the pixel values ​​of the area one by one to the generated reference image, and restore the masked area to the real lip area of ​​the face.

[0102] The quality of the generated image is generally evaluated based on the facial area in the image. The facial area, especially the lip area, is more important than the background area. Therefore, the facial image is divided into the following categories according to the importance from high to low: Figure 8 The different regions shown include the lip region, the face region, and the background region. At the same time, a weight is set for each region, with the lip region having a greater weight than the face region, and the face region having a greater weight than the background region.

[0103] Figure 8 In the figure, the solid-line frame, dashed-line frame, and dotted-line frame represent the background area, the face area, and the lip area, respectively. In this embodiment, the solid-line frame covers the entire face image; the dashed-line frame is a rectangular area enclosed by 68 facial key points, including the entire head information including the facial features; and the dotted-line frame is a rectangular area enclosed by 20 lip key points. The solid-line frame contains the dashed-line frame, and the dashed-line frame contains the dotted-line frame. This does not mean that the entire image is divided into several independent areas, but rather that the divided areas have a certain containment relationship.

[0104] When calculating the background area image reconstruction loss in the subsequent step 3.4, λ3·Loss in formula (5) MAE (x bg ,G(z) bg ) will calculate the image reconstruction loss of the face area and lip area with weight λ3. Similarly, when calculating the image reconstruction loss of the face area in the subsequent step 3.4, λ4·Loss in formula (5) MAE (x f ,G(z) f ) will calculate the image reconstruction loss for the lip region with a weight of λ4. In other words, when processing facial images in this way, the face and lip regions are included in the image reconstruction loss calculation multiple times. These two regions are precisely where the model focuses the most attention, and this is essentially a method of assigning weights to the face and lip regions. Therefore, using this inclusion relationship to divide facial images is well suited for facial image conversion in the TFGAN model.

[0105] The dotted area is actually drawn based on the information of 68 key points of the face, including the location information of the facial features and facial contours. Based on the following considerations, the facial area represented by the dotted area in the present invention does not include the entire face. This is because: (1) the head area is an irregular shape, which is not convenient for calculating the image reconstruction loss of the face area in step 3.4; (2) the face area is not the area of ​​most concern to the model, and the rest of the head can be treated as the background area and is also well suited for the face image conversion task.

[0106] The lip area represented by the dotted box is actually the mask area, which extends in width and height with the lip area as the center. The area in the dotted box is the area that the model pays most attention to. The quality of the generated reference image depends largely on the accuracy of this area, including whether the lips are accurate, the details of the teeth and tongue areas, and whether the boundaries are smooth.

[0107] Step 3.3 Face Transformation Generative Adversarial Network (FTGAN) modeling and training

[0108] Construct a Face Transformation Generative Adversarial Network (FTGAN). Combined with the subsequent step 3.4, the FTGAN model is trained to convert the mask image obtained by predicting lip key points into a high-quality reference face image. This method defines a reference image as an image that restores the mask image to a realistic facial image. This involves steps 3.3.1 and 3.3.2.

[0109] Step 3.3.1 FTGAN modeling

[0110] The Generative Adversarial Network (GAN) (2014) is an unsupervised deep learning model that achieves good output through a game of learning between a generator (G) and a discriminator (D). The generator creates images that can deceive the discriminator. Through adversarial learning, the discriminator becomes stronger and better able to distinguish real images from fake ones. This paper designs the FTGAN model by modifying the internal architecture of the GAN.

[0111] like Figure 9As shown, the FTGAN model consists of a generator network and a discriminator network. Taking into account the uncertainty of the face's size and position relative to the image, the present invention proposes introducing the STN module (Spatial Transformer Network) (2015) and the CBAM module (Convolutional Block Attention Module) (2018) into the traditional generator when designing the generator network structure. The combination of STN and CBAM improves the quality of reference image generation. As lightweight, universal modules, integrating STN and CBAM into the network effectively enhances the model's expressiveness without adding any additional computational overhead. When designing the discriminator network structure, a multi-scale discriminator (Patch-based Generative Adversarial Network, PatchGAN) network architecture (2017) is adopted. Unlike traditional discriminator networks that only output a single evaluation value, PatchGAN evaluates the generated image in multiple dimensions, which can better guide the generator in outputting a reference image of the face.

[0112] 1. Generator network structure

[0113] The generator network consists of a face encoder / decoder module, an STN module, and a CBAM module. It receives a mask image as input and produces a face reference image as output. From input to output, the STN module, face encoder module, CBAM module, and face encoder module are connected sequentially. The encoder / decoder module uses the face encoder to extract feature maps representing the mask image during encoding and restore the image information during decoding. Since the size and position of a face in an image frame are often uncertain, the STN module can eliminate this problem during face image transformation. The CBAM module focuses on intermediate channels that have a greater impact on the task during convolution operations.

[0114] The encoder / decoder module consists of a downsampling convolutional layer, a residual connection block, and an upsampling convolutional layer. The downsampling convolutional layer extracts features and reduces their dimensionality. The residual connection block includes a direct mapping part and a residual part to address the problems of vanishing gradients and network degradation. The upsampling convolutional layer is used for feature decoding and image recovery.

[0115] STN module. The STN network structure is as follows Figure 10As shown in the figure, it consists of three parts: a localization network, a grid generator, and a sampler. The input mask image undergoes a series of affine transformations in the localization network to obtain the affine transformation parameters θ, which are then input into the network generator; the grid generator performs spatial data transformation based on θ to obtain Tθ(G), which is then input into the sampler; the input mask image is also input into the sampler, which multiplies and combines the image and outputs the feature map U1, which handles the situation where the corresponding pixel point cannot be indexed when the output coordinates are decimals. The STN module performs an affine transformation on the input and output to obtain a mapping relationship between the two, and uses network backpropagation to optimize the parameters to obtain the optimal value of the data in spatial position.

[0116] CBAM module. The CBAM network structure is as follows Figure 11 As shown in the figure, it consists of a Channel Attention Module (CAM) and a Spatial Attention Module (SAM). CAM uses the outputs of global max pooling and average pooling with a shared multilayer perceptron to implement the channel attention mechanism; SAM uses global max pooling and average pooling to aggregate channel features to implement the spatial attention mechanism. Input feature U2 is fed into the Channel Attention Module CAM. The output of the Channel Attention Module CAM is multiplied by the input feature U2, and the resulting intermediate result is fed into the Spatial Attention Module SAM. The output of the Spatial Attention Module SAM is multiplied by the intermediate result to obtain the refined feature U3. In this way, CBAM obtains the locally refined feature U3 through the attention mechanism.

[0117] The multiplication and combination refers to the element-level multiplication operation of the feature map.

[0118] 2. Discriminator network structure

[0119] The discriminator network structure mainly includes a frame discriminator module. Preferably, the frame discriminator adopts the network structure of PatchGAN. PatchGAN is a discriminator of the generative adversarial network with a fully convolutional network. The input true value image is cut into an N×N matrix X. After cutting, each image block patch is sent to the PatchGAN discriminator for discrimination. Each x in the matrix ij Represents the evaluation value (true or false) of a region with coordinates (i, j) in the image. ij The discriminator output is then averaged to evaluate the image generated by the generator. A typical GAN ​​discriminator maps the input to a real number and outputs a single evaluation value (true or false). Unlike a GAN discriminator, PatchGAN uses a fully convolutional architecture, allowing it to evaluate the generated image in multiple dimensions, guiding the generator to focus on more details during the image generation process.

[0120] Step 3.3.2 FTGAN training

[0121] When training the FTGAN model, such as Figure 9 As shown in the figure, the mask image is input into the STN module of the generator layer. The mask image undergoes an affine transformation to obtain feature map U1. U1 is input into the face encoder, where it passes through a downsampling convolution layer and a residual convolution block to obtain feature map U2. U2 is input into the CBAM module, where the CAM and SAM functions together to obtain feature map U3. U3 is input into the face decoder, where it passes through an upsampling convolution layer to obtain a preliminary reference image.

[0122] On this basis, the reference image generated by the generator layer and the corresponding ground-truth image are input into the frame discriminator network. The mean squared error between the generated label and the ground-truth label is calculated to evaluate the quality of the generated image. Combined with step 3.4, the model parameters are updated through backpropagation, and the FTGAN model is repeatedly trained to obtain reference images with good training results. Here, the ground-truth image refers to a digital representation that truly reflects the characteristics of the original image.

[0123] Step 3.4 FTGAN model parameter optimization

[0124] The image reconstruction loss function based on face area division is used to calculate the image reconstruction loss of the reference image. The reference image and the corresponding true image are input into the frame discriminator to calculate the error between the two, and the model parameters are updated through back propagation.

[0125] Loss function of the FTGAN model total From the loss function of general GAN ​​Loss gan And image reconstruction loss function Loss rec It consists of two parts and is calculated by formula (3), namely

[0126] Loss total =λ1·Loss gan +λ2·Loss rec (3)

[0127] Among them, λ1 and λ2 represent the weights of GAN loss and image reconstruction loss, respectively.

[0128] Loss gan With Loss rec Calculate by formula (4) and formula (5) respectively, that is

[0129] Loss gan =Loss MSE (D(x),D(G(z)) (4)

[0130] Loss rec=λ3·Loss MAE (x bg ,G(z) bg )+λ4·Loss MAE (x f ,G(z) f )+λ5·Loss MAE (x m ,G(z) m ) (5)

[0131] Among them, x and z represent the true image and the noise image respectively, G(z) represents the generated image, x bg 、x f with x m Respectively represent the image background area, face area, and lip area after face segmentation, G(z) bg Represents the background area of ​​the generated image G(z), G(z) f Represents the face area of ​​the generated image G(z), G(z) m represents the lip region of the generated image G(z); λ3, λ4, and λ5 represent the loss weights for the background, face, and lip regions, respectively. When setting the weights for the loss function calculation, follow the following principles: ensure that the model weights the lip region more than the face region, and vice versa.

[0132] Loss MSE With Loss MAE They represent the minimum mean square error and the minimum absolute value error respectively, which are calculated by equations (6) and (7), namely.

[0133]

[0134]

[0135] Where n1 and n2 represent the dimension of the discriminator evaluation and the number of pixels in the image, respectively.

[0136] The loss function uses mean squared error to calculate the generative adversarial loss and absolute value error to calculate the image reconstruction loss, and assigns different weights to these two parts.

[0137] The model training in step 3.3 is an iterative process. The error is calculated in step 3.4, and the model parameters are updated through back propagation to finally obtain a well-trained FTGAN model.

[0138] Step 4: Generate facial animation based on audio features and reference images

[0139] This step builds upon the reference image obtained in step 3 and again utilizes the audio features obtained in step 2 to guide the generation of facial animation. To achieve lip sync for the facial animation, a speech-to-face generative adversarial network model is established and trained, performing facial animation synthesis to produce the final facial animation. The examples further address specific steps 4.1 and 4.2.

[0140] Step 4.1 Speech-to-Face Generative Adversarial Network A2FGAN Modeling and Training

[0141] Build an Audio-to-Face Generative Adversarial Network (A2FGAN), which receives the DSAudio_features audio features and the reference image as input. Combined with the subsequent step 4.2, the A2FGAN model is trained to obtain animated facial image frames. This involves steps 4.1.1 and 4.1.2.

[0142] Step 4.1.1 A2FGAN modeling

[0143] The preferred solution of the embodiment of the present invention is to use the GAN architecture to design the A2FGAN model. However, it is different from the traditional GAN ​​architecture. Figure 12 As shown in Figure 1, it consists of a generator network and two discriminator networks. When designing the generator network, we also introduce a combination of STN and CBAM modules to improve image generation quality. When designing the discriminator network, we build a frame discriminator based on the PatchGAN network structure. Furthermore, to determine whether audio and video are synchronized, we further leverage the advantages of SyncNet (2016) in determining lip sync and design a SyncNet-based lip sync discriminator in the discriminator.

[0144] 1. Generator network structure

[0145] The generator network consists of a face encoder, decoder module, STN module and CBAM module.

[0146] The encoder module consists of a face encoder and a DSAudio_features audio encoder. The face encoder shares the same task as the face encoder in the FTGAN model in step 3.3.1: extracting feature maps of the input image and their encoded representations. The DSAudio_features audio encoder extracts feature maps of the DSAudio_features audio features and their encoded representations. It consists of convolutional layers with residual connections. The face decoder has a similar structure to that in the FTGAN model. However, the face decoder in the A2FGAN model concatenates the feature maps extracted by the two encoders before decoding.

[0147] The reference image is fed into the STN module, where it undergoes an affine transformation to generate feature map U1. U1 is fed into the face encoder to generate feature map U2. The DSAudio_features audio features are then fed into the DSAudio_features audio encoder to generate feature map U3. U2 and U3 are concatenated and fed into the CBAM module to generate feature map U4. U4 is fed into the face decoder to generate an animated facial image frame.

[0148] 2. Discriminator network structure

[0149] The two discriminator networks are composed of a frame discriminator module and a lip synchronization discriminator module respectively.

[0150] Frame discriminator module. It adopts the same structure as the PatchGAN discriminator in the FTGAN model.

[0151] Lip synchronization discriminator module. The lip synchronization discriminator network based on SyncNet is as follows: Figure 13 As shown in the figure, it determines the similarity between audio features and facial image features in a common parameter space to evaluate the synchronization effect between the generated facial animation image frame and the corresponding audio frame. Taking into account the uncertainty of the position and size of the face relative to the entire image and the practical application of audio features corresponding to lip movements, the SyncNet model is adaptively modified. Instead of using the lower half of the facial image as input, the lip mask area centered on the lip coordinates of the facial image data is used as input to calculate the similarity between the audio frame and the image frame in the common parameter space. In this way, the area can cover the lip area and can be migrated using the model parameters pre-trained in this dimension.

[0152] In this network, Figure 13 As shown, two convolutional neural networks (CNN1 and CNN2) are used to extract feature information from audio frames and facial image frames of the same length, respectively. A weight-sharing module assigns shared weights to these two pieces of information, transforming the features of audio and image frames of different modalities into the same parameter space. A loss function is then used to calculate the difference between the cosine similarity label vector and the true label vector.

[0153] Step 4.1.2 A2FGAN training

[0154] When training the A2FGAN model, Figure 12As shown in the figure, the reference image is first input into the STN module, where it undergoes an affine transformation to obtain feature map U1. U1 is then input into the face encoder, where it passes through a downsampling convolution layer and a residual convolution block to obtain feature map U2. The DSAudio_features audio features are then input into the DSAudio_features audio encoder, where they pass through a convolution layer with a residual connection to obtain feature map U3. U2 and U3 are then concatenated and input into the CBAM module, where they pass through the CAM and SAM modules to obtain feature map U4. U4 is then input into the face decoder, where it passes through an upsampling convolution layer to obtain a preliminary animated facial image frame.

[0155] The present invention preferably proposes to train the lip sync discriminator and the frame discriminator separately, that is, first train the lip sync discriminator separately, and then train the frame discriminator.

[0156] 1. Lip synchronization discriminator training

[0157] First, we train the lip sync discriminator to determine whether the person's mouth shape and voice are synchronized. If they are, the model is trained well; otherwise, there is an error, so we need to repeat the training.

[0158] When training a lip sync discriminator, such as Figure 12 As shown in Figure 1, the facial animation image frame generated by the generator layer and the corresponding true value audio are input into the lip synchronization discriminator. After CNN1, CNN2 and weight sharing modules, the feature representation of the facial image frame and the audio frame in the common parameter space is obtained. The cosine similarity is used to calculate the similarity between the feature representations of the facial image frame and the audio frame in the common parameter space, and the cosine similarity label vector between the two is obtained. The cosine similarity is calculated by formula (8), that is,

[0159]

[0160] Among them, n represents the dimension of the image frame and the audio frame in the common parameter space, A i With V i They represent the feature values ​​of the audio frame and image frame sequence on the i-th dimension in the common parameter space.

[0161] 2. Frame discriminator training

[0162] When training the frame discriminator, the synchronized facial animation image frames that were trained when training the lip synchronization discriminator are used. Although lip synchronization has been achieved at this time, there will still be differences with the true value image, so training must continue until the effect of the true value image is completely achieved.

[0163] The lip-synced animated facial image frames and their corresponding ground-truth images are fed into the frame discriminator. The mean squared error between the generated and ground-truth labels is calculated to evaluate the quality of the generated images. Combined with step 4.2, the model parameters are updated through backpropagation, and the A2FGAN model is repeatedly trained to produce animated facial image frames that are lip-synced and retain the ground-truth quality.

[0164] Step 4.2A2FGAN model parameter optimization

[0165] Loss function of the A2FGAN model total It consists of three parts, namely the generation of adversarial loss Loss of the frame discriminator gan , lip synchronization loss of lip synchronization discriminator sync And the reconstruction loss of face image Loss rec , the calculation formula is shown in formula (9).

[0166] Loss total =λ1·Loss gan +λ2·Loss sync +λ3·Loss rec (9)

[0167] Among them, λ1, λ2 and λ3 represent the weights of each part of the loss.

[0168] Loss gan With Loss rec The calculation of is the same as that of the FTGAN model, using formula (4) and formula (5) respectively.

[0169] Calculate the binary cross entropy loss between the cosine similarity label vector and the true value label vector to get Loss sync , which is calculated by formula (10).

[0170] Loss sync =Loss BCE (cosine_similarity(D sync (audio,G(x))),y) (10)

[0171] Among them, audio represents the audio corresponding to the true value image, G(x) represents the image generated by the generator based on the reference image, and D sync Denotes the lip synchronization discriminator built based on the SyncNet model, D sync (audio, G(x)) returns the representation of audio and G(x) in the same parameter space. The cosine_similarity() function calculates the cosine similarity between the two. The calculation formula is (8), where y represents the true value label vector corresponding to lip synchronization. LossBCE Calculated by formula (11),

[0172]

[0173] Among them, N represents the dimension of the label vector, y i With p i Represent the corresponding true value label and cosine similarity label in the i-th dimension respectively.

[0174] The model training in step 4.1 is an iterative process. The error is calculated in step 4.2, and the model parameters are updated through backpropagation to finally obtain a well-trained A2FGAN model.

[0175] Step 4.3 Facial animation synthesis

[0176] Facial input generally comes in two forms: face videos and face images. For the former, given a sequence of facial image frames, the face images in the video already exhibit natural head movement. For the latter, however, the images require certain processing to generate facial animations with natural head movement. Therefore, when synthesizing facial animations, we address the following two different scenarios.

[0177] 1. When the face image is given in the form of a video

[0178] Since the facial images in the video already have natural head movements, facial animations can be directly generated based on the trained A2FGAN model.

[0179] 2. When the face image is given in the form of a single image

[0180] When a facial image is given as a single image, to ensure natural head movement in the facial animation generated from the speech-driven single facial image, a standard intermediate facial image frame sequence is selected for the given speech and a single target facial image. Based on the given audio and intermediate facial image frame sequence, an intermediate facial animation synchronized with the audio is generated. The Few_Shot_Vid2Vid model (2019) is then used to construct a facial pose transfer model, which transfers the intermediate facial animation to the given single facial image, generating and outputting the facial animation. The Few_Shot_Vid2Vid model is a conditional video synthesis network based on a Generative Adversarial Network (GAN). It can synthesize identical live-action videos using a small number of target images and has scene generalization capabilities, making it suitable for face transfer and generating facial animation videos.

[0181] Abbreviation Table

[0182] Table: Abbreviation table

[0183]

[0184]

[0185] In specific implementation, the method proposed in the technical solution of the present invention can be automatically run by those skilled in the art using computer software technology. System devices that implement the method, such as computer-readable storage media that store the corresponding computer program of the technical solution of the present invention and computer equipment that runs the corresponding computer program, should also be within the scope of protection of the present invention.

[0186] In some possible embodiments, a voice-driven facial animation generation system is provided, including a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a voice-driven facial animation generation method as described above.

[0187] In some possible embodiments, a voice-driven facial animation generation system is provided, including a readable storage medium having a computer program stored thereon. When the computer program is executed, the voice-driven facial animation generation method as described above is implemented.

[0188] The specific embodiments described herein are merely illustrative of the spirit of the present invention. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments without departing from the spirit of the present invention or exceeding the scope of the appended claims.

Claims

1. A voice-driven facial animation generation method, characterized by: The following steps are included to automatically generate facial animation: Step S1, extracting and standardizing facial key points, wherein the facial key point standardization process includes using the lips as a primary reference and utilizing the positional relationship between the eyes and lips to perform geometric position correction on the facial key points; Step S2, predicting lip key points from audio features, including audio feature extraction, data preprocessing, modeling and training of the lip key point prediction model Audio2MKP, and parameter optimization of the lip key point prediction model Audio2MKP; the lip key point prediction model Audio2MKP is a model that realizes the mapping from speech to lip key points; Step S3, generating a reference image based on lip key points, including mask image generation, face region segmentation, face transformation generative adversarial network (FTGAN) model building and training, and face transformation generative adversarial network (FTGAN) model parameter optimization; the face transformation generative adversarial network (FTGAN) model is a model for converting a face mask image into a face reference image; Based on the architecture improvement of the generative adversarial network, a face conversion generative adversarial network model FTGAN is established to convert the face mask image into a face reference image. The implementation method includes the following steps: 1) FTGAN modeling, including converting a mask image obtained by predicting lip key points into a face reference image based on the face conversion generative adversarial network model FTGAN; the face conversion generative adversarial network model FTGAN is composed of a generator network and a discriminator network, the generator network receives an input mask image and generates an output face reference image, wherein the STN module, the face encoder module, the CBAM module and the face encoder module are connected in sequence; the discriminator network includes a frame discriminator module, the reference image generated by the generator network and the corresponding true value image are input to the frame discriminator module together, and the mean square error between the generated label and the true value label is calculated to evaluate the quality of the generated image; 2) FTGAN training: receiving the input mask image, training the face transformation generative adversarial network model FTGAN, optimizing the model parameters through backpropagation, and obtaining a high-quality face reference image; Step S4, based on the reference image obtained in step S3 and using the audio features obtained in step S2 to guide the generation of facial animation, including speech-to-face generative adversarial network A2FGAN modeling and training, speech-to-face generative adversarial network A2FGAN parameter optimization, and facial animation synthesis; the speech-to-face generative adversarial network A2FGAN is a model for achieving facial animation with lip synchronization effect.

2. The voice-driven facial animation generation method according to claim 1, wherein: When extracting facial key points, the key facial parts of the face image are first located, including eyebrows, eyes, nose, lips and facial contours, and the basic facial key points are determined.

3. The voice-driven facial animation generation method according to claim 1, wherein: In step S2, the lip key point prediction model Audio2MKP is implemented based on the convolutional neural network to obtain accurate lip key point coordinates from the voice information. The implementation method includes the following steps: 1) Audio2MKP modeling, including implementing a lip keypoint prediction model Audio2MKP based on a convolutional neural network to achieve mapping from speech to lip keypoints; the lip keypoint prediction model Audio2MKP includes several convolutional layers and several fully connected layers connected in sequence, and residual connections from the previous convolutional layer are added to some convolutional layers to form residual blocks; 2) Audio2MKP training, which includes receiving input audio features, training the lip key point prediction model Audio2MKP, and optimizing the model parameters through backpropagation to obtain accurate predicted lip key points.

4. The voice-driven facial animation generation method according to claim 1, wherein: When the mask image is generated in step S3, the size of the mask area is determined by extending the lip area outward.

5. The voice-driven facial animation generation method according to claim 1, wherein: When performing facial area division in step S3, the facial image is divided into different areas according to the importance from high to low, including the lip area, the face area and the background area; at the same time, a weight is set for each area, the weight of the lip area is greater than that of the face area, and the weight of the face area is greater than that of the background area.

6. The voice-driven facial animation generation method according to claim 1, 2, 3, 4, or 5, characterized in that: In step S4, based on the architecture improvement of the generative adversarial network, a speech-to-face generative adversarial network A2FGAN is proposed to obtain the lip synchronization effect of facial animation. The implementation method includes the following steps: 1) A2FGAN modeling, including using speech information to guide facial animation generation based on a reference image based on a speech-to-face generative adversarial network (A2FGAN); the speech-to-face generative adversarial network (A2FGAN) includes a generator network and two discriminator networks. The generator network consists of a face encoder / decoder module, an STN module, and a CBAM module. The two discriminator networks consist of a frame discriminator module and a lip synchronization discriminator module, respectively. 2) A2FGAN training: receiving input audio features and reference images, training the speech-to-face generative adversarial network (A2FGAN), optimizing model parameters through backpropagation, and obtaining lip-synced facial animation image frames.

7. The voice-driven facial animation generation method according to claim 6, wherein: When training the discriminator of A2FGAN, the lip synchronization discriminator module is first trained separately. Then, based on the trained lip synchronization facial animation images, the frame discriminator module is trained. Finally, the facial animation image frames with lip synchronization and maintaining the true value image effect are obtained.

8. A voice-driven facial animation generation system, characterized by: Used to implement a voice-driven facial animation generation method as described in any one of claims 1 to 7.

9. The voice-driven facial animation generation system according to claim 8, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the voice-driven facial animation generation method according to any one of claims 1 to 7.

Images