Speaking face generation method, system and device based on multi-frame lip reference and lip optical flow module and medium
Through the multi-frame lip reference and lip optical flow module combined with the audio-perceived cross attention module, the method of generating a conversation face is optimized, which solves the problem of not fully considering lip details in the prior art, and achieves a more realistic and synchronous The effect of speaking face generation.
Patent Information
- Application Number
- CN202411134657.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-12-03
AI Technical Summary
When generating audio and video synchronization and character image quality, the existing speaking face generation method fails to fully consider the lip details of the reference character, making it difficult to achieve real visualization results.
The multi-frame lip reference and lip optical flow module are adopted, combined with the audio-perceptual cross attention module, the alignment model and rendering model are designed, the model weight is updated through backpropagation, and the generation results are optimized to ensure the alignment and synchronization of lip features.
The generated speaking face images perform excellently in audio and video synchronization and lip details authenticity, significantly improving the overall speaking face fidelity.
Smart Images

Figure CN119068526A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of multimodal image translation, and in particular relates to a method, system, device and medium for generating a speaking face based on a multi-frame lip reference and a lip optical flow module. Background Art
[0002] Audio-driven talking face video generation aims to generate high-quality talking faces synchronized with the input speech. It has important value in a wide range of applications such as visual dubbing, digital assistants, and animated films, and has received widespread attention from industry and academia in the past few decades.
[0003] Many methods for generating talking faces have been proposed. For example, Wav2Lip uses an encoder-decoder based structure to synthesize talking face videos under the guidance of a lip synchronization discriminator; PC-AVS classifies talking faces into feature spaces of speech content, head posture, and identity, and assembles these features to generate talking face videos; IPLAP generates talking face videos by aligning reference face images using a two-stage model structure based on intermediate representation.
[0004] Although the above methods can generate more reasonable results under the guidance of reference face images, they only focus on audio and video synchronization and the quality of the generated character images, and do not fully consider the lip detail information of the reference character, which makes it difficult to produce realistic visualization results. Therefore, the present invention provides a speaking face generation method, system, device and medium based on multi-frame lip reference and lip optical flow module, which enables the present invention to generate more realistic lip details while ensuring audio and video synchronization and character image quality, thereby making the overall speaking face more realistic. Summary of the invention
[0005] The present invention aims at the deficiencies in the prior art, focuses on generating speaker face and lip details synchronized with audio, and provides a speaking face generation method, system, device and medium based on multi-frame lip reference and lip optical flow module.
[0006] In a first aspect, the present invention provides a method for generating a speaking face based on a multi-frame lip reference and a lip optical flow module, comprising the following steps:
[0007] Step 1: Extract the audio information of the face video and crop the facial key points, and standardize the facial key points and face video frames.
[0008] Step 2: Design the alignment model and rendering model. The rendering model generates faces under the guidance of the alignment model results.
[0009] Step 3: Train the model and construct the discriminator, further optimize the final result, and back-propagate to update the weights of the alignment model, rendering model, and discriminator.
[0010] Step 4: Repeat step 3 until the preset number of iterations is reached. All face images are traversed in each round and the model parameters are saved. After each round of training, the indicators of the generated face images are calculated using an independent validation set.
[0011] Step 5: Select a model parameter with the best indicator, import it, and use the unmatched video, audio, and facial key points to generate the corresponding speaking face video.
[0012] In a second aspect, the present invention provides a speaking face generation system based on multi-frame lip references and a lip optical flow module, comprising:
[0013] Data extraction unit: used to extract audio information of face video and crop facial key points, and perform data standardization on facial key points and face video frames.
[0014] Design unit: used to design the alignment model and rendering model. The rendering model generates the final face under the guidance of the alignment model results.
[0015] Training unit: used to train the model and construct the discriminator, further optimize the final result, and back-propagate to update the weights of the alignment model, rendering model, and discriminator.
[0016] Traversal unit: used to repeatedly execute step 3 until the preset number of iterations is reached, traversing all face images in each round and saving model parameters, and using an independent validation set to calculate the indicators of the generated face images after each round of training.
[0017] Select generation unit: It is used to select a model parameter with the best indicator, import it and use the unmatched video, audio and facial key points to generate the corresponding speaking face video.
[0018] In a third aspect, the present invention provides a speaking face generation device based on multi-frame lip references and a lip optical flow module, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned speaking face generation method based on multi-frame lip references and a lip optical flow module is implemented.
[0019] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program for executing the above-mentioned method for generating a speaking face based on a multi-frame lip reference and a lip optical flow module.
[0020] Beneficial effects of the present invention:
[0021] The present invention introduces a multi-frame lip reference module and a lip optical flow module, fully considers the lip details of the reference character, and inputs aligned and non-aligned lip features into the model. At the same time, the present invention introduces an audio-aware cross-attention module, combines image and audio features, and generates more realistic and natural speaking face results, showing excellent generation effects on multiple face video datasets. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 Generating a structural diagram of the speaking face of the alignment model and the rendering model of the embodiment of the present application;
[0023] Figure 2 This is a flow chart of model training for an embodiment of the present application;
[0024] Figure 3 This is a comparison chart of the results of the embodiment of the present application and other methods. DETAILED DESCRIPTION
[0025] The present invention is further described below with reference to the accompanying drawings and specific examples:
[0026] The present application provides a method for generating a speaking face based on a multi-frame lip reference and a lip optical flow module, comprising the following steps:
[0027] Step 1: Extract the audio information of the face video and crop the facial key points, and standardize the facial key points and face video frames.
[0028] Step 2: Design the alignment model and rendering model. The rendering model generates faces under the guidance of the alignment model results.
[0029] Step 3: Train the model and construct the discriminator, further optimize the final result, and back-propagate to update the weights of the alignment model, rendering model, and discriminator.
[0030] Step 4: Repeat step 3 until the preset number of iterations is reached. All face images are traversed in each round and the model parameters are saved. After each round of training, the indicators of the generated face images are calculated using an independent validation set.
[0031] Step 5: Select a model parameter with the best indicator, import it, and use the unmatched video, audio, and facial key points to generate the corresponding speaking face video.
[0032] Further, the step 1 comprises:
[0033] Step 1-1: Extract the audio information from the face video and process it into a Mel-spectrogram.
[0034] Step 1-2: Recognize each face frame in the face video, extract key points of the face and crop it.
[0035] Step 1-3: Save the facial key points and facial video frames and perform data standardization on the images.
[0036] In particular, in steps 1-3:
[0037] Saving the face video as frame-by-frame pictures; and
[0038] Normalize the image data to ensure that the pixel values are in the range [0,1].
[0039] Further, in step 2:
[0040] The alignment model mainly includes a lip optical flow module, which is used to warp and align the lip features of the reference image to the lip features of the generated result.
[0041] The rendering model mainly includes a multi-frame lip reference module and an audio-aware cross-attention module, which are respectively used to combine multi-frame lip reference images with audio information.
[0042] Furthermore, the training model in step 3 is specifically:
[0043] For the alignment model, the reference face, the occluded real face image and the predicted face key points are input into the alignment model. After obtaining the aligned reference face image, the VGG loss is calculated and input into the rendering model as the guiding face image.
[0044] For the rendering model, the occluded real face image, reference lip image and predicted facial key points are input into the rendering model, and the final result is generated by combining the aligned reference face of the alignment model, and the VGG loss and GAN loss are calculated. After all losses are obtained, the alignment model and rendering model parameters are updated by backpropagation.
[0045] Example 1
[0046] This preferred example provides a method for generating a speaking face based on a multi-frame lip reference and a lip optical flow module, which specifically includes:
[0047] Step 1: Extract the audio information of the face video and crop the facial key points, and standardize the facial key points and face video frames.
[0048] Step 1-1: Prepare a dataset containing a certain number of face videos, extract the audio from the video, and process it into a 80*16 Mel-spectrogram at a sampling rate of 16kHz.
[0049] Step 1-2: Use MediaPipe to identify faces in each frame of the video, extract facial key points and crop them.
[0050] Step 1-3: Save the facial key points and face video frames in .png format, and use the resize function to standardize the image to 128*128 pixels. Normalize the image data to ensure that the pixel value is in the range of [0,1].
[0051] Step 2: In order to consider both the non-aligned lip reference features and the aligned lip reference features and combine the audio information, this preferred embodiment designs an alignment model (M a ) and the rendering model (M r ), the model includes a multi-frame lip reference module, a lip optical flow module, and an audio-aware cross-attention module, such as Figure 1 shown.
[0052] The alignment model mainly includes a lip optical flow module.
[0053] First, use two encoders E r and E m Extract reference image I respectively r And the occluded real face image Next, the two features are concatenated in the channel dimension and input into the feature fusion module composed of Resblock. Subsequently, the facial key points L are injected into the alignment model through the SPADE layer to guide the prediction of optical flow. The function of the alignment model can be expressed as:
[0054]
[0055] Among them, F i represents the obtained lip optical flow.
[0056] At the same time, in order to aggregate the visual features of multiple reference images distorted by optical flow, this preferred embodiment adds an output layer to the last layer of the alignment model to predict a 2D weight for each reference image. The calculation of aligned images can be expressed as:
[0057]
[0058] in Represents the aligned image after aggregation.
[0059] In order to further guide the high-dimensional features of the rendering model, this preferred embodiment converts the encoder E r The features extracted at two spatial resolutions and Perform warp alignment. The calculation of alignment features can be expressed as:
[0060]
[0061] in Represents the aligned high-dimensional features after aggregation.
[0062] Finally, in order to further reduce the impact of chromatic aberration and artifacts caused by large head movements, this preferred embodiment splices the occluded real face image features and the distorted features on the channel to jointly guide the generation of the rendering model. Specifically, it can be expressed as:
[0063]
[0064] in, and Indicated by the encoder E m The extracted features of the occluded real face image at two spatial resolutions, I c and Respectively represent the joint guidance image and features after channel splicing. Finally, I c and It is input into the decoder of the rendering model through the SPADE layer.
[0065] The rendering model mainly includes a multi-frame lip reference module and an audio perception cross-attention module.
[0066] For the multi-frame lip reference module, two encoders are first used to extract the occluded real face image features. Lip features compared to reference Then, the facial image features and the reference lip features are spliced along the channel and input into the feature fusion module F composed of Resblock to obtain a set of fused features. Then the group of features is averaged to obtain the multi-frame lip reference features of the person's identity. Specifically, it can be expressed as:
[0067]
[0068]
[0069] in This is the final multi-frame lip reference feature.
[0070] Next, the preferred embodiment converts the Mel spectrum m corresponding to the video frame into t The audio-aware cross-attention module is integrated with the image features to further guide the generation of corresponding lip shapes synchronized with the audio. Specifically, it can be expressed as:
[0071]
[0072] in, Indicates that the audio encoder E a The obtained audio features of the corresponding t frames. q , W k and W v There are three different 1×1 convolutional layers.
[0073] Finally, the obtained image feature f g and the feature I obtained from the alignment model c , The decoder generates the final result The function of rendering the model can be specifically expressed as:
[0074]
[0075] After building the alignment model and rendering model, we use the structure-aware loss L based on VGG19. w , by comparing the real face I t Image after alignment with the warp To get a more accurate playing field and weights.
[0076]
[0077] where φ i represents the activation output of the i-th layer in the VGG-19 network.
[0078] Further, for the rendering model, first use the same W The reconstruction loss L is similar to r and style loss L s , the generated results are compared by the reconstruction loss and style loss and real face I t In order to achieve better generation effect.
[0079]
[0080] in Represents the output φ from the activation layer i The resulting Gram matrix.
[0081] In addition, in order to make the rendering result more realistic, this preferred example also introduces the GAN loss L g And feature matching loss L f .
[0082] Finally, the loss function is summarized as follows:
[0083] L=λw L w +λ r L r +λ s L s +λ g L g +λ f L f
[0084] In particular, in this preferred embodiment, the setting parameters are: w =2.5,λ r =4,λ s =1000,λ g =0.25,λ f =2.5.
[0085] Step 3: The model training process is as follows Figure 2 As shown in the figure, the left side shows the alignment model training process, and the right side shows the rendering model training process. Green represents the alignment model, and yellow represents the loss calculation result of the alignment model; red represents the rendering model, and blue represents the loss calculation result of the rendering model. The model adopts a joint training method.
[0086] For the alignment model, first the occluded real face image and multiple frames of reference faces And the corresponding facial key points Input into the model, align the optical flow results obtained by the lip optical flow module in the model, and then use the distortion method to distort the reference image through the optical flow to the approximate reference image to be generated Then, the features of real images and generated images are extracted through the pre-trained VGG19 network and L1 loss is performed.
[0087] For the rendering model, first use the occluded real face image and the reference lip image And the predicted facial key points obtained through the pre-trained model The lip reference module of the rendering model is input. After the fused features are obtained, the audio features are fused with the image features through the audio perception cross attention module. Then, in the decoder of the rendering model, the results obtained by the alignment model are used to guide the generation of the rendering model to obtain the final result. Then the final result and the real image are input into the pre-trained VGG19 network to extract features, and the L1 loss between the features and the L1 loss of the Gram matrix are calculated.
[0088] In addition, a discriminator is constructed, and the final result is further optimized using GAN loss and feature matching loss. After obtaining the above loss results, backpropagation is used to update the weights of the alignment model, rendering model, and discriminator.
[0089] Step 4: Repeat step 3 until the preset number of iterations is reached (300 rounds in this preferred example), and traverse all face images in each round. At the end of each round of training, save the parameters of the model. At the same time, use an independent validation set to calculate the index of the generated image after each round of training.
[0090] Furthermore, the indicators include FID indicator, SSIM indicator and PSNR indicator.
[0091] Among them, the evaluation index Fréchet Inception Distance (FID) is a commonly used indicator to measure the quality of generated images. It evaluates by measuring the Wasserstein distance between the feature space representation of the real image and the output of the generated model. The calculation formula of FID is as follows:
[0092]
[0093] Evaluation index Structural Similarity (SSIM), that is, structural similarity, is an indicator to measure the similarity between two images. The SSIM algorithm is mainly used to detect the similarity between two images of the same size, or to detect the degree of image distortion. The calculation formula of SSIM is as follows:
[0094] SSIM(x,g)=[l(x,g)] α [c(x,g)] β [s(x,g)] γ
[0095] The evaluation index PSNR stands for Peak Signal to Noise Ratio, which is an objective standard for evaluating images. It is often used to calculate the visual error between two images, such as the difference between the image generated by the generative network and the real image. The calculation formula of PSNR is as follows:
[0096]
[0097] Step 5: Select a model parameter with the best FID index, SSIM index and PSNR index, load the selected best parameters into the rendering model and the alignment model, and generate the corresponding speaking face video by inputting the original video, the audio that does not match the original video, and the facial key points obtained by the pre-trained model.
[0098] Example 2
[0099] Figure 3 The comparison of the generation results of this preferred example and other existing methods is shown.
[0100] Among them, the last column is the real faces of different characters, the second to last column is the result generated by this preferred example, and the remaining columns are the results of the existing method.
[0101] The specific details of this embodiment include:
[0102] (1) This embodiment uses the LRS2 dataset as the video dataset.
[0103] It should be noted that LRS2 is a video slice dataset, which contains 48164 video clips, including 45839 training videos, 1082 verification videos and 1243 test videos. This embodiment samples 45 videos from the test set of the dataset as the test and verification of the results.
[0104] For each frame of the video, this embodiment uses center cropping to crop the video frame to a size of 128*128 pixels, and uses a window size of 800 and a frame shift of 200 for the audio, and processes it into a Mel-spectrogram of size 80*16 at a sampling rate of 16 kHz.
[0105] (2) For the alignment model, the number of reference images is 3 during training and one fifth of the original video length during inference. For the rendering model, the number of reference lips is set to 5. All tests are performed on 4 L40 images, with a training batch size of 72, training for approximately 72 hours, and an iteration period of 300 epochs.
[0106] (3) This embodiment uses the training parameters of the 300th epoch for image testing. On the LRS2 dataset, the PSNR index of this embodiment is 32.93, the SSIM index is 0.9261, the LPIPS index is 0.0355, and the FID index is 9.89.
[0107] In addition, in order to further prove that the present application has excellent performance in audio and video synchronization, this embodiment also adds two indicators, LipLMD and LSE-D, which are 1.128 and 6.97 respectively.
[0108] In order to further verify the generalization of this embodiment, the embodiment of the present application also uses the LRW dataset to test the model trained on the LRS2 dataset. As shown in Table 1, the PSNR, SSIM, LPIPS, and FID indicators of this embodiment are 30.88, 0.9327, 0.0356, and 15.05, respectively, all exceeding the existing methods. In terms of the two indicators of LipLMD and LSE-D, the results of this embodiment are also close to the existing methods, which are 0.985 and 8.14, respectively. The test results show that this embodiment can be well generalized to the identity of people that do not appear in the dataset.
[0109] Table 1 shows the comparison of the performance indicators of the embodiments of the present application and the existing methods. The comparison results show that the effects of the embodiments of the present application are generally better than those of the previous methods.
[0110] Table 1 Comparison of indicators between Example 2 and existing methods
[0111]
[0112]
[0113] The embodiment of the present application also provides a system for generating a speaking face based on a multi-frame lip reference and a lip optical flow module, and the method for generating a speaking face based on a multi-frame lip reference and a lip optical flow module described in the preferred embodiment includes:
[0114] Data extraction unit: used to extract audio information of face video and crop facial key points, and perform data standardization on facial key points and face video frames.
[0115] Design unit: used to design the alignment model and rendering model. The rendering model generates the final face under the guidance of the alignment model results.
[0116] Training unit: used to train the model and construct the discriminator, further optimize the final result, and back-propagate to update the weights of the alignment model, rendering model, and discriminator.
[0117] Traversal unit: used to repeatedly execute step 3 until the preset number of iterations is reached, traversing all face images in each round and saving model parameters, and using an independent validation set to calculate the indicators of the generated face images after each round of training.
[0118] Select generation unit: It is used to select a model parameter with the best indicator, import it and use the unmatched video, audio and facial key points to generate the corresponding speaking face video.
[0119] At the same time, the embodiment of the present application discloses a speaking face generation device based on multi-frame lip reference and lip optical flow module, including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the intelligent recommendation method for hospital registration departments based on text mining technology when executing the computer program, wherein the memory may include a memory, such as a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk memory, etc.; the processor, the network interface, and the memory are interconnected through an internal bus, and the internal bus may be an industrial standard architecture bus, a peripheral component interconnection standard bus, an extended industrial standard structure bus, etc., and the bus may be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs, and specifically, the program may include a program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.
[0120] The embodiment of the present application also discloses a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of a method for generating a speaking face based on a multi-frame lip reference and a lip optical flow module are implemented. Specifically, the computer-readable storage medium includes, but is not limited to, for example, a volatile memory and / or a non-volatile memory. The volatile memory may include a random access memory (RAM) and / or a cache memory, etc. The non-volatile memory may include a read-only memory (ROM), a hard disk, a flash memory, an optical disk, a magnetic disk, etc.
[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.
Claims
1. A speaking face generation method based on multi-frame lip reference and lip optical flow module, It is characterized in that The following steps are involved: Step 1: Extract the audio information of the face video and crop the face key points, and standardize the face key points and face video frames; Step 2: Design the alignment model and rendering model. The rendering model generates faces under the guidance of the alignment model results. Step 3: Train the model and construct the discriminator, further optimize the final result, and back-propagate to update the weights of the alignment model, rendering model, and discriminator; Step 4: Repeat step 3 until the preset number of iterations is reached. All face images are traversed in each round and the model parameters are saved. After each round of training, the indicators of the generated face images are calculated using an independent validation set. Step 5: Select a model parameter with the best indicator, import it, and use the unmatched video, audio, and facial key points to generate the corresponding speaking face video.
2. According to claim 1, a method for generating a speaking face based on a multi-frame lip reference and a lip optical flow module, It is characterized in that The step 1 comprises: Step 1-1: extract the audio information from the face video and process it into a Mel-spectrogram; Step 1-2: Recognize each face frame in the face video, extract key points of the face and crop it; Step 1-3: Save the facial key points and facial video frames and perform data standardization on the images.
3. The method for generating a speaking face based on a multi-frame lip reference and a lip optical flow module according to claim 2, It is characterized in that In the steps 1-3: Saving the face video as frame-by-frame pictures; and Normalize the image data to ensure that the pixel values are in the range [0,1].
4. The method for generating a speaking face based on a multi-frame lip reference and a lip optical flow module according to claim 1, It is characterized in that In step 2: The alignment model mainly includes a lip optical flow module, which is used to warp and align the lip features of the reference image to the lip features of the generated result; The rendering model mainly includes a multi-frame lip reference module and an audio-aware cross-attention module, which are respectively used to combine multi-frame lip reference images with audio information.
5. The method for generating a speaking face based on a multi-frame lip reference and a lip optical flow module according to claim 1, It is characterized in that The training model in step 3 is specifically: For the alignment model, the reference face, the occluded real face image and the predicted face key points are input into the alignment model. After obtaining the aligned reference face image, the VGG loss is calculated and input into the rendering model as the guiding face image. For the rendering model, the occluded real face image, reference lip image and predicted facial key points are input into the rendering model, and the final result is generated by combining the aligned reference face of the alignment model, and the VGG loss and GAN loss are calculated. After all losses are obtained, the alignment model and rendering model parameters are updated by backpropagation.
6. The method for generating a speaking face based on a multi-frame lip reference and a lip optical flow module according to claim 5, Features: The rendering model is based on a rendering generation model of a multi-frame reference lip module and an audio-aware cross-attention module, and generates character features and lip details by utilizing multi-reference lip features and corresponding audio features.
7. The method for generating a speaking face based on a multi-frame lip reference and a lip optical flow module according to claim 1, It is characterized in that In the step 4: the indicators include the FID indicator, the SSIM indicator and the PSNR indicator.
8. A speaking face generation system based on multi-frame lip reference and lip optical flow module, It is characterized in that include: Data extraction unit: used to extract audio information of face video and crop key points of face, and perform data standardization on key points of face and face video frames; Design unit: used to design the alignment model and rendering model. The rendering model generates the final face under the guidance of the alignment model results. Training unit: used to train the model and construct the discriminator, further optimize the final result, and back-propagate to update the weights of the alignment model, rendering model, and discriminator; Traversal unit: used to repeatedly execute step 3 until the preset number of iterations is reached, traversing all face images in each round and saving model parameters, and using an independent validation set to calculate the indicators of the generated face images after each round of training; Select generation unit: It is used to select a model parameter with the best indicator, import it and use the unmatched video, audio and facial key points to generate the corresponding speaking face video.
9. A speaking face generation device based on multi-frame lip reference and lip optical flow module, It is characterized in that include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for generating a speaking face based on a multi-frame lip reference and a lip optical flow module as described in any one of claims 1 to 8 is implemented.
10. A computer-readable storage medium, It is characterized in that The storage medium stores a computer program, and the computer program is used to execute the method for generating a speaking face based on a multi-frame lip reference and a lip optical flow module as described in any one of claims 1 to 8.
Citation Information
Cited By
Digital human video generation quality control method and device
CN122476248A
