Voice driving method and system for improving video quality

Abstract

The invention discloses a voice driving method and system for improving video quality, and the method comprises the steps: carrying out the preprocessing of a face speaking video containing a Chinese video, and obtaining a face video data set; pre-training a lip shape synchronous discriminator by using the face video data set; according to the method, a face video data set is used as a generator, a loss function and a network architecture of the generator are reconstructed, the generator is trained by using the face video data set, a pre-trained lip shape synchronization discriminator is introduced in a training process, and the generator and the lip shape synchronization discriminator are alternately optimized to improve a training effect, so that the trained generator generates a high-quality and high-synchronization-rate voice-driven face video. According to the method, the quality problems such as blurred lip-shaped area pictures can be effectively improved, and a face video with higher definition is generated.

Description

Technical Field

[0001] The present invention relates to the field of computer information technology, and in particular to a voice-driven method and system for improving video quality. Background Art

[0002] Voice-driven face generation is mainly based on deep learning technology. This method constructs a neural network that can encode the features of a certain video and audio input, and continuously learns the video features and audio features to generate a new speaking face video that matches the audio. This method can be effectively applied in the live broadcast industry of online celebrities. The research and development of virtual anchors can bring great convenience to enterprises and human resources. In practical applications, only text information or voice information needs to be input to synthesize new video applications through the synthesis system.

[0003] In recent years, the generation technology of face videos driven by voice has penetrated into fields including but not limited to virtual cartoon characters, virtual anchors selling goods, intelligent face recognition, video face-changing special effects, and beauty applications. At present, the specific implementation methods of face video generation driven by voice are mainly divided into two categories: the generation method based on facial feature points to the speaking face, and the generation method based on facial images to the speaking face.

[0004] The face video generation method based on facial feature points uses recurrent neural network learning. This method maps the original audio features to the mouth shape, synthesizes high-quality mouth textures, and generates high-quality synchronized facial videos of the mouth shape. However, this method requires a large number of data sets of specific people for training in order to generate realistic face videos of the specific person. The generation method based on face images to talking faces uses an encoder-decoder convolutional neural network model, which uses a joint embedding of face and audio. Its features are extracted by the identity encoder and audio encoder respectively, and the face image generation decoder is used to reconstruct the character features and audio features to generate a video frame of the talking face. This method only calculates the L1 loss for each video frame, resulting in a blurry output image and poor visual quality.

[0005] In general, in the task of speech-driven face video generation, the current main research content focuses on the synchronization accuracy of audio and face lip shape, and the naturalness of facial expressions. Traditional methods only consider it as the generation of multiple single-frame videos, without considering the temporal relationship between video frames, which leads to the incoherence of the generated video, and is prone to problems such as pixel jitter and poor visual perception. In addition, traditional speech-driven face video methods are prone to problems such as blurred images, poor detail quality, and obvious rectangular frames in the mouth reconstruction area. Therefore, how to provide a multi-scale speech-driven face video method so that the mouth area of ​​the generated face video has higher clarity and can effectively improve the problems of a large number of green noise points, slight artifacts, matrix contours, etc. in the lip area has become a technical problem to be solved. Summary of the invention

[0006] The technical problem to be solved by the present invention is as follows: In view of the above-mentioned problems in the prior art, a voice-driven method and system for improving video quality are provided, which can generate a face video with higher definition.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A voice-driven method for improving video quality comprises the following steps:

[0009] Preprocess the face speaking videos including Chinese videos to obtain a face video dataset;

[0010] Pre-training lip sync discriminator using face video dataset;

[0011] The loss function and network architecture of the generator are reconstructed, and the generator is trained using a face video dataset. During the training process, a pre-trained lip synchronization discriminator is introduced. The generator and lip synchronization discriminator are alternately optimized to improve the training effect, so that the trained generator can generate high-quality and high-synchronization speech-driven face videos.

[0012] Furthermore, when preprocessing the face speech video containing the Chinese video, it includes:

[0013] Eliminate low-quality videos, unify the video frame rate, denoise videos with background noise, and cut out irrelevant clips in the video;

[0014] Divide the video into segments, delete the segments that do not meet the required length, and save the video segments of the same person in the same face folder;

[0015] Use the trained audio-video synchronization detection model to detect the audio and video synchronization of each video clip, and delete the video clips with audio and video offset information or face size that does not meet the requirements;

[0016] Use the trained face detector to detect and segment the face images in each video clip.

[0017] Furthermore, the lip synchronization discriminator includes a facial encoder, which uses stacked convolution modules to extract features from an input feature map, wherein convolution module 1 uses a 7x7 convolution kernel to increase the number of channels, and the size of the feature map remains unchanged; convolution module 2 reduces the width of the feature map by 1 / 2 and doubles the number of channels; convolution modules 3 to 8 reduce the size of the feature map by 1 / 2 module by module, and the number of channels doubles layer by layer until convolution module 7 remains unchanged; convolution modules 3 to 8 use stacked 3x3 convolution kernels to capture higher-level and global features; convolution module 9 further scales the feature map by padding with 0; and the last convolution layer compresses the number of channels and applies Sigmoid as the activation function to output a binary classification result.

[0018] Furthermore, when the face video dataset is used to pre-train the lip sync discriminator, the loss function of the lip sync discriminator uses the cosine similarity loss function with binary cross entropy. At the same time, the lip sync discriminator uses the Relu activation function to calculate the dot product between the video and speech embeddings to obtain the input audio-video synchronization rate P of the face sample. sync , the expression is as follows:

[0019]

[0020] Where v and s represent the feature vectors of video and audio respectively, P sync Represents the probability that the video and audio are synchronized in this time series.

[0021] Furthermore, when reconstructing the generator's loss function and network architecture, the reconstructed generator loss function is as follows:

[0022] L total =α1L sync +α2L perceptual +α3L ms +(1-α1-α2-α3)L1

[0023] Among them, L sync represents the synchronization loss function, L perceptual represents the perceptual loss function, L1 represents the reconstruction loss function, and L ms is the multi-scale structural similarity loss function, and α1, α2, and α3 are hyperparameter weights.

[0024] Furthermore, the multi-scale structural similarity loss function decomposes the input face image into different scales and calculates the SSIM loss at each scale. The expression is as follows:

[0025]

[0026] Among them, l M (x, y) represents the contrast of the original image at the highest scale, c j (x, y) represents the structural similarity of the original image at the jth iteration scale, s j (x, y) represents the brightness similarity of the original image at the jth iteration scale, α M , β j , γ j Represents an index used to adjust the relative importance of different components.

[0027] Furthermore, when reconstructing the loss function and network architecture of the generator, the reconstructed generator network architecture includes a facial image encoder, which uses stacked convolutional layer sub-modules to extract features from the spliced ​​image, wherein sub-module 1 uses a 7x7 convolution kernel to increase the number of channels; sub-modules 2 to 8 gradually reduce the spatial size by 2 times and increase the number of channels by 2 times; sub-module 9 further scales the feature map by padding with 0, and sub-modules 2 to 8 respectively use stacked 3x3 convolution kernels and introduce residual connections to help gradient propagation and learn complex features.

[0028] Furthermore, when reconstructing the loss function and network architecture of the generator, the reconstructed generator network architecture also includes a decoder, which uses stacked deconvolution module layers to gradually upsample the input feature map to restore the size of the original facial image, wherein deconvolution module layer 1 to deconvolution module layer 7 first upsample the feature map through the deconvolution layer, doubles the size of the feature map, reduces the number of channels, and then processes the feature map through stacked 3x3 convolution layers, and the output module of the decoder further reduces the number of channels and limits the output to a specified range through the Tanh activation function.

[0029] Furthermore, the generator is trained using a face video dataset, a pre-trained lip sync discriminator is introduced into the training process, and the generator and the lip sync discriminator are alternately optimized to improve the training effect, including:

[0030] In the current training iteration, freeze the parameters of the lip sync discriminator and train the generator. According to the defined loss function, calculate the gradients of the trainable parameters in the generator network, and use the back-propagation algorithm to propagate the gradients back to the generator network to update the network parameters to minimize the overall loss function.

[0031] In the next training iteration, the parameters of the generator are frozen and the lip sync discriminator is trained. The parameters of the lip sync discriminator are updated by calculating the gradient of the loss function. During the back-propagation process, the prediction results of the lip sync discriminator are compared with the true labels to optimize the ability of the lip sync discriminator so that it can better distinguish the authenticity of the generated lip sync videos.

[0032] The present invention also proposes a voice-driven system for improving video quality, comprising a microprocessor and a computer-readable storage medium connected to each other, wherein the microprocessor is programmed or configured to execute any one of the voice-driven methods for improving video quality.

[0033] Compared with the prior art, the advantages of the present invention are:

[0034] The present invention introduces Chinese videos to address the problem that the language coverage of current face speaking videos lacks Chinese, resulting in insufficient adaptation and fitting of Chinese videos. Based on this data set, the lip synchronization discriminator of Wav2lip is retrained to improve the discrimination rate of the discriminator for lip synchronization. In addition, a multi-scale structural similarity loss is introduced in the generator of Wav2lip, and the loss function is reconstructed to avoid the generator focusing only on the global structure and ignoring details or local structures. At the same time, the network input layer of the generator and the lip synchronization discriminator is broadened, and multiple 3x3 convolution kernels are stacked to optimize the network architecture. Compared with the benchmark method, the present invention can effectively improve the quality problems such as blurred lip area and generate a face video with higher definition. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.

[0036] Figure 2 FIG. 4 is a schematic diagram of a network architecture of a facial encoder in a lip synchronization discriminator according to an embodiment of the present invention.

[0037] Figure 3 Schematic diagram of the network architecture of the facial encoder of the generator according to an embodiment of the present invention.

[0038] Figure 4 Schematic diagram of the network architecture of the decoder of the generator according to an embodiment of the present invention.

[0039] Figure 5 The figure is a flow chart of training a generator and introducing a pre-trained lip sync discriminator during the training process in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0041] Before introducing the specific embodiments of the present invention, the relevant concepts are first explained:

[0042] Wav2Lip: Wav2Lip is a deep learning model based on generative adversarial networks (GANs), which is mainly used to generate facial animations that match audio waveforms, especially lip movements. The model consists of the following two core components:

[0043] Generator: The task of the generator is to receive audio information as input and then generate the corresponding facial animation sequence, where special attention is paid to the movement of the lips to achieve synchronization with the audio.

[0044] Lip Sync Discriminator: This discriminator determines the degree of lip sync and audio synchronization between the generated facial animation and the real facial animation. It guides the generator to produce more accurate results by analyzing the input continuous video frames and audio signals to determine whether the two are synchronized.

[0045] During the training process, the generator and the lip sync discriminator compete with each other, the generator tries to generate animations as realistic as possible to fool the discriminator, while the discriminator tries to accurately distinguish between the generated animations and real animations. Through this adversarial process, the model gradually improves its ability to generate realistic lip sync animations.

[0046] Embodiment 1

[0047] This embodiment proposes a voice-driven method for improving video quality. Based on the traditional public Wav2lip algorithm, the input layer of the network is broadened and multiple 3X3 convolution kernels are stacked. The newly added convolution layer can help the generator network better capture the various features of the image and further improve the performance of the generated image. In order to allow the convolution layer to obtain a larger receptive field, the pooling layer is not added to the network architecture. Residual connections are introduced to help gradient propagation and learn complex features. Through the above method, it is possible to avoid the situation where the generator only focuses on the global structure and ignores the details or local structure, so that the generated image can better maintain the structural similarity with the real image, thereby producing a clearer and more natural video output.

[0048] like Figure 1 As shown, the method of this embodiment includes the following steps:

[0049] S1) obtaining a human face speaking video, and introducing a Chinese video into the human face speaking video;

[0050] S2) preprocessing the face speaking videos including the Chinese videos to obtain a face video dataset;

[0051] S3) pre-training a lip sync discriminator using a face video dataset;

[0052] S4) reconstructing the loss function and network architecture of the generator;

[0053] S5) Using a face video dataset to train a generator, introducing a pre-trained lip synchronization discriminator during the training process, and alternately optimizing the generator and the lip synchronization discriminator to improve the training effect, so that the trained generator can generate high-quality and high-synchronization rate speech-driven face videos.

[0054] Each step is described below.

[0055] Step S1 of this embodiment obtains a large number of face speaking videos with different backgrounds, different postures, and different lighting conditions, and introduces Chinese videos into the collected face speaking videos, thereby overcoming the problem that the language coverage of current face speaking videos lacks Chinese, resulting in insufficient adaptation of the voice-driven method to Chinese videos. The specific process is as follows:

[0056] S1.1) Experimental analysis of the currently more recognized and widely used public datasets Avspeech, CelebA, LFW, and FDDB;

[0057] S1.2) Through experimental analysis, it is found that the annotation quality of the CelebA dataset varies, and some attribute annotations contain errors and noise; the image quality and angle of the LFW dataset vary greatly, which is not ideal for more sophisticated face analysis tasks; the scale of the FDDB dataset is relatively small, which is not sufficient for tasks that require large-scale data. The Avspeech dataset has the following advantages and is recognized:

[0058] (1) Avspeech is a large dataset of human face speech videos, containing more than 10,000 hours of audio and video data. This large-scale dataset can provide better training and evaluation capabilities for speech-driven face video methods.

[0059] (2) The Avspeech dataset covers multiple languages, including but not limited to English, French, German, Spanish, Japanese, etc. This multi-language coverage enables researchers to conduct cross-language speech-driven human-video method research and development;

[0060] (3) The Avspeech dataset contains a variety of audio and video data from YouTube, covering a wide range of topics and fields. This diversity of topics makes the dataset more representative and can be used for the study of speech-driven human-video methods in various practical application scenarios;

[0061] (4) The Avspeech dataset has become one of the important resources in the field of speech technology research. Many research methods and projects use the Avspeech dataset as a benchmark for model training, evaluation, and comparison.

[0062] S1.3) The Avspeech dataset still has some shortcomings. Since the Avspeech dataset is extracted from YouTube videos, the quality of the audio and video data may vary depending on factors such as the video screen, recording equipment, and environmental conditions. At the same time, since the Avspeech dataset lacks Chinese data, the trained audio is not highly adaptable to Chinese. To this end, the Avspeech dataset is optimized as follows:

[0063] (1) Manually clean the Avspeech dataset to remove low-quality audio and video;

[0064] (2) We manually recorded some high-quality 4K Chinese facial audio and video with professional models, professional recording scenes, and professional shooting conditions to mix and fill the dataset to ensure that the overall dataset is sufficient in quantity and meets the quality requirements.

[0065] Step S2 of this embodiment preprocesses the optimized Avspeech dataset filled with Chinese videos to achieve better training results and construct a dataset file, which is specifically performed as follows:

[0066] S2.1) Eliminate low-quality videos. Specifically, in this embodiment, the low-quality videos are eliminated manually by eliminating the audio and video with a resolution lower than 1080P, multiple faces, and a small proportion of face size in the Avspeech dataset;

[0067] Then, the video frame rate is unified, the video with background noise is denoised, and irrelevant clips in the video are cut out. Specifically, the Avspeech dataset after removing the low-quality video and the mixed 4K high-definition face video material are processed as follows: (1) Open the video file using OpenCV and obtain the video frame rate (FPS), and use Ffmpeg to convert the video frame rate to 25; (2) De-noise the video with obvious background noise; (3) The video can only have shots of people facing the camera. If there is any discrepancy, irrelevant clips must be cut out;

[0068] S2.2) Divide the video into segments and delete the video segments whose duration does not meet the requirement. Specifically, use FFmpeg to divide the video into multiple segments and delete the videos whose duration is less than 4 seconds;

[0069] Save the video clips of the same person in the same face folder. Specifically, traverse each face folder and video file in the root directory of the dataset, rename the video files in a standardized way, and the video files in each face folder correspond to the same person.

[0070] S2.3) Use the trained audio-video synchronization detection model to detect the audio and video synchronization of each video clip, and delete the video clips with audio and video offset information or face size that does not meet the requirements;

[0071] Specifically, load the trained and exported audio-video synchronization detection model vad_moedl to process the video file, extract the audio information, analyze the audio, detect the audio-video synchronization, and return the audio-video offset information. If the offset information is not obtained or the face size is smaller than the threshold, skip the video file and write its path to the output file; if the obtained offset exceeds the preset range (-3 to 3), write its path to the output file. Check the output file and manually delete the audio and video clips that do not meet the requirements;

[0072] S2.4) Use the trained face detector to detect and cut the face images in each video clip. Specifically, use the face detector pre-trained based on deep learning to detect and cut the face images. Finally, construct the training set, validation set, and test set based on all the pre-processed face speaking videos.

[0073] In step S3 of this embodiment, the original face image and audio after face segmentation are passed through the discriminator to obtain the synchronization loss between the original audio and video, and generate a discriminator for supervising and detecting the lip synchronization of the face video. Specifically, the following process is performed:

[0074] The constructed face video dataset is passed as input through the discriminator, and a lip sync discriminator specifically for this dataset is retrained.

[0075] The lip sync discriminator consists of two parts, a facial encoder and an audio encoder. The facial encoder and audio encoder of the lip sync discriminator convert the input face video image and audio into effective features respectively. The lip sync discriminator distinguishes the difference between the lip sync accuracy of the generated video frames and the real video frames by learning effective feature representations.

[0076] In this embodiment, the facial encoder network architecture diagram of the lip sync discriminator is shown in the attached figure. Figure 2 As shown in the figure, the numbers on the convolution module represent the image length x width x number of channels. The K in the lower right corner represents the convolution kernel size, S represents the step size, and P represents padding. The facial encoder of the lip sync discriminator extracts features from the input feature map through a series of stacked convolution modules without using pooling layers to obtain the largest receptive field.

[0077] like Figure 2As shown in the figure, convolution module 1 uses a 7x7 convolution kernel to increase the number of channels from 3 to 16, and the size of the feature map remains unchanged; convolution module 2 reduces the width of the feature map by 1 / 2 and doubles the number of channels; convolution modules 3 to 8 reduce the size of the feature map by 1 / 2 module by module, and convolution modules 3 to 8 use stacked 3x3 convolution kernels to capture more advanced and global features by stacking multiple convolution layers, where the number of channels doubles layer by layer and remains unchanged after the seventh convolution module; convolution module 9 further scales the feature map to 1024x1x1 by padding with 0; the last convolution layer compresses the number of channels to 1, and applies Sigmoid as the activation function to output the binary classification result.

[0078] In this embodiment, the audio encoder network architecture of the lip sync discriminator is similar to that of the source model Wav2lip, with minor adjustments made to the new input size, which will not be described in detail in this embodiment.

[0079] In this embodiment, the lip sync discriminator uses a new type of loss function - cosine similarity loss function with binary cross entropy. That is, the lip sync discriminator uses the ReLU activation function to calculate the dot product between the video embedding v of the video image valid features and the speech embedding s of the audio valid features. Each face sample will produce a single value between [0,1], indicating the synchronization rate P of the input audio-video. sync , the expression is as follows:

[0080]

[0081] Where v and s represent the feature vectors of video and audio respectively, sync Represents the probability that the video and audio are synchronized in this time series.

[0082] Step S4 of this embodiment introduces a structural similarity loss for enhanced effect into the original Wav2lip generator, reconstructs the generator loss function, broadens the network input layer of the generator and the discriminator, and optimizes the network architecture by stacking multiple 3x3 convolution kernels. Specifically, the following process is performed:

[0083] Considering that the Wav2lip method mainly focuses on the alignment between speech and vision to generate realistic lip animation, the overall picture clarity of this method may be lacking. Compared with the public general method, this embodiment reconstructs the loss function of the generator and reconstructs the overall loss function L total as follows:

[0084] L total =α1L sync +α2L perceptual +α3L ms +(1-α1-α2-α3)L1 (2)

[0085] Among them, L ms is the introduced multi-scale structural similarity loss function, L sync , L perceptual , L1 is the original loss function of Wav2lip, L sync represents the synchronization loss function: compares facial video frames and related audio features to evaluate the correlation between them, obtained by the lip sync discriminator; L perceptual represents the perceptual loss function: measures the perceptual difference between the generated video frames and the real video frames, L1 represents the reconstruction loss function: the generator improves the quality of the generated frames by minimizing the L1 loss between the generated frames and the real frames; α1, α2, and α3 are hyperparameter weights respectively.

[0086] This embodiment converts the multi-scale structural similarity loss to the same order of magnitude as the original Wav2lip loss through the hyperparameter weight α3 corresponding to the multi-scale structural similarity loss function to avoid the gradient explosion phenomenon, and maintains the sum of the total weight coefficients of each loss to 1 through (1-α1-α2-α3).

[0087] The parameter values ​​of α1, α2, and α3 can be obtained through hyperparameter search and multiple pre-training. An excessively large α3 parameter will cause the generated face image to be overfitted and cause extra images to appear. Therefore, it is more appropriate to keep α3 between 0.003 and 0.007, and α2 needs to be appropriately reduced according to the size of α3. If α2 remains at its original size, the generated image will be blurred.

[0088] In this embodiment, the multi-scale structural similarity loss function of the enhanced effect decomposes the input face image into three different scales: contrast, structural similarity, and brightness similarity, and calculates the SSIM loss at each scale. The expression is as follows:

[0089]

[0090] Among them, l M (x, y) represents the contrast of the original image at the highest scale, c j (x, y) represents the structural similarity of the original image at the jth iteration scale, s j (x, y) represents the brightness similarity of the original image at the jth iteration scale, α M , β j , γ j Represents an index used to adjust the relative importance of different components.

[0091] The original image scale is Scale1, and the highest scale ScaleM is obtained after M-1 iterations. At the jth scale, only contrast and structural similarity are calculated, and only brightness similarity l is calculated at ScaleM.M (x, y), the final SSIM index is to connect the results of each scale as shown in Formula 3. M , β j , γ j It is used to adjust the relative importance of different components. In order to simplify the selection of parameters, let all α M =β j =γ j .

[0092] In this embodiment, the contrast calculation formula is as follows:

[0093]

[0094] where μ x , μ y Represents the mean of x and y, where x and y represent two given images, namely the real image and the generated image, and C1 is a constant.

[0095] The formula for calculating structural similarity is as follows:

[0096]

[0097] where σ x , σ y represents the standard deviation of x and y, represents the variance of x, and C2 is a constant.

[0098] The formula for calculating brightness similarity is as follows:

[0099]

[0100] where σ xy is the covariance between x and y.

[0101] The generator of this embodiment is divided into the following three parts:

[0102] The first part is the facial image encoder, whose input consists of the original image with the lower half of the area masked (from the real label GroudTruth) and the image for lip posture reference. This module splices the two sets of images together and inputs them into the facial image encoder, and then outputs the image features. The reconstructed generator facial encoder network architecture is shown in the attached figure. Figure 3As shown in the figure, the numbers on the convolution module represent the image length x width x number of channels. In the note in the lower right corner, K represents the convolution kernel size, S represents the step size, and P represents padding. The input is a 384x384x6 face image, which is extracted through a sequence of nine convolutional layer submodules. Submodule 1 uses a 7x7 convolution kernel to increase the number of channels from 6 to 8; submodules 2 to 8 reduce the spatial size by 2 times and increase the number of channels by 2 times; submodule 9 further scales the feature map by 0 padding to obtain a 1x1 face feature vector. Submodules 2 to 8 stack multiple 3x3 convolution layers and introduce residual connections to help gradient propagation and learn complex features.

[0103] The second part is the audio encoder, which takes as input the audio corresponding to the original image with the lower half masked, and outputs audio features. The audio encoder module structure is similar to the source model Wav2lip, only modified to match the new size.

[0104] The third part is the decoder, which takes in image and audio features, outputs an audio-driven image, and compares the reconstruction loss with the true label GroudTruth. The decoder receives the fused audio and video features, learns how to generate lip movements that match the audio content, and applies them to static images to generate a video frame sequence with lip-sync effects. The decoder network architecture is shown in the attached figure. Figure 4 As shown in the figure, the numbers on the convolution module represent the image length x width x number of channels. In the notes in the lower right corner, K represents the convolution kernel size, S represents the step size, and P represents padding. Through the deconvolution module composed of multiple convolutional neural network layers, the input feature map is gradually upsampled to restore the size of the original facial image. When the feature to be processed passes through convolution module 1, the output feature map size is 1024x1x1; deconvolution module layer 1 to deconvolution module layer 7 first upsample the feature map through the deconvolution layer, the feature map size is doubled, the number of channels is reduced, and then the feature map is processed through the stacked 3x3 convolution layer; finally, when passing through the output module, the number of channels is further reduced to 3, corresponding to the RGB channel of the image, the feature map size is 3x384x384, and the output is limited to the range of [-1,1] through the Tanh activation function.

[0105] In step S5 of this embodiment, after reconstructing the loss function and network architecture of the generator, the generator is retrained based on the face video dataset file in step S2. The overall process is shown in the attached figure. Figure 5 As shown in the figure, it includes the reconstructed generator, loss function and pre-trained lip sync discriminator. During the training process, the generator and lip sync discriminator are optimized alternately to improve the overall training effect. The mutual punishment between the two promotes learning, so as to generate high-quality and high-synchronization speech-driven face videos. Specifically, it includes:

[0106] In the first stage of the current training iteration, the parameters of the lip sync discriminator are frozen and the generator is trained. As mentioned above, when training the generator, the face video dataset is input into the generator. The facial image encoder of the generator obtains the original image with the lower half of the area masked and the image used for lip posture reference and then outputs the image features. The audio encoder of the generator obtains the audio corresponding to the original image with the lower half of the area masked and outputs the audio features. The decoder of the generator receives the fused audio and video features, learns how to generate lip movements that match the audio content, and applies it to the static image to generate a video frame sequence with lip sync effect. After obtaining the face video generated by the generator, the reconstructed overall loss is used to compare the loss of the face video with the corresponding original image, and the relevant parameters of the generator are adjusted accordingly. Specifically, according to the loss function defined in the overall loss function, the gradient of the trainable parameters in the generator network is calculated, and the gradient is propagated back to the generator network using the back propagation algorithm to update the network parameters to minimize the overall loss function;

[0107] In the second stage of the current training iteration, the parameters of the generator are frozen and the lip sync discriminator is trained. As mentioned above, when training the lip sync discriminator, the face video image generated by the generator and the corresponding original audio in the face video dataset are input into the lip sync discriminator. The face encoder and audio encoder of the lip sync discriminator will convert the input face video image and audio into valid features respectively. Then, the lip sync discriminator uses the Relu activation function to calculate the dot product between the video embedding v of the effective features of the video image and the speech embedding s of the effective features of the audio, and obtains the synchronization rate P of the input audio-video. sync After obtaining the synchronization rate, the difference between the prediction result of the lip sync discriminator and the true label is calculated through the loss function, and the relevant parameters of the lip sync discriminator are adjusted accordingly. Specifically, the parameters of the lip sync discriminator are updated by calculating the gradient of the loss function. During the back propagation process, the prediction result of the lip sync discriminator is compared with the true label to optimize the ability of the lip sync discriminator so that it can better distinguish the authenticity of the generated lip videos.

[0108] The method of this embodiment is verified and evaluated below. Under the same parameters and experimental conditions, it is compared with the most popular and widely used Wav2lip method. Specifically, it is performed according to the following process:

[0109] This process is intended to verify the performance of the voice-driven method for improving the quality of face videos proposed in this embodiment. The method of this embodiment and the benchmark lip synchronization algorithm Wav2lip were trained on the Avspeech+4K mixed data set.

[0110] The performance of the method is evaluated using the FID and PSNR indicators. FID (Fréchet Inception Distance) is an indicator used to evaluate the difference between images generated by the generative model and real images. It quantifies the difference between the generated image and the real image by calculating the statistical distance between the features extracted from the pre-trained Inception model. Specifically, FID captures the statistical difference between the generated image and the real image by considering the mean and covariance of the feature distribution. It is defined as follows:

[0111]

[0112] where μ x and∑ x Represent the mean and covariance matrix of the real image features, μ g and∑ g Respectively represent the mean and covariance matrix of the generated image features. The smaller the FID index, the smaller the difference between the generated image and the real image, and the better the generation effect;

[0113] Compared with the common MSE (MeanSquared Error) and SSIM (Structural Similarity Index) indicators, PSNR (Peak Signal-to-Noise Ratio) is applicable to any type of image, including grayscale and color images, without the need for specific analysis of image content or structure. Secondly, the process of calculating PSNR is relatively simple and fast. It only requires calculating the mean square error and applying logarithmic operations, unlike SSIM, which needs to consider complexities such as covariance and brightness contrast between pixels. Therefore, PSNR is an intuitive indicator that is easy to understand and explain. It is based on the mean square error between image pixel values ​​and expresses the signal-to-noise ratio of the image in decibels. Higher PSNR means less information loss and is usually associated with higher image quality.

[0114] The method of this embodiment is compared with the benchmark algorithm Wav2lip on a 1080P dataset collected by an individual. Each algorithm is trained for 600,000 steps under the same hyperparameters. After the training is completed, 100 inference videos are generated respectively. The face areas of the inference video images are compared with the real video images to calculate the FID and PSNR. As shown in Table 1, in the 1080P dataset, the FID score and PSNR score of the voice-driven method for improving the quality of face videos proposed in this embodiment are both optimal.

[0115] Table 1 Comparison of FID performance and PSNR performance of different algorithms on 1080P dataset

[0116]

[0117] The FID score is improved by 61.2% and the PSNR score is improved by 4.29% compared with the original Wav2lip. The experimental results show that the image quality generated by the method of this embodiment is much higher than the benchmark algorithm.

[0118] Embodiment 2

[0119] This embodiment proposes a voice-driven system for improving video quality, including a microprocessor and a computer-readable storage medium connected to each other, wherein the microprocessor is programmed or configured to execute the voice-driven method for improving video quality described in the first embodiment.

[0120] In summary, the present invention proposes a voice-driven method and system for improving video quality. Aiming at the problem that the language coverage lacks Chinese, resulting in insufficient adaptation and fitting of Chinese videos, a high-quality speaking face video dataset is obtained after collection and preprocessing, and a professional 4K high-definition Chinese video is introduced into it. Based on this dataset, the lip synchronization discriminator of Wav2lip is retrained to improve the discrimination rate of the discriminator for lip synchronization. In addition, a multi-scale structural similarity loss is introduced in the original Wav2lip generator, and the loss function is reconstructed to avoid the generator focusing only on the global structure and ignoring the details or local structure. At the same time, the network input layer of the generator and the discriminator is broadened, and the network architecture is optimized by stacking multiple 3x3 convolution kernels in the submodule. A voice-driven method for improving the quality of face videos is verified and evaluated using FID and PSNR indicators. Compared with the benchmark method, this method can effectively improve the quality problems such as blurred lip area and generate a face video with higher definition.

[0121] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A voice-driven method for improving video quality, characterized in that: The following steps are involved: Preprocess the face speaking videos including Chinese videos to obtain a face video dataset; Pre-training lip sync discriminator using face video dataset; The loss function and network architecture of the generator are reconstructed, and the generator is trained using a face video dataset. During the training process, a pre-trained lip synchronization discriminator is introduced. The generator and lip synchronization discriminator are alternately optimized to improve the training effect, so that the trained generator can generate high-quality and high-synchronization speech-driven face videos.

2. The voice-driven method for improving video quality according to claim 1, characterized in that: When preprocessing the face speech video containing Chinese video, it includes: Eliminate low-quality videos, unify the video frame rate, denoise videos with background noise, and cut out irrelevant clips in the video; Divide the video into segments, delete the segments that do not meet the required length, and save the video segments of the same person in the same face folder; Use the trained audio-video synchronization detection model to detect the audio and video synchronization of each video clip, and delete the video clips with audio and video offset information or face size that does not meet the requirements; Use the trained face detector to detect and segment the face images in each video clip.

3. The voice-driven method for improving video quality according to claim 1, characterized in that: The lip sync discriminator includes a facial encoder, which uses stacked convolution modules to extract features from an input feature map, wherein convolution module 1 uses a 7x7 convolution kernel to increase the number of channels, and the size of the feature map remains unchanged; convolution module 2 reduces the width of the feature map by 1 / 2 and doubles the number of channels; convolution modules 3 to 8 reduce the size of the feature map by 1 / 2 module by module, and the number of channels doubles layer by layer until convolution module 7 remains unchanged; convolution modules 3 to 8 use stacked 3x3 convolution kernels to capture higher-level and global features; convolution module 9 further scales the feature map by 0 padding; the last convolution layer compresses the number of channels and applies Sigmoid as the activation function to output a binary classification result.

4. The voice-driven method for improving video quality according to claim 1, characterized in that: When pre-training the lip sync discriminator using the face video dataset, the loss function of the lip sync discriminator uses the cosine similarity loss function with binary cross entropy. At the same time, the lip sync discriminator uses the Relu activation function to calculate the dot product between the video and speech embeddings to obtain the input audio-video synchronization rate P of the face sample. sync , the expression is as follows: Where v and s represent the feature vectors of video and audio respectively, P sync Represents the probability that the video and audio are synchronized in this time series.

5. The voice-driven method for improving video quality according to claim 1, characterized in that: When reconstructing the generator's loss function and network architecture, the reconstructed generator loss function is as follows: L total =α1L sync +α2L perceptual +α3L ms +(1-α1-α2-α3)L1 in, sync represents the synchronization loss function, L perceptual represents the perceptual loss function, L1 represents the reconstruction loss function, and L ms is the multi-scale structural similarity loss function, and α1, α2, and α3 are hyperparameter weights.

6. The voice-driven method for improving video quality according to claim 5, characterized in that: The multi-scale structural similarity loss function decomposes the input face image into different scales and calculates the SSIM loss at each scale. The expression is as follows: Among them, l M (x, y) represents the contrast of the original image at the highest scale, c j (x, y) represents the structural similarity of the original image at the jth iteration scale, s j (x, y) represents the brightness similarity of the original image at the jth iteration scale, α M , β j , γ j Represents an index used to adjust the relative importance of different components.

7. The voice-driven method for improving video quality according to claim 1, characterized in that: When reconstructing the loss function and network architecture of the generator, the reconstructed generator network architecture includes a facial image encoder, which uses stacked convolutional layer sub-modules to extract features from the spliced ​​image, wherein sub-module 1 uses a 7x7 convolution kernel to increase the number of channels; sub-modules 2 to 8 gradually reduce the spatial size by 2 times and increase the number of channels by 2 times; sub-module 9 further scales the feature map by padding with 0, and sub-modules 2 to 8 respectively use stacked 3x3 convolution kernels and introduce residual connections to help gradient propagation and learn complex features.

8. The voice-driven method for improving video quality according to claim 1, characterized in that: When reconstructing the loss function and network architecture of the generator, the reconstructed generator network architecture also includes a decoder, which uses stacked deconvolution module layers to gradually upsample the input feature map to restore the size of the original facial image, wherein deconvolution module layer 1 to deconvolution module layer 7 first upsample the feature map through the deconvolution layer, doubles the size of the feature map, reduces the number of channels, and then processes the feature map through stacked 3x3 convolution layers. The output module of the decoder further reduces the number of channels and limits the output to a specified range through the Tanh activation function.

9. The voice-driven method for improving video quality according to claim 1, characterized in that: The generator is trained using a face video dataset. The pre-trained lip sync discriminator is introduced during the training process. The generator and lip sync discriminator are alternately optimized to improve the training effect, including: In the first stage of the current training iteration, the parameters of the lip sync discriminator are frozen and the generator is trained. According to the defined loss function, the gradients of the trainable parameters in the generator network are calculated, and the gradients are propagated back to the generator network to update the network parameters using the back-propagation algorithm to minimize the overall loss function. In the second stage of the current training iteration, the parameters of the generator are frozen and the lip sync discriminator is trained. The parameters of the lip sync discriminator are updated by calculating the gradient of the loss function. During the back-propagation process, the prediction results of the lip sync discriminator are compared with the true labels to optimize the ability of the lip sync discriminator so that it can better distinguish the authenticity of the generated lip sync videos.

10. A voice-driven system for improving video quality, characterized in that: The invention comprises a microprocessor and a computer-readable storage medium connected to each other, wherein the microprocessor is programmed or configured to execute the voice-driven method for improving video quality as claimed in any one of claims 1 to 9.

Images