A voice-driven method and system for improving video quality

Abstract

This invention discloses a speech-driven method and system for improving video quality. The method includes: preprocessing a facial speaking video containing Chinese characters to obtain a facial video dataset; pre-training a lip-sync discriminator using the facial video dataset; reconstructing the generator's loss function and network architecture; training the generator using the facial video dataset; introducing the pre-trained lip-sync discriminator during training; and alternately optimizing the generator and the lip-sync discriminator to improve training performance, enabling the trained generator to produce high-quality, high-synchronization-rate speech-driven facial videos. This invention can effectively improve quality issues such as blurred lip areas, generating clearer facial videos.

Description

Technical Field

[0001] This invention relates to the field of computer information technology, and specifically to a voice-driven method and system for improving video quality. Background Technology

[0002] Voice-driven face generation is primarily based on deep learning technology. This method constructs a neural network that encodes features from input video and audio clips, continuously learning these features to generate new speaking face videos that match the audio. This method can be effectively applied in the influencer livestreaming industry, and the development of virtual anchors can bring significant convenience to businesses and human resources. In practical applications, only text or voice information needs to be input to synthesize new video applications through a synthesis system.

[0003] In recent years, voice-driven facial video generation technology has permeated fields including, but not limited to, virtual anime characters, virtual livestreaming e-commerce, intelligent facial recognition, video face-swapping effects, and beauty applications. Currently, the specific implementation methods of voice-driven facial video generation are mainly divided into two categories: generation methods based on facial feature points to the speaking face, and generation methods based on facial images to the speaking face.

[0004] Face video generation methods based on facial feature points use recurrent neural network learning. This method maps raw audio features to mouth shapes, synthesizes high-quality mouth textures, and generates high-quality synchronized facial videos with mouth shapes. However, this method requires a large dataset of specific individuals for training to generate realistic face videos for those individuals. Methods based on face images to speaking faces use an encoder-decoder convolutional neural network model. This model uses joint embeddings of face and audio, with features extracted by an identity encoder and an audio encoder, respectively. A face image generation decoder reconstructs the person's features and audio features to generate video frames of speaking faces. This method only calculates L1 loss for each video frame, resulting in blurry output images and poor visual quality.

[0005] In summary, current research in speech-driven face video generation primarily focuses on the synchronization accuracy between audio and lip movements, as well as the naturalness of facial expressions. Traditional methods only consider the generation of multiple single-frame videos without considering the temporal relationships between frames, leading to discontinuities in the generated videos and issues such as pixel jitter and poor visual quality. Furthermore, traditional speech-driven face video methods are prone to problems such as blurriness, poor detail quality, and obvious rectangular frames in the mouth reconstruction area. Therefore, providing a multi-scale speech-driven face video method that achieves higher clarity in the mouth region of the generated face video and effectively improves issues such as excessive green noise, slight artifacts, and matrix contours in the lip area remains a pressing technical challenge. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a voice-driven method and system for improving video quality, which can generate higher-resolution facial videos, in view of the above-mentioned problems in the prior art.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] A voice-driven method for improving video quality includes the following steps:

[0009] Preprocessing is performed on videos of people speaking with their faces, which contain Chinese characters, to obtain a face video dataset;

[0010] A lip-sync discriminator was pre-trained using a face video dataset;

[0011] The loss function and network architecture of the generator are reconstructed. The generator is trained using a face video dataset. A pre-trained lip-sync discriminator is introduced during the training process. The generator and lip-sync discriminator are alternately optimized to improve the training effect, so that the trained generator can generate high-quality, high-synchronization speech-driven face videos.

[0012] Furthermore, preprocessing of facial speaking videos containing Chinese characters includes:

[0013] Low-quality videos are removed, then the video frame rate is unified, videos with background noise are denoised, and irrelevant segments in the video are cut out.

[0014] Divide the video into segments, delete video segments that do not meet the length requirements, and save video segments of the same person in the same face folder;

[0015] The trained audio-video synchronization detection model is used to detect the audio-video synchronization of each video segment, and video segments with audio-video offset information or face size that do not meet the requirements are deleted.

[0016] The trained face detector is used to detect and segment face images in each video clip.

[0017] Furthermore, the lip-sync discriminator includes a facial encoder, which uses stacked convolutional modules to extract features from the input feature map. Convolutional module 1 increases the number of channels using 7x7 convolutional kernels while keeping the feature map size unchanged. Convolutional module 2 reduces the feature map width by half and doubles the number of channels. Convolutional modules 3 through 8 reduce the feature map size by half layer by layer while doubling the number of channels layer by layer until convolutional module 7 remains unchanged. Convolutional modules 3 through 8 use stacked 3x3 convolutional kernels to capture higher-level and global features. Convolutional module 9 further scales the feature map by padding with zeros. The last convolutional layer compresses the number of channels and applies a sigmoid function as the activation function to output a binary classification result.

[0018] Furthermore, when pre-training the lip-sync discriminator using a face video dataset, the loss function of the lip-sync discriminator uses a cosine similarity loss function with binary cross-entropy. Simultaneously, the lip-sync discriminator uses a ReLU activation function to calculate the dot product between the video and speech embeddings, obtaining the synchronization rate P of the input audio-video for the face sample. sync The expression is as follows:

[0019]

[0020] Where v and s represent the feature vectors of video and audio, respectively, and P sync This indicates the probability that the video and audio are synchronized within this time series.

[0021] Furthermore, when reconstructing the generator's loss function and network architecture, the reconstructed generator loss function is as follows:

[0022] L total =α1L sync +α2L perceptual +α3L ms +(1-α1-α2-α3)L1

[0023] Among them, L sync L represents the synchronization loss function. perceptual L1 represents the perceptual loss function, and L2 represents the reconstruction loss function. ms Let α1, α2, and α3 be the multi-scale structural similarity loss function, and let α1, α2, and α3 be the hyperparameter weights, respectively.

[0024] Furthermore, the multi-scale structural similarity loss function decomposes the input face image into different scales and calculates the SSIM loss at each scale, as shown in the following expression:

[0025]

[0026] Among them, l M (x,y) represents the contrast of the original image at its highest scale, c j (x,y) represents the structural similarity of the original image at the j-th iteration, s j (x,y) represents the brightness similarity of the original image at the j-th iteration, α M ,β j γ j An index used to adjust the relative importance of different components.

[0027] Furthermore, when reconstructing the loss function and network architecture of the generator, the reconstructed generator network architecture includes a facial image encoder. The facial image encoder uses stacked convolutional layer sub-modules to extract features from the stitched image. Sub-module 1 uses 7x7 convolutional kernels to increase the number of channels; sub-modules 2 to 8 gradually reduce the spatial size by a factor of 2 and increase the number of channels by a factor of 2; sub-module 9 further scales the feature map by zero padding. Sub-modules 2 to 8 use stacked 3x3 convolutional kernels and introduce residual connections to help gradient propagation and learn complex features.

[0028] Furthermore, when reconstructing the generator's loss function and network architecture, the reconstructed generator network architecture also includes a decoder. The decoder uses stacked deconvolutional module layers to progressively upsample the input feature map to restore the size of the original facial image. Deconvolutional module layers 1 to 7 first upsample the feature map through deconvolutional layers, doubling the feature map size and reducing the number of channels. Then, the feature map is processed through stacked 3x3 convolutional layers. The decoder's output module further reduces the number of channels and limits the output to a specified range using the Tanh activation function.

[0029] Furthermore, the generator is trained using a face video dataset. During training, a pre-trained lip-sync discriminator is introduced. The generator and lip-sync discriminator are alternately optimized to improve training performance, including:

[0030] In the current training iteration, the parameters of the lip-sync discriminator are frozen and the generator is trained. According to the defined loss function, the gradients of the trainable parameters in the generator network are calculated. The gradients are propagated back to the generator network using the backpropagation algorithm to update the network parameters in order to minimize the overall loss function.

[0031] In the next training iteration, the generator's parameters are frozen and the lip-sync discriminator is trained. The parameters of the lip-sync discriminator are updated by calculating the gradient of the loss function. During backpropagation, the prediction results of the lip-sync discriminator are compared with the real labels to optimize the ability of the lip-sync discriminator so that it can better distinguish the authenticity of the generated lip-sync videos.

[0032] The present invention also proposes a voice-driven system for improving video quality, comprising a microprocessor and a computer-readable storage medium interconnected thereto, wherein the microprocessor is programmed or configured to execute any of the voice-driven methods for improving video quality described herein.

[0033] Compared with the prior art, the advantages of the present invention are as follows:

[0034] This invention addresses the issue of insufficient adaptation and fitting of Chinese videos due to the lack of Chinese language coverage in current facial speech videos by introducing Chinese videos. Based on this dataset, the Wav2lip lip-sync discriminator is retrained to improve its discrimination rate for lip-sync. Furthermore, a multi-scale structural similarity loss is introduced into the Wav2lip generator, and the loss function is reconstructed to prevent the generator from focusing solely on global structure while ignoring details or local structures. Simultaneously, the input layers of the generator and lip-sync discriminator are broadened, and multiple 3x3 convolutional kernels are stacked to optimize the network architecture. Compared to benchmark methods, this invention effectively improves quality issues such as blurred lip areas, generating clearer facial videos. Attached Figure Description

[0035] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0036] Figure 2 This is a schematic diagram of the network architecture of the facial encoder in the lip-sync discriminator of this invention.

[0037] Figure 3 This is a schematic diagram of the network architecture of the face encoder of the generator in an embodiment of the present invention.

[0038] Figure 4 This is a schematic diagram of the network architecture of the decoder of the generator in an embodiment of the present invention.

[0039] Figure 5 This is a schematic diagram of the process of training the generator and introducing a pre-trained lip-sync discriminator during the training process in an embodiment of the present invention. Detailed Implementation

[0040] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0041] Before introducing specific embodiments of the present invention, the relevant concepts will be explained first:

[0042] Wav2Lip: Wav2Lip is a deep learning model based on Generative Adversarial Networks (GANs) primarily used to generate facial animations, especially lip movements, that match audio waveforms. The model consists of two core components:

[0043] Generator: The generator's task is to receive audio information as input and then generate a corresponding facial animation sequence, paying particular attention to lip movements to achieve synchronization with the audio.

[0044] Lip-sync discriminator: This discriminator determines the degree of synchronization between the generated facial animation and the real facial animation in terms of lip movements and audio. It analyzes consecutive input video frames and audio signals to determine if they are synchronized, thus guiding the generator to produce more accurate results.

[0045] During training, the generator and the lip-sync discriminator compete against each other. The generator attempts to produce the most realistic animation possible to fool the discriminator, while the discriminator tries to accurately distinguish between the generated animation and the real animation. Through this adversarial process, the model gradually improves its ability to generate realistic synchronized lip-sync animations.

[0046] Example 1

[0047] This embodiment proposes a speech-driven method to improve video quality. Based on the commonly used Wav2lip algorithm, it expands the network's input layer by stacking multiple 3x3 convolutional kernels. The newly added convolutional layers help the generator network better capture various features of the image, further improving the performance of generated images. To allow the convolutional layers to have a larger receptive field, pooling layers are not included in the network architecture. Residual connections are introduced to aid gradient propagation and the learning of complex features. This approach avoids the generator focusing only on global structure while ignoring details or local structures, allowing the generated images to better maintain structural similarity to real images, thus producing clearer and more natural video output.

[0048] like Figure 1 As shown, the method in this embodiment includes the following steps:

[0049] S1) Obtain the facial speaking video and import Chinese video into it;

[0050] S2) Preprocess the facial speaking videos containing Chinese videos to obtain a facial video dataset;

[0051] S3) Use a face video dataset to pre-train a lip-sync discriminator;

[0052] S4) Reconstruct the generator's loss function and network architecture;

[0053] S5) The generator is trained using a face video dataset. During the training process, a pre-trained lip-sync discriminator is introduced. The generator and the lip-sync discriminator are alternately optimized to improve the training effect, so that the trained generator can generate high-quality, high-synchronization speech-driven face videos.

[0054] Each step will be explained below.

[0055] Step S1 of this embodiment acquires a large number of facial speaking videos with different backgrounds, poses, and lighting conditions, and introduces Chinese videos into the collected facial speaking videos. This overcomes the problem that the current facial speaking videos lack Chinese language coverage, resulting in insufficient adaptation and fitting of the voice-driven method to Chinese videos. The specific process is as follows:

[0056] S1.1) Experimental analysis was conducted on the currently recognized and widely used public datasets Avspeech, CelebA, LFW, and FDDB;

[0057] S1.2) Experimental analysis revealed that the annotation quality of the CelebA dataset is inconsistent, with some attribute annotations containing errors and noise; the image quality and angle of the LFW dataset vary significantly, making it less than ideal for more detailed face analysis tasks; and the FDDB dataset is relatively small, insufficient for tasks requiring large-scale data. In contrast, the Avspeech dataset possesses the following advantages and enjoys widespread recognition:

[0058] (1) Avspeech is a large dataset of face speaking videos, containing more than 10,000 hours of audio and video data. This large-scale dataset can provide a better ability to train and evaluate speech-driven face video methods.

[0059] (2) The Avspeech dataset covers a variety of languages, including but not limited to English, French, German, Spanish, Japanese and so on. This multilingual coverage enables researchers to conduct research and development on cross-language speech-driven human video methods.

[0060] (3) The Avspeech dataset contains a variety of audio and video data from YouTube, covering a wide range of topics and fields. This diversity of topics makes the dataset more representative and can be used for research on speech-driven human video methods in various practical application scenarios;

[0061] (4) The Avspeech dataset has become one of the important resources in the field of speech technology research. Many research methods and projects use the Avspeech dataset as a benchmark for model training, evaluation and comparison.

[0062] S1.3) The Avspeech dataset still has some shortcomings. Since it is extracted from YouTube videos, the quality of the audio and video data may vary due to factors such as video quality, recording equipment, and environmental conditions. Additionally, the lack of Chinese data in the Avspeech dataset results in low adaptability of the trained audio to Chinese. Therefore, the following optimizations are made to the Avspeech dataset:

[0063] (1) Manually clean the Avspeech dataset to remove low-quality audio and video;

[0064] (2) High-quality 4K Chinese face audio and video recordings with professional models, professional recording scenes and professional shooting conditions were manually recorded to fill the dataset, so as to ensure that the overall dataset is sufficient in quantity and meets the quality standards.

[0065] Step S2 of this embodiment preprocesses the optimized Avspeech dataset filled with Chinese videos to achieve better training results and constructs the dataset file. Specifically, it is performed as follows:

[0066] S2.1) Remove low-quality videos. Specifically, in this embodiment, removing low-quality videos involves manually removing audio and video files from the Avspeech dataset that have a resolution lower than 1080P, contain multiple faces, or have small face sizes.

[0067] Then, the video frame rate was unified, the video with background noise was denoised, and irrelevant segments in the video were cut off. Specifically, the Avspeech dataset after removing low-quality videos and the mixed 4K high-definition face video material were processed as follows: (1) Open the video file with OpenCV and obtain the video frame rate (FPS), and use Ffmpeg to convert the video frame rate to 25; (2) The video with obvious background noise was denoised; (3) Only people facing the camera lens in the video can be seen. If there is any discrepancy, irrelevant segments should be cut off.

[0068] S2.2) Divide the video into segments and delete video segments that do not meet the duration requirements. Specifically, use FFmpeg to divide the video into multiple segments and delete videos with a duration of less than 4 seconds.

[0069] Video clips of the same person are saved to the same face folder. Specifically, each face folder and video file in the root directory of the dataset is traversed, and the video files are renamed in a standardized manner. The video files in each face folder correspond to the same person.

[0070] S2.3) Use the trained audio-video synchronization detection model to detect the audio-video synchronization of each video segment, and delete video segments with audio-video offset information or face size that does not meet the requirements.

[0071] Specifically, the trained and exported audio-video synchronization detection model vad_moedl is loaded to process the video file, extract audio information, analyze the audio, detect audio-video synchronization, and return audio-video offset information. If no offset information is obtained or the face size is less than a threshold, the video file is skipped and its path is written to the output file; if the obtained offset exceeds a preset range (-3 to 3), its path is written to the output file. The output file is then reviewed and non-compliant audio-video segments are manually deleted.

[0072] S2.4) Use a trained face detector to detect and segment face images in each video segment. Specifically, a deep learning-based pre-trained face detector is used to detect and segment face images. Finally, training, validation, and test sets are constructed based on all pre-processed face-speaking videos.

[0073] In this embodiment, step S3 involves passing the original face image after face segmentation and the audio through a discriminator to obtain the synchronization loss between the original audio and video, and generating a discriminator for supervising the detection of lip-sync in face videos. Specifically, this is performed according to the following process:

[0074] The constructed face video dataset is used as input to pass through the discriminator, and a lip-sync discriminator specifically designed for this dataset is retrained.

[0075] The lip-sync discriminator consists of two parts: a face encoder and an audio encoder. The face encoder and audio encoder of the lip-sync discriminator convert the input face video image and audio into effective features, respectively. The lip-sync discriminator learns the effective feature representations to distinguish the difference between the lip-sync accuracy of the generated video frames and the real video frames.

[0076] In this embodiment, the facial encoder network architecture diagram of the lip-sync discriminator is attached. Figure 2 As shown, the numbers on the convolutional modules represent the image's length x width x number of channels. In the lower right corner, K represents the kernel size, S represents the stride, and P represents padding. The lip-sync discriminator's face encoder extracts features from the input feature map through a series of stacked convolutional modules, without using pooling layers to obtain the maximum receptive field.

[0077] like Figure 2As shown, convolutional module 1 uses a 7x7 convolutional kernel to increase the number of channels from 3 to 16, while keeping the feature map size unchanged; convolutional module 2 reduces the feature map width by 1 / 2 and doubles the number of channels; convolutional modules 3 to 8 reduce the feature map size by 1 / 2 layer by layer, using stacked 3x3 convolutional kernels to capture higher-level and global features by stacking multiple convolutional layers, where the number of channels doubles layer by layer until it remains unchanged after the seventh convolutional module; convolutional module 9 further scales the feature map to 1024x1x1 by padding with zeros; the last convolutional layer compresses the number of channels to 1 and applies Sigmoid as the activation function to output the binary classification result.

[0078] In this embodiment, the audio encoder network architecture of the lip-sync discriminator is similar to the source model Wav2lip, with minor adjustments made for the new input size, which will not be elaborated in this embodiment.

[0079] In this embodiment, the lip-sync discriminator uses a novel loss function—the cosine similarity loss function with binary cross-entropy. Specifically, the lip-sync discriminator uses a ReLU activation function to compute the dot product between the video embedding v of the effective features of the video image and the speech embedding s of the effective features of the audio. Each face sample generates a single value between [0,1] representing the synchronization rate P of the input audio-video. sync The expression is as follows:

[0080]

[0081] Where v and s represent the feature vectors of video and audio, respectively, and P sync This indicates the probability that the video and audio are synchronized within this time series.

[0082] Step S4 of this embodiment introduces an enhanced structural similarity loss into the original Wav2lip generator, reconstructs the generator loss function, broadens the network input layers of the generator and discriminator, and optimizes the network architecture by stacking multiple 3x3 convolutional kernels. Specifically, it is performed as follows:

[0083] While the Wav2lip method primarily focuses on aligning speech and vision to generate realistic lip-sync animation, its overall image clarity may be somewhat lacking. Compared to publicly available methods, this embodiment reconstructs the generator's loss function, resulting in a new overall loss function L. total as follows:

[0084] L total =α1L sync +α2L perceptual +α3L ms +(1-α1-α2-α3)L1 (2)

[0085] Among them, L ms To introduce a multi-scale structural similarity loss function, L sync L perceptual L1 is the original loss function of Wav2lip, L sync The synchronization loss function is represented by comparing facial video frames and associated audio features to assess their correlation, obtained through a lip-sync discriminator; L perceptual The perceptual loss function represents the difference between the generated video frame and the real video frame. The reconstruction loss function represents the generator improving the quality of the generated frame by minimizing the L1 loss between the generated frame and the real frame. α1, α2, and α3 are the hyperparameter weights, respectively.

[0086] In this embodiment, the multi-scale structural similarity loss is transformed to the same order of magnitude as the original Wav2lip loss by using the hyperparameter weight α3 corresponding to the multi-scale structural similarity loss function, so as to avoid the phenomenon of gradient explosion, and the sum of the total weight coefficients of each loss is kept to 1 by (1-α1-α2-α3).

[0087] The parameter values ​​of α1, α2, and α3 can be obtained through hyperparameter search and multiple pre-training sessions. An excessively large α3 parameter can lead to overfitting of the generated face image, resulting in extraneous images. Therefore, keeping α3 between 0.003 and 0.007 is more suitable, and α2 needs to be appropriately reduced based on the value of α3. Maintaining α2 at its original value will result in a blurry generated image.

[0088] In this embodiment, the multi-scale structural similarity loss function for enhancing the effect decomposes the input face image into three different scales: contrast, structural similarity, and brightness similarity, and calculates the SSIM loss at each scale, as shown in the following expression:

[0089]

[0090] Among them, l M (x,y) represents the contrast of the original image at its highest scale, c j (x,y) represents the structural similarity of the original image at the j-th iteration, s j (x,y) represents the brightness similarity of the original image at the j-th iteration, α M ,β j γ j An index used to adjust the relative importance of different components.

[0091] The original image scale is Scale1, and the highest scale ScaleM is obtained through M-1 iterations. At the j-th scale, only contrast and structural similarity are calculated, and brightness similarity is calculated only at ScaleM.M (x,y), the final SSIM index is obtained by connecting the results of each scale as shown in Formula 3. Where the exponent α... M ,β j γ j Used to adjust the relative importance of different components, to simplify the selection of parameters, let all α M =β j =γ j .

[0092] In this embodiment, the contrast ratio calculation formula is as follows:

[0093]

[0094] Where μ x μ y C1 represents the mean of x and y, where x and y represent two given images, namely the real image and the generated image, and C1 is a constant.

[0095] The formula for calculating structural similarity is as follows:

[0096]

[0097] Where σ x , σ y Represents the standard deviation of x and y. C represents the variance of x, and C2 is a constant.

[0098] The formula for calculating brightness similarity is as follows:

[0099]

[0100] Where σ xy Let x be the covariance between x and y.

[0101] The generator in this embodiment consists of the following three parts:

[0102] The first part is the face image encoder, whose input consists of the original image with the lower half of the region occluded (from the ground truth label, GroudTruth) and an image for lip pose reference. This module concatenates the two sets of images and inputs them into the face image encoder, then outputs image features. The network architecture diagram of the reconstructed generator face encoder is attached. Figure 3As shown, the numbers on the convolutional modules represent the image's length x width x number of channels. In the lower right corner, K represents the kernel size, S represents the stride, and P represents padding. The input is a 384x384x6 face image, which undergoes feature extraction through a sequence of nine convolutional layer sub-modules. Sub-module 1 uses a 7x7 convolutional kernel to increase the number of channels from 6 to 8; sub-modules 2 through 8 reduce the spatial size by a factor of 2 and increase the number of channels by a factor of 2; sub-module 9 further scales the feature map using zero padding to obtain a 1x1 face feature vector. Sub-modules 2 through 8 all stack multiple 3x3 convolutional layers and introduce residual connections to aid gradient propagation and the learning of complex features.

[0103] The second part is the audio encoder, which takes as input the audio corresponding to the original image with the lower half of the region occluded, and outputs audio features. The audio encoder module structure is similar to the source model Wav2lip, with only matching modifications made for the new size.

[0104] The third part is the decoder. It takes the image and audio features as input and outputs an audio-driven image, comparing the reconstruction loss (Loss) with the ground truth label, Ground Truth. The decoder receives the fused audio and video features, learns how to generate matching lip-sync actions based on the audio content, and applies them to the still image to generate a sequence of video frames with lip-sync effects. The decoder network architecture diagram is attached. Figure 4 As shown, the numbers on the convolutional modules represent the image's length x width x number of channels. In the lower right corner, K represents the kernel size, S represents the stride, and P represents padding. The deconvolutional module, composed of multiple convolutional neural network layers, progressively upsamples the input feature map to restore the original facial image size. When the features to be processed pass through convolutional module 1, the output feature map size is 1024x1x1. Deconvolutional module layers 1 to 7 first upsample the feature map through deconvolutional layers, doubling the feature map size and reducing the number of channels. Then, they are processed through stacked 3x3 convolutional layers. Finally, when passing through the output module, the number of channels is further reduced to 3, corresponding to the image's RGB channels, with a feature map size of 3x384x384. The Tanh activation function then limits the output to the range [-1, 1].

[0105] In this embodiment, after reconstructing the generator's loss function and network architecture, step S5 retrains the generator based on the face video dataset file from step S2. The overall process is shown in the appendix. Figure 5 As shown, it includes a reconstructed generator, a loss function, and a pre-trained lip-sync discriminator. During training, the generator and lip-sync discriminator are alternately optimized to improve the overall training effect. They penalize each other to promote learning, thereby achieving the generation of high-quality, high-synchronization speech-driven face videos. Specifically, it includes:

[0106] In the first phase of the current training iteration, the parameters of the lip-sync discriminator are frozen and the generator is trained. As mentioned earlier, during generator training, a face video dataset is input into the generator. The generator's facial image encoder acquires the original image with the lower half of the region occluded and an image used for lip pose reference, then outputs image features. The generator's audio encoder acquires the audio corresponding to the original image with the lower half of the region occluded and outputs audio features. The generator's decoder receives the fused audio and video features, learns how to generate matching lip movements based on the audio content, and applies them to static images to generate a video frame sequence with lip-sync effect. After obtaining the face video generated by the generator, the overall loss function is used to compare the face video with the corresponding original image, and the relevant parameters of the generator are adjusted accordingly. Specifically, based on the loss function defined in the overall loss function, the gradients of the trainable parameters in the generator network are calculated, and the backpropagation algorithm is used to propagate the gradients back to the generator network to update the network parameters to minimize the overall loss function.

[0107] In the second phase of the current training iteration, the generator parameters are frozen and the lip-sync discriminator is trained. As mentioned earlier, during lip-sync discriminator training, the face video image generated by the generator and the corresponding original audio from the face video dataset are input into the lip-sync discriminator. The face encoder and audio encoder of the lip-sync discriminator convert the input face video image and audio into effective features, respectively. Then, the lip-sync discriminator uses the ReLU activation function to calculate the dot product between the video embedding v of the effective features of the video image and the speech embedding s of the effective features of the audio, obtaining the synchronization rate P of the input audio-video. sync After obtaining the synchronization rate, the difference between the prediction results of the lip-sync discriminator and the real labels is calculated using a loss function, and the relevant parameters of the lip-sync discriminator are adjusted accordingly. Specifically, the parameters of the lip-sync discriminator are updated by calculating the gradient of the loss function. During backpropagation, the prediction results of the lip-sync discriminator are compared with the real labels to optimize the ability of the lip-sync discriminator to better distinguish the authenticity of the generated lip-sync videos.

[0108] The method of this embodiment will now be verified and evaluated. Under the same parameters and experimental conditions, it will be compared with the most popular and widely used Wav2lip method. The specific process is as follows:

[0109] This process aims to verify the performance of the voice-driven method for improving face video quality proposed in this embodiment. The method of this embodiment was trained on the Avspeech+4K hybrid dataset along with the benchmark lip-sync algorithm Wav2lip.

[0110] The performance of the method is evaluated using FID and PSNR metrics. FID (Fréchet Inception Distance) is a metric used to evaluate the difference between images generated by a generative model and real images. It quantifies the difference between the generated and real images by calculating the statistical distance between the features extracted from the pre-trained Inception model. Specifically, FID captures the statistical difference between the generated and real images by considering the mean and covariance of the feature distribution, and its definition is as follows:

[0111]

[0112] Where μ x and ∑ x Let μ represent the mean and covariance matrices of the real image features, respectively. g and ∑ g These represent the mean and covariance matrix of the generated image features, respectively. A smaller FID index indicates a smaller difference between the generated image and the real image, resulting in better generation quality.

[0113] Compared to common metrics like MSE (Mean Squared Error) and SSIM (Structural Similarity Index), PSNR (Peak Signal-to-Noise Ratio) is applicable to any type of image, including grayscale and color images, without requiring specific analysis of image content or structure. Secondly, calculating PSNR is relatively simple and fast, requiring only the calculation of the mean squared error and the application of logarithmic operations, unlike SSIM which needs to consider the complexity of pixel covariance and brightness contrast. Therefore, PSNR is an intuitive metric, easy to understand and interpret. It is based on the mean squared error between image pixel values, expressed in decibels as the image's signal-to-noise ratio. A higher PSNR means less information loss and is generally associated with higher image quality.

[0114] The method of this embodiment was compared with the benchmark algorithm Wav2lip on a personally collected 1080P dataset. Each algorithm was trained for 600,000 steps under the same hyperparameters. After training, 100 inference videos were generated for each algorithm. The face regions of the inference video images and the real video images were compared to calculate FID and PSNR. As shown in Table 1, in the 1080P dataset, the speech-driven method for improving face video quality proposed in this embodiment achieved the best FID and PSNR scores.

[0115] Table 1 Comparison of FID and PSNR performance of different algorithms on the 1080P dataset.

[0116]

[0117] The FID score was improved by 61.2% compared to the original Wav2lip, and the PSNR score was improved by 4.29%. Experimental results show that the image quality generated by the method in this embodiment is far superior to the benchmark algorithm.

[0118] Example 2

[0119] This embodiment proposes a voice-driven system for improving video quality, including a microprocessor and a computer-readable storage medium interconnected, wherein the microprocessor is programmed or configured to execute the voice-driven method for improving video quality as described in Embodiment 1.

[0120] In summary, this invention proposes a speech-driven method and system to improve video quality. Addressing the issue of insufficient adaptation and fitting to Chinese videos due to the lack of Chinese language coverage, a high-quality speaking face video dataset was obtained after collection and preprocessing, incorporating professional 4K high-definition Chinese videos. Based on this dataset, the Wav2lip lip-sync discriminator was retrained to improve its discrimination rate for lip-sync. Furthermore, a multi-scale structural similarity loss was introduced into the original Wav2lip generator, and the loss function was reconstructed to prevent the generator from focusing solely on global structure while ignoring details or local structures. Simultaneously, the input layers of the generator and discriminator were broadened, and the network architecture was optimized by stacking multiple 3x3 convolutional kernels in submodules. The speech-driven method for improving face video quality was validated and evaluated using FID and PSNR metrics. Compared to benchmark methods, this method effectively improves quality issues such as blurred lip regions, generating clearer face videos.

[0121] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A voice-driven method for improving video quality, characterized in that, Includes the following steps: Preprocessing is performed on videos of people speaking with their faces, which contain Chinese characters, to obtain a face video dataset; A lip-sync discriminator is pre-trained using a face video dataset. The lip-sync discriminator includes a face encoder that extracts features from the input feature map using stacked convolutional modules. Convolutional module 1 increases the number of channels using a 7x7 kernel while maintaining the same feature map size. Convolutional module 2 reduces the feature map width by half while doubling the number of channels. Convolutional modules 3 through 8 reduce the feature map size by half layer by layer while doubling the number of channels layer by layer until convolutional module 7 remains unchanged. Convolutional modules 3 through 8 use stacked 3x3 kernels to capture higher-level and global features. Convolutional module 9 further scales the feature map by padding with zeros. The final convolutional layer compresses the number of channels and applies a sigmoid activation function to output a binary classification result. The generator loss function and network architecture are reconstructed. The reconstructed generator loss function is as follows: in, Represents the synchronization loss function. Represents the perceptual loss function. Represents the reconstruction loss function. For multi-scale structural similarity loss function, These are the hyperparameter weights; The reconstructed generator network architecture includes a face image encoder that uses stacked convolutional layer sub-modules to extract features from the stitched image. Sub-module 1 uses 7x7 convolutional kernels to increase the number of channels; sub-modules 2 to 8 progressively reduce the spatial size by a factor of 2 and increase the number of channels by a factor of 2; sub-module 9 further scales the feature map by zero padding. Sub-modules 2 to 8 each use stacked 3x3 convolutional kernels and introduce residual connections to aid gradient propagation and learning complex features. The reconstructed generator network architecture also includes a decoder, which uses stacked deconvolutional module layers to progressively upsample the input feature map to restore the size of the original facial image. Deconvolutional module layers 1 to 7 first upsample the feature map through deconvolutional layers, doubling the feature map size and reducing the number of channels. Then, the feature map is processed through stacked 3x3 convolutional layers. The output module of the decoder further reduces the number of channels and limits the output to a specified range through the Tanh activation function. The generator is trained using a face video dataset. A pre-trained lip-sync discriminator is introduced during the training process. The generator and the lip-sync discriminator are alternately optimized to improve the training effect, so that the trained generator can generate high-quality, high-synchronization speech-driven face videos.

2. The voice-driven method for improving video quality according to claim 1, characterized in that, Preprocessing of facial speaking videos containing Chinese characters includes: Low-quality videos are removed, then the video frame rate is unified, videos with background noise are denoised, and irrelevant segments in the video are cut out. Divide the video into segments, delete video segments that do not meet the length requirements, and save video segments of the same person in the same face folder; The trained audio-video synchronization detection model is used to detect the audio-video synchronization of each video segment, and video segments with audio-video offset information or face size that do not meet the requirements are deleted. The trained face detector is used to detect and segment face images in each video clip.

3. The voice-driven method for improving video quality according to claim 1, characterized in that, When pre-training a lip-sync discriminator using a face video dataset, the loss function of the lip-sync discriminator uses a cosine similarity loss function with binary cross-entropy. Simultaneously, the lip-sync discriminator uses a ReLU activation function to compute the dot product between the video and speech embeddings, obtaining the synchronization rate of the input audio-video for the face samples. The expression is as follows: in v and s These represent the feature vectors of video and audio, respectively. This indicates the probability that the video and audio are synchronized within this time series.

4. The voice-driven method for improving video quality according to claim 1, characterized in that, The multi-scale structural similarity loss function decomposes the input face image into different scales and calculates the SSIM loss at each scale, as shown in the following expression: in, This represents the contrast of the original image at its highest scale. The structural similarity of the original image at the j-th iteration scale is represented by... This represents the brightness similarity of the original image at the scale of the j-th iteration. An index used to adjust the relative importance of different components.

5. The voice-driven method for improving video quality according to claim 1, characterized in that, When training a generator using a face video dataset, and introducing a pre-trained lip-sync discriminator during training, and alternately optimizing the generator and lip-sync discriminator to improve training performance, the following steps are taken: In the first phase of the current training iteration, the parameters of the lip-sync discriminator are frozen and the generator is trained. According to the defined loss function, the gradients of the trainable parameters in the generator network are calculated, and the gradients are propagated back to the generator network using the backpropagation algorithm to update the network parameters in order to minimize the overall loss function. In the second phase of the current training iteration, the generator's parameters are frozen and the lip-sync discriminator is trained. The parameters of the lip-sync discriminator are updated by calculating the gradient of the loss function. During backpropagation, the prediction results of the lip-sync discriminator are compared with the real labels to optimize the ability of the lip-sync discriminator to better distinguish the authenticity of the generated lip-sync videos.

6. A voice-driven system for improving video quality, characterized in that, It includes an interconnected microprocessor and a computer-readable storage medium, the microprocessor being programmed or configured to perform the voice-driven method for improving video quality as described in any one of claims 1 to 5.

Images