A lip sync video generation method, apparatus, device and medium

Abstract

The application provides a lip-synchronous video generation method and device, equipment and medium, relates to the lip-synchronous video generation technical field, and the main contributions include two aspects: first, the scale-invariant face detector (single shot scale-invariant face detector, S3FD) algorithm is used to remove the noise data of the data set, the convergence time of the lip-synchronous model is significantly reduced, and the lip-synchronous accuracy is also improved to a certain extent; second, the residual network (residual network, resNet) algorithm is used to identify the gender and age of the person in the data set, and a lip-synchronous submodel of the classified person is generated, so that the quality of the generated lip image is improved under the premise of ensuring the generalization ability of the overall model.

Description

Technical Field

[0001] This invention relates to the field of lip-sync video generation technology, specifically to a method, apparatus, device, and medium for generating lip-sync video. Background Technology

[0002] In recent years, with the rapid development of virtual reality, metaverse, and other fields, wave-to-lip (Wav2Lip) technology has become a hot research topic in the fusion of computer vision and audio processing, and has made great progress. It can now basically achieve the level of generating complete, unrestricted human speech videos. Wave-to-lip technology is a cross-modal generation technology based on speech signals and video images. It can convert speech signals into human lip movements and apply them to video generation. Virtual digital humans built using wave-to-lip technology have begun to contribute to productivity in various industries, such as bank customer service and internet live streaming.

[0003] Currently, the Wav2Lip model has become one of the most widely adopted algorithms in this field. It utilizes deep learning technology to generate lip-shape images based on speech and achieves effective lip-sound synchronization. However, in practical applications, this model still has some shortcomings: 1) When the characteristics of the dataset samples are poor, the lip-sound synchronization model may fail to converge or the convergence time may be very long. In the field of speech-driven lip-shape research, which requires a large number of video samples, manually correcting the samples is very difficult. Therefore, this algorithm has high requirements for the dataset; 2) In order to improve the generalization ability of the lip-sound synchronization model, the Wav2Lip dataset contains samples of different people with a wide range of facial features. However, this also leads to low-quality lips and incorrect tooth information in the generated lip-shape videos. Therefore, how to generate high-quality lip-shape videos remains a challenge.

[0004] In view of the above, this application is hereby submitted. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method, apparatus, device and medium for generating lip-sync videos, which can effectively solve the problems of the existing Wav2Lip model failing to converge or having a long convergence time when the characteristics of the dataset samples are poor. In the field of speech-driven lip-shape research, which requires a large number of video samples, it is very difficult to manually correct the samples. Therefore, this algorithm has high requirements for the dataset. It also addresses the problem of low-quality lip and tooth information.

[0006] This invention discloses a method for generating lip-synced video, comprising:

[0007] Multiple solo speaking videos are acquired, and each solo speaking video is segmented to generate an initial dataset, wherein the original video length of each solo speaking video is greater than 20 seconds. The initial dataset is optimized using the S3FD algorithm to generate a character dataset. The optimization process includes frame extraction and judgment processing. The trained ResNet-50 deep residual network model is used to preprocess the dataset of people to generate lip-sync videos. The ResNet-50 deep residual network model consists of 50 convolutional layers and fully connected layers, and includes four lip-sync sub-models for adult women, adult men, children and the elderly.

[0008] This invention also discloses a lip-sync video generation device, comprising: The dataset acquisition unit is used to acquire multiple single-person speaking videos, and to segment each single-person speaking video to generate an initial dataset, wherein the original video length of the single-person speaking video is greater than 20 seconds. The dataset optimization unit is used to optimize the initial dataset using the S3FD algorithm to generate a character dataset. The optimization process includes frame extraction and judgment processing. The lip-sync video generation unit is used to call the trained ResNet-50 deep residual network model to preprocess the dataset of people and generate lip-sync videos. The ResNet-50 deep residual network model consists of 50 convolutional layers and fully connected layers, and includes four lip-sync sub-models for adult women, adult men, children and the elderly.

[0009] The present invention also discloses a lip-sync video generation device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a lip-sync video generation method as described above.

[0010] The present invention also discloses a readable storage medium storing a computer program, which can be executed by the processor of the device where the storage medium is located to implement a lip-sync video generation method as described in any of the above claims.

[0011] In summary, the lip-sync video generation method, apparatus, device, and medium provided in this embodiment employ a scale-invariant face detector (S3FD) algorithm to remove noisy data from the dataset, significantly reducing the convergence time of the lip-sync model and improving the accuracy of lip-sync. Furthermore, it proposes using a residual network (resNet) algorithm to identify the gender and age of individuals in the dataset, generating lip-sync sub-models for classifying these individuals. This improves the quality of the generated lip images while ensuring the overall model's generalization ability. This addresses the problems of existing Wav2Lip models failing to converge or experiencing very long convergence times when the dataset samples have poor characteristics. In speech-driven lip-sync research requiring a large number of video samples, manual sample correction is extremely difficult, thus the algorithm has high requirements for the dataset. It also resolves the issue of low-quality lip and tooth information that may occur. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a lip-sync video generation method provided in an embodiment of the present invention.

[0013] Figure 2 This is a schematic diagram of the 3FD algorithm principle provided in the embodiment of the present invention.

[0014] Figure 3 This is a schematic diagram illustrating the data cleaning effect provided in an embodiment of the present invention.

[0015] Figure 4 This is a schematic diagram of the Wav2Lip model flow provided in an embodiment of the present invention.

[0016] Figure 5 This is a schematic diagram of the ResNet-50 model principle provided in the embodiment of the present invention.

[0017] Figure 6 This is a schematic diagram comparing the results of the Wav2Lip algorithm and the face classifier sub-model algorithm provided in this embodiment of the invention.

[0018] Figure 7 This is a schematic diagram of a lip-sync video generation device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] Please see Figure 1 The first embodiment of the present invention provides a lip-sync video generation method, which can be executed by a lip-sync video generation device (hereinafter referred to as the generation device), and in particular by one or more processors within the generation device, to achieve the following steps: In this embodiment, the generating device may be a user terminal device (such as a smartphone, smart computer, or other smart device), which can establish a communication connection with a server in the cloud to achieve data interaction.

[0022] The existing Wav2Lip model algorithm also has some shortcomings, mainly: (1) When the characteristics of the dataset samples are poor, the lip-sync model may not converge or the convergence time is very long. For the field of speech-driven lip-shape research that requires a large number of video samples, it is difficult to manually correct the samples. Therefore, the algorithm has high requirements for the dataset; (2) In order to improve the generalization ability of the lip-sync model, the training set used by the Wav2Lip model contains different samples of people with a large range of facial features, which may result in low-quality lips and incorrect tooth information in the generated lip-shape video.

[0023] S101, acquire multiple single-person speaking videos, and segment each single-person speaking video to generate an initial dataset, wherein the original video length of the single-person speaking video is greater than 20 seconds; Specifically, in this embodiment, firstly, to ensure the model's accuracy in generating lip movements for non-specific individuals, this paper uses widely available single-person speaking videos from the internet as the dataset. These speaking videos should possess the following characteristics: single-person broadcasting; each original video segment is longer than 20 seconds; the subjects are adults (female), adults (male), children, and the elderly, totaling 12,000 videos, with a relatively balanced number of videos for each of the four types of individuals; the dataset primarily consists of face data frames. Since internet videos vary in length, each video in the dataset needs to be segmented to ensure effective model training. A script is used to automatically segment each video into short videos of approximately 10 seconds. Shorter videos facilitate rapid model training and allow for segmented observation of the loss function's changes during training.

[0024] S102, The initial dataset is optimized using the S3FD algorithm to generate a character dataset. The optimization process includes frame extraction and judgment processing. Specifically, step S102 includes: performing frame-by-frame extraction processing on the initial dataset using a preset script; The S3FD algorithm is used to determine the facial feature values ​​of different frames in order to remove images with large interference, and the data after removal is used as a person dataset.

[0025] Specifically, in this embodiment, S3FD is used to extract and judge video frames from the dataset. This optimization method can automatically generate a high-quality dataset, achieving the goal of removing noisy data and reducing interference with model training. S3FD is a deep learning-based face detection algorithm, characterized by its ability to detect faces of different scales using a single image, exhibiting consistent detection performance across different scales. This algorithm is based on a convolutional neural network architecture, utilizing multi-level feature representations and achieving target detection at different scales through changes in the receptive field. The principle of the S3FD algorithm is as follows: Figure 2 As shown.

[0026] In the S3FD algorithm, the convolutional neural network transforms the input into feature maps of different scales, trains the model using a multi-task loss function, and achieves good face detection results. The specific implementation is as follows:

[0027] in, It is a priori box index; It is a priori box It represents the predicted probability of a face. If the prior bounding box is positive, The truth label is 1 if true, otherwise 0. It is a vector of four parametric coordinates of the predicted bounding box. It is a vector of the four parameterized coordinates of the positive prior box, and the classification loss is... It uses the softmax loss for the face and background (softmax is a function that maps vectors to probability distributions), and the regression loss. This is the smoothed L1 loss. It's important to note here that these two terms are expressed in terms of sample size. and positive sample number Normalization is performed because the number of positive and negative prior boxes is unbalanced, so a parameter is introduced. To balance these two loss terms.

[0028] The algorithm steps are as follows: First, the collected and segmented dataset is extracted frame by frame using a script; second, the S3FD algorithm is used to determine the facial feature values ​​of different frames, removing images that significantly interfere with the training of the lip-sync model. In actual detection, these mainly include images whose faces are interfered with by hand waving, occluded by objects, or occluded by text; third, the cleaned data is used as the training dataset for the lip-sync model. The data cleaning effect is as follows. Figure 3 As shown, data cleaning based on the S3FD algorithm improves lip-sound synchronization, lip shape image quality, and model training time.

[0029] In this embodiment, the effect of dataset optimization on the algorithm model is measured by the following four parameters: lip-sync error with confidence (LSE-C), lip-sync error distance (LSE-D), fréchet inception distance (FID), and model training convergence time. LSE-C is used to detect lip-sync; it predicts the confidence level of synchronicity between lip movements and audio. Higher confidence levels indicate a better fit of the model. Its calculation formula is:

[0030] The confidence level is expressed as a unit time step. The calculation is the ratio of the number of samples whose lip readings match the true value to the total number of samples within a unit time step. The total length of the video sequence. A higher value indicates better lip-synchronization. The LSE-D parameter uses a displacement field to describe the deformation of lip movements. It calculates the average Euclidean distance of the lip contour line between adjacent frames of speech and video. The calculation formula is as follows:

[0031] in, For the truth value of the lip shape Frame motion features To generate the lip shape Frame motion features. A smaller LSE-D value indicates higher generated video quality. The FID parameter measures the distance between the generated and real images of the lip-sync video, based on Euclidean distance and the covariance matrix. Its calculation formula is:

[0032] The trace of a matrix is ​​the sum of the elements on its diagonal. and For real and generated images, This represents the mean. This is the covariance matrix. A smaller FID value indicates a closer similarity between the two distributions, resulting in higher quality generated images. Following the above formula, the LSE-C, LSE-D, FID, and training model convergence time before and after optimization are shown in Table 1: (The training model convergence time statistics in this paper are based on the dataset in Section 3.1 and a 3090Ti graphics card.) Table 1: Comparison of the effects of optimization based on S3FD algorithm

[0033] As shown in Table 1, LSE-C increased from 6.01 to 7.77 (a higher LSE-C value indicates better synchronization), LSE-D decreased from 6.99 to 6.82 (a lower LSE-D value indicates better synchronization), FID decreased from 5.18 to 5.01 (a lower FID value indicates better lip-sync image quality), and convergence time decreased from 288.20 hours to 173.60 hours (a shorter convergence time indicates higher algorithm efficiency). The conclusion is that all parameters have been optimized to some extent, especially LSE-C and model convergence time, which have been significantly improved. Using the S3FD face detection model to remove images with significant interference to faces in the dataset is an effective technique for improving lip-sync video.

[0034] S103, the trained ResNet-50 deep residual network model is called to preprocess the dataset of people to generate lip-sync videos. The ResNet-50 deep residual network model consists of 50 convolutional layers and fully connected layers, and includes four lip-sync sub-models for adult women, adult men, children and the elderly.

[0035] Specifically, step S103 includes: determining the feature values ​​of the people in the video to be generated in the person dataset, and calling the corresponding lip-sync sub-model according to the feature values; The operation steps for the adult female lip-sync sub-model, adult male lip-sync sub-model, child lip-sync sub-model, and elderly lip-sync sub-model are the same, specifically: The speech signals of the aforementioned dataset are converted into a frequency domain representation and then represented using Mel spectrum for better processing. Each speech frame is encoded using a pre-trained recurrent neural network model, wherein the recurrent neural network model is capable of converting each speech frame into a fixed-length vector representation; By using a facial keypoint detector, key points of the lips are extracted from the input video frames for subsequent lip shape synthesis; A lip-sync discriminator is generated by training a model using information about the position, shape, and pose of the lips. Each video frame is encoded using a pre-trained convolutional neural network model that can convert each video frame into a fixed-length vector representation. By concatenating video frame features and speech features to form a joint embedding vector, which can capture the correlation between speech and lips, the joint embedding vector is converted into the shape and posture of the lips by a decoder to form the final lip shape and generate lip-sync video.

[0036] Specifically, in this embodiment, before calling the trained ResNet-50 deep residual network model to preprocess the person dataset, the following steps are also included: Multiple solo speaking videos were acquired, each solo speaking video was segmented, and the S3FD algorithm was used to optimize the segmented solo speaking videos to generate a training dataset. Create an original ResNet-50 model, and train the original ResNet-50 model based on the preset VGGFace2 dataset to generate a pre-trained ResNet-50 model. The VGGFace2 dataset is a large-scale face recognition dataset. The pre-trained ResNet-50 model was used to classify people's gender and age on the training dataset, resulting in four sub-datasets: adult women, adult men, children, and the elderly. The lip-sync discriminator was trained using four subsets of datasets: adult women, adult men, children, and the elderly. This resulted in four lip-sync sub-models, which were then used to generate a ResNet-50 deep residual network model.

[0037] Specifically, in this embodiment, in the Wav2Lip algorithm, to improve the generalization ability of the lip-shape driven model, the dataset contains different samples of people with a wide range of facial features. Therefore, the accuracy of lip-shape fitting for specific people cannot be well solved, and the generated lip-shape video may have low-quality lip and teeth information errors. The proposed lip-sync video generation method addresses this problem by using a face classifier algorithm to generate lip-sync videos, thereby further improving the accuracy of generated lip shapes while ensuring the generalization ability of the overall model. A method based on the ResNet-50 algorithm for video gender and age recognition is proposed, classifying people into four typical types according to their feature values: adult (female), adult (male), child, and elderly. This generates a new lip-sync pre-trained model containing four sub-models for each type of person. When performing lip-shape generation, the ResNet-50 algorithm is used again to determine the person type, and the appropriate sub-model is called to generate the lip-shape image. Evaluation results show that this method can improve the generalization ability and accuracy of the overall model.

[0038] In this embodiment, the ResNet-50 deep residual network model consists of 50 convolutional layers and fully connected layers. It can extract high-level image features for image classification problems. Training on the VGGface2 dataset, which covers a wide range of ethnicities and ages, improves the model's robustness. The principle of the ResNet-50 model is as follows: Figure 4 As shown. From Figure 4 As can be seen, deepening the network with 50 convolutional layers and fully connected layers for feature extraction, and adding residual blocks and residual connections alleviates the gradient vanishing problem caused by excessive network depth, thus accelerating network training. After training on the VGGFace2 dataset, it can predict gender and age from photo input. During training, the binary cross-entropy loss for classification is used to calculate the loss between the true and predicted values, and the formula is as follows:

[0039] in, For binary tags 0 or 1, To predict the probability of the output value, The total number of samples is denoted as . By using a weighted average of the two loss functions, the predicted values ​​are made closer to the true values. At the same time, the Adam optimizer is used to dynamically adjust the learning rate to accelerate the model convergence speed, achieving better training results.

[0040] In this embodiment, the model is trained first. Firstly, using the VGGFace2 dataset, a ResNet-50 model is trained according to the principles and formulas in Section 3.1. A pre-trained ResNet-50 model is generated using the VGGFace2 dataset. VGGFace2 is a large-scale face recognition dataset containing 3.31 million images, with the following three characteristics: 1) It contains 9131 individuals, a relatively large number, and each individual has an average of 362.6 images; 2) It covers a wide range of poses, ages, and ethnicities; 3) The images generally have low noise. Secondly, using the pre-trained ResNet-50 model, a gender and age classifier is used to determine the gender of individuals in the dataset processed in Section 3.2, resulting in four subsets: adult (female), adult (male), child, and elderly. Finally, a lip-sync discriminator is trained using each of the four subsets, resulting in four lip-sync sub-models.

[0041] The trained model is then used to generate the dataset. Before image generation, a pre-trained ResNet-50 model is added to determine the feature values ​​of the people in the video to be generated. Based on the feature values, the corresponding lip-sync sub-model is called to finally generate the lip-sync video. The steps of the lip-sync sub-model are the same as those of the Wav2Lip model.

[0042] The Wav2Lip model is a widely used algorithm in the field of speech-driven lip-syncing. It consists of two deep learning models: lip-sound synchronization and lip-syncing video generation. Its aim is to synthesize speech and video signals into realistic and natural lip-sound synchronized video. The algorithm flow is as follows: Figure 5 As shown. The principle and process of the Wav2Lip model algorithm are as follows: (1) Preprocessing speech signals: Convert speech signals into frequency domain representations by applying Mel spectrum so that the model can process them better; (2) Extracting speech features: Each speech frame is encoded using a pre-trained recurrent neural network (RNN). This RNN model is often called a "speech encoder," which can convert each speech frame into a fixed-length vector representation; (3) Extracting lip key points: Using a facial key point detector, lip key points are extracted from the input video frames for subsequent lip shape synthesis. These key points represent information such as the position, shape, and pose of the lips; (4) Generate a lip-sync discriminator: Use the position, shape and pose information of the lips to train a lip-sync discriminator through model training; (5) Extracting video frame features: To extract the features of the lips in the video frames, a pre-trained convolutional neural network (CNN) is used to encode each video frame. This CNN model is often called a "visual encoder," which can convert each video frame into a fixed-length vector representation; (6) Joint Embedding: By concatenating video frame features and speech features, a joint embedding vector is formed, which can capture the correlation between speech and lips. Finally, a decoder converts the joint embedding vector into the shape and pose of the lips to form the final lip shape. This decoder is usually a deconvolutional neural network (DeCNN) or a generative adversarial network (GAN).

[0043] The model training yields synthesized lip shapes and voice, as well as lip-sync loss and image quality loss values. The core of the algorithm is to combine speech and lip information, using a joint embedding vector to predict lip shape.

[0044] Experimental evaluations were conducted on the proposed lip-sync video generation method and the existing Wav2Lip model algorithm. The lip-sync generation model based on a face classifier algorithm shows improvements in both subjective image quality and objective video evaluation parameters; the image generation effect is comparable to... Figure 6 As shown. By Figure 6 It can be seen that, for the same image input, the image generated by the Wav2Lip model in the upper right corner has obvious errors in lip shape information, while the image generated by the face classifier algorithm in the lower right corner has higher lip shape accuracy.

[0045] To measure the generation performance of the improved model, LSE-C and LSE-D were used to evaluate the lip-sync effect of the generated video, the FID metric was used to evaluate the video quality, and the stability of the model output quality was measured by the fluctuation of the model training loss value, where the loss value fluctuation was calculated using the mean-normalized mean square error formula. Evaluation Conclusion: As shown in Table 2, LSE-C increased from 7.77 to 7.81 (a higher LSE-C value indicates better synchronization), LSE-D decreased from 6.82 to 6.03 (a lower LSE-D value indicates better synchronization), FID decreased from 5.01 to 4.01 (a lower FID value indicates better lip-sync image quality), and the mean normalized variance of the loss value decreased from 0.1255 to 0.0041 (a smaller mean normalized variance of loss indicates less fluctuation and better quality stability; the significant decrease in mean normalized variance is due to better clustering of training data and fewer unexpected data). The conclusion is that all parameters have been optimized to some extent, especially FID and the mean normalized variance of loss. Classifying gender and age based on the face classifier algorithm allows for more concentrated facial features in individual sub-models, thereby improving the accuracy and quality of lip-sync video generation.

[0046] Table 2: Comparison of Lip Shape Generation Results Based on Face Classifier Algorithms

[0047] In summary, the proposed lip-sync video generation method employs a face detection model to remove noise from the dataset, effectively shortening the convergence time of model training. Specifically, the S3FD face detection model is used to remove noise from the dataset, significantly reducing the convergence time of the lip-sync model and improving lip-sync accuracy. Furthermore, the method proposes using the ResNet-50 algorithm to identify gender and age, generating lip-sync sub-models for different types of people. This method improves the model's accuracy while ensuring its overall generalization ability. Specifically, based on the ResNet-50 algorithm, gender and age are identified, and lip-sync sub-models for different types of people are generated, improving the quality of the generated lip images while ensuring the overall model's generalization ability. These improvements enable the model to fit various types of people well, and the videos generated by the algorithm closely resemble the actual person speaking. This technology has broad application value in the metaverse scenario. In short, the proposed lip-sync video generation method effectively improves upon the problems of lip-sync accuracy, lip quality, and model speed in existing technologies, resulting in more realistic and natural videos.

[0048] Please see Figure 7 A second embodiment of the invention provides a lip-sync video generation apparatus, comprising: The dataset acquisition unit 201 is used to acquire multiple single-person speaking videos, and to segment each single-person speaking video to generate an initial dataset, wherein the original video length of the single-person speaking video is greater than 20 seconds. The dataset optimization unit 202 is used to optimize the initial dataset using the S3FD algorithm to generate a character dataset. The optimization process includes frame extraction and judgment processing. The lip-sync video generation unit 203 is used to call the trained ResNet-50 deep residual network model to preprocess the person dataset and generate lip-sync videos. The ResNet-50 deep residual network model consists of 50 convolutional layers and fully connected layers, and includes four lip-sync sub-models for adult women, adult men, children and the elderly.

[0049] Preferably, the dataset optimization unit 202 is specifically used for: The initial dataset is processed frame by frame using a preset script; The S3FD algorithm is used to determine the facial feature values ​​of different frames in order to remove images with large interference, and the data after removal is used as a person dataset.

[0050] Preferably, the lip-sync video generation unit 203 is specifically used for: Determine the feature values ​​of the people in the video to be generated from the dataset, and call the corresponding lip-sync sub-model based on the feature values; The operation steps for the adult female lip-sync sub-model, adult male lip-sync sub-model, child lip-sync sub-model, and elderly lip-sync sub-model are the same, specifically: The speech signals of the aforementioned dataset are converted into a frequency domain representation and then represented using Mel spectrum for better processing. Each speech frame is encoded using a pre-trained recurrent neural network model, wherein the recurrent neural network model is capable of converting each speech frame into a fixed-length vector representation; By using a facial keypoint detector, key points of the lips are extracted from the input video frames for subsequent lip shape synthesis; A lip-sync discriminator is generated by training a model using information about the position, shape, and pose of the lips. Each video frame is encoded using a pre-trained convolutional neural network model that can convert each video frame into a fixed-length vector representation. By concatenating video frame features and speech features to form a joint embedding vector, which can capture the correlation between speech and lips, the joint embedding vector is converted into the shape and posture of the lips by a decoder to form the final lip shape and generate lip-sync video.

[0051] A third embodiment of the present invention provides a lip-sync video generation device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a lip-sync video generation method as described in any of the above embodiments.

[0052] A fourth embodiment of the present invention provides a readable storage medium storing a computer program that can be executed by a processor of the device in which the storage medium is located, to implement a lip-sync video generation method as described in any of the above embodiments.

[0053] Exemplary examples show that the computer program described in the third and fourth embodiments of the present invention can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the lip-sync video generation device. For example, the apparatus described in the second embodiment of the present invention.

[0054] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the lip-sync video generation method, connecting all parts of the method via various interfaces and lines.

[0055] The memory can be used to store the computer program and / or modules. The processor implements various functions of a lip-sync video generation method by running or executing the computer program and / or modules stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, text conversion function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, text message data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD card), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0056] If the implemented module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0057] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0058] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions that fall within the scope of the present invention are within the scope of protection of the present invention.

Claims

1. A method for generating lip-synced video, characterized in that, include: Multiple solo speaking videos are acquired, and each solo speaking video is segmented to generate an initial dataset, wherein the original video length of each solo speaking video is greater than 20 seconds. The initial dataset is optimized using the S3FD algorithm to generate a character dataset. The optimization process includes frame extraction and judgment processing. The trained ResNet-50 deep residual network model is used to preprocess the dataset of people to generate lip-sync videos. The ResNet-50 deep residual network model consists of 50 convolutional layers and fully connected layers, and includes four lip-sync sub-models for adult women, adult men, children and the elderly. Before calling the trained ResNet-50 deep residual network model to preprocess the aforementioned person dataset, the process also includes: Multiple solo speaking videos were acquired, each solo speaking video was segmented, and the S3FD algorithm was used to optimize the segmented solo speaking videos to generate a training dataset. Create an original ResNet-50 model, and train the original ResNet-50 model based on the preset VGGFace2 dataset to generate a pre-trained ResNet-50 model. The VGGFace2 dataset is a large-scale face recognition dataset. The pre-trained ResNet-50 model was used to classify people's gender and age on the training dataset, resulting in four sub-datasets: adult women, adult men, children, and the elderly. The lip-sync discriminator was trained using four subsets of datasets: adult women, adult men, children, and the elderly. This resulted in four lip-sync sub-models, which were then used to generate a ResNet-50 deep residual network model.

2. The lip-sync video generation method according to claim 1, characterized in that, The initial dataset is optimized using the S3FD algorithm to generate a people dataset, specifically as follows: The initial dataset is processed frame by frame using a preset script; The S3FD algorithm is used to determine the facial feature values ​​of different frames in order to remove images with large interference, and the data after removal is used as a person dataset.

3. The method for generating lip-sync video according to claim 1, characterized in that, The trained ResNet-50 deep residual network model is used to preprocess the aforementioned dataset to generate lip-sync videos, specifically: Determine the feature values ​​of the people in the video to be generated from the dataset, and call the corresponding lip-sync sub-model based on the feature values; The operation steps for the adult female lip-sync sub-model, adult male lip-sync sub-model, child lip-sync sub-model, and elderly lip-sync sub-model are the same, specifically: The speech signals of the aforementioned dataset are converted into a frequency domain representation and then represented using Mel spectrum for better processing. Each speech frame is encoded using a pre-trained recurrent neural network model, wherein the recurrent neural network model is capable of converting each speech frame into a fixed-length vector representation; By using a facial keypoint detector, key points of the lips are extracted from the input video frames for subsequent lip shape synthesis; A lip-sync discriminator is generated by training a model using information about the position, shape, and pose of the lips. Each video frame is encoded using a pre-trained convolutional neural network model that can convert each video frame into a fixed-length vector representation. By concatenating video frame features and speech features to form a joint embedding vector, which can capture the correlation between speech and lips, the joint embedding vector is converted into the shape and posture of the lips by a decoder to form the final lip shape and generate lip-sync video.

4. A lip-sync video generation device, characterized in that, include: The dataset acquisition unit is used to acquire multiple single-person speaking videos, and to segment each single-person speaking video to generate an initial dataset, wherein the original video length of the single-person speaking video is greater than 20 seconds. The dataset optimization unit is used to optimize the initial dataset using the S3FD algorithm to generate a character dataset. The optimization process includes frame extraction and judgment processing. The lip-sync video generation unit is used to call the trained ResNet-50 deep residual network model to preprocess the dataset of people and generate lip-sync videos. The ResNet-50 deep residual network model consists of 50 convolutional layers and fully connected layers, and includes four lip-sync sub-models for adult women, adult men, children and the elderly. Before calling the trained ResNet-50 deep residual network model to preprocess the aforementioned person dataset, the process also includes: Multiple solo speaking videos were acquired, each solo speaking video was segmented, and the S3FD algorithm was used to optimize the segmented solo speaking videos to generate a training dataset. Create an original ResNet-50 model, and train the original ResNet-50 model based on the preset VGGFace2 dataset to generate a pre-trained ResNet-50 model. The VGGFace2 dataset is a large-scale face recognition dataset. The pre-trained ResNet-50 model was used to classify people's gender and age on the training dataset, resulting in four sub-datasets: adult women, adult men, children, and the elderly. The lip-sync discriminator was trained using four subsets of datasets: adult women, adult men, children, and the elderly. This resulted in four lip-sync sub-models, which were then used to generate a ResNet-50 deep residual network model.

5. The lip-sync video generation device according to claim 4, characterized in that, The dataset optimization unit is specifically used for: The initial dataset is processed frame by frame using a preset script; The S3FD algorithm is used to determine the facial feature values ​​of different frames in order to remove images with large interference, and the data after removal is used as a person dataset.

6. The lip-sync video generation device according to claim 4, characterized in that, The lip-sync video generation unit is specifically used for: Determine the feature values ​​of the people in the video to be generated from the dataset, and call the corresponding lip-sync sub-model based on the feature values; The operation steps for the adult female lip-sync sub-model, adult male lip-sync sub-model, child lip-sync sub-model, and elderly lip-sync sub-model are the same, specifically: The speech signals of the aforementioned dataset are converted into a frequency domain representation and then represented using Mel spectrum for better processing. Each speech frame is encoded using a pre-trained recurrent neural network model, wherein the recurrent neural network model is capable of converting each speech frame into a fixed-length vector representation; By using a facial keypoint detector, key points of the lips are extracted from the input video frames for subsequent lip shape synthesis; A lip-sync discriminator is generated by training a model using information about the position, shape, and pose of the lips. Each video frame is encoded using a pre-trained convolutional neural network model that can convert each video frame into a fixed-length vector representation. By concatenating video frame features and speech features to form a joint embedding vector, which can capture the correlation between speech and lips, the joint embedding vector is converted into the shape and posture of the lips by a decoder to form the final lip shape and generate lip-sync video.

7. A lip-sync video generation device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement a lip-sync video generation method as described in any one of claims 1 to 3.

8. A readable storage medium, characterized in that, The device contains a computer program that can be executed by a processor of the device where the storage medium is located, to implement a lip-sync video generation method as described in any one of claims 1 to 3.

Images