Lip shape synchronization model training method, digital human video generation method and device

Abstract

The invention provides a training method of a lip shape synchronization model and a generation method and device of a digital human video. The invention relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, virtual reality, large models, digital human broadcast, digital human live broadcast and the like. According to the specific scheme, the method comprises the steps of determining a two-dimensional face feature truth value and a three-dimensional face feature truth value based on a video of a target object; extracting a three-dimensional face feature prediction value from the sample voice data through the first network sub-model; generating a two-dimensional face feature predicted value from the three-dimensional face feature predicted value through a second network sub-model; based on the two-dimensional face feature truth value, the three-dimensional face feature truth value, the two-dimensional face feature predicted value and the three-dimensional face feature predicted value, a first network sub-model and a second network sub-model are trained, a lip shape synchronization model is obtained, and the lip shape synchronization model comprises the first network sub-model and the second network sub-model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, virtual reality, large models, digital population broadcasting, digital person broadcasting and the like. BACKGROUND

[0002] Three-dimensional digital person lip synchronization needs the participation of professionals such as artists and modelers, and needs engine rendering, with high production cost. Two-dimensional digital person lip synchronization can be obtained through model reasoning, but the lip synchronization effect is poor. SUMMARY

[0003] The present disclosure provides a training method of a lip synchronization model, a generation method and device of a digital person video.

[0004] According to a first aspect of the present disclosure, a training method of a lip synchronization model is provided, comprising:

[0005] determining a two-dimensional face feature ground truth and a three-dimensional face feature ground truth based on a video of a target object;

[0006] extracting a three-dimensional face feature prediction value from sample speech data through a first network submodel;

[0007] generating a two-dimensional face feature prediction value from the three-dimensional face feature prediction value through a second network submodel;

[0008] training the first network submodel and the second network submodel based on the two-dimensional face feature ground truth, the three-dimensional face feature ground truth, the two-dimensional face feature prediction value and the three-dimensional face feature prediction value, to obtain a lip synchronization model, the lip synchronization model comprising the first network submodel and the second network submodel.

[0009] According to a second aspect of the present disclosure, a generation method of a digital person video is provided, comprising:

[0010] obtaining target speech data;

[0011] inputting the target speech data into the lip synchronization model to generate speech features through the first network submodel of the lip synchronization model, and generating three-dimensional face features based on the speech features; generating two-dimensional face features simulating the target object based on the three-dimensional face features through the second network submodel of the lip synchronization model; wherein the lip synchronization model is obtained by training the method of the first aspect;

[0012] generating a digital person video based on the two-dimensional face features and the target speech data.

[0013] According to a third aspect of the present disclosure, a training device of a lip synchronization model is provided, comprising:

[0014] determining a two-dimensional face feature ground truth and a three-dimensional face feature ground truth based on a video of the target object;

[0015] extracting a three-dimensional face feature prediction value from the sample speech data by the first network sub-model;

[0016] generating a two-dimensional face feature prediction value from the three-dimensional face feature prediction value by the second network sub-model;

[0017] training the first network sub-model and the second network sub-model based on the two-dimensional face feature ground truth, the three-dimensional face feature ground truth, the two-dimensional face feature prediction value, and the three-dimensional face feature prediction value to obtain a lip-sync model, the lip-sync model comprising the first network sub-model and the second network sub-model.

[0018] According to a fourth aspect of the present disclosure, a digital human video generation apparatus is provided, comprising:

[0019] an obtaining module configured to obtain target speech data;

[0020] an inputting module configured to input the target speech data into the lip-sync model to generate speech features by the first network sub-model of the lip-sync model, generate three-dimensional face features based on the speech features, and generate two-dimensional face features simulating the target object based on the three-dimensional face features by the second network sub-model of the lip-sync model; wherein the lip-sync model is trained by the method of the first aspect;

[0021] a second generating module configured to generate a digital human video based on the two-dimensional face features and the target speech data.

[0022] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the training method of the lip-sync model provided in the first aspect and / or the generation method of the digital human video provided in the second aspect.

[0023] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the training method of the lip-sync model provided in the first aspect and / or the generation method of the digital human video provided in the second aspect.

[0024] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program stored on a storage medium, which, when executed by a processor, implements the training method of the lip-sync model provided in the first aspect and / or the generation method of the digital human video provided in the second aspect.

[0025] According to the scheme of the present disclosure, the lip synchronization model can learn the digital human personalized features of the target object, and improve the synchronization effect of the trained model.

[0026] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0028] Figure 1 is a flowchart of a training method of a lip synchronization model according to an embodiment of the present disclosure;

[0029] Figure 2 is a schematic diagram of training a first network submodel according to an embodiment of the present disclosure;

[0030] Figure 3 is a schematic diagram of training a second network submodel according to an embodiment of the present disclosure;

[0031] Figure 4 is a test schematic diagram of a lip synchronization model according to an embodiment of the present disclosure;

[0032] Figure 5 is a schematic diagram of preparing true value data according to an embodiment of the present disclosure;

[0033] Figure 6 is a schematic diagram of generating a digital human video according to an embodiment of the present disclosure;

[0034] Figure 7 is a schematic diagram of generating a mouth broadcasting digital human video according to an embodiment of the present disclosure;

[0035] Figure 8 is a schematic diagram of generating an anchor digital human video according to an embodiment of the present disclosure;

[0036] Figure 9 is a flowchart of a digital human video generation method according to an embodiment of the present disclosure;

[0037] Figure 10 is a structural schematic diagram of a training device of a lip synchronization model according to an embodiment of the present disclosure;

[0038] Figure 11 is a structural schematic diagram of a digital human video generation device according to an embodiment of the present disclosure;

[0039] Figure 12is a scenario schematic diagram of a training method of a lip-sync model according to an embodiment of the disclosure;

[0040] Figure 13 is a scenario schematic diagram of a method for generating a digital human video according to an embodiment of the disclosure;

[0041] Figure 14 is a structural schematic diagram of an electronic device for implementing the training method of the lip-sync model and / or the method for generating a digital human video according to an embodiment of the disclosure. DETAILED DESCRIPTION

[0042] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Thus, those of ordinary skill in the art will recognize various changes and modifications of the embodiments described herein, which will not depart from the scope of the present disclosure. Also, descriptions of known functions and constructions are omitted in the following description for clarity and conciseness.

[0043] The terms "first", "second", and "third" and the like in the description of the specification embodiments and claims of the present disclosure and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a particular order or sequence. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, inclusion of a series of steps or units. The method, system, product, or device is not necessarily limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the processes, methods, products, or devices.

[0044] In related technologies, there are mainly two kinds of two-dimensional digital human lip-sync. The first method is trained through an end-to-end network, and the obtained lip-sync is only a general lip shape result, without fine-tuning training for a specific digital human image, so as to fail to reflect the individualized features of each different digital human, such as mouth shape, speaking rhythm, and expression, etc. The second method learns not only the individualized features of the digital human, but also the individualized features of the video, such as the face orientation and position change of the digital human in the input video, etc., and fails to decouple the individualized features of the digital human and the individualized features of the video. When another video of the same digital human is tested, the output result in the test stage cannot be aligned with the new video.

[0045] The present disclosure proposes a lip synchronization method that can learn both digital human personalized features and decouple video personalized features, to at least partially solve one or more of the above problems and other potential problems. The lip synchronization model learns the digital human personalized features of the target object based on the video of the target object to obtain training data true values, so as to improve the synchronization effect of the trained model. The first network sub-model extracts three-dimensional face feature prediction values from sample speech data, and the second network sub-model generates two-dimensional face feature prediction values from the three-dimensional face feature prediction values, thereby improving the training accuracy of the model.

[0046] The present disclosure provides a lip synchronization model training method, Figure 1 is a flowchart of the lip synchronization model training method according to an embodiment of the present disclosure. The lip synchronization model training method can be applied to a lip synchronization model training device. The lip synchronization model training device is located in an electronic device. The electronic device includes but is not limited to a fixed device and / or a mobile device. For example, the fixed device includes but is not limited to a server, which can be a cloud server or a general server. For example, the mobile device includes but is not limited to a mobile phone, a tablet computer, etc. In some possible implementation manners, the lip synchronization model training method can also be realized by a processor calling computer readable instructions stored in a memory. As shown in Figure 1 The lip synchronization model training method includes:

[0047] S101: determining two-dimensional face feature true values and three-dimensional face feature true values based on a video of a target object; wherein the target object is an object simulated by a digital human;

[0048] S102: extracting three-dimensional face feature prediction values from sample speech data by a first network sub-model;

[0049] S103: generating two-dimensional face feature prediction values from the three-dimensional face feature prediction values by a second network sub-model;

[0050] S104: training the first network sub-model and the second network sub-model based on the two-dimensional face feature true values, the three-dimensional face feature true values, the two-dimensional face feature prediction values, and the three-dimensional face feature prediction values to obtain a lip synchronization model, wherein the lip synchronization model includes the first network sub-model and the second network sub-model.

[0051] In the present disclosure, the target object is an object simulated by a digital human. For example, the target object can be a specific person or a certain type of person.

[0052] In the present disclosure, the digital human is a virtual character created in computer graphics, which can simulate the actions and expressions of real humans.

[0053] In the embodiments of the present disclosure, the two-dimensional face feature ground truth is two-dimensional face feature data directly extracted from the video of the target object, serving as the standard answer of the two-dimensional face feature in the model training.

[0054] In the embodiments of the present disclosure, the three-dimensional face feature ground truth is feature data of a three-dimensional face model obtained through three-dimensional scanning or reconstruction technology, serving as the standard answer of the three-dimensional face feature in the training process.

[0055] In the embodiments of the present disclosure, the first network sub-model is a deep learning model responsible for predicting three-dimensional face features from speech data.

[0056] In the embodiments of the present disclosure, the second network sub-model is a deep learning model responsible for converting the three-dimensional face feature prediction value into a two-dimensional face feature prediction value.

[0057] In some embodiments, advanced face recognition technology is used to extract two-dimensional face feature ground truth from video data, such as using the Open Source Computer Vision Library (OpenCV), an open source library containing machine learning algorithms for face detection and feature point positioning.

[0058] In some embodiments, three-dimensional face models are generated from videos or photos using three-dimensional scanning or three-dimensional reconstruction technology, and three-dimensional face feature ground truth is extracted therefrom.

[0059] In some embodiments, the task of the first network sub-model is to extract three-dimensional face feature prediction values from sample speech data. For example, the first network sub-model can be a deep learning model, such as a Convolutional Neural Network (CNN) or a Recurrent Neural Network (RNN), which learns the complex relationship from speech feature mapping to three-dimensional face action.

[0060] In some embodiments, the second network sub-model is responsible for generating two-dimensional face feature prediction values from the three-dimensional face feature prediction values output by the first network sub-model. The second network sub-model is a deep learning model that learns the mapping of three-dimensional face features to two-dimensional face features.

[0061] In some embodiments, the true value data and the predicted data are input into a loss function, such as Mean Squared Error (MSE) or other suitable loss function, using a backpropagation algorithm and an optimizer (such as gradient descent) to update the network weights to minimize the difference between the predicted values and the true values. This process is repeated until the model performance reaches a preset standard or the number of training rounds reaches an upper limit. For example, the performance of the model is evaluated on an independent validation set, and necessary model adjustments or optimizations are made based on the evaluation results.

[0062] In some embodiments, the trained lip-sync model is integrated into a digital human system; when the system receives a voice input, the lip-sync model is used to generate corresponding lip animation.

[0063] In the embodiments of the present disclosure, the training method of the lip-sync model is not only applicable to target objects such as specific characters, but also can be adapted to different characters by changing the data set.

[0064] Suppose we want to create a digital human model for an actor. We collect multiple videos of the actor and corresponding voice data. Through the above training method, we can obtain a lip-sync model that can simulate the changes in the actor's lips when speaking. When this model is integrated into a digital human system, the digital human can mimic the actor's lip movements in a very realistic way.

[0065] The technical solutions of the embodiments of the present disclosure determine two-dimensional face feature true values and three-dimensional face feature true values based on the video of the target object; extract three-dimensional face feature predicted values from the sample voice data through the first network sub-model; generate two-dimensional face feature predicted values from the three-dimensional face feature predicted values through the second network sub-model; train the first network sub-model and the second network sub-model based on the two-dimensional face feature true values, the three-dimensional face feature true values, the two-dimensional face feature predicted values, and the three-dimensional face feature predicted values to obtain a lip-sync model; in this way, the lip-sync model trained in the above manner can generate lip animation highly consistent with the target object, improving the realism of the digital human. The lip-sync model can adapt to different voice inputs and generate corresponding lip changes, making the expression of the digital human more natural and lively.

[0066] In some embodiments, extracting three-dimensional face feature predicted values from the sample voice data through the first network sub-model comprises: extracting voice features from the sample voice data through a first branch network in the first network sub-model; predicting three-dimensional face feature predicted values from the voice features through a second branch network in the second network sub-model.

[0067] In some embodiments, in the first network sub-model, the output end of the first branch network is connected to the input end of the second branch network.

[0068] In some embodiments, the network structure of the first branch network is a speech feature extraction model, and the network structure of the second branch network is a three-dimensional face feature extraction model.

[0069] In some implementations, the speech feature extraction model based on self-supervised learning is used as the first branch network, which can directly extract high-level speech features from the original speech waveform without any labels or supervision information.

[0070] In some implementations, the deep learning model based on self-attention mechanism is used as the second branch network, which can process sequence data and capture long-range dependencies in the data.

[0071] In some implementations, during the training process, the first branch network and the second branch network are jointly trained. The weights of the first branch network and the second branch network can be updated simultaneously to minimize the difference between the predicted three-dimensional face features and the true values. A suitable loss function is used to measure the gap between the predicted values and the true values, and the network parameters are updated through backpropagation and optimization algorithms.

[0072] For example, suppose there is a speech data of someone's speech, and it is desired to generate a corresponding three-dimensional face animation according to the speech. By using the first branch network in the first network sub-model to extract speech features, and then inputting these speech features into the second branch network in the first network sub-model, the predicted three-dimensional face features that match the speech can be obtained. These three-dimensional face feature predictions can then be used to drive the three-dimensional face model of the digital person to generate realistic lip animation.

[0073] The technical solutions of the embodiments of the present disclosure, the first branch network in the first network sub-model can automatically learn meaningful feature representations from the original speech without the need for artificial design of features or reliance on a large amount of labeled data. The second branch network in the first network sub-model effectively captures the complex relationship between speech features and three-dimensional face features using the self-attention mechanism, improving the accuracy of the predictions. The entire process is end-to-end, i.e., the three-dimensional face feature predictions are obtained directly from the original speech data, and the first network sub-model does not require intermediate steps or human intervention, improving the efficiency of the training.

[0074] In the embodiments of the present disclosure, the network structure of the first branch network is a speech feature extraction model based on self-supervised learning, and the network structure of the second branch network is a deep learning model based on self-attention mechanism.

[0075] In some embodiments, the first branch network can be an unsupervised pre-training speech recognition model (Wav2vec) for learning high-level feature representations from raw speech waveforms.

[0076] In some embodiments, the second branch network can be a Transformer model for processing sequential data and capable of capturing long-range dependencies in the data. In this embodiment, the second branch network is used to learn the mapping relationship between speech features and three-dimensional face features.

[0077] In some embodiments, the sample speech data is input into the Wav2vec model, and through multiple convolutional layers and self-attention mechanisms of the model, a feature vector containing rich speech information is extracted.

[0078] In some embodiments, the speech feature vector extracted from the Wav2vec model is input into the Transformer model. The Transformer model learns the complex mapping relationship between speech features and three-dimensional face features through self-attention mechanisms and outputs a three-dimensional face feature prediction value.

[0079] For example, suppose there is a piece of speech data performed by someone, and it is desired to generate a corresponding three-dimensional face animation according to this speech. By using the Wav2vec model to extract speech features and then inputting these speech features into the Transformer model, a three-dimensional face feature prediction value matching the speech can be obtained. These three-dimensional face feature prediction values can then be used to drive the three-dimensional face model of the digital person to generate realistic lip animation.

[0080] The technical solution of the embodiments of the present disclosure is that the Wav2vec model can automatically learn meaningful feature representations from raw speech without the need for artificial design of features or reliance on a large amount of labeled data. The Transformer model effectively captures the complex relationship between speech features and three-dimensional face features using self-attention mechanisms, improving the accuracy of the prediction. The entire process is end-to-end, i.e., directly obtaining a three-dimensional face feature prediction value from raw speech data without intermediate steps or human intervention, improving the speed of training.

[0081] In the embodiments of the present disclosure, generating a two-dimensional face feature prediction value from the three-dimensional face feature prediction value through the second network sub-model includes: generating a two-dimensional face feature prediction value from the three-dimensional face feature prediction value through a third branch network in the second network sub-model.

[0082] In some embodiments, the first network sub-model obtains a three-dimensional facial feature prediction value. The three-dimensional facial feature prediction value will be used as input of the second network sub-model.

[0083] In some embodiments, the U-shaped neural network model is trained to learn the mapping from three-dimensional features to two-dimensional features using labeled two-dimensional facial feature data as supervision information. A suitable loss function, such as mean square error or cross-entropy loss, is selected to measure the difference between the predicted value and the true two-dimensional facial feature. The parameters of the U-shaped neural network model are updated through backpropagation and optimization algorithms such as gradient descent to minimize the prediction error.

[0084] In some embodiments, after the model training is completed, the three-dimensional facial feature prediction value is input into the trained U-shaped neural network model. Through the forward propagation of the U-shaped neural network model, the prediction value of the two-dimensional facial feature is obtained.

[0085] For example, suppose there is a three-dimensional facial data set and we want to generate corresponding two-dimensional facial images from these data. First, we use the first network sub-model to extract three-dimensional facial feature prediction values from the speech data. Then, these three-dimensional features are input into the third branch network (U-shaped neural network model). Through the conversion of the U-shaped neural network model, we obtain the two-dimensional facial feature prediction value corresponding to the original three-dimensional facial data. These two-dimensional features can be used to generate realistic two-dimensional facial images, or for subsequent tasks such as face recognition, expression analysis, etc.

[0086] The technical solution of the embodiments of the present disclosure can effectively convert three-dimensional facial features into two-dimensional facial features while preserving key facial information and details. The second network sub-model can learn complex nonlinear mapping relationships, thereby achieving efficient feature conversion and accurate prediction.

[0087] In the embodiments of the present disclosure, the network structure of the third branch network adopts a U-shaped neural network model.

[0088] In the embodiments of the present disclosure, in the second network sub-model, the input end of the third branch network is connected to the output end of the second branch network in the first network sub-model.

[0089] In the embodiments of the present disclosure, the lip shape synchronization model includes a first network sub-model and a second network sub-model, the first network sub-model includes a first branch network and a second branch network, and the second network sub-model includes a third branch network; the output end of the first branch network is connected to the input end of the second branch network, and the output end of the second branch network is connected to the input end of the third branch network.

[0090] Here, the U-Net is a classic convolutional neural network architecture, particularly suitable for image segmentation and feature transformation tasks. In the U-Net model, a series of convolution, pooling and up-sampling operations can effectively map the input three-dimensional face features to a two-dimensional face feature space. It consists of a contraction path (encoder) and an expansion path (decoder), with information transmission between the two through a jump connection. U-Net can effectively capture the context information in the image and generate high-quality segmentation or feature transformation results. In this example, U-Net is used to convert three-dimensional face features to two-dimensional face features.

[0091] For example, suppose we have a three-dimensional face dataset and want to generate corresponding two-dimensional face images from the data. First, we use the first network sub-model to extract three-dimensional face feature prediction values from the speech data. Then, input these three-dimensional features into the third branch network (U-Net model). Through the conversion of the U-Net model, we get the two-dimensional face feature prediction values corresponding to the original three-dimensional face data. These two-dimensional features can be used to generate realistic two-dimensional face images, or for subsequent tasks such as face recognition, expression analysis, etc.

[0092] The technical solution of the embodiments of the present disclosure, the U-Net model can effectively convert three-dimensional face features to two-dimensional face features while preserving key face information and details. Through deep learning technology, U-Net can learn complex nonlinear mapping relationships, thereby achieving efficient feature conversion and accurate prediction.

[0093] In the embodiments of the present disclosure, based on the two-dimensional face feature true value, the three-dimensional face feature true value, the two-dimensional face feature prediction value and the three-dimensional face feature prediction value, the first network sub-model and the second network sub-model are trained, including: based on the three-dimensional face feature true value and the three-dimensional face feature prediction value, constructing a first loss function; training the first network sub-model based on the first loss function; based on the two-dimensional face feature true value and the two-dimensional face feature prediction value, constructing a second loss function; training the second network sub-model based on the second loss function.

[0094] In some embodiments, the first network sub-model and the second network sub-model can be trained simultaneously, such as jointly training the first network sub-model and the second network sub-model.

[0095] In some embodiments, the first network sub-model and the second network sub-model can be trained separately, such as first training the first network sub-model and then training the second network sub-model.

[0096] In some embodiments, a dataset containing 2D facial feature ground truth and 3D facial feature ground truth is collected and prepared. These data are usually from high-precision facial scanning or capturing systems. Necessary preprocessing is performed on the data, such as normalization, denoising, etc., to ensure the quality and consistency of the data.

[0097] In some embodiments, for the first network submodel, a first loss function is constructed based on the 3D facial feature ground truth and the 3D facial feature prediction. The first loss function includes but is not limited to MSE or Mean Absolute Error (MAE), which is used to measure the difference between the prediction and the ground truth.

[0098] In some embodiments, for the second network submodel, a second loss function is constructed based on the 2D facial feature ground truth and the 2D facial feature prediction. Similarly, loss functions such as MSE or MAE can be used.

[0099] In some embodiments, the first network submodel is trained using the first loss function. The weights and parameters of the model are updated through the backpropagation algorithm and the optimizer (such as gradient descent, Adam, etc.) to minimize the first loss function. Similarly, the second network submodel is trained using the second loss function. The parameters of the model are adjusted through the optimization algorithm to reduce the difference between the 2D facial feature prediction and the ground truth.

[0100] In some embodiments, after training the two submodels separately, joint training can be considered to further optimize the performance of the entire system. In the joint training process, both the first loss function and the second loss function can be considered, and the weights of the two are adjusted to balance the training of the two tasks.

[0101] In some embodiments, an independent validation set or test set is used to evaluate the performance of the model. The error indicators between the prediction and the ground truth are calculated, such as accuracy, precision, recall, etc. According to the evaluation results, the model is fine-tuned to improve its prediction performance and generalization ability.

[0102] Figure 2 A schematic diagram of training the first network submodel is shown as follows: Figure 2 As shown, the input speech is processed by the first branch network of the first network submodel, such as Wav2vec, to extract the speech features, and the second branch network of the first network submodel, such as Transformer, is used to realize the conversion of the speech features to 3D facial features, and the 3D facial feature prediction is output. Based on the 3D facial feature prediction and the 3D facial feature ground truth, the first network submodel is trained based on the first loss function, such as the Mean Absolute Error loss function (also known as the L1 loss function).

[0103] Figure 3 A schematic diagram of training the second network sub-model is shown as follows: Figure 3 As shown, input the three-dimensional face feature prediction value, and use the third branch network of the second network sub-model, such as a U-shaped neural network model, to extract the two-dimensional face feature prediction value; based on the two-dimensional face feature prediction value and the two-dimensional face feature true value, the second network sub-model is trained based on the second loss function, such as a generative adversarial network (GAN) loss function.

[0104] Figure 4 A test schematic diagram of the lip synchronization model is shown as follows: Figure 4 As shown, input the speech, obtain the three-dimensional face feature through the first network sub-model of the lip synchronization model, and then obtain the two-dimensional face feature consistent with the input speech through the second network sub-model of the lip synchronization model.

[0105] For example, suppose we have a dataset containing speech, three-dimensional face features, and two-dimensional face features. We first train two network sub-models using the first loss function and the second loss function, respectively. During the training process, we continuously adjust the parameters of the model to minimize the value of the loss function. After training is completed, we can use a validation set to evaluate the performance of the model. If we find that the performance of a certain sub-model is poor, we can fine-tune the sub-model or add more training data to improve its performance. Finally, we obtain a model system that can accurately predict two-dimensional and three-dimensional face features from speech data.

[0106] The technical solutions of the embodiments of the present disclosure can significantly improve the prediction accuracy of two-dimensional and three-dimensional face features by constructing loss functions for specific tasks and training corresponding network sub-models. By jointly training two network sub-models, the system can learn the mapping relationship from speech to three-dimensional face features and from three-dimensional face features to two-dimensional face features, enhancing the multi-task learning ability of the model.

[0107] In the embodiments of the present disclosure, determining the two-dimensional face feature true value and the three-dimensional face feature true value based on the video of the target object includes: performing video decoding based on the video to obtain a video decoding result; performing face detection based on the video decoding result to obtain a face detection result; performing face cropping based on the face detection result to obtain the two-dimensional face feature true value; and performing face reconstruction based on the face detection result to obtain the three-dimensional face feature true value.

[0108] In some embodiments, a suitable video decoder is used to decode the video file, converting it into a processable image sequence, i.e., a video decoding result.

[0109] In some implementations, for each frame of the image obtained after video decoding, a face detection algorithm (such as a cascaded classifier, a deep learning model, etc.) is applied to identify the face region in the image.

[0110] In some implementations, the face detection algorithm outputs the location information of the face, such as the bounding box coordinates, as the face detection result.

[0111] In some implementations, the face region is cropped from the original image based on the bounding box coordinates in the face detection results. The cropped face image can serve as a ground truth of two-dimensional face features, directly reflecting the appearance features of the face in the video.

[0112] In some implementations, 3D face reconstruction techniques (such as Shape From Shading (SFS) algorithm, 3D deformable models, etc.) are used to process the face detection results. By analyzing information such as the texture, shape, and lighting of the face, the 3D structure of the face is reconstructed. The reconstructed 3D face model contains the depth information and geometry of the face and can be used as the ground truth of 3D face features.

[0113] Figure 5 A schematic diagram of truth data preparation is shown, such as... Figure 5 As shown, the input is a video of the target object, and the length of the video is not less than a certain threshold. During data preprocessing, the video is first decoded using an audio / video codec tool (Fast Forward Moving Picture Experts Group, FFmpeg), then face detection is performed, and face cropping is performed based on the face detection results to obtain cropped 2D face features; finally, 3D face reconstruction is performed based on the face detection results to obtain 3D face features.

[0114] For example, suppose a user records a video and wants to generate a digital human-like video simulating themselves. First, the video is decoded, resulting in a series of image frames. Then, face detection is performed on each frame to locate the user's face region. Next, a 2D face image is cropped based on the face detection results and saved as 2D face feature ground truth. Finally, 3D face reconstruction technology is used to reconstruct the detected face, obtaining a 3D face model of the user, which is saved as 3D face feature ground truth. These extracted 2D and 3D face feature ground truths will be used as the basis for generating a customized lip-sync model for the user.

[0115] The technical solutions of the embodiments of the present disclosure can accurately identify and extract the face region from the video through professional video decoding and face detection technology, and provide accurate data basis for subsequent feature extraction and reconstruction. Through the three-dimensional face reconstruction technology, more abundant three-dimensional geometric information than two-dimensional images can be obtained, which helps to more comprehensively understand and analyze the face features.

[0116] Figure 6 A generation schematic diagram of a digital human video is shown as Figure 6 shown, input the video of the target object into the to-be-trained model to obtain a trained model; input the voice into the trained model to obtain a digital human video simulating the target object. Since the video of each target object is trained, the individualized features of each different target object, such as mouth shape, speaking rhythm, expression, etc., can be learned, so that the lip synchronization effect in the digital human video is more natural and realistic.

[0117] The method described in the embodiments of the present disclosure can be applied to multiple fields such as face recognition, expression analysis, virtual reality, game production, and digital human video generation, and provides real and accurate face feature data for various application scenarios.

[0118] The embodiments of the present disclosure provide a digital human video generation method. Figure 7 is a flowchart of the digital human video generation method according to the embodiments of the present disclosure, which can be applied to a digital human video generation device. The digital human video generation device is located in an electronic device. The electronic device includes but is not limited to fixed devices and / or mobile devices. For example, the fixed device includes but is not limited to a server, which can be a cloud server or a general server. For example, the mobile device includes but is not limited to a mobile phone, a tablet computer, a notebook computer, etc. In some possible implementation manners, the digital human video generation method can also be realized by the way of calling the computer readable instructions stored in the memory by the processor. As Figure 7 shown, the digital human video generation method includes:

[0119] S701: obtaining target voice data;

[0120] S702: inputting the target voice data into a lip synchronization model to generate voice features through a first network submodel of the lip synchronization model, generating three-dimensional face features based on the voice features, and generating two-dimensional face features simulating the target object based on the three-dimensional face features through a second network submodel of the lip synchronization model;

[0121] S703: generating a digital human video based on the two-dimensional face features and the target voice data.

[0122] The lip synchronization model is obtained by training through any one of the methods described above.

[0123] In some embodiments, the lip-sync model is used to generate facial expressions and mouth shape features that are synchronized with the input speech data.

[0124] In some embodiments, the first network sub-model is part of the lip-sync model responsible for extracting features from the speech data and generating three-dimensional face features.

[0125] In some embodiments, the second network sub-model is another part of the lip-sync model responsible for converting the three-dimensional face features into two-dimensional face features for video generation.

[0126] In some embodiments, a digital human video is a video containing a digital human character, where the digital human's facial expressions and mouth shape changes are synchronized with speech data.

[0127] In some embodiments, the target speech data is obtained from a recording device, an audio file, or other sources. These data can be any form of audio, such as speeches, conversations, or singing, etc.

[0128] In some embodiments, the obtained target speech data is input into a pre-trained lip-sync model. The lip-sync model contains two main network sub-models: the first network sub-model and the second network sub-model. The target speech data is processed by the first network sub-model of the lip-sync model to generate corresponding speech features. Based on these speech features, the first network sub-model further generates three-dimensional face features that capture the facial expressions and mouth shape changes corresponding to the target speech. In some embodiments, the second network sub-model of the lip-sync model receives the three-dimensional face features as input and generates two-dimensional face features that simulate the target object. These two-dimensional face features accurately reflect the subtle changes in mouth shape and other facial expressions synchronized with the target speech.

[0129] In some embodiments, the generated two-dimensional face features and target speech data are combined to create a digital human video using video synthesis techniques such as face replacement technology or animation rendering engines. In this video, the digital human's facial expressions and mouth shape changes are perfectly synchronized with the target speech data, presenting a natural communication effect.

[0130] For example, suppose a game development company wants to create a promotional video for its new game, which contains a dialogue segment of a digital human character. They can use this method to first obtain the speech data of a professional voice actor and the digital human character, then generate two-dimensional face features that simulate the digital human character synchronized with the speech through the lip-sync model. Finally, combine these two-dimensional face features and speech data to generate a high-quality digital human promotional video, where the digital human's mouth shape changes and facial expressions perfectly match the voice.

[0131] The method for generating a digital human video according to the embodiments of the present disclosure can be applied to many fields such as virtual character production, games, movie trailers, advertisements, etc., and provides a powerful tool for creative content production.

[0132] The technical solution of the embodiments of the present disclosure generates a digital human video with a precise lip shape synchronization model, which not only has facial expressions and lip shape changes of the target object that are highly matched with the target voice, significantly improving the realism of the video, but also can quickly generate a digital human video synchronized with the voice, greatly improving the production efficiency.

[0133] Figure 8 A generation schematic diagram of a lip-synch digital human video is shown as shown in Figure 8 The product introduction and promotion voice is input into the trained model to obtain a lip-synch digital human video. The lip-synch digital human video can be used for product introduction and promotion.

[0134] Figure 9 A generation schematic diagram of a live digital human video is shown as shown in Figure 9 The answer corresponding to the user question is obtained in combination with a knowledge question and answer system; the answer is synthesized into voice in combination with text-to-speech (TTS) technology, and then input into the trained model to generate a live digital human video, which answers the user's question online and forms an interactive feedback with the user.

[0135] It should be understood that Figures 2 to 6 , Figures 8 to 9 The schematic diagram shown is only exemplary and not limiting, and it is extensible, and those skilled in the art can make various obvious changes and / or replacements based on the examples of Figures 2 to 6 , Figures 8 to 9 The resulting technical solution still belongs to the disclosure range of the embodiments of the present disclosure.

[0136] The embodiments of the present disclosure provide a training device of a lip shape synchronization model, as shown in Figure 10 The training device of the lip shape synchronization model can include: a determination module 1001 configured to determine two-dimensional face feature true values and three-dimensional face feature true values based on a video of a target object; an extraction module 1002 configured to extract three-dimensional face feature predicted values from sample voice data through a first network sub-model; a first generation module 1003 configured to generate two-dimensional face feature predicted values from the three-dimensional face feature predicted values through a second network sub-model; and a training module 1004 configured to train the first network sub-model and the second network sub-model based on the two-dimensional face feature true values, the three-dimensional face feature true values, the two-dimensional face feature predicted values, and the three-dimensional face feature predicted values to obtain a lip shape synchronization model, wherein the lip shape synchronization model includes the first network sub-model and the second network sub-model.

[0137] In some embodiments, the extraction module 1002 is configured to: extract speech features from the sample speech data through a first branch network in the first network submodel; and predict the three-dimensional face feature prediction value from the speech features through a second branch network in the second network submodel.

[0138] In some embodiments, the network structure of the first branch network is a speech feature extraction model based on self-supervised learning, and the network structure of the second branch network is a deep learning model based on self-attention mechanism.

[0139] In some embodiments, the generation module 1003 is configured to: generate a two-dimensional face feature prediction value from the three-dimensional face feature prediction value through a third branch network in the second network submodel.

[0140] In some embodiments, the network structure of the third branch network is a U-shaped neural network model.

[0141] In some embodiments, the training module 1004 is configured to: construct a first loss function based on the three-dimensional face feature ground truth and the three-dimensional face feature prediction value; train the first network submodel based on the first loss function; construct a second loss function based on the two-dimensional face feature ground truth and the two-dimensional face feature prediction value; and train the second network submodel based on the second loss function.

[0142] In some embodiments, the determination module 1001 is configured to: perform video decoding based on the video to obtain a video decoding result; perform face detection based on the video decoding result to obtain a face detection result; perform face cropping based on the face detection result to obtain the two-dimensional face feature ground truth; and perform face reconstruction based on the face detection result to obtain the three-dimensional face feature ground truth.

[0143] Those skilled in the art should understand that the functions of the processing modules in the training device of the lip shape synchronization model of the embodiments of the present disclosure can be understood with reference to the foregoing description of the training method of the lip shape synchronization model, and the processing modules in the training device of the lip shape synchronization model of the embodiments of the present disclosure can be implemented by generating circuits that implement the functions of the embodiments of the present disclosure, or by running software that implements the functions of the embodiments of the present disclosure on an electronic device.

[0144] The training device of the lip shape synchronization model of the embodiments of the present disclosure can generate a lip shape animation highly consistent with a target object, thereby improving the realism of a digital person. The lip shape synchronization model can adapt to different speech inputs and generate corresponding lip shape changes, making the expression of the digital person more natural and lively.

[0145] The embodiments of the present disclosure provide a digital person video generation device, which comprises a receiving module, a determination module, an extraction module, a generation module, and a training module. Figure 11As shown, the obtaining module 1101 is configured to obtain target voice data; the input module 1102 is configured to input the target voice data into a lip synchronization model to generate voice features through a first network submodel of the lip synchronization model, generate three-dimensional face features based on the voice features, generate two-dimensional face features simulating a target object based on the three-dimensional face features through a second network submodel of the lip synchronization model; wherein the lip synchronization model is obtained by training through any one of the methods described above; and the second generation module 1103 is configured to generate a digital human video based on the two-dimensional face features and the target voice data.

[0146] Those skilled in the art should understand that the functions of the processing modules in the digital human video generation apparatus of the embodiments of the present disclosure can be understood with reference to the related descriptions of the aforementioned digital human video generation method. The processing modules in the digital human video generation apparatus of the embodiments of the present disclosure can be implemented by generation circuits that implement the functions of the embodiments of the present disclosure, or can be implemented by software that implements the functions of the embodiments of the present disclosure running on an electronic device.

[0147] The digital human video generation apparatus of the embodiments of the present disclosure can make the generated digital human video not only have facial expressions and mouth shape changes of the target object that are highly matched with the target voice, significantly improving the realism of the video, but also can quickly generate a digital human video that is synchronized with the voice, greatly improving the production efficiency.

[0148] The embodiments of the present disclosure provide a scene schematic diagram of a training method of a lip synchronization model, as shown in Figure 12 As described above, the training method of the lip synchronization model provided by the embodiments of the present disclosure is applied to an electronic device. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices.

[0149] Specifically, the electronic device can specifically perform the following operations:

[0150] determining two-dimensional face feature ground truths and three-dimensional face feature ground truths based on a video of a target object, the target object being an object simulated by a digital human;

[0151] extracting three-dimensional face feature prediction values from sample voice data through the first network submodel;

[0152] generating two-dimensional face feature prediction values from the three-dimensional face feature prediction values through the second network submodel;

[0153] The first network sub-model and the second network sub-model are trained based on the two-dimensional face feature ground truth, the three-dimensional face feature ground truth, the two-dimensional face feature prediction value, and the three-dimensional face feature prediction value to obtain a lip synchronization model. The lip synchronization model includes the first network sub-model and the second network sub-model.

[0154] It should be understood that Figure 12 The scenario diagram shown is merely illustrative and not limiting. Those skilled in the art can make various obvious changes and / or replacements based on the examples Figure 12 The resulting technical solutions still belong to the disclosure range of the embodiments of the present disclosure.

[0155] The embodiments of the present disclosure also provide a scenario diagram of a digital human video generation method, as shown in Figure 13 As described above, the digital human video generation method provided by the embodiments of the present disclosure is applied to an electronic device. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices.

[0156] Specifically, the electronic device can specifically perform the following operations:

[0157] Obtain target speech data:

[0158] Input the target speech data into the lip synchronization model to generate speech features through the first network sub-model of the lip synchronization model, generate three-dimensional face features based on the speech features, generate two-dimensional face features simulating a target object based on the three-dimensional face features through the second network sub-model of the lip synchronization model. The lip synchronization model is obtained by training the method of any one of the above;

[0159] Generate a digital human video based on the two-dimensional face features and the target speech data.

[0160] It should be understood that Figure 13 The scenario diagram shown is merely illustrative and not limiting. Those skilled in the art can make various obvious changes and / or replacements based on the examples Figure 13 The resulting technical solutions still belong to the disclosure range of the embodiments of the present disclosure.

[0161] According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0162] It should be noted that in the technical solutions of the present disclosure, the acquisition, storage and application of user personal information involved all comply with relevant laws and regulations and do not violate public order and good customs.

[0163] Figure 14 A schematic block diagram of an example electronic device 1400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0164] As shown in Figure 14 The device 1400 includes a computing unit 1401 that can perform various appropriate actions and processes in accordance with a computer program stored in a Read-Only Memory (ROM) 1402 or a computer program loaded into a Random Access Memory (RAM) 1403 from a storage unit 1408. Various programs and data required for the operation of the device 1400 can also be stored in the RAM 1403. The computing unit 1401, the ROM 1402, and the RAM 1403 are connected to each other through a bus 1404. An Input / Output (I / O) interface 1405 is also connected to the bus 1404.

[0165] Various components in the device 1400 are connected to the I / O interface 1405, including an input unit 1406, such as a keyboard, a mouse, etc.; an output unit 1407, such as various types of displays, speakers, etc.; a storage unit 1408, such as a magnetic disk, an optical disk, etc.; and a communication unit 1409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1409 allows the device 1400 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0166] The computing unit 1401 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1401 performs various methods and processes described above, such as the training method of the lip-sync model and / or the generation method of the digital human video. For example, in some embodiments, the training method of the lip-sync model and / or the generation method of the digital human video can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1408. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1400 via the ROM 1402 and / or the communication unit 1409. When the computer program is loaded onto the RAM 1403 and executed by the computing unit 1401, one or more steps of the training method of the lip-sync model and / or the generation method of the digital human video described above can be performed. Alternatively, in other embodiments, the computing unit 1401 can be configured to perform the training method of the lip-sync model and / or the generation method of the digital human video by any other appropriate means, such as by means of firmware.

[0167] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system on chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0168] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable digital electronic device to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine or server, or entirely on a remote machine or server.

[0169] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0170] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a Cathode Ray Tube (CRT) or Liquid Crystal Display (LCD) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0171] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0172] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0173] It should be understood that the various forms of flow shown above can be re-ordered, added to, or have steps deleted, using the steps described above. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure can be achieved, which is not limited herein.

[0174] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a lip-sync model, comprising: determining a two-dimensional facial feature ground truth and a three-dimensional facial feature ground truth based on a video of a target object, the target object being an object simulated by a digital human; extracting a three-dimensional facial feature prediction from sample speech data by a first network sub-model; generating a two-dimensional facial feature prediction from the three-dimensional facial feature prediction by a second network sub-model; training the first network sub-model and the second network sub-model based on the two-dimensional facial feature ground truth, the three-dimensional facial feature ground truth, the two-dimensional facial feature prediction, and the three-dimensional facial feature prediction, to obtain a lip-sync model, the lip-sync model comprising the first network sub-model and the second network sub-model.

2. The method of claim 1, wherein, The extracting of the three-dimensional facial feature prediction from the sample speech data by the first network sub-model comprises: extracting a speech feature from the sample speech data by a first branch network in the first network sub-model; predicting the three-dimensional facial feature prediction from the speech feature by a second branch network in the second network sub-model.

3. The method of claim 2, wherein, The network structure of the first branch network is a speech feature extraction model based on self-supervised learning, and the network structure of the second branch network is a deep learning model based on self-attention mechanism.

4. The method of claim 1, wherein, The generating of the two-dimensional facial feature prediction from the three-dimensional facial feature prediction by the second network sub-model comprises: generating the two-dimensional facial feature prediction from the three-dimensional facial feature prediction by a third branch network in the second network sub-model.

5. The method of claim 4, wherein, The network structure of the third branch network is a U-shaped neural network model.

6. The method of claim 1, wherein, The training of the first network sub-model and the second network sub-model based on the two-dimensional facial feature ground truth, the three-dimensional facial feature ground truth, the two-dimensional facial feature prediction, and the three-dimensional facial feature prediction comprises: constructing a first loss function based on the three-dimensional facial feature ground truth and the three-dimensional facial feature prediction, training the first network sub-model based on the first loss function; and constructing a second loss function based on the two-dimensional facial feature ground truth and the two-dimensional facial feature prediction, training the second network sub-model based on the second loss function. The determining of the two-dimensional facial feature ground truth and the three-dimensional facial feature ground truth based on the video of the target object comprises:

7. The method of claim 1, wherein, performing video decoding based on the video to obtain a video decoding result; performing face detection based on the video decoding result to obtain a face detection result; performing face cropping based on the face detection result to obtain the two-dimensional facial feature ground truth; performing face reconstruction based on the face detection result to obtain the three-dimensional facial feature ground truth.

8. A method for generating a digital human video, comprising: obtaining target speech data; ​ inputting the target voice data into a lip-sync model to generate voice features through a first network sub-model of the lip-sync model, generating three-dimensional face features based on the voice features; generating two-dimensional face features simulating a target object based on the three-dimensional face features through a second network sub-model of the lip-sync model; wherein the lip-sync model is obtained by training through the method in any one of claims 1 to 7; generating a digital human video based on the two-dimensional face features and the target voice data.

9. A training device of a lip-sync model, comprising: a determination module configured to determine two-dimensional face feature ground truth and three-dimensional face feature ground truth based on a video of a target object, the target object being an object simulated by a digital human; an extraction module configured to extract three-dimensional face feature prediction from sample voice data through a first network sub-model; a first generation module configured to generate two-dimensional face feature prediction from the three-dimensional face feature prediction through a second network sub-model; a training module configured to train the first network sub-model and the second network sub-model based on the two-dimensional face feature ground truth, the three-dimensional face feature ground truth, the two-dimensional face feature prediction and the three-dimensional face feature prediction, to obtain a lip-sync model, the lip-sync model comprising the first network sub-model and the second network sub-model.

10. The apparatus of claim 9, wherein, The extraction module is configured to: extract voice features from the sample voice data through a first branch network in the first network sub-model; predict three-dimensional face feature prediction from the voice features through a second branch network in the second network sub-model.

11. The apparatus of claim 10, wherein, The network structure of the first branch network is a voice feature extraction model based on self-supervised learning, and the network structure of the second branch network is a deep learning model based on self-attention mechanism.

12. The apparatus of claim 9, wherein, The generation module is configured to: generate two-dimensional face feature prediction from the three-dimensional face feature prediction through a third branch network in the second network sub-model.

13. The apparatus of claim 12, wherein, The network structure of the third branch network is a U-shaped neural network model.

14. The apparatus of claim 9, wherein, The training module is configured to: construct a first loss function based on the three-dimensional face feature ground truth and the three-dimensional face feature prediction; and train the first network sub-model based on the first loss function; and construct a second loss function based on the two-dimensional face feature ground truth and the two-dimensional face feature prediction; and train the second network sub-model based on the second loss function. The determination module is configured to: decode the video based on the video to obtain a video decoding result; 15. The apparatus of claim 9, wherein, perform face detection based on the video decoding result to obtain a face detection result; perform face cropping based on the face detection result to obtain two-dimensional face feature ground truth; perform face reconstruction based on the face detection result to obtain three-dimensional face feature ground truth.

16. A digital human video generation device, comprising: an acquisition module configured to acquire target voice data; ​ ​ The input module is configured to input the target voice data into the lip-sync model to generate voice features by a first network sub-model of the lip-sync model, generate three-dimensional face features based on the voice features, generate two-dimensional face features simulating a target object based on the three-dimensional face features by a second network sub-model of the lip-sync model, and wherein the lip-sync model is trained by the method in any one of claims 1 to 7. The second generation module is configured to generate a digital human video based on the two-dimensional face features and the target voice data.

17. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are configured to cause the computer to perform the method of any one of claims 1-8.

19. A computer program product comprising a computer program stored on a storage medium, the computer program, when executed by a processor, implementing the method of any one of claims 1-8.

Images