Digital human model training method and device, digital human model generation method and device, equipment and medium

Abstract

The invention relates to a digital human model training method and device, a digital human model generation method and device, equipment and a medium, and the training method comprises the steps: obtaining a training set which comprises human face image data, human face key point feature data and mouth image feature data; inputting the training set into an initial generator to obtain a generated image; training the initial generator according to the loss of the generated image compared with a real image to obtain a digital human model; wherein the step of obtaining the generated image comprises obtaining source image feature data according to the face image feature data and the face key point feature data; and fusing the mouth image feature data and the source image feature data to obtain the generated image, and by adopting the training method provided by the invention, the problem of low training speed of a digital human model in the prior art can be improved.

Description

Technical Field

[0001] The present application relates to the field of virtual application technology, and in particular to a digital human model training method, generation method, device, equipment and medium. Background Art

[0002] Digital human is a digital character image close to human image created by digital technology. It is widely used in education and training, finance and insurance, medical health, classroom lectures, games, e-commerce and other fields.

[0003] In the related technology, the method of generating video mainly uses audio to drive images. The model has a huge amount of parameters and calculations, and the training is very difficult and slow. Summary of the invention

[0004] Based on this, a digital human model training method, generation method, device, equipment and medium are provided to improve the problem of slow digital human model training speed in the prior art.

[0005] On the one hand, a digital human model training method is provided, comprising:

[0006] Acquire a training set, the training set including face image data, face key point feature data, and mouth image feature data, wherein the face image feature data and face key point feature data are obtained based on a training video data set, and the mouth image feature data is obtained based on the face image feature data;

[0007] Inputting the training set into the initial generator to obtain a generated image;

[0008] Training the initial generator according to the loss of the generated image compared to the real image to obtain a digital human model;

[0009] Wherein, obtaining the generated image includes obtaining source image feature data according to the facial image feature data and facial key point feature data; and fusing the mouth image feature data with the source image feature data to obtain the generated image.

[0010] In one embodiment, the fusing the mouth image feature data and the source image feature data comprises:

[0011] Aligning the mouth image feature data with the source image feature data to obtain aligned image data;

[0012] Performing an affine transformation on the aligned image data to obtain affine transformed image data;

[0013] Upsampling is performed according to the affine transformed image data to obtain the generated image.

[0014] In one embodiment, the training of the initial generator according to the loss of the generated image compared to the real image includes:

[0015] Inputting the generated image into an initial discriminator to obtain a first output of the discriminator;

[0016] Based on the first output of the discriminator, the generated image loss is obtained;

[0017] The initial generator is trained according to the generated image loss.

[0018] In one embodiment, the training of the initial generator according to the loss of the generated image compared to the real image further includes:

[0019] Obtaining a perceptual loss based on the generated image and the real image;

[0020] The initial generator is trained according to the sum of the generated image loss and the perceptual loss.

[0021] In one embodiment, the training of the initial generator according to the loss of the generated image compared to the real image further includes:

[0022] Inputting the real image in the face image data into the initial discriminator to obtain the second output of the discriminator;

[0023] According to the second output of the discriminator, obtaining a real image loss;

[0024] Obtaining a regularization loss according to the real image and the second output of the discriminator;

[0025] The initial discriminator is trained according to the generated image loss, the real image loss and the regularization loss.

[0026] In one embodiment, obtaining the regularization loss includes:

[0027] According to the real image and the second output of the discriminator, obtaining a gradient of the second output of the discriminator with respect to the real image;

[0028] According to the gradient, obtaining the L2 norm square of the gradient;

[0029] The regularization loss is obtained by summing and averaging the squared L2 norm of the gradient.

[0030] On the other hand, a method for generating a digital human is provided, comprising:

[0031] Acquire inference data, the inference data including character audio data, reference video data including a face image, and reference face key point feature data, wherein the reference face key point feature data is obtained based on the reference video data;

[0032] Obtaining mouth image feature data according to the character audio data;

[0033] Inputting the reference video data, the reference facial key point feature data and the mouth image feature data into a digital human model to obtain a digital human;

[0034] Wherein, the digital human model is obtained according to the digital human model training method.

[0035] In another aspect, a digital human model training device is provided, the device comprising:

[0036] A training set acquisition module, used to acquire a training set, wherein the training set includes face image data, face key point feature data, and mouth image feature data, wherein the face image feature data and face key point feature data are acquired based on a training video data set, and the mouth image feature data is acquired based on the face image feature data;

[0037] A training module, used for inputting the training set into an initial generator to obtain a generated image, and training the initial generator according to the loss of the generated image compared with the real image to obtain a digital human model;

[0038] Wherein, obtaining the generated image includes obtaining source image feature data according to the facial image feature data and facial key point feature data; and fusing the mouth image feature data with the source image feature data to obtain the generated image.

[0039] A computer device is also provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a digital human model training method or a digital human generation method when executing the computer program.

[0040] A computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, a digital human model training method or a digital human generation method is implemented.

[0041] In the above-mentioned digital human model training method, facial image feature data and facial key point feature data are obtained according to the training video data set, source image feature data is obtained according to the facial image data and the facial key point feature data, and then the mouth image feature data and the source image feature data are fused to obtain the generated image, and the initial generator is trained according to the loss of the generated image compared to the real image. Audio is no longer used as a driver during the training process, and there is no need to train the audio, which improves the training speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the DINet architecture in the related technology;

[0043] Figure 2 A schematic diagram of a flow chart of a digital human model training method in one embodiment;

[0044] Figure 3 A schematic diagram of an improved DINet architecture in an embodiment;

[0045] Figure 4 A schematic diagram of a discriminator structure in one embodiment;

[0046] Figure 5 A schematic diagram of a process for generating a digital human in one embodiment

[0047] Figure 6 is a structural block diagram of a digital human model training device in one embodiment;

[0048] Figure 7 It is a structural block diagram of a digital human generating device in one embodiment;

[0049] Figure 8 is an internal structure diagram of a computer device in one embodiment;

[0050] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] Digital human is a virtual character created based on artificial intelligence technology and computer graphics technology. This technology can simulate human appearance, voice, movement and communication ability, bringing new application possibilities to multiple fields.

[0053] In the related technologies, the main method of generating video from images is to use audio-driven methods, such as wav2lips, DINet (deformable inpainting network), musetalk, sadtalker and ER-Nerf, etc., to achieve stable and controllable generation of the speaker's mouth shape and expression.

[0054] However, the actual deployment of smart cockpits faces great challenges. First, the process of generating video from images requires high-performance hardware, which is almost impossible to achieve in the environment of the vehicle. Secondly, from the perspective of the model, the current image-generated video algorithm model has a huge number of parameters and a large amount of calculation. The digital human generation technology is explained using DINet as an example. Its architecture is as follows: Figure 1 As shown, the training and reasoning process is described below:

[0055] Input data: The video dataset is video data of people speaking captured by a mobile phone or camera.

[0056] Data preprocessing: Preprocess the video dataset. First, use the openface tool to obtain the facial key points of the video, a total of 68 key points, and save them as a facial key point dataset. Then extract each frame of the video dataset and save them as an image frame dataset. Then extract the audio data of the video dataset and save them as an audio dataset. Then use the DeepSpeech model to extract the deep speech features of the audio dataset. Finally, crop the face image of each frame according to the above facial key points to construct a face cropped image dataset. At this point, the dataset is ready.

[0057] Feature encoder: Input the face cropped image dataset and the image frame dataset into the image feature encoder for feature extraction to obtain face cropped image features and source image frame features; input the deep speech feature data into the speech feature encoder for extraction to obtain speech features.

[0058] Feature fusion: First, align the features of the face cropped image with the features of the source image frame; then fuse them with the speech features.

[0059] Feature decoder: The fused features are input into the feature decoder for upsampling to finally obtain the generated image.

[0060] Gradient update: First, the generated image and the real image are input into the discriminator. The discriminator can distinguish between real data and generated data, calculate the generator loss and the discriminator loss, and then use the discriminator loss to update the discriminator network; use the generator loss to back propagate and update the DINet network.

[0061] Training phase: The algorithm is trained in four steps. The first step focuses on low-resolution face data (104x80 resolution face, 64x64 resolution mouth), which takes about 8 hours on the RTX409024GB platform; the second step will load the pre-trained model in the first step, and then train with higher resolution (208x160 resolution face and 128x128 resolution mouth data), which takes about 24 hours; the third step also needs to load the pre-trained model in the previous step and train with higher resolution (416x320 resolution face and 256x256 resolution mouth), which takes longer; the fourth step also loads the pre-trained model in the previous step, as well as the pre-trained model for audio and lip synchronization, and performs lip synchronization training, and finally the training ends.

[0062] Inference stage: To generate a digital human, you first need to prepare a reference video, an audio that drives the digital human to speak, and the key points of the face processed by OpenFace. After the data is prepared, load the trained model and perform inference to generate a digital human video.

[0063] During the above training process, on the one hand, an additional DeepSpeech model was used for audio feature extraction. The large size of the model resulted in a slow generation speed, and audio training was required during training, which increased the training time. On the other hand, during the above training process, the face image input was not fused with the face key points, and the key point features were used during lip synchronization training, which required multiple steps of training for the model, increasing the training time.

[0064] The present application provides a digital human model training method that only requires one step of training, simplifies the data processing method, eliminates the need for audio training, simplifies the model, and reduces training costs.

[0065] In one embodiment, the digital human model training method is as follows: Figure 2 As shown, the following steps are included:

[0066] Step 110, obtaining a training set, the training set includes face image data, face key point feature data, and mouth image feature data.

[0067] The face image data includes a number of face images, the face key point feature data is obtained in advance by performing key point analysis on the face images, and the mouth image feature data is obtained by dividing the mouth in the face images.

[0068] Exemplarily, the facial image feature data and the facial key point feature data are obtained based on a training video data set, and the mouth image feature data are obtained based on the facial image feature data.

[0069] The training video dataset is high-resolution video data of people speaking, shot by a mobile phone or camera. The video is set up with a green screen and dual cameras to collect videos from multiple angles to obtain richer facial information and enhance the generalization performance of the model.

[0070] Then the training video dataset is preprocessed. (1) First, the ffmpeg tool is used to edit and crop the video. The ffmpeg tool is an open source audio and video codec tool. Only one line of code is needed to edit a long video into a video segment of 30 seconds. The sliding window method is used instead of continuous fixed 30-second capture. The length of the sliding window is 30 seconds. In this way, the video segments edited from a long video can not only increase the richness of the dataset, but also maintain the continuity of the video segment. (2) MediaPipe is used to extract the facial key points corresponding to each frame of the video and save them as facial key point feature data. MediaPipe is an open source multimedia machine learning model application framework. The extracted facial key points contain 478 key point features, which is 410 more than the original key points. These key points contain the three-dimensional features and depth features of the face, which can enable the model to learn complete and comprehensive facial features. And using this tool only requires a few lines of code to extract, without the need for manual collection using openface. (3) The cut frames of each video segment are saved as facial image data. (4) Based on the extracted 478 key points, the key point features of the lower half of the face and the important information of the mouth shape features of the lower half of the face are segmented to obtain the mouth image feature data.

[0071] Step 120, input the training set into the initial generator to obtain a generated image.

[0072] Generators can include various neural network models based on machine learning and deep learning, such as convolutional neural networks (CNNs) and generative adversarial networks (GANs). Convolutional neural networks can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training. Based on pattern analysis of training data, the performance of the neural network can be improved by updating weights.

[0073] In the field of image generation, image generation can also be achieved based on generative adversarial networks. The generator is usually composed of multiple convolutional generators (Generator): layers and convolutional inversion layers to learn the feature representation of generated images.

[0074] In the initial generator, the initial generator performs feature extraction based on the facial image feature data and the facial key point feature data to obtain the source image feature data; and then fuses the mouth image feature data with the source image feature data to obtain the generated image.

[0075] In this embodiment, training is performed based on the improved DINet. DINet includes feature encoding, feature fusion, and feature decoding steps, such as Figure 3 shown.

[0076] Feature encoding: The feature encoder based on DINet can learn the implicit features of the input data, concatenate the facial image feature data and the facial key point feature data as the input of the feature encoder, and extract the features to obtain the source image feature data;

[0077] Feature fusion: Align the mouth image feature data with the source image feature data so that the mouth image is aligned and covered at the mouth position of the source image to obtain aligned image data. In some implementations, perform an affine transformation on the aligned image data to obtain affine transformed image data to enhance the robustness of the features, such as through AdaAT (adaptive affine transformation), which typically includes operations such as rotation, scaling, translation, and shearing. Based on these operations, AdaAT performs random or adaptive transformations on the aligned features to make it difficult for the model to grasp the pattern of input disturbances, thereby improving the robustness of the model.

[0078] Feature decoding: Based on the feature decoder implementation of DINet, the fused features (aligned image data or affine transformed image data) are input into the feature decoder for upsampling to generate image frames. The generated image frames are the generated images of the digital human.

[0079] The generated images obtained in the early stage of training are of poor quality and have large differences from the real images, so iterative training is performed subsequently.

[0080] Step 130: training the initial generator according to the loss of the generated image compared to the real image to obtain a digital human model.

[0081] The loss of the generated image compared to the real image can be measured by calculating the difference between the two, including pixel-level differences, structural differences, and perceptual differences. The real image comes from the facial image data obtained by frame cutting mentioned above. During each training process, the real image and the generated image obtained by the generator based on the real image are compared to obtain the loss, and the loss is back-propagated to update the parameters of the generator.

[0082] In GAN technology, a generator and a discriminator are trained at the same time. The discriminator can distinguish between real images and generated images and calculate GAN Loss.

[0083] In this embodiment, the discriminator network structure adopts the discriminator network Discriminator of StyleGAN2, such as Figure 4As shown in the figure, the overall model is lightweight. The discriminator mainly uses the residual network structure ResBlock, the convolutional neural network convBlock and the EqualLinear linear layer. By using ResBlock, the discriminator can more effectively deal with the gradient vanishing problem and improve the training efficiency of deep networks. EqualLinear implements weight normalization. By scaling the weights proportionally in the forward propagation, it avoids manually adjusting the learning rate during training and ensures a more stable training process. Standard deviation feature enhancement is used in the network. Through standard deviation calculation, the discriminator can better detect the mode collapse phenomenon in the image and ensure that the images generated by the generator are more diverse. In summary, the Discriminator of StyleGAN2 enables the generator to generate higher quality and more stable images. It is more friendly to high-resolution image generation.

[0084] The discriminator works as follows:

[0085] The generated image frames are input into the discriminator of StyleGAN2 together with the real image frames. The discriminator outputs corresponding discriminant values ​​for the generated images and the real images respectively. For the real images, the expected output is close to 1 (real); for the generated images, the expected output is close to 0 (fake). Then, based on these outputs, the losses of the discriminator and the generator (GAN Loss) are calculated respectively. The discriminator loss is used to optimize its ability to distinguish between real and generated images, and the generator loss adjusts the parameters of the generator through back propagation to make the generated images more realistic to deceive the discriminator. Finally, the loss value is used to update the network parameters to improve the model performance.

[0086] In this embodiment, the initial generator is trained according to the loss of the generated image compared to the real image, including the following steps:

[0087] 1. Input the generated image into the initial discriminator to obtain the first output of the discriminator, that is, the generated image output;

[0088] 2. Based on the first output of the discriminator, obtain the generated image loss;

[0089] 3. Train the initial generator according to the generated image loss and back-propagate to update the generator network parameters.

[0090] In some other implementations of this embodiment, multi-scale features are extracted from the generated image and the real image, and perceptual loss is calculated. Perceptual loss is a loss function for image conversion tasks that aims to capture high-level feature differences between images, rather than just pixel-level differences. The perceptual loss and the generated image loss are added together as the loss of the generator. Finally, back propagation allows the generator network to adjust parameters to minimize the generator loss, i.e., to make the generated image more realistic. For example, the generated image and the real image are input into the VGG19 network to calculate the perceptual loss.

[0091] On the other hand, by training the initial discriminator, its ability to identify real images and generated images is improved. The process is as follows:

[0092] ① Inputting the real image in the face image data into the initial discriminator to obtain the second output of the discriminator, i.e., the real image output;

[0093] ②According to the second output of the discriminator, the real image loss is obtained;

[0094] ③ Obtaining a regularization loss based on the real image and the second output of the discriminator;

[0095] ④ According to the generated image loss, the real image loss and the regularization loss, the initial discriminator is trained.

[0096] Regularization limits the complexity of the model by adding constraints to the model parameters. For example, L1 and L2 regularization penalize large parameter values ​​so that the model parameters do not become too large. This prevents the model from being too complex and avoids overfitting to noise and outliers in the training data.

[0097] For example, the R1 regularizer is used to calculate the regularized loss R1 loss, which is combined with the GAN loss to update the discriminator network. R1 regularization is a regularization method for the discriminator gradient, which is applied to real data samples. By constraining the gradient of the discriminator, R1 regularization can prevent the discriminator from overfitting and improve the stability of training.

[0098] The R1 regularizer obtains the gradient of the second output of the discriminator to the real image according to the real image and the second output of the discriminator; obtains the L2 norm square of the gradient according to the gradient; and obtains the regularization loss by summing and averaging the L2 norm square of the gradient. The calculation expression is as follows:

[0099]

[0100] Where x represents a real data sample, i.e., a real image; D(x) represents the output of the discriminator to x, i.e., the second output of the discriminator; represents the L2 norm square of the gradient of the discriminator output to the input, which indicates the sensitivity of the discriminator output to input changes. Indicates the square of the norm corresponding to the real data sample x in the face image data P real The expectation in the range is the mean of the sum of the squared L2 norms.

[0101] Through the above process, the digital human generator obtained by the present invention can fuse the mouth shape with the facial image to obtain a realistic digital human effect. In the inference process, the mouth shape image corresponding to the voice and the facial image of the person to be used are used as input, and the digital human model can be fused to generate a corresponding, realistic digital human, and the training process has the following advantages:

[0102] The present invention is based on an improved DINet model architecture, aiming to balance high resolution and light weight when generating digital humans, which is particularly beneficial for deploying digital human interaction systems in resource-constrained cockpit environments.

[0103] The present invention uses a more comprehensive and efficient data processing method to extract more abundant facial key point features (478 key points), which is very beneficial for the mouth shape effect generated by digital humans. It enables the model to learn complete and comprehensive facial features. This data processing method is not only fast, but also enhances the generalization ability of the model.

[0104] The present invention splices image data with key features of facial points, which greatly shortens the training time of the model and only requires one step of training to achieve the ideal effect.

[0105] The present invention removes the audio training model in the model, greatly reducing the training cost. This is lightweight enough for deployment in a resource-constrained cockpit environment, and the generation speed is 82% of the audio duration, which can achieve real-time performance.

[0106] This paper introduces the Discriminator of StyleGAN2, which can capture local details and global structures in high-resolution images thanks to ResBlock, EqualLinear and standard deviation feature enhancement, making the generator network more efficient and stable. It can generate videos with richer mouth details.

[0107] The R1 regularization introduced in the present invention constrains the discriminator's response to the real input, thereby preventing it from overfitting. The advantage of this regularization technique is that it improves the stability of training and prevents the discriminator from becoming too strong during the confrontation process.

[0108] In another embodiment, based on the digital human model obtained by the digital human model training method provided in the above embodiment, a digital human generation method is provided, such as Figure 5 As shown, the following steps are included:

[0109] Step 210, obtaining inference data, the inference data including character audio data, reference video data including a face image, and reference face key point feature data, wherein the reference face key point feature data is obtained based on the reference video data.

[0110] Exemplarily, the inference stage first needs to prepare a section of reference video data, and the reference video data, such as the aforementioned MediaPipe, extracts facial key point features to obtain reference facial key point feature data, and in addition, prepares a section of audio as character audio data.

[0111] Step 220, obtaining mouth image feature data based on the character audio data.

[0112] Exemplarily, a mouth shape image model for generating the lower half of the face mouth shape based on the character audio is pre-trained for business scenarios. The model is composed of a downsampling convolutional neural network and a bidirectional LSTM network. In the inference stage, the character audio data is input into the mouth shape image model to obtain the mouth image feature data.

[0113] Step 230: input the reference video data, the reference facial key point feature data and the mouth image feature data into a digital human model to obtain a digital human.

[0114] The time consumed by the entire inference process of this model is 82% of that of the audio-driven method, which speeds up the generation and can meet real-time interaction.

[0115] It should be understood that although Figure 2 , Figure 5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , Figure 5 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0116] In one embodiment, Figure 6 As shown, a digital human model training device is provided, including: a training set acquisition module 301 and a training module 302, wherein:

[0117] A training set acquisition module 301 is used to acquire a training set, wherein the training set includes face image data, face key point feature data, and mouth image feature data, wherein the face image feature data and face key point feature data are acquired based on a training video data set, and the mouth image feature data is acquired based on the face image feature data;

[0118] A training module 302 is used to input the training set into an initial generator to obtain a generated image, and train the initial generator according to the loss of the generated image compared to the real image to obtain a digital human model;

[0119] Wherein, obtaining the generated image includes obtaining source image feature data according to the facial image feature data and facial key point feature data; and fusing the mouth image feature data with the source image feature data to obtain the generated image.

[0120] The above-mentioned digital human model training device no longer uses audio as a drive during the training process, and there is no need to train the audio, which improves the training speed. It also splices the image data with the key point features of the face, which greatly shortens the training time of the model. Only one step of training is required to achieve the ideal effect.

[0121] The training module 302 fuses the mouth image feature data with the source image feature data, including aligning the mouth image feature data with the source image feature data to obtain aligned image data, performing affine transformation based on the aligned image data to obtain affine transformed image data; and upsampling based on the affine transformed image data to obtain the generated image.

[0122] The training module 302 trains the initial generator according to the loss of the generated image compared to the real image, including inputting the generated image into the initial discriminator to obtain the first output of the discriminator; obtaining the generated image loss based on the first output of the discriminator; and training the initial generator according to the generated image loss.

[0123] The training module 302 trains the initial generator according to the loss of the generated image compared to the real image, and also includes obtaining the perceptual loss according to the generated image and the real image; and training the initial generator according to the sum of the generated image loss and the perceptual loss.

[0124] The training module 302 trains the initial generator according to the loss of the generated image compared to the real image, and also includes inputting the real image in the face image data into the initial discriminator to obtain the second output of the discriminator; obtaining the real image loss according to the second output of the discriminator; obtaining the regularization loss according to the real image and the second output of the discriminator; and training the initial discriminator according to the generated image loss, the real image loss and the regularization loss.

[0125] The step of obtaining the regularization loss in the training module 302 includes obtaining the gradient of the second output of the discriminator to the real image based on the real image and the second output of the discriminator; obtaining the L2 norm square of the gradient based on the gradient; and obtaining the regularization loss by summing and averaging the L2 norm squares of the gradient.

[0126] In another embodiment, a digital human generation device is also provided, such as Figure 7 As shown, the digital human generation device includes an inference set acquisition module 401, a mouth image acquisition module 402 and an inference module 403, wherein:

[0127] The inference set acquisition module 401 is used to acquire inference data, wherein the inference data includes character audio data, reference video data including a face image, and reference face key point feature data, wherein the reference face key point feature data is obtained based on the reference video data;

[0128] The mouth image acquisition module 402 is used to obtain mouth image feature data according to the character audio data;

[0129] The inference module 403 is used to configure the digital human model, and input the reference video data, the reference facial key point feature data and the mouth image feature data into the digital human model to obtain a digital human.

[0130] For the specific definition of the digital human model training device, please refer to the definition of the digital human model training method above, and for the specific definition of the digital human generation device, please refer to the definition of the digital human generation method above, which will not be repeated here. The various modules in the above-mentioned digital human model training device and digital human generation device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0131] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store training set data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a digital human model training method is implemented.

[0132] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0133] Acquire a training set, the training set including face image data, face key point feature data, and mouth image feature data, wherein the face image feature data and face key point feature data are obtained based on a training video data set, and the mouth image feature data is obtained based on the face image feature data;

[0134] Inputting the training set into the initial generator to obtain a generated image;

[0135] Training the initial generator according to the loss of the generated image compared to the real image to obtain a digital human model;

[0136] Wherein, obtaining the generated image includes obtaining source image feature data according to the facial image feature data and facial key point feature data; and fusing the mouth image feature data with the source image feature data to obtain the generated image.

[0137] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0138] The fusing the mouth image feature data and the source image feature data comprises:

[0139] Aligning the mouth image feature data with the source image feature data to obtain aligned image data;

[0140] Performing an affine transformation on the aligned image data to obtain affine transformed image data;

[0141] Upsampling is performed according to the affine transformed image data to obtain the generated image.

[0142] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0143] The training of the initial generator according to the loss of the generated image compared to the real image comprises:

[0144] Inputting the generated image into an initial discriminator to obtain a first output of the discriminator;

[0145] Based on the first output of the discriminator, the generated image loss is obtained;

[0146] The initial generator is trained according to the generated image loss.

[0147] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0148] The training of the initial generator according to the loss of the generated image compared to the real image also includes:

[0149] Obtaining a perceptual loss based on the generated image and the real image;

[0150] The initial generator is trained according to the sum of the generated image loss and the perceptual loss.

[0151] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0152] The training of the initial generator according to the loss of the generated image compared to the real image also includes:

[0153] Inputting the real image in the face image data into the initial discriminator to obtain the second output of the discriminator;

[0154] According to the second output of the discriminator, obtaining a real image loss;

[0155] Obtaining a regularization loss according to the real image and the second output of the discriminator;

[0156] The initial discriminator is trained according to the generated image loss, the real image loss and the regularization loss.

[0157] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0158] The regularized loss is obtained, including:

[0159] According to the real image and the second output of the discriminator, obtaining a gradient of the second output of the discriminator with respect to the real image;

[0160] According to the gradient, obtaining the L2 norm square of the gradient;

[0161] The regularization loss is obtained by summing and averaging the squared L2 norm of the gradient.

[0162] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for generating a digital human is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0163] The terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0164] Acquire inference data, the inference data including character audio data, reference video data including a face image, and reference face key point feature data, wherein the reference face key point feature data is obtained based on the reference video data;

[0165] Obtaining mouth image feature data according to the character audio data;

[0166] The reference video data, the reference facial key point feature data and the mouth image feature data are input into a digital human model to obtain a digital human.

[0167] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0168] Acquire a training set, the training set including face image data, face key point feature data, and mouth image feature data, wherein the face image feature data and face key point feature data are obtained based on a training video data set, and the mouth image feature data is obtained based on the face image feature data;

[0169] Inputting the training set into the initial generator to obtain a generated image;

[0170] Training the initial generator according to the loss of the generated image compared to the real image to obtain a digital human model;

[0171] Wherein, obtaining the generated image includes obtaining source image feature data according to the facial image feature data and facial key point feature data; and fusing the mouth image feature data with the source image feature data to obtain the generated image.

[0172] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0173] Aligning the mouth image feature data with the source image feature data to obtain aligned image data;

[0174] Performing an affine transformation on the aligned image data to obtain affine transformed image data;

[0175] Upsampling is performed according to the affine transformed image data to obtain the generated image.

[0176] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0177] Inputting the generated image into an initial discriminator to obtain a first output of the discriminator;

[0178] Based on the first output of the discriminator, the generated image loss is obtained;

[0179] The initial generator is trained according to the generated image loss.

[0180] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0181] Obtaining a perceptual loss based on the generated image and the real image;

[0182] The initial generator is trained according to the sum of the generated image loss and the perceptual loss.

[0183] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0184] Inputting the real image in the face image data into the initial discriminator to obtain the second output of the discriminator;

[0185] According to the second output of the discriminator, obtaining a real image loss;

[0186] Obtaining a regularization loss according to the real image and the second output of the discriminator;

[0187] The initial discriminator is trained according to the generated image loss, the real image loss and the regularization loss.

[0188] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0189] According to the real image and the second output of the discriminator, obtaining a gradient of the second output of the discriminator with respect to the real image;

[0190] According to the gradient, obtaining the L2 norm square of the gradient;

[0191] The regularization loss is obtained by summing and averaging the squared L2 norm of the gradient.

[0192] In another embodiment, when the computer program is executed by a processor, the following steps are implemented:

[0193] Acquire inference data, the inference data including character audio data, reference video data including a face image, and reference face key point feature data, wherein the reference face key point feature data is obtained based on the reference video data;

[0194] Obtaining mouth image feature data according to the character audio data;

[0195] Inputting the reference video data, the reference facial key point feature data and the mouth image feature data into a digital human model to obtain a digital human;

[0196] Wherein, the digital human model is obtained according to the digital human model training method.

[0197] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0198] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0199] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A digital human model training method, characterized in that: include: Acquire a training set, the training set including face image data, face key point feature data, and mouth image feature data, wherein the face image feature data and face key point feature data are obtained based on a training video data set, and the mouth image feature data is obtained based on the face image feature data; Inputting the training set into the initial generator to obtain a generated image; Training the initial generator according to the loss of the generated image compared to the real image to obtain a digital human model; Wherein, obtaining the generated image includes obtaining source image feature data according to the facial image feature data and facial key point feature data; and fusing the mouth image feature data with the source image feature data to obtain the generated image.

2. The digital human model training method according to claim 1, characterized in that: The fusing the mouth image feature data and the source image feature data comprises: Aligning the mouth image feature data with the source image feature data to obtain aligned image data; Performing an affine transformation on the aligned image data to obtain affine transformed image data; Upsampling is performed according to the affine transformed image data to obtain the generated image.

3. The digital human model training method according to claim 1, characterized in that: The training of the initial generator according to the loss of the generated image compared to the real image comprises: Inputting the generated image into an initial discriminator to obtain a first output of the discriminator; Based on the first output of the discriminator, the generated image loss is obtained; The initial generator is trained according to the generated image loss.

4. The digital human model training method according to claim 3, characterized in that: The training of the initial generator according to the loss of the generated image compared to the real image also includes: Obtaining a perceptual loss based on the generated image and the real image; The initial generator is trained according to the sum of the generated image loss and the perceptual loss.

5. The digital human model training method according to claim 3, characterized in that: The training of the initial generator according to the loss of the generated image compared to the real image also includes: Inputting the real image in the face image data into the initial discriminator to obtain the second output of the discriminator; According to the second output of the discriminator, obtaining a real image loss; Obtaining a regularization loss according to the real image and the second output of the discriminator; The initial discriminator is trained according to the generated image loss, the real image loss and the regularization loss.

6. The digital human model training method according to claim 5, characterized in that: The regularized loss is obtained, including: According to the real image and the second output of the discriminator, obtaining a gradient of the second output of the discriminator with respect to the real image; According to the gradient, obtaining the L2 norm square of the gradient; The regularization loss is obtained by summing and averaging the squared L2 norm of the gradient.

7. A method for generating a digital human, characterized in that: include: Acquire inference data, the inference data including character audio data, reference video data including a face image, and reference face key point feature data, wherein the reference face key point feature data is obtained based on the reference video data; Obtaining mouth image feature data according to the character audio data; Inputting the reference video data, the reference facial key point feature data and the mouth image feature data into a digital human model to obtain a digital human; Wherein, the digital human model is obtained according to the digital human model training method according to any one of claims 1-6.

8. A digital human model training device, characterized in that: The device comprises: A training set acquisition module, used to acquire a training set, wherein the training set includes face image data, face key point feature data, and mouth image feature data, wherein the face image feature data and face key point feature data are acquired based on a training video data set, and the mouth image feature data is acquired based on the face image feature data; A training module, used for inputting the training set into an initial generator to obtain a generated image, and training the initial generator according to the loss of the generated image compared with the real image to obtain a digital human model; Wherein, obtaining the generated image includes obtaining source image feature data according to the facial image feature data and facial key point feature data; and fusing the mouth image feature data with the source image feature data to obtain the generated image.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the digital human model training method of any one of claims 1 to 6 or the digital human generation method of claim 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the digital human model training method of any one of claims 1 to 6 or the digital human generation method of claim 7 is implemented.

Images