Training methods for digital human generation models, digital human video generation methods and systems

Abstract

This application provides a training method for a digital human generation model, a digital human video generation method, and a system. A first model and a second model generate a first fitted image and a second fitted image, respectively, based on reference training data. The parameters of the first model are adjusted based on the similarity difference between the first fitted image and the reference image. The output differences between the first and second models are determined based on the first and second fitted images, thereby adjusting the parameters of the second model. The first and second models are trained iteratively simultaneously until convergence. In this way, the second model does not need to directly learn from real images in the reference training data; it only needs to fit the first fitted image output by the first model. This achieves a near-first model output effect using fewer model parameters and less computation, making the second model applicable to low-computing devices and thus enriching the application scenarios of digital humans.

Description

Technical Field

[0001] This application relates to the fields of computer and image processing, and in particular to a training method for a digital human generation model, a digital human video generation method, and a system. Background Technology

[0002] As the demand for digital experiences grows, the trend of using digital characters (such as virtual assistants) is accelerating. Digital humans have been a captivating topic, and can be virtual characters, humanoid robots, or any digital entity with voice, vision, touch, and mobility capabilities. They are widely used in various fields such as healthcare, financial services, industrial automation, and customer service.

[0003] Digital human synthesis often employs digital human synthesis models, which can synthesize digital human videos based on user-input speech. However, the process of digital human synthesis requires users to upload high-quality videos, presenting a certain barrier to entry and limiting the target audience. Furthermore, the large number of parameters and computational demands of digital human synthesis models place significant demands on device computing resources, preventing the widespread application of digital humans in various scenarios. Summary of the Invention

[0004] This application provides a training method for a digital human generation model, a digital human video generation method and system, which can solve the problem that the existing digital human synthesis process requires too much computing resources from the device and cannot be widely applied in various scenarios.

[0005] Firstly, this application provides a training method for a digital human generation model, the method comprising:

[0006] Obtain reference training data, which includes reference images and reference audio.

[0007] The reference training data is input into a preset first model and a second model respectively, so as to instruct the first model to generate a first fitted image based on the reference training data, and to instruct the second model to generate a second fitted image based on the reference training data; wherein the parameter values ​​of the second model are smaller than the parameter values ​​of the first model;

[0008] The first true value corresponding to the first model is determined based on the similarity between the first fitted image and the reference image, and the first model is trained based on the first true value.

[0009] The second true value corresponding to the second model is determined based on the similarity between the first fitted image and the second fitted image, and the second model is trained based on the second true value.

[0010] The first model and the second model are trained until convergence, and the trained second model is output as the target model; wherein, the target model is used to generate digital human videos based on target audio input by the user.

[0011] In this application, the first model and the second model generate a first fitted image and a second fitted image respectively based on the reference training data. The parameters of the first model are adjusted based on the similarity difference between the first fitted image and the reference image. The output difference between the first model and the second model is determined based on the similarity between the first fitted image and the second fitted image, thereby adjusting the parameters of the second model. The first model and the second model are trained iteratively simultaneously until they converge. In this way, the second model does not need to directly learn the real images in the reference training data, but only needs to fit the first fitted image output by the first model. This allows it to achieve an output effect close to that of the first model with fewer model parameters and less computation, making the second model applicable to low-computing devices and thus enriching the application scenarios of digital humans.

[0012] In some possible implementations, determining the second true value corresponding to the second model based on the similarity between the first fitted image and the second fitted image, and training the second model based on the second true value includes:

[0013] Obtain the first fitted image and the second fitted image;

[0014] The second true value corresponding to the second model is calculated based on the similarity between the first fitted image and the second fitted image; the second true value is used to characterize the difference between the output results of the first model and the second model.

[0015] If the second true value is less than the second preset threshold, the parameter values ​​of the second model are adjusted according to the similarity between the first fitted image and the second fitted image.

[0016] The second model, with its parameter values ​​adjusted, is trained until convergence, and the next set of reference training data is input into the second model.

[0017] In some possible implementations, calculating the second true value corresponding to the second model based on the similarity between the first fitted image and the second fitted image includes:

[0018] Calculate structural similarity and perceptual similarity based on the first fitted image and the second fitted image, respectively.

[0019] A first weight value is obtained by using the structural similarity and the first coefficient, and a second weight value is obtained by using the perceptual similarity and the second coefficient;

[0020] The second true value is obtained based on the first weight value and the second weight value.

[0021] In some possible implementations, the method further includes:

[0022] If the second true value is equal to or greater than a preset threshold, the second model is trained until convergence, and the second model is output as the target model.

[0023] In some possible implementations, determining the first true value corresponding to the first model based on the similarity between the first fitted image and the reference image includes:

[0024] Calculate the first loss value based on the similarity between the first fitted image and the reference image;

[0025] The second loss value is calculated based on the difference between the first fitted image and the reference image;

[0026] The first true value is obtained based on the first loss value and the second loss value; the first true value is used to characterize the difference between the first fitted image output by the first model and the reference image.

[0027] In some possible implementations, after obtaining the first true value based on the first loss value and the second loss value, the method further includes:

[0028] When training the first model, if the first true value is less than the first preset threshold, the parameter values ​​of the first model are adjusted according to the similarity between the first fitted image and the reference image.

[0029] The first model, after its parameter values ​​have been adjusted, is trained until convergence, and the next set of reference training data is input into the first model.

[0030] Secondly, this application provides a method for generating digital human videos, including:

[0031] Obtain the image and target audio input by the user;

[0032] A template video is synthesized based on the image, and a target image is obtained by preprocessing the template video; wherein, the target image is a face image with the mouth area removed;

[0033] The target image and the target audio are imported into the target model to generate a predicted face image;

[0034] The predicted face image is spliced ​​with the target audio to obtain a digital human video.

[0035] In this application, after the target model obtains the image, it outputs a template video. The template video undergoes preprocessing: first, the video is converted into an image; then, face detection is performed, and the detected faces are cropped and saved. Next, the user's voice input is obtained, and the voice and face images are input into the trained target model. The target model outputs the face corresponding to the voice based on the voice and image, and the output face image is then stitched back into the original video to obtain the digital human video.

[0036] Thirdly, this application provides a digital human video generation system, the system comprising:

[0037] The acquisition module is used to acquire reference training data, which includes reference images and reference audio.

[0038] The training module inputs the reference training data into a preset first model and a second model, respectively, to instruct the first model to generate a first fitted image based on the reference training data, and to instruct the second model to generate a second fitted image based on the reference training data; wherein the parameter values ​​of the second model are smaller than the parameter values ​​of the first model;

[0039] The first true value corresponding to the first model is determined based on the similarity between the first fitted image and the reference image, and the first model is trained based on the first true value.

[0040] The second true value corresponding to the second model is determined based on the similarity between the first fitted image and the second fitted image, and the second model is trained based on the second true value.

[0041] The first model and the second model are trained until convergence, and the trained second model is output as the target model; wherein, the target model is used to generate digital human videos based on target audio input by the user.

[0042] In some possible implementations, the system further includes:

[0043] A generation module is used to synthesize a template video based on images input by the user, and to preprocess the template video to obtain a target image; wherein the target image is a face image with the mouth area removed;

[0044] The target image and the target audio input by the user are imported into the target model to generate a predicted face image;

[0045] The predicted face image is spliced ​​with the target audio to obtain a digital human video.

[0046] This application also provides a digital human video generation system, the system including an acquisition module, a training module, and a generation module; the acquisition module is used to acquire reference training data and user-input images and target audio; the training module is used to train a preset second model to obtain a target model with low parameter count and capable of running on low-computing-power devices; the generation module is used to control the target model to synthesize and output a digital human video based on the user-input images and target audio.

[0047] Fourthly, this application also provides an electronic device, comprising:

[0048] processor;

[0049] Memory for storing the executable instructions of the processor;

[0050] The processor is configured to execute the training method for the digital human generation model described in the first aspect and / or the digital human video generation method described in the second aspect by executing the executable instructions.

[0051] As described above, this application provides a training method for a digital human generation model, a digital human video generation method, and a system. The training method includes: acquiring reference training data; inputting the reference training data into a preset first model and a second model, respectively, to instruct the first model to generate a first fitted image based on the reference training data, and to instruct the second model to generate a second fitted image based on the reference training data; determining a first real value corresponding to the first model based on the similarity between the first fitted image and the reference image, and training the first model based on the first real value; determining a second real value corresponding to the second model based on the similarity between the first fitted image and the second fitted image, and training the second model based on the second real value; training the first model and the second model until convergence, and outputting the trained second model as the target model. The second model does not need to directly learn the real images in the reference training data, but only needs to fit the first fitted image output by the first model, so as to achieve an output effect close to that of the first model with fewer model parameters and less computation, making the second model applicable to low-computing devices, thereby enriching the application scenarios of digital humans. Attached Figure Description

[0052] Figure 1 A schematic diagram illustrating a training method for a digital human generation model provided in some embodiments of this application;

[0053] Figure 2 This is a schematic diagram of the first model training process provided in some embodiments of this application;

[0054] Figure 3 A schematic diagram illustrating the determination of the first true value corresponding to the first model, provided in some embodiments of this application;

[0055] Figure 4 This is a schematic diagram of the second model training process provided in some embodiments of this application;

[0056] Figure 5 This is a schematic diagram illustrating the interaction process between the first model and the second model provided in some embodiments of this application;

[0057] Figure 6 A schematic diagram of a digital human video generation system provided in some embodiments of this application. Figure 1 ;

[0058] Figure 7 A schematic diagram of a digital human video generation system provided in some embodiments of this application. Figure 2 . Detailed Implementation

[0059] To facilitate the explanation of the technical solution of this application, some concepts involved in this application will be explained first below.

[0060] As the demand for digital experiences grows, the trend of using digital characters (such as virtual assistants) is accelerating. Digital humans have been a captivating topic, and can be virtual characters, humanoid robots, or any digital entity with voice, vision, touch, and mobility capabilities. They are widely used in various fields such as healthcare, financial services, industrial automation, and customer service.

[0061] Digital human synthesis primarily employs digital human synthesis models, which can synthesize digital human videos based on user-input speech. Currently, these models only support training via video, requiring users to record high-quality video data for the synthesis process. Since the training effect is directly related to the quality of the video, the video must be clear, free of background noise, and unobstructed. This creates a high barrier to entry, hindering the widespread adoption of digital human technology. Therefore, the requirement for users to upload high-quality videos in the digital human synthesis process presents a significant barrier to entry, limiting the target audience.

[0062] At the same time, the current digital human synthesis models have a large number of parameters and computational load. The training and use of the models require high-end graphics cards such as A100 and 3090ti. Both the training and use of the models require a lot of computing power. The high demand for computing resources greatly limits the application scenarios of digital humans, causing them to be used only on computers or in environments with high-speed networks, which prevents digital humans from being widely used in various scenarios.

[0063] Therefore, this application provides a training method for a digital human generation model, a digital human video generation method, and a system. This application includes two preset models: a first model and a second model. The first model is responsible for learning from real images in the reference training data. The second model, which generates digital human videos, does not need to directly learn from real images in the reference training data; it only needs to fit the first fitted image output by the first model. By comparing the similarity difference between the first fitted image output by the first model and the real image, and by comparing the similarity difference between the second fitted image output by the second model and the first fitted image output by the first model, the parameters of the first and second models are adjusted respectively. The first and second models are trained iteratively simultaneously until convergence. The trained second model is used as the target model, achieving an output effect close to that of the first model with fewer model parameters and less computation. This allows the second model to be applicable to low-computing devices, thereby enriching the application scenarios of digital humans.

[0064] See Figure 1 This application provides a training method for a digital human generation model, the method comprising:

[0065] S100: Obtain reference training data, which includes reference images and reference audio.

[0066] In the digital human model learning process, there is a clear correspondence between human lip movements and semantic information in speech, independent of factors such as loudness, timbre, and noise. Furthermore, facial features must be considered, as different people have individual characteristics such as skin color, tooth shape, and lip shape. Therefore, the model requires two types of input information: speech and face. Speech information is used to predict the output lip movement information, while face images control the output facial features. These two features together determine the output result.

[0067] In some embodiments of this application, reference training data is used to enable the preset first and second models to learn the correspondence between speech and lip movements, while keeping the facial features of the input image unchanged, in order to obtain accurate results and good stability. Therefore, to ensure better training results for the subsequent first and second models, the selected reference training data needs to be clear, showing the mouth area, and the lip movements must accurately correspond to the speech without any deviation, excessive noise, or the sounds of other people speaking, and must not obscure the mouth.

[0068] The reference training data was obtained through self-recording and network download. Data cleaning was performed before training to delete problematic segments and ensure the cleanliness of the training data.

[0069] After obtaining the reference training data, data preprocessing is required. The data preprocessing work in this application mainly includes audio feature extraction and face cropping.

[0070] The audio feature extraction uses an audio feature extraction model, such as the ProsodyEncoder module, which can extract audio features by overcoming the bottleneck of word-level vector quantization.

[0071] Data preprocessing also includes cropping the video. First, the video is converted into an image sequence. Then, a face detection program is used to detect the face position. The face is then cropped based on the position information and used as the real image for training. After that, the mouth area of ​​the face image is masked and used as the input image, i.e., the reference image, for subsequent training. The model repairs the masked area based on the masked image and voice information to obtain the complete face image.

[0072] S200: The reference training data is input into a preset first model and a second model respectively, so as to instruct the first model to generate a first fitted image based on the reference training data, and to instruct the second model to generate a second fitted image based on the reference training data; wherein the parameter values ​​of the second model are less than the parameter values ​​of the first model.

[0073] This application includes two pre-defined models: a first model and a second model. The first model employs a large amount of ResNet architecture in its audio and image coding networks. This network architecture superimposes the input content onto the output convolutional results, which can significantly alleviate the gradient vanishing problem and enhance the network's learning ability, but it also greatly increases the number of model parameters and computational cost. The second model, on the other hand, uses a standard convolutional structure. The number of channels in the intermediate network layers of the second model is 0.25 to 0.4 times that of the first model, and the parameter values ​​of the second model are smaller than those of the first model. In some embodiments, the second model runs 19 times faster than the first model, meeting the requirements for operation on low-computing-power devices such as mobile phones.

[0074] The reference training data is input into the first model and the second model respectively, resulting in a first fitted image generated by the first model based on the reference training data, and a second fitted image generated by the second model based on the reference training data.

[0075] S300: Determine the first true value corresponding to the first model based on the similarity between the first fitted image and the reference image, and train the first model based on the first true value.

[0076] After obtaining the first fitted image, the similarity between the first fitted image and the reference image is compared to determine the difference between the first fitted image output by the first model and the reference image, so as to adjust the parameters of the first model and facilitate the subsequent training and learning of the first model based on the current reference image.

[0077] like Figure 2 As shown, in some embodiments, determining the first true value corresponding to the first model based on the similarity between the first fitted image and the reference image includes:

[0078] S301: Calculate a first loss value based on the similarity between the first fitted image and the reference image;

[0079] S302: Calculate a second loss value based on the difference between the first fitted image and the reference image;

[0080] S303: The first true value is obtained based on the first loss value and the second loss value; the first true value is used to characterize the difference between the first fitted image output by the first model and the reference image.

[0081] During the training and learning process, the first model has two loss functions, namely the first loss function (concat) and the second loss function. Based on the first loss function and the second loss function, the first true value corresponding to the first model is determined. The first true value is used to characterize the difference between the first fitted image output by the first model and the reference image.

[0082] like Figure 3 As shown, in some embodiments, the first loss function can be the reconstruction loss L. Recon The second loss function can be the GAN loss L. GAN The calculation formula is as follows:

[0083] L GAN (G T ) = E x,y [logD(x,y)]+E x [log(1-D(x,G T (x)))] (1)

[0084] L Recon (G T ) = E x,y [||yG T (x)||1] (2)

[0085]

[0086] Among them G TLet G represent the generator in the first model, D represent the discriminator in the first model, y represent the real face image, x represent the input audio features and image features (i.e., the reference training data), and G represent the data in the first model. T (x) represents the first fitted image; the generator is used to learn the correspondence between semantics and lip shape, as well as the face fitting ability, while the discriminator is used to evaluate the quality of the image synthesized by the generator.

[0087] L GAN (G T The loss function (L) is used in GANs. Its calculation requires inputting a reference image and a first fitted image into a discriminator for encoding, resulting in a label used to calculate the loss. Recon (G T The reconstruction loss, also known as L1 loss, is calculated by subtracting the reference image from the first fitted image to determine the absolute difference, which is used to evaluate the realism of the first fitted image. For the overall loss, containing L GAN and L Recon Two loss functions are used, and the purpose of training the generator is to reduce L. GAN The loss function aims to make the first fitted image resemble the reference image more closely. The discriminator's goal is to maximize L. GAN The goal is to distinguish between the reference image and the first fitted image as much as possible. The generator during training is also a process that continuously decreases L. Recon The process.

[0088] In some embodiments, after obtaining the first true value based on the first loss value and the second loss value, the method further includes:

[0089] S304: When training the first model, if the first true value is less than the first preset threshold, adjust the parameter values ​​of the first model according to the similarity between the first fitted image and the reference image.

[0090] S305: Train the first model with adjusted parameter values ​​until convergence, and input the next set of reference training data into the first model.

[0091] After obtaining the first true value, before iteratively training the first model, the first true value needs to be compared with a first preset threshold to determine whether the parameters of the first model need to be adjusted. If the first true value is less than the first preset threshold, the parameter values ​​of the first model are adjusted according to the similarity between the first fitted image and the reference image. The first model with adjusted parameter values ​​is trained until convergence, and the next set of reference training data is input into the first model. If the first true value is not less than the first preset threshold, the current first model is trained until convergence, and the next set of reference training data is input into the first model.

[0092] S400: Determine the second true value corresponding to the second model based on the similarity between the first fitted image and the second fitted image, and train the second model based on the second true value.

[0093] like Figure 4 and Figure 5 As shown, after obtaining the first fitted image and the second fitted image, the similarity between the first fitted image and the second fitted image is compared to determine the difference between the second fitted image output by the second model and the first fitted image output by the first model, so as to adjust the parameters of the second model and facilitate the subsequent training and learning of the second model based on the current first fitted image.

[0094] In this application, the second model no longer fits to the real image, i.e., the reference image, but rather to the first fitted image output by the first model. It only needs to fit a structure similar to the output of the first model, which greatly reduces the learning difficulty. This makes the second model run faster and meets the requirements for running on low-computing-power devices such as mobile phones.

[0095] In some embodiments, determining the second true value corresponding to the second model based on the similarity between the first fitted image and the second fitted image, and training the second model based on the second true value includes:

[0096] S401: Obtain the first fitted image and the second fitted image;

[0097] S402: Calculate the second true value corresponding to the second model based on the similarity between the first fitted image and the second fitted image; the second true value is used to characterize the difference between the output results of the first model and the second model;

[0098] During the training and learning process, the second model needs to calculate the structural similarity and perceptual similarity between the second model and the first model to evaluate the difference between the first fitted image output by the first model and the second fitted image output by the second model. The second true value is used to characterize the difference between the output results of the first model and the second model.

[0099] In some embodiments, calculating the second true value corresponding to the second model based on the similarity between the first fitted image and the second fitted image includes:

[0100] S4021: Calculate the structural similarity and perceptual similarity based on the first fitted image and the second fitted image, respectively;

[0101] S4022: Obtain a first weight value through the structural similarity and the first coefficient, and obtain a second weight value through the perceptual similarity and the second coefficient;

[0102] S4023: Obtain the second true value based on the first weight value and the second weight value.

[0103] In some embodiments, structural similarity, perceptual similarity, and the second true value can be calculated using the following loss function:

[0104]

[0105]

[0106] L KD (p t ,p s )=λ SSIM L SSIM +λ feature L feature +λ e L e (6)

[0107] Where p t ,p s L represents the first fitted image output by the first model and the second fitted image output by the second model. SSIM Structural similarity loss and perceptual loss are used to evaluate p t ,p s The difference between them, the perceptual loss is the feature reconstruction loss L feature and style reconstruction loss L e Two types, u t u s The average value representing brightness. σ represents the standard deviation of contrast. ts Structural similarity covariance, where C2C1 is a fixed constant. This is a pre-trained VGG network, used here to evaluate p. t ,p s Feature similarity, L e We also use the intermediate layer of the VGG network for evaluation. λ SSIM , λ feature , λ e These are the coefficients for various losses, used to define the weights of each loss.

[0108] After obtaining the second true value, before iteratively training the second model, it is necessary to compare the second true value with the first preset threshold to determine whether the parameters of the second model need to be adjusted.

[0109] S403: If the second true value is less than the second preset threshold, adjust the parameter values ​​of the second model according to the similarity between the first fitted image and the second fitted image;

[0110] S404: Train the second model with adjusted parameter values ​​until convergence, and input the next set of reference training data into the second model.

[0111] S405: If the second true value is equal to or greater than a preset threshold, train the second model until convergence, and output the second model as the target model.

[0112] If the second true value is less than a second preset threshold, the parameter values ​​of the second model are adjusted according to the similarity between the first fitted image and the second fitted image. The second model with adjusted parameter values ​​is trained until convergence, and the next set of reference training data is input into the second model. If the second true value is equal to or greater than the preset threshold, the second model is trained until convergence, and the second model is output as the target model.

[0113] It is worth noting that the first and second models in this application are trained simultaneously, iterating over both models without any specific order. During training, the first and second models are trained together, with the second model learning based on the first model. The first model is used to assist the second model's learning, allowing the second model to achieve a similar fitting ability with fewer parameters and less computation. After training, the model is deployed to low-computing-power devices. If the first model is trained first, and then the second model is trained based on the first model, to avoid model collapse and gradient vanishing phenomena due to a large difference in model size, the second model needs to maintain a dynamic balance with the first model. If the number of parameters in the first and second models is similar, the second model will not be applicable to low-computing-power devices. If the first model is a pre-trained model, it cannot guide the learning of the second model in the initial stage, leading to problems in the early stages and increasing the risk of overfitting.

[0114] This application trains the first model and the second model together. The first model can determine the fitting of the second model in the initial stage of training and learn iteratively in sync. The two models learn together and are closely combined to obtain a good fitting effect. The second model can fit the parameters more flexibly. Therefore, there is no problem that the first model and the second model must be balanced and the number of parameters cannot be too different. In this way, the second model can use a smaller number of parameters, so that the trained second model can be deployed on low computing power devices, enriching the application scenarios of digital humans.

[0115] S500: Train the first model and the second model until convergence, and output the trained second model as the target model; wherein, the target model is used to generate digital human videos based on target audio input by the user.

[0116] In this application, the first model and the second model generate a first fitted image and a second fitted image based on reference training data, respectively. The parameters of the first model are adjusted based on the similarity difference between the first fitted image and the reference image. The output difference between the first model and the second model is determined based on the first fitted image and the second fitted image, thereby adjusting the parameters of the second model. The first model and the second model are trained iteratively simultaneously until they converge. In this way, the second model does not need to directly learn the real images in the reference training data, but only needs to fit the first fitted image output by the first model. This allows it to achieve an output effect close to that of the first model with fewer model parameters and less computation, making the second model applicable to low-computing devices and thus enriching the application scenarios of digital humans.

[0117] In some embodiments, this application provides a digital human video generation method, including:

[0118] S600: Acquires the image and target audio input by the user;

[0119] S601: Based on the image, a template video is synthesized, and the template video is preprocessed to obtain a target image; wherein, the target image is a face image with the mouth area removed;

[0120] S602: Import the target image and the target audio into the target model to generate a predicted face image;

[0121] S603: The predicted face image and the target audio are spliced ​​together to obtain a digital human video.

[0122] After the trained second model is output as the target model, the user only needs to upload a clear, frontal photo to synthesize a digital human video with accurate lip movements and realistic results. The digital human workflow is detailed below. After the user uploads an image, a template video is synthesized based on it. The template video then undergoes preprocessing, similar to the training process, to obtain a face crop image and a mask image. The user also needs to provide audio data, which can be recorded directly with a mobile phone or uploaded existing audio data. Finally, the face mask image and audio data are fed into the target model to obtain the digital human video.

[0123] In this application, after the target model obtains the image, it outputs a template video. The template video undergoes preprocessing: first, the video is converted into an image; then, face detection is performed, and the detected faces are cropped and saved. Next, the user's voice input is obtained, and the voice and face images are input into the trained target model. The target model outputs the face corresponding to the voice based on the voice and image, and the output face image is then stitched back into the original video to obtain the digital human video.

[0124] This application can synthesize digital human videos with accurate lip movements and natural actions. It can synthesize a video corresponding to the audio with only one image. Because the target model has low parameter count and fast running speed, it can run offline on low computing power devices such as mobile phones. It can be widely used in scenarios such as news broadcasting, customer service, and AI assistants, greatly improving the interactive effect.

[0125] like Figure 6 As shown, in some embodiments, this application provides a digital human video generation system, the system comprising:

[0126] The acquisition module is used to acquire reference training data, which includes reference images and reference audio.

[0127] The training module inputs the reference training data into a preset first model and a second model, respectively, to instruct the first model to generate a first fitted image based on the reference training data, and to instruct the second model to generate a second fitted image based on the reference training data; wherein the parameter values ​​of the second model are smaller than the parameter values ​​of the first model;

[0128] The first true value corresponding to the first model is determined based on the similarity between the first fitted image and the reference image, and the first model is trained based on the first true value.

[0129] The second true value corresponding to the second model is determined based on the similarity between the first fitted image and the second fitted image, and the second model is trained based on the second true value.

[0130] The first model and the second model are trained until convergence, and the trained second model is output as the target model; wherein, the target model is used to generate digital human videos based on target audio input by the user.

[0131] The reference training data in the acquisition module includes high-definition videos recorded by the database itself and high-definition data downloaded from the internet. Data cleaning was performed before training, removing issues such as misaligned lip movements, multiple voices, significant environmental noise, no human figures or faces, and severe occlusion. Clean data helps the model converge faster and achieve better video synthesis results. It should be noted that the reference training data here refers to general training video data, not videos of specific models for digital human synthesis.

[0132] This application provides a digital human video generation system, which includes an acquisition module and a training module. The acquisition module is used to acquire reference training data. The training module is used to train a preset second model to obtain a target model with low parameter count and capable of running on low-computing-power devices. Users can generate digital human videos based on target audio input from the target model.

[0133] like Figure 7 As shown, in some embodiments, the digital human video generation system provided in this application further includes:

[0134] A generation module is used to synthesize a template video based on images input by the user, and to preprocess the template video to obtain a target image; wherein the target image is a face image with the mouth area removed;

[0135] The target image and the target audio input by the user are imported into the target model to generate a predicted face image;

[0136] The predicted face image is spliced ​​with the target audio to obtain a digital human video.

[0137] The generation module is equipped with an image driving unit and a lip-sync generation unit. The generation module controls the image driving unit to drive the image to synthesize a template video, and controls the lip-sync generation unit to drive the template video according to the input speech to synthesize a digital human video that matches the audio.

[0138] The image-driven unit is used to convert input images into videos with head movements and expressions. This unit is equipped with an image-driven model, which is trained using a large amount of unlabeled video data. This model can drive an image with a video, transferring the head movements and expressions from the video to the image, and synthesizing a video with movements and expressions. This video is used as input for subsequent modules and as a template video for subsequent digital human-driven processes.

[0139] The generation module also includes a digital human driving unit, which uses input audio data to drive the template video synthesized by the image driving unit, synthesizing a video that matches the speech. By training with recorded video data, it can learn the correspondence between speech and lip movements.

[0140] This application also provides a digital human video generation system, the system including an acquisition module, a training module, and a generation module; the acquisition module is used to acquire reference training data and user-input images and target audio; the training module is used to train a preset second model to obtain a target model with low parameter count and capable of running on low-computing-power devices; the generation module is used to control the target model to synthesize and output a digital human video based on the user-input images and target audio.

[0141] The digital human video generation system proposed in this application can train a digital human model and generate digital human videos based on only a single image. Furthermore, the digital human model only requires server-side training; the video generation process can be performed on low-computing-power devices such as mobile phones, without the need for high-performance graphics cards. Based on this system, accurate lip-syncing and natural-looking digital human videos can be generated from audio. Moreover, this technology can run on low-computing-power devices and offline, significantly lowering the barrier to entry and enabling the application of digital humans in more scenarios.

[0142] In some embodiments, this application also provides an electronic device, including:

[0143] processor;

[0144] Memory for storing the executable instructions of the processor;

[0145] The processor is configured to execute the above-described training method for the digital human generation model and the digital human video generation method by executing the executable instructions.

[0146] As can be seen from the above embodiments, this application provides a training method for a digital human generation model, a digital human video generation method, and a system. The training method includes: acquiring reference training data; inputting the reference training data into a preset first model and a second model respectively, to instruct the first model to generate a first fitted image based on the reference training data, and to instruct the second model to generate a second fitted image based on the reference training data; determining a first real value corresponding to the first model based on the similarity between the first fitted image and the reference image, and training the first model based on the first real value; determining a second real value corresponding to the second model based on the similarity between the first fitted image and the second fitted image, and training the second model based on the second real value; training the first model and the second model until convergence, and outputting the trained second model as the target model. The second model does not need to directly learn the real images in the reference training data, but only needs to fit the first fitted image output by the first model, so as to achieve an output effect close to that of the first model with fewer model parameters and less computation, making the second model applicable to low-computing-power devices, thereby enriching the application scenarios of digital humans.

Claims

1. A training method for a digital human generation model, characterized in that, The method includes: Acquire reference training data and preprocess the reference training data. The preprocessed reference training data includes reference images and reference audio. The reference training data is input into a preset first model and a second model, respectively, to instruct the first model to generate a first fitted image based on the reference training data, and to instruct the second model to generate a second fitted image based on the reference training data; wherein the parameter values ​​of the second model are less than the parameter values ​​of the first model; the audio and image coding networks of the first model adopt a ResNet structure, the second model adopts a regular convolutional structure, and the number of channels in the intermediate network layers of the second model is 0.25 to 0.4 times that of the first model; The first true value corresponding to the first model is determined based on the similarity between the first fitted image and the reference image, and the first model is trained based on the first true value. The second true value corresponding to the second model is determined based on the similarity between the first fitted image and the second fitted image, and the second model is trained based on the second true value. The first model and the second model are trained until convergence, and the trained second model is output as the target model; wherein, the target model is used to generate digital human videos based on target audio input by the user; After obtaining the reference training data, the preprocessing of the reference training data includes: The reference audio is obtained by using an audio feature extraction model to extract audio features from the reference training data based on the word-level vector quantization bottleneck. The video in the reference training data is converted into an image sequence, and the face detection program is used to detect the face location; The face is cropped based on its position, and the mouth area of ​​the face image is masked to obtain a reference image.

2. The training method for the digital human generation model according to claim 1, characterized in that, The second model is then trained based on the similarity between the first fitted image and the second fitted image to determine the second true value corresponding to the second model, including: Obtain the first fitted image and the second fitted image; The second true value corresponding to the second model is calculated based on the similarity between the first fitted image and the second fitted image; the second true value is used to characterize the difference between the output results of the first model and the second model. If the second true value is less than the second preset threshold, the parameter values ​​of the second model are adjusted according to the similarity between the first fitted image and the second fitted image. The second model, with its parameter values ​​adjusted, is trained until convergence, and the next set of reference training data is input into the second model.

3. The training method for the digital human generation model according to claim 2, characterized in that, The calculation of the second true value corresponding to the second model based on the similarity between the first fitted image and the second fitted image includes: Calculate structural similarity and perceptual similarity based on the first fitted image and the second fitted image, respectively. A first weight value is obtained by using the structural similarity and the first coefficient, and a second weight value is obtained by using the perceptual similarity and the second coefficient; The second true value is obtained based on the first weight value and the second weight value.

4. The training method for the digital human generation model according to claim 2, characterized in that, The method further includes: If the second true value is equal to or greater than a preset threshold, the second model is trained until convergence, and the second model is output as the target model.

5. The training method for the digital human generation model according to claim 1, characterized in that, The first true value corresponding to the first model is determined based on the similarity between the first fitted image and the reference image, including: A first loss value is calculated based on the similarity between the first fitted image and the reference image; The second loss value is calculated based on the difference between the first fitted image and the reference image; The first true value is obtained based on the first loss value and the second loss value; the first true value is used to characterize the difference between the first fitted image output by the first model and the reference image.

6. The training method for the digital human generation model according to claim 5, characterized in that, After obtaining the first true value based on the first loss value and the second loss value, the process further includes: When training the first model, if the first true value is less than the first preset threshold, the parameter values ​​of the first model are adjusted according to the similarity between the first fitted image and the reference image. The first model, after its parameter values ​​have been adjusted, is trained until convergence, and the next set of reference training data is input into the first model.

7. A method for generating digital human videos, characterized in that, The digital human video generation method uses the target model described in any one of claims 1 to 6, and the method includes: Obtain the image and target audio input by the user; A template video is synthesized based on the image, and a target image is obtained by preprocessing the template video; wherein, the target image is a face image with the mouth area removed; The target image and the target audio are imported into the target model to generate a predicted face image; The predicted face image is spliced ​​with the target audio to obtain a digital human video.

8. A digital human video generation system, characterized in that, The system includes: The acquisition module is used to acquire reference training data and preprocess the reference training data. The preprocessed reference training data includes reference images and reference audio. The preprocessing of the reference training data includes: using an audio feature extraction model to extract audio features from the reference training data based on word-level vector quantization bottleneck to obtain reference audio; converting the video in the reference training data into an image sequence and using a face detection program to detect face positions; cropping the face according to the face position and masking the mouth area of ​​the face image to obtain a reference image. The training module inputs the reference training data into a preset first model and a second model, respectively, to instruct the first model to generate a first fitted image based on the reference training data, and to instruct the second model to generate a second fitted image based on the reference training data; wherein the parameter values ​​of the second model are less than the parameter values ​​of the first model; the audio and image coding networks of the first model adopt a ResNet structure, the second model adopts a regular convolutional structure, and the number of channels in the intermediate network layers of the second model is 0.25 to 0.4 times that of the first model; The first true value corresponding to the first model is determined based on the similarity between the first fitted image and the reference image, and the first model is trained based on the first true value. The second true value corresponding to the second model is determined based on the similarity between the first fitted image and the second fitted image, and the second model is trained based on the second true value. The first model and the second model are trained until convergence, and the trained second model is output as the target model; wherein, the target model is used to generate digital human videos based on target audio input by the user.

9. The digital human video generation system according to claim 8, characterized in that, The system also includes: A generation module is used to synthesize a template video based on images input by the user, and to preprocess the template video to obtain a target image; wherein the target image is a face image with the mouth area removed; The target image and the target audio input by the user are imported into the target model to generate a predicted face image; The predicted face image is spliced ​​with the target audio to obtain a digital human video.

10. An electronic device, characterized in that, include: processor; Memory for storing the executable instructions of the processor; The processor is configured to execute the training method of the digital human generation model according to any one of claims 1-6, and / or execute the digital human video generation method according to claim 7, by executing the executable instructions.

Images