Digital human generation model training method, digital human generation method and related device

Abstract

The invention discloses a digital human generation model training method, a digital human generation method and a related device, and relates to the technical field of machine learning, and the method comprises the steps: obtaining a first image sample and a lip-shaped driving audio sample, inputting the first image sample and the lip-shaped driving audio sample into an initial digital human generation model, a digital human image corresponding to a lip-shaped driving audio sample is obtained, fuzzy detection is performed on a lip area of the digital human image to obtain a lip fuzzy degree predicted value, an initial digital human generation model is trained according to at least a lip fuzzy loss value, and the lip fuzzy loss value is used for measuring the difference between the lip fuzzy degree predicted value and a target value. The target value is a minimum fuzzy degree threshold value enabling the lip region to meet a preset detail definition requirement. According to the invention, model training is carried out based on the lip fuzzy loss value, so that the model can master the capability of depicting the details of the lip region, thereby improving the definition of the lip details of the digital human image generated by the model in the reasoning stage.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a method for training a digital human generation model, a digital human generation method, and related apparatus. Background Technology

[0002] Digital humans have applications in many scenarios such as entertainment, education, finance, and healthcare. For example, digital humans can act as virtual anchors to achieve 24-hour uninterrupted live streaming, and digital human characters in film and animation can reduce the cost of live shooting.

[0003] Existing digital human generation technologies, such as those based on generative adversarial networks and those based on diffusion models, can generate digital humans with excellent overall effects. However, in the lip representation area of ​​digital humans, there are still problems such as blurred tooth edges, loss of lip gloss details, and unclear lip lines, which have become key issues restricting the practical application of digital human technology. Summary of the Invention

[0004] In view of the above problems, this application provides a digital human generation model training method, a digital human generation method, and related apparatus to improve the clarity of lip details in digital human images. The specific solution is as follows:

[0005] The first aspect of this application provides a method for training a digital human generation model, including:

[0006] A first image sample and a lip-driven audio sample are obtained, wherein the first image sample is obtained by masking the lip region of the original image;

[0007] The first image sample and the lip-driven audio sample are input into the initial digital human generation model to obtain the digital human image corresponding to the lip-driven audio sample.

[0008] Blur detection is performed on the lip region of the digital human image to obtain a predicted value of the degree of lip blur.

[0009] The initial digital human generation model is trained based on at least a lip blur loss value, wherein the lip blur loss value is used to measure the difference between the predicted lip blur degree and the target value, wherein the target value is the minimum blur degree threshold that makes the lip region meet the preset detail clarity requirements.

[0010] In one possible implementation, the blur detection of the lip region of the digital human image to obtain a predicted value of the degree of lip blur includes:

[0011] The digital human image is subjected to facial key point detection using a pre-trained facial key point detection model to obtain facial key point detection results. The facial key point detection model is trained using training images labeled with facial key point tags as training data.

[0012] Based on the facial key point detection results, the lip region is identified and cropped from the digital human image to obtain the target lip image;

[0013] The target lip image is input into a pre-trained target lip blur prediction model to obtain the predicted value of the lip blur degree.

[0014] In one possible implementation, the training process of the target lip blur prediction model includes:

[0015] A second image sample is obtained, and the lip region in the second image sample is identified and cropped to obtain a lip image sample. The second image sample is a real face image or digital human image in which the lip region meets the required level of detail.

[0016] The lip image sample is blurred at different levels using pre-configured blur parameters to obtain one or more blurred image samples corresponding to the lip image sample, and the blur level is marked for each blurred image sample.

[0017] Using one or more blurred image samples and their corresponding blur levels, the initial lip blur prediction model is trained to obtain the trained lip blur prediction model, which is then used as the target lip blur prediction model.

[0018] In one possible implementation, the target lip blur prediction model consists of a sequentially cascaded input layer, one or more feature extraction layers, one or more fully connected layers, and an output layer, wherein the feature extraction layer consists of convolutional blocks and pooling layers.

[0019] When the last fully connected layer outputs 1D data, the loss function used for training the target lip blur prediction model is the mean squared error loss function, and the evaluation metric on the validation set is the mean absolute error.

[0020] When the last fully connected layer outputs multidimensional data, the loss function used for training the target lip fuzziness prediction model is the cross-entropy loss function, and the evaluation metric on the validation set is the fuzziness level accuracy. The total dimension of the multidimensional data is the same as the number of fuzziness levels.

[0021] In one possible implementation, the process of determining the lip blurring loss value includes:

[0022] When the last fully connected layer outputs 1D data, the distance between the predicted value of lip blur and the target value is calculated, and the absolute value of the distance is taken as the lip blur loss value.

[0023] When the last fully connected layer outputs multidimensional data, the cross-entropy between the predicted lip blur degree and the target value is calculated as the lip blur loss value.

[0024] In one possible implementation, the initial digital human generation model is a digital human generator;

[0025] The step of training the initial digital human generation model based on at least the lip blur loss value includes:

[0026] The discriminator corresponding to the digital human generator is used to perform discrimination processing on the digital human image to obtain first discrimination information, and the discriminator is used to perform discrimination processing on the original image corresponding to the digital human image to obtain second discrimination information;

[0027] An adversarial loss value is generated based on the first discrimination information and the second discrimination information;

[0028] The digital human generator and the discriminator are jointly trained based on the lip blurring loss value and the adversarial loss value.

[0029] In one possible implementation, the joint training of the digital human generator and the discriminator based on the lip blurring loss value and the adversarial loss value includes:

[0030] The features of the digital human image are extracted using a pre-trained Visual Geometric Group Network (VGG) model to obtain the first image features, and the features of the original image corresponding to the digital human image are extracted using the VGG model to obtain the second image features.

[0031] Generate a perceptual loss value based on the first image features and the second image features;

[0032] The digital human generator and the discriminator are jointly trained based on the adversarial loss value, the lip blurring loss value, and the perceptual loss value.

[0033] In one possible implementation, the training termination condition of the initial digital human generation model consists of one or more of the following conditions: the predicted lip blur level on the validation set stabilizes within a preset blur level range, the total loss value converges, the maximum number of training rounds is reached, and a training stop signal is received.

[0034] A second aspect of this application provides a method for generating a digital human, comprising:

[0035] Acquire reference image and target lip-driven audio;

[0036] The lip region of the reference image is masked to obtain a masked image that covers the lip region;

[0037] The mask image and the target lip-driven audio are input into a pre-trained target digital human generation model to obtain a digital human image corresponding to the target lip-driven audio. The target digital human generation model is obtained by training an initial digital human generation model using the digital human generation model training method of the first aspect or any implementation of the first aspect described above.

[0038] A third aspect of this application provides a computer program product, including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the digital human generation model training method of the first aspect or any implementation thereof, or the digital human generation method of the second aspect.

[0039] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0040] The memory is used to store computer programs;

[0041] The processor is used to execute the computer program so that the electronic device can implement the digital human generation model training method of the first aspect or any implementation thereof, or the digital human generation method of the second aspect.

[0042] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the digital human generation model training method of the first aspect or any implementation thereof, or the digital human generation method of the second aspect.

[0043] By employing the aforementioned technical solution, the digital human generation model training method provided in this application acquires a first image sample and a lip-driven audio sample, and inputs the acquired first image sample and lip-driven audio sample into an initial digital human generation model to obtain a digital human image corresponding to the lip-driven audio sample. To enable the model to generate digital human images with clearer details in the lip region, this application can further perform blur detection on the lip region of the digital human image to obtain a predicted value for lip blur. The initial digital human generation model is then trained based on at least a lip blur loss value, which measures the difference between the predicted lip blur value and a target value. The target value is the minimum blur threshold that makes the lip region meet the preset detail clarity requirements. Because a lip blur loss value is incorporated into the model training process to penalize lip region blur, the model can learn how to minimize the lip blur loss value during training, thereby generating digital human images where the predicted lip blur value is closer to the target value. Therefore, after training, the model can master the ability to depict lip region details, thus improving the lip detail clarity of the digital human image generated by the model during the inference stage, and consequently improving the practicality of the digital human image. Attached Figure Description

[0044] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0045] Figure 1 A schematic diagram of a system architecture provided for this application;

[0046] Figure 2 A flowchart illustrating a digital human generation model training method provided in this application;

[0047] Figure 3 A flowchart illustrating a digital human generation method provided in this application;

[0048] Figure 4 A schematic diagram of the structure of a digital human generation model training device provided in this application;

[0049] Figure 5 A schematic diagram of the structure of a digital human generation device provided in this application;

[0050] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0051] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0052] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0053] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0054] Currently, digital human generation technology mainly includes the following methods:

[0055] First, digital human generation methods based on Generative Adversarial Networks (GANs), such as StyleGAN3 (Algorithmic Latent Space Exploration GAN) and ProGAN (Progressive Growing of Generative Adversarial Networks), generate high-resolution digital human images through adversarial training between the generator and the discriminator. While the introduction of GANs can better capture overall facial contours and skin color features, it lacks the ability to depict details in fine areas such as the lips and eyes, easily leading to problems like blurred lip edges and the merging of teeth and lip boundaries.

[0056] Second, digital human generation methods based on diffusion models, such as Stable Diffusion and DALLE-2, control digital human generation through text prompts. Since the text prompts can be flexibly changed, the digital human generation process can adjust facial styles more flexibly. However, due to the "global smoothness" of the diffusion process, local details in the lip area (such as lip lines and tooth textures) are easily blurred.

[0057] Third, digital human generation methods based on 3D reconstruction, such as EVA3D and NeRF (Neural Radiance Fields), generate digital humans through 3D geometric structures. Although they can ensure the three-dimensionality of the face, the generation of 2D textures in the lip area still relies on 2D image models, and there is also a problem of loss of detail.

[0058] In summary, current digital human generation methods suffer from blurred details in the lips, which can only be addressed through post-processing. This can be achieved using super-resolution models such as ESRGAN (Enhanced Super-Resolution Generative Adversarial Networks) and Real-ESRGAN (Real-World ESRGAN) to improve the resolution of the lip region, or by enhancing the gloss and contrast of the lips using the Retinex algorithm. However, these methods cannot solve the fundamental problem of "detail loss during the generation stage." If the generated digital human image lacks details such as tooth edges and lip lines, post-processing can only amplify the blurred areas; it cannot create details from scratch and is prone to producing jagged artifacts.

[0059] In order to directly generate digital human images with clear lip details during the digital human generation stage, this application provides a digital human generation model training method, a digital human generation method, and related apparatus.

[0060] Optionally, the digital human generation model training method, digital human generation method, and related devices can be applied to scenarios that require the generation of digital humans, such as generating virtual anchors to achieve 24-hour uninterrupted live streaming; generating virtual teachers to teach students; and generating digital human characters in film and animation to increase the fun and viewing effect of film and animation.

[0061] It should be noted that the above scenarios are merely examples and are not intended to limit this application.

[0062] Optionally, the digital human generation model training method and digital human generation method provided in this application can be applied to, for example... Figure 1 The system architecture shown includes a terminal 100 and a server 200. The server 200 may include one or more servers (…). Figure 1 (This example uses a server as an illustration).

[0063] Either terminal 100 or server 200 can be used independently to execute the digital human generation model training method and the digital human generation method provided in the embodiments of this application. Alternatively, terminal 100 and server 200 can also be used collaboratively to execute the digital human generation model training method and the digital human generation method provided in the embodiments of this application.

[0064] The following description Figure 1 The product form of the mid-terminal 100;

[0065] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, wearable device, vehicle-mounted device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0066] To enable those skilled in the art to better understand this application, the digital human generation model training method of this application embodiments will be described in detail below with reference to the accompanying drawings.

[0067] Reference Figure 2 , Figure 2 This application provides a flowchart illustrating a digital human generation model training method, as shown in the embodiments below. Figure 2 As shown, the training method for this digital human generation model may include:

[0068] Step S101: Obtain a first image sample and a lip-driven audio sample. The first image sample is obtained by masking the lip region of the original image.

[0069] Here, the original image can be a single frame containing a facial region, or it can be a frame from a video sequence containing a facial region.

[0070] For example, a facial image of user A can be extracted from a video sequence taken by user A in the past, and this facial image can be used as the original image to generate a digital human image of user A according to this embodiment.

[0071] In this embodiment, the lip area of ​​the original image can be masked to obtain an image that covers the lip area, which serves as the first image sample for training the initial digital human generation model. In the scenario of digital human generation, the first image sample determines the external appearance of the digital human.

[0072] To obtain a vivid, believable, and audio-visual synchronized digital human image, it is also necessary to acquire lip-driven audio samples, which determine how the digital human's lips change.

[0073] Step S102: Input the first image sample and the lip-driven audio sample into the initial digital human generation model to obtain the digital human image corresponding to the lip-driven audio sample.

[0074] It is understandable that different lip shapes can produce different sounds. In order to make the lip region in the digital human image more consistent with the lip-driven audio sample, the first image sample and the lip-driven audio sample can be input into the initial digital human generation model. Through the internal processing of the model, the lip shape adapted to the audio change is generated in the lip region covered in the first image sample, so as to obtain a digital human image that is highly consistent with the lip-driven audio sample.

[0075] It should also be noted that if the digital human image corresponding to the lip-driven audio sample can include one or more frames of digital human, the specific number of frames depends on the length of the lip-driven audio sample, and this application does not impose a specific limitation.

[0076] Step S103: Perform blur detection on the lip region of the digital human image to obtain the predicted value of the lip blur degree.

[0077] In order to train the model to generate digital human images with clear lip details, this embodiment can perform blur detection on the lip region of the digital human image generated based on the samples mentioned above, and obtain a predicted value of lip blur degree that represents the degree of blur of the lip region of the digital human image.

[0078] Optionally, the predicted blur level of the lips can refer to a preset blur level value, such as a preset blur level of 1-10. Then, through blur detection, a number from 1 to 10 can be output to represent the blur level of the lip region in the current digital human image.

[0079] Of course, the predicted value for the degree of lip blur can also be other values, such as the probability distribution of each blur level or a real value that can be mapped to the blur level value, such as 1.1 rounded to level 1 blur, 1.8 rounded to level 2 blur, etc. This application does not impose specific limitations.

[0080] Step S104: Train the initial digital human generation model based on at least the lip blur loss value. The lip blur loss value is used to measure the difference between the predicted value of the lip blur degree and the target value. The target value is the minimum blur degree threshold that makes the lip region meet the preset detail clarity requirements.

[0081] Specifically, in order to penalize the model when it generates digital human images with unclear lip details, this embodiment can generate a lip blur loss value based on the predicted value and target value of the lip blur degree, and then use the lip blur loss value to train the initial digital human generation model.

[0082] The greater the difference between the predicted value and the target value of lip blur, the more blurred the lip area of ​​the digital human image is, and the greater the lip blur loss value is; conversely, the smaller the difference between the predicted value and the target value of lip blur, the clearer the lip area of ​​the digital human image is, and the smaller the lip blur loss value is.

[0083] In other words, the more blurred the lip area of ​​the digital human image, the greater the penalty is given to the model, so that the model learns the ability to generate digital human images with clear lip areas during the training process. This ensures that the clearest possible digital human image is generated during the digital human generation stage, avoiding artifacts introduced by post-processing that result in clear but unrealistic lips in the digital human.

[0084] Taking the initial digital human generation model as the digital human generator (such as the generator in StyleGAN3) as an example, the process of "training the initial digital human generation model based on at least the lip blur loss value" mentioned earlier will be introduced.

[0085] In this embodiment, the discriminator corresponding to the digital human generator can be used to discriminate the digital human image to obtain first discrimination information, and the discriminator can also be used to discriminate the original image corresponding to the digital human image to obtain second discrimination information. Then, an adversarial loss value is generated based on the first and second discrimination information. The digital human generator and the discriminator are jointly trained based on the lip blurring loss value and the adversarial loss value. That is, the parameters of the target lip blurring prediction model are fixed, and only the digital human generator and the discriminator are trained; the adversarial loss value can measure the adversarial effect between the digital human generator G and the discriminator D.

[0086] Optionally, the input to the digital human generator is a 128-dimensional vector, and the output is a 1024×1024×3 digital human image. That is, in step S102, the first image sample and the lip-driven audio sample can be processed into 128-dimensional vectors and then input into the digital human generator to obtain a 1024×1024×3 digital human image.

[0087] It should be noted that the above dimensions are merely examples and are not intended to limit this application.

[0088] Furthermore, the digital human image output by the digital human generator and the original image corresponding to the digital human image (i.e. the image corresponding to the first image sample that does not cover the lip area) are respectively input into the discriminator to obtain the first discrimination information and the second discrimination information.

[0089] Optionally, the first and second discriminant information respectively include the discriminant information output by each layer of the discriminator. Based on this, the adversarial loss value can be calculated using the following formula.

[0090] Formula (1);

[0091] in, This represents the adversarial loss value (i.e., the Wasserstein distance loss value). This indicates that the second discrimination information contains the discriminator's first... Features of the layer This indicates that the first discrimination information contains the discriminator's first... Features of the layer The value range is 1-L; Indicates the distribution of real data Medium-sampled original image samples Find the expected value.

[0092] To encourage the model to generate structures and textures consistent with the target in the feature space, thereby producing clearer and more visually persuasive digital human images, optionally, a pre-trained VisualGeometry Group (VGG) model can be used to extract features from the digital human image to obtain the first image features, and the VGG model can be used to extract features from the original image corresponding to the digital human image to obtain the second image features. Then, a perceptual loss value is generated based on the first and second image features. Based on the perceptual loss value, adversarial loss value, and lip blurring loss value, the digital human generator and discriminator are jointly trained.

[0093] Optionally, the VGG model can specifically be the VGG16 model.

[0094] Optionally, the formula for calculating the perceptual loss value is:

[0095] Formula (2);

[0096] in, This represents the perceived loss value. Indicates the first image feature, This represents the second image feature.

[0097] Optionally, the process of "jointly training the digital human generator and the discriminator based on the perceptual loss value, the adversarial loss value, and the lip blurring loss value" may include: weighting and summing the perceptual loss value, the adversarial loss value, and the lip blurring loss value to obtain the total loss value, and jointly training the digital human generator and the discriminator based on the total loss value.

[0098] For example, the formula for calculating the total loss value is:

[0099] Formula (3);

[0100] in, This represents the total loss value. Indicates the adversarial loss value. This represents the perceived loss value. This represents the loss value for blurred lips. The weighting coefficients represent the adversarial loss values. The weighting coefficients represent the perceived loss values. The three weighting coefficients represent the loss value of lip blurring. They can be adjusted according to the actual generated effect.

[0101] In summary, this embodiment allows the first image sample and lip-driven audio sample to be input into the digital human generator in each iteration of model training to obtain a digital human image. Then, the lip area is cropped and input into the target lip blur prediction model to obtain the predicted lip blur degree P, and then calculated according to the previous process. The parameters of the digital human generator and discriminator are updated through backpropagation.

[0102] Optionally, the training termination condition of the initial digital human generation model (such as a digital human generator) consists of one or more of the following conditions: the predicted lip blur level on the validation set stabilizes within a preset blur level range (such as level 1-2), the total loss value converges, the maximum number of training rounds is reached, and a training stop signal is received.

[0103] Optionally, the training stop signal can be a signal triggered when the user subjectively evaluates the digital human image output during the model training process, and the subjective evaluation indicates that the teeth have clear edges, visible lip lines, and natural lip gloss.

[0104] Of course, if the initial digital human generation model is another model architecture, such as a diffusion model, other loss functions adapted to the model can be used, such as L1 loss function, L2 loss function, etc. This application does not impose specific limitations.

[0105] The digital human generation model training method provided in this application acquires a first image sample and a lip-driven audio sample, and inputs the acquired first image sample and lip-driven audio sample into an initial digital human generation model to obtain a digital human image corresponding to the lip-driven audio sample. To enable the model to generate digital human images with clearer details in the lip region, this application can further perform blur detection on the lip region of the digital human image to obtain a predicted value of lip blur. The initial digital human generation model is trained based on at least a lip blur loss value, which measures the difference between the predicted lip blur value and a target value. The target value is the minimum blur threshold that makes the lip region meet the preset detail clarity requirements. Because a lip blur loss value is incorporated into the model training process to penalize lip region blur, the model can learn how to minimize the lip blur loss value during training to generate digital human images with lip blur prediction values ​​closer to the target value. Therefore, after training, the model can master the ability to depict lip region details, thereby improving the lip detail clarity of the digital human image generated by the model during the inference stage, and thus improving the practicality of the digital human image.

[0106] Meanwhile, the lip blurring loss value of this application is compatible with various existing frameworks such as GAN, diffusion model, and 3D digital human generation. It does not require reconstruction of the entire generation network, but only the addition of a fuzzy supervision module, which reduces the cost of engineering implementation.

[0107] In some embodiments of this application, the process of step S103, "performing blur detection on the lip region of a digital human image to obtain a predicted value of lip blur level", is described in detail.

[0108] This embodiment can employ various methods for fuzz detection, and the following methods are provided, but are not limited to:

[0109] Method 1: Collect a high-resolution image with the same or similar lip shape as the digital human image as a reference. Use one or more methods, such as mean square error, peak signal-to-noise ratio, structural similarity, and feature similarity, to calculate the distortion of the lip region of the digital human image and the lip region of the high-resolution image in the pixel domain or transform domain, and use it as the predicted value of lip blur.

[0110] Method 2 also requires the aforementioned high-resolution image as a reference. A portion of the features is extracted from the lip region of the high-resolution image and compared with the features of the lip region of the digital human image to obtain a predicted value of the lip blur level.

[0111] However, high-resolution images may be difficult to acquire, limiting the application scenarios of Method 1 and Method 2. To achieve blur detection of the lip region in a digital human image without a reference, this application also provides the following method.

[0112] Method 3: Pre-train a lip blur prediction model to achieve blur detection. For ease of introduction below, the trained lip blur prediction model is defined as the target lip blur prediction model.

[0113] Considering that digital human images contain regions other than the lips, the data in these other regions may affect the blur detection results of the target lip blur prediction model for the lip region of the digital human image. Therefore, in a preferred implementation, the lip region can be first identified and cropped from the digital human image to obtain a target lip image, and then the target lip image can be input into the target lip blur prediction model to obtain a predicted value for the degree of lip blur.

[0114] In order to identify the lip region from a digital human image, this embodiment can optionally use a pre-trained facial landmark detection model to perform facial landmark detection on the digital human image and obtain the facial landmark detection results.

[0115] Here, the facial landmark detection model is trained using training images labeled with facial landmarks as training data.

[0116] For example, the facial landmark detection model can be an MTCNN (Multi-Task Cascaded Convolutional Networks) model or a MediaPipe Face Mesh model.

[0117] Furthermore, the lip region can be identified and cropped from the digital human image based on the facial key point detection results to obtain the target lip image.

[0118] Optionally, the range of the lip region can be: from the base of the nose to the mentolabial sulcus, and from the corners of the mouth to the sides. Then, based on the various facial key points detected by the model, the lip region can be located, and the target lip image can be cropped.

[0119] Optionally, the target lip image can be a lip ROI (Region of Interest) image with a resolution of 256×256, which can ensure that the ROI only includes the lips and a small amount of skin within an acceptable range around them, avoiding interference from other facial features.

[0120] The training process of the target lip blur prediction model is described below.

[0121] First, a high-resolution second image sample is obtained. Optionally, the second image sample is a real face image or a digital human image that meets the above-mentioned detail clarity requirements for the lip area.

[0122] For example, real face images can be collected from CelebA-HQ (Celebrities Attributes High-Quality) and FFHQ (Flickr-Faces-HQ) datasets as second image samples; and / or, existing digital human images can be collected as second image samples.

[0123] Preferably, the real face image can include the lip region of different expressions (such as smiling, speaking, pursing lips, etc.) to train the initial digital human generation model to learn the continuous rules of lip deformation, and decouple identity from expression to improve the model's robustness to various expressions.

[0124] Preferably, the digital human images can cover digital humans of different styles (such as realistic, cartoon, 3D, etc.) to train the initial digital human generation model to capture semantic text rather than the stylistic representation of the digital human, which greatly expands the applicability of the model.

[0125] Next, the lip region in the second image sample can be identified and cropped to obtain the lip image sample. The process of identifying and cropping the lip region in the second image sample can be referred to the previous description and will not be repeated here.

[0126] In order to train the target lip blur prediction model to accurately predict the blur level of lip image samples, this embodiment can optionally use the configured blur parameters to perform different levels of blur processing on the lip image samples to obtain more than one blurred image sample corresponding to the lip image sample, and mark the blur level for each blurred image sample.

[0127] For example, in this embodiment, the configured Gaussian blur kernel and Gaussian standard deviation can be used to perform different levels of Gaussian blur processing on the lip image sample to obtain one or more blurred image samples corresponding to the lip image sample.

[0128] For example, taking Gaussian blur levels 1-10 as an example, the correspondence between blur level, blur parameters, and blur effect can be as follows:

[0129] Level 1 blur: Gaussian blur kernel size is 3×3, Gaussian standard deviation (σ) is 0.5, and the blur effect is "slight blur, with clear edges of lips and teeth".

[0130] Level 2 blur: Gaussian blur kernel size is 3×3, Gaussian standard deviation (σ) is 1.0, and the blur effect is "slight blur, lip lines begin to fade";

[0131] Level 3 blur: Gaussian blur kernel size is 3×3, Gaussian standard deviation (σ) is 1.5, and the blur effect is "moderate blur, slight fusion of tooth edges";

[0132] Level 4 blur: Gaussian blur kernel size is 3×3, Gaussian standard deviation (σ) is 2.0, and the blur effect is "moderate blur, lip lines basically disappear";

[0133] Level 5 blur: Gaussian blur kernel size is 5×5, Gaussian standard deviation (σ) is 2.5, and the blur effect is "medium to high blur, with blurred boundaries between teeth and lips";

[0134] Level 6 blur: Gaussian blur kernel size is 5×5, Gaussian standard deviation (σ) is 3.0, and the blur effect is "high blur, tooth morphology is difficult to distinguish";

[0135] Level 7 blur: Gaussian blur kernel size is 7×7, Gaussian standard deviation (σ) is 3.5, and the blur effect is "high blur, the entire lip area is blurred";

[0136] Level 8 blur: Gaussian blur kernel size is 7×7, Gaussian standard deviation (σ) is 4.0, and the blur effect is "extremely high blur, with the lips and skin boundary blended together";

[0137] Level 9 blur: Gaussian blur kernel size is 7×7, Gaussian standard deviation (σ) is 4.5, and the blur effect is "extremely high blur, with complete loss of lip details";

[0138] Level 10 blur: Gaussian blur kernel size is 9×9, Gaussian standard deviation (σ) is 5.0, and the blur effect is "extreme blur, the general outline of the lips can only be distinguished".

[0139] It should be noted that the above correspondence is merely an example and is not intended to limit this application.

[0140] For example, if Gaussian blurring of lip image sample 1 is performed at level 1, resulting in 10 blurred image samples, then the blurred image sample obtained by processing with level 1 blur parameters can be labeled "1", the blurred image sample obtained by processing with level 2 blur parameters can be labeled "2", and so on.

[0141] Therefore, the initial lip blur prediction model can be trained using one or more blurred image samples and their corresponding blur levels to obtain the trained lip blur prediction model, which can then be used as the target lip blur prediction model.

[0142] Specifically, one or more blurred image samples are used as training samples, and the corresponding blur level is used as the sample label to train the initial lip blur prediction model, thereby obtaining the target lip blur prediction model.

[0143] Optionally, one or more blurred image samples and their corresponding blur levels can be divided into training, validation and test sets according to a preset ratio (e.g., 7:2:1) for subsequent training and evaluation of the initial lip blur prediction model.

[0144] To address the issue of blurred lip details in existing digital humans, this application constructs targeted training data for the lip region of digital humans. The initial lip blur prediction model is trained to learn the mapping relationship between lip image samples and blur levels. This ensures that the initial digital human generation model, after training, guarantees the generation of lip details (such as tooth texture and lip lines) from the generation stage, solving the problem of "creating something out of nothing" to supplement details, avoiding post-processing artifacts, and fundamentally optimizing the lip clarity of digital human images.

[0145] In one possible implementation, the target lip blur prediction model can be a lightweight convolutional neural network (CNN) model. Optionally, the target lip blur prediction model can consist of a sequentially cascaded input layer, one or more feature extraction layers, one or more fully connected layers, and an output layer, wherein the feature extraction layer consists of convolutional blocks and pooling layers.

[0146] For example, a target lip blur prediction model can consist of an input layer, a first convolutional block, a first pooling layer, a second convolutional block, a second pooling layer, a third convolutional block, a third pooling layer, a first fully connected layer, a second fully connected layer, and an output layer, all cascaded in sequence.

[0147] Optionally, each convolutional block consists of a 3×3 convolutional layer (with ReLU activation function) and a batch normalization (BatchNorm) layer.

[0148] Optionally, the first convolutional block has 64 output channels, the second convolutional block has 128 output channels, and the third convolutional block has 256 output channels.

[0149] Optionally, each pooling layer can use 2×2 max pooling with a stride of 2 to reduce the resolution of the input feature map while retaining key features.

[0150] Taking a blurred image sample with dimensions of 256×256×3 as an example, the input dimension of the first fully connected layer can be 256×32×32 (the size of the feature map after pooling), and the output dimension is 1024; the input dimension of the second fully connected layer is 1024, and the output dimension can be determined according to the specific task type.

[0151] If the target lip fuzz prediction model is a regression task, the output dimension of the second fully connected layer (i.e., the last fully connected layer) is 1 (i.e., the continuous value used by the regression task to predict the fuzz level); if the target lip fuzz prediction model is a classification task, the output dimension of the second fully connected layer (i.e., the probability distribution used by the classification task to predict the fuzz level) is multi-dimensional.

[0152] Optionally, when the last fully connected layer outputs 1D data (regression task), the loss function used for training the target lip blur prediction model is the mean squared error loss function, and the evaluation metric on the validation set is the mean absolute error; when the last fully connected layer outputs multi-dimensional data (classification task), the loss function used for training the target lip blur prediction model is the cross-entropy loss function, and the evaluation metric on the validation set is the fuzziness level accuracy, with the total dimension of the multi-dimensional data being the same as the number of fuzziness levels (such as 10 mentioned above).

[0153] Specifically, if the task type of the target lip blur prediction model is a regression task, then the mean squared error loss function is used to calculate the squared difference between the predicted value of the lip blur degree of the model for the blurred image sample and the true label. That is, the difference between the predicted value of the lip blur degree and the true label is calculated and then the difference is squared to obtain the mean squared error loss value. The target lip blur prediction model is trained based on the mean squared error loss value.

[0154] If the target lip blur prediction model is a classification task, then the cross-entropy loss function is used to calculate the cross-entropy between the model's predicted lip blur degree of the blurred image sample and the one-hot encoding result of the real label, and the cross-entropy loss value is obtained. The target lip blur prediction model is then trained based on the cross-entropy loss value.

[0155] After several epochs of model training, the model's performance can be validated using evaluation metrics on the validation set until the evaluation metrics on the validation set converge (generally requiring 30-50 epochs of training).

[0156] If the target lip blur prediction model is a regression task, the evaluation metric is the mean absolute error (MAE). The mean absolute error is used to measure the average deviation between the model's predicted value of the lip blur degree of the blurred image sample and the true label. Optionally, 0.3 can be used as the threshold, that is, if MAE≤0.3, the verification is passed.

[0157] If the target lip blur prediction model is a classification task, the evaluation metric is the blur level accuracy. The blur level accuracy measures the proportion of samples in which the model's predicted lip blur degree differs from the true label by less than a preset value (e.g., 1). Optionally, 95% can be used as the threshold, that is, if the blur level accuracy is ≥95%, the validation is passed.

[0158] Optionally, the target lip model prediction model training process can use the Adam optimizer with an initial learning rate of 1e-4, which decays to 0.5 of the original rate every 5 epochs.

[0159] In this embodiment, the target lip blur prediction model can be designed to adapt to different task types according to specific application scenarios, thus expanding the application scenarios of the target lip blur prediction model.

[0160] In one possible implementation, the process of determining the lip blur loss value can be implemented in different ways for different task types.

[0161] For example, in the case where the last fully connected layer outputs 1D data, i.e., for a regression task, the process of determining the lip blurring loss value may include: calculating the distance between the predicted lip blurring degree value in step S103 and the target value, and taking the absolute value of this distance as the lip blurring loss value. For example, taking a target value of 1 as an example... ,in, The target value represents the lip blurring loss value, and P represents the predicted lip blurring degree value in step S103. Of course, the target value of 1 is only an example. In addition, the target value can be other values, such as 0. This application does not impose specific limitations, and the same applies below.

[0162] When the last fully connected layer outputs multidimensional data, i.e., for a classification task, the process of determining the lip blurring loss value can include: calculating the cross-entropy between the predicted lip blurring degree and the target value, as the lip blurring loss value. For example, taking a target value of 1 as an example... ,in, This indicates the calculation of cross-entropy.

[0163] In this embodiment, different methods for determining the lip blur loss value are provided for different task types to penalize digital human images with excessively blurred lips. By setting the lip pattern loss value, a lip region-oriented supervision method is provided for the initial digital human generation model, enabling the model to learn lip detail features more accurately and optimizing the lip clarity of the generated digital human images.

[0164] This application also provides a method for generating digital humans, such as Figure 3 The diagram shown is a flowchart illustrating the digital human generation method provided in this application.

[0165] See Figure 3 The digital human generation method may include:

[0166] Step S301: Obtain the reference image and the target lip-shaped driven audio.

[0167] Step S302: Mask the lip region of the reference image to obtain a masked image that covers the lip region.

[0168] Specifically, the lip region in the reference image can be identified first, and then the identified lip region can be masked to obtain the mask image corresponding to the reference image.

[0169] The process of identifying the lip region in the reference image can be referred to the previous description and will not be repeated here.

[0170] Step S303: Input the mask image and the target lip-driven audio into the pre-trained target digital human generation model to obtain the digital human image corresponding to the target lip-driven audio.

[0171] Here, the target digital human generation model is obtained by training the initial digital human generation model using the digital human generation model training method described above.

[0172] This application uses supervised training with lip blur loss values ​​to enable the target digital human generation model to stably output digital human images with low blur in the lip region (e.g., blur level within the range of 1-5). Experimental verification shows that the teeth of the digital human images generated by this application are more distinct and the lip gloss is more natural. Validated by objective metrics PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index), the subjective realism is improved by more than 30%.

[0173] The above describes a digital human generation model training method and a digital human generation method provided by embodiments of this application. The apparatus for performing the above method will be described below.

[0174] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a digital human generation model training device provided in an embodiment of this application. Figure 4 As shown, the digital human generation model training device may include:

[0175] The first data acquisition unit 401 is used to acquire a first image sample and a lip-driven audio sample. The first image sample is obtained by masking the lip region of the original image.

[0176] Data processing unit 402 is used to input the first image sample and the lip-driven audio sample into the initial digital human generation model to obtain the digital human image corresponding to the lip-driven audio sample;

[0177] The data detection unit 403 is used to perform blur detection on the lip region of the digital human image and obtain a predicted value of the degree of lip blur.

[0178] The model training unit 404 is used to train the initial digital human generation model based on at least the lip blur loss value. The lip blur loss value is used to measure the difference between the predicted value of the lip blur degree and the target value. The target value is the minimum blur degree threshold that makes the lip region meet the preset detail clarity requirements.

[0179] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a digital human generation device provided in an embodiment of this application. Figure 5 As shown, the digital human generation device may include:

[0180] The second data acquisition unit 501 is used to acquire reference images and target lip-shaped driven audio.

[0181] Image processing unit 502 is used to mask the lip region of the reference image to obtain a mask image that covers the lip region;

[0182] The model uses unit 503 to input the mask image and the target lip-driven audio into the pre-trained target digital human generation model to obtain the digital human image corresponding to the target lip-driven audio. The target digital human generation model is obtained by training the initial digital human generation model using the digital human generation model training method described above.

[0183] The aforementioned digital human generation model training device and various modules within the digital human generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0184] This application also provides an electronic device, which may include at least one processor and a memory connected to the processor, wherein:

[0185] Memory is used to store computer programs;

[0186] The processor is used to execute computer programs to enable electronic devices to implement any of the digital human generation model training device methods or any of the digital human generation methods provided in the embodiments of this application.

[0187] refer to Figure 6 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0188] like Figure 6As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0189] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0190] This application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device causes the electronic device to implement any of the digital human generation model training device methods or any of the digital human generation methods provided in this application.

[0191] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device is able to implement any of the digital human generation model training device methods or any of the digital human generation methods provided in this application.

[0192] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0193] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0194] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0195] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for training a digital human generation model, characterized in that, include: A first image sample and a lip-driven audio sample are obtained, wherein the first image sample is obtained by masking the lip region of the original image; The first image sample and the lip-driven audio sample are input into the initial digital human generation model to obtain the digital human image corresponding to the lip-driven audio sample. Blur detection is performed on the lip region of the digital human image to obtain a predicted value of the degree of lip blur. The initial digital human generation model is trained based on at least a lip blur loss value, wherein the lip blur loss value is used to measure the difference between the predicted lip blur degree and the target value, wherein the target value is the minimum blur degree threshold that makes the lip region meet the preset detail clarity requirements.

2. The digital human generation model training method according to claim 1, characterized in that, The step of performing blur detection on the lip region of the digital human image to obtain a predicted value for the degree of lip blur includes: The digital human image is subjected to facial key point detection using a pre-trained facial key point detection model to obtain facial key point detection results. The facial key point detection model is trained using training images labeled with facial key point tags as training data. Based on the facial key point detection results, the lip region is identified and cropped from the digital human image to obtain the target lip image; The target lip image is input into a pre-trained target lip blur prediction model to obtain the predicted value of the lip blur degree.

3. The digital human generation model training method according to claim 2, characterized in that, The training process of the target lip blur prediction model includes: A second image sample is obtained, and the lip region in the second image sample is identified and cropped to obtain a lip image sample. The second image sample is a real face image or a digital human image whose lip region meets the required level of detail. The lip image sample is blurred at different levels using pre-configured blur parameters to obtain one or more blurred image samples corresponding to the lip image sample, and the blur level is marked for each blurred image sample. Using one or more blurred image samples and their corresponding blur levels, the initial lip blur prediction model is trained to obtain the trained lip blur prediction model, which is then used as the target lip blur prediction model.

4. The digital human generation model training method according to claim 2 or 3, characterized in that, The target lip blur prediction model consists of a sequentially cascaded input layer, one or more feature extraction layers, one or more fully connected layers, and an output layer. The feature extraction layer consists of convolutional blocks and pooling layers. When the last fully connected layer outputs 1D data, the loss function used for training the target lip blur prediction model is the mean squared error loss function, and the evaluation metric on the validation set is the mean absolute error. When the last fully connected layer outputs multidimensional data, the loss function used for training the target lip fuzzy prediction model is the cross-entropy loss function, and the evaluation metric on the validation set is the fuzziness level accuracy. The total dimension of the multidimensional data is the same as the number of fuzziness levels.

5. The digital human generation model training method according to claim 4, characterized in that, The process of determining the lip blur loss value includes: When the last fully connected layer outputs 1D data, the distance between the predicted lip blur degree and the target value is calculated, and the absolute value of the distance is used as the lip blur loss value. When the last fully connected layer outputs multidimensional data, the cross-entropy between the predicted lip blur degree and the target value is calculated as the lip blur loss value.

6. The digital human generation model training method according to claim 1, characterized in that, The initial digital human generation model is a digital human generator; The step of training the initial digital human generation model based on at least the lip blur loss value includes: The discriminator corresponding to the digital human generator is used to perform discrimination processing on the digital human image to obtain first discrimination information, and the discriminator is used to perform discrimination processing on the original image corresponding to the digital human image to obtain second discrimination information; An adversarial loss value is generated based on the first discrimination information and the second discrimination information; The digital human generator and the discriminator are jointly trained based on the lip blurring loss value and the adversarial loss value.

7. The digital human generation model training method according to claim 6, characterized in that, The step of jointly training the digital human generator and the discriminator based on the lip blurring loss value and the adversarial loss value includes: The features of the digital human image are extracted using a pre-trained Visual Geometric Group Network (VGG) model to obtain the first image features, and the features of the original image corresponding to the digital human image are extracted using the VGG model to obtain the second image features. A perceptual loss value is generated based on the first image features and the second image features; The digital human generator and the discriminator are jointly trained based on the adversarial loss value, the lip blurring loss value, and the perceptual loss value.

8. The digital human generation model training method according to claim 1, characterized in that, The training termination condition of the initial digital human generation model consists of one or more of the following conditions: the predicted lip blur level on the validation set stabilizes within a preset blur level range, the total loss value converges, the maximum number of training rounds is reached, and a training stop signal is received.

9. A method for generating a digital human, characterized in that, include: Acquire reference image and target lip-driven audio; The lip region of the reference image is masked to obtain a masked image that covers the lip region; The mask image and the target lip-driven audio are input into a pre-trained target digital human generation model to obtain a digital human image corresponding to the target lip-driven audio. The target digital human generation model is obtained by training an initial digital human generation model using the digital human generation model training method according to any one of claims 1-8.

10. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the digital human generation model training method as described in any one of claims 1 to 8 or the digital human generation method as described in claim 9.

11. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the digital human generation model training method as described in any one of claims 1 to 8 or the digital human generation method as described in claim 9.

12. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the digital human generation model training method as described in any one of claims 1 to 8 or the digital human generation method as described in claim 9.

Images