Generative model training method of digital human, generation method and related device

Abstract

The invention provides a generative model training method of a digital human, a generation method and a related device, and the method comprises the steps: removing the identity information of a first object from a first image, so as to obtain a posture feature; the first image is any frame in the first video segment; extracting audio features from the first audio data; extracting a first identity feature of the first object from the second image; the second image is a frame corresponding to the first image in the second video segment; according to the method, the posture feature, the audio feature and the first identity feature are fused to obtain the first fusion feature, the digital human generation model is trained based on the first fusion feature, the digital human generation model with the controllable face posture can be obtained, and a more vivid and more real digital human video can be generated by using the digital human generation model.

Description

Technical Field

[0001] The present application relates to the technical field of digital human generation in artificial intelligence, and in particular to a digital human generation model training method, generation method and related devices. Background Art

[0002] The two-dimensional (2D) digital human generation technology refers to collecting a video or a photo, quickly identifying and binding facial expressions and body movements through machine learning, and then using voice drive to generate an anthropomorphic video. The existing 2D digital human generation ideas include encoding-decoding ideas based on Generative Adversarial Network (GAN), audio to lip synchronization (wav2lip) and deformable-iterative network (DINet) are its representative models. Due to the limitations of wav2lip and DINet's training data, the posture of the image they drive is uncontrollable, that is, some existing pre-trained models are difficult to achieve posture-controllable 2D digital human generation. Summary of the invention

[0003] In response to the above problems, the present application provides a digital human generation model training method, a generation method and related devices, which can generate digital human videos with controllable postures.

[0004] In a first aspect, an embodiment of the present application provides a method for training a digital human generation model, the method comprising:

[0005] Acquire a first video segment, a second video segment, a third video segment and first audio data of a first object; the first video segment has the same number of frames as the second video segment and the third video segment;

[0006] For any first image frame in the first video segment, remove the identity information of the first object from the first image to obtain a posture feature of the first object;

[0007] extracting an audio feature of a first object from the first audio data;

[0008] For a second image in the second video segment corresponding to the first image frame sequence, extracting a first identity feature of the first object from the second image;

[0009] Fusing the posture feature, the audio feature, and the first identity feature to obtain a first fused feature;

[0010] For a third image in the third video segment corresponding to the second image frame sequence, generating a simulated digital human image corresponding to the third image based on the first fusion feature;

[0011] The simulated digital human image corresponding to each frame of the third image in the third video segment is used to form a first simulated digital human video;

[0012] The neural network is iteratively trained based on the first simulated digital human video to obtain a trained digital human generation model.

[0013] In a second aspect, an embodiment of the present application provides a method for generating a digital human, the method comprising:

[0014] Acquire the second audio data, the sixth video segment of the second object, the seventh video segment and the eighth video segment of the third object; the sixth video segment has the same number of frames as the seventh video segment and the eighth video segment;

[0015] inputting the second audio data, the sixth video segment, the seventh video segment, and the eighth video segment into the digital human generation model to generate a third simulated digital human video;

[0016] The digital human generation model is obtained by training based on the training method of any one of the embodiments of the first aspect described above.

[0017] In a third aspect, an embodiment of the present application provides a digital human generation model training device, the device comprising a first acquisition unit and a first processing unit, wherein:

[0018] A first acquisition unit is used to acquire a first video segment, a second video segment, a third video segment and first audio data of a first object; the first video segment has the same number of frames as the second video segment and the third video segment;

[0019] The first processing unit is used to remove the identity information of the first object from any frame of the first image in the first video segment to obtain the posture feature of the first object; extract the audio feature of the first object from the first audio data; for the second image corresponding to the first image frame sequence in the second video segment, extract the first identity feature of the first object from the second image; fuse the posture feature, the audio feature and the first identity feature to obtain the first fusion feature; for the third image corresponding to the second image frame sequence in the third video segment, generate a simulated digital human image corresponding to the third image based on the first fusion feature; form a first simulated digital human video with the simulated digital human images corresponding to each frame of the third image in the third video segment; iteratively train the neural network based on the first simulated digital human video to obtain a trained digital human generation model.

[0020] It should be understood that since the method embodiment and the device embodiment are different presentation forms of the same technical concept, the content of the first aspect of the embodiment of the present application should be synchronously adapted to the third aspect of the embodiment of the present application, and can achieve the same or similar beneficial effects, which will not be repeated here.

[0021] In a fourth aspect, an embodiment of the present application provides a device for generating a digital human, the device comprising a second acquisition unit and a second processing unit, wherein:

[0022] A second acquisition unit is used to acquire second audio data, a sixth video segment of the second object, and a seventh and eighth video segments of the third object; the sixth video segment has the same number of frames as the seventh and eighth video segments;

[0023] a second processing unit, configured to input the second audio data, the sixth video segment, the seventh video segment, and the eighth video segment into the digital human generation model to generate a third simulated digital human video;

[0024] The digital human generation model is obtained by training based on the training method of any one of the embodiments of the first aspect described above.

[0025] It should be understood that since the method embodiment and the device embodiment are different presentation forms of the same technical concept, the content of the second aspect of the embodiment of the present application should be synchronously adapted to the fourth aspect of the embodiment of the present application, and can achieve the same or similar beneficial effects, and will not be repeated here.

[0026] In a fifth aspect, an embodiment of the present application provides a computer device, the computer device comprising:

[0027] Processor; and

[0028] A memory storing computer executable instructions, the executable instructions are configured to be executed by a processor, and the executable instructions include instructions for executing the steps in the method described in any embodiment of the first aspect or the second aspect above.

[0029] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program for execution by a device. When the computer program is executed, the steps in the method described in any embodiment of the first aspect or the second aspect above are implemented.

[0030] In the seventh aspect, an embodiment of the present application provides a computer program product. When the computer program product is run by a device, the device executes the steps of the method described in any embodiment of the first aspect or the second aspect above.

[0031] The above solution of the present application includes at least the following beneficial effects:

[0032] In the embodiment of the present application, the image frame in the first video segment is the posture frame of the first object in the sample data, and the image frame in the second video segment is the reference frame corresponding to the posture frame. After obtaining the first video segment, the second video segment, the third video segment and the first audio data of the first object, the computer device removes the identity information of the first object from the posture frame (such as the first image) to obtain the posture feature, which is equivalent to enhancing the posture feature, that is, the posture feature can explicitly characterize the posture change between each posture frame; the computer device fuses the enhanced posture feature, the audio feature extracted from the first audio data and the identity feature (i.e., the first identity feature) extracted from the corresponding reference frame (such as the second image), which is conducive to the neural network to perceive the subtle changes in the posture feature in the fused feature (i.e., the first fused feature), so that the image (such as the simulated digital human image) corresponding to the input image (such as the third image) generated based on the fused feature can be more distinguished in terms of facial posture. In this way, supervising the neural network with additional enhanced posture features is conducive to reducing the influence of the facial posture in the input image on the neural network, thereby achieving posture controllability. By iteratively training the neural network based on the generated video (such as the first simulated digital human video), a digital human generation model with controllable facial posture can be trained. The digital human generation model can be used to generate more vivid and realistic digital human videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A schematic diagram of a scenario architecture provided for an embodiment of the present application;

[0035] Figure 2 A schematic diagram of a flow chart of a method for training a digital human generation model provided in an embodiment of the present application;

[0036] Figure 3 A schematic diagram of the structure of a neural network provided in an embodiment of the present application;

[0037] Figure 4 A schematic diagram of generating a digital human video using a neural network provided in an embodiment of the present application;

[0038] Figure 5 A schematic diagram of a lightweight U-shaped network in a gesture encoder provided in an embodiment of the present application;

[0039] Figure 6A schematic diagram of calculating the L1 loss of a style transfer matrix provided in an embodiment of the present application;

[0040] Figure 7 A schematic diagram of obtaining a mouth area image provided in an embodiment of the present application;

[0041] Figure 8 A schematic diagram of image semantic segmentation provided in an embodiment of the present application;

[0042] Fig. 9 A schematic diagram of a flow chart of another method for training a digital human generation model provided in an embodiment of the present application;

[0043] Fig.10 A flowchart of a method for generating a digital human provided in an embodiment of the present application.

[0044] Fig.11 A schematic diagram of the structure of a digital human generation model training device provided in an embodiment of the present application;

[0045] Fig.12 A schematic diagram of the structure of a digital human generation device provided in an embodiment of the present application;

[0046] Fig.13 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0048] The terms "including" and "having" and any variations thereof that appear in the specification, claims and drawings of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally further include steps or units that are not listed, or may optionally further include other steps or units that are inherent to these processes, methods, products or devices. In addition, the terms "first", "second" and "third" are used to distinguish different objects, rather than to describe a specific order.

[0049] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0050] The following is a brief description of the relevant technologies involved in this application.

[0051] One of the related technologies is to generate 2D face images based on the idea of ​​GAN. Its representative models include wav2lip and DINet, as well as the Progressive Disentangled Fine-Grained Controllable Talking Head (PD-FGC) model. If wav2lip is trained on its original training data set, the size of the face area it generates is limited. If you want to increase the size of the face area it generates, the amount of data it relies on will increase exponentially, which will bring stability problems to the training. When focusing more on generating static face images, the posture of the image driven by wav2lip is uncontrollable. DINet aims to decouple various attributes in the generated image, but ignores the possible association between posture and other attributes (such as expression, lighting, etc.), which will also make the posture of the driven image uncontrollable. PD-FGC decouples head posture, eye movement, and lip movement in a fine-grained manner, extracts the feature vectors of the three, and generates a 2D image based on them. However, its accuracy for the detailed textures of the synthesized images (beard, facial texture) is not high, and it also has difficulty maintaining identity. In the generated speaking video, the identity information of the person will change, and the consistency of the identity information cannot be maintained.

[0052] The second related technology uses the 3D Morphable Face Model (3DMM), which usually uses a set of parameters to represent the shape and texture of the face. The shape parameters control the overall geometry of the face, such as the position and size of the eyes, nose, mouth, etc. The texture parameters contain information such as skin color and skin texture, which are used to describe the appearance of the face. The related technology of 3DMM requires a large amount of training data with different identity information to achieve good generalization; during the training process, 3D intermediate states are needed to supervise the loss. The 3D intermediate states are generated by the pre-training model and are prone to errors; the driven head posture is also derived from the inference frame, and it is also difficult to achieve controllable posture.

[0053] In order to overcome the defects or shortcomings in the related art, the present application embodiment provides a method for training a digital human generation model, which can be applied to Figure 1 The scene architecture shown in Figure 1 is as follows. Figure 1 As shown, the architecture includes at least a terminal device and a computer device, wherein the terminal device is used to obtain training sample data, wherein the sample data in the training sample data is embodied in the form of a sample pair, wherein a sample pair includes two video segments of the same identity information (Identity, ID), wherein one video segment is a posture video segment corresponding to the audio data, and the other video segment is a reference video segment of the posture video segment, the image frame in the posture video segment can be called a posture frame, the image frame in the reference video segment is called a reference frame, the posture video segment has the same number of image frames as the reference video segment, that is, each posture frame has a corresponding reference frame in the reference video segment. The video segments in the sample data can be obtained by dividing the video downloaded from a third-party platform, or can be obtained by recruiting professional broadcasters and hosts to record the text and dividing the recorded video. The entire sample data set includes videos of multiple identity IDs, and the head of the object in each video has posture changes. A neural network is deployed in the computer device, and the terminal device can send the collected sample data and reasoning videos to the computer device in batches. The computer device executes the preset algorithm, calls the various parts of the neural network for forward reasoning, and performs back propagation based on at least the prediction loss to update the parameters of the neural network, and obtains a trained digital human generation model after multiple iterations of training. Exemplarily, the terminal device can also send the collected original data set to the computer device, and the computer device constructs the sample pair. Exemplarily, the terminal device can be a laptop, a desktop computer, a vehicle-mounted device, a camera device, a smart phone, etc., and the computer device can be a server, a cloud device, a supercomputer, a computer host, etc. The terminal device and the computer device can communicate through a wired or wireless network.

[0054] The present application is described in detail below with reference to specific embodiments.

[0055] See also Figure 2 , Figure 2 The flowchart of a method for training a digital human generation model provided in an embodiment of the present application is shown in FIG. The method can be executed by a computer device or a chip in the computer device, such as Figure 2 As shown, at least the following steps are included:

[0056] 201: Acquire a first video segment, a second video segment, a third video segment, and first audio data of a first object;

[0057] Among them, the first audio data corresponds to the first video segment; the first video segment, the second video segment and the third video segment have the same number of frames, that is, each frame image in the first video segment has a frame corresponding to the frame sequence in the second video segment and the third video segment, for example: the first frame in the first video segment corresponds to the first frame in the second video segment and the third video segment, the second frame in the first video segment corresponds to the second frame in the second video segment and the third video segment, and so on. In addition, each frame in the first video segment contains a complete face of the first object, and each frame in the second video segment also contains a complete face of the first object. The facial posture of the first object in any frame of the first video segment is different from the facial posture of the first object in a frame corresponding to the arbitrary frame in the second video segment; each frame in the third video segment contains a partial face of the first object.

[0058] The first video segment and the second video segment are from the same sample pair, the first video segment is a posture video segment, and the second video segment is a reference video segment. The first video segment and the second video segment can be video segments of different times obtained by segmenting a video in which each frame contains the face of the first object. The first audio data can be extracted from the first video segment, or it can be audio recording data corresponding to the first video segment. The third video segment is an inference video segment (that is, each frame in the third video segment is a frame to be predicted). In the training stage, the inference video segment, the posture video segment, and the reference video segment are all from the same object. Training with the posture video segment of the same object is conducive to reducing the amount of training data. Without considering the amount of training data, the posture video segment of another object can be used for training. In the application stage, it is only necessary that the inference video segment and the reference video segment come from the same object, and the posture video segment can be unrestricted, that is, it is only necessary that the identity ID of the inference frame is consistent with that of the reference frame.

[0059] Among them, each frame of the third video segment only includes the upper half of the face of the first object, aiming to allow the model to predict the lower half of its face to generate a predicted image containing the entire face of the first object, and supervise the posture and texture details of the predicted image through the true value image corresponding to each frame of the third video segment.

[0060] 202: For any first image frame in the first video segment, remove the identity information of the first object from the first image to obtain a posture feature of the first object.

[0061] The first image contains a complete face of the first object.

[0062] For example, see Figure 3 , Figure 3 A schematic diagram of the structure of a neural network provided in an embodiment of the present application. Figure 3As shown in the figure, the backbone network of the neural network is a U-type network, which also includes an audio encoder, a posture encoder and a style convolution (styledconv) module. The output of the last downsampling layer of the U-type network, the output of the audio encoder and the output of the posture encoder are used as the input of the styledconv module, and the output of the styledconv module is the encoding feature of the U-type network. Among them, the U-type is divided into two parts: downsampling and upsampling. The downsampling part is the encoder part of the U-type network, and the upsampling part is the decoder part of the U-type network. Each layer of the encoder is downsampled by 2 times, and each layer of the downsampling has an upsampling layer with a jump connection. The output of each downsampling layer is connected to the previous layer feature of the corresponding upsampling layer as the input of the corresponding upsampling layer.

[0063] In the present application embodiment, Figure 4 As shown, the neural network has four inputs, namely the first video, the second video, the third video and the first audio data in a batch. For any gesture frame in the first video segment, that is, the first image, it is input into the gesture encoder for feature extraction to obtain the corresponding gesture feature.

[0064] Specifically, if Figure 5 As shown, the posture encoder includes a lightweight U-type network. For the input first image, the posture encoder performs an affine transformation on it through a randomly generated affine transformation matrix, and then performs nonlinear modification of the image color gamut space on the first image after the affine transformation to remove the identity information of the first object; for the image with the identity information removed, it is downsampled through a lightweight U-type network to obtain a 486*32 posture feature. Wherein, when constructing the affine transformation matrix, 3 pixel points are selected in the first image, and for these three pixel points, their positions are randomly offset (for example: positive and negative offsets of 10 pixels) to obtain 3 target pixel points, and the affine transformation matrix is ​​constructed with these 3 target pixel points and the 3 pixel points selected in the first image. The nonlinear modification of the image color gamut space can be achieved through a Bezier curve, such as including the following steps:

[0065] (1) determining a Bezier curve for modifying the color gamut;

[0066] (2) Calculate the color gamut mapping and obtain a mapping function;

[0067] (3) Each pixel value in the first image after affine transformation is converted into a corresponding output pixel value through a mapping function.

[0068] In this implementation, since the pose frame and the reference frame are both from a video of the same object (the video before division), the pose frame is subjected to affine transformation and nonlinear modification of the color gamut space, so that the color distribution of the pose frame and the reference frame is different, thereby removing the model's ability to learn image color distribution from the pose frame, thereby constraining the model to only learn pose information from the pose frame, which is equivalent to enhancing the pose features.

[0069] 203: Extracting audio features of the first object from the first audio data.

[0070] In the embodiment of the present application, the waveform sound file (ie, the first audio data) is input into an audio encoder for feature extraction to obtain corresponding audio features.

[0071] Specifically, the audio encoder also includes a lightweight U-type network. For the first input audio data, after converting it into a Mel spectrum, it is downsampled through the U-type network to obtain a 486*1 audio feature.

[0072] 204: For a second image in the second video segment corresponding to the first image frame sequence, extract a first identity feature of the first object from the second image.

[0073] The second image contains a complete face of the first object, and the facial posture is different from the facial posture of the first object in the first image.

[0074] In the embodiment of the present application, the reference frame corresponding to the first image (ie, the second image) is input into the backbone network for layer-by-layer downsampling to obtain a 510*6*6 identity information feature map (ie, the first identity feature).

[0075] 205: Fusing the posture feature, the audio feature, and the first identity feature to obtain a first fused feature.

[0076] In an embodiment of the present application, the 486*32 posture feature is converted into a 972*1 feature vector through one-dimensional convolution and adaptive pooling, the 486*1 audio feature is concatenated with the 972*1 feature vector to obtain a 1458*1 feature vector, and the feature vector is converted into a 510*1*1 feature map (i.e., the second fused feature) through a fully connected layer, and the second fused feature is fused with the first identity feature to obtain the first fused feature.

[0077] In this implementation, the posture feature is converted into a feature vector and then fused with the audio feature to obtain a weight driven by audio and posture. The weight is applied to the first identity feature, which enables the audio feature and posture feature to control the reference frame with identity information and texture information.

[0078] Exemplarily, in fusing the second fused feature with the first identity feature to obtain the first fused feature, the second fused feature is first multiplied by the first identity feature to obtain a third fused feature, and then the elements in the third fused feature are normalized to obtain the first fused feature. The normalization method may be to obtain a normalization factor of the elements in the third fused feature, and multiply the normalization factor by the elements in the third fused feature. Of course, other normalization methods may also be used, which are not limited here.

[0079] In this implementation, normalizing the third fused feature can ensure that the norms of the first fused features of each channel of the output are closer, thereby ensuring that the features of each channel are represented and are not dominated by features with excessively large values. This helps the network learn richer and more diverse feature representations, avoids problems such as gradient explosion and gradient disappearance, and ensures the stability of training.

[0080] 206: For a third image in the third video segment corresponding to the second image frame sequence, generate a simulated digital human image corresponding to the third image based on the first fusion feature.

[0081] 207: The simulated digital human image corresponding to each frame of the third image in the third video segment is combined into a first simulated digital human video.

[0082] In the embodiment of the present application, the backbone network decodes the first fusion feature through upsampling of each layer, and predicts the lower half of the face of the third image at each upsampling layer based on the feature map obtained by decoding, and obtains the predicted and completed image output by each upsampling layer, and uses the simulated digital human image output by the last upsampling layer as the final predicted image, and the size of the simulated digital human image is the same as that of the second image and the third image. The first simulated digital human video, i.e., the generated video segment, is composed of the simulated digital human images corresponding to each frame in the third video segment.

[0083] 208: Iteratively train the neural network based on the first simulated digital human video to obtain a trained digital human generation model.

[0084] In the embodiment of the present application, the prediction loss (i.e., the first loss) can be determined based on the first simulated digital human video output from a single round of training, and the neural network is iteratively trained based on the first loss to obtain a trained digital human generation model. The process of determining the first loss includes:

[0085] (1) Using a pre-trained audio-visual synchronization detection model to perform audio-visual synchronization detection on the first simulated digital human video, obtaining an audio-visual synchronization detection result, and determining the synchronization loss based on the audio-visual synchronization detection result of the fourth video and a target synchronization label; the target synchronization label is a synchronization label between the first video segment and the first audio data, for example: synchronization is 1, and asynchrony is 0. The target synchronization label can be obtained based on the pre-trained audio-visual synchronization detection model.

[0086] (2) Determine the first L1 loss based on each frame of the simulated digital human image in the first simulated digital human video and the true value image corresponding to each frame of the simulated digital human image. Specifically, calculate the average absolute error between the RGB value of each frame of the simulated digital human image and the RGB value of the true value image corresponding to each frame of the simulated digital human image; determine the first L1 loss based on the average absolute error between the RGB value of each frame of the simulated digital human image and the RGB value of the true value image corresponding to each frame of the simulated digital human image. For example: the first video segment, the second video segment, and the third video segment each contain 5 frames of images, then calculate the sum or weighted sum of the 5 average absolute errors to obtain the first L1 loss. In this implementation, the model is supervised by the L1 loss between the predicted image and the true value image, which is conducive to making the color of the digital human image generated by the model closer to the true value image.

[0087] (3) Determine a first perceptual loss based on each frame of simulated digital human image and the true value image corresponding to each frame of simulated digital human image. Specifically, perform feature extraction on each frame of simulated digital human image through pre-trained VGG (Visual Geometry Group Network) 19 to obtain at least one layer of first features of each frame of simulated digital human image. Similarly, perform feature extraction on the true value image corresponding to each frame of simulated digital human image through VGG 19 to obtain at least one layer of second features of the true value image. The number of second features is the same as that of first features, that is, at least one layer of second features corresponds to at least one layer of first features, and then calculate the second L1 loss between each layer of first features and the corresponding second features. For example, Figure 6 As shown, for each layer of the first feature, the first style transfer matrix (Grammatrix) is obtained by convolution calculation, and for the corresponding second feature, the second style transfer matrix is ​​obtained by convolution calculation, and the third L1 loss between the first style transfer matrix and the second style transfer matrix is ​​calculated. Based on the second L1 loss and the third L1 loss, the perceptual loss corresponding to each frame of the simulated digital human image is determined, and the formula can be expressed as:

[0088]

[0089] Among them, Perception loss represents the perceptual loss corresponding to each frame of simulated digital human image, P layeriprerepresents the value of the first feature of the i-th layer, P layerigt Indicates the value of the second feature of the i-th layer, GM(layer i pre) represents the first style transfer matrix of the first feature of the i-th layer, GM (layer i gt) represents the second style transfer matrix of the second feature of the i-th layer.

[0090] The computer device determines the first perceptual loss based on the perceptual loss corresponding to each frame of the simulated digital human image, for example, by summing or weighted summing the perceptual losses corresponding to each frame of the simulated digital human image. In this implementation, by calculating the second L1 loss between the first feature of each layer of the simulated digital human image and the second feature of the corresponding true value image, the feature representation of the generated image can be supervised, which is conducive to improving the model's perception of audio features and posture features. By calculating the third L1 loss between the style transfer matrix of the first feature of each layer and the style transfer matrix of the corresponding second feature, the style of the generated image can be supervised, which is conducive to improving the model's perception of audio style and posture style.

[0091] (4) Based on each frame of the simulated digital human image and the true value image corresponding to each frame of the simulated digital human image, determine the first structural similarity loss. Specifically, perform mouth key point detection on each frame of the simulated digital human image and the true value image corresponding to each frame of the simulated digital human image; and determine the first mouth region image based on the mouth key points obtained in each frame of the simulated digital human image. Figure 7 As shown, the mouth key points include the pixel point with the largest upper lip height value, the pixel point with the smallest lower lip height value, and the left and right corners of the mouth. A rectangular area is determined based on these four points, and the area covered by the rectangular area is determined as the mouth area image. Similarly, based on the mouth key points obtained in the true value image corresponding to each frame of the simulated digital human image, the second mouth area image is determined. Since the mouth structure in the generated simulated digital human image is not necessarily consistent with the mouth structure of the true value image, this may cause the sizes of the first mouth area image and the second mouth area image to be inconsistent. Therefore, the first mouth area image and the second mouth area image need to be normalized to make the sizes of the two consistent. Then, the structural similarity (Structural Similarity, SSIM) between the normalized first mouth area image and the normalized second mouth area image is calculated, and the second structural similarity loss corresponding to each frame of the simulated digital human image is determined based on the structural similarity. The formula can be expressed as:

[0092] Mouse ssim loss=(1-ssim(pred,gt))*area weightmatrix

[0093] Among them, Mouse ssim loss represents the second structural similarity loss of a single frame simulated digital human image, ssim(pred,gt) represents the structural similarity between the normalized first mouth area image and the normalized second mouth area image, area weightmatrix It represents a weight matrix that is consistent with the normalized mouth region image size, in which the weight of the teeth region > the weight of the lips region > the weight of the background region.

[0094] Based on the second structural similarity loss corresponding to each frame of the simulated digital human image, the first structural similarity loss is determined. For example, the second structural similarity loss corresponding to each frame of the simulated digital human image is summed or weighted. In this implementation, by calculating the structural similarity of the mouth between the simulated digital human image and the true value image, and multiplying the mouth area weight matrix by the structural similarity, the mouth area can be more distinguished from the background area, so that the model pays more attention to the mouth structure to optimize the mouth structure of the generated image.

[0095] (5) Determine a first texture loss based on each frame of the simulated digital human image and the true value image corresponding to each frame of the simulated digital human image. Specifically, perform semantic segmentation on each frame of the simulated digital human image to obtain multiple first semantic segmentation regions, such as Figure 8 As shown, the multiple first semantic segmentation areas include skin, left eyebrow, right eyebrow, left eye, right eye, lips, teeth, etc. In order to supervise the detailed texture of the generated image, at least one first target area is determined from the multiple first semantic segmentation areas, such as skin, lips, and teeth; at least one second target area is determined from the multiple second semantic segmentation areas, and at least one second target area has the same category as at least one first target area, such as skin, lips, and teeth. At least one first target area is divided into a preset rectangle to obtain a plurality of first rectangular areas. Similarly, at least one second target area is divided into a preset rectangle to obtain a plurality of second rectangular areas, and the plurality of second rectangular areas correspond to the plurality of first rectangular areas one by one. For example, the preset rectangle can be a 16*16 rectangle, which is not limited here. The high-frequency information distribution of each first rectangular area is calculated, and the high-frequency information distribution of the second rectangular area corresponding to each first rectangular area is calculated. Based on the high-frequency information distribution of each first rectangular area and the high-frequency information distribution of the second rectangular area corresponding to each first rectangular area (specifically L1 loss), the second texture loss corresponding to each frame of the simulated digital human image is determined, and the formula can be expressed as:

[0096] Loss texture = ΣL1(Areapatch j gt, Areapatch j pred)

[0097] Among them, Loss texture represents the second texture loss corresponding to a single frame simulated digital human image, Areapatch j pred represents the high-frequency information distribution of the jth first rectangular area, Areapatch j gt represents the high-frequency information distribution of the j-th second rectangular area.

[0098] Exemplarily, the high-frequency information distribution of each rectangular area can be calculated by the following formula:

[0099] Areapatch=(patch-patchmean) / patchvar

[0100] Among them, Areapatch represents the high-frequency information distribution of the rectangular area, patch represents the RGB value of the rectangular area, patchmean represents the average value of the RGB value of the rectangular area, and patchvar represents the variance of the RGB value of the rectangular area.

[0101] Based on the second texture loss corresponding to each frame of the simulated digital human image, the first texture loss is determined. For example, the second texture loss corresponding to each frame of the simulated digital human image is summed or weighted. In this implementation, by calculating the L1 loss of the high-frequency information distribution of the semantic segmentation area of ​​each frame of the simulated digital human image and the high-frequency information distribution of the semantic segmentation area of ​​the corresponding true value image, the texture loss of the generated image can be obtained, so that the detailed texture of the generated image can be supervised to improve the model's ability to generate detailed textures.

[0102] Exemplarily, each batch of sample data input during the training phase of the embodiment of the present application includes sample pairs of at least two objects, and there are at least two sample pairs for each object. Among them, a posture video segment and a reference video segment constitute a sample pair. That is, in the currently input batch, there are at least two reference video segments of the first object, wherein the first video segment and the second video segment are a sample pair of the first object in the sample data of the same batch. In this way, it can be ensured that during identity ID supervision, the reference frames in at least two reference video segments in a batch are positive samples of each other, and the images between different IDs are negative samples, so that the identity information of the same object is distinguishable within the batch, and then through the construction of sample data, the identity information of the first object in the generated first simulated digital human video does not change, and at the same time, the model can also achieve generalization on a small number of identity IDs.

[0103] Furthermore, before determining the first loss based on the first simulated digital human video, the method further includes:

[0104] A first contrastive learning loss (InfoNCE loss) of identity information is determined based on the first identity feature and the reference video segment in each sample pair of the sample data.

[0105] Specifically, the second identity feature of the first object is extracted from the fifth image; the fifth image is an image frame other than the second image in the reference video segment of the first object. For example: there are two reference video segments of the first object in the current batch, and each reference video segment includes 5 reference frames. For any reference frame, its second image has 9 positive samples, that is, the other 9 reference frames, and the second identity feature is extracted from each of them. The third identity feature is extracted from the sixth image; the sixth image is an image frame in the target reference video segment of the sample data; the target reference video segment is a reference video segment other than the reference video segment of the first object. For example: there are 4 target reference video segments in the current batch, and each reference video segment includes 5 reference frames. Then, the second image has 20 negative samples, and the third identity feature is extracted from each of them. Among them, the extraction method of the second identity feature and the third identity feature can refer to the extraction method of the first identity feature. Based on the similarity between the first identity feature and the second identity feature and the similarity between the first identity feature and the third identity feature, the second contrastive learning loss of the identity information corresponding to each frame of the simulated digital human image is determined, and its formula can be expressed as:

[0106]

[0107] Among them, id infoNce loss represents the second contrastive learning loss, q·k + represents the similarity between the first identity feature and each second identity feature, p represents the number of positive samples of the second image, q·k i represents the similarity between the first identity feature and the identity features (including the second identity feature and the third identity feature) of the i-th reference frame (including the fifth image and the sixth image) in the batch, k represents the total number of positive and negative samples of the second image, and exp is a natural exponential function.

[0108] Based on the second contrastive learning loss corresponding to each frame of the simulated digital human image, the first contrastive learning loss is determined. For example, the second contrastive learning loss corresponding to each frame of the simulated digital human image is summed or weighted. In this implementation, within a batch, by calculating the similarity between any reference frame and the positive sample, and the similarity between the reference frame and the negative sample, the identity information contrastive learning loss of the generated image is determined based on these two similarities, and the identity information of the generated image is supervised by the contrastive learning loss of the identity information, which is beneficial to the model's perception of the identity information and ensures that the generated video does not have changes in the identity information.

[0109] The computer device may determine the first loss based on the synchronization loss, the first L1 loss, the first perceptual loss, the first structural similarity loss, the first texture loss, and the first contrastive learning loss, such as by summing or weighted summing. In one possible implementation, the first loss may be used as the loss of the entire model, and the total loss may be expressed as:

[0110] Total loss = first loss = synchronization loss + first L1 loss * weight + first perceptual loss + first structural similarity loss + first texture loss + first contrastive learning loss

[0111] Among them, Total loss represents the total loss of the model.

[0112] In an embodiment of the present application, the computer device can use a back-propagation algorithm to perform gradient calculation based on the first loss, thereby updating the parameters of the neural network, and after multiple rounds of iterative training, a trained digital human generation model is obtained.

[0113] It can be seen that in the embodiment of the present application, the image frame in the first video segment is the posture frame of the first object in the sample data, and the image frame in the second video segment is the reference frame corresponding to the posture frame. After obtaining the first video segment, the second video segment, the third video segment and the first audio data of the first object, the computer device removes the identity information of the first object from the posture frame (such as the first image) to obtain the posture feature, which is equivalent to enhancing the posture feature, that is, the posture feature can explicitly characterize the posture change between each posture frame; the computer device fuses the enhanced posture feature, the audio feature extracted from the first audio data and the identity feature (i.e., the first identity feature) extracted from the corresponding reference frame (such as the second image), which is conducive to the neural network to perceive the subtle changes in the posture feature in the fused feature (i.e., the first fused feature), so that the image (such as the simulated digital human image) corresponding to the input image (such as the third image) generated based on the fused feature can be more distinguished in terms of facial posture. In this way, supervising the neural network with additional enhanced posture features is conducive to reducing the influence of the facial posture in the input image on the neural network, thereby achieving posture controllability. By determining the prediction loss based on the generated video (such as the first simulated digital human video), and iteratively training the neural network based on the prediction loss, a digital human generation model with controllable facial posture can be trained, and the digital human generation model can be used to generate more vivid and realistic digital human videos.

[0114] See also Fig. 9 , Fig. 9 A flowchart of another method for training a digital human generation model provided in an embodiment of the present application. The method can be executed by a computer device or a chip in the computer device, such as Fig. 9 As shown, at least the following steps are included:

[0115] 901: Acquire a first video segment, a second video segment, a third video segment, and first audio data of a first object;

[0116] The first audio data corresponds to the first video segment; the first video segment, the second video segment and the third video segment have the same number of frames, that is, the first video segment has the same number of frames as the second video segment and the third video segment.

[0117] 902: For any first image frame in the first video segment, remove identity information of a first object from the first image to obtain a posture feature of the first object;

[0118] 903: extracting audio features of a first object from the first audio data;

[0119] 904: extracting a first identity feature of the first object from a second image in the second video segment corresponding to the first image frame sequence;

[0120] 905: Fusing the posture feature, the audio feature, and the first identity feature to obtain a first fused feature;

[0121] 906: for a third image in the third video segment corresponding to the second image frame sequence, generating a simulated digital human image corresponding to the third image based on the first fusion feature;

[0122] 907: compose a first simulated digital human video with the simulated digital human image corresponding to each frame of the third image in the third video segment;

[0123] The implementation of steps 901-907 can refer to Figure 2 The related descriptions of steps 201-207 in the illustrated embodiment are similar and can achieve the same or similar beneficial effects.

[0124] 908: Determine at least one second loss based on at least one second simulated digital human video; the at least one second simulated digital human video corresponds one-to-one to the at least one second loss; the at least one second simulated digital human video is a video segment output by at least one layer except the last layer in the decoder; the at least one second simulated digital human video corresponds one-to-one to the at least one layer;

[0125] In the embodiment of the present application, the video segments output by the upsampling layers other than the last upsampling layer are recorded as the second simulated digital human video. Accordingly, the neural network will also output multiple second simulated digital human videos, and the computer device selects at least one second simulated digital human video from the multiple second simulated digital human videos, such as: the second simulated digital human video output by the 1 / 2 upsampling layer, the second simulated digital human video output by the 1 / 4 upsampling layer, the second simulated digital human video output by the 1 / 8 upsampling layer, the second simulated digital human video output by the 1 / 16 upsampling layer, etc. For any second simulated digital human video H in the at least one second simulated digital human video, based on each frame of the seventh image in the second simulated digital human video H and the true value image corresponding to each frame of the seventh image, the fourth L1 loss and the second perceptual loss are determined; the second loss of the second simulated digital human video H is determined based on the fourth L1 loss and the second perceptual loss to obtain the at least one second loss.

[0126] Specifically, the fourth L1 loss may be determined by referring to the first L1 loss, and the second perceptual loss may be determined by referring to the first perceptual loss. The second loss of the second simulated digital human video H may be the sum or weighted sum of the fourth L1 loss and the second perceptual loss. In this implementation, at least one second loss is used as an auxiliary loss to supervise the audio features and posture features in each upsampling layer, so that the model's perception of the audio features and posture features will not gradually disappear as the number of network layers increases, which is conducive to ensuring the quality of the feature representation obtained by the last layer of the model.

[0127] 909: Iteratively train the neural network based on the first loss and at least one second loss to obtain a trained digital human generation model.

[0128] In the embodiment of the present application, the total loss of the model can be obtained based on the first loss and at least one second loss, and the total loss can be expressed as:

[0129] Total loss = first loss + second loss 1 + ... + second loss y

[0130] Second loss = fourth L1 loss * weight + second perceptual loss

[0131] Where y represents the amount of the second loss.

[0132] Based on the total loss of the model, the computer device uses a back-propagation algorithm to update the parameters of the neural network to obtain a trained digital human generation model.

[0133] See also Fig.10 , Fig.10A flowchart of a method for generating a digital human provided in an embodiment of the present application. The method can be applied to a computer device (the computer device and the computer device of the method for training a digital human generation model can be the same device or different devices) or a chip in the computer device, such as Fig.10 As shown, at least the following steps are included:

[0134] 1001: Acquire second audio data, a sixth video segment of a second object, and a seventh video segment and an eighth video segment of a third object;

[0135] The sixth video segment, the seventh video segment, and the eighth video segment have the same number of frames; each frame in the sixth video segment includes a complete face of the second subject, each frame in the seventh video segment includes a complete face of the third subject, and each frame in the eighth video segment includes a partial face of the third subject;

[0136] 1002: Input the second audio data, the sixth video segment, the seventh video segment and the eighth video segment into the digital human generation model to generate a third simulated digital human video.

[0137] Specifically, the audio feature of the second audio data is extracted through the audio encoder of the digital human generation model, the posture feature of the image frame in the sixth video segment is extracted through the posture encoder, and the reference frame corresponding to the image frame in the sixth video segment (that is, the corresponding image frame in the seventh video segment) is obtained, the identity feature is extracted through the encoder part of the backbone network, the audio feature, posture feature and identity feature are fused, and the other part of the face of the corresponding image frame in the eighth video segment is predicted through the decoder part of the backbone network to obtain the predicted image frame, and the predicted image frame constitutes a third simulated digital human video.

[0138] Among them, the digital human generation model is based on the above Figure 2 or Fig. 9 The training is obtained by training using the training method described in any one of the embodiments.

[0139] It can be seen that in the embodiment of the present application, by Figure 2 or Fig. 9 The digital human generation model trained by the training method described in any one of the embodiments can generate digital human videos with controllable postures, and the generated 2D digital human images have high detail texture quality.

[0140] Corresponding to the above Figures 2 to 9 The method for training a digital human generation model described above is based on the same technical concept. The embodiment of the present application also provides a digital human generation model training device. Fig.11 A schematic diagram of the module composition of a digital human generation model training device provided in an embodiment of the present application, which can be used to perform Figures 2 to 9The generative model training method for digital humans described in Fig.11 As shown, the device includes a first acquisition unit and a first processing unit, wherein:

[0141] The first acquisition unit 1101 is used to acquire a first video segment, a second video segment, a third video segment and first audio data of a first object; the first video segment has the same number of frames as the second video segment and the third video segment;

[0142] The first processing unit 1102 is used to remove the identity information of the first object from any frame of the first image in the first video segment to obtain the posture feature of the first object; extract the audio feature of the first object from the first audio data; for the second image corresponding to the first image frame sequence in the second video segment, extract the first identity feature of the first object from the second image; fuse the posture feature, the audio feature and the first identity feature to obtain the first fusion feature; for the third image corresponding to the second image frame sequence in the third video segment, generate a simulated digital human image corresponding to the third image based on the first fusion feature; form a first simulated digital human video with the simulated digital human images corresponding to each frame of the third image in the third video segment; iteratively train the neural network based on the first simulated digital human video to obtain a trained digital human generation model.

[0143] It can be seen that in the embodiment of the present application, the image frame in the first video segment is the posture frame of the first object in the sample data, and the image frame in the second video segment is the reference frame corresponding to the posture frame. After obtaining the first video segment, the second video segment, the third video segment and the first audio data of the first object, the digital human generation model training device removes the identity information of the first object from the posture frame (such as the first image) to obtain the posture feature, which is equivalent to enhancing the posture feature, that is, the posture feature can explicitly characterize the posture change between each posture frame; the digital human generation model training device fuses the enhanced posture feature, the audio feature extracted from the first audio data and the identity feature (i.e., the first identity feature) extracted from the corresponding reference frame (such as the second image), which is conducive to the neural network to perceive the subtle changes in the posture feature in the fused feature (i.e., the first fused feature), so that the image (such as the simulated digital human image) corresponding to the input image (such as the third image) generated based on the fused feature can be more distinguished in terms of facial posture. In this way, supervising the neural network with additional enhanced posture features is conducive to reducing the influence of the facial posture in the input image on the neural network, thereby achieving posture controllability. By iteratively training the neural network based on the generated video (such as the first simulated digital human video), a digital human generation model with controllable facial posture can be trained. The digital human generation model can be used to generate more vivid and realistic digital human videos.

[0144] In a possible implementation, in fusing the posture feature, the audio feature, and the first identity feature to obtain the first fused feature, the first processing unit 1102 is specifically configured to:

[0145] Convert posture features into feature vectors;

[0146] Concatenate the feature vector with the audio feature and convert it into the second fusion feature;

[0147] Multiply the second fused feature by the first identity feature to obtain a third fused feature;

[0148] The third fusion feature is normalized to obtain the first fusion feature.

[0149] In a possible implementation, the first video segment and the second video segment are sample pairs of a first object in sample data of the same batch; the sample data includes sample pairs of at least two objects, and each object has at least two sample pairs.

[0150] In a possible implementation, the second video segment is a reference video segment of the first video segment, and each sample pair of the sample data includes a reference video segment;

[0151] The first processing unit 1102 is further configured to: determine a first contrastive learning loss of the identity information based on the first identity feature and the reference video segment in each sample pair of the sample data;

[0152] In determining the first loss based on the first simulated digital human video, the first processing unit 1102 is specifically configured to:

[0153] determining a synchronization loss, a first L1 loss, a first perceptual loss, a first structural similarity loss, and a first texture loss based on the first simulated digital human video;

[0154] A first loss is determined based on a synchronization loss, a first L1 loss, a first perceptual loss, a first structural similarity loss, a first texture loss, and a first contrastive learning loss.

[0155] In a possible implementation, in determining the synchronization loss, the first L1 loss, the first perception loss, the first structural similarity loss, and the first texture loss based on the first simulated digital human video, the first processing unit 1102 is specifically configured to:

[0156] Determine the synchronization loss based on the audio and video synchronization detection result of the first simulated digital human video and the target synchronization tag; the target synchronization tag is the synchronization tag of the first video segment and the first audio data;

[0157] Determine a first L1 loss based on each frame of the simulated digital human image in the first simulated digital human video and a true value image corresponding to each frame of the simulated digital human image;

[0158] Determine a first perceptual loss based on each frame of the simulated digital human image and the true value image;

[0159] Determine a first structural similarity loss based on each frame of the simulated digital human image and the true value image;

[0160] Based on each frame of the simulated digital human image and the true value image, a first texture loss is determined.

[0161] In a possible implementation, in determining the first L1 loss based on each frame of the simulated digital human image in the first simulated digital human video and a true value image corresponding to each frame of the simulated digital human image, the first processing unit 1102 is specifically configured to:

[0162] Calculate the mean absolute error between the RGB value of each frame of the simulated digital human image and the RGB value of the true value image;

[0163] The first L1 loss is determined based on the mean absolute error between the RGB values ​​of each frame of the simulated digital human image and the RGB values ​​of the true value image.

[0164] In a possible implementation, in determining the first perceptual loss based on each frame of the simulated digital human image and the true value image, the first processing unit 1102 is specifically configured to:

[0165] Extracting features from each frame of simulated digital human image to obtain at least one first feature layer of each frame of simulated digital human image;

[0166] Performing feature extraction on the true value image to obtain at least one layer of second features of the true value image; the at least one layer of second features corresponds to the at least one layer of first features in a one-to-one manner;

[0167] Calculate the second L1 loss between the first feature of each layer and the corresponding target second feature;

[0168] Calculate the third L1 loss between the first style transfer matrix of the first feature of each layer and the second style transfer matrix of the second feature of the target;

[0169] Based on the second L1 loss and the third L1 loss, a first perceptual loss is determined.

[0170] In a possible implementation, in determining the first structural similarity loss based on each frame of the simulated digital human image and the true value image, the first processing unit 1102 is specifically configured to:

[0171] Detect the mouth key points of each frame of simulated digital human image and true value image;

[0172] Determine a first mouth region image based on the mouth key points obtained in each frame of the simulated digital human image;

[0173] Determine a second mouth region image based on the mouth key points obtained in the true value image;

[0174] Performing normalization processing on the first mouth area image and the second mouth area image;

[0175] Calculating the structural similarity between the normalized first mouth region image and the normalized second mouth region image;

[0176] Determine a second structural similarity loss corresponding to each frame of the simulated digital human image based on the structural similarity and the mouth area weight;

[0177] The first structural similarity loss is determined based on the second structural similarity loss corresponding to each frame of the simulated digital human image.

[0178] In a possible implementation, in determining the first texture loss based on each frame of the simulated digital human image and the true value image, the first processing unit 1102 is specifically configured to:

[0179] Performing semantic segmentation and region division on each frame of the simulated digital human image to obtain a plurality of first rectangular regions;

[0180] Performing semantic segmentation and region division on the true value image to obtain a plurality of second rectangular regions; the plurality of second rectangular regions correspond one-to-one to the plurality of first rectangular regions;

[0181] Calculating the high-frequency information distribution of each first rectangular area, and calculating the high-frequency information distribution of the target second rectangular area corresponding to each first rectangular area;

[0182] Determine a second texture loss corresponding to each frame of the simulated digital human image based on the high-frequency information distribution of each first rectangular area and the high-frequency information distribution of the target second rectangular area;

[0183] The first texture loss is determined based on the second texture loss corresponding to each frame of the simulated digital human image.

[0184] In a possible implementation, in determining the first contrastive learning loss of the identity information based on the first identity feature and the reference video segment in each sample pair of the sample data, the first processing unit 1102 is specifically configured to:

[0185] extracting a second identity feature of the first object from a fifth image; the fifth image is an image frame other than the second image in the reference video segment of the first object;

[0186] Extracting a third identity feature from a sixth image; the sixth image is an image frame in a target reference video segment of the sample data; the target reference video segment is a reference video segment other than the reference video segment of the first object;

[0187] Determining a second contrastive learning loss of identity information corresponding to each frame of the simulated digital human image based on a similarity between the first identity feature and the second identity feature and a similarity between the first identity feature and the third identity feature;

[0188] The first contrastive learning loss is determined based on the second contrastive learning loss corresponding to each frame of the simulated digital human image.

[0189] In a possible implementation, the backbone network of the neural network is a U-shaped network, and the first simulated digital human video is a video segment output by the last layer of a decoder of the U-shaped network;

[0190] The first processing unit 1102 is further configured to: determine at least one second loss based on at least one second simulated digital human video; the at least one second simulated digital human video corresponds one-to-one to the at least one second loss; the at least one second simulated digital human video is a video segment output by at least one layer except the last layer in the decoder; the at least one second simulated digital human video corresponds one-to-one to the at least one layer;

[0191] In terms of iteratively training the neural network based on the first loss to obtain a trained digital human generation model, the first processing unit 1102 is specifically used to:

[0192] The neural network is iteratively trained based on the first loss and at least one second loss to obtain a trained digital human generation model.

[0193] In a possible implementation, in determining at least one second loss based on at least one second simulated digital human video, the first processing unit 1102 is specifically configured to:

[0194] For any second simulated digital human video H of at least one second simulated digital human video, determine a fourth L1 loss and a second perceptual loss based on each frame of the seventh image in the second simulated digital human video H and a true value image corresponding to each frame of the seventh image;

[0195] A second loss of the second simulated digital human video H is determined based on the fourth L1 loss and the second perceptual loss to obtain at least one second loss.

[0196] It should be noted that the embodiment of the digital human generative model training device in the present application and the embodiment of the digital human generative model training method in the present application are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the aforementioned corresponding digital human generative model training method.

[0197] Corresponding to the above Fig.10 The method for generating a digital human described above is based on the same technical concept. The embodiment of the present application also provides a device for generating a digital human, such as Fig.12 As shown, the device includes a second acquisition unit 1201 and a second processing unit 1202, wherein:

[0198] The second acquisition unit 1201 is used to acquire second audio data, a sixth video segment of the second object, and a seventh and eighth video segments of the third object; the sixth video segment has the same number of frames as the seventh and eighth video segments;

[0199] The second processing unit 1202 is used to input the second audio data, the sixth video segment, the seventh video segment and the eighth video segment into the digital human generation model to generate a third simulated digital human video;

[0200] Among them, the digital human generation model is based on the above Figure 2 or Fig. 9 The training is obtained by training using the training method described in any one of the embodiments.

[0201] It should be noted that the embodiment of the digital human generation device in this application and the embodiment of the digital human generation method in this application are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the aforementioned corresponding digital human generation method.

[0202] Furthermore, corresponding to the above Figures 2 to 10 Based on the same technical concept, the present application also provides a computer device for executing the above-mentioned digital human generation model training method or digital human generation method, such as Fig.13As shown, the computer device may have relatively large differences due to different configurations or performances, and may include one or more processors 1301 and memory 1302, and the memory 1302 may store one or more storage applications or data. Among them, the memory 1302 may be a short-term storage or a persistent storage. The application stored in the memory 1302 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the computer device. Furthermore, the processor 1301 may be configured to communicate with the memory 1302 to execute a series of computer executable instructions in the memory 1302 on the computer device. The computer device may also include one or more power supplies 1303, one or more wired or wireless network interfaces 1304, one or more input and output interfaces 1305, one or more keyboards 1306, etc. The memory 1302 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or compact disc read-only memory (CD-ROM).

[0203] In a specific embodiment, a computer device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions in the computer device, and the one or more programs configured to be executed by one or more processors include computer executable instructions for performing the following:

[0204] Acquire a first video segment, a second video segment, a third video segment and first audio data of a first object; the first video segment has the same number of frames as the second video segment and the third video segment;

[0205] For any first image frame in the first video segment, remove the identity information of the first object from the first image to obtain a posture feature of the first object;

[0206] extracting an audio feature of a first object from the first audio data;

[0207] For a second image in the second video segment corresponding to the first image, extracting a first identity feature of the first object from the second image;

[0208] Fusing the posture feature, the audio feature, and the first identity feature to obtain a first fused feature;

[0209] For a third image in the third video segment corresponding to the second image, generating a simulated digital human image corresponding to the third image based on the first fusion feature;

[0210] The simulated digital human image corresponding to each frame of the third image in the third video segment is used to form a first simulated digital human video;

[0211] The neural network is iteratively trained based on the first simulated digital human video to obtain a trained digital human generation model.

[0212] In another specific embodiment, a computer device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions in the computer device, and the one or more programs are configured to be executed by one or more processors, including computer-executable instructions for performing the following:

[0213] Acquire the second audio data, the sixth video segment of the second object, the seventh video segment and the eighth video segment of the third object; the sixth video segment has the same number of frames as the seventh video segment and the eighth video segment;

[0214] inputting the second audio data, the sixth video segment, the seventh video segment, and the eighth video segment into the digital human generation model to generate a third simulated digital human video;

[0215] The digital human generation model is based on Figures 2 to 9 The training is obtained by training using the training method of any one of the embodiments.

[0216] It should be noted that the implementation of each operation can also refer to Figures 2 to 10 The corresponding description of the method of any one of the embodiments shown.

[0217] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device, and of course can also include extended storage media supported by the computer device, or can also include storage media used in conjunction with the chip. The computer-readable storage medium provides a storage space, which stores the operating system of the computer device. In addition, one or more instructions suitable for being loaded and executed by the chip are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be RAM, double data rate synchronous dynamic random access memory (Double Data Rate Synchronous Dynamic Random Access Memory, DDR), etc., or it can be a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned chip. In one embodiment, one or more instructions stored in the computer-readable storage medium can be loaded and executed by the device to implement Figure 2 , Fig. 9 or Fig.10 The method shown in .

[0218] The present application also provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a device to execute Figure 2 , Fig. 9 or Fig.10 The computer program product may be a software installation package.

[0219] It should be understood that the processor mentioned in the embodiments of the present application may be a CPU, or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0220] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a ROM, a programmable read-only memory (Programmable ROM, PROM), an EPROM, an electrically erasable programmable read-only memory (ElectricallyEPROM, EEPROM) or a flash memory. The volatile memory may be a RAM, which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory (Synchlink DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DRRAM).

[0221] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) is integrated in the processor.

[0222] It should be noted that the memory described herein is intended to include, without being limited to, these and any other suitable types of memory.

[0223] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0224] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely exemplary. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0225] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0226] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium.

[0227] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. In the text description of this application, the character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0228] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0229] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0230] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a generative model of a digital human, characterized in that: The method comprises: Acquire a first video segment, a second video segment, a third video segment and first audio data of a first object; the first video segment has the same number of frames as the second video segment and the third video segment; For any first image frame in the first video segment, remove the identity information of the first object from the first image to obtain a posture feature of the first object; extracting audio features of the first object from the first audio data; For a second image in the second video segment corresponding to the first image frame sequence, extracting a first identity feature of the first object from the second image; Fusing the posture feature, the audio feature, and the first identity feature to obtain a first fused feature; For a third image in the third video segment corresponding to the second image frame sequence, generating a simulated digital human image corresponding to the third image based on the first fusion feature; The simulated digital human image corresponding to each frame of the third image in the third video segment is used to form a first simulated digital human video; The neural network is iteratively trained based on the first simulated digital human video to obtain a trained digital human generation model.

2. The method according to claim 1, characterized in that The iterative training of the neural network based on the first simulated digital human video to obtain a trained digital human generation model includes: determining a first loss based on the first simulated digital human video; The neural network is iteratively trained based on the first loss to obtain a trained digital human generation model.

3. The method according to claim 1 or 2, characterized in that: The step of fusing the posture feature, the audio feature and the first identity feature to obtain a first fused feature includes: Converting the posture feature into a feature vector; Concatenate the feature vector with the audio feature and convert them into a second fusion feature; Multiplying the second fused feature by the first identity feature to obtain a third fused feature; The third fusion feature is normalized to obtain the first fusion feature.

4. The method according to claim 2, characterized in that: The first video segment and the second video segment are sample pairs of the first object in the same batch of sample data; the sample data includes sample pairs of at least two objects, and each object has at least two sample pairs.

5. The method according to claim 4, characterized in that The second video segment is a reference video segment of the first video segment, and each sample pair of the sample data includes a reference video segment; Before determining a first loss based on the first simulated digital human video, the method further includes: Determining a first contrastive learning loss of identity information based on the first identity feature and a reference video segment in each sample pair of the sample data; The determining a first loss based on the first simulated digital human video comprises: determining a synchronization loss, a first L1 loss, a first perceptual loss, a first structural similarity loss, and a first texture loss based on the first simulated digital human video; The first loss is determined based on the synchronization loss, the first L1 loss, the first perceptual loss, the first structural similarity loss, the first texture loss, and the first contrastive learning loss.

6. The method according to claim 5, characterized in that The determining of a synchronization loss, a first L1 loss, a first perceptual loss, a first structural similarity loss, and a first texture loss based on the first simulated digital human video comprises: Determine the synchronization loss based on the audio and video synchronization detection result of the first simulated digital human video and the target synchronization tag; the target synchronization tag is the synchronization tag of the first video segment and the first audio data; Determine a first L1 loss based on each frame of simulated digital human image in the first simulated digital human video and a true value image corresponding to each frame of simulated digital human image; Determine a first perceptual loss based on each frame of the simulated digital human image and the true value image; Determine a first structural similarity loss based on each frame of the simulated digital human image and the true value image; A first texture loss is determined based on each frame of the simulated digital human image and the true value image.

7. The method according to claim 6, characterized in that The determining a first L1 loss based on each frame of the simulated digital human image in the first simulated digital human video and a true value image corresponding to each frame of the simulated digital human image comprises: Calculating the mean absolute error between the RGB value of each frame of the simulated digital human image and the RGB value of the true value image; The first L1 loss is determined based on the mean absolute error between the RGB value of each frame of the simulated digital human image and the RGB value of the true value image.

8. The method according to claim 6, characterized in that The determining of the first perceptual loss based on each frame of the simulated digital human image and the true value image comprises: Performing feature extraction on each frame of the simulated digital human image to obtain at least one first feature layer of each frame of the simulated digital human image; Performing feature extraction on the true value image to obtain at least one layer of second features of the true value image; the at least one layer of second features corresponds to the at least one layer of first features in a one-to-one manner; Calculate a second L1 loss between the first feature of each layer and the corresponding target second feature; Calculate a third L1 loss between a first style transfer matrix of the first feature of each layer and a second style transfer matrix of the target second feature; Based on the second L1 loss and the third L1 loss, the first perceptual loss is determined.

9. The method according to claim 6, characterized in that Determining a first structural similarity loss based on each frame of the simulated digital human image and the true value image includes: Performing mouth key point detection on each frame of the simulated digital human image and the true value image; Determine a first mouth region image based on the mouth key points obtained in each frame of the simulated digital human image; Determining a second mouth region image based on the mouth key points obtained in the true value image; performing normalization processing on the first mouth area image and the second mouth area image; Calculating the structural similarity between the first mouth region image after normalization and the second mouth region image after normalization; Determining a second structural similarity loss corresponding to each frame of the simulated digital human image based on the structural similarity and the mouth region weight; The first structural similarity loss is determined based on the second structural similarity loss corresponding to each frame of the simulated digital human image.

10. The method according to claim 6, characterized in that The determining of the first texture loss based on each frame of the simulated digital human image and the true value image comprises: Performing semantic segmentation and region division on each frame of the simulated digital human image to obtain a plurality of first rectangular regions; Performing semantic segmentation and region division on the true value image to obtain a plurality of second rectangular regions; the plurality of second rectangular regions correspond one-to-one to the plurality of first rectangular regions; Calculating the high-frequency information distribution of each of the first rectangular areas, and calculating the high-frequency information distribution of the target second rectangular area corresponding to each of the first rectangular areas; Determining a second texture loss corresponding to each frame of the simulated digital human image based on the high-frequency information distribution of each of the first rectangular regions and the high-frequency information distribution of the target second rectangular region; The first texture loss is determined based on the second texture loss corresponding to each frame of the simulated digital human image.

11. The method according to claim 5, characterized in that The determining a first contrastive learning loss of identity information based on the first identity feature and a reference video segment in each sample pair of the sample data comprises: extracting a second identity feature of the first object from a fifth image; the fifth image is an image frame other than the second image in a reference video segment of the first object; Extracting a third identity feature from a sixth image; the sixth image is an image frame in a target reference video segment of the sample data; the target reference video segment is a reference video segment other than the reference video segment of the first object; Determining a second contrastive learning loss of the identity information corresponding to each frame of the simulated digital human image based on the similarity between the first identity feature and the second identity feature and the similarity between the first identity feature and the third identity feature; The first contrastive learning loss is determined based on the second contrastive learning loss corresponding to each frame of the simulated digital human image.

12. The method according to any one of claims 1 to 11, characterized in that: The backbone network of the neural network is a U-shaped network, and the first simulated digital human video is a video segment output by the last layer of the decoder of the U-shaped network; Before iteratively training the neural network based on the first loss to obtain a trained digital human generation model, the method further includes: Determine at least one second loss based on at least one second simulated digital human video; the at least one second simulated digital human video corresponds one-to-one to the at least one second loss; the at least one second simulated digital human video is a video segment output by at least one layer in the decoder except the last layer; the at least one second simulated digital human video corresponds one-to-one to the at least one layer; The iterative training of the neural network based on the first loss to obtain a trained digital human generation model includes: The neural network is iteratively trained based on the first loss and the at least one second loss to obtain a trained digital human generation model.

13. The method according to claim 12, characterized in that The determining at least one second loss based on at least one second simulated digital human video comprises: For any one of the at least one second simulated digital human videos H, determining a fourth L1 loss and a second perceptual loss based on each frame of the seventh image in the second simulated digital human video H and a true value image corresponding to each frame of the seventh image; A second loss of the second simulated digital human video H is determined based on the fourth L1 loss and the second perceptual loss to obtain the at least one second loss.

14. A method for generating a digital human, characterized in that: The method comprises: Acquire second audio data, a sixth video segment of the second object, a seventh video segment and an eighth video segment of the third object; the sixth video segment has the same number of frames as the seventh video segment and the eighth video segment; inputting the second audio data, the sixth video segment, the seventh video segment and the eighth video segment into a digital human generation model to generate a third simulated digital human video; Wherein, the digital human generation model is obtained by training based on the training method described in any one of claims 1 to 13.

15. A digital human generative model training device, characterized in that: The device comprises a first acquisition unit and a first processing unit, wherein: The first acquisition unit is used to acquire a first video segment, a second video segment, a third video segment and first audio data of a first object; the first video segment has the same number of frames as the second video segment and the third video segment; The first processing unit is used to remove the identity information of the first object from any frame of the first image in the first video segment to obtain the posture feature of the first object; extract the audio feature of the first object from the first audio data; for the second image corresponding to the first image frame sequence in the second video segment, extract the first identity feature of the first object from the second image; fuse the posture feature, the audio feature and the first identity feature to obtain a first fusion feature; for the third image corresponding to the second image frame sequence in the third video segment, generate a simulated digital human image corresponding to the third image based on the first fusion feature; form a first simulated digital human video with the simulated digital human images corresponding to each frame of the third image in the third video segment; iteratively train the neural network based on the first simulated digital human video to obtain a trained digital human generation model.

16. A digital human generation device, characterized in that: The device comprises a second acquisition unit and a second processing unit, wherein: The second acquisition unit is used to acquire the second audio data, the sixth video segment of the second object, the seventh video segment and the eighth video segment of the third object; the sixth video segment has the same number of frames as the seventh video segment and the eighth video segment; The second processing unit is used to input the second audio data, the sixth video segment, the seventh video segment and the eighth video segment into the digital human generation model to generate a third simulated digital human video; Wherein, the digital human generation model is obtained by training based on the training method described in any one of claims 1 to 13.

17. A computer device, characterized in that: The computer device comprises: Processor; and A memory storing computer executable instructions, wherein the executable instructions are configured to be executed by the processor, and the executable instructions include instructions for executing the steps in the method according to any one of claims 1 to 13 or claim 14.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for execution by a device, and when the computer program is executed, the steps in the method according to any one of claims 1 to 13 or claim 14 are implemented.

Images