Digital human video generation method for synchronizing three-dimensional portrait expressions and gestures of photo digital human under driving of voice

Abstract

The invention discloses a voice-driven photo digital human three-dimensional portrait expression and gesture synchronization digital human video generation method, which integrates automatic generation of a voice-to-facial expression coefficient sequence, automatic generation of a voice-to-head posture motion coefficient sequence and automatic generation of a voice-to-gesture sequence. A three-dimensional portrait coefficient is used as an intermediate value to synchronously render an expression feature, a head motion feature and a gesture feature in voice to a three-dimensional portrait, a two-dimensional image is mapped according to the three-dimensional portrait to serve as a video frame, and an image enhancement model is introduced to enhance the video frame, so that the video frame is enhanced. And writing the enhanced video frame into the video, fusing the audio and the video, and finally generating an audio-driven synchronous video. The generated video has facial expressions, head motion postures and gestures at the same time, and the digital human video has synchronism, identity consistency and clearer and more natural face.

Description

Technical Field

[0001] The invention relates to the field of digital human video generation, in particular to a method for generating digital human videos with synchronization of three-dimensional human portrait expressions and gestures driven by voice. Background Art

[0002] In the digital age, digital human technology is gaining increasing attention and showing great potential in many fields. With the booming development of social media platforms such as WeChat, Weibo, Douyin, Bilibili, and Xiaohongshu, the application of digital humans in the fields of entertainment and social interaction has become more extensive and diversified. They are gradually changing people's entertainment and social habits by providing innovative interactive experiences and personalized services. In addition, in the fields of education, training, customer service, etc., digital human technology also provides a more natural and realistic interactive experience. The following are some specific application examples:

[0003] (1) Virtual anchors: In the entertainment field, digital humans can appear as virtual anchors. They can replace real people in live broadcasts and video production, and provide services such as news broadcasting, education and training, and product display.

[0004] (2) Social interaction: Digital humans are an important symbol of next-generation social interaction. Users can use their virtual avatars to experience a variety of immersive social scenarios in the virtual world. This social method allows users to establish social connections with like-minded people based on interest maps or recommendations.

[0005] (3) Film, television and animation: In the film and television industry, digital human technology is used to create virtual characters and special effects, making the production of movies and animations more realistic and efficient.

[0006] (4) Brand marketing: Brands can use digital humans as virtual image spokespersons to establish connections with young consumers and enhance their brand image and market influence. Digital humans can help brands quickly attract the attention of young people and help brands capture more target users.

[0007] In addition, digital humans can be divided into two categories: 2D and 3D. Due to different internal driving methods, 2D digital humans do not require complex modeling and can be generated with only pictures and video materials, while 3D digital humans require modeling, motion capture and other technologies to achieve interaction, and the cost is relatively expensive. Therefore, in actual application, 2D digital humans are more widely used in short videos, live broadcasts and other scenes. By creating short videos with digital humans, bloggers can indeed save a lot of creation time, and can also complete the appearance without showing their faces. In the technology of 2D digital humans, voice is used to drive a portrait to move and generate a video with synchronized voice and video. This type of technology has shown significant advantages in many fields due to its high efficiency, low cost and ease of use. Among them, the technology of converting sound into facial expressions is the key to achieving natural and realistic artificial interaction.

[0008] Currently, the technology of using audio to drive a portrait to move is mainly divided into two branches:

[0009] One branch is to use voice to drive static portraits to produce lip movements, such as Oculus Lipsync and Wav2Lip. Because voice and lips have a strong relationship, the technology is relatively mature and has high real-time performance.

[0010] Another branch is to use voice-driven generation of different head postures and expressions; such as PoseTalk, RAD-NeRF. The generated video not only includes lip movements, but also generates different head postures and expressions, providing more comprehensive facial animation. Through natural and realistic head movement effects, the realism of interaction is improved, and the corresponding emotions can be expressed according to the voice content, making the digital human more expressive.

[0011] Although some current technologies have achieved some results, they still have some defects, such as

[0012] 1. In the technology of using voice to drive static portraits to generate lip movements, because it only focuses on lip movements, the expression changes in other parts of the face are not handled finely enough, and the differences in facial features and speaking states of different people are not handled well enough.

[0013] 2. In the technology of using voice-driven generation of different head postures and expressions, the technology is more complicated to implement because more facial features and movements need to be processed, and higher computing resources are required in real-time applications, but distorted faces may be generated because head movements and expressions are not fully unraveled; inaccurate expressions and unnatural sequences may be produced.

[0014] 3. In the field of voice-driven photo digital humans, research and development often focus on a single interaction dimension, such as focusing only on the dynamic generation of facial expressions, or focusing on one of expressions, head postures, and gestures. This focus optimizes the performance of digital humans in specific interaction tasks, but also limits their naturalness and realism in multimodal interactions. Summary of the invention

[0015] The present invention aims at the problems of unnatural head movement, distorted expression and identity inconsistency in the existing voice-driven digital human facial expression synchronization in photos, and proposes a method for generating digital human videos with synchronized expression and gestures of three-dimensional portraits driven by voice in photos. The method improves the naturalness and synchronization of head movement and expression, reduces identity inconsistency and facial distortion, and increases the diversity of gesture synchronization and head movement while synchronizing voice, expression and head movement. At the same time, an image enhancement model is introduced to enhance the generated video frames and write them into the video; thus, a method for generating digital human videos with synchronized expression, head posture and gestures driven by voice in photos is realized.

[0016] The present invention proposes a method for generating a digital human video with a voice-driven three-dimensional portrait expression and gesture synchronization in a digital human in a photo, comprising the following steps:

[0017] Step 1: Collect anchor broadcast videos, and pre-process the anchor broadcast videos to obtain pre-processed videos;

[0018] Step 2: Extract the audio file from the preprocessed video obtained in step 1;

[0019] Step 3: Use video-expression coefficient model; video-gesture coefficient model; audio-gesture model; image enhancement model;

[0020] Step 4: The expression coefficient generation model includes an audio encoder composed of a residual network and an output layer composed of a linear network; the video preprocessed in step 1 is used to obtain the real expression coefficient sequence of the video through the video-expression coefficient model in step 3, and the audio file extracted in step 2 is sequentially passed through the audio encoder and output layer of the expression coefficient generation model to obtain the generated expression coefficient sequence; the loss value of the real expression coefficient sequence and the generated expression coefficient sequence is obtained through the mean square error loss function, and the expression coefficient generation model is obtained by minimizing the loss value and iteratively optimizing;

[0021] Step 5: define a generative adversarial network, including a generative model and a discriminative model, wherein the generative model includes a posture coefficient generative model and a generative loss module connected in series, and the discriminative model includes a discriminator and a discriminative loss module connected in series;

[0022] The video preprocessed in step 1 is used to obtain the true posture coefficient sequence through the video-posture coefficient model in step 3. The video preprocessed in step 1 is used to generate the posture coefficient sequence through the posture coefficient generation model of the generation model to obtain the generated posture coefficient sequence. The true posture coefficient sequence and the generated posture coefficient sequence are used to obtain the generated loss value through the KL divergence loss function and the mean square error loss function in the generation loss module.

[0023] The probability value of the true value is obtained by the discriminator for the real posture coefficient sequence, and the probability value of the false value is obtained by the discriminator for the generated posture coefficient sequence. The discriminant loss module obtains the discriminant loss value by comparing the difference between the probability value of the false value and the value 0 and the difference between the probability value of the true value and the value 1. The generated loss value and the discriminant loss value constitute the loss value of the adversarial generation network;

[0024] First, the generative model is fixed, the loss value of the adversarial generative network is maximized, and the discriminative model is trained; then the discriminative model is fixed, the loss value of the adversarial generative network is minimized, and the generative model is trained. The pose coefficient generative model in the optimized generative model is obtained by alternately training the generative model and the discriminative model;

[0025] Step 6: Input the required audio and portrait, input the audio into the expression coefficient generation model in step 4 to obtain an audio expression coefficient sequence; input the audio into the posture coefficient generation model in step 5 to obtain an audio posture coefficient sequence; input the audio into the audio-gesture model in step 3 to generate a gesture sequence; obtain the audio expression coefficient sequence, audio posture coefficient sequence, gesture sequence and portrait by using a three-dimensional rendering model to obtain a three-dimensional portrait model with facial expressions, postures and gestures synchronization, obtain a two-dimensional image by three-dimensional to two-dimensional mapping of the three-dimensional portrait model, obtain a generated image by the image enhancement model, and a series of generated images constitute a digital human video.

[0026] In step 1, the anchor broadcast video is collected and preprocessed to obtain the preprocessed video, which specifically includes:

[0027] The frame number and audio sampling rate of the anchor's broadcast video are adjusted to obtain a preprocessed video.

[0028] In step 2, the audio file is extracted from the preprocessed video obtained in step 1, including:

[0029] Use ffmpeg tool to batch extract audio files from pre-processed videos;

[0030] In step 3, prepare the pre-trained model, including the video-expression coefficient model, the video-posture coefficient model, the audio-gesture model, and the image enhancement model. Specifically:

[0031] 3.1) Video-expression coefficient model uses 3D Morphable Models (3D Morphable Models,

[0032] 3DMM), extracting 3DMM facial expression coefficients from the input portrait video;

[0033] 3.2) Video-Gesture Coefficient Model Using 3D Morphable Models

[0034] 3DMM), extract 3DMM head motion coefficients by inputting portrait video;

[0035] 3.3) Audio-gesture model uses voice-driven gesture template model to obtain gesture sequence through input audio;

[0036] 3.4) The image enhancement model uses GFPGAN to improve the resolution of the input image and output a high-resolution image;

[0037] In step 4, the expression coefficient generation model includes an audio encoder composed of a residual network and an output layer composed of a linear network; the video preprocessed in step 1 is used to obtain the real expression coefficient sequence of the video through the video-expression coefficient model in step 3, and the audio file extracted in step 2 is sequentially passed through the audio encoder and output layer of the expression coefficient generation model to obtain the generated expression coefficient sequence; the loss value of the real expression coefficient sequence and the generated expression coefficient sequence is obtained through the mean square error loss function, and the expression coefficient generation model is obtained by minimizing the loss value and iteratively optimizing, specifically:

[0038] 4.1) The expression coefficient generation model includes an audio encoder composed of a residual network and an output layer composed of a linear network; the audio encoder is denoted as Φ A ; The output layer is denoted as Φ M ;

[0039] 4.2) The video preprocessed in step 1 is used to obtain the true expression coefficient sequence through the video-expression coefficient model in step 2, denoted as β′ {1,...,t} ;

[0040] 4.3) Obtain the 0.2s Mel spectrum feature sequence from the audio file extracted in step 2, also called the audio feature sequence, denoted as a {1,...t} , specifically:

[0041] 4.3.1) Load the audio file and obtain the audio signal and sampling rate;

[0042] 4.3.2) Perform pre-emphasis processing on the audio signal to highlight the high frequency part;

[0043] 4.3.3) The audio signal is divided into frames using a Hamming window, and each frame is subjected to a fast Fourier transform to obtain a frequency domain representation. The frequency domain representation is obtained by calculating the square sum of the real part and the imaginary part to obtain a power spectrum;

[0044] 4.3.4) Use the mel function in the audio analysis tool librosa library to create a Mel filter bank;

[0045] 4.3.5) Apply the Mel filter bank to the power spectrum to obtain a Mel spectrum graph, take the logarithm of each element of the Mel spectrum graph to obtain a Mel spectrum feature; intercept the Mel spectrum feature corresponding to 0.2s as the Mel spectrum feature sequence;

[0046] 4.4) Use the audio encoder to encode the audio feature sequence into the latent space to obtain the audio coding feature; the audio coding feature is denoted as aud emb ;

[0047] 4.5) Input aud to the output layer of the linear network emb The expression coefficient is denoted as β {1,...,t} ; The network formula is as follows:

[0048] β {1,...,t} =φ M (φ A (α {1,...,t} )) (1)

[0049] This formula represents the audio feature sequence a {1,...t} Through the audio encoder Φ A and the output layer Φ M Get the expression coefficient β {1,...,t} ;

[0050] 4.5) Calculate the true expression coefficient β′ {1,...,t} and the generated expression coefficient β {1,...,t} The mean square error between is used as the loss function, denoted as loss_exp, and the formula is as follows.

[0051]

[0052] This formula represents the loss between the generated expression coefficient and the true expression coefficient calculated using the mean square error function;

[0053] where β′ i Represents the true expression coefficient β′ {1,...,t} ;

[0054] where β i Represents the generated expression coefficient β {1,...,t} ;

[0055] Where N represents the number of training data;

[0056] 4.6) The expression coefficient generation model is obtained by minimizing loss_exp, back propagation, and iterative optimization.

[0057] In step 5, a generative adversarial network is defined, including a generative model and a discriminative model, wherein the generative model includes a posture coefficient generative model and a generative loss module connected in series, and the discriminative model includes a discriminator and a discriminative loss module connected in series;

[0058] The video preprocessed in step 1 is used to obtain the true posture coefficient sequence through the video-posture coefficient model in step 3. The video preprocessed in step 1 is used to generate the posture coefficient sequence through the posture coefficient generation model of the generation model to obtain the generated posture coefficient sequence. The true posture coefficient sequence and the generated posture coefficient sequence are used to obtain the generated loss value through the KL divergence loss function and the mean square error loss function in the generation loss module.

[0059] The probability value of the true value is obtained by the discriminator for the real posture coefficient sequence, and the probability value of the false value is obtained by the discriminator for the generated posture coefficient sequence. The discriminant loss module obtains the discriminant loss value by comparing the difference between the probability value of the false value and the value 0 and the difference between the probability value of the true value and the value 1. The generated loss value and the discriminant loss value constitute the loss value of the adversarial generation network;

[0060] First, fix the generative model, maximize the loss value of the adversarial generative network, and train the discriminative model; then fix the discriminative model, minimize the loss value of the adversarial generative network, and train the generative model. By alternately training the generative model and the discriminative model, the pose coefficient generative model in the optimized generative model is obtained, which specifically includes:

[0061] 5.1) Define the adversarial generative network generation model, denoted as G, which includes the posture coefficient generation model and the generation loss module. Specifically:

[0062] 5.1.1) The posture coefficient generation model consists of the encoder and decoder of the conditional variational autoencoder;

[0063] 5.1.2) Using the video-pose coefficient model in step 2, extract the true pose coefficient sequence from the preprocessed video, denoted as ρ′ {1,...,t} The first frame of the preprocessed video is denoted as I;

[0064] 5.1.3) The encoder in the conditional variational autoencoder is denoted as Encoder. The encoder inputs the true posture coefficient sequence ρ′ {1,...,t} , the corresponding audio feature sequence a {1,...t} And the first frame image I, the encoder extracts features from the input data through a multi-layer neural network, mapping the high-dimensional input data to a mean vector μ and a variance vector σ2 In the Gaussian distribution of , it is recorded as the generation probability distribution, which can be expressed by the following formula (2);

[0065] N(μ,σ 2 )=Encoder(a {1,...t} ,I,ρ′ {1,...,t} ) (2)

[0066] This formula represents the mapping of the input data of the encoder in the conditional variational autoencoder to the mean vector μ and the variance vector σ 2 In the Gaussian distribution of

[0067] Where N(μ,σ 2 ) indicates that the mean vector is μ and the variance vector is σ 2 Gaussian distribution of

[0068] where ρ′ {1,...,t} Represents the true posture coefficient sequence;

[0069] where a {1,...t} Represents the preprocessed video and audio feature sequence;

[0070] Where I represents the first frame of the preprocessed video;

[0071] 5.1.3) The decoder in the conditional variational autoencoder, denoted as Decoder, samples the latent variable z from the Gaussian distribution, inputs the latent variable z and the first frame of the conditional information I into the decoder composed of a multi-layer convolutional neural network to obtain the generated posture coefficient sequence, denoted as ρ {1,...t} ; It is expressed by the following formula (3) and formula (4):

[0072] z~N(μ,σ 2 ) (3)

[0073] ρ {1,...t} =Decoder(z,I) (4)

[0074] Formula (3) represents the equation from the mean vector μ and the variance vector σ 2 The latent variable z is sampled from the Gaussian distribution of

[0075] Formula (4) represents the decoder in the conditional variational autoencoder inputting the latent variable z and the first frame image I to obtain the generated posture coefficient sequence ρ {1,...t} ;

[0076] 5.1.4) The generation loss function of the adversarial generative network generation model is denoted as Loss gen , including KL divergence loss function and mean square error loss function;

[0077] 5.1.4.1) Use KL divergence to measure the difference between the probability distribution generated in 5.1.3) and the standard normal distribution to encourage the distribution of the encoder output to be consistent with the standard normal distribution; denoted as Loss KL , the formula of KL divergence loss function is expressed as follows (5):

[0078]

[0079] This formula represents the use of the relative entropy function (i.e., KL divergence loss function) to calculate the generated probability distribution N(μ,σ 2 ) and the standard normal distribution N(0,1);

[0080] Where N(μ,σ 2 ) represents the generated probability distribution;

[0081] Where N(0,1) is the standard normal distribution;

[0082] Where log represents the logarithmic function;

[0083] Where μ represents the mean, σ represents the standard deviation, and σ 2 represents the variance, i represents the index of the training data, represents the variance of the ith data point, represents the square of the mean of the i-th data point;

[0084] Where k represents the number of training data;

[0085] 5.1.4.2) Use the mean square error loss to measure the difference between the posture coefficients predicted by the model and the actual posture coefficients, so as to make the posture coefficients generated by the model as similar as possible to the actual posture coefficients; denoted as Loss MSE ; The formula is expressed by the following formula (6):

[0086]

[0087] This formula represents the use of the mean square error loss function to calculate the predicted posture coefficient ρ {1,...t} and the true attitude coefficient ρ′ {1,...,t} The difference between

[0088] where ρ′ i Represents the true attitude coefficient sequence ρ′ {1,...,t} , ρ i Represents the predicted posture coefficient sequence ρ {1,...t} ;

[0089] Where N represents the number of training data;

[0090] 5.1.5) Generative Loss Function of Adversarial Generative Network Generative Model gen The formula is as follows:

[0091] Loss gen = Loss KL +Loss MSE (7)

[0092] This formula represents the generation loss function Loss of the adversarial generation network generation model gen By the relative entropy function Loss KL And the mean square error loss function Loss MSE composition;

[0093] 5.2) Define the discriminative model of the adversarial generative network. The discriminative model includes a discriminator and a discriminative loss module, where the discriminator is denoted as D;

[0094] The discriminator is composed of multi-layer convolutional neural networks and sigmoid functions, and the discriminant loss module is a binary cross loss function. The real posture coefficient sequence is input to the discriminator to obtain the probability value of the true value; the generated posture coefficient sequence is input to the discriminator to obtain the probability value of the false value; the probability value of the true value and the probability value of the false value are input to the binary cross loss function to obtain the discriminant loss value, recorded as Loss GAN ; The formula is expressed by the following formula (8):

[0095]

[0096] This formula represents the binary cross loss function, which calculates the difference between the output and the actual label;

[0097] Where D represents the discriminator;

[0098] where y i Represents the true label of the i-th sample, which takes a value of 0 or 1. It is 0 when the generated posture coefficient sequence is input, and 1 when the true posture coefficient sequence is input;

[0099] where ρ′ i That is, the predicted attitude coefficient sequence ρ′ {1,...,t} , ρ i That is, the true attitude coefficient sequence ρ {1,...t} ;

[0100] Where N represents the number of training data;

[0101] 5.3) The loss value of the adversarial generation network is recorded as loss_pose, which is given by Loss KL 、Loss MSE 、Loss GAN Composition: The formula is as follows:

[0102]

[0103] This formula represents the composition of the loss function of the adversarial generation network;

[0104] in Indicates optimizing the discriminant network by maximizing loss_pose;

[0105] in Indicates optimizing the generated network by minimizing loss_pose;

[0106] 5.4) By alternately training the generator and the discriminator, the conditional variational autoencoder in the final optimized generation model is obtained as the posture coefficient generation model.

[0107] First, fix the generative model, maximize loss_pose, and train the discriminative model; then fix the discriminative model, minimize loss_pose, and train the generative model. By alternately training the generative model and the discriminator, the final optimized conditional variational autoencoder is obtained as the pose coefficient generative model.

[0108] In step 6, the required audio and portrait are input, the audio is input into the expression coefficient generation model in step 4 to obtain an audio expression coefficient sequence; the audio is input into the posture coefficient generation model in step 5 to obtain an audio posture coefficient sequence; the audio is input into the audio-gesture model in step 3 to generate a gesture sequence; the obtained audio expression coefficient sequence, audio posture coefficient sequence, gesture sequence and portrait are used to obtain a facial expression, posture and gesture synchronized 3D portrait model by using a 3D rendering model, the 3D portrait model is converted into a 2D image by 3D to 2D mapping, the 2D image is generated by an image enhancement model, and a series of generated images constitute a digital human video, specifically including:

[0109] 6.1) Prepare the required audio and portrait;

[0110] 6.2) The audio generates a 3DMM expression coefficient sequence through the expression coefficient generation model;

[0111] 6.3) The audio generates a 3DMM attitude coefficient sequence through an attitude coefficient generation model;

[0112] 6.4) Audio generates gesture sequences through audio-gesture model;

[0113] 6.5) Rendering the expression coefficient sequence, posture coefficient sequence, gesture sequence and portrait onto the three-dimensional portrait model using a three-dimensional rendering model to obtain a three-dimensional portrait model with synchronized facial expressions, postures and gestures;

[0114] 6.6) Mapping the portrait in the three-dimensional portrait model into a two-dimensional image as a video frame;

[0115] 6.7) Use the pre-trained image enhancement model GFPGAN to enhance the video frames, improve the clarity and quality of the video frames, and make the processed faces clearer and more natural while retaining the identity features;

[0116] 6.8) Use cv2.VideoWriter to write the video frames into the video to get the video;

[0117] 6.9) Use ffmpeg tool to fuse audio and video to get digital human video;.

[0118] Compared with the prior art, the present invention has the following advantages:

[0119] The present invention uses a three-dimensional expression coefficient as an intermediate value to synchronously render the expression features, head posture features, and gesture features in speech onto a three-dimensional portrait, integrates the automatic generation of speech to facial expression coefficients, speech to head posture motion coefficients, and speech to gestures, and renders a three-dimensional portrait into a 2D image as a video frame. At the same time, a video frame enhancement model is introduced to finally synthesize a digital human video with higher accuracy, more naturalness, and clarity, which is synchronized with speech and has identity consistency. Specifically:

[0120] 1. Propose an intelligent method for generating facial expression coefficients from speech. Use a pre-trained video-expression coefficient model to extract real expression coefficient sequences from training data. Generate expression coefficient sequences from speech through a designed expression coefficient generation model. Compare the real expression coefficient sequence with the generated expression coefficient sequence, calculate the loss, and optimize the iterative expression coefficient generation model. In this way, highly realistic facial synchronization can be achieved, so that the expression of the digital human is synchronized with the input speech, improving the realism and naturalness of the interactive experience.

[0121] 2. Propose an intelligent generation method for the speech-to-head posture coefficient generation model. Use the pre-trained video-posture coefficient model to extract the real posture coefficient sequence from the training data. Use the designed posture coefficient generation model to generate posture coefficients from the speech and the first frame of the portrait. Compare the real posture coefficients with the generated posture coefficients, calculate the loss, and optimize the iterative posture coefficient generation model. In this way, using the first frame of the portrait as the input for prediction can achieve the consistency of the person's identity; while ensuring that the head movement of the digital person is synchronized with the input speech, it increases the diversity of head movement and improves the realism and naturalness of the interactive experience.

[0122] 3. A method is proposed to synchronously render the facial expression features, head movement features, and gesture features in the speech onto a three-dimensional portrait model to generate a synchronized digital human video. In this way, the generated video contains facial expressions, head movement postures, and gestures. At the same time, the face is clearer and more natural, and the consistency of the character's identity is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0123] Figure 1 The present invention is a schematic diagram of a model training process of a method for generating a digital human video with synchronized expressions and gestures of a three-dimensional portrait of a digital human in a voice-driven photo.

[0124] Figure 2 The present invention is a schematic diagram of a digital human video generation process of a method for generating digital human videos with synchronization of expressions and gestures of a three-dimensional portrait of a digital human in a voice-driven manner.

[0125] Figure 3 A schematic diagram showing a comparison between video frames generated on the market and video frames generated by the present invention in a method for generating digital human videos with synchronization of expressions and gestures of three-dimensional portraits of digital humans in voice-driven photos of the present invention. DETAILED DESCRIPTION

[0126] The following further describes the method for generating a digital human video with synchronized expressions and gestures of a three-dimensional digital human portrait driven by voice in a photo with reference to the accompanying drawings.

[0127] like Figure 1 As shown in the figure, the process of training the expression coefficient generation model and the posture coefficient generation model is: first collect the character broadcast video as training data, extract the audio from the training data, use the designed expression coefficient generation model to generate the expression coefficient sequence, compare it with the real expression coefficient sequence, calculate the loss, and train and optimize to obtain the expression coefficient generation model; use the designed adversarial generation network to alternately train the generation model and the discriminant model to obtain the optimized posture coefficient generation model, which specifically includes the following steps:

[0128] Step 1: Collect anchor broadcast videos, and pre-process the anchor broadcast videos to obtain pre-processed videos;

[0129] Step 2: Extract the audio file from the preprocessed video obtained in step 1;

[0130] Step 3: Prepare pre-trained models, including video-expression coefficient model, video-gesture coefficient model, audio-gesture model, and image enhancement model.

[0131] Step 4: The expression coefficient generation model includes an audio encoder composed of a residual network and an output layer composed of a linear network; the video preprocessed in step 1 is used to obtain the real expression coefficient sequence of the video through the video-expression coefficient model in step 3, and the audio file extracted in step 2 is sequentially passed through the audio encoder and output layer of the expression coefficient generation model to obtain the generated expression coefficient sequence; the loss value of the real expression coefficient sequence and the generated expression coefficient sequence is obtained through the mean square error loss function, and the expression coefficient generation model is obtained by minimizing the loss value and iteratively optimizing;

[0132] Step 5: define a generative adversarial network, including a generative model and a discriminative model, wherein the generative model includes a posture coefficient generative model and a generative loss module connected in series, and the discriminative model includes a discriminator and a discriminative loss module connected in series;

[0133] The video preprocessed in step 1 is used to obtain the true posture coefficient sequence through the video-posture coefficient model in step 3. The video preprocessed in step 1 is used to generate the posture coefficient sequence through the posture coefficient generation model of the generation model to obtain the generated posture coefficient sequence. The true posture coefficient sequence and the generated posture coefficient sequence are used to obtain the generated loss value through the KL divergence loss function and the mean square error loss function in the generation loss module.

[0134] The probability value of the true value is obtained by the discriminator for the real posture coefficient sequence, and the probability value of the false value is obtained by the discriminator for the generated posture coefficient sequence. The discriminant loss module obtains the discriminant loss value by comparing the difference between the probability value of the false value and the value 0 and the difference between the probability value of the true value and the value 1. The generated loss value and the discriminant loss value constitute the loss value of the adversarial generation network;

[0135] First, the generative model is fixed, the loss value of the adversarial generative network is maximized, and the discriminative model is trained; then the discriminative model is fixed, the loss value of the adversarial generative network is minimized, and the generative model is trained. The pose coefficient generative model in the optimized generative model is obtained by alternately training the generative model and the discriminative model;

[0136] Step 6: Input the required audio and portrait, input the audio into the expression coefficient generation model in step 4 to obtain an audio expression coefficient sequence; input the audio into the posture coefficient generation model in step 5 to obtain an audio posture coefficient sequence; input the audio into the audio-gesture model in step 3 to generate a gesture sequence; obtain the audio expression coefficient sequence, audio posture coefficient sequence, gesture sequence and portrait by using a three-dimensional rendering model to obtain a three-dimensional portrait model with facial expressions, postures and gestures synchronization, obtain a two-dimensional image by three-dimensional to two-dimensional mapping of the three-dimensional portrait model, obtain a generated image by the image enhancement model, and a series of generated images constitute a digital human video.

[0137] In step 1, the anchor broadcast video is collected and preprocessed to obtain the preprocessed video, which specifically includes:

[0138] The frame number and audio sampling rate of the anchor's broadcast video are adjusted to obtain a preprocessed video.

[0139] In step 2, the audio file is extracted from the preprocessed video obtained in step 1, including:

[0140] Use ffmpeg tool to batch extract audio files from pre-processed videos;

[0141] In step 3, prepare the pre-trained model, including the video-expression coefficient model, the video-posture coefficient model, the audio-gesture model, and the image enhancement model. Specifically:

[0142] 3.1) Video-expression coefficient model uses 3D Morphable Models (3D Morphable Models,

[0143] 3DMM), extracting 3DMM facial expression coefficients from the input portrait video;

[0144] 3.2) Video-Gesture Coefficient Model Using 3D Morphable Models

[0145] 3DMM), extract 3DMM head motion coefficients by inputting portrait video;

[0146] 3.3) Audio-gesture model uses voice-driven gesture template model to obtain gesture

[0147] Potential sequence

[0148] 3.4) The image enhancement model uses GFPGAN to improve the resolution of the input image and output a high-resolution image;

[0149] In step 4, the expression coefficient generation model includes an audio encoder composed of a residual network and an output layer composed of a linear network; the video preprocessed in step 1 is used to obtain the real expression coefficient sequence of the video through the video-expression coefficient model in step 3, and the audio file extracted in step 2 is sequentially passed through the audio encoder and output layer of the expression coefficient generation model to obtain the generated expression coefficient sequence; the loss value of the real expression coefficient sequence and the generated expression coefficient sequence is obtained through the mean square error loss function, and the expression coefficient generation model is obtained by minimizing the loss value and iteratively optimizing, specifically:

[0150] 4.1) The expression coefficient generation model includes an audio encoder composed of a residual network and an output layer composed of a linear network; the audio encoder is denoted as Φ A ; The output layer is denoted as Φ M;

[0151] 4.2) The video preprocessed in step 1 is used to obtain the true expression coefficient sequence through the video-expression coefficient model in step 2, denoted as β′ {1,...,t} ;

[0152] 4.3) Obtain the 0.2s Mel spectrum feature sequence from the audio file extracted in step 2, also called the audio feature sequence, denoted as a {1,...t} , specifically:

[0153] 4.3.1) Load the audio file and obtain the audio signal and sampling rate;

[0154] 4.3.2) Perform pre-emphasis processing on the audio signal to highlight the high frequency part;

[0155] 4.3.3) The audio signal is divided into frames using a Hamming window, and each frame is subjected to a fast Fourier transform to obtain a frequency domain representation. The frequency domain representation is obtained by calculating the square sum of the real part and the imaginary part to obtain a power spectrum;

[0156] 4.3.4) Use the mel function in the audio analysis tool librosa library to create a Mel filter bank;

[0157] 4.3.5) Apply the Mel filter bank to the power spectrum to obtain a Mel spectrum graph, take the logarithm of each element of the Mel spectrum graph to obtain a Mel spectrum feature; intercept the Mel spectrum feature corresponding to 0.2s as the Mel spectrum feature sequence;

[0158] 4.4) Use the audio encoder to encode the audio feature sequence into the latent space to obtain the audio coding feature; the audio coding feature is denoted as aud emb ;

[0159] 4.5) Input aud to the output layer of the linear network emb The expression coefficient is denoted as β {1,...,t} ; The network formula is as follows:

[0160] β {1,...,t} =φ M (φ A (α {1,...,t} )) (1)

[0161] This formula represents the audio feature sequence a {1,...t} Through the audio encoder Φ A and the output layer Φ M Get the expression coefficient β {1,...,t} ;

[0162] 4.5) Calculate the true expression coefficient β′ {1,...,t} and the generated expression coefficient β {1,...,t}The mean square error between is used as the loss function, denoted as loss_exp, and the formula is as follows.

[0163]

[0164] This formula represents the loss between the generated expression coefficient and the true expression coefficient calculated using the mean square error function;

[0165] where β′ i Represents the true expression coefficient β′ {1,...,t} ;

[0166] where β i Represents the generated expression coefficient β {1,...,t} ;

[0167] Where N represents the number of training data;

[0168] 4.6) The expression coefficient generation model is obtained by minimizing loss_exp, back propagation, and iterative optimization.

[0169] In step 5, a generative adversarial network is defined, including a generative model and a discriminative model, wherein the generative model includes a posture coefficient generative model and a generative loss module connected in series, and the discriminative model includes a discriminator and a discriminative loss module connected in series;

[0170] The video preprocessed in step 1 is used to obtain the true posture coefficient sequence through the video-posture coefficient model in step 3. The video preprocessed in step 1 is used to generate the posture coefficient sequence through the posture coefficient generation model of the generation model to obtain the generated posture coefficient sequence. The true posture coefficient sequence and the generated posture coefficient sequence are used to obtain the generated loss value through the KL divergence loss function and the mean square error loss function in the generation loss module.

[0171] The probability value of the true value is obtained by the discriminator for the real posture coefficient sequence, and the probability value of the false value is obtained by the discriminator for the generated posture coefficient sequence. The discriminant loss module obtains the discriminant loss value by comparing the difference between the probability value of the false value and the value 0 and the difference between the probability value of the true value and the value 1. The generated loss value and the discriminant loss value constitute the loss value of the adversarial generation network;

[0172] First, fix the generative model, maximize the loss value of the adversarial generative network, and train the discriminative model; then fix the discriminative model, minimize the loss value of the adversarial generative network, and train the generative model. By alternately training the generative model and the discriminative model, the pose coefficient generative model in the optimized generative model is obtained, which specifically includes:

[0173] 5.1) Define the adversarial generative network generation model, denoted as G, which includes the posture coefficient generation model and the generation loss module. Specifically:

[0174] 5.1.1) The posture coefficient generation model consists of the encoder and decoder of the conditional variational autoencoder;

[0175] 5.1.2) Using the video-pose coefficient model in step 2, extract the true pose coefficient sequence from the preprocessed video, denoted as ρ′ {1,...,t} The first frame of the preprocessed video is denoted as I;

[0176] 5.1.3) The encoder in the conditional variational autoencoder is denoted as Encoder. The encoder inputs the true posture coefficient sequence ρ′ {1,...,t} , the corresponding audio feature sequence a {1,...t} And the first frame image I, the encoder extracts features from the input data through a multi-layer neural network, mapping the high-dimensional input data to a mean vector μ and a variance vector σ 2 In the Gaussian distribution of , it is recorded as the generation probability distribution, which can be expressed by the following formula (2);

[0177] N(μ,σ 2 )=Encoder(a {1,...t} ,I,ρ′ {1,...,t} ) (2)

[0178] This formula represents the mapping of the input data of the encoder in the conditional variational autoencoder to the mean vector μ and the variance vector σ 2 In the Gaussian distribution of

[0179] Where N(μ,σ 2 ) indicates that the mean vector is μ and the variance vector is σ 2 Gaussian distribution of

[0180] where ρ′ {1,...,t} Represents the true posture coefficient sequence;

[0181] where a {1,...t} Represents the preprocessed video and audio feature sequence;

[0182] Where I represents the first frame of the preprocessed video;

[0183] 5.1.3) The decoder in the conditional variational autoencoder, denoted as Decoder, samples the latent variable z from the Gaussian distribution, inputs the latent variable z and the first frame of the conditional information I into the decoder composed of a multi-layer convolutional neural network to obtain the generated posture coefficient sequence, denoted as ρ {1,...t} ; It is expressed by the following formula (3) and formula (4):

[0184] z~N(μ,σ 2 )(3)

[0185] ρ{1,...t} =Decoder(z,I) (4)

[0186] Formula (3) represents the equation from the mean vector μ and the variance vector σ 2 The latent variable z is sampled from the Gaussian distribution of

[0187] Formula (4) represents the decoder in the conditional variational autoencoder inputting the latent variable z and the first frame image I to obtain the generated posture coefficient sequence ρ {1,...t} ;

[0188] 5.1.4) The generation loss function of the adversarial generative network generation model is denoted as Loss gen , including KL divergence loss function and mean square error loss function;

[0189] 5.1.4.1) Use KL divergence to measure the difference between the probability distribution generated in 5.1.3) and the standard normal distribution to encourage the distribution of the encoder output to be consistent with the standard normal distribution; denoted as Loss KL , the formula of KL divergence loss function is expressed as follows (5):

[0190]

[0191] This formula represents the use of the relative entropy function (i.e., KL divergence loss function) to calculate the generated probability distribution N(μ,σ 2 ) and the standard normal distribution N(0,1);

[0192] Where N(μ,σ 2 ) represents the generated probability distribution;

[0193] Where N(0,1) is the standard normal distribution;

[0194] Where log represents the logarithmic function;

[0195] Where μ represents the mean, σ represents the standard deviation, and σ 2 represents variance;

[0196] Where k represents the number of training data;

[0197] 5.1.4.2) Use the mean square error loss to measure the difference between the posture coefficients predicted by the model and the actual posture coefficients, so as to make the posture coefficients generated by the model as similar as possible to the actual posture coefficients; denoted as Loss MSE ; The formula is expressed by the following formula (6):

[0198]

[0199] This formula represents the use of the mean square error loss function to calculate the predicted posture coefficient ρ {1,...t} and the true attitude coefficient ρ′ {1,...,t} The difference between

[0200] where ρ′ i Represents the true attitude coefficient sequence ρ′ {1,...,t} , ρ i Represents the predicted posture coefficient sequence ρ {1,...t} ;

[0201] Where N represents the number of training data;

[0202] 5.1.5) Generative Loss Function of Adversarial Generative Network Generative Model gen The formula is as follows:

[0203] Loss gen =Loss KL +Loss MSE (7)

[0204] This formula represents the generation loss function Loss of the adversarial generation network generation model gen By the relative entropy function Loss KL And the mean square error loss function Loss MSE composition;

[0205] 5.2) Define the discriminative model of the adversarial generative network. The discriminative model includes a discriminator and a discriminative loss module, where the discriminator is denoted as D;

[0206] The discriminator is composed of multi-layer convolutional neural networks and sigmoid functions, and the discriminant loss module is a binary cross loss function. The real posture coefficient sequence is input to the discriminator to obtain the probability value of the true value; the generated posture coefficient sequence is input to the discriminator to obtain the probability value of the false value; the probability value of the true value and the probability value of the false value are input to the binary cross loss function to obtain the discriminant loss value, recorded as Loss GAN ; The formula is expressed by the following formula (7):

[0207]

[0208] This formula represents the binary cross loss function, which calculates the difference between the output and the actual label;

[0209] Where D represents the discriminator;

[0210] where y i Represents the true label of the i-th sample, which takes a value of 0 or 1. It is 0 when the generated posture coefficient sequence is input, and 1 when the true posture coefficient sequence is input;

[0211] where ρ′i That is, the predicted attitude coefficient sequence ρ′ {1,...,t} , ρ i That is, the true attitude coefficient sequence ρ {1,...t} ;

[0212] Where N represents the number of training data;

[0213] 5.3) The loss value of the adversarial generation network is recorded as loss_pose, which is given by Loss KL 、Loss MSE 、Loss GAN Composition: The formula is as follows:

[0214]

[0215] This formula represents the composition of the loss function of the adversarial generation network;

[0216] in Indicates optimizing the discriminant network by maximizing loss_pose;

[0217] in Indicates optimizing the generated network by minimizing loss_pose;

[0218] 5.4) By alternately training the generator and the discriminator, the conditional variational autoencoder in the final optimized generation model is obtained as the posture coefficient generation model.

[0219] First, fix the generative model, maximize loss_pose, and train the discriminative model; then fix the discriminative model, minimize loss_pose, and train the generative model. By alternately training the generative model and the discriminator, the final optimized conditional variational autoencoder is obtained as the pose coefficient generative model.

[0220] In step 6, if Figure 2 As shown, the required audio and portrait are input, the audio is input into the expression coefficient generation model in step 4 to obtain an audio expression coefficient sequence; the audio is input into the posture coefficient generation model in step 5 to obtain an audio posture coefficient sequence; the audio is input into the audio-gesture model in step 3 to generate a gesture sequence; the obtained audio expression coefficient sequence, audio posture coefficient sequence, gesture sequence and portrait are obtained by using a three-dimensional rendering model to obtain a facial expression, posture and gesture synchronized three-dimensional portrait model, the three-dimensional portrait model is converted from three-dimensional to two-dimensional mapping to obtain a two-dimensional image, the two-dimensional image is obtained by an image enhancement model to obtain a generated image, and a series of generated images constitute a digital human video, specifically including:

[0221] 6.1) Prepare the required audio and portrait;

[0222] 6.2) The audio generates a 3DMM expression coefficient sequence through the expression coefficient generation model;

[0223] 6.3) The audio generates a 3DMM attitude coefficient sequence through an attitude coefficient generation model;

[0224] 6.4) Audio generates gesture sequences through audio-gesture model;

[0225] 6.5) Rendering the expression coefficient sequence, posture coefficient sequence, gesture sequence and portrait onto the three-dimensional portrait model using a three-dimensional rendering model to obtain a three-dimensional portrait model with synchronized facial expressions, postures and gestures;

[0226] 6.6) Mapping the portrait in the three-dimensional portrait model into a two-dimensional image as a video frame;

[0227] 6.7) Use the pre-trained image enhancement model GFPGAN to enhance the video frames, improve the clarity and quality of the video frames, and make the processed faces clearer and more natural while retaining the identity features;

[0228] 6.8) Use cv2.VideoWriter to write the video frames into the video to get the video;

[0229] 6.9) Use ffmpeg tool to merge audio and video to get digital human video;

[0230] like Figure 3 As shown, the digital human video contains gestures and the video frames are clearer;.

[0231] The above is only an illustration of the present invention, and is not intended to limit the present invention. A person skilled in the art should recognize that any changes or modifications made to the present invention will fall within the protection scope of the present invention.

Claims

1. A method for generating a digital human video with synchronized expressions and gestures of a three-dimensional portrait of a digital human in a voice-driven photo, characterized in that: The following steps are involved: Step 1: Collect anchor broadcast videos, and pre-process the anchor broadcast videos to obtain pre-processed videos; Step 2: Extract the audio file from the preprocessed video obtained in step 1; Step 3: Use video-expression coefficient model, video-gesture coefficient model, audio-gesture model, and image enhancement model; Step 4: For the video preprocessed in step 1, the real expression coefficient sequence of the video is obtained through the video-expression coefficient model in step 3, and the expression coefficient generation model is used to generate the expression coefficient sequence for the audio file extracted in step 2; The loss value is obtained by using the mean square error loss function for the real expression coefficient sequence and the generated expression coefficient sequence. The optimized expression coefficient generation model is obtained by minimizing the loss value and iteratively optimizing. Step 5: adversarial generative network, including a generative model and a discriminative model, wherein the generative model includes a posture coefficient generative model and a generative loss module connected in series, and the discriminative model includes a discriminator and a discriminative loss module connected in series; Train the generative model and the discriminative model to obtain an optimized posture coefficient generative model; Step 6: Input the required audio and portrait, input the audio into the optimized expression coefficient generation model in step 4 to obtain an audio expression coefficient sequence; input the audio into the optimized posture coefficient generation model in step 5 to obtain an audio posture coefficient sequence; input the audio into the audio-gesture model in step 3 to generate a gesture sequence; obtain the obtained audio expression coefficient sequence, audio posture coefficient sequence, gesture sequence and input portrait by using a three-dimensional rendering model to obtain a three-dimensional portrait model with facial expression, posture and gesture synchronization, obtain a two-dimensional image by three-dimensional to two-dimensional mapping of the three-dimensional portrait model, obtain a generated image by the image enhancement model, and a series of generated images constitute a digital human video.

2. The method for generating a digital human video with synchronized expressions and gestures of a three-dimensional portrait of a digital human in a voice-driven photo according to claim 1, characterized in that: In step 1, the anchor broadcast video is collected and preprocessed to obtain the preprocessed video, which specifically includes: The frame number and audio sampling rate of the anchor's broadcast video are adjusted to obtain a preprocessed video.

3. The method for generating a digital human video with synchronized expressions and gestures of a three-dimensional portrait of a digital human in a voice-driven photo according to claim 1, characterized in that: In step 2, the audio file is extracted from the preprocessed video obtained in step 1, including: Use the ffmpeg tool to batch extract audio files from the preprocessed video.

4. The method for generating a digital human video with synchronized expressions and gestures of a three-dimensional portrait of a digital human in a voice-driven photo according to claim 1, characterized in that: In step 4, the expression coefficient generation model includes an audio encoder composed of a residual network and an output layer composed of a linear network.

5. The method for generating a digital human video with synchronized expressions and gestures of a three-dimensional portrait of a digital human in a voice-driven photo according to claim 1, characterized in that: In step 5, the generative model and the discriminative model are trained to obtain an optimized posture coefficient generative model, including: For the video preprocessed in step 1, the real posture coefficient sequence is obtained through the video-posture coefficient model in step 3. For the video preprocessed in step 1, the generated posture coefficient sequence is obtained through the generation loss module of the adversarial generation network. For the real posture coefficient sequence and the generated posture coefficient sequence, the relative entropy function and the mean square error loss function in the generation loss module of the adversarial generation network are used to obtain the generation loss value; The probability value of the true value is obtained by the discriminator of the adversarial generation network for the real posture coefficient sequence, and the probability value of the false value is obtained by the discriminator for the generated posture coefficient sequence. The discriminant loss module of the adversarial generation network obtains the discriminant loss value by comparing the difference between the probability value of the false value and the value 0 and the difference between the probability value of the true value and the value 1. The generated loss value and the discriminant loss value constitute the loss value of the adversarial generation network; First, the generative model of the adversarial generative network is fixed, the loss value of the adversarial generative network is maximized, and the discriminative model is trained; then the discriminative model of the anti-generative network is fixed, the loss value of the adversarial generative network is minimized, and the generative model is trained. By alternately training the generative model and the discriminative model, the optimized posture coefficient generative model is obtained.

6. The method for generating a digital human video with synchronized expressions and gestures of a three-dimensional portrait of a digital human in a voice-driven photo according to claim 5, characterized in that: In step 5, the generated loss value is calculated by the following formula: Loss gen = Loss KL +Loss MSE (7) Among them, N(μ,σ 2 ) represents the generated probability distribution, N(0,1) is the standard normal distribution, log represents the logarithmic function, μ represents the mean, σ represents the standard deviation, σ 2 represents the variance, i represents the index of the training data, represents the variance of the ith data point, represents the square of the mean of the i-th data point, k represents the number of training data, ρ′ i represents the true posture coefficient sequence, ρ i represents the predicted posture coefficient sequence, N represents the number of training data, and Loss gen Represents the generation loss function of the adversarial generation network generation model, Loss KL Represents the relative entropy function, Loss MSE Represents the mean square error loss function Loss MSE .

Images