A method for generating a talking digital human based on a large model learnable text latent code

Abstract

The application discloses a method for generating a talking digital human based on a learnable text latent code of a large model, acquires a plurality of face voice videos according to actual needs, extracts training samples to form a training sample set, each training sample comprising a reference video and a corresponding mel spectrum graph sequence, and a mouth reference image; a general talking digital human generation model based on the learnable text latent code is constructed, and the training sample set is used for training; when a talking digital human video needs to be generated, the voice data to be generated is first framed and the mel spectrum graph sequence is extracted, the mouth reference image and the reference video are set according to actual conditions, and the trained general talking digital human generation model is inputted together to generate the talking digital human video corresponding to the voice data to be generated. The application continuously learns a stable text latent code from the mouth reference images of different speakers, which is used to guide the synthesis process of the general talking digital human video, and the quality of the talking digital human generation model is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image generation, and more specifically relates to a method for generating a talking digital human based on a learnable text latent code of a large model. BACKGROUND

[0002] The generation of talking digital humans aims to map acoustic features to time-aligned facial movements. Currently, this method can be divided into two categories: specific talking digital human generation methods for individuals and general talking digital human generation methods for anyone. Some methods can generate high-fidelity face videos for specific individuals, and these methods mostly rely on neural radiance fields (NeRF) and three-dimensional morphable models (3DMM). In contrast, general talking digital human generation methods can use a few seconds of video clips or a single static face image to match any speech content. In general talking digital human generation methods, there are three aspects closely related to face generation quality, namely visual quality (VQ), lip sync (LS), and reading intelligibility (RI).

[0003] In recent years, large models have been widely used in various research fields. CLIP is a large visual language pre-training model that is very effective for visual tasks using text representation. This powerful model has been proven effective in many tasks, such as domain transfer, 3D digital humans, and image editing. Multimodal features also facilitate robust feature learning. In the TFG field, there are currently only a few works that attempt to use the CLIP model to extract emotional features to increase the expressiveness of talking digital humans.

[0004] However, current general talking digital human methods have difficulty in maintaining good visual quality while improving reading intelligibility and lip sync, and the features learned from limited speech face data are difficult to adapt to different speakers' lip shapes, which hinders the performance of general talking digital human generation methods. Lip sync and visual quality have become two major challenges faced by talking digital human generation.

[0005] To solve the problem of lip consistency, some techniques use a mutual information estimator to alleviate the problem of lip synchronization difference. Computing the synchronization loss between the visual lip features of the real face and the output video sequence is also a technique to improve lip consistency. In addition to using auxiliary models to assist in controlling lip consistency, encoder-decoder structures can effectively improve the performance of speech and video consistency. In addition to lip consistency, poor visual quality (i.e., blur and untruthfulness) can also affect the performance of the generated speaker video. This problem can be alleviated by using a skip connection and a generative adversarial network. The generative adversarial network can generate realistic speaker avatars while maintaining the facial features of the target person. In addition to lip consistency and visual quality, reading clarity is also indispensable in speaker digital human generation. It reflects how much text can be translated from the face video of the speaker through lip reading technology. Adjusting the lip shape to match the corresponding speech content can improve reading clarity, and using a lip synchronization discriminator and contrastive learning can improve lip consistency. However, these methods can cause unexpected changes in the fine-grained mouth region, resulting in poor visual quality performance. Current techniques can only improve reading clarity and lip consistency, or only improve visual quality, making it difficult to improve the performance of all three aspects simultaneously. SUMMARY

[0006] The purpose of the present application is to overcome the shortcomings of the prior art and provide a speaker digital human generation method based on a large model learnable text latent code. A general speaker digital human generation model is designed to continuously learn robust text latent codes from different speakers' mouth reference images to guide the synthesis process of general speaker digital human videos and improve the quality of speaker digital human generation models.

[0007] To achieve the above-mentioned application purpose, the speaker digital human generation method based on a large model learnable text latent code comprises the following steps:

[0008] S1: Obtain a plurality of face voice videos according to actual needs, extract voice data from each face voice video, then obtain face video data after removing the voice data, process the obtained plurality of pairs of voice data and face video data to obtain training samples, and the specific method is:

[0009] For each pair of voice data and face video data, extract the region of interest on each video frame in the face video data, normalize the region of interest of each video frame to a preset size to obtain the processed face video data V; select N frames of video frames from the face video data V, and select one as a mouth reference image M r Mask the lower part of the N frames of video frames to obtain a posture reference video P r; then select N video frames from the remaining video frames as identity reference frames to form an identity reference video I r ; concatenate the pose reference video P r and the identity reference video I r along the channel dimension to obtain a reference video F;

[0010] frame the voice data, and extract the mel spectrogram of each voice frame to form a mel spectrogram sequence S;

[0011] concatenate each reference video F with the corresponding mel spectrogram sequence S, mouth reference image M r , to form a training sample, thereby obtaining a training sample set;

[0012] S2: Construct a general speaking digital human generation model, including a voice encoder, a face encoder, an image encoder, a text encoder, a text feature learning module, a voice joint learning space module, a face joint learning space module, a feature splicing module, and a face generator, wherein:

[0013] The voice encoder is used for feature extraction on the mel spectrogram sequence S, and the feature vectors of the corresponding voice frames of the obtained pose reference video P r are filtered out to form a voice feature w s which is sent to the voice joint learning space module;

[0014] The face encoder is used for feature extraction on the concatenated video F containing 2N face video frames, and the obtained face feature w f is sent to the face joint learning space module;

[0015] The image encoder is used for feature extraction on the mouth reference image M r , and the obtained mouth image feature c m is sent to the text feature learning module;

[0016] The text encoder is used for feature extraction on the preset mouth text prompt T, and the obtained mouth text feature c t is sent to the text feature learning module;

[0017] The text feature learning module is used to extract a text latent code c according to the mouth image feature c m and the mouth text feature c t and send it to the voice joint learning space module and the face joint learning space module, respectively;

[0018] The voice joint learning space module is used to inject the text latent code c into the voice feature w s using the following formula to obtain an optimized voice feature and send it to the feature splicing module;

[0019] The face joint learning space module is configured to inject the text latent code c into the face feature w by using the following formula f to obtain the optimized face feature and send it to the feature splicing module.

[0020] The feature splicing module is configured to splice the optimized speech feature and the optimized face feature and then send the spliced feature W to the face generator.

[0021] The face generator is configured to generate N speaker face video frames according to the spliced feature W to form a speaker digital human video.

[0022] S3: using the training sample set obtained in step S1 to train the general speaker digital human generation model constructed in step S2 to obtain a trained general speaker digital human generation model.

[0023] S4: when a speaker digital human video needs to be generated, first frame the to-be-generated speech data and extract the mel spectrogram of each speech frame, divide the speech frame sequence into sub-sequences with a length of N to obtain a mel spectrogram sequence S' of each sub-sequence; set N face video frames according to actual conditions, and mask the lower part of the frames to obtain a posture reference frame to form a posture reference video P r ', and select N face video frames of the same person as identity reference frames to form an identity reference video I r '; input the mel spectrogram sequence S', the posture reference video P r ' and the identity reference video I r ' into the trained general speaker digital human generation model to generate a speaker digital human video with a length of N, and then splice to obtain a speaker digital human video corresponding to the to-be-generated speech data.

[0024] The speaker digital human generation method based on a large model that can learn a text latent code, according to actual needs, acquires a plurality of face speech videos and extracts training samples to form a training sample set, each training sample including a reference video and a corresponding mel spectrogram sequence and a mouth reference image; a general speaker digital human generation model based on a learnable text latent code is constructed, and the training sample set is used for training; when a speaker digital human video needs to be generated, first frame the to-be-generated speech data and extract the mel spectrogram sequence, set the mouth reference image and the reference video according to actual conditions, and input them into the trained general speaker digital human generation model to generate a speaker digital human video corresponding to the to-be-generated speech data.

[0025] The present application has the following advantages:

[0026] 1) This invention utilizes the shared semantics in the speech-visual multimodal space and the text-image multimodal space as a bridge. In the general speaking digital human generation model, a speech joint learning space module and a face joint learning space module are set up to integrate the speech and visual features obtained from highly aligned data pairs and the robust text and image features based on extensive data. This connects the robust feature space learned from large-scale data with the feature space of specific tasks, thereby improving the quality of speaking digital human videos.

[0027] 2) The general speaking digital human generation model of this invention uses a learnable text latent code for guidance, so as to use a unified text latent code to capture the dynamic changes in the oral cavity region caused by different speakers and pronunciation variations. Unlike previous methods that excelled in specific aspects (such as reading clarity, lip-sound synchronization or visual quality), this model not only further improves reading clarity and lip-sound consistency, but also improves visual quality. It can not only enable different speakers to produce lip shapes similar to the ground reality when pronouncing the same words, but also preserve the unique features of the oral cavity region.

[0028] 3) In training the general speaking digital human generation model, this invention also proposes an improved text feature learning strategy and loss function, thereby further improving the model performance. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating a specific implementation of the present invention's method for generating general speaking digital humans based on large-scale model-learnable text latent codes;

[0030] Figure 2 This is a structural diagram of the universal speaking digital human generation model in this invention;

[0031] Figure 3 This is a structural diagram of the joint learning space module in this embodiment;

[0032] Figure 4 This is a diagram showing the result of uniformly sampling two generated face dialogue videos generated using the present invention in this embodiment. Detailed Implementation

[0033] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0034] Example

[0035] Figure 1 This is a flowchart illustrating a specific implementation of the present invention's method for generating a general speaking digital human based on a large model and learnable text latent codes.Figure 1 As shown, the specific steps of the universal speaking digital human generation method based on learnable text latent codes of the present invention include:

[0036] S101: Obtain training samples:

[0037] To obtain several facial voice and video recordings as needed, extract voice data from each recording, then remove the voice data to obtain facial video data. Process these pairs of voice and video data to obtain training samples. The specific method is as follows:

[0038] For each pair of voice data and face video data, extract the region of interest (ROI) from each video frame of the face video data. Normalize the ROI of each video frame to a preset size to obtain the processed face video data V. Select N video frames from the face video data V, and choose one of them as the mouth reference image M. r The lower halves of these frames are masked and used as pose reference frames to form the pose reference video P. r Then, N video frames are selected from the remaining video frames as identity reference frames to form the identity reference video I. r Please refer to the posture video. r Identity Reference Video I r The reference video F is obtained by stitching along the channel dimension.

[0039] The speech data is segmented into frames, and the Mel spectrogram of each speech frame is extracted to form a Mel spectrogram sequence S. The specific parameters of the Mel spectrogram can be set according to actual needs. In this embodiment, the transition and window length are 125 milliseconds and 50 milliseconds, respectively.

[0040] Each reference video F and its corresponding Mel spectrogram sequence S, and mouth reference image M are used to... r This is used to construct a training sample, thus obtaining the training sample set.

[0041] S102: Constructing a general speaking digital human generation model:

[0042] To improve the visual quality of speaking digital human videos, this invention introduces learnable text latent codes into a general speaking digital human generation model to guide the generation of speaking digital human video frames. Figure 2 This is a structural diagram of the universal speaking digital human generation model in this invention. (For example...) Figure 2As shown, the general speaking digital human generation model in this invention includes a speech encoder, a face encoder, an image encoder, a text encoder, a text feature learning module, a speech joint learning space module, a face joint learning space module, a feature splicing module, a face generator, and a face discriminator. Each module will be described in detail below.

[0043] The speech encoder is used to extract features from the Mel spectrogram sequence S, and selects the pose reference video P from the obtained speech feature matrix. r The feature vectors of the corresponding speech frames constitute the speech features w s Send to the Voice Joint Learning Space module.

[0044] A face encoder is used to extract features from a stitched video F containing 2N individual face video frames, and the resulting face features w f Send to the Face Joint Learning Space module.

[0045] Image encoder used for mouth reference image M r Feature extraction is performed, and the resulting mouth image features c m The image is sent to the text feature learning module. In this embodiment, the image encoder uses the image encoder in the CLIP model.

[0046] The text encoder is used to extract features from a preset mouth text prompt T ("mouth" in this embodiment), and the resulting mouth text features c t The text is sent to the text feature learning module. In this embodiment, the text encoder uses the text encoder in the CLIP model.

[0047] The text feature learning module is used to learn from the features of the mouth image. m and mouth text features c t The text latent code c is extracted and sent to the speech joint learning space module and the face joint learning space module respectively.

[0048] like Figure 2 As shown, in this embodiment, the text feature learning module includes a mouth image feature noise addition module, a mouth text feature noise addition module, a first dot product module, a projection module, and a second dot product module, wherein:

[0049] The mouth image feature noise reduction module is used to add noise to mouth image features c m Add Gaussian noise G a Obtain the features of the mouth image with added noise. And send it to the first dot product module. In this embodiment, Gaussian noise G a Gaussian noise that conforms to a normal distribution is used.

[0050] The mouth text feature noise addition module is used to add noise to mouth text features ct Add Gaussian noise G a Obtain the text features of the mouth area with added noise. And send it to the first dot product module and the second dot product module.

[0051] The first dot product module is used to process the features of the noisy mouth image. Noisy mouth text features Perform a dot product operation to obtain the enhanced feature 'a' and send it to the projection module.

[0052] The projection module is used to employ a preset projection function f c () Perform a projection operation on the enhanced feature a to obtain the mouth weight λ and send it to the second dot product module.

[0053] The second dot product module is used to calculate the mouth weight λ and the noisy mouth text features. Performing a dot product operation yields the text latent code c. Introducing a mouth weight λ effectively enhances the features. It learns task-related details while suppressing task-irrelevant information, thus making the textual code more accurate.

[0054] The speech joint learning space module is used to inject the text latent code c into the speech feature w using the following formula. s Optimized speech features And send it to the feature splicing module.

[0055] The face joint learning space module is used to inject the text latent code c into the face feature w using the following formula. f Optimized facial features And send it to the feature splicing module.

[0056] In this invention, a joint learning space module for speech and a joint learning space module for face are used to integrate speech-visual features obtained from highly aligned data pairs and robust text-image features based on extensive data. This allows for the use of a unified text latent code to capture dynamic changes in the oral cavity region caused by different speakers and pronunciation variations, thereby improving the quality of the generated speaking digital human video. In this embodiment, the joint learning space module for speech and the joint learning space module for face adopt the same joint learning space module structure. Figure 3 This is a structural diagram of the joint learning space module in this embodiment. For example... Figure 3 As shown, in this embodiment, the joint learning space module includes a first text latent code feature extraction module, a second text latent code feature extraction module, a third text latent code feature extraction module, a fourth text latent code feature extraction module, an input feature extraction module, a first feature multiplication module, a first feature summation module, a fusion feature extraction module, a second feature multiplication module, a second feature summation module, and a third feature summation module, wherein:

[0057] The first text latent code feature extraction module is used to extract features from the text latent code c, and send the obtained feature c1 to the first feature multiplication module.

[0058] The second text latent code feature extraction module is used to extract features from the text latent code c, and sends the obtained feature c2 to the first feature summation module.

[0059] The third text latent code feature extraction module is used to extract features from the text latent code c, and the obtained feature c3 is sent to the second feature multiplication module.

[0060] The fourth text latent code feature extraction module is used to extract features from the text latent code c, and sends the obtained feature c4 to the second feature summation module.

[0061] In this embodiment, the four text latent code feature extraction modules all adopt the same structure, which is obtained by cascading one-dimensional convolutional layers, layer normalization layers and Leaky ReLU layers.

[0062] The input feature extraction module is used to extract features from the input feature w, where w∈{w s ,w f The obtained feature w1 is sent to the first feature multiplication module.

[0063] The first feature multiplication module is used to multiply feature c1 and feature w1 to obtain fused feature z1 and send it to the first feature summation module.

[0064] The first feature summation module is used to sum features w1, z1, and c2 to obtain the fused feature z2, which is then sent to the fused feature extraction module.

[0065] The fusion feature extraction module is used to extract features from the fusion feature z2, obtain feature w2, and send it to the second feature multiplication module.

[0066] The second feature multiplication module is used to multiply feature z3 and feature c3 to obtain the fused feature z3 and send it to the second feature summation module.

[0067] The second feature summation module is used to sum features w2, z3 and c4 to obtain the fused feature z4 and send it to the third feature summation module.

[0068] The third feature summation module is used to sum the input features w and z4 to obtain the optimized features w. c And output it.

[0069] The feature concatenation module is used to optimize speech features. and optimize facial features The features are then stitched together and the stitched features W are sent to the face generator.

[0070] The face generator is used to generate N speaking face video frames based on the splicing features W, forming a speaking digital human video. In this embodiment, the face generator consists of a set of cascaded transposed convolutional blocks, used for upsampling to obtain speaking face video frames.

[0071] The face discriminator is used to distinguish between the input pose reference video frame and the generated speaking face video frame, and obtain the discrimination result.

[0072] S103: Training a general speaking digital human generation model:

[0073] The training sample set obtained in step S101 is used to train the general speaking digital human generation model constructed in step S102, resulting in a trained general speaking digital human generation model.

[0074] In order to map multimodal spaces from different domains into a unified joint learning space and improve the training effect of the general speaking digital human generation model, this embodiment improves the loss function of the general speaking digital human generation model by calculating text-speech-visual loss, reading clarity loss, lip-sync loss and visual quality loss, and then fusing them to obtain the model loss function.

[0075] 1) Text-speech-visual loss

[0076] In the two joint learning space modules, the goal is to ensure that features with similar semantics from different multimodal spaces are close to each other. Therefore, this embodiment uses cosine similarity to design a text-speech-visual loss L... tsv The details are as follows:

[0077] L tsv =L ts +L tv

[0078] Among them, L ts =1-cos(c,w s ) represents text-to-speech loss, L tv =1-cos(c,w f ) represents the visual loss of the text, and cos() represents solving the cosine similarity.

[0079] 2) Loss of reading clarity

[0080] N speaking face video frames generated from the current training samples are inserted into the original face video data V to replace the N pose reference frames, thus obtaining face video data V′. A lip-reading network is pre-built and trained. The face video data V′ is input into the lip-reading network, which identifies the word matrix based on lip movements. Extracting word matrix from speech data In the comparison, L represents the length of the word sequence, and C represents the word category. The reading clarity loss L is calculated using the following formula. lri :

[0081]

[0082] Among them, A i,j A i ′ ,j Let A and B represent the probabilities that the i-th word in word matrices A and A′ belongs to the j-th category, respectively, i = 1, 2, ..., L, j = 1, 2, ..., C.

[0083] Using reading acuity loss can effectively improve the reading acuity of spoken digital human-generated videos.

[0084] 3) Loss of lip-sound synchronization

[0085] In this embodiment, contrastive learning loss (InfoNCE loss) is used to attract audio features extracted by the speech encoder and temporally aligned visual context features extracted by the lip-reading network, while rejecting audio features from other frames, thereby improving lip-sound synchronization capability. The lip-sound synchronization loss L... lsl The calculation formula is as follows:

[0086]

[0087] in, This represents the intermediate audio features extracted by the speech encoder from the nth pose reference video in the current training sample. This represents the intermediate visual features extracted by the lip-reading network from the nth speaking face video frame generated from the current training samples. The intermediate audio and video features are outputs of a specific layer in the speech encoder or lip-reading network; the choice of which layer's output to use depends on the specific situation. θ(x,y) represents the feature similarity, calculated using the following formula:

[0088]

[0089] Here, exp() represents the exponential function, Nor() represents the normalization operation, FC() represents the fully connected layer operation, and τ represents the preset temperature coefficient.

[0090] 4) Loss of visual quality

[0091] Calculate the L1 loss between the original frames corresponding to the N pose reference frames in the current training sample and the generated N speaking face video frames as the visual quality loss L1:

[0092]

[0093] Where, x n 、x′ n These represent the original frame and the generated video frame showing the speaking face, respectively, n = 1, 2, ..., N.

[0094] 5) Generate adversarial network loss

[0095] Similar to ordinary generative adversarial networks, the general speaking digital human generation model of this invention also involves two sets of losses, namely the generation loss L. gen and identification loss L dis The calculation formulas are as follows:

[0096]

[0097] Where Ε[] represents the expected value, D() represents the identification result, and x n 、x′ n This represents the original frame corresponding to the input pose reference frame and the generated speaking face video frame.

[0098] Then, the model training loss function Loss is calculated using the following formula:

[0099] Loss=λ tsv L tsv +λ lri L lri +λ lsl L lsl +λ1L1+λ gen L gen +λ dis L dis

[0100] Where, λ tsv , λ lri , λ lsl ,λ1,λ gen , λ dis These represent the weights of the corresponding loss functions.

[0101] S104: Generate a talking digital human video:

[0102] When generating a speaking digital human video, the audio data to be generated is first segmented into frames, and the Mel spectrogram of each audio frame is extracted. The audio frame sequence is then divided into subsequences of length N, resulting in a Mel spectrogram sequence S′ for each subsequence. N face video frames are then selected based on the actual situation, and one is chosen as the mouth reference image M. r The lower half of N frames of face video is masked and used as pose reference frames to form pose reference video P. r Additionally, select N facial video frames of the same person as identity reference frames to construct identity reference video I.r ′, refer to the posture video P r ′ and Identity Reference Video I r The reference video F′ is obtained by stitching along the channel dimension. The mermaid spectrogram sequence S′ and the mouth reference image M′ are then combined. r The training general speaking digital human generation model takes F′ and reference video F′ as inputs to generate a speaking digital human video of length N, and then splices them together to obtain the speaking digital human video corresponding to the speech data to be generated.

[0103] To better illustrate the technical solution of the present invention, specific examples are used to experimentally verify the present invention.

[0104] This embodiment uses two audiovisual datasets, LRS2 and LRW, to validate the performance of the proposed method. These datasets are very popular for spoken digital human generation. LRS2 is an audio-visual speech recognition dataset collected from the real world, with each sample containing pairs of video, text, and sequences. It has two training subsets containing 60,000 speech samples: a pre-training subset (195 hours) and a training subset (29 hours). LRW is a dataset containing 500 distinct words for audiovisual word classification, with each sample including a video, a synchronized speech, and a spoken word from the middle of the sample. The audio and video sampling rates for both datasets are 16kHz and 25fps, respectively.

[0105] In this embodiment, the proposed general speaking digital human generation model (denoted as MouthTalk) is trained on the LRS2 training set and evaluated on the LRS2 validation set. After training, the MouthTalk model is tested using the LRS2 and LRW test sets and compared with other existing methods. The comparison methods include:

[0106] For details on Wav2Lip and Wav2Lip-2, please refer to the literature "KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In ACM MM, pages 484–492, 2020."

[0107] TalkLip and TalkLip-2, for details, please refer to the document "Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T Tan, and Haizhou Li. Seeing what you said: Talking face generation guided by a lip reading expert. In CVPR, pages 14653–14662, 2023."

[0108] Compared to most related models, Wav2Lip and TalkLip perform better in terms of lip-sync, reading clarity, and visual quality, and are therefore the main comparison methods in this embodiment.

[0109] In this embodiment, three sets of evaluation metrics were used to evaluate the performance of the present invention and the comparative method: (1) PSNR and SSIM for evaluating visual quality; (2) LSE-C and LSE-D for evaluating lip-sync; and (3) WER and ACC for evaluating reading acuity. Specifically, PSNR and SSIM were calculated by comparing generated faces with real faces. LSE-C and LSE-D were quantized using a pre-trained synchronization network. WER on LRS2 was evaluated using AV-Hubert (large), and ACC on LRW was measured using an MVM model. Table 1 is a performance comparison table of the present invention and the comparative method in this embodiment.

[0110]

[0111] Table 1

[0112] As shown in Table 1, CLIP-related losses With the help of CLIP's text and image encoders, text feature learning module, and joint feature space, the MouthTalk model of this invention achieves state-of-the-art performance on LS (LSE-C and LSE-D), RI (WER and ACC), and VQ (PSNR and SSIM). This demonstrates that using unified text prompts to guide universal human-speaking digital human generation is effective. Although TalkLip introduces attention to lip movements to enhance RI and LS, both TalkLip and TalkLip-2 show a downward trend in VQ compared to Wav2Lip and Wav2lip-2. By incorporating robust text-image features from large-scale data and domain-specific speech-visual features into a unified joint learning space, the MouthTalk model not only enhances LS and RI but also improves VQ. This shows that the present invention is more reliable and robust when facing universal human-speaking digital human generation tasks. Ablation experiments were also conducted on the proposed method, and the results demonstrate the rationality and effectiveness of our proposed method.

[0113] Figure 4 This is a diagram showing the result of uniformly sampling two generated face dialogue videos generated using the present invention in this embodiment. Figure 4 Ground Truth represents real video frames. From the two sets of video frames, it can be observed that Wav2Lip can generate a mouth region with moderate similarity to real video frames, but there are blurry and unrealistic mouth transformations; TalkLip can change the lip shape to enhance RI, but there are problems with mouth shape incoordination and poor mouth shape similarity; compared with Wav2Lip and TalkLip, the MouthTalk model proposed in this invention can provide smoother lip changes, more accurate lip-sound consistency, and better visual quality in the oral cavity region.

[0114] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A method for generating speaking digital humans based on large-scale model-learnable text latent codes, characterized in that, Includes the following steps: S1: Acquire several facial voice videos according to actual needs, extract voice data from each facial voice video, then remove the voice data to obtain facial video data. Process the obtained pairs of voice data and facial video data to obtain training samples. The specific method is as follows: For each pair of voice data and face video data, extract the region of interest (ROI) from each video frame of the face video data, normalize the ROI of each video frame to a preset size, and obtain the processed face video data V; select N video frames from the face video data V, and choose one of them as the mouth reference image M. r The lower half of N video frames is masked and used as the pose reference frame to form the pose reference video P. r Then, N video frames are selected from the remaining video frames as identity reference frames to form the identity reference video I. r ;Please refer to the posture video r Identity Reference Video I r The reference video F is obtained by stitching along the channel dimension; The speech data is divided into frames, and the Mel spectrogram of each speech frame is extracted to form a Mel spectrogram sequence S; Each reference video F and its corresponding Mel spectrogram sequence S, and mouth reference image M are used to... r Construct a training sample to obtain the training sample set; S2: Construct a general speaking digital human generation model, including a speech encoder, a face encoder, an image encoder, a text encoder, a text feature learning module, a speech joint learning space module, a face joint learning space module, a feature concatenation module, and a face generator, wherein: The speech encoder is used to extract features from the Mel spectrogram sequence S, and selects the pose reference video P from the obtained speech feature matrix. r The feature vectors of the corresponding speech frames constitute the speech features w s Send to the speech joint learning space module; A face encoder is used to extract features from a stitched video F containing 2N face video frames, and the resulting face features w f Send to the face joint learning space module; Image encoder used for mouth reference image M r Feature extraction is performed, and the resulting mouth image features c m Send to the text feature learning module; The text encoder is used to extract features from a preset mouth text c. t Send to the text feature learning module; The text feature learning module is used to learn from the features of the mouth image. m and mouth text features c t The text latent code c is extracted and sent to the speech joint learning space module and the face joint learning space module respectively. The speech joint learning space module is used to inject the text latent code c into the speech feature w using the following formula. s Optimized speech features And send it to the feature splicing module; The face joint learning space module is used to inject the text latent code c into the face feature w using the following formula. f Optimized facial features And send it to the feature splicing module; The feature concatenation module is used to optimize speech features. and optimize facial features The features are then stitched together, and the stitched features W are sent to the face generator. The face generator is used to generate N speaking face video frames based on the splicing feature W, forming a speaking digital human video; S3: Use the training sample set obtained in step S1 to train the general speaking digital human generation model constructed in step S2, and obtain the trained general speaking digital human generation model. S4: When generating a speaking digital human video, first, the speech data to be generated is segmented into frames and the Mel spectrogram of each speech frame is extracted. The speech frame sequence is divided into subsequences of length N, and the Mel spectrogram sequence S′ of each subsequence is obtained. N face video frames are set according to the actual situation. The lower half of these frames is masked and used as pose reference frames to form the pose reference video P. r Additionally, select N facial video frames of the same person as identity reference frames to construct identity reference video I. r ′; The Mel spectrogram sequence S′ and the pose reference video P r ′ and Identity Reference Video I r Input the trained general speaking digital human generation model to generate a speaking digital human video of length N, and then concatenate them to obtain the speaking digital human video corresponding to the speech data to be generated.

2. The universal speaking digital human generation method according to claim 1, characterized in that, The text feature learning module includes a mouth image feature noise addition module, a mouth text feature noise addition module, a first dot product module, a projection module, and a second dot product module, wherein: The mouth image feature noise reduction module is used to add noise to mouth image features c m Add Gaussian noise G a Obtain the features of the mouth image with added noise. And send it to the first dot product module; The mouth text feature noise addition module is used to add noise to mouth text features c t Add Gaussian noise G a Obtain the text features of the mouth area with added noise. And send it to the first dot product module and the second dot product module; The first dot product module is used to process the features of the noisy mouth image. Noisy mouth text features Perform a dot product operation to obtain the enhanced feature 'a' and send it to the projection module; The projection module is used to employ a preset projection function f c () Perform a projection operation on the enhanced feature a to obtain the mouth weight λ and send it to the second dot product module; The second dot product module is used to calculate the mouth weight λ and the noisy mouth text features. Perform a dot product operation to obtain the text latent code c.

3. The universal speaking digital human generation method according to claim 1, characterized in that, The speech joint learning space module and the face joint learning space module adopt the same joint learning space module structure, including a first text latent code feature extraction module, a second text latent code feature extraction module, a third text latent code feature extraction module, a fourth text latent code feature extraction module, an input feature extraction module, a first feature multiplication module, a first feature summation module, a fusion feature extraction module, a second feature multiplication module, a second feature summation module, and a third feature summation module, wherein: The first text latent code feature extraction module is used to extract features from the text latent code c, and send the obtained feature c1 to the first feature multiplication module; The second text latent code feature extraction module is used to extract features from the text latent code c, and send the obtained feature c2 to the first feature summation module; The third text latent code feature extraction module is used to extract features from the text latent code c, and sends the obtained feature c3 to the second feature multiplication module; The fourth text latent code feature extraction module is used to extract features from the text latent code c, and sends the obtained feature c4 to the second feature summation module; The input feature extraction module is used to extract features from the input feature w, where w∈{w s ,w f The obtained feature w1 is sent to the first feature multiplication module; The first feature multiplication module is used to multiply feature c1 and feature w1 to obtain fused feature z1 and send it to the first feature summation module; The first feature summation module is used to sum features w1, z1, and c2 to obtain the fused feature z2 and send it to the fused feature extraction module. The fusion feature extraction module is used to extract features from the fusion feature z2 to obtain feature w2, which is then sent to the second feature multiplication module. The second feature multiplication module is used to multiply feature z3 and feature c3 to obtain fused feature z3 and send it to the second feature summation module; The second feature summation module is used to sum features w2, z3 and c4 to obtain the fused feature z4 and send it to the third feature summation module; The third feature summation module is used to sum the input features w and z4 to obtain the optimized features w. c And output it.

4. The universal speaking digital human generation method according to claim 1, characterized in that, The formula for calculating the loss function Loss during the training of the general speaking digital human generation model in step S3 is as follows: Loss=λ tsv L tsv +λ lri L lri +λ lsl L lsl +λ1L1+λ gen L gen +λ dis L dis Among them, L tsv L represents text-to-speech-to-visual loss. lri Indicating a loss of reading clarity, L lsl L1 represents the loss of lip-sound synchronization, and L2 represents the loss of visual quality. gen L represents the generation loss. dis λ represents the discrimination loss. tsv , λ lri , λ lsl ,λ1,λ gen , λ dis These represent the weights of the corresponding losses, and the formulas for calculating each loss are as follows: Text-to-speech-to-visual loss L tsv The calculation formula is: L tsv =L ts +L tv Among them, L ts =1-cos(c,w s ) represents text-to-speech loss, L tv =1-cos(c,w f ) represents the visual loss of the text, and cos() represents solving the cosine similarity; Reading clarity loss L lri The calculation method is as follows: N speaking face video frames generated from the current training samples are inserted into the original face video data V to replace the N pose reference frames, thus obtaining face video data V′. A lip-reading network is pre-built and trained, and the face video data V′ is input into the lip-reading network to identify the word matrix based on lip movements. Extracting word matrix from speech data In the comparison, L represents the length of the word sequence, and C represents the word category. The reading clarity loss L is calculated using the following formula. lri ; Among them, A i,j A i ′ ,j Let A and B represent the probabilities that the i-th word in word matrices A and A′ belongs to the j-th category, respectively, i = 1, 2, ..., L, j = 1, 2, ..., C; Lip-sound synchronization loss L lsl The calculation formula is: in, This represents the intermediate audio features extracted by the speech encoder from the nth pose reference video in the current training sample. θ represents the intermediate visual features extracted by the lip-reading network from the nth speaking face video frame generated from the current training samples; θ(x,y) represents the feature similarity, and its calculation formula is: Where exp() represents the exponential function, Nor() represents the normalization operation, FC() represents the fully connected layer operation, and τ represents the preset temperature coefficient. The formula for calculating visual quality loss L1 is: Where, x n 、x′ n These represent the original frame and the generated video frame showing the speaking face, respectively, n = 1, 2, ..., N; Generation loss L gen and identification loss L dis The calculation formulas are as follows: L gen =E[log(1-D(x′ n ))] L gen =E[log(1-D(x n ))]+Ε[log(D(x′ n ))] Where Ε[] represents the expected value, D() represents the identification result, and x n 、x′ n This represents the original frame corresponding to the input pose reference frame and the generated speaking face video frame.

Images