Speaking digital human generation method based on large model learnable text latent codes

Abstract

The invention discloses a speaking digital human generation method based on a large model learnable text latent code, which comprises the following steps: acquiring a plurality of face voice videos according to actual needs and extracting training samples to form a training sample set, each training sample comprising a reference video and a corresponding Mel spectrogram sequence and mouth reference image; constructing a general speaking digital human generation model based on learnable text latent codes, and training by adopting the training sample set; and when a speaking digital person video needs to be generated, framing is carried out on to-be-generated voice data, a Mel spectrogram sequence is extracted, a mouth reference image and a reference video are set according to actual conditions, the reference image and the reference video are input into the trained general speaking digital person generation model together, and the speaking digital person video corresponding to the to-be-generated voice data is generated. According to the method, robust text latent codes are continuously learned from mouth reference images of different speakers, the method is used for guiding the synthesis process of a general speaking digital human video, and the quality of a speaking digital human generation model is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image generation, and more specifically, relates to a method for generating a speaking digital human based on a large model capable of learning text latent codes. Background Art

[0002] Talking digital human generation aims to map acoustic features to time-aligned facial movements. Currently, the methods can be divided into two categories: specific talking digital human generation methods for individuals and general talking digital human generation methods for anyone. Some methods can generate high-fidelity face videos for specific people, most of which rely on neural radiance fields (NeRFs) and three-dimensional deformable models (3DMMs). In contrast, general talking digital human generation methods can use a few seconds of video clips or a single static face image to match arbitrary speech content. In general talking digital human generation methods, there are three aspects that are closely related to the quality of face generation, namely visual quality (Visual Quality, VQ), lip sync (Lip Sync, LS) and reading intelligibility (Reading Intelligibility, RI).

[0003] In recent years, large models have become popular in various research fields. CLIP is a large visual language pre-training model that is very effective for visual tasks using text representation. This powerful model has been proven to be effective in many tasks, such as domain transfer, 3D digital humans, and image editing. Multimodal features are also conducive to robust feature learning. In the field of TFG, there are currently a few works that attempt to use the CLIP model for emotional feature extraction to increase the richness of the expressions of speaking digital humans.

[0004] However, it is difficult for current general speaking digital human methods to improve reading clarity and lip-sound consistency while maintaining good visual quality. In addition, the features learned from limited speech-face data pairs are difficult to adapt to the lip shapes of different speakers, which hinders the performance of general speaking digital human generation methods. Lip-sound consistency and visual quality have become the two main challenges faced by speaking digital human generation.

[0005] To address the problem of lip-sync consistency, some techniques use a mutual information estimator to alleviate the problem of poor lip synchronization. Calculating the synchronization loss between the visual lip features of the real face and the output video sequence is also a technique to improve lip-sync consistency. In addition to using auxiliary models to assist in controlling lip-sync consistency, the encoder-decoder structure can also effectively improve the performance of speech and video consistency. In addition to lip-sync consistency, poor visual quality (i.e., blurry and unrealistic) also affects the performance of generated talking person videos. This problem can be alleviated by skip connections and generative adversarial networks. Generative adversarial networks can generate realistic talking heads while maintaining the facial features of the target person. In addition to lip-sync consistency and visual quality, reading clarity is also indispensable in the generation of speaking digital humans. It reflects how much text can be translated from the talking face video through lip reading technology. Adjusting the lip shape to match the corresponding speech content can improve reading clarity, and using lip sync discriminators and contrastive learning can improve lip-sync consistency. However, these methods can cause unexpected changes in fine-grained mouth areas, resulting in poor visual quality performance. Current technologies can either only improve reading clarity and lip-sound consistency, or only improve visual quality; it is difficult to improve all three aspects at the same time. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for generating a speaking digital human based on a large model that can learn text latent codes. A universal speaking digital human generation model is designed to continuously learn robust text latent codes from mouth reference images of different speakers to guide the synthesis process of universal speaking digital human videos and improve the quality of the speaking digital human generation model.

[0007] In order to achieve the above-mentioned invention object, the method for generating a speaking digital human based on a large model capable of learning textual latent codes comprises the following steps:

[0008] S1: According to actual needs, several face and voice videos are obtained, voice data is extracted from each face and voice video, and then face video data is obtained after removing the voice data. Several pairs of voice data and face video data are processed to obtain training samples. The specific method is as follows:

[0009] For each pair of speech data and face video data, extract the region of interest on each video frame in the face video data, normalize the region of interest of each video frame to a preset size, and obtain the processed face video data V; select N video frames from the face video data V, and select one of them as the mouth reference image M r , the lower half of the N frames of video are masked and used as the posture reference frame to form the posture reference video P r; Then select N video frames from the remaining video frames as identity reference frames to form identity reference video I r ; Reference the pose video P r and identity reference video I r Splice along the channel dimension to get the reference video F;

[0010] The speech data is divided into frames, and the Mel-spectrogram of each speech frame is extracted to form a Mel-spectrogram sequence S;

[0011] Each reference video F and the corresponding Mel-spectrogram sequence S, mouth reference image M r Construct a training sample to obtain a training sample set;

[0012] S2: Construct a general speaking digital human generation model, including a speech encoder, a face encoder, an image encoder, a text encoder, a text feature learning module, a speech joint learning space module, a face joint learning space module, a feature splicing module and a face generator, wherein:

[0013] The speech encoder is used to extract features from the Mel-spectrogram sequence S and filter out the posture reference video P from the obtained speech feature matrix. r The feature vector of the corresponding speech frame constitutes the speech feature w s Send to the speech joint learning space module;

[0014] The face encoder is used to extract features from the spliced ​​video F containing 2N face video frames, and the obtained face features w f Send to the face joint learning space module;

[0015] The image encoder is used to align the mouth reference image M r Perform feature extraction and obtain the mouth image feature c m Send to the text feature learning module;

[0016] The text encoder is used to extract features from the preset mouth text prompt T and convert the obtained mouth text features c t Send to the text feature learning module;

[0017] The text feature learning module is used to learn the mouth image features c m and mouth text feature c t The text latent code c is extracted and sent to the speech joint learning space module and the face joint learning space module respectively;

[0018] The speech joint learning space module is used to inject the textual latent code c into the speech feature w using the following formula s Get optimized voice features And send it to the feature stitching module;

[0019] The face joint learning space module is used to inject the text latent code c into the face feature w using the following formula f Get optimized facial features And send it to the feature stitching module;

[0020] The feature concatenation module is used to optimize the speech features and optimize facial features Perform splicing, and then send the splicing feature W to the face generator;

[0021] The face generator is used to generate N talking face video frames according to the splicing feature W to form a talking digital human video;

[0022] S3: using the training sample set obtained in step S1 to train the general speaking digital human generation model constructed in step S2, to obtain a trained general speaking digital human generation model;

[0023] S4: When it is necessary to generate a speaking digital human video, firstly divide the speech data to be generated into frames and extract the Mel-spectrogram of each speech frame, divide the speech frame sequence into subsequences of length N, and obtain the Mel-spectrogram sequence S′ of each subsequence; according to the actual situation, set N human face video frames, mask the lower half of these frames and use them as posture reference frames to form a posture reference video P r ′, and select N face video frames of the same person as the identity reference frames to form the identity reference video I r ′; Mel frequency spectrum sequence S′, posture reference video P r ′ and identity reference video I r ' Input the trained general speaking digital human generation model, generate speaking digital human videos of length N, and then splice them to obtain the speaking digital human videos corresponding to the speech data to be generated.

[0024] The present invention is based on a method for generating a speaking digital human with a large model that can learn textual latent codes. According to actual needs, a number of face voice videos are obtained and training samples are extracted to form a training sample set, each training sample includes a reference video and a corresponding mel-spectrogram sequence and a mouth reference image; a universal speaking digital human generation model based on a learnable textual latent code is constructed, and the training sample set is used for training; when it is necessary to generate a speaking digital human video, the voice data to be generated is firstly framed and the mel-spectrogram sequence is extracted, and a mouth reference image and a reference video are set according to actual conditions, and the mouth reference image and the reference video are input into the trained universal speaking digital human generation model together to generate a speaking digital human video corresponding to the voice data to be generated.

[0025] The present invention has the following beneficial effects:

[0026] 1) The present invention uses the shared semantics in the speech-vision multimodal space and the text-image multimodal space as a bridge, sets up a speech joint learning space module and a face joint learning space module in the general speaking digital human generation model, integrates the speech-vision features obtained from highly aligned data pairs and the robust text-image features based on extensive data, thereby connecting the robust feature space learned from large-scale data with the task-specific feature space, thereby improving the quality of speaking digital human videos;

[0027] 2) The universal speaking digital human generation model of the present invention is guided by a learnable text latent code so as to use a unified text latent code to capture the dynamic changes of the oral region caused by different speakers and pronunciation changes; unlike previous methods that excel in specific aspects (such as reading clarity, lip synchronization or visual quality), this model not only further improves reading clarity and lip consistency, but also improves visual quality, and can not only allow different speakers to produce lip shapes similar to the ground truth when pronouncing the same word, but also retain the unique characteristics of the oral region.

[0028] 3) The present invention also proposes an improved text feature learning strategy and loss function when training a general speaking digital human generation model, thereby further improving the model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flowchart of a specific implementation method of the method for generating a universal speaking digital human with learnable textual latent code based on a large model of the present invention;

[0030] Figure 2 It is a structural diagram of a general speaking digital human generation model in the present invention;

[0031] Figure 3 is a structural diagram of the joint learning space module in this embodiment;

[0032] Figure 4 This is a result diagram of uniformly sampling two segments of generated face dialogue videos generated by the present invention in this embodiment. DETAILED DESCRIPTION

[0033] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0034] Example

[0035] Figure 1 This is a flowchart of a specific implementation of the method for generating a general speaking digital human with learnable text latent code based on a large model of the present invention. Figure 1 As shown, the specific steps of the method for generating a general speaking digital human based on learnable textual latent codes of the present invention include:

[0036] S101: Obtain training samples:

[0037] According to actual needs, several face and voice videos are obtained, voice data is extracted from each face and voice video, and then face video data is obtained after removing the voice data. Several pairs of voice data and face video data are processed to obtain training samples. The specific method is as follows:

[0038] For each pair of speech data and face video data, extract the region of interest (ROI) on each video frame in the face video data, normalize the region of interest of each video frame to a preset size, and obtain the processed face video data V. Select N video frames from the face video data V, and select one of them as the mouth reference image M r , the lower half of these frames are masked and used as pose reference frames to form pose reference video P r Then, N video frames are selected from the remaining video frames as identity reference frames to form the identity reference video I r . Refer to the pose video P r and identity reference video I r The reference video F is obtained by splicing along the channel dimension.

[0039] The speech data is divided into frames, and the mel-spectrogram of each speech frame is extracted to form a mel-spectrogram sequence S. The specific parameters of the mel-spectrogram can be set according to actual needs. In this embodiment, the jump and window lengths are 125 milliseconds and 50 milliseconds respectively.

[0040] Each reference video F and the corresponding Mel-spectrogram sequence S, mouth reference image M r Construct it into a training sample to obtain a training sample set.

[0041] S102: Constructing a general speaking digital human generation model:

[0042] In order to improve the visual quality of talking digital human videos, the present invention introduces a learnable textual latent code into a general talking digital human generation model to guide the generation of talking digital human video frames. Figure 2 : is a structural diagram of the general speaking digital human generation model in the present invention. Figure 2As shown, the general speaking digital human generation model in the present invention includes a speech encoder, a face encoder, an image encoder, a text encoder, a text feature learning module, a speech joint learning space module, a face joint learning space module, a feature splicing module, a face generator and a face discriminator. Next, each module is described in detail.

[0043] The speech encoder is used to extract features from the Mel-spectrogram sequence S and filter out the posture reference video P from the obtained speech feature matrix. r The feature vector of the corresponding speech frame constitutes the speech feature w s Send to the speech joint learning space module.

[0044] The face encoder is used to extract features from the spliced ​​video F containing 2N face video frames, and the obtained face features w f Send to the face joint learning space module.

[0045] The image encoder is used to align the mouth reference image M r Perform feature extraction and obtain the mouth image feature c m The image encoder in this embodiment adopts the image encoder in the CLIP model.

[0046] The text encoder is used to extract features of the preset mouth text prompt T (in this embodiment, "mouth"), and extract the obtained mouth text feature c t The text encoder is sent to the text feature learning module. In this embodiment, the text encoder is the text encoder in the CLIP model.

[0047] The text feature learning module is used to learn the mouth image features c m and mouth text feature c t The text latent code c is extracted and sent to the speech joint learning space module and the face joint learning space module respectively.

[0048] like Figure 2 As shown, the text feature learning module in this embodiment includes a mouth image feature denoising module, a mouth text feature denoising module, a first dot product module, a projection module, and a second dot product module, wherein:

[0049] The mouth image feature denoising module is used to add noise to the mouth image feature c m Add Gaussian noise G a , get the noisy mouth image features And sent to the first point multiplication module. In this embodiment, Gaussian noise G a A Gaussian noise with normal distribution is used.

[0050] The mouth text feature denoising module is used to add noise to the mouth text feature ct Add Gaussian noise G a , get the noisy mouth text feature And sent to the first dot product module and the second dot product module.

[0051] The first dot multiplication module is used to add noise to the mouth image features And noisy mouth text features Perform a dot product operation to obtain the enhanced feature a and send it to the projection module.

[0052] The projection module is used to use the preset projection function f c () Perform a projection operation on the enhanced feature a to obtain the mouth weight λ and send it to the second dot product module.

[0053] The second dot product module is used to add the mouth weight λ and the noisy mouth text feature Perform a dot multiplication operation to obtain the text latent code c. By introducing the mouth weight λ, the feature can be effectively enhanced. It can extract task-related details and suppress task-irrelevant information, making the learned text latent code more accurate.

[0054] The speech joint learning space module is used to inject the textual latent code c into the speech feature w using the following formula s Get optimized voice features And sent to the feature stitching module.

[0055] The face joint learning space module is used to inject the text latent code c into the face feature w using the following formula f Get optimized facial features And sent to the feature stitching module.

[0056] In the present invention, the speech visual features obtained from highly aligned data pairs and the robust text image features based on extensive data are integrated through the speech joint learning space module and the face joint learning space module, so as to use the unified text latent code to capture the dynamic changes of the oral region caused by different speakers and pronunciation changes, thereby improving the quality of the generated speaking digital human video. In this embodiment, the speech joint learning space module and the face joint learning space module adopt the same joint learning space module structure. Figure 3 is a structural diagram of the joint learning space module in this embodiment. Figure 3 As shown, the joint learning space module in this embodiment includes a first text latent feature extraction module, a second text latent feature extraction module, a third text latent feature extraction module, a fourth text latent feature extraction module, an input feature extraction module, a first feature multiplication module, a first feature summation module, a fusion feature extraction module, a second feature multiplication module, a second feature summation module and a third feature summation module, wherein:

[0057] The first text latent code feature extraction module is used to extract features from the text latent code c, and send the obtained feature c1 to the first feature multiplication module.

[0058] The second text latent code feature extraction module is used to extract features from the text latent code c, and send the obtained feature c2 to the first feature summation module.

[0059] The third text latent code feature extraction module is used to extract features from the text latent code c, and send the obtained feature c3 to the second feature multiplication module.

[0060] The fourth text latent code feature extraction module is used to extract features from the text latent code c, and send the obtained feature c4 to the second feature summation module.

[0061] In this embodiment, the four text latent code feature extraction modules all adopt the same structure, which is obtained by cascading a one-dimensional convolution layer, a layer normalization layer and a leaky Relu layer.

[0062] The input feature extraction module is used to extract the input feature w, w∈{w s ,w f}, and send the obtained feature w1 to the first feature multiplication module.

[0063] The first feature multiplication module is used to multiply feature c1 and feature w1 to obtain fused feature z1 and send it to the first feature summation module.

[0064] The first feature summation module is used to sum the feature w1, feature z1, and feature c2 to obtain the fused feature z2 and send it to the fused feature extraction module.

[0065] The fusion feature extraction module is used to extract the fusion feature z2, obtain the feature w2 and send it to the second feature multiplication module.

[0066] The second feature multiplication module is used to multiply feature z3 and feature c3 to obtain fused feature z3 and send it to the second feature summation module.

[0067] The second feature summation module is used to sum the feature w2, feature z3 and feature c4 to obtain a fused feature z4 and send it to the third feature summation module.

[0068] The third feature summation module is used to sum the input feature w and feature z4 to obtain the optimized feature w c And output,

[0069] The feature concatenation module is used to optimize the speech features and optimize facial features The splicing is performed and then the splicing feature W is sent to the face generator.

[0070] The face generator is used to generate N talking face video frames according to the splicing feature W to form a talking digital human video. In this embodiment, the face generator is composed of a group of cascaded transposed convolution blocks, which are used to upsample to obtain talking face video frames.

[0071] The face discriminator is used to discriminate the input posture reference video frame and the generated speaking face video frame respectively to obtain a discriminant result.

[0072] S103: Training a general speaking digital human generation model:

[0073] The training sample set obtained in step S101 is used to train the general speaking digital human generation model constructed in step S102 to obtain a trained general speaking digital human generation model.

[0074] In order to map the multimodal spaces of different domains into a unified joint learning space and improve the training effect of the general speaking digital human generation model, the loss function of the general speaking digital human generation model is improved in this embodiment by calculating the text-speech-vision loss, reading clarity loss, lip synchronization loss and visual quality loss, and then fusing them to obtain the model loss function.

[0075] 1) Text-Speech-Vision Loss

[0076] In the two joint learning space modules, the goal is to ensure that features with similar semantics from different multimodal spaces are close to each other. Therefore, this embodiment uses cosine similarity to design a text-speech-vision loss L tsv , as follows:

[0077] L tsv =L ts +L tv

[0078] Among them, L ts =1-cos(c,w s ) represents the text-to-speech loss, L tv =1-cos(c,w f ) represents text visual loss, and cos() represents solving cosine similarity.

[0079] 2) Loss of reading clarity

[0080] Insert the N speaking face video frames generated by the current training sample into the original face video data V to replace the N posture reference frames, thereby obtaining the face video data V′. Pre-build and train the lip reading network, input the face video data V′ into the lip reading network, and recognize the word matrix based on the lip movements. Extract word matrix from speech data For comparison, L represents the length of the word sequence, C represents the word category, and the reading clarity loss L is calculated using the following formula lri :

[0081]

[0082] Among them, A i,j , A i ' ,j They represent the probability that the i-th word in the word matrix A and A′ belongs to the j-th category, i = 1, 2, …, L, j = 1, 2, …, C.

[0083] Using reading clarity loss can effectively improve the reading clarity of videos generated by speaking digital humans.

[0084] 3) Lip sync loss

[0085] In this embodiment, the contrastive learning loss (InfoNCE loss) is used to attract the audio features extracted by the speech encoder and the time-aligned visual context features extracted by the lip reading network, while rejecting the audio features of other frames, thereby improving the lip synchronization ability. lsl The calculation formula is as follows:

[0086]

[0087] in, represents the intermediate audio features extracted by the speech encoder from the nth posture reference video in the current training sample, It represents the intermediate visual features extracted by the lip reading network for the nth speaker face video frame generated by the current training sample. The intermediate audio features and intermediate video features are the outputs of a certain layer in the speech encoder and lip reading network. The specific output of which layer is selected depends on the actual situation. θ(x,y) represents the feature similarity, and its calculation formula is:

[0088]

[0089] Among them, exp() represents the exponential function, Nor() represents the normalization operation, FC() represents the fully connected layer operation, and τ represents the preset temperature coefficient.

[0090] 4) Loss of visual quality

[0091] Calculate the L1 loss between the original frames corresponding to the N pose reference frames in the current training sample and the generated N speaking face video frames as the visual quality loss L1:

[0092]

[0093] Among them, x n , x′ n Represent the original frame and the generated speaking face video frame respectively, n=1,2,…,N.

[0094] 5) Generate adversarial network loss

[0095] Similar to the common generative adversarial network, the universal speaking digital human generation model of the present invention also involves two sets of losses, namely, the generation loss L gen and the identification loss L dis , the calculation formulas are as follows:

[0096]

[0097] Among them, Ε[] represents the expectation, D() represents the identification result, x n , x′ n Represents the original frame corresponding to the input pose reference frame and the generated speaking face video frame.

[0098] Then the following formula is used to calculate the model training loss function Loss:

[0099] Loss = λ tsv L tsv +λ lri L lri +λ lsl L lsl +λ1L1+λ gen L gen +λ dis L dis

[0100] Among them, λ tsv , lri , lsl ,λ1,λ gen , dis They represent the weights of the corresponding loss functions respectively.

[0101] S104: Generate a speaking digital human video:

[0102] When it is necessary to generate a speaking digital human video, first divide the speech data to be generated into frames and extract the Mel-spectrogram of each speech frame, divide the speech frame sequence into subsequences of length N, and obtain the Mel-spectrogram sequence S′ of each subsequence. According to the actual situation, set N human face video frames and select one of them as the mouth reference image M r ′, the lower half of the N frames of face video frames are masked and used as the pose reference frame to form the pose reference video P r ′, and select N face video frames of the same person as the identity reference frames to form the identity reference video Ir ′, the pose reference video P r ′ and identity reference video I r ′ is spliced ​​together along the channel dimension to obtain the reference video F′. The Mel-spectrogram sequence S′ and the mouth reference image M r ′ and the reference video F′ are input into the trained general speaking digital human generation model to generate a speaking digital human video of length N, which is then spliced ​​to obtain the speaking digital human video corresponding to the speech data to be generated.

[0103] In order to better illustrate the technical solution of the present invention, a specific example is used to experimentally verify the present invention.

[0104] This embodiment uses two audio-visual datasets LRS2 and LRW to verify the performance of the proposed method. These two datasets are very popular in the generation of speaking digital humans. LRS2 is an audio-visual speech recognition dataset collected from the real world, and each sample contains a pair of videos, texts, and sequences. It has two training subsets containing 60,000 voices: a pre-training subset (195 hours) and a training subset (29 hours). LRW is a dataset containing 500 different words for audio-visual word classification. Each sample includes a video, a synchronized voice, and a spoken word in the middle of the sample. The audio and video sampling rates of the two datasets are 16Khz and 25fps, respectively.

[0105] In this embodiment, the universal speaking digital human generation model (denoted as MouthTalk) proposed in the present invention is trained in the training set of LRS2 and evaluated in the validation set of LRS2. After the training is completed, the MouthTalk model is tested using the test sets of LRS2 and LRW to compare with other existing methods. The comparison method includes:

[0106] Wav2Lip and Wav2Lip-2, for details, see the literature "KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild. In ACM MM, pages 484–492, 2020."

[0107] TalkLip and TalkLip-2, for details, please refer to the document "Jiadong Wang, Xinyuan Qian, Malu Zhang, Robby T Tan, and Haizhou Li. Seeing what you said: Talking face generation guided by a lip reading expert. In CVPR, pages 14653–14662, 2023."

[0108] Compared with most related models, Wav2Lip and TalkLip have better performance in lip synchronization, reading clarity and visual quality, and are therefore the main comparison methods in this example.

[0109] In this embodiment, three sets of evaluation indicators are used to evaluate the performance of the present invention and the comparative method: (1) PSNR and SSIM for evaluating visual quality; (2) LSE-C and LSE-D for evaluating lip synchronization; (3) WER and ACC for evaluating reading clarity. Specifically, PSNR and SSIM are calculated by comparing the generated face and the real face. LSE-C and LSE-D quantify lip synchronization using a pre-trained synchronization network. The WER on LRS2 is evaluated by AV-Hubert (large), and the ACC on LRW is measured by the MVM model. Table 1 is a performance comparison table of the present invention and the comparative method in this embodiment.

[0110]

[0111] Table 1

[0112] As shown in Table 1, the CLIP-related loss With the help of CLIP's text and image encoders, text feature learning module, and joint feature space, the MouthTalk model of the present invention achieves optimal performance in LS (LSE-C and LSE-D) and RI (WER and ACC) as well as VQ (PSNR and SSIM). This shows that it is effective to use unified text prompts to guide the generation of general speaking digital humans. Although TalkLip introduces attention to lip movements to enhance RI and LS, both TalkLip and TalkLip-2 show a downward trend in VQ compared with Wav2Lip and Wav2lip-2. After incorporating robust text image features from large-scale data and domain-specific speech visual features into a unified joint learning space, the MouthTalk model can not only enhance LS and RI, but also improve VQ. This shows that the present invention is more reliable and robust when facing the task of general speaking digital human generation. At the same time, ablation experiments are also carried out on the proposed method, and the ablation experiment results can prove the rationality and effectiveness of our proposed method.

[0113] Figure 4 This is a result diagram of uniformly sampling two segments of generated face dialogue videos generated by the present invention in this embodiment. Figure 4 The Ground Truth is the real video frame. From the two sets of video frames, it can be observed that Wav2Lip can generate a mouth area with moderate similarity to the real video frame, but there are blurry and unrealistic mouth changes; TalkLip can change the lip shape to enhance RI, but there are problems of inconsistent mouth shape and poor mouth shape similarity; compared with Wav2Lip and TalkLip, the MouthTalk model proposed in the present invention can provide smoother lip changes, more accurate lip-sound consistency, and retain better visual quality in the oral region.

[0114] Although the above describes the illustrative specific embodiments of the present invention to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

Claims

1. A method for generating a speaking digital human based on a large model capable of learning text latent codes, characterized in that: The following steps are involved: S1: According to actual needs, several face and voice videos are obtained, voice data is extracted from each face and voice video, and then face video data is obtained after removing the voice data. Several pairs of voice data and face video data are processed to obtain training samples. The specific method is as follows: For each pair of speech data and face video data, extract the region of interest on each video frame in the face video data, normalize the region of interest of each video frame to a preset size, and obtain the processed face video data V; select N video frames from the face video data V, and select one of them as the mouth reference image M r , the lower half of the N frames of video frames are masked and used as the posture reference frame to form the posture reference video P r ; Then select N video frames from the remaining video frames as identity reference frames to form identity reference video I r ; Reference the pose video P r and identity reference video I r Splice along the channel dimension to get the reference video F; The speech data is divided into frames, and the Mel-spectrogram of each speech frame is extracted to form a Mel-spectrogram sequence S; Each reference video F and the corresponding Mel-spectrogram sequence S, mouth reference image M r Construct a training sample to obtain a training sample set; S2: Construct a general speaking digital human generation model, including a speech encoder, a face encoder, an image encoder, a text encoder, a text feature learning module, a speech joint learning space module, a face joint learning space module, a feature splicing module and a face generator, wherein: The speech encoder is used to extract features from the Mel-spectrogram sequence S and filter out the posture reference video P from the obtained speech feature matrix. r The feature vector of the corresponding speech frame constitutes the speech feature w s Send to the speech joint learning space module; The face encoder is used to extract features from the spliced ​​video F containing 2N face video frames, and the obtained face features w f Send to the face joint learning space module; The image encoder is used to align the mouth reference image M r Perform feature extraction and obtain the mouth image feature c m Send to the text feature learning module; The text encoder is used to extract features from the preset mouth text prompt T and convert the obtained mouth text features c t Send to the text feature learning module; The text feature learning module is used to learn the mouth image features c m and mouth text feature c t The text latent code c is extracted and sent to the speech joint learning space module and the face joint learning space module respectively; The speech joint learning space module is used to inject the textual latent code c into the speech feature w using the following formula s Get optimized voice features And send it to the feature stitching module; The face joint learning space module is used to inject the text latent code c into the face feature w using the following formula f Get optimized facial features And send it to the feature stitching module; The feature concatenation module is used to optimize the speech features and optimize facial features Perform splicing, and then send the splicing feature W to the face generator; The face generator is used to generate N talking face video frames according to the splicing feature W to form a talking digital human video; S3: using the training sample set obtained in step S1 to train the general speaking digital human generation model constructed in step S2, to obtain a trained general speaking digital human generation model; S4: When it is necessary to generate a speaking digital human video, firstly divide the speech data to be generated into frames and extract the Mel-spectrogram of each speech frame, divide the speech frame sequence into subsequences of length N, and obtain the Mel-spectrogram sequence S′ of each subsequence; according to the actual situation, set N human face video frames, mask the lower half of these frames and use them as posture reference frames to form a posture reference video P r ′, and select N face video frames of the same person as the identity reference frames to form the identity reference video I r ′; Mel frequency spectrum sequence S′, posture reference video P r ′ and identity reference video I r ' Input the trained general speaking digital human generation model, generate speaking digital human videos of length N, and then splice them to obtain the speaking digital human videos corresponding to the speech data to be generated.

2. The method for generating a general speaking digital human according to claim 1, characterized in that: The text feature learning module includes a mouth image feature denoising module, a mouth text feature denoising module, a first dot multiplication module, a projection module and a second dot multiplication module, wherein: The mouth image feature denoising module is used to add noise to the mouth image feature c m Add Gaussian noise G a , get the noisy mouth image features And send it to the first point product module; The mouth text feature denoising module is used to add noise to the mouth text feature c t Add Gaussian noise G a , get the noisy mouth text feature And send it to the first dot product module and the second dot product module; The first dot multiplication module is used to add noise to the mouth image features And the noisy mouth text features Perform a dot product operation to obtain the enhanced feature a and send it to the projection module; The projection module is used to use the preset projection function f c () Perform a projection operation on the enhanced feature a to obtain the mouth weight λ and send it to the second point multiplication module; The second dot product module is used to add the mouth weight λ and the noisy mouth text feature Perform a dot product operation to obtain the text latent code c.

3. The method for generating a universal speaking digital human according to claim 1, characterized in that: The speech joint learning space module and the face joint learning space module adopt the same joint learning space module structure, including a first text latent feature extraction module, a second text latent feature extraction module, a third text latent feature extraction module, a fourth text latent feature extraction module, an input feature extraction module, a first feature multiplication module, a first feature summation module, a fusion feature extraction module, a second feature multiplication module, a second feature summation module and a third feature summation module, wherein: The first text latent code feature extraction module is used to extract features from the text latent code c, and send the obtained feature c1 to the first feature multiplication module; The second text latent code feature extraction module is used to extract features from the text latent code c, and send the obtained feature c2 to the first feature summation module; The third text latent code feature extraction module is used to extract features from the text latent code c, and send the obtained feature c3 to the second feature multiplication module; The fourth text latent code feature extraction module is used to extract features of the text latent code c, and send the obtained feature c4 to the second feature summation module; The input feature extraction module is used to extract the input feature w, w∈{w s ,w f }, send the obtained feature w1 to the first feature multiplication module; The first feature multiplication module is used to multiply the feature c1 and the feature w1 to obtain the fused feature z1 and send it to the first feature summation module; The first feature summation module is used to sum the feature w1, feature z1, and feature c2 to obtain the fused feature z2 and send it to the fused feature extraction module; The fusion feature extraction module is used to extract the fusion feature z2, obtain the feature w2 and send it to the second feature multiplication module; The second feature multiplication module is used to multiply feature z3 and feature c3 to obtain fused feature z3 and send it to the second feature summation module; The second feature summation module is used to sum the feature w2, feature z3 and feature c4 to obtain a fused feature z4 and send it to the third feature summation module; The third feature summation module is used to sum the input feature w and feature z4 to obtain the optimized feature w c And output, 4. The method for generating a universal speaking digital human according to claim 1, characterized in that: The calculation formula of the loss function Loss during the training of the general speaking digital human generation model in step S3 is: Loss=λ tsv L tsv +λ lri L lri +λ lsl L lsl +λ1L1+λ gen L gen +λ dis L dis Among them, L tsv represents the text-speech-vision loss, L lri Indicates the loss of reading clarity, L lsl represents lip synchronization loss, L1 represents visual quality loss, and L gen represents the generation loss, L dis represents the identification loss, λ tsv , lri , lsl ,λ1,λ gen , dis They represent the weights of the corresponding losses, and the calculation formulas for each loss are: Text-Speech-Vision Loss L tsv The calculation formula is: L tsv =L ts +L tv Among them, L ts =1-cos(c,w s ) represents the text-to-speech loss, L tv =1-cos(c,w f ) represents text visual loss, cos() represents solving cosine similarity; Reading clarity loss L lri The calculation method is: Insert the N talking face video frames generated by the current training sample into the original face video data V to replace the N posture reference frames, thereby obtaining the face video data V′; pre-build and train the lip reading network, input the face video data V′ into the lip reading network, and recognize the word matrix according to the lip movements Extract word matrix from speech data For comparison, L represents the length of the word sequence, C represents the word category, and the reading clarity loss L is calculated using the following formula lri ; Among them, A i,j , A i ' ,j Respectively represent the probability that the i-th word in the word matrix A and A′ belongs to the j-th category, i = 1, 2, ..., L, j = 1, 2, ..., C; Lip sync loss L lsl The calculation formula is: in, represents the intermediate audio features extracted by the speech encoder from the nth posture reference video in the current training sample, represents the intermediate visual features extracted by the lip reading network for the nth speaking face video frame generated by the current training sample; θ(x,y) represents the feature similarity, which is calculated as: Among them, exp() represents the exponential function, Nor() represents the normalization operation, FC() represents the fully connected layer operation, and τ represents the preset temperature coefficient; The calculation formula of visual quality loss L1 is: Among them, x n , x′ n Respectively represent the original frame and the generated speaking face video frame, n = 1, 2, ..., N; Generate loss L gen and the identification loss L dis The calculation formulas are: L gen =E[log(1-D(x′ n ))] L gen =E[log(1-D(x n ))]+Ε[log(D(x′ n ))] Among them, Ε[] represents the expectation, D() represents the identification result, x n , x′ n Represents the original frame corresponding to the input pose reference frame and the generated speaking face video frame.

Images