Voice-driven high-timeliness digital population type synthesis algorithm

Abstract

The invention relates to a voice-driven high-timeliness digital population type synthesis algorithm, and the algorithm improves the digital population type synthesis effect through introducing lip-reading expert and a pre-trained video automatic encoder as supervision, carefully designing a feature fusion module, training a loss function and other skills. The speed of digital population type synthesis is improved through skills such as channel-by-channel convolution and linear attention. And the device has the functions of controllable mouth opening amplitude and multi-language support.

Description

Technical Field

[0001] The present invention relates to the technical field of mouth shape synthesis, and particularly relates to a voice-driven digital human mouth shape synthesis algorithm with strong timeliness. Background Art

[0002] Digital humans can be applied to customer service digital humans, teacher digital humans, virtual anchors, digital immortality, and can be applied to various industries such as the film and television production industry, the anchor industry, the advertising industry, and the education industry.

[0003] In the prior art, there are the following deficiencies in digital human mouth shape synthesis:

[0004] (1) Poor lip synchronization: The lip synchronization of the mouth shapes synthesized by the existing methods is poor. For example, the mouth opens in advance.

[0005] (2) Inaccurate mouth movements: The mouth shapes synthesized by the existing methods do not conform to the lip movements during pronunciation. For example, for some words, the pronunciation requires pursing the lips, but the synthesized mouth shape does not have the pursed lips.

[0006] (3) Unable to control the opening amplitude of the mouth: Sometimes the volume of the input audio is relatively low, resulting in a relatively small opening amplitude of the digital human's mouth.

[0007] (4) Blurry image: The existing methods cannot synthesize fine facial skin textures, resulting in a relatively blurry image.

[0008] (5) High computational cost: The existing methods generally use a large number of transformers or diffusions, which are time-consuming and computationally intensive modules.

[0009] (6) Lack of multi-language support: In the existing methods, one model can only be driven by one language, and it is impossible to share the same model for multiple languages.

[0010] Therefore, it is necessary to provide a voice-driven digital human mouth shape synthesis algorithm with strong timeliness to improve the inference speed and timeliness of digital human mouth shape synthesis. Summary of the Invention

[0011] The purpose of the present invention is to provide a voice-driven digital human mouth shape synthesis algorithm with strong timeliness to improve the inference speed and timeliness of digital human mouth shape synthesis.

[0012] In order to solve the problems existing in the prior art, the present invention provides a voice-driven digital human mouth shape synthesis algorithm with strong timeliness, including the following steps:

[0013] S1: Obtain and preprocess the data set;

[0014] S2: Design the model structure using an audio encoder, an image encoder, an audio feature selector, a feature fusion module, and an image decoder;

[0015] Among them, the image encoder is stacked by several layers of two-dimensional convolution, several batch normalizations, and several activation functions. The two-dimensional convolution uses depthwise convolution and pointwise convolution; the feature fusion module performs linear attention operation;

[0016] S3: Use pytorch as the training framework to train the digital mouth shape synthesis model. Among them, the calculation formula of the loss function of the model is as follows:

[0017] L total = λ s × L sync + λ c × L lip + λ r × L rec + λ p × L p + λ g × L gan + λ v × L vid ;

[0018] Among them, L total is the loss function value of the model, L sync is the lip synchronization loss value, λ s is the weight of the lip synchronization loss, L lip is the lip reading loss value, λ c is the weight of the lip reading loss, L rec is the reconstruction loss value, λ r is the weight of the reconstruction loss, L p is the perceptual loss value, λ p is the weight of the perceptual loss, L gan is the generative adversarial network loss value, λ g is the weight of the generative adversarial network loss, L vid is the video coherence loss value, λ v is the weight of the video coherence loss.

[0019] Optionally, in the digital mouth shape synthesis algorithm with strong timeliness driven by speech, the method for obtaining the dataset is as follows:

[0020] Record videos of people speaking for several hours, covering different ages, genders, face shapes, and skin colors, with a resolution of 720P and a frame rate of 25FPS. Cut the long video into 5-second short video segments, extract the audio of each short video segment to obtain a standard digital audio file with a sampling rate of 16K.

[0021] Optionally, in the speech-driven digital lip-sync synthesis algorithm with strong timeliness, the preprocessing method of the dataset is as follows:

[0022] Extract the facial key point coordinates in each frame of the short video clip, calculate the affine transformation matrix that can make the face stand upright using the key point coordinates, apply the affine transformation matrix to the frame image to obtain the image after the face stands upright, detect the facial rectangle coordinates in this image, and crop out a 256×256 facial image from the image using the facial rectangle coordinates.

[0023] Optionally, in the speech-driven digital lip-sync synthesis algorithm with strong timeliness,

[0024] The audio encoder inputs a standard digital audio file and outputs audio features with a tensor shape of [B, L, D], where B is the batch size, L is the time length, and D is the feature dimension.

[0025] Optionally, in the speech-driven digital lip-sync synthesis algorithm with strong timeliness, the image encoder performs downsampling by setting the stride to 2 in some two-dimensional convolutional layers;

[0026] The input of the image encoder is the concatenation of two images in the channel dimension, and the output is image features with a tensor shape of [B, H, W, C], where B represents the number of inputs, H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map.

[0027] Optionally, in the speech-driven digital lip-sync synthesis algorithm with strong timeliness,

[0028] The audio feature selector selects some audio features from the audio features output by the audio encoder to obtain audio features with a tensor shape of [B, F, D], where F is the selection parameter.

[0029] Optionally, in the speech-driven digital lip-sync synthesis algorithm with strong timeliness, the working method of the feature fusion module is as follows:

[0030] For the selected audio features with a tensor shape of [B, F, D], multiply the tensor of the audio features by the mouth-opening amplitude coefficient;

[0031] Use an adapter to map the features in the audio domain to a common space with the image features;

[0032] The tensor undergoes sinusoidal absolute position encoding;

[0033] Input the tensor encoded by sine absolute position into the multi-head self-attention module, which is used to learn the audio context information at different times. The fully connected layer in the multi-head self-attention module maps the audio features into Query, Key, and Value;

[0034] Perform linear attention operations on Query, Key, and Value as follows:

[0035] where Q is Query, K is Key, V is Value, m is the token index of Query, j is the token index of Key and Value, T is the transpose operation, and f is the total number of tokens;

[0036] Perform residual connection addition and layer normalization, and the resulting tensor is used as the Key and Value of the multi-head cross-attention module;

[0037] For image features, the image features with a tensor shape of [B, H, W, C] are reshaped, and the tensor shape changes from [B, H, W, C] to [B, H×W, C]. The changed tensor shape is used as the Query of the multi-head cross-attention module, so that the image features can extract information from the audio features and achieve multi-modal information fusion.

[0038] Optionally, in the digital human mouth shape synthesis algorithm with strong speech-driven timeliness, the model structure of the image decoder is stacked by M upsamplings, M convolutions, and M SPADEResnetBlocks, where M is a natural number, and is also connected with Leaky ReLU function, Conv2d function, and Sigmoid function.

[0039] Optionally, in the digital human mouth shape synthesis algorithm with strong speech-driven timeliness, use pytorch as the training framework to train the digital human mouth shape synthesis model as follows:

[0040] The training framework uses pytorch, the batch size is set to 16, the optimizer uses Adam, the learning rate regulator uses CosineAnnealingLR, and the initial learning rate is set to 1×10^-4;

[0041] During training, the parameters of the lip-sync expert and the lip-reading expert are frozen, and data augmentation uses the strategy of randomly changing the gamma and brightness of the pictures.

[0042] Optionally, in the digital human mouth shape synthesis algorithm with strong speech-driven timeliness,

[0043] The expression of the lip-sync loss value is as follows:

[0044]

[0045] Among them, a is the audio vector after the S-frame audio Mel spectrogram is encoded by the lip-sync expert, v is the lower half of the image generated by the S-frame model, the image vector after being encoded by the lip-sync expert, S is a natural number, and ε is a small positive number constant to prevent the denominator from being zero;

[0046] The expression of the lip-reading loss value is as follows:

[0047]

[0048] Among them, Y is the real text, is the text predicted by the lip reading expert according to the given video V, and P is the probability;

[0049] The expression of the reconstruction loss value is as follows:

[0050]

[0051] Among them, γ is the enhancement coefficient, M{mouth} is the number of mouth region masks, N is the number of elements in the tensor, i is the element serial number in the tensor, and v is the value of each element in the tensor;

[0052] The expression of the perceptual loss value is as follows:

[0053]

[0054] Among them, Vgg i represents the i-th layer in the VGG-19 network, Io is the frame image output by the model, Ir is the real reference frame image, W i represents the width of the i-th layer tensor, H i represents the height of the i-th layer tensor, C i represents the number of channels of the i-th layer tensor;

[0055] The expression of the generative adversarial network loss value is as follows:

[0056]

[0057] Among them, D j represents the GAN discriminator, E represents the expected value, Io is the frame image output by the model, and Ir is the real reference frame image;

[0058] The expression of the video coherence loss value is as follows:

[0059]

[0060] Among them, T is the pre-trained video autoencoder, and V pred is the number of consecutive video frames output by the model, and V gt is the number of real consecutive video frames.

[0061] Compared with the prior art, the present invention has the following advantages:

[0062] (1) Improved lip-sync accuracy: It can more accurately synchronize audio and lip movements.

[0063] (2) Accurate lip movements: It can synthesize lip movements consistent with the pronunciation.

[0064] (3) Clear facial images: It can synthesize clear skin texture materials, consistent with the skin in the input images.

[0065] (4) Multilingual support: The same model can simultaneously support lip driving in multiple languages, such as Chinese, English, Japanese, and French, easily handling the situation where multiple languages are mixed in the input audio.

[0066] (5) Low computational overhead: Through a carefully designed model architecture, the computational cost of the model is controlled, enabling this technology to be implemented in real-time on edge devices, suitable for resource-constrained devices such as mobile phones and tablets.

[0067] (6) Wide application prospects: This algorithm can be widely applied to digital humans in various industries (including but not limited to customer service representatives, live streamers, teachers, relatives, self-media bloggers, advertising spokespersons), with high commercial potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is a schematic diagram of the model structure provided by an embodiment of the present invention;

[0069] Figure 2 is a structural diagram of the feature fusion module provided by an embodiment of the present invention;

[0070] Figure 3 is a structural diagram of the audio encoder provided by an embodiment of the present invention;

[0071] Figure 4 is a structural diagram of the image decoder provided by an embodiment of the present invention;

[0072] Figure 5 is a structural diagram of the SPADEResnetBlock provided by an embodiment of the present invention;

[0073] Figure 6 is a structural diagram of the SPADE provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0074] The specific embodiments of the present invention will be described in more detail below in conjunction with the schematic diagrams. According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the attached drawings are all in a very simplified form and use non-precise scales, only for conveniently and clearly assisting in explaining the purpose of the embodiments of the present invention.

[0075] In the following text, if the methods described herein include a series of steps, the order of these steps presented herein is not necessarily the only order in which these steps can be executed, and some of the described steps can be omitted and / or some other steps not described herein can be added to the method.

[0076] In the prior art, there are many deficiencies in digital mouth shape synthesis.

[0077] To solve the problems existing in the prior art, the present invention provides a voice-driven digital mouth shape synthesis algorithm with strong timeliness, including the following steps:

[0078] S1: Obtain and preprocess the data set;

[0079] Specifically, the way of S11 to obtain the data set is as follows:

[0080] Record the speaking videos of several people for several hours, covering different ages, genders, face shapes and skin colors, with a resolution of 720P and a frame rate of 25FPS. Cut the long video into 5-second short video segments, extract the audio of each short video segment to obtain a standard digital audio file with a sampling rate of 16K.

[0081] The way of S12 to preprocess the data set is as follows:

[0082] Use mediapipe to extract the facial key point coordinates in each frame of the short video segment, calculate the affine transformation matrix that can make the face stand upright using the key point coordinates, apply the affine transformation matrix to the frame image to obtain the image of the face after standing upright, detect the face rectangle coordinates in this image using yolov8-face, and crop out a 256×256 face picture from the image using the face rectangle coordinates.

[0083] S2: Design the model structure using an audio encoder, an image encoder, an audio feature selector, a feature fusion module, and an image decoder, as Figure 1 shown;

[0084] Specifically, (1) The audio encoder inputs a standard digital audio file and outputs audio features with a tensor shape of [B, L, D], where B is the batch size, L is the time length, and D is the feature dimension. The audio encoder can select a pre-trained ASR model, such as wav2vec, whisper, HuBERT, etc. By using speech recognition models in different languages, it is possible to recognize lip drives in multiple languages. It is also possible to build an audio encoder from scratch for training, as Figure 3 shown. By using the audio encoder of pre-trained multilingual speech recognition, the algorithm of the present invention can handle input audio in multiple languages and maintain a certain degree of accuracy and stability.

[0085] (2) The image encoder is composed of several layers of two-dimensional convolution, several batch normalizations, and several activation functions stacked together, and downsampling is performed by setting the stride to 2 in some two-dimensional convolution layers; for lightweight, depthwise convolution and pointwise convolution are used instead of ordinary convolution in two-dimensional convolution.

[0086] The input of the image encoder is the concatenation of two pictures in the channel dimension, and the output is image features with a tensor shape of [B, H, W, C], where B represents the number of inputs, H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map.

[0087] Furthermore, the input of the image encoder in the inference stage is the concatenation of two pictures of 256×256×3 in the channel dimension, with a tensor shape of [B, H, W, 6], and the output is image features with a tensor shape of [B, H, W, C]. One of the two input pictures is the frame Ir of the current mouth shape to be synthesized, which provides lip reference for the model, and the other is the frame Ii with the lower half set to zero, which provides head pose prior. The main task of the model is to synthesize the lower half of the face with the correct mouth shape. In the training stage, its input is S groups of consecutive frames. Each group of consecutive frames is the same as in the inference stage. The purpose of inputting S groups simultaneously is that the model will finally output S pictures for the lip-sync expert to calculate the lip sync loss.

[0088] (3) The audio feature selector selects some audio features from the audio features output by the audio encoder to obtain audio features with a tensor shape of [B, F, D], where F is the selection parameter. Usually, there can be various forms of selection strategies.

[0089] In one embodiment, for example, with different spacings and different lengths, assuming the index of the current frame is i and the audio feature sequence is {a_0, a_1, …, a_{i - 1}, a_i, a_{i + 1}, …, a_L}, then the selected audio feature sequence can be a sequence with a spacing of 1 and a length of 11: {a_{i - 5}, …, a_{i - 1}, a_i, a_{i + 1}, …, a_{i + 5}}, or a sequence with a spacing of 2 and a length of 11: {a_{i - 10}, …, a_{i - 2}, a_i, a_{i + 2}, …, a_{i + 10}}, and so on.

[0090] (4) As Figure 2 shown, the feature fusion module works as follows:

[0091] Step a: For the selected audio features with a tensor shape of [B, F, D], multiply the tensor of the audio features by the mouth-opening amplitude coefficient. The mouth-opening amplitude coefficient is usually 1. When the mouth-opening amplitude coefficient > 1, it will make the mouth-opening amplitude finally output by the model larger, and vice versa, it will make the mouth-opening amplitude finally output by the model smaller; the present invention supports inputting a parameter for controlling the mouth-opening amplitude to achieve the control of the mouth-opening amplitude of the mouth shape.

[0092] Step b: Use an adapter to map the features in the audio domain to a space common with the image features. The adapter is mainly composed of several stacked fully connected layers and activation functions;

[0093] Step c: The tensor undergoes sine absolute position encoding, and the formula is as follows:

[0094]

[0095] where PE represents the position encoding, pos is the position index of the token in the sequence, i is the index of the channel dimension, and d model is the number of channels.

[0096] Step d: Input the tensor after sine absolute position encoding into the multi-head self-attention (MHSA = Multi Head Self Attention) module. The multi-head self-attention module is used to learn the audio context information at different times. The fully connected layers in the multi-head self-attention module map the audio features into Query, Key, and Value;

[0097] Step e: Perform a linear attention operation on Query, Key, and Value in the following manner:

[0098] Among them, Q is the Query, K is the Key, V is the Value, m is the token index of the Query, j is the token index of the Key and Value, T is the transpose operation, and f is the total number of tokens.

[0099] Step f: Perform residual connection addition and layer normalization, and the resulting tensor is used as the Key and Value of the multi-head cross-attention module;

[0100] Step g: For image features, the image features with a tensor shape of [B, H, W, C] are reshaped, and the tensor shape changes from [B, H, W, C] to [B, H×W, C]. The changed tensor shape is used as the Query of the multi-head cross-attention module, so that the image features can extract information from the audio features to achieve multi-modal information fusion.

[0101] (5) The model structure of the image decoder is as Figure 4 shown. The model structure of the image decoder is stacked by M upsamplings, M convolutions, and M SPADEResnetBlocks. M is a natural number. For example, M is 5. It is also connected with Leaky ReLU function, Conv2d function, and Sigmoid function. The internal structure of the SPADEResnetBlock is as Figure 5 shown. The SPADEResnetBlock contains SPADE, and the internal structure of SPADE is as Figure 6 shown. To reduce the computational complexity, the SPADEResnetBlock can be replaced by a convolution. The output of the image decoder is an image of 256×256×3.

[0102] The modality fusion module proposed by the present invention deeply fuses audio features and image features with each other, and also improves the synchronization and accuracy of lip-sync synthesis. The image decoder part of the model uses the SPADE module, which can synthesize clear pictures. The model does not use a large number of transformers and diffusions, so the computational cost is low and it is developed for real-time inference.

[0103] S3: Use PyTorch as the training framework to train the digital mouth shape synthesis model. Set the batch size to 16, use Adam as the optimizer, and use CosineAnnealingLR as the learning rate scheduler with an initial learning rate of 1×10^-4. During training, the parameters of the lip-sync expert and the lip-reading expert are frozen. Data augmentation uses the strategy of randomly changing the gamma and brightness of the images. The synchronization of mouth shape synthesis is improved by using the lip-sync expert. The accuracy of mouth shape synthesis is improved by using the lip-reading expert.

[0104] Among them, the calculation formula of the loss function of the model is as follows:

[0105] L total = λ s × L sync + λ c × L lip + λ r × L rec + λ p × L p + λ g × L gan + λ v × L vid ;

[0106] Among them, L total is the loss function value of the model, L sync is the lip-sync loss value, λ s is the weight of the lip-sync loss, L lip is the lip-reading loss value, λ c is the weight of the lip-reading loss, L rec is the reconstruction loss value, λ r is the weight of the reconstruction loss, L p is the perceptual loss value, λ p is the weight of the perceptual loss, L gan is the generative adversarial network loss value, λ g is the weight of the generative adversarial network loss, L vid is the video coherence loss value, λ v is the weight of the video coherence loss.

[0107] Specifically, (1) The expression of the lip-sync loss value is as follows:

[0108]

[0109] Among them, a is the audio vector after encoding the S-frame audio Mel spectrogram by the lip-sync expert, v is the lower half of the image generated by the S-frame model, the image vector after encoding by the lip-sync expert, S is a natural number, here S can be 5, ε is a small positive number constant to prevent the denominator from being zero, here ε takes 10^(-6); the lip-sync loss value is a contrastive loss, which can make the synchronous audio features and image features attract each other, while making the asynchronous audio features and image features repel each other.

[0110] (2) The expression of the lip-reading loss value is as follows:

[0111]

[0112] Among them, Y is the true text, is the text predicted by the lip reading expert according to the given video V, V is the video, and P is the probability;

[0113] (3) The expression of the reconstruction loss value is as follows:

[0114]

[0115] Among them, γ is the enhancement coefficient, M{mouth} is the number of mouth region masks, N is the number of elements in the tensor, i is the element serial number in the tensor, and v is the value of each element in the tensor; the reconstruction loss value is used to enhance the loss of the mouth region.

[0116] (4) The perceptual loss value is used to calculate the perceptual loss of the features of the VGG model at two image scales (3×H×W and 3×H / 2×W / 2), and its expression is as follows:

[0117]

[0118] Among them, Vgg i represents the i-th layer in the VGG-19 network, Io is the frame image output by the model, Ir is the true reference frame image, W i represents the width of the i-th layer tensor, H i represents the height of the i-th layer tensor, C i represents the number of channels of the i-th layer tensor;

[0119] (5) Regarding the generative adversarial network loss, it includes the discriminator loss and the generator loss, and the expression of the generative adversarial network loss value is as follows:

[0120]

[0121] The expressions of the generator loss and the discriminator loss are respectively:

[0122]

[0123] Among them, D j represents the GAN discriminator, which is a general term in the field of the generative adversarial network GAN. E represents the expected value, Io is the frame image output by the model, and Ir is the real reference frame image;

[0124] (6)Regarding the video coherence loss, the present invention uses the pre-trained video autoencoder VideoMAE V2 to encode the generated consecutive frames and the real consecutive frames, and then calculates their L1 norm. The expression is as follows:

[0125]

[0126] Among them, T is the pre-trained video autoencoder, and V pred is the number of consecutive video frames output by the model, and V gt is the number of real consecutive video frames. The video autoencoder is also a general term, and here VideoMAE V2 is used. By using the pre-trained video autoencoder as the supervision loss, the present invention increases the temporal consistency.

[0127] Compared with the prior art, the present invention has the following advantages:

[0128] (1)Improved lip-sync accuracy: It can more accurately synchronize audio and lip movements.

[0129] (2)Accurate lip movements: It can synthesize lip movements consistent with the pronunciation.

[0130] (3)Clear facial images: It can synthesize clear skin texture materials, which are consistent with the skin in the input images.

[0131] (4)Multi-language support: The same model can simultaneously support lip driving in multiple languages, such as Chinese, English, Japanese, and French, easily handling the situation where multiple languages are mixed in the input audio.

[0132] (5)Low computational overhead: Through a carefully designed model architecture, the computational cost of the model is controlled, enabling this technology to be implemented in real time on edge devices and being suitable for resource-constrained devices such as mobile phones and tablets.

[0133] (6)Wide application prospects: This algorithm can be widely applied to digital humans in various industries (including but not limited to customer service representatives, live streamers, teachers, relatives, self-media bloggers, advertising spokespersons), and has high commercial potential.

[0134] The above are only the preferred embodiments of the present invention and do not impose any limitation on the present invention. Any person skilled in the art, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification and other changes to the technical solution and technical content disclosed by the present invention, which are all within the content of the technical solution of the present invention and still fall within the protection scope of the present invention.

Claims

1. A voice-driven, time-sensitive digital population synthesis algorithm, characterized in that: The following steps are involved: S1: Obtain and preprocess the dataset; S2: Design the model structure using audio encoder, image encoder, audio feature selector, feature fusion module and image decoder; The image encoder is composed of several layers of two-dimensional convolution, several batch normalization and several activation functions. The two-dimensional convolution uses channel-by-channel convolution and point-by-point convolution. The feature fusion module uses linear attention operation. S3: Pytorch is used as the training framework to train the digital population synthesis model, where the calculation formula of the model's loss function is as follows: L total =λ s ×L sync +λ c ×L lip +λ r ×L rec +λ p ×L p +λ g ×L gan +λ v ×L vid ; Among them, L total is the loss function value of the model, L sync is the lip synchronization loss value, λ s is the weight of lip synchronization loss, L lip is the lip reading loss, λ c is the weight of lip reading loss, L rec is the reconstruction loss value, λ r is the weight of the reconstruction loss, L p is the perceptual loss value, λ p is the weight of perceptual loss, L gan is the loss value of the generated adversarial network, λ g is the weight of the generative adversarial network loss, L vid is the video coherence loss value, λ v is the weight of the video coherence loss.

2. The voice-driven, time-sensitive digital population synthesis algorithm as claimed in claim 1, characterized in that: The way to obtain the dataset is as follows: Record several hours of talking videos of people, covering different ages, genders, face shapes and skin colors, with a resolution of 720P and a frame rate of 25FPS. Cut the long videos into 5-second short video clips, extract the audio of each short video clip, and obtain a standard digital audio file with a sampling rate of 16K.

3. The speech-driven, time-sensitive digital population synthesis algorithm as claimed in claim 2, characterized in that: The dataset is preprocessed as follows: The coordinates of the key points of the face in each frame of the short video clip are extracted, and the coordinates of the key points are used to calculate an affine transformation matrix that can make the face stand at attention. The affine transformation matrix is ​​applied to the frame to obtain a picture with the face standing at attention, the coordinates of the face rectangular frame are detected in the picture, and a 256×256 face image is cropped from the picture using the coordinates of the face rectangular frame.

4. The speech-driven, time-sensitive digital population synthesis algorithm of claim 1, characterized in that: The audio encoder inputs a standard digital audio file and outputs audio features with a tensor shape of [B,L,D], where B is the batch size, L is the time length, and D is the feature dimension.

5. The speech-driven, time-sensitive digital population synthesis algorithm of claim 4, characterized in that: The image encoder performs downsampling by setting the stride to 2 in some 2D convolutional layers; The input of the image encoder is two pictures concatenated in the channel dimension, and the output is the image feature with a tensor shape of [B,H,W,C], where B represents the number of inputs, H represents the height of the feature map, W represents the width of the feature map, and C represents the number of channels of the feature map.

6. The speech-driven, time-sensitive digital population synthesis algorithm of claim 5, characterized in that: The audio feature selector selects some audio features from the audio features output by the audio encoder to obtain audio features with a tensor shape of [B, F, D], where F is the selection parameter.

7. The speech-driven, time-sensitive digital population synthesis algorithm of claim 6, characterized in that: The feature fusion module works as follows: For the audio feature whose tensor shape is [B, F, D], the tensor of the audio feature is multiplied by the mouth opening amplitude coefficient; Use an adapter to map audio domain features to a common space with image features; The tensor is sinusoidally absolute position encoded; Input the tensor encoded by the sinusoidal absolute position into a multi-head self-attention module, which is used to learn audio context information at different times. The fully connected layer in the multi-head self-attention module maps the audio features into Query, Key and Value; Perform linear attention operations on Query, Key, and Value as follows: Where Q is Query, K is Key, V is Value, m is the token index of Query, j is the token index of Key and Value, T is the transposition operation, and f is the total number of tokens; Perform residual connection addition and layer normalization, and the resulting tensors are used as the Key and Value of the multi-head cross attention module; For image features, the tensor shape of [B, H, W, C] is reshaped from [B, H, W, C] to [B, H × W, C]. The changed tensor shape is used as the query of the multi-head cross attention module to enable the image features to extract information from the audio features and realize multimodal information fusion.

8. The speech-driven, time-sensitive digital population synthesis algorithm of claim 7, characterized in that: The model structure of the image decoder is composed of M upsampling, M convolutions and M SPADEResnetBlock stacks, where M is a natural number, and is also connected with Leaky ReLU function, Conv2d function and Sigmoid function.

9. The speech-driven, time-sensitive digital population synthesis algorithm of claim 1, characterized in that: Use pytorch as the training framework to train the digital population synthesis model as follows: The training framework uses pytorch, the batch size is set to 16, the optimizer uses Adam, the learning rate regulator uses CosineAnnealingLR, and the initial learning rate is set to 1×10^-4; During training, the parameters of the lip-sync expert and lip-reading expert are frozen, and data augmentation uses a strategy of randomly changing the gamma and brightness of the image.

10. The speech-driven, time-sensitive digital population synthesis algorithm of claim 9, characterized in that: The expression for the lip sync loss value is as follows: Where a is the audio vector of the S-frame audio Mel-spectrogram after lip-sync expert encoding, v is the image vector of the lower half of the image generated by the S-frame model after lip-sync expert encoding, S is a natural number, and ε is a small positive constant to prevent the denominator from being zero; The expression of lip reading loss value is as follows: Among them, Y is the real text, is the text predicted by the lip reading expert based on the given video V, and P is the probability; The expression of the reconstruction loss value is as follows: Where γ is the enhancement coefficient, M{mouth} is the number of mouth area masks, N is the number of elements in the tensor, i is the element index in the tensor, and v is the value of each element in the tensor; The expression of the perceptual loss value is as follows: Among them, Vgg i represents the i-th layer in the VGG-19 network, Io is the frame image output by the model, Ir is the real reference frame image, and W i Represents the width of the i-th layer tensor, H i Represents the height of the i-th layer tensor, C i Represents the number of channels of the i-th layer tensor; The expression for the loss value of the generated adversarial network is as follows: Among them, D j represents the GAN discriminator, E represents the expected value, Io is the frame image output by the model, and Ir is the real reference frame image; The expression of the video coherence loss value is as follows: Where T is the pre-trained video autoencoder, V pred is the number of consecutive video frames output by the model, V gt It is the actual number of continuous video frames.

Images