Human face video generation method of human face dialogue generation model based on audio driving

By introducing the lip sound synchronization discrimination network and the Quality-Attention-Wav2Lip network into the face animation technology, the problems of exaggeration and blurring of lip shape are solved, the naturalness and synchronization effect of face videos are improved, and high-quality real face videos are generated.

CN119028369AActive Publication Date: 2024-11-26JINHUA INSTITUTE OF ZHEJIANG UNIVERSITY

Patent Information

Application Number
CN202411029024.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-26
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

Existing face animation technology has problems with exaggerated, blurred and artifacts in generating natural lip movement and head synchronization effects, and the details of the lower half of the face are unnatural, making it difficult to generate high-quality face videos.

Method used

Design a face dialogue generation model based on audio-driven, using a lip-sound synchronization discriminant network and a quality-Attention-Wav2Lip-based audio-driven lip-shaped network. Through the lip-sound synchronization training set and quality attention mechanism, lip-shaped generation is optimized, reducing exaggeration and non-examination. Natural lower profile to improve image quality.

Benefits of technology

It effectively improves the synchronization effect of lip shape generation and the overall face image quality, and generates real face videos with natural head movement and good lip sound synchronization effects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119028369A_ABST
    Figure CN119028369A_ABST
Patent Text Reader

Abstract

The invention discloses a face video generation method based on an audio-driven face dialogue generation model. The method comprises the following steps: establishing a lip sound synchronous judgment network and an audio-driven lip network based on quality attention; training the network by using the lip sound synchronization training set, and constructing an overall loss function of the audio-driven lip network based on quality attention according to a discrimination loss function of the lip sound synchronization discrimination network until the overall loss function converges to complete training; obtaining a reply audio according to the to-be-replied text or audio; and inputting the reply audio and the face image of the to-be-generated character into the trained network, outputting a face video of the current character when the current reply audio is read, and finally displaying the face video on a display. According to the method, the synchronization effect of lip shape generation and the image quality of the whole face are effectively improved, conversation with clients can be carried out, and the real face video with natural head movement and good lip sound synchronization effect is generated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a face video generation method, and in particular to a face video generation method based on an audio-driven face dialogue generation model. Background Art

[0002] Facial animation technology aims to generate a series of highly natural facial sequences given speech or text. The generated video must not only ensure the realism of the video frames and textures, but also the temporal continuity between video frames. Audio-driven facial animation technology is the focus of most researchers. People are sensitive to subtle changes in the face, and facial movements that are inconsistent with speech will make users feel uncomfortable. How to improve the realism of facial animation technology is still a difficult problem that needs to be solved in computer animation.

[0003] In order to solve the problem of unnatural facial movements of characters in the model-synthesized facial animation while speaking, Wav2Lip receives a series of image frame sequences as input, and generates facial animation by guiding the changes in lip movements through audio features while retaining the facial movement information of the characters in the original frame sequence. Compared with previous models, the Wav2Lip model has made significant progress in generating natural lip movements. The model can generate real face videos with natural head movement and good lip synchronization. However, there is still the problem of exaggerated lip shape generation and mismatch with previous frames. In addition, the pixel value of the generated lip shape is too low, resulting in blurring and artifacts. The detailed features and overall features generated in the lower half of the face are still unnatural, and even post-image enhancement cannot generate high-quality face sequences. Summary of the invention

[0004] In order to solve the problems existing in the background technology, the present invention provides a face video generation method based on an audio-driven face dialogue generation model.

[0005] The technical solution adopted by the present invention is:

[0006] The face video generation method based on the audio-driven face dialogue generation model of the present invention comprises:

[0007] S1) Establish a lip synchronization discrimination network and a quality-attention-based audio-driven lip shape network QAW (Quality-Attention-Wav2Lip).

[0008] S2) collecting several videos of people speaking and dividing each video into person audio and continuous frames of face images, and constructing each group of person audio and continuous frames of face images together as a lip synchronization training set.

[0009] S3) inputting the lip synchronization training set into the lip synchronization discrimination network for training, so that it can more accurately discriminate lip synchronization faces, until the discrimination loss function of the lip synchronization discrimination network converges, and a trained lip synchronization discrimination network is obtained. The discrimination loss function convergence of the lip synchronization discrimination network is constructed according to the synchronization discrimination weight file and similarity function output by the lip synchronization discrimination network during training.

[0010] S4) Inputting the lip synchronization training set into the quality-attention-based audio-driven lip-patterning network QAW for training, constructing the overall loss function of the quality-attention-based audio-driven lip-patterning network QAW based on the discriminant loss function, obtaining the overall loss value of the quality-attention-based audio-driven lip-patterning network QAW and using the gradient descent method to update the network parameters until the loss value of the overall loss function converges, thereby obtaining the trained quality-attention-based audio-driven lip-patterning network QAW.

[0011] S5) Input the text or audio to be replied into the dialogue system. For each piece of text or audio to be replied, the dialogue system processes and outputs the reply text of the text or audio to be replied, and converts the reply text into reply audio; input the reply audio and the facial image of the character to be generated into the trained quality-attention-based audio-driven lip network QAW. The trained quality-attention-based audio-driven lip network QAW processes and outputs the facial video of the current character when reading the current reply audio, and finally displays it on the display to achieve the generation of the facial video.

[0012] In the step S1), the lip synchronization discrimination network includes an audio encoder and an image encoder, the audio encoder includes a plurality of convolutional layers Conv and a plurality of two-dimensional convolutional Conv2d connected in sequence, the image encoder includes a plurality of two-dimensional convolutional Conv2d and a plurality of convolutional layers Conv connected in sequence, each convolutional layer Conv is composed of three two-dimensional convolutional Conv2d connected in sequence, each two-dimensional convolutional Conv2d includes a convolutional block connected in sequence, a batch normalization layer BN (BatchNormalization) and a ReLU activation function; for each group of character audio and continuous frame face images in the lip synchronization training set, the character audio is converted into a Mel-spectrogram and then input into the audio encoder for processing and then outputting a first audio feature vector, and the continuous frame face images are input into the image encoder to obtain a first lip shape feature vector; the cosine similarity method is used to obtain a similarity function between the first audio feature vector and the first lip shape feature vector to represent the matching degree of audio and video, as follows:

[0013]

[0014] Wherein, Similarity(A,B) represents the similarity function between the first audio feature vector A and the first lip shape feature vector B; A i and B i They respectively represent the i-th dimension first audio feature vector output by the audio encoder and the i-th dimension first lip shape feature vector output by the image encoder.

[0015] In the step S1), the audio-driven lip shape network QAW based on quality attention includes a face generator and a video quality discriminator connected in sequence, the lip synchronization training set is input into the face generator, the face generator outputs the generated face video after processing, the generated face video and the real face video are input into the video quality discriminator together, and the adversarial loss function and quality assessment loss function of the lower half of the face including the lip shape of the video quality discriminator are obtained. When the adversarial loss function of the video quality discriminator converges, the training of the video quality discriminator is completed; the reconstruction loss function is obtained according to the generated face video and the real face video, and finally the overall loss function of the audio-driven lip shape network QAW based on quality attention is constructed according to the discriminant loss function of the lip synchronization discriminant network, the quality assessment loss of the video quality discriminator, the FSIM loss function of the face dialogue generation model and the reconstruction loss function.

[0016] The quality-attention-based audio-driven lip network (QAW) extracts sound waveform information from audio and character lip movement information from video, and uses a generative adversarial network to train the model to obtain the mapping relationship between the two, so that the model can generate a series of lip movement sequences that match the input audio.

[0017] The face generator includes an audio encoder, a face encoder and a face decoder. The audio encoder includes a first convolution layer Conv, a second convolution layer Conv, a third convolution layer Conv, a fourth convolution layer Conv, a fifth convolution layer Conv, a sixth convolution layer Conv, and a first two-dimensional convolution Conv2d connected in sequence. Each convolution layer Conv is composed of three two-dimensional convolution Conv2d connected in sequence. Each two-dimensional convolution Conv2d includes a convolution block, a batch normalization layer BN and a ReLU activation function connected in sequence. The face encoder includes a second two-dimensional convolution Conv2d, a seventh convolution layer Conv, an eighth convolution layer Conv, a ninth convolution layer Conv, a first spatial self-attention mechanism SSA (SpatialSelfAttention), a tenth convolution layer Conv, a second spatial self-attention mechanism SSA, an eleventh convolution layer Conv, and a twelfth convolution layer Conv connected in sequence. , the thirteenth convolution layer Conv, the third two-dimensional convolution Conv2d and the fourth two-dimensional convolution Conv2d; the face decoder includes the fifth two-dimensional convolution Conv2d, the first spatial attention mechanism SA (SpatialAttention), the first fusion function Concat, the fourteenth convolution layer Conv, the second spatial attention mechanism SA, the second fusion function Concat, the fifteenth convolution layer Conv, the third spatial attention mechanism SA, the third fusion function Concat, the sixteenth convolution layer Conv, the fourth spatial attention mechanism SA, the fourth fusion function Concat, the seventeenth convolution layer Conv, the fifth spatial attention mechanism SA, the fifth fusion function Concat, the eighteenth convolution layer Conv, the sixth spatial attention mechanism SA, the sixth fusion function Concat, the nineteenth convolution layer Conv, the first position attention mechanism CA (Coordinate Attention), seventh spatial attention mechanism SA, seventh fusion function Concat, twentieth convolution layer Conv, second position attention mechanism CA, eighth spatial attention mechanism SA, eighth fusion function Concat, twenty-first convolution layer Conv, third position attention mechanism CA, ninth spatial attention mechanism SA, ninth fusion function Concat, sixth two-dimensional convolution Conv2d and seventh two-dimensional convolution Conv2d.

[0018] For each group of character audio and continuous frame face images in the lip synchronization training set, the character audio is converted into a Mel-spectrogram and then input into the audio encoder of the face generator for processing to output a second audio feature vector, the continuous frame face images are input into the face encoder for processing to output a second lip shape feature vector, the second audio feature vector is input into the fifth two-dimensional convolution Conv2d of the face decoder for processing, the output of the fifth two-dimensional convolution Conv2d and the second lip shape feature vector are input into the first spatial attention mechanism SA of the face encoder for processing, the output of the first spatial attention mechanism SA and the second lip shape feature vector are input into the first fusion function Concat for processing and then input into the fourteenth convolution layer Conv for processing, the output of the fourteenth convolutional layer Conv and the output of the thirteenth convolutional layer Conv are input into the second spatial attention mechanism SA for processing, the output of the second spatial attention mechanism SA and the output of the thirteenth convolutional layer Conv are input into the second fusion function Concat for processing and then output to the fifteenth convolutional layer Conv for processing, the output of the fifteenth convolutional layer Conv and the output of the twelfth convolutional layer Conv are input into the third spatial attention mechanism SA for processing, the output of the third spatial attention mechanism SA and the output of the twelfth convolutional layer Conv are input into the third fusion function Concat for processing and then output to the sixteenth convolutional layer Conv for processing, The output of the sixteenth convolutional layer Conv and the output of the eleventh convolutional layer Conv are input into the fourth spatial attention mechanism SA for processing. The output of the fourth spatial attention mechanism SA and the output of the eleventh convolutional layer Conv are input into the fourth fusion function Concat for processing and then output to the seventeenth convolutional layer Conv for processing. The output of the seventeenth convolutional layer Conv and the output of the tenth convolutional layer Conv are input into the fifth spatial attention mechanism SA for processing. The output of the fifth spatial attention mechanism SA and the output of the tenth convolutional layer Conv are input into the fifth fusion function Concat for processing and then output to the eighteenth convolutional layer Conv for processing. The output of the eighteenth convolutional layer Conv The output of the sixth spatial attention mechanism SA and the output of the ninth convolutional layer Conv are input together to the sixth spatial attention mechanism SA for processing. The output of the sixth spatial attention mechanism SA and the output of the ninth convolutional layer Conv are input together to the sixth fusion function Concat for processing and then output to the nineteenth convolutional layer Conv and the first position attention mechanism CA for processing in sequence. The output of the first position attention mechanism CA and the output of the eighth convolutional layer Conv are input together to the seventh spatial attention mechanism SA for processing. The output of the seventh spatial attention mechanism SA and the output of the eighth convolutional layer Conv are input together to the seventh fusion function Concat for processing and then output to the twentieth convolutional layer Conv and the second position attention mechanism CA for processing.The output of the second position attention mechanism CA and the output of the seventh convolution layer Conv are input into the eighth spatial attention mechanism SA for processing. The output of the eighth spatial attention mechanism SA and the output of the seventh convolution layer Conv are input into the eighth fusion function Concat for processing and then output to the twenty-first convolution layer Conv and the third position attention mechanism CA for processing. The output of the third position attention mechanism CA and the output of the second two-dimensional convolution Conv2d are input into the ninth spatial attention mechanism SA for processing. The output of the ninth spatial attention mechanism SA and the output of the second two-dimensional convolution Conv2d are input into the ninth fusion function Concat for processing and then output to the sixth two-dimensional convolution Conv2d and the seventh two-dimensional convolution Conv2d for processing, and the final output is used as the output of the face generator. ,

[0019] a) The face encoder is composed of convolutional layers with residuals. Its main task is to encode the face reference frame F in the face video. The reference frame F obtained in the face video needs to be concatenated with the pose frame that covers the lower half of the face in the channel dimension. In particular, in order to make the face generation more continuous and reduce the appearance of exaggerated lip shape and unnatural lower contour, the reference frame is generally the first T frames of the training frame. If the training frame is less than T frames before, it is randomly selected. b) The audio encoder is a standard convolutional neural network. Its job is to encode the corresponding speech segment. Its input is the Mel-frequency Cepstral Coefficient (MFCC) heat map of size M×T×1, and its output is an audio embedding of size h. c) The face decoder connects the feature vector generated by the audio encoder of size h, the fused feature vector and the face embedding to produce a joint embedding of size 2×h. The joint embedding is used as the input of the face decoder. The face decoder is composed of convolutional layers with residuals and up-sampled transposed convolutions.

[0020] The spatial attention mechanism SA has two input terminals, namely the face feature vector input and the audio feature vector input; in addition, it also includes the average pooling layer AvgPool, the maximum pooling layer MaxPool, the fusion function Concat, the two-dimensional convolution Conv2d with a convolution kernel size of 7×7, the activation function sigmoid and the output terminal. After the vector is input to the input terminal, the face feature vector is processed by the average pooling layer AvgPool and the maximum pooling layer MaxPool respectively, and the processed output is then input to the fusion function Concat for splicing in the channel dimension. The spliced ​​vector is then input to the two-dimensional convolution Conv2d and the activation function sigmoid for processing. The output of the activation function sigmoid is multiplied element by element with the audio feature vector and then added, and finally output to the output terminal. In particular, the dimension of the final output vector is consistent with the audio feature vector.

[0021] The spatial self-attention mechanism SSA includes an input end, an average pooling layer AvgPool, a maximum pooling layer MaxPool, a fusion function Concat, a two-dimensional convolution Conv2d with a convolution kernel size of 7×7, an activation function sigmoid, and an output end. After the feature vector is input to the input end, it is processed by the average pooling layer AvgPool and the maximum pooling layer MaxPool respectively, and then input to the fusion function Concat for splicing in the channel dimension. The spliced ​​vector is sequentially input to the two-dimensional convolution Conv2d with a convolution kernel size of 7×7 and the activation function Sigmoid for processing; the output of the Sigmoid layer is multiplied element by element with the feature vector of the input end and then added, and finally output to the output end.

[0022] The position attention mechanism CA includes an input end, two average pooling layers AvgPool, a fusion function Concat, a two-dimensional convolution Conv2d, two convolution layers Conv, two activation functions Sigmoid and an output end. After the feature vector is input to the input end, it passes through two average pooling layers AvgPool in the width direction and the height direction respectively, and then the output is input to the fusion function Concat together for splicing in the channel dimension. The spliced ​​vector is input to the two-dimensional convolution Conv2d. The output of the two-dimensional convolution Conv2d is input to the two convolution layers Conv in the H direction and the W direction according to the feature vector of size H in the width direction and the feature vector of size W. The output of the two convolution layers Conv is then input to each activation function Sigmoid in the chain. The output of the two activation functions Sigmoid is multiplied by the feature vector of the input end and then added to the feature vector of the input end, and finally output to the output end.

[0023] The video quality discriminator includes a plurality of convolution groups connected in sequence, each convolution group includes convolution blocks connected in sequence and an activation function LeakyReLU. In a specific implementation, the video quality discriminator includes 17 convolution groups connected in sequence.

[0024] The overall loss function L all The details are as follows:

[0025] L all =(1-w s -w g -w f )L recon +w s ·L sync +w g ·L gen +w f ·L FSIM

[0026] Among them, w s 、w g and w f They represent the synchronization penalty weight, adversarial loss weight and FSIM loss weight respectively; L recon , L sync , L gen and L FSIM They respectively represent the reconstruction loss function, the discriminant loss function of the trained lip synchronization discriminant network, the quality assessment loss function of the lower half of the face including the lip shape of the video quality discriminator, and the quality assessment loss index loss FSIM of the face dialogue generation model.

[0027] The discriminant loss function of the lip synchronization discriminant network is obtained according to the synchronization discriminant weight file, and the generated lip shape and audio are input into the lip synchronization discriminant network loaded with the synchronization discriminant weight file to obtain the audio feature vector and the lip shape feature vector respectively. Then, the discriminant loss function of the lip synchronization discriminant network is further calculated using the similarity function.

[0028] The loss function generated by the video quality discriminator consists of two parts: a) the adversarial loss of the generator, that is, the original adversarial loss, the generator hopes that the generated video will be considered real by the discriminator; b) the adversarial loss of the discriminator, the discriminator can correctly distinguish between real videos and generated videos.

[0029] The reconstruction loss function L recon The loss function is composed of the loss obtained based on the Manhattan L1 distance between the generated face video and the corresponding face image of each frame in the real face video.

[0030] The quality assessment loss function L of the lower half of the face including the lip shape of the video quality discriminator is gen The details are as follows:

[0031]

[0032] The adversarial loss function L of the lower half of the face including the lip shape of the video quality discriminator disc The details are as follows:

[0033]

[0034] Among them, x i~Lg represents the i-th number of the one-dimensional vector generated by the generated face video input to the video quality discriminator, N represents the total number of elements in the one-dimensional vector generated by the video quality discriminator, and D() represents the probability that the video quality discriminator judges the input to be true; x i~LG Represents the i-th number of the one-dimensional vector generated by the real face video input to the video quality discriminator.

[0035] The quality assessment loss index loss L of the face dialogue generation model FSIM The details are as follows:

[0036]

[0037] Among them, G g and G G They represent the gradient amplitude of the corresponding video frames in the generated face video and the real face video calculated using the Sobel operator, respectively. g and P G They represent the phase consistency of the corresponding video frames calculated according to the gradient amplitude of the corresponding video frames in the generated face video and the real face video respectively; T1 and T2 represent the first and second mediation parameters, which are used to control the weights of phase consistency and gradient amplitude respectively; feature similarity index (FSIM, featuresimilarity index mersure) is the quality assessment loss index loss L FSIM The second half of the face video. L represents the video frame in the generated face video or the real face video, Sobel x and Sobel y They represent the gradient maps of the Sobel operator in the horizontal X direction and the vertical Y direction of the image respectively, and ∈ represents the preset minimum value.

[0038] FSLM loss L FSIM , the image quality evaluation of the image features similarity between the video generated by the generator and the real video is performed, with the aim of making some detail features of the generated video similar to the real features.

[0039] In the step S5), the dialogue system includes a natural language processing module and a chatgpt module. After the text to be replied is input into the dialogue system, it is directly input into the chatgpt module to generate the reply text. After the audio to be replied is input into the dialogue system, it is first input into the natural language processing module for processing and then converted into the text in the audio to be replied, and then input into the chatgpt module to generate the reply text; each reply text is input into the text-to-speech conversion model TTS (Text to Speech) for processing and then the reply audio is output. The TTS model is a model based on deep learning and can generate high-quality and natural speech.

[0040] The reply audio and the facial image of the character to be generated are input into the trained quality attention-based audio-driven lip network QAW face generator. After processing, the trained face generator outputs the facial video of the current character when reading the current reply audio.

[0041] The present invention designs a lip synchronization discrimination network and an audio-driven lip shape network based on Quality-Attention-Wav2lip; the method selects the first T frames of the pose frame as the reference frame F of the face decoder in the Quality-Attention-Wav2Lip network based on quality attention audio-driven lip shape, so that the face generation is more continuous and the appearance of exaggerated lip shape and unnatural lower contour is reduced. The input and output facial feature dimensions of the network are increased to 384×384 to ensure the clarity of lip shape generation. In addition, a spatial self-attention structure is added to the face encoder to increase the self-attention of the channel to obtain sufficiently effective reference frame features; position attention is added to the face decoder part to encode the channel relationship and long-term dependency through precise position information, thereby improving the image quality and connection between the lower half of the face and the overall facial features. In view of artifacts and unnatural lip shapes, the present invention introduces FSLM loss to optimize image generation and restoration.

[0042] The beneficial effects of the present invention are:

[0043] The method of the invention effectively improves the synchronization effect of lip shape generation and the image quality of the overall human face, and is capable of having conversations with customers, aiming to generate a real human face video with natural head movement and good lip-sync effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of the method of the present invention;

[0045] Figure 2 is a structural diagram of a lip sync discrimination network of the present invention;

[0046] Figure 3The structure diagram of the quality attention-based audio-driven lip network QAW of the present invention;

[0047] Figure 4 This is a structural diagram of the spatial attention mechanism SA, spatial self-attention mechanism SSA and position attention mechanism CA of the present invention. DETAILED DESCRIPTION

[0048] The following describes the embodiments of the present invention through specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0049] like Figure 1 As shown, the face video generation method based on the audio-driven face dialogue generation model of the present invention is specifically as follows:

[0050] S1) Establish a lip synchronization discrimination network and a quality attention-based audio-driven lip shape network QAW.

[0051] like Figure 2 As shown, the lip synchronization discrimination network includes an audio encoder and an image encoder, the audio encoder includes several layers of convolutional layers Conv and several two-dimensional convolutional layers Conv2d connected in sequence, the image encoder includes several two-dimensional convolutional layers Conv2d and several layers of convolutional layers Conv connected in sequence, each convolutional layer Conv is composed of three two-dimensional convolutional layers Conv2d connected in sequence, and each two-dimensional convolutional layer Conv2d includes convolutional blocks, batch normalization layers BN and ReLU activation functions connected in sequence; for each group of character audio and continuous frame face images in the lip synchronization training set, the character audio is converted into a Mel spectrum map and then input into the audio encoder for processing and then outputting a first audio feature vector, and the continuous frame face images are input into the image encoder to obtain a first lip shape feature vector; the cosine similarity method is used to obtain the similarity function between the first audio feature vector and the first lip shape feature vector to represent the matching degree of audio and video, as follows:

[0052]

[0053] Wherein, Similarity(A,B) represents the similarity function between the first audio feature vector A and the first lip shape feature vector B; A i and B iThey respectively represent the i-th dimension first audio feature vector output by the audio encoder and the i-th dimension first lip shape feature vector output by the image encoder, and n=1024.

[0054] The lip sync discrimination network can help the face generator generate more accurate lip sync face videos. In the specific implementation, the audio encoder includes five layers of convolutional layers Conv and two layers of two-dimensional convolution Conv2d connected in sequence, and the dimensions of the input audio are changed from (80,16,1) to (80,16,32), (27,16,64), (9,6,128), (3,3,256), (3,3,512), (1,1,1024), (1,1,1024); the image encoder includes one layer of two-dimensional convolution Conv2d and seven layers of convolution Conv2d connected in sequence. In the convolution layer Conv, the dimensions of the input image change from (384,384,3*T) to (384,384,16), (192,192,32), (96,96,64), (48,48,128), (24,24,256), (12,12,512), (6,6,1024), and (1,1,1024), where T represents the duration of the character audio and the number of consecutive frames of the face image, T=5, and n=1024.

[0055] like Figure 3 As shown, the audio-driven lip shape network QAW based on quality attention includes a face generator and a video quality discriminator connected in sequence, the lip synchronization training set is input into the face generator, the face generator outputs the generated face video after processing, the generated face video and the real face video are input into the video quality discriminator together, and the adversarial loss function and quality assessment loss function of the lower half of the face including the lip shape of the video quality discriminator are obtained. When the adversarial loss function of the video quality discriminator converges, the training of the video quality discriminator is completed; the reconstruction loss function is obtained according to the generated face video and the real face video, and finally the overall loss function of the audio-driven lip shape network QAW based on quality attention is constructed according to the discriminant loss function of the lip synchronization discriminant network, the quality assessment loss of the video quality discriminator, the FSIM loss function of the face dialogue generation model and the reconstruction loss function.

[0056] The quality-attention-based audio-driven lip network (QAW) extracts sound waveform information from audio and character lip movement information from video, and uses a generative adversarial network to train the model to obtain the mapping relationship between the two, so that the model can generate a series of lip movement sequences that match the input audio.

[0057] The face generator includes an audio encoder, a face encoder and a face decoder. The audio encoder includes a first convolution layer Conv, a second convolution layer Conv, a third convolution layer Conv, a fourth convolution layer Conv, a fifth convolution layer Conv, a sixth convolution layer Conv, and a first two-dimensional convolution Conv2d connected in sequence. Each convolution layer Conv is composed of three two-dimensional convolution Conv2d connected in sequence. Each two-dimensional convolution Conv2d includes a convolution block, a batch normalization layer BN and a ReLU activation function connected in sequence. The face encoder includes a second two-dimensional convolution Conv2d, a seventh convolution layer Conv, an eighth convolution layer Conv, a ninth convolution layer Conv, a first spatial self-attention mechanism SSA, a tenth convolution layer Conv, a second spatial self-attention mechanism SSA, an eleventh convolution layer Conv, a twelfth convolution layer Conv, a thirteenth convolution layer Conv, a third two-dimensional convolution Conv2d and a fourth two-dimensional convolution Conv2d connected in sequence. The face decoder includes a fifth two-dimensional convolution Conv2d, a first spatial attention mechanism SA, the first fusion function Concat, the fourteenth convolution layer Conv, the second spatial attention mechanism SA, the second fusion function Concat, the fifteenth convolution layer Conv, the third spatial attention mechanism SA, the third fusion function Concat, the sixteenth convolution layer Conv, the fourth spatial attention mechanism SA, the fourth fusion function Concat, the seventeenth convolution layer Conv, the fifth spatial attention mechanism SA, the fifth fusion function Concat, the eighteenth convolution layer Conv, the sixth spatial attention mechanism SA, the sixth fusion function Concat, the nineteenth convolution layer Conv, the first position attention mechanism CA, the seventh spatial attention mechanism SA, the seventh fusion function Concat, the twentieth convolution layer Conv, the second position attention mechanism CA, the eighth spatial attention mechanism SA, the eighth fusion function Concat, the twenty-first convolution layer Conv, the third position attention mechanism CA, the ninth spatial attention mechanism SA, the ninth fusion function Concat, the sixth two-dimensional convolution Conv2d and the seventh two-dimensional convolution Conv2d.

[0058] For each group of character audio and continuous frame face images in the lip synchronization training set, the character audio is converted into a Mel-spectrogram and then input into the audio encoder of the face generator for processing to output a second audio feature vector, the continuous frame face images are input into the face encoder for processing to output a second lip shape feature vector, the second audio feature vector is input into the fifth two-dimensional convolution Conv2d of the face decoder for processing, the output of the fifth two-dimensional convolution Conv2d and the second lip shape feature vector are input into the first spatial attention mechanism SA of the face encoder for processing, the output of the first spatial attention mechanism SA and the second lip shape feature vector are input into the first fusion function Concat for processing and then input into the fourteenth convolution layer Conv for processing, the output of the fourteenth convolutional layer Conv and the output of the thirteenth convolutional layer Conv are input into the second spatial attention mechanism SA for processing, the output of the second spatial attention mechanism SA and the output of the thirteenth convolutional layer Conv are input into the second fusion function Concat for processing and then output to the fifteenth convolutional layer Conv for processing, the output of the fifteenth convolutional layer Conv and the output of the twelfth convolutional layer Conv are input into the third spatial attention mechanism SA for processing, the output of the third spatial attention mechanism SA and the output of the twelfth convolutional layer Conv are input into the third fusion function Concat for processing and then output to the sixteenth convolutional layer Conv for processing, The output of the sixteenth convolutional layer Conv and the output of the eleventh convolutional layer Conv are input into the fourth spatial attention mechanism SA for processing. The output of the fourth spatial attention mechanism SA and the output of the eleventh convolutional layer Conv are input into the fourth fusion function Concat for processing and then output to the seventeenth convolutional layer Conv for processing. The output of the seventeenth convolutional layer Conv and the output of the tenth convolutional layer Conv are input into the fifth spatial attention mechanism SA for processing. The output of the fifth spatial attention mechanism SA and the output of the tenth convolutional layer Conv are input into the fifth fusion function Concat for processing and then output to the eighteenth convolutional layer Conv for processing. The output of the eighteenth convolutional layer Conv The output of the sixth spatial attention mechanism SA and the output of the ninth convolutional layer Conv are input together to the sixth spatial attention mechanism SA for processing. The output of the sixth spatial attention mechanism SA and the output of the ninth convolutional layer Conv are input together to the sixth fusion function Concat for processing and then output to the nineteenth convolutional layer Conv and the first position attention mechanism CA for processing in sequence. The output of the first position attention mechanism CA and the output of the eighth convolutional layer Conv are input together to the seventh spatial attention mechanism SA for processing. The output of the seventh spatial attention mechanism SA and the output of the eighth convolutional layer Conv are input together to the seventh fusion function Concat for processing and then output to the twentieth convolutional layer Conv and the second position attention mechanism CA for processing.The output of the second position attention mechanism CA and the output of the seventh convolution layer Conv are input into the eighth spatial attention mechanism SA for processing. The output of the eighth spatial attention mechanism SA and the output of the seventh convolution layer Conv are input into the eighth fusion function Concat for processing and then output to the twenty-first convolution layer Conv and the third position attention mechanism CA for processing. The output of the third position attention mechanism CA and the output of the second two-dimensional convolution Conv2d are input into the ninth spatial attention mechanism SA for processing. The output of the ninth spatial attention mechanism SA and the output of the second two-dimensional convolution Conv2d are input into the ninth fusion function Concat for processing and then output to the sixth two-dimensional convolution Conv2d and the seventh two-dimensional convolution Conv2d for processing, and the final output is used as the output of the face generator. ,

[0059] a) The face encoder is composed of convolutional layers with residuals. Its main task is to encode the face reference frame F in the face video. The reference frame F obtained in the face video needs to be concatenated with the pose frame that covers the lower half of the face in the channel dimension. In particular, in order to make the face generation more continuous and reduce the appearance of exaggerated lip shape and unnatural lower contour, the reference frame is generally the first T frames of the training frame. If there are less than T frames before the training frame, it is randomly selected. b) The audio encoder is a standard convolutional neural network. Its job is to encode the corresponding speech segment. Its input is the Mel frequency cepstral coefficient heat map MFCC of size M×T×1, and its output is the audio embedding of size h. c) The face decoder takes the feature vector generated by the audio encoder of size h, connects the fused feature vector with the face embedding, and generates a joint embedding of size 2×h. The joint embedding is used as the input of the face decoder. The face decoder is composed of convolutional layers with residuals and up-sampled transposed convolutions.

[0060] The dimensions of the audio input to the audio encoder change from (80,16,1) to (80,16,32), (27,16,64), (9,6,128), (3,3,256), (3,3,512), (1,1,1024), and (1,1,1024); the dimensions of the image input to the face encoder change from (384,384,6) to (384,384,8), (192,192,16), (96,96,32), (48,48,64), (24,24,128), (12,12,256), (6,6,512), (3,3,1024), (1,1,1024), (1,1,1024); the input dimension of the face decoder changes from (1,1,1024) to (1,1,1024), (1,1,1024), (1,1,2048), (3,3,1024), (3,3,1024), (3,3,204 8)、(6,6,1024)、(6,6,1024)、(6,6,1536)、(12,12,768)、(12,12,768)、(12,12,1024)、(24,24,512)、(24,24,512)、(24,24,640)、(48,48,256)、(48,48,256)、(48,48,320)、(96,96,128)、(96,96,128 The third dimension of a picture is 3, representing RGB, and the third dimension is 6, which means two pictures, that is, one is the face frame and reference frame of the lower half of the face without the need to generate the corresponding lip shape.

[0061] like Figure 4As shown in the figure, the spatial attention mechanism SA has two input terminals, namely the face feature vector input and the audio feature vector input; in addition, it also includes the average pooling layer AvgPool, the maximum pooling layer MaxPool, the fusion function Concat, the two-dimensional convolution Conv2d with a convolution kernel size of 7×7, the activation function sigmoid and the output terminal. After the vector is input to the input terminal, the face feature vector is processed by the average pooling layer AvgPool and the maximum pooling layer MaxPool respectively, and the processed output is then input to the fusion function Concat for splicing in the channel dimension. The spliced ​​vector is then input to the two-dimensional convolution Conv2d and the activation function sigmoid for processing. The output of the activation function sigmoid is multiplied element by element with the audio feature vector and then added, and finally output to the output terminal. In particular, the dimension of the final output vector is consistent with the audio feature vector.

[0062] The spatial self-attention mechanism SSA includes an input end, an average pooling layer AvgPool, a maximum pooling layer MaxPool, a fusion function Concat, a two-dimensional convolution Conv2d with a convolution kernel size of 7×7, an activation function sigmoid, and an output end. After the feature vector is input to the input end, it is processed by the average pooling layer AvgPool and the maximum pooling layer MaxPool respectively, and then input to the fusion function Concat for splicing in the channel dimension. The spliced ​​vector is sequentially input to the two-dimensional convolution Conv2d with a convolution kernel size of 7×7 and the activation function Sigmoid for processing; the output of the Sigmoid layer is multiplied element by element with the feature vector of the input end and then added, and finally output to the output end.

[0063] The position attention mechanism CA includes an input end, two average pooling layers AvgPool, a fusion function Concat, a two-dimensional convolution Conv2d, two convolution layers Conv, two activation functions Sigmoid and an output end. After the feature vector is input to the input end, it passes through two average pooling layers AvgPool in the width direction and the height direction respectively, and then the output is input to the fusion function Concat together for splicing in the channel dimension. The spliced ​​vector is input to the two-dimensional convolution Conv2d. The output of the two-dimensional convolution Conv2d is input to the two convolution layers Conv in the H direction and the W direction according to the feature vector of size H in the width direction and the feature vector of size W. The output of the two convolution layers Conv is then input to each activation function Sigmoid in the chain. The output of the two activation functions Sigmoid is multiplied by the feature vector of the input end and then added to the feature vector of the input end, and finally output to the output end.

[0064] The video quality discriminator includes several convolution groups connected in sequence, each of which includes convolution blocks connected in sequence and activation function LeakyReLU. In specific implementation, the video quality discriminator includes 17 convolution groups connected in sequence. When training the generator and discriminator, the Adam optimizer is used with an initial learning rate of 1E–4.

[0065] The overall loss function L all The details are as follows:

[0066] L all =(1-w s -w g -w f )L recon +w s ·L sync +w g ·L gen +w f ·L FSIM

[0067] Among them, w s 、w g and w f They represent the synchronization penalty weight, adversarial loss weight, and FSIM loss weight, respectively, and are set to 0.03, 0.04, and 0.03 by default; L recon , L sync , L gen and L FSIM They respectively represent the reconstruction loss function, the discriminant loss function of the trained lip synchronization discriminant network, the quality assessment loss function of the lower half of the face including the lip shape of the video quality discriminator, and the quality assessment loss index loss FSIM of the face dialogue generation model.

[0068] The discriminant loss function of the lip synchronization discriminant network is obtained according to the synchronization discriminant weight file, and the generated lip shape and audio are input into the lip synchronization discriminant network loaded with the synchronization discriminant weight file to obtain the audio feature vector and the lip shape feature vector respectively. Then, the discriminant loss function of the lip synchronization discriminant network is further calculated using the similarity function.

[0069] The loss function generated by the video quality discriminator consists of two parts: a) the adversarial loss of the generator, that is, the original adversarial loss, the generator hopes that the generated video will be considered real by the discriminator; b) the adversarial loss of the discriminator, the discriminator can correctly distinguish between real videos and generated videos.

[0070] Reconstruction loss function L recon The loss function is composed of the loss obtained based on the Manhattan L1 distance between the generated face video and the corresponding face image of each frame in the real face video.

[0071] The quality assessment loss function L of the lower half of the face including the lip shape of the video quality discriminator gen The details are as follows:

[0072]

[0073] The adversarial loss function L of the lower half of the face including the lip shape of the video quality discriminator disc The details are as follows:

[0074]

[0075] Among them, x i~Lg represents the i-th number of the one-dimensional vector generated by the generated face video input to the video quality discriminator, N represents the total number of elements in the one-dimensional vector generated by the video quality discriminator, and D() represents the probability that the video quality discriminator judges the input to be true; x i~LG Represents the i-th number of the one-dimensional vector generated by the real face video input to the video quality discriminator.

[0076] Quality Assessment Loss Index Loss L for Face Dialogue Generation Model FSIM The details are as follows:

[0077]

[0078]

[0079] Among them, G g and G G They represent the gradient amplitude of the corresponding video frames in the generated face video and the real face video calculated using the Sobel operator, respectively. g and P G They represent the phase consistency of the corresponding video frames calculated according to the gradient amplitude of the corresponding video frames in the generated face video and the real face video respectively; T1 and T2 represent the first and second mediation parameters, which are used to control the weights of phase consistency and gradient amplitude, and are set to 0.85 and 160 by default; the feature similarity index (FSIM) is the quality assessment loss index loss L FSIM The second half of the face video. L represents the video frame in the generated face video or the real face video, Sobel x and Sobel y They represent the gradient maps of the Sobel operator in the horizontal X direction and the vertical Y direction of the image respectively, and ∈ represents the preset minimum value.

[0080] FSLM loss L FSIM, the image quality evaluation of the image features similarity between the video generated by the generator and the real video is performed, with the aim of making some detail features of the generated video similar to the real features.

[0081] S2) collecting several videos of people speaking and dividing each video into person audio and continuous frames of face images, and constructing each group of person audio and continuous frames of face images together as a lip synchronization training set.

[0082] S3) inputting the lip synchronization training set into the lip synchronization discrimination network for training, so that it can more accurately discriminate lip synchronization faces, until the discrimination loss function of the lip synchronization discrimination network converges, and a trained lip synchronization discrimination network is obtained. The discrimination loss function convergence of the lip synchronization discrimination network is constructed according to the synchronization discrimination weight file and similarity function output by the lip synchronization discrimination network during training.

[0083] S4) Inputting the lip synchronization training set into the quality-attention-based audio-driven lip-patterning network QAW for training, constructing the overall loss function of the quality-attention-based audio-driven lip-patterning network QAW based on the discriminant loss function, obtaining the overall loss value of the quality-attention-based audio-driven lip-patterning network QAW and using the gradient descent method to update the network parameters until the loss value of the overall loss function converges, thereby obtaining the trained quality-attention-based audio-driven lip-patterning network QAW.

[0084] S5) Input the text or audio to be replied into the dialogue system. For each piece of text or audio to be replied, the dialogue system processes and outputs the reply text of the text or audio to be replied, and converts the reply text into reply audio; input the reply audio and the facial image of the character to be generated into the trained quality-attention-based audio-driven lip network QAW. The trained quality-attention-based audio-driven lip network QAW processes and outputs the facial video of the current character when reading the current reply audio, and finally displays it on the display to achieve the generation of the facial video.

[0085] The dialogue system includes a natural language processing module and a chatgpt module. After the text to be replied is input into the dialogue system, it is directly input into the chatgpt module to generate the reply text. After the audio to be replied is input into the dialogue system, it is first input into the natural language processing module for processing and then converted into the text in the audio to be replied, and then input into the chatgpt module to generate the reply text. Each reply text is input into the text-to-speech conversion model TTS for processing and then the reply audio is output. The TTS model is a deep learning-based model that can generate high-quality, natural speech.

[0086] The reply audio and the facial image of the character to be generated are input into the trained quality attention-based audio-driven lip network QAW face generator. After processing, the trained face generator outputs the facial video of the current character when reading the current reply audio.

[0087] The embodiments described above are merely descriptions of preferred implementations of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope of the present invention.

Claims

1. A method for generating face videos based on an audio-driven face dialogue generation model, characterized in that: include: S1) Establishing a lip synchronization discrimination network and a quality attention-based audio-driven lip shape network QAW; S2) collecting videos of several people speaking and dividing each video into person audio and continuous frames of face images, and constructing each group of person audio and continuous frames of face images together as a lip synchronization training set; S3) inputting the lip synchronization training set into the lip synchronization discrimination network for training until the discrimination loss function of the lip synchronization discrimination network converges, thereby obtaining a trained lip synchronization discrimination network; S4) inputting the lip synchronization training set into the quality-attention-based audio-driven lip shape network QAW for training, constructing the overall loss function of the quality-attention-based audio-driven lip shape network QAW based on the discriminant loss function, obtaining the overall loss value of the quality-attention-based audio-driven lip shape network QAW and using the gradient descent method to update the network parameters until the loss value of the overall loss function converges, thereby obtaining the trained quality-attention-based audio-driven lip shape network QAW; S5) inputting the text or audio to be replied into the dialogue system, and for each piece of text or audio to be replied, the dialogue system processes and outputs a reply text of the text or audio to be replied, and converts the reply text into a reply audio; The reply audio and the facial image of the person to be generated are input into the trained quality-attention-based audio-driven lip network QAW. After processing, the trained quality-attention-based audio-driven lip network QAW outputs the facial video of the current person when reading the current reply audio, and finally displays it on the monitor to realize the generation of the facial video.

2. The method for generating face video based on the audio-driven face dialogue generation model according to claim 1, characterized in that: In the step S1), the lip synchronization discrimination network includes an audio encoder and an image encoder, the audio encoder includes a plurality of convolutional layers Conv and a plurality of two-dimensional convolutional layers Conv2d connected in sequence, the image encoder includes a plurality of two-dimensional convolutional layers Conv2d and a plurality of convolutional layers Conv connected in sequence, each convolutional layer Conv is composed of three two-dimensional convolutional layers Conv2d connected in sequence, and each two-dimensional convolutional layer Conv2d includes a convolutional block, a batch normalization layer BN and a ReLU activation function connected in sequence; for each group of character audio and continuous frame facial images in the lip synchronization training set, the character audio is converted into a Mel-spectrogram and then input into the audio encoder for processing to output a first audio feature vector, and the continuous frame facial images are input into the image encoder to obtain a first lip shape feature vector.

3. The method for generating face video based on the audio-driven face dialogue generation model according to claim 1, characterized in that: In the step S1), the audio-driven lip shape network QAW based on quality attention includes a face generator and a video quality discriminator connected in sequence, the lip synchronization training set is input into the face generator, the face generator outputs the generated face video after processing, the generated face video and the real face video are input into the video quality discriminator together, the adversarial loss function and the quality assessment loss function of the video quality discriminator are obtained, when the adversarial loss function of the video quality discriminator converges, the video quality discriminator training is completed; the reconstruction loss function is obtained according to the generated face video and the real face video, and finally the overall loss function of the audio-driven lip shape network QAW based on quality attention is constructed according to the discriminant loss function of the lip synchronization discriminant network, the quality assessment loss of the video quality discriminator, the FSIM loss function of the face dialogue generation model and the reconstruction loss function.

4. The method for generating face video based on the audio-driven face dialogue generation model according to claim 3, characterized in that: The face generator includes an audio encoder, a face encoder and a face decoder. The audio encoder includes a first convolution layer Conv, a second convolution layer Conv, a third convolution layer Conv, a fourth convolution layer Conv, a fifth convolution layer Conv, a sixth convolution layer Conv, and a first two-dimensional convolution Conv2d connected in sequence. Each convolution layer Conv is composed of three two-dimensional convolution Conv2d connected in sequence. Each two-dimensional convolution Conv2d includes a convolution block, a batch normalization layer BN and a ReLU activation function connected in sequence. The face encoder includes a second two-dimensional convolution Conv2d, a seventh convolution layer Conv, an eighth convolution layer Conv, a ninth convolution layer Conv, a first spatial self-attention mechanism SSA, a tenth convolution layer Conv, a second spatial self-attention mechanism SSA, an eleventh convolution layer Conv, a twelfth convolution layer Conv, a thirteenth convolution layer Conv, a third two-dimensional convolution Conv2d and a fourth two-dimensional convolution Conv2d connected in sequence. The face decoder includes a fifth two-dimensional convolution Conv2d, a first spatial attention mechanism SSA, a first spatial attention mechanism SSA, a second spatial self-attention mechanism SSA, a first convolution layer Conv, a second spatial self-attention mechanism SSA, a second convolution layer Conv, a second convolution layer Conv, a second convolution layer Conv, a third convolution Conv2d and a fourth two-dimensional convolution Conv2d connected in sequence. system SA, the first fusion function Concat, the fourteenth convolution layer Conv, the second spatial attention mechanism SA, the second fusion function Concat, the fifteenth convolution layer Conv, the third spatial attention mechanism SA, the third fusion function Concat, the sixteenth convolution layer Conv, the fourth spatial attention mechanism SA, the fourth fusion function Concat, the seventeenth convolution layer Conv, the fifth spatial attention mechanism SA, the fifth fusion function Concat, the eighteenth convolution layer Conv, the sixth spatial attention mechanism SA, the sixth fusion function Concat, the nineteenth convolution layer Conv, the first position attention mechanism CA, the seventh spatial attention mechanism SA, the seventh fusion function Concat, the twentieth convolution layer Conv, the second position attention mechanism CA, the eighth spatial attention mechanism SA, the eighth fusion function Concat, the twenty-first convolution layer Conv, the third position attention mechanism CA, the ninth spatial attention mechanism SA, the ninth fusion function Concat, the sixth two-dimensional convolution Conv2d and the seventh two-dimensional convolution Conv2d; For each group of character audio and continuous frame face images in the lip synchronization training set, the character audio is converted into a Mel-spectrogram and then input into the audio encoder of the face generator for processing to output a second audio feature vector, the continuous frame face images are input into the face encoder for processing to output a second lip shape feature vector, the second audio feature vector is input into the fifth two-dimensional convolution Conv2d of the face decoder for processing, the output of the fifth two-dimensional convolution Conv2d and the second lip shape feature vector are input into the first spatial attention mechanism SA of the face encoder for processing, the output of the first spatial attention mechanism SA and the second lip shape feature vector are input into the first fusion function Concat for processing and then input into the fourteenth convolution layer Conv for processing, the output of the fourteenth convolutional layer Conv and the output of the thirteenth convolutional layer Conv are input into the second spatial attention mechanism SA for processing, the output of the second spatial attention mechanism SA and the output of the thirteenth convolutional layer Conv are input into the second fusion function Concat for processing and then output to the fifteenth convolutional layer Conv for processing, the output of the fifteenth convolutional layer Conv and the output of the twelfth convolutional layer Conv are input into the third spatial attention mechanism SA for processing, the output of the third spatial attention mechanism SA and the output of the twelfth convolutional layer Conv are input into the third fusion function Concat for processing and then output to the sixteenth convolutional layer Conv for processing, The output of the sixteenth convolutional layer Conv and the output of the eleventh convolutional layer Conv are input into the fourth spatial attention mechanism SA for processing. The output of the fourth spatial attention mechanism SA and the output of the eleventh convolutional layer Conv are input into the fourth fusion function Concat for processing and then output to the seventeenth convolutional layer Conv for processing. The output of the seventeenth convolutional layer Conv and the output of the tenth convolutional layer Conv are input into the fifth spatial attention mechanism SA for processing. The output of the fifth spatial attention mechanism SA and the output of the tenth convolutional layer Conv are input into the fifth fusion function Concat for processing and then output to the eighteenth convolutional layer Conv for processing. The output of the eighteenth convolutional layer Conv The output of the sixth spatial attention mechanism SA and the output of the ninth convolutional layer Conv are input together to the sixth spatial attention mechanism SA for processing. The output of the sixth spatial attention mechanism SA and the output of the ninth convolutional layer Conv are input together to the sixth fusion function Concat for processing and then output to the nineteenth convolutional layer Conv and the first position attention mechanism CA for processing in sequence. The output of the first position attention mechanism CA and the output of the eighth convolutional layer Conv are input together to the seventh spatial attention mechanism SA for processing. The output of the seventh spatial attention mechanism SA and the output of the eighth convolutional layer Conv are input together to the seventh fusion function Concat for processing and then output to the twentieth convolutional layer Conv and the second position attention mechanism CA for processing.The output of the second position attention mechanism CA and the output of the seventh convolution layer Conv are input into the eighth spatial attention mechanism SA for processing. The output of the eighth spatial attention mechanism SA and the output of the seventh convolution layer Conv are input into the eighth fusion function Concat for processing and then output to the twenty-first convolution layer Conv and the third position attention mechanism CA for processing. The output of the third position attention mechanism CA and the output of the second two-dimensional convolution Conv2d are input into the ninth spatial attention mechanism SA for processing. The output of the ninth spatial attention mechanism SA and the output of the second two-dimensional convolution Conv2d are input into the ninth fusion function Concat for processing and then output to the sixth two-dimensional convolution Conv2d and the seventh two-dimensional convolution Conv2d for processing, and the final output is used as the output of the face generator. , 5. The method for generating face video based on the audio-driven face dialogue generation model according to claim 3, characterized in that: The video quality discriminator includes a plurality of sequentially connected convolution groups, and each convolution group includes sequentially connected convolution blocks and an activation function LeakyReLU.

6. The method for generating face video based on the audio-driven face dialogue generation model according to claim 3, characterized in that: The overall loss function L all The details are as follows: L all =(1-w s -w g -w f )L recon +w s ·L sync +w g ·L gen +w f ·L FSIM Among them, w s 、w g and w f They represent the synchronization penalty weight, adversarial loss weight and FSIM loss weight respectively; L recon , L sync , L gen and L FSIM They respectively represent the reconstruction loss function, the discriminant loss function of the trained lip synchronization discriminant network, the quality assessment loss function of the video quality discriminator, and the quality assessment loss index loss FSIM of the face dialogue generation model.

7. The method for generating face video based on the audio-driven face dialogue generation model according to claim 6, characterized in that: The reconstruction loss function L recon The loss function is composed of the loss obtained based on the Manhattan L1 distance between the generated face video and the corresponding face image of each frame in the real face video.

8. The method for generating face video based on the audio-driven face dialogue generation model according to claim 6, characterized in that: The quality assessment loss function L of the video quality discriminator is gen The details are as follows: The adversarial loss function L of the video quality discriminator is disc The details are as follows: Among them, x i~Lg represents the i-th number of the one-dimensional vector generated by the generated face video input to the video quality discriminator, N represents the total number of elements in the one-dimensional vector generated by the video quality discriminator, and D() represents the probability that the video quality discriminator judges the input to be true; x i~LG Represents the i-th number of the one-dimensional vector generated by the real face video input to the video quality discriminator.

9. The method for generating face video based on the audio-driven face dialogue generation model according to claim 6, characterized in that: The quality assessment loss index loss L of the face dialogue generation model FSIM The details are as follows: Among them, G g and G G They represent the gradient amplitude of the corresponding video frames in the generated face video and the real face video calculated using the Sobel operator, respectively. g and P G They respectively represent the corresponding video frame phase consistency calculated according to the gradient amplitude of the corresponding video frames in the generated face video and the real face video; T1 and T2 respectively represent the first and second mediation parameters.

10. The method for generating face video based on the audio-driven face dialogue generation model according to claim 3, characterized in that: In the step S5), the dialogue system includes a natural language processing module and a chatgpt module. After the text to be replied is input into the dialogue system, it is directly input into the chatgpt module to generate the reply text. After the audio to be replied is input into the dialogue system, it is first input into the natural language processing module for processing and then converted into the text in the audio to be replied, and then input into the chatgpt module to generate the reply text; each reply text is input into the text-to-speech conversion model TTS for processing and then the reply audio is output; The reply audio and the facial image of the character to be generated are input into the trained quality attention-based audio-driven lip network QAW face generator. After processing, the trained face generator outputs the facial video of the current character when reading the current reply audio.

Citation Information

Patent Citations

  • Face feature point positioning method and device

    CN108596093A

  • Lip shape synchronization video generation method and device, equipment and storage medium

    CN112562720A

  • Speaking face video generation method and device based on convolutional neural network

    CN113378697A

  • Video processing method and device, and neural network training method and device

    CN113723385A

  • Video generation method and device, server and storage medium

    CN113901894A

Cited By

  • Method and device for generating face image based on voice

    CN119851656A

  • Lip type generation method and device based on voice driving and medium

    CN120708647A