Digital human construction method based on voice drive

Abstract

The present invention discloses a method for constructing a digital human based on voice drive. By improving the audio encoder, face encoder and face decoder of the wav2lip model, the audio encoder is optimized with deep separable convolution DSC and time-frequency decomposition mechanism TFD, the number of parameters and the amount of calculation of the model are reduced, and the ability of the model to capture key features is enhanced. The processing capability of the face encoder for multi-scale facial information is further improved by adding feature pyramid FPN, especially the ability to capture subtle changes in lip movements and facial expressions. In addition, a multi-scale generation strategy is adopted to improve the clarity and detail expression of the output image of the face decoder. The overall system also includes a lip synchronization discriminator and a visual quality discriminator, which are respectively used to ensure the high synchronization of lip movements with audio and optimize the visual quality of the generated image, thereby achieving a more natural and accurate digital human interaction experience.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and machine vision, and in particular relates to a method for constructing a digital human based on voice drive. Background Art

[0002] Voice-driven digital human technology is an important direction in the development of digital humans. It uses voice input to drive the expressions and behaviors of virtual characters, enhancing the naturalness and interactivity of user experience. This technology shows great potential and broad application prospects in many fields such as service industry, entertainment, virtual assistants and distance education.

[0003] The importance of digital human technology lies in its ability to provide a seamless interactive experience, especially in scenarios that require highly personalized services. In the field of customer service, digital humans can serve as virtual assistants to answer customer inquiries 24 hours a day and improve user experience; in the field of education, they can simulate historical figures or fictional characters to provide students with an immersive learning experience; in addition, digital humans can also play the role of virtual doctors in the field of medical health, providing basic diagnostic support and health consultation.

[0004] The mainstream voice-driven digital human construction models include Wav2Lip, Audio2Face, GeneFace, EmotionAI, etc. Wav2Lip synchronizes the lip movements of virtual characters by analyzing voice signals, ensuring the consistency of voice and mouth shape and enhancing the sense of reality; Audio2Face generates corresponding facial expressions based on voice signals, without the need for complex manual animation, and can automatically adjust expressions according to the emotional changes of the sound; GeneFace uses generative adversarial networks (GAN) to create realistic facial images and animations, which can achieve highly personalized facial expressions and movements; EmotionAI analyzes the emotional content in the voice, allowing the virtual human to make corresponding expressions and reactions according to the emotions of the conversation.

[0005] Although the current voice-driven digital human technology has great application prospects and many excellent models have been found, it also has some technical challenges and limitations. The lip synchronization accuracy is not high enough, especially in the case of multiple languages ​​and dialects; the generated video clarity is relatively low and cannot meet the requirements of input and output of different resolutions; complex emotional changes and subtle non-verbal expressions are still difficult to accurately capture and reproduce. Summary of the invention

[0006] Purpose of the invention: To address the above problems, the present invention uses an improved wav2lip model to improve the lip sync of digital humans and the clarity of videos. It mainly introduces a time-frequency decomposition mechanism and a depth-separable convolution to optimize the capture and processing of facial features; combines feature pyramid layers to enhance the model's ability to process multi-scale facial information, further improving the model's performance and adaptability; and introduces a multi-scale generation strategy to improve the clarity and detail of the output video.

[0007] Technical solution: The present invention proposes a method for constructing a digital human based on voice drive, comprising the following steps:

[0008] Step 1: Obtain a Chinese lip reading video dataset and perform preprocessing to obtain a preprocessed Chinese lip reading video dataset;

[0009] Step 2: Build a generator, which consists of three modules: wav2lip audio encoder, wav2lip face encoder, and wav2lip face decoder; improve the wav2lip audio encoder, and use the deep separable convolution and time-frequency decomposition mechanism to optimize the audio encoder to obtain audio features;

[0010] Step 3: Add feature pyramid FPN to the wav2lip face encoder to obtain lip shape features;

[0011] Step 4: A multi-scale generation strategy is used for the wav2lip face decoder to improve the clarity and detail of the output image of the face decoder, which is used to combine the input audio signal with the extracted facial features, including facial identity and posture information, to generate lip movements that accurately match the input audio;

[0012] Step 5: Build a wav2lip lip sync discriminator to determine whether the lip movements in the generated video frames are synchronized with the input audio signal;

[0013] Step 6: Build a wav2lip visual quality discriminator to evaluate the overall visual quality of the generated video frames, including clarity, color fidelity, and the presence of visual artifacts, to ensure that the generated results are both realistic and high-quality;

[0014] Step 7: Input the preprocessed Chinese lip reading video dataset into the wav2lip network for training.

[0015] Furthermore, the specific method of step 1 is:

[0016] Step 1.1: Get the Chinese lip reading video dataset, separate each video from the audio, and save them in two folders respectively;

[0017] Step 1.2: Divide the video into frames and save each frame as a picture;

[0018] Step 1.3: Use Dlib to detect faces in the video frames, crop and align them;

[0019] Step 1.3: Split the data into training, validation, and test sets.

[0020] Furthermore, the specific method of step 2 is:

[0021] The wav2lip audio encoder replaces the existing standard convolution with a depth-separable convolution and adds a time-frequency decomposition module after the 9th convolution and the 12th convolution layer;

[0022] The time-frequency decomposition module consists of a time dimension convolution, a frequency dimension convolution and a feature fusion module; the time dimension convolution consists of a 1-dimensional convolution, which processes the time dimension of the audio spectrum to capture the dynamic changes of the audio signal time series; the frequency dimension convolution consists of a 2-dimensional convolution, which processes the frequency dimension of the audio spectrum to capture the frequency component relationship of the audio signal; the feature fusion module adds and fuses the features of the time dimension and the frequency dimension.

[0023] Furthermore, the specific method of step 3 is:

[0024] The wav2lip face encoder module uses depthwise separable convolution to replace the existing standard convolution and adds feature pyramid layers after the 9th and 15th convolution layers;

[0025] The feature pyramid layer is composed of an underlying feature extraction network ResNet, a multi-scale feature fusion module and an upsampling module.

[0026] Furthermore, the specific method of step 4 is:

[0027] The wav2lip face decoder module adopts a multi-scale feature fusion mechanism to implement a multi-scale generation strategy;

[0028] The multi-scale generation strategy consists of three decoders of different dimensions and a feature fusion module, wherein the three decoders process a high-resolution feature map of 512*512, a medium-resolution feature map of 256*256, and a low-resolution feature map of 128*128 respectively; the feature fusion module consists of two convolutional layers and an activation function, and the feature fusion module fuses the feature maps of the three decoders to output the final high-definition image.

[0029] Furthermore, the specific method of step 5 is:

[0030] The wav2lip lip sync discriminator is composed of an audio encoder, a video encoder, and a synchronization discriminant network. Five frames of continuous output images of the face decoder and the audio corresponding to the five frames of images are randomly extracted and fed to the lip sync discriminator. The audio encoder and the video encoder extract audio and video features, and then the audio and video features are sent to the synchronization discriminant network for feature fusion, and the cosine similarity between the audio features and the video features is calculated to evaluate the lip sync.

[0031] The formula for calculating the cosine similarity of audio and video features is as follows:

[0032]

[0033] Among them, ε is a very small positive number, ensuring that no matter what the value of ||v||2·||s||2 is, the denominator is always greater than zero to avoid the problem of division by zero, v represents the video feature s represents the audio feature;

[0034] The loss function of the lip synchronization discriminator is as follows:

[0035]

[0036] Where N is the batch size, is the cosine similarity between the lip shape of the i-th frame image and the corresponding audio.

[0037] Furthermore, the specific method of step 6 is:

[0038] The wav2lip visual quality discriminator is based on the discriminator principle in the generative adversarial network (GAN), and is composed of a convolutional layer, a batch normalization layer, an activation layer, and a fully connected layer. The image output by the face decoder is sent to the wav2lip visual quality discriminator, and firstly, image features are extracted through a series of convolutional layers. The extracted features are evaluated through a fully connected layer, and finally a scalar output is generated. The scalar output represents the probability of the image being a real image, which is used to guide the optimization of the generation model;

[0039] The loss function used by the wav2lip visual quality discriminator is the cross entropy loss function:

[0040]

[0041] in, Represents the distribution of real images L g The expectation of sampling in the image, D(x) represents the probability that the discriminator D predicts that the image x is a real image, L gen is the adversarial loss of the generator;

[0042]

[0043] Furthermore, the specific method of step 7 is:

[0044] Step 7.1: Use the preprocessed video frames and corresponding audio to train the wav2lip lip sync discriminator so that it can accurately assess synchronization;

[0045] Step 7.2: Use the preprocessed video frames to train the wav2lip visual quality discriminator so that it can distinguish between generated frames and real frames;

[0046] Step 7.3: Train the generator using the preprocessed video frames and corresponding audio along with lip sync loss and visual quality loss.

[0047] Beneficial effects:

[0048] The present invention adopts deep learning technology, combined with an improved wav2lip model, to achieve a digital human with higher lip synchronization and clearer video. By replacing the standard convolutional layers in the audio encoder and the face encoder with deep separable convolution, the number of parameters and computational complexity of the model are greatly reduced, while maintaining efficient feature extraction capabilities; the introduction of a video decomposition mechanism in the audio encoder enhances the model's ability to capture key audio information, thereby more accurately synchronizing lip movements; adding a feature pyramid layer in the face encoder further optimizes the integration of multi-scale features and enhances the model's ability to characterize facial features. Finally, a multi-scale generation strategy is introduced to process the output of the face decoder, which improves the clarity and details of the generated image, making the facial expressions of the digital human more natural and realistic. These integrated improvements not only improve the visual quality of the digital human, but also optimize the operating efficiency of the model, making it more suitable for real-time application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The overall flow chart of the method and device for constructing a voice-driven digital human;

[0050] Figure 2 This is the overall model diagram of wav2lip;

[0051] Figure 3 This is the structure diagram of the audio encoder module;

[0052] Figure 4 This is the structural diagram of the face encoder module;

[0053] Figure 5 This is the structural diagram of the face decoder module. DETAILED DESCRIPTION

[0054] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0055] The method for constructing a digital human based on voice drive disclosed in the present invention comprises the following steps:

[0056] Step 1: Obtain a Chinese lip reading video dataset and perform preprocessing to obtain a preprocessed Chinese lip reading video dataset.

[0057] Step 1.1: Get the Chinese lip reading video dataset, separate each video from the audio, and save them in two folders respectively.

[0058] Step 1.2: Divide the video into frames and save each frame as an image.

[0059] Step 1.3: Use Dlib to detect faces in the video frames, crop them and align them.

[0060] Step 1.3: Split the data into training, validation, and test sets.

[0061] Step 2: Build an improved wav2lip audio encoder module to obtain audio features.

[0062] The improved wav2lip audio encoder module replaces the existing standard convolution with depth-wise separable convolution and adds a time-frequency decomposition module after the 9th and 12th convolution layers.

[0063] The time-frequency decomposition module consists of a time dimension convolution, a frequency dimension convolution, and a feature fusion module. The time dimension convolution consists of a 1-dimensional convolution, which processes the time dimension of the audio spectrogram to capture the dynamic changes of the audio signal time series; the frequency dimension convolution consists of a 2-dimensional convolution, which processes the frequency dimension of the audio spectrogram to capture the frequency component relationship of the audio signal; the feature fusion module adds and fuses the features of the time dimension and the frequency dimension to obtain a richer audio feature representation, which helps to more comprehensively represent the audio signal.

[0064] Step 3: Build an improved wav2lip face encoder module to obtain lip shape features.

[0065] Improved wav2lip face encoder module, using depthwise separable convolution to replace the existing standard convolution and adding feature pyramid layers after the 9th and 15th convolution layers;

[0066] The feature pyramid layer is composed of the underlying feature extraction network ResNet, the multi-scale feature fusion module, and the upsampling module. The feature pyramid layer can significantly enhance the model's ability to process multi-scale information, especially in capturing subtle changes in mouth movements and facial expressions, and can improve the accuracy and naturalness of speech-to-lip synchronization.

[0067] Step 4: Build an improved wav2lip face decoder module to combine the input audio signal and the extracted facial features, including facial identity and pose information, to generate lip movements that accurately match the input audio.

[0068] The improved wav2lip face decoder module adopts a multi-scale feature fusion mechanism to implement a multi-scale generation strategy to improve the clarity and detail of the generated image, especially in the detail processing of key areas such as the edges of the lips.

[0069] The multi-scale generation strategy consists of three decoders of different dimensions and a feature fusion module. The three decoders process 512*512 high-resolution feature maps, 256*256 medium-resolution feature maps, and 128*128 low-resolution feature maps respectively.

[0070] The feature fusion module consists of two convolutional layers and an activation function. The feature fusion module fuses the feature maps of the three decoders and outputs the final high-definition image.

[0071] Step 5: Build a wav2lip lip sync discriminator module to determine whether the lip movements in the generated video frames are synchronized with the input audio signal.

[0072] The lip sync discriminator module of wav2lip consists of three submodules: audio encoder, video encoder, and synchronization discriminant network. Five consecutive output images of the face decoder and the audio corresponding to these five frames are randomly extracted and fed to the lip sync discriminator. The audio encoder and video encoder extract audio and video features, and then the audio and video features are sent to the synchronization discriminant network for feature fusion. The cosine similarity between the audio features and the video features is calculated to evaluate the lip sync.

[0073] The formula for calculating the cosine similarity of audio and video features is as shown in Formula 1.1.

[0074]

[0075] ε is a very small positive number, ensuring that no matter what the value of ||v||2·||s||2 is, the denominator is always greater than zero to avoid the problem of division by zero. v represents video features and s represents audio features.

[0076] The loss function of the lip synchronization discriminator is shown in formula 1.2.

[0077]

[0078] N is the batch size, is the cosine similarity between the lip shape of the i-th frame image and the corresponding audio, and its formula is as shown in 1.1.

[0079] Step 6: Build a wav2lip visual quality discriminator module to evaluate the overall visual quality of the generated video frames, including clarity, color fidelity, and the presence of visual artifacts, to ensure that the generated results are both realistic and high-quality.

[0080] The wav2lip visual quality discriminator module is mainly based on the discriminator principle in the Generative Adversarial Network (GAN), and consists of convolutional layers, batch normalization layers, activation layers, and fully connected layers. The image output by the face decoder is sent to the visual quality discriminator, and first the image features are extracted through a series of convolutional layers; the extracted features are evaluated through the fully connected layer, and finally a scalar output is generated, which represents the probability of the image being a real image, and is used to guide the optimization of the generation model.

[0081] The loss function used by the visual quality discriminator is the cross entropy loss function, and the calculation formula is as follows:

[0082]

[0083] Represents the distribution of real images L g The expectation of sampling in the image, D(x) represents the probability that the discriminator D predicts that the image x is a real image, L gen The adversarial loss formula of the generator is as shown in 1.4. The generator consists of three modules: the audio encoder, the face encoder, and the face decoder.

[0084]

[0085] Step 7: Input the preprocessed Chinese lip reading video dataset into the wav2lip network for training.

[0086] Step 7.1: Use the preprocessed video frames and corresponding audio to train a lip sync discriminator so that it can accurately assess synchronization.

[0087] Step 7.2: Use the preprocessed video frames to train a visual quality discriminator so that it can distinguish between generated frames and real frames.

[0088] Step 7.3: Train the generator using the preprocessed video frames and corresponding audio along with lip sync loss and visual quality loss.

[0089] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent transformation or decoration made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a digital human based on voice drive, characterized in that: The steps include: Step 1: Obtain a Chinese lip reading video dataset and perform preprocessing to obtain a preprocessed Chinese lip reading video dataset; Step 2: Build a generator, which consists of three modules: wav2lip audio encoder, wav2lip face encoder, and wav2lip face decoder; improve the wav2lip audio encoder, and use the deep separable convolution and time-frequency decomposition mechanism to optimize the audio encoder to obtain audio features; The wav2lip audio encoder replaces the existing standard convolution with a depth-separable convolution and adds a time-frequency decomposition module after the 9th convolution and the 12th convolution layer; The time-frequency decomposition module is composed of a time dimension convolution, a frequency dimension convolution and a feature fusion module; the time dimension convolution is composed of a 1-dimensional convolution, which processes the time dimension of the audio spectrum to capture the dynamic changes of the audio signal time series; the frequency dimension convolution is composed of a 2-dimensional convolution, which processes the frequency dimension of the audio spectrum to capture the frequency component relationship of the audio signal; the feature fusion module adds and fuses the features of the time dimension and the frequency dimension; Step 3: Add feature pyramid FPN to the wav2lip face encoder to obtain lip shape features; The wav2lip face encoder module uses depthwise separable convolution to replace the existing standard convolution and adds feature pyramid layers after the 9th and 15th convolution layers; The feature pyramid layer is composed of the underlying feature extraction network ResNet, a multi-scale feature fusion module and an upsampling module; Step 4: A multi-scale generation strategy is used for the wav2lip face decoder to improve the clarity and detail of the output image of the face decoder, which is used to combine the input audio signal with the extracted facial features, including facial identity and posture information, to generate lip movements that accurately match the input audio; Step 5: Build a wav2lip lip sync discriminator to determine whether the lip movements in the generated video frames are synchronized with the input audio signal; Step 6: Build a wav2lip visual quality discriminator to evaluate the overall visual quality of the generated video frames, including clarity, color fidelity, and the presence of visual artifacts, to ensure that the generated results are both realistic and high-quality; Step 7: Input the preprocessed Chinese lip reading video dataset into the wav2lip network for training.

2. The method for constructing a digital human based on voice drive according to claim 1, characterized in that: The specific method of step 1 is: Step 1.1: Get the Chinese lip reading video dataset, separate each video from the audio, and save them in two folders respectively; Step 1.2: Divide the video into frames and save each frame as a picture; Step 1.3: Use Dlib to detect faces in the video frames, crop and align them; Step 1.3: Split the data into training, validation, and test sets.

3. The method for constructing a digital human based on voice drive according to claim 1, characterized in that: The specific method of step 4 is: The wav2lip face decoder module adopts a multi-scale feature fusion mechanism to implement a multi-scale generation strategy; The multi-scale generation strategy consists of three decoders of different dimensions and a feature fusion module, wherein the three decoders process a high-resolution feature map of 512*512, a medium-resolution feature map of 256*256, and a low-resolution feature map of 128*128 respectively; the feature fusion module consists of two convolutional layers and an activation function, and the feature fusion module fuses the feature maps of the three decoders to output the final high-definition image.

4. The method for constructing a digital human based on voice drive according to claim 1, characterized in that: The specific method of step 5 is: The wav2lip lip sync discriminator is composed of an audio encoder, a video encoder, and a synchronization discriminant network. Five frames of continuous output images of the face decoder and the audio corresponding to the five frames of images are randomly extracted and fed to the lip sync discriminator. The audio encoder and the video encoder extract audio and video features, and then the audio and video features are sent to the synchronization discriminant network for feature fusion, and the cosine similarity between the audio features and the video features is calculated to evaluate the lip sync. The formula for calculating the cosine similarity of audio and video features is as follows: Among them, ε is a very small positive number, ensuring that no matter what the value of ||v||2·|||s||2 is, the denominator is always greater than zero to avoid the problem of division by zero, v represents the video feature s represents the audio feature; The loss function of the lip synchronization discriminator is as follows: Where N is the batch size, is the cosine similarity between the lip shape of the i-th frame image and the corresponding audio.

5. The method for constructing a digital human based on voice drive according to claim 1, characterized in that: The specific method of step 6 is: The wav2lip visual quality discriminator is based on the discriminator principle in the generative adversarial network (GAN), and is composed of a convolutional layer, a batch normalization layer, an activation layer, and a fully connected layer. The image output by the face decoder is sent to the wav2lip visual quality discriminator, and firstly, image features are extracted through a series of convolutional layers. The extracted features are evaluated through a fully connected layer, and finally a scalar output is generated. The scalar output represents the probability of the image being a real image, which is used to guide the optimization of the generation model; The loss function used by the wav2lip visual quality discriminator is the cross entropy loss function: in, Represents the distribution of real images L g The expectation of sampling in the image, D(x) represents the probability that the discriminator D predicts that the image x is a real image, L gen is the adversarial loss of the generator; 6. The method for constructing a digital human based on voice drive according to claim 1, characterized in that: The specific method of step 7 is: Step 7.1: Use the preprocessed video frames and corresponding audio to train the wav2lip lip sync discriminator so that it can accurately assess synchronization; Step 7.2: Use the preprocessed video frames to train the wav2lip visual quality discriminator so that it can distinguish between generated frames and real frames; Step 7.3: Train the generator using the preprocessed video frames and corresponding audio along with lip sync loss and visual quality loss.

Images