Rapid lip shape synthesis method and system for high-definition digital human

Abstract

The invention provides a rapid lip-shape synthesis method and system for a high-definition digital human, and the method comprises the steps: S1, synthesizing voice content according to the text content, storing an audio file, and providing voice feature input for a Wav2Lip lip-shape synthesis algorithm after a user inputs a text; s2, converting the audio waveform into a facial animation by using Wav2Lip, learning the relationship between the audio and the image based on GAN, aligning the face in the audio and the face in the static image, and synthesizing into a dynamic video; and S3, converting the face restoration task into a Code sequence prediction task through the VQGAN, and carrying out global modeling by adopting Transform. According to the rapid lip shape synthesis method for the high-definition digital human, the lip shape of a general figure can be generated for a static picture or video, and mouth shape actions corresponding to voice are synthesized by utilizing an extracted voice frequency spectrum and a detected face sequence.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for fast lip shape synthesis of a high-definition digital human. Background Art

[0002] Digital human technology can not only achieve low-cost and high-efficiency content output to replace manual services, but also realize intelligent interaction between people and digital humans based on a variety of AI technologies. Digital humans are currently widely used in a variety of industries such as education, finance, entertainment, government and public utilities, providing capabilities including digital human endorsement, digital human presentation, exhibition hall explanation, virtual customer service, digital human broadcasting, etc.

[0003] The basic ability of digital human explanation and broadcasting requires that the virtual character has a sense of reality and presence, and can bring a natural interactive experience to users in the real world. The accurate lip movements and facial expressions of the digital character image are the key to achieving this ability. With the development of artificial intelligence technology, the lip movements of virtual characters have gradually evolved from the past animation production, 3D model movement production or motion capture mode to the use of generation algorithms and diffusion models in the field of artificial intelligence to automatically synthesize lip shapes and facial expressions based on speech. The Wav2Lip lip synthesis algorithm can effectively use the relationship between audio features and facial movements to convert audio waveforms into facial animations. The Wav2Lip model can achieve high-speed reasoning from audio to facial animation, synchronize audio and lip movements, and achieve better results.

[0004] In actual application scenarios, although the Wav2Lip model can complete smooth and accurate lip shape synthesis, the lip area accounts for less than 4% of the entire image, and the model's original data lacks high-definition lip shape data, so it is impossible to focus on the lip details, resulting in very blurry synthesized digital human facial animations, which is difficult to directly apply to customer interaction scenarios. Therefore, we try to combine multiple artificial intelligence algorithms and algorithm optimization technologies to achieve rapid generation of high-definition facial expressions and lip movements, giving users a good interactive experience.

[0005] Sadtalker, VideoRetalking and other algorithms can achieve high-definition facial expression driving through voice, but there are problems such as too slow generation speed, separation of face and body, no head movement or unnatural movement. The First OrderMotion expression migration algorithm has problems such as inability to complete lip movement migration well, fixed input size and difficulty in migrating back to the original image, and smearing of generated images. In order to restore the high-definition face of the generated face, the GFPGAN algorithm can restore high-quality faces from low-quality images, thereby solving the problem of low resolution in the lip synthesis algorithm, but due to the extremely slow generation speed, it cannot be generated in time for the user's digital content services, affecting the user experience.

[0006] Patent document CN115546362A discloses a digital human voice lip shape movement training method and a voice synthesis method. It relates to the field of human-computer interaction technology, and specifically includes: obtaining voice data, and building a voice lip shape prediction model based on the voice data; obtaining voice data in real time, generating the lip shape movement of the current voice data through the voice lip shape prediction model, and training the voice lip shape prediction model. However, the invention does not have the ability to output the blurred face synthesized by lip shape in high definition. Summary of the invention

[0007] In view of the defects in the prior art, the object of the present invention is to provide a method and system for fast lip shape synthesis of high-definition digital human.

[0008] A method for fast lip shape synthesis of a high-definition digital human provided by the present invention comprises:

[0009] Step S1: after the user inputs text, synthesize the speech content according to the text content, save the audio file, and provide speech feature input for the Wav2Lip lip synthesis algorithm;

[0010] Step S2: Use Wav2Lip to convert the audio waveform into facial animation, learn the relationship between audio and image based on GAN, align the audio with the face in the static image and synthesize the dynamic video;

[0011] Step S3: Convert the face restoration task into a code sequence prediction task through VQGAN, and use Transformer global modeling.

[0012] Preferably, in step S1:

[0013] According to the text content input by the user and the voice timbre and speed selected by the user, the speech synthesis algorithm uses the autoregressive model of the Text-to-Speech Model to convert the input text into corresponding speech features, and uses the high-quality generator of HiFi-GAN to convert the speech features into actual audio waveforms. The speech synthesis algorithm saves the waveforms as speech audio files.

[0014] After the user selects the digital human synthetic background and digital human image, the Mel spectrum is extracted according to the generated voice audio file, and the face detection model is trained using the pre-processed face image data. The detection accuracy and robustness of the model are improved by adjusting the model parameters and optimizing the algorithm. The trained face detection model is used to perform face detection on each frame in the pre-recorded digital human template video, and the detected face area is cropped out from the video frame. After using the face detection model to detect each frame, a series of face image sequences are obtained. The face image sequence and the Mel spectrum sequence are jointly generated into Wav2Lip input data.

[0015] Preferably, in step S2:

[0016] The spectrum sequence and image batch sequence are input into the Wav2Lip algorithm. Wav2Lip consists of a generator and two discriminators. The generator is responsible for generating image frames synchronized with lip movements based on the input audio feature Mel spectrum and face image. The two discriminators are the expert audio and lip synchronization discriminator and the visual quality discriminator, which are used to evaluate the synchronization of the generated lip movements with the audio and the quality of the image. Wav2Lip generates a talking face video frame by frame based on the input audio feature Mel spectrum and face image, realizes the synchronous synthesis of audio and image, and obtains a digital human video corresponding to the speech lip shape.

[0017] Preferably, in step S3:

[0018] According to the generated digital human video, the digital human head image is cropped and scaled to 512x512 size and normalized and color space converted, and then input into the CodeFormer face super-resolution model. The CodeFormer face super-resolution model is a model based on VQGAN and Transformer. The CodeFormer face super-resolution model receives low-quality face images as input, and the images are encoded through the VQGAN encoder to obtain discrete Code sequences. The Code sequences are input into the Transformer model for global modeling and prediction. After being decoded by the VQGAN decoder, high-quality face images are output. The model parameters are kept closest to the original image parameters to maintain the original face features and output the preset high-definition digital human video.

[0019] Preferably, in step S3:

[0020] According to the output preset high-definition digital human audio, the digital human body is extracted using the chroma extraction method, and the virtual human image is converted into a 4-channel transparent layer. The input digital human background is merged to complete the background and digital human fusion effect, and the preset high-definition dynamic digital human video with background is output.

[0021] Merge the audio file and the video file, generate a subtitle file based on the audio, and hard-code it into the video using the srt subtitle format to complete the digital human video output.

[0022] A high-definition digital human fast lip shape synthesis system provided by the present invention comprises:

[0023] Module M1: After the user inputs text, it synthesizes the speech content according to the text content, saves the audio file, and provides speech feature input for the Wav2Lip lip synthesis algorithm;

[0024] Module M2: Use Wav2Lip to convert audio waveforms into facial animations, learn the relationship between audio and images based on GAN, align audio with faces in static images and synthesize dynamic videos;

[0025] Module M3: Convert the face restoration task into a code sequence prediction task through VQGAN, and use Transformer global modeling.

[0026] Preferably, in the module M1:

[0027] According to the text content input by the user and the voice timbre and speed selected by the user, the speech synthesis algorithm uses the autoregressive model of the Text-to-Speech Model to convert the input text into corresponding speech features, and uses the high-quality generator of HiFi-GAN to convert the speech features into actual audio waveforms. The speech synthesis algorithm saves the waveforms as speech audio files.

[0028] After the user selects the digital human synthetic background and digital human image, the Mel spectrum is extracted according to the generated voice audio file, and the face detection model is trained using the pre-processed face image data. The detection accuracy and robustness of the model are improved by adjusting the model parameters and optimizing the algorithm. The trained face detection model is used to perform face detection on each frame in the pre-recorded digital human template video, and the detected face area is cropped out from the video frame. After using the face detection model to detect each frame, a series of face image sequences are obtained. The face image sequence and the Mel spectrum sequence are jointly generated into Wav2Lip input data.

[0029] Preferably, in the module M2:

[0030] The spectrum sequence and image batch sequence are input into the Wav2Lip algorithm. Wav2Lip consists of a generator and two discriminators. The generator is responsible for generating image frames synchronized with lip movements based on the input audio feature Mel spectrum and face image. The two discriminators are the expert audio and lip synchronization discriminator and the visual quality discriminator, which are used to evaluate the synchronization of the generated lip movements with the audio and the quality of the image. Wav2Lip generates a talking face video frame by frame based on the input audio feature Mel spectrum and face image, realizes the synchronous synthesis of audio and image, and obtains a digital human video corresponding to the speech lip shape.

[0031] Preferably, in the module M3:

[0032] According to the generated digital human video, the digital human head image is cropped and scaled to 512x512 size and normalized and color space converted, and then input into the CodeFormer face super-resolution model. The CodeFormer face super-resolution model is a model based on VQGAN and Transformer. The CodeFormer face super-resolution model receives low-quality face images as input, and the images are encoded through the VQGAN encoder to obtain discrete Code sequences. The Code sequences are input into the Transformer model for global modeling and prediction. After being decoded by the VQGAN decoder, high-quality face images are output. The model parameters are kept closest to the original image parameters to maintain the original face features and output the preset high-definition digital human video.

[0033] Preferably, in the module M3:

[0034] According to the output preset high-definition digital human audio, the digital human body is extracted using the chroma extraction method, and the virtual human image is converted into a 4-channel transparent layer. The input digital human background is merged to complete the background and digital human fusion effect, and the preset high-definition dynamic digital human video with background is output.

[0035] Merge the audio file and the video file, generate a subtitle file based on the audio, and hard-code it into the video using the srt subtitle format to complete the digital human video output.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The present invention uses a fast lip shape synthesis method for high-definition digital humans, which can perform speech synthesis through text input. Through the speech synthesis algorithm, it can provide digital humans with multi-timbre and different speech speed speech file generation to cope with different digital human synthesis scenarios;

[0038] 2. The present invention uses a fast lip shape synthesis method for high-definition digital humans, which can generate the lip shape of a general person for a static picture or video, and synthesize the lip shape corresponding to the speech by using the extracted speech spectrum and the detected face sequence;

[0039] 3. The present invention applies a fast lip shape synthesis method for high-definition digital humans, which can output the fuzzy face synthesized by lip shape in high definition, and uses a face super-resolution model based on VQGAN and Transfomer to quickly output the high-definition result of the face, bringing a better digital content generation experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0041] Figure 1 This is a schematic diagram of the high-definition digital human synthesis process. DETAILED DESCRIPTION

[0042] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0043] Embodiment 1:

[0044] The present invention relates to the field of speech synthesis. The purpose of the present invention is to provide a fast lip shape synthesis algorithm for a high-definition digital human, so as to solve the problem of low lip resolution existing in common lip shape synthesis algorithms and the problem of slow synthesis speed when optimizing a super-resolution model for high-definition output.

[0045] The main objectives of the present invention are: (1) to design a lip shape synthesis algorithm that can generate high-definition human faces, generate corresponding lip movements according to audio features, and naturally restore them to the human face part, and output high-definition digital human videos through a super-resolution algorithm for the human face; (2) to design a high-speed digital human high-definition generation algorithm that can optimize lip shape synthesis and human face super-resolution stages, provide faster and more efficient reasoning performance, guarantee user experience, and ensure efficient use of resources.

[0046] According to the invention, a method for fast lip shape synthesis of a high-definition digital human is provided. Figure 1 As shown, including:

[0047] Step S1: after the user inputs text, synthesize the speech content according to the text content, save the audio file, and provide speech feature input for the Wav2Lip lip synthesis algorithm;

[0048] Specifically, in step S1:

[0049] According to the text content input by the user and the voice timbre and speed selected by the user, the speech synthesis algorithm uses the autoregressive model of the Text-to-Speech Model to convert the input text into corresponding speech features, and uses the high-quality generator of HiFi-GAN to convert the speech features into actual audio waveforms. The speech synthesis algorithm saves the waveforms as speech audio files.

[0050] After the user selects the digital human synthetic background and digital human image, the Mel spectrum is extracted according to the generated voice audio file, and the face detection model is trained using the pre-processed face image data. The detection accuracy and robustness of the model are improved by adjusting the model parameters and optimizing the algorithm. The trained face detection model is used to perform face detection on each frame in the pre-recorded digital human template video, and the detected face area is cropped out from the video frame. After using the face detection model to detect each frame, a series of face image sequences are obtained. The face image sequence and the Mel spectrum sequence are jointly generated into Wav2Lip input data.

[0051] Step S2: Use Wav2Lip to convert the audio waveform into facial animation, learn the relationship between audio and image based on GAN, align the audio with the face in the static image and synthesize the dynamic video;

[0052] Specifically, in step S2:

[0053] The spectrum sequence and image batch sequence are input into the Wav2Lip algorithm. Wav2Lip consists of a generator and two discriminators. The generator is responsible for generating image frames synchronized with lip movements based on the input audio feature Mel spectrum and face image. The two discriminators are the expert audio and lip synchronization discriminator and the visual quality discriminator, which are used to evaluate the synchronization of the generated lip movements with the audio and the quality of the image. Wav2Lip generates a talking face video frame by frame based on the input audio feature Mel spectrum and face image, realizes the synchronous synthesis of audio and image, and obtains a digital human video corresponding to the speech lip shape.

[0054] Step S3: Convert the face restoration task into a code sequence prediction task through VQGAN, and use Transformer global modeling.

[0055] Specifically, in step S3:

[0056] According to the generated digital human video, the digital human head image is cropped and scaled to 512x512 size and normalized and color space converted, and then input into the CodeFormer face super-resolution model. The CodeFormer face super-resolution model is a model based on VQGAN and Transformer. The CodeFormer face super-resolution model receives low-quality face images as input, and the images are encoded through the VQGAN encoder to obtain discrete Code sequences. The Code sequences are input into the Transformer model for global modeling and prediction. After being decoded by the VQGAN decoder, high-quality face images are output. The model parameters are kept closest to the original image parameters to maintain the original face features and output the preset high-definition digital human video.

[0057] Specifically, in step S3:

[0058] According to the output preset high-definition digital human audio, the digital human body is extracted using the chroma extraction method, and the virtual human image is converted into a 4-channel transparent layer. The input digital human background is merged to complete the background and digital human fusion effect, and the preset high-definition dynamic digital human video with background is output.

[0059] Merge the audio file and the video file, generate a subtitle file based on the audio, and hard-code it into the video using the srt subtitle format to complete the digital human video output.

[0060] Embodiment 2:

[0061] Embodiment 2 is a preferred example of Embodiment 1, and is used to illustrate the present invention in more detail.

[0062] The present invention also provides a fast lip shape synthesis system for a high-definition digital human. The fast lip shape synthesis system for a high-definition digital human can be implemented by executing the process steps of the fast lip shape synthesis method for a high-definition digital human. That is, those skilled in the art can understand the fast lip shape synthesis method for a high-definition digital human as a preferred implementation of the fast lip shape synthesis system for a high-definition digital human.

[0063] A high-definition digital human fast lip shape synthesis system provided by the present invention comprises:

[0064] Module M1: After the user inputs text, it synthesizes the speech content according to the text content, saves the audio file, and provides speech feature input for the Wav2Lip lip synthesis algorithm;

[0065] Specifically, in the module M1:

[0066] According to the text content input by the user and the voice timbre and speed selected by the user, the speech synthesis algorithm uses the autoregressive model of the Text-to-Speech Model to convert the input text into corresponding speech features, and uses the high-quality generator of HiFi-GAN to convert the speech features into actual audio waveforms. The speech synthesis algorithm saves the waveforms as speech audio files.

[0067] After the user selects the digital human synthetic background and digital human image, the Mel spectrum is extracted according to the generated voice audio file, and the face detection model is trained using the pre-processed face image data. The detection accuracy and robustness of the model are improved by adjusting the model parameters and optimizing the algorithm. The trained face detection model is used to perform face detection on each frame in the pre-recorded digital human template video, and the detected face area is cropped out from the video frame. After using the face detection model to detect each frame, a series of face image sequences are obtained. The face image sequence and the Mel spectrum sequence are jointly generated into Wav2Lip input data.

[0068] Module M2: Use Wav2Lip to convert audio waveforms into facial animations, learn the relationship between audio and images based on GAN, align audio with faces in static images and synthesize dynamic videos;

[0069] Specifically, in the module M2:

[0070] The spectrum sequence and image batch sequence are input into the Wav2Lip algorithm. Wav2Lip consists of a generator and two discriminators. The generator is responsible for generating image frames synchronized with lip movements based on the input audio feature Mel spectrum and face image. The two discriminators are the expert audio and lip synchronization discriminator and the visual quality discriminator, which are used to evaluate the synchronization of the generated lip movements with the audio and the quality of the image. Wav2Lip generates a talking face video frame by frame based on the input audio feature Mel spectrum and face image, realizes the synchronous synthesis of audio and image, and obtains a digital human video corresponding to the speech lip shape.

[0071] Module M3: Convert the face restoration task into a code sequence prediction task through VQGAN, and use Transformer global modeling.

[0072] Specifically, in the module M3:

[0073] According to the generated digital human video, the digital human head image is cropped and scaled to 512x512 size and normalized and color space converted, and then input into the CodeFormer face super-resolution model. The CodeFormer face super-resolution model is a model based on VQGAN and Transformer. The CodeFormer face super-resolution model receives low-quality face images as input, and the images are encoded through the VQGAN encoder to obtain discrete Code sequences. The Code sequences are input into the Transformer model for global modeling and prediction. After being decoded by the VQGAN decoder, high-quality face images are output. The model parameters are kept closest to the original image parameters to maintain the original face features and output the preset high-definition digital human video.

[0074] Specifically, in the module M3:

[0075] According to the output preset high-definition digital human audio, the digital human body is extracted using the chroma extraction method, and the virtual human image is converted into a 4-channel transparent layer. The input digital human background is merged to complete the background and digital human fusion effect, and the preset high-definition dynamic digital human video with background is output.

[0076] Merge the audio file and the video file, generate a subtitle file based on the audio, and hard-code it into the video using the srt subtitle format to complete the digital human video output.

[0077] Embodiment 3:

[0078] Embodiment 3 is a preferred example of Embodiment 1, and is used to illustrate the present invention in more detail.

[0079] The present invention proposes a fast lip shape synthesis method for high-definition digital human, which mainly consists of three parts: speech synthesis algorithm, lip shape synthesis algorithm, and face super-resolution algorithm:

[0080] After the user inputs text, the speech synthesis algorithm synthesizes speech content with different speaking speeds, timbres and other characteristics based on the text content, saves the audio file, and provides speech feature input for the Wav2Lip lip synthesis algorithm.

[0081] The lip synthesis algorithm uses Wav2Lip, which is an algorithm that converts audio waveforms into facial animations. It learns the relationship between audio and images based on GAN, aligns the audio with the face in the static image, and synthesizes dynamic video.

[0082] The face super-resolution algorithm uses CodeFormer, a face restoration model based on VQGAN and Transformer. VQGAN converts the face restoration task into a Code sequence prediction task, and uses Transformer global modeling to further increase the model's robustness to severe degradation, making the restored face more realistic.

[0083] The present invention provides a method for fast lip shape synthesis of a high-definition digital human, comprising the following steps:

[0084] Step 1: Based on the text content input by the user, and the voice timbre and voice speed selected by the user, a speech audio file is first generated using a speech synthesis algorithm.

[0085] Step 2: After the user selects the digital human synthetic background and digital human image, the Mel spectrum is extracted according to the speech audio generated in step 1, and the face detection model is trained using the pre-processed face image data. The detection accuracy and robustness of the model are improved by adjusting the model parameters and optimizing the algorithm. The trained face detection model is used to perform face detection on each frame in the pre-recorded digital human template video, and the detected face area is cropped out from the video frame. After using the face detection model to detect each frame, a series of face image sequences are obtained. The face image sequence and the Mel spectrum sequence are jointly generated into Wav2Lip input data.

[0086] Step 3: According to step 2, the spectrum sequence and image batch sequence are input into the Wav2Lip algorithm. Wav2Lip contains a generator and two discriminators. The generator is responsible for generating image frames synchronized with lip movements based on the input audio features (Mel spectrum) and face images. The two discriminators are the expert audio and lip synchronization discriminator and the visual quality discriminator, which are used to evaluate the synchronization of the generated lip movements with the audio and the quality of the image. Wav2Lip generates a talking face video frame by frame based on the input audio features (Mel spectrum) and face images, realizes the synchronous synthesis of audio and image, and obtains the digital human video corresponding to the speech lip shape.

[0087] Step 4: Based on the digital human video generated in step 3, crop the digital human head image, scale it to 512x512, complete normalization and color space conversion, and input it into the CodeFormer face super-resolution model. CodeFormer is a model based on VQGAN and Transformer. CodeFormer receives low-quality face images as input. The image is encoded by the VQGAN encoder to obtain a discrete Code sequence. The Code sequence is input into the Transformer model for global modeling and prediction. After being decoded by the VQGAN decoder, a high-quality face image is output. The model parameters are kept closest to the original image parameters to maintain the original facial features and output high-definition digital human videos.

[0088] Step 5: Based on the high-definition digital human audio output in step 4, use the chroma extraction method to extract the digital human body, convert the virtual human image into a 4-channel transparent layer, merge the input digital human background, complete the background and digital human fusion effect, and output a high-definition dynamic digital human video with background.

[0089] Step 6: According to step 5, merge the audio file and the video file, generate a subtitle file according to the audio, and hard-code it into the video using the srt subtitle format to complete the final high-definition digital human video output.

[0090] Figure 1It is the high-definition digital human synthesis method used in the present invention. The whole technical scheme is mainly divided into three levels. The first level is the digital human data input, including the digital human background, the digital human template video and the text input. The text input obtains the digital human voice file with different timbre and different speech speed selection through the speech synthesis algorithm. The second level is the digital human lip shape synthesis. The voice file is extracted as the Mel spectrum feature. The S3FD face detection model in step 2 is used to predict using multiple feature maps of different scales. By laying equally spaced anchor points on different feature maps, effective detection of faces of different scales is achieved, and the face image and face image position information are obtained. The Mel spectrum and face image sequence are aggregated, and the lip shape synthesis algorithm Wav2Lip is input to synthesize the lip movements under the voice feature. The third level is the face super-resolution network. The fuzzy lip shape video synthesized by the digital human is first encoded into a visual code sequence using VQGAN through the CodeFormer image restoration model, and the restored image is generated using Transformer decoding to obtain a high-quality digital human face image. Finally, the digital human is extracted as a transparent channel using chroma extraction, synthesized with the input digital human background, and synthesized and encoded with the audio track and subtitle file to finally output a high-definition digital human video file.

[0091] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0092] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for fast lip shape synthesis of high-definition digital human, characterized in that: include: Step S1: after the user inputs text, synthesize the speech content according to the text content, save the audio file, and provide speech feature input for the Wav2Lip lip synthesis algorithm; Step S2: Use Wav2Lip to convert the audio waveform into facial animation, learn the relationship between audio and image based on GAN, align the audio with the face in the static image and synthesize the dynamic video; Step S3: Convert the face restoration task into a code sequence prediction task through VQGAN, and use Transformer global modeling.

2. The method for rapid lip shape synthesis of high-definition digital human according to claim 1, characterized in that: In step S1: According to the text content input by the user and the voice timbre and speed selected by the user, the speech synthesis algorithm uses the autoregressive model of the Text-to-Speech Model to convert the input text into corresponding speech features, and uses the high-quality generator of HiFi-GAN to convert the speech features into actual audio waveforms. The speech synthesis algorithm saves the waveforms as speech audio files. After the user selects the digital human synthetic background and digital human image, the Mel spectrum is extracted according to the generated voice audio file, and the face detection model is trained using the pre-processed face image data. The detection accuracy and robustness of the model are improved by adjusting the model parameters and optimizing the algorithm. The trained face detection model is used to perform face detection on each frame in the pre-recorded digital human template video, and the detected face area is cropped out from the video frame. After using the face detection model to detect each frame, a series of face image sequences are obtained. The face image sequence and the Mel spectrum sequence are jointly generated into Wav2Lip input data.

3. The method for rapid lip shape synthesis of high-definition digital human according to claim 1, characterized in that: In step S2: The spectrum sequence and image batch sequence are input into the Wav2Lip algorithm. Wav2Lip consists of a generator and two discriminators. The generator is responsible for generating image frames synchronized with lip movements based on the input audio feature Mel spectrum and face image. The two discriminators are the expert audio and lip synchronization discriminator and the visual quality discriminator, which are used to evaluate the synchronization of the generated lip movements with the audio and the quality of the image. Wav2Lip generates a talking face video frame by frame based on the input audio feature Mel spectrum and face image, realizes the synchronous synthesis of audio and image, and obtains a digital human video corresponding to the speech lip shape.

4. The method for rapid lip shape synthesis of high-definition digital human according to claim 1, characterized in that: In step S3: According to the generated digital human video, the digital human head image is cropped and scaled to 512x512 size and normalized and color space converted, and then input into the CodeFormer face super-resolution model. The CodeFormer face super-resolution model is a model based on VQGAN and Transformer. The CodeFormer face super-resolution model receives low-quality face images as input, and the images are encoded through the VQGAN encoder to obtain discrete Code sequences. The Code sequences are input into the Transformer model for global modeling and prediction. After being decoded by the VQGAN decoder, high-quality face images are output. The model parameters are kept closest to the original image parameters to maintain the original face features and output the preset high-definition digital human video.

5. The method for rapid lip shape synthesis of high-definition digital human according to claim 1, characterized in that: In step S3: According to the output preset high-definition digital human audio, the digital human body is extracted using the chroma extraction method, and the virtual human image is converted into a 4-channel transparent layer. The input digital human background is merged to complete the background and digital human fusion effect, and the preset high-definition dynamic digital human video with background is output. Merge the audio file and the video file, generate a subtitle file based on the audio, and hard-code it into the video using the srt subtitle format to complete the digital human video output.

6. A high-definition digital human fast lip shape synthesis system, characterized in that: include: Module M1: After the user inputs text, it synthesizes the speech content according to the text content, saves the audio file, and provides speech feature input for the Wav2Lip lip synthesis algorithm; Module M2: Use Wav2Lip to convert audio waveforms into facial animations, learn the relationship between audio and images based on GAN, align audio with faces in static images and synthesize dynamic videos; Module M3: Convert the face restoration task into a code sequence prediction task through VQGAN, and use Transformer global modeling.

7. The high-definition digital human rapid lip shape synthesis system according to claim 6 is characterized in that: In the module M1: According to the text content input by the user and the voice timbre and speed selected by the user, the speech synthesis algorithm uses the autoregressive model of the Text-to-Speech Model to convert the input text into corresponding speech features, and uses the high-quality generator of HiFi-GAN to convert the speech features into actual audio waveforms. The speech synthesis algorithm saves the waveforms as speech audio files. After the user selects the digital human synthetic background and digital human image, the Mel spectrum is extracted according to the generated voice audio file, and the face detection model is trained using the pre-processed face image data. The detection accuracy and robustness of the model are improved by adjusting the model parameters and optimizing the algorithm. The trained face detection model is used to perform face detection on each frame in the pre-recorded digital human template video, and the detected face area is cropped out from the video frame. After using the face detection model to detect each frame, a series of face image sequences are obtained. The face image sequence and the Mel spectrum sequence are jointly generated into Wav2Lip input data.

8. The high-definition digital human rapid lip shape synthesis system according to claim 6 is characterized in that: In the module M2: The spectrum sequence and image batch sequence are input into the Wav2Lip algorithm. Wav2Lip consists of a generator and two discriminators. The generator is responsible for generating image frames synchronized with lip movements based on the input audio feature Mel spectrum and face image. The two discriminators are the expert audio and lip synchronization discriminator and the visual quality discriminator, which are used to evaluate the synchronization of the generated lip movements with the audio and the quality of the image. Wav2Lip generates a talking face video frame by frame based on the input audio feature Mel spectrum and face image, realizes the synchronous synthesis of audio and image, and obtains a digital human video corresponding to the speech lip shape.

9. The high-definition digital human rapid lip shape synthesis system according to claim 6, characterized in that: In the module M3: According to the generated digital human video, the digital human head image is cropped and scaled to 512x512 size and normalized and color space converted, and then input into the CodeFormer face super-resolution model. The CodeFormer face super-resolution model is a model based on VQGAN and Transformer. The CodeFormer face super-resolution model receives low-quality face images as input, and the images are encoded through the VQGAN encoder to obtain discrete Code sequences. The Code sequences are input into the Transformer model for global modeling and prediction. After being decoded by the VQGAN decoder, high-quality face images are output. The model parameters are kept closest to the original image parameters to maintain the original face features and output the preset high-definition digital human video.

10. The high-definition digital human rapid lip shape synthesis system according to claim 6, characterized in that: In the module M3: According to the output preset high-definition digital human audio, the digital human body is extracted using the chroma extraction method, and the virtual human image is converted into a 4-channel transparent layer. The input digital human background is merged to complete the background and digital human fusion effect, and the preset high-definition dynamic digital human video with background is output. Merge the audio file and the video file, generate a subtitle file based on the audio, and hard-code it into the video using the srt subtitle format to complete the digital human video output.

Images