Real-time virtual digital human broadcasting method based on SadTalker

Abstract

The invention relates to a real-time virtual digital human broadcasting method based on SadTalker, belongs to video generation in the technical field of computer vision and artificial intelligence, and aims to improve the real-time performance and natural fidelity of a virtual digital human, and the method comprises the following steps: step 1, generating a voice-free video frame from an original picture; 2, cutting the video frame picture according to a result of a face detection algorithm to obtain a face picture, and storing the face picture as input of a SadTalker model; step 3, continuously reading the voice stream and preprocessing the voice stream; 4, the face picture and the real-time voice stream are transmitted to a SadTalker model after data scheduling, real-time short frame reasoning adaptability modification is added, and the model is limited to use short voice reasoning; 5, restoring the position of the human face, and forming a complete video frame; step 6, matching timestamps of the video frame and the voice stream; the REF video is added, so that the virtual digital human is more natural and vivid, and the short frame reasoning technology is adopted, so that the video generation process of the virtual digital human is more efficient, and the real-time performance is remarkably improved.

Description

Technical Field

[0001] The invention belongs to video generation in the technical field of computer vision and artificial intelligence. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and deep learning technologies, virtual digital humans have been widely used in various fields, including the entertainment and media industries, such as virtual anchors and virtual idols, and can also provide innovative solutions in education, medical care, and service fields. The core technology of virtual digital humans involves the comprehensive application of multiple technologies such as computer vision, natural language processing, and deep learning.

[0003] SadTalker is a deep learning model widely used in generating videos. It aims to generate talking face animation from a static face photo and a reference audio or reference video (REF video). The model uses a 3D deformable model to extract 3D shape and initial expression coefficients in the face image, and then uses ResNet to extract audio features from the Mel spectrum. After that, these data are used to generate 3D motion coefficients, namely expression coefficients and head coefficients. Finally, the 3D motion coefficients are used to generate a series of facial image frames through the face-vid2vid model.

[0004] However, SadTalker does not support real-time face video synthesis and relies on the input speech to be longer than 2 seconds, otherwise the quality of continuously generated videos will deteriorate, which limits the flexibility and interactivity of the system. At the same time, SadTalker cannot introduce REF video when the input speech clip is too short. REF video is a reference video clip used to provide dynamic information such as facial expressions, head movements, and speaking style. This video is an important reference for the model to generate realistic speaking videos. The timing information in the REF video helps the model maintain the continuity and synchronization of movements when generating videos, avoiding unnatural jumps or freezes. Therefore, the inability to introduce REF video will cause the natural realism of the generated video to be severely reduced. Summary of the invention

[0005] In order to improve the real-time performance and natural fidelity of virtual digital human, the present invention provides a real-time virtual digital human broadcasting method based on SadTalker.

[0006] like Figure 1 As shown, the present invention comprises the following steps:

[0007] Step 1: Generate a video frame without speech from the original image: Based on the SadTalker model, the original image and the empty voice file are used to generate a long video of a person speaking. The REF video is introduced in the process of generating the long video, and each frame of the generated long video is saved as a series of images.

[0008] Step 2: Cut out the face images from the series of video frames saved in step 1 according to the result of the face detection algorithm, and save them as the input of the SadTalker model;

[0009] Step 3: Continue to read the speech stream and perform preprocessing and feature extraction to convert the audio signal into a time-frequency signal as the input of the SadTalker model;

[0010] Step 4: The face image of step 2 and the real-time voice stream of step 3 are passed into the SadTalker model after data scheduling, and the real-time short frame reasoning adaptability modification is added to limit the model to use short voice reasoning, calculate the time consumed by each batch of reasoning, set the image used for reasoning according to the calculation time result, and generate a video frame with voice;

[0011] The combination of the video frame with voice generated in step 5 and step 4 and the video frame without voice generated in step 1 can restore the face position, and form a complete video frame after performing transition frame generation processing;

[0012] Step 6: The timestamps of the video frames generated in step 5 and the voice stream in step 3 match.

[0013] Technical effects:

[0014] Compared with the prior art, the present invention modifies the logic of single-image reasoning of the video generation model, first generates video with images, and then uses each frame of the video as the reasoning logic for reasoning the image. This can better simulate real-life facial expressions and movements, and by adding REF videos, make virtual digital humans more natural and realistic. The short-frame reasoning technology is used to make the video generation process of virtual digital humans more efficient, thereby significantly improving real-time performance. The video frames generated by voice-based video generation technology often have certain edge effects. After using short-frame reasoning, the edge effects are more obvious. In order to make the video flow coherent, the method of generating transition frames is used to solve the edge effect problem of the video, thereby significantly improving the smoothness of the video.

[0015] The present invention not only solves the shortcomings of the existing methods in terms of real-time performance and natural fidelity, but also provides new ideas and methods for the wide application of virtual digital human technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is an overall flow chart of the present invention and serves as an abstract drawing.

[0017] Figure 2 The figure is a flow chart of the system of the present invention. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by ordinary technicians in the field without creative work by adopting the embodiments of the present invention are within the scope of protection of the present invention.

[0019] The series of images generated in step 1 cannot be used directly, because the face in the image accounts for a small proportion of the entire image. For the face-generated video model, a high confidence result can only be obtained when the input image only contains the face, so the face image needs to be cropped from the entire image. Optionally, use RetinaFaceResnet50 as the face detection algorithm, crop the image according to the result of the face detection algorithm, and save it as a series of new images as the input of the subsequent SadTalker model.

[0020] In step 3, the voice continuously generated by the text is obtained by reading the wav file. As the original voice data, the voice has many parameters in the reading process, such as mono and multi-channel, different sampling rates and bit depths, so unified voice data is obtained through voice preprocessing. The preprocessed data is mono, with a sampling rate of 16000 and a bit depth of 16 bits.

[0021] The feature extraction is as follows:

[0022] Framing: Divide the audio signal into multiple small time frames to ensure that the signal is relatively static within each frame. The frame length here is 40 milliseconds;

[0023] Windowing: Apply a window function to each frame, optionally using a Hamming window to reduce spectral leakage effects;

[0024] Calculate the fast Fourier transform: Perform a fast Fourier transform on each frame to convert the time domain signal into the frequency domain signal. The formula is as follows:

[0025]

[0026] Where X[k] is the frequency domain signal, x[n] is the input time domain signal, N is the total length of x, j is the imaginary unit, k is the frequency variable, indicating the frequency index in the frequency domain, and n is the time variable, indicating the sampling point index of the time domain signal;

[0027] Power spectrum: Calculate the power spectrum of each frame by taking the modulus of the result of the previous step, squaring it, and finally normalizing it;

[0028] Mel filter: Mel filter bank is a set of triangular filters, usually evenly distributed on the Mel scale. Each filter covers a specific frequency range and performs a weighted sum of the frequencies within the range. By applying these filters, the linear frequency spectrum can be converted to a Mel frequency spectrum, which can better capture the important features in the speech signal.

[0029] Logarithmic compression: Take the logarithm of each Mel filter output to compress the dynamic range and simulate the human voice's perception of sound intensity;

[0030] Mel spectrum graph: Arrange the Mel spectrum of all frames to form a two-dimensional image, which is the Mel spectrum feature extraction result we finally get.

[0031] In step 4, before the face image in step 2 and the real-time voice stream in step 3 enter the model, data scheduling processing is added to clarify the location of the two types of data and the calling process. Specifically, first, the face image, as a limited data that needs to be input each time the model is loaded, can be pre-loaded in the model, and then, based on the time step, determine which face image to use as the input for the model to generate the video; secondly, the real-time voice stream is streamed into the input queue of different message queues according to the time step, and the activated SadTalker model will stream the voice stream data from the input queue of the message queue, and finally pass the pre-loaded face image list into the output queue of the message queue.

[0032] The specific modifications to the real-time short-frame reasoning adaptability are: adding a verification function,

[0033]

[0034] Here, Fn is the nth frame of speech received by the SadTalker model. The model is restricted to use short speech reasoning. The number of image frames corresponding to each input speech is less than or equal to 5 frames. Otherwise, the reasoning time is too long, which will cause high reasoning latency. Secondly, the time consumed by each batch of reasoning is calculated, and the image used for reasoning is set according to this time. This will make the inferenced image look more natural in other parts except the mouth shape. Finally, ONNX is used to accelerate model reasoning to achieve real-time broadcasting effect of synchronous text generation.

[0035] In step 5, the video frame generated in step 4 is a picture containing only faces. According to the face picture and the cropping information of the face detection in step 2, the face picture is used to replace the face position of the original picture to form a new picture.

[0036] In step 5, the transition frames are generated using linear interpolation. The last frame of each batch and the first frame of the next batch are collected, and three transition frames are generated using linear interpolation.

[0037] In step 6, multiple batches of video frames generated in step 5 are transmitted back to the client via the message queue. Each batch of video frames carries time step information. In order to ensure the synchronization of audio and video during video playback, the voice of the corresponding time step is called immediately after each batch of video frames arrives at the client to complete the timestamp matching of the video frames generated in step 5 and the voice stream in step 3, thereby achieving real-time broadcasting.

[0038] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, a variety of simple modifications can be made to the technical solution of the present invention, and these simple modifications all belong to the protection scope of the present invention.

Claims

1. A real-time virtual digital human broadcasting method based on SadTalker, characterized in that: The following steps are included: Step 1: Generate a video frame without speech from the original image: Based on the SadTalker model, the original image and the empty voice file are used to generate a long video of a person speaking. The REF video is introduced in the process of generating the long video, and each frame of the generated long video is saved as a series of images. Step 2: Cut out the face images from the series of video frames saved in step 1 according to the result of the face detection algorithm, and save them as the input of the SadTalker model; Step 3: Continue to read the speech stream and perform preprocessing and feature extraction to convert the audio signal into a time-frequency signal as the input of the SadTalker model; Step 4: The face image of step 2 and the real-time voice stream of step 3 are passed into the SadTalker model after data scheduling, and the real-time short frame reasoning adaptability modification is added to limit the model to use short voice reasoning, calculate the time consumed by each batch of reasoning, set the image used for reasoning according to the calculation time result, and generate a video frame with voice; The combination of the video frame with voice generated in step 5 and step 4 and the video frame without voice generated in step 1 can restore the face position, and form a complete video frame after performing transition frame generation processing; Step 6: The timestamps of the video frames generated in step 5 and the voice stream in step 3 match.

2. The real-time virtual digital human broadcasting method based on SadTalker according to claim 1 is characterized in that: The face detection algorithm described in step 2 is RetinaFace Resnet50.

3. The real-time virtual digital human broadcasting method based on SadTalker according to claim 1 is characterized in that: In step 3, the speech continuously generated by the text is obtained by reading the wav file as the original speech data, and the speech data with mono channel, sampling rate of 16000 and bit depth of 16 bits is obtained through speech preprocessing.

4. The real-time virtual digital human broadcasting method based on SadTalker according to claim 1 is characterized in that: The data scheduling described in step 4 is specifically as follows: first, the face image is used as the limited data required for each model loading, and is pre-loaded in the model. Then, according to the time step, it is determined which face image is used as the input for the model to generate the video; secondly, the real-time voice stream is passed into the input queue of different message queues according to the time step, and the activated SadTalker model will stream the voice stream data from the input queue of the message queue, and finally the pre-loaded face image list is passed into the output queue of the message queue; the real-time short frame reasoning adaptability is modified to add a verification function, Among them, Fn is the nth frame of speech received by the SadTalker model. The model is restricted to use short speech reasoning. The number of image frames corresponding to each input speech is less than or equal to 5 frames, and ONNX is used to accelerate model reasoning.

5. The real-time virtual digital human broadcasting method based on SadTalker according to claim 1 is characterized in that: In step 5, the transition frames are generated using linear interpolation. The last frame of each batch and the first frame of the next batch are collected, and three transition frames are generated using linear interpolation.

6. The real-time virtual digital human broadcasting method based on SadTalker according to claim 1 is characterized in that: In step 6, multiple batches of video frames generated in step 5 are transmitted back to the client via the message queue. Each batch of video frames carries time step information. When each batch of video frames arrives at the client, the voice of the corresponding time step is immediately called to complete the timestamp matching of the video frames generated in step 5 and the voice stream in step 3.

Images