A high-definition voiceover video generation method based on super-resolution

Abstract

The application relates to a high-definition lip-sync video generation method based on super-resolution, and belongs to the technical field of computer vision.The application comprises the following steps: 1, preprocessing video data; 2, using a neural radiance field network to perform rendering training on a low-resolution lip-sync video; 3, super-resolution is performed on the low-resolution lip-sync video frame by frame to obtain high-resolution video frames, and high-resolution video is formed by combination; 4, end-to-end joint training of the super-resolution network and the neural radiance field is performed; and 5, the lip-sync video is rendered; compared with the prior art, the application uses the super-resolution technology, can generate high-definition lip-sync video, greatly reduces the cost of obtaining high-definition lip-sync video in the field of video production and game creation, and improves the video production efficiency.

Description

TECHNICAL FIELD

[0001] The application relates to a high-definition lip-sync video generation method based on super-resolution, and belongs to the technical field of computer vision. BACKGROUND

[0002] The lip-sync video generation task is to use artificial intelligence technology to make the characters in the video make mouth movements consistent with the pronunciation of the given audio. The task has wide application in virtual reality and video and game production fields. Many application scenarios of the lip-sync video generation task, such as high-definition commercial film and television production and high-quality game production, have high requirements for the resolution and clarity of images and videos. Previous methods such as AD-NeRF, RAD-NeRF and ER-NeRF want to obtain high-quality rendered videos. The method can be realized by directly training the model using higher resolution pictures, however, this strategy not only brings more computing cost, but also does not significantly improve the effect of generated videos.

[0003] Another technology that can improve video quality is super-resolution technology. Super-resolution technology is a technology that improves the resolution of images or videos through software algorithms. In the case of only low-resolution input images, super-resolution technology can output high-resolution images that are consistent with low-resolution images in content, but are visually clearer and more detailed and realistic. With the development of deep learning technology, convolutional neural networks, generative adversarial networks and deep reinforcement learning are widely used in super-resolution reconstruction. However, the training of these methods for super-resolution is very complex, and a large amount of data set is needed to train the super-resolution network. For the 3D video rendering task, previous super-resolution networks do not fully utilize the prior information in rendering, so directly using previous super-resolution methods is costly and ineffective.

[0004] Therefore, how to improve the high definition and high resolution of the lip-sync video has become a problem to be solved. SUMMARY

[0005] The application aims to solve the technical problem of improving the high definition and high resolution of the lip-sync video, and proposes a high-definition lip-sync video generation method based on super-resolution.

[0006] The working principle of the present application is as follows: firstly, the present application trains a model using neural radiance fields for three-dimensional reconstruction of a character and audio-driven, and renders a low-resolution lip-synch video. Then, a lightweight conditional super-resolution method is used to render high-resolution videos of dynamic character models. This method uses a densely connected residual network and combines the depth prior condition and audio prior condition obtained in the volume rendering process in this paper to fuse the features of the image and the prior condition in the network, guiding the network to realize the fine restoration of the high-frequency details and mouth shape of the character in the super-resolution process, and finally realizing the efficient generation of high-resolution clear lip-synch videos. Finally, the neural radiance field and the super-resolution network are jointly trained to obtain a super-resolution network with good super-resolution capability. Input any audio, the neural radiance field can simulate the mouth shape of the character, and then the result of the neural radiance field is rendered through the super-resolution network to obtain a high-resolution and high-definition lip-synch video.

[0007] The purpose of the present application is realized by the following technical solutions:

[0008] The present application is a high-definition lip-synch video generation method based on super-resolution, comprising the following steps:

[0009] Step 1: preprocessing video data;

[0010] Step 1.1: extracting the original video frames of the video, reducing the resolution of the original video frames to form low-resolution video frames, and performing semantic segmentation on the low-resolution video frames;

[0011] Step 1.2: extracting the pixel coordinates of the face key region of the low-resolution video frames using face key points;

[0012] Step 1.3: estimating the camera pose of the low-resolution video frames using the method of 3DMM and face tracking;

[0013] Step 1.4: extracting the audio features of the video using a language recognition model;

[0014] Step 2: rendering and training the low-resolution lip-synch video using a neural radiance field network;

[0015] Step 2.1: inputting the low-resolution video frames as a training set and encoding the audio features into the neural radiance field network to form an audio-driven character model;

[0016] Step 2.2: inputting new audio to the audio-driven character model to obtain a low-resolution lip-synch video after rendering and training;

[0017] Step 3: super-resolving the low-resolution lip-synch video by frame to obtain high-resolution video frames, and combining to form a high-resolution video.

[0018] Step 3.1: inputting the low-resolution image into a feature decoder, super-resolving the low-resolution image by using a depth map and an audio feature map to obtain a high-resolution image.

[0019] Step 3.2: extracting image features from the low-resolution image by using a convolutional network;

[0020] Step 3.3: decoding the image features by using a feature operation decoder of a neural network to obtain a decoded image; and sampling the decoded image by using a sampler with a RRDB (Residual in Residual Dense Block) for image super-resolution;

[0021] Step 3.4: adding a feature fusion block to the decoder, wherein the feature fusion block is defined as comprising a dense block and a conditional information fusion block; the feature fusion block is used for fusing the image features and prior condition features by using convolution; the dense block is defined as comprising five convolution layers and Leaky ReLu (activation function) for inputting the image features; the conditional information fusion block is used for inputting the condition features; and residual connection is performed after embedding the conditional information fusion block between the dense blocks;

[0022] Step 4: performing end-to-end joint training of the super-resolution network and the neural radiance field;

[0023] Step 4.1: training the super-resolution network by using an adversarial loss, and performing supervision by using an L1 loss;

[0024] Step 4.2: measuring the similarity between the super-resolved image block and the ground truth image block in the feature space by using a pre-trained VGG network;

[0025] Step 5: rendering the lip-sync video;

[0026] Step 5.1: inputting the audio into the neural radiance field to render a low-resolution image.

[0027] Step 5.2: inputting the low-resolution image into the super-resolution network to obtain a high-resolution image by super-resolution, and obtaining a high-definition lip-sync video by rendering and combining.

[0028] Beneficial effects:

[0029] Compared with the prior art, the super-resolution technology is used to generate a high-definition lip-sync video, which greatly reduces the cost of obtaining a high-definition lip-sync video in the field of video production and game creation, and improves the efficiency of video production. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The super-resolution network flowchart of the application;

[0031] Figure 2 The super-resolution visualization result of the method of the application;

[0032] Figure 3 This is a comparison between the method of the present invention and previous methods. Detailed Implementation

[0033] To better illustrate the purpose and advantages of this invention, the invention will be further described below with reference to the accompanying drawings and examples. It should be noted that the implementation of this invention is not limited to the following embodiments, and any modifications or alterations made to this invention will fall within the scope of protection of this invention.

[0034] Example

[0035] like Figure 1 As shown, the specific implementation steps of the high-definition voice-over video generation method based on super-resolution of the present invention are as follows:

[0036] Step 1: Preprocess video data;

[0037] Step 1.1: Extract the original video frames, reduce the resolution of the original video frames to form low-resolution video frames, and perform semantic segmentation on the low-resolution video frames;

[0038] In this embodiment, the present invention uses a 3-5 minute monocular camera video as training data to train the neural network. The video records the actions and audio of a person speaking from a fixed perspective. The video frame rate is 25fps, and the video frames are divided into 4500 images. The video frames are down-resolution for 3D reconstruction. Semantic segmentation is performed on the down-resolution images to extract the head and torso of the person, while the original high-resolution video frames are retained for training the super-resolution network.

[0039] Step 1.2: Extract the pixel coordinates of key facial regions from low-resolution video frames using facial key points;

[0040] Step 1.3: Use 3DMM and face tracking methods to estimate camera pose for low-resolution video frames;

[0041] Step 1.4: Extract audio features from the video using a speech recognition model;

[0042] Step 2: Use a neural radiation field network to render and train low-resolution audio-visual videos;

[0043] Step 2.1: Use low-resolution video frames as the training set, encode the audio features, and input them into the neural radiation field network to form an audio-driven character model;

[0044] Step 2.2: Input new audio into the audio-driven character model to obtain a low-resolution spoken video after rendering and training;

[0045] In an embodiment, 4500 video frames are used for training to realize dynamic three-dimensional reconstruction of the person. After a new audio is input into the trained model, the model can render a corresponding low-resolution lip-sync video.

[0046] Step 3: Super-resolution is performed on the low-resolution lip-sync video frame by frame to obtain high-resolution video frames, which are combined to form a high-resolution video.

[0047] Step 3.1: The low-resolution image is input into a feature decoder, and the low-resolution image is super-resolved using a depth map and an audio feature map to obtain a high-resolution image.

[0048] In an embodiment, the low-resolution image rendered by the model in step 2 is super-resolved to obtain a high-resolution image, which is combined to form a high-resolution video. The feature decoder takes the low-resolution image as input and takes the depth map and the audio feature map as conditions to perform image super-resolution to obtain a high-resolution image.

[0049] Step 3.2: A convolutional network is used to extract image features from the low-resolution image.

[0050] Step 3.3: A decoder of the neural network is used to decode the image features to obtain a decoded image; and a sampler with a RRDB (Residual Nested Dense Block) is used to sample the decoded image for image super-resolution.

[0051] Step 3.4: A feature fusion block is added to the decoder, and the feature fusion block is defined as including a dense block and a conditional information fusion block; the feature fusion block is used to fuse the image features and the prior condition features by convolution; the dense block is defined as including five convolutional layers and Leaky ReLu (activation function) for input of the image features; the conditional information fusion block is used for input of the condition features; and the conditional information fusion block is embedded between the dense blocks for residual connection.

[0052] In an embodiment, the depth map features and the audio feature map features are input into the conditional information fusion block for conditional information fusion; the present application uses a convolution to extract features of both, then channels are spliced, and the features are input into the conditional information fusion block to realize fusion of the image features and the condition features; the conditions used include two: one is the face geometry feature of the person, i.e., the depth map; and the other is the audio feature, i.e., the audio feature map.

[0053] Step 4: The super-resolution network and the neural radiance field are jointly trained end to end.

[0054] Step 4.1: The super-resolution network is trained using an adversarial loss, and is supervised using an L1 loss.

[0055] Step 4.2: Measure the similarity between the super-resolved image patch and the ground truth image patch in the feature space using a pre-trained VGG network;

[0056] In an embodiment, the present application takes a low-resolution face picture rendered by a neural radiance field as input, and takes a depth map and an audio feature map obtained in a volume rendering process of a neural radiance field as prior condition input. A learnable discriminator is implemented to judge the difference between the predicted image and the training image in the data distribution, and the discriminator is trained together with the super-resolution network.

[0057] Step 5: Render the lip-sync video;

[0058] Step 5.1: Input the audio into the neural radiance field to render a low-resolution image.

[0059] Step 5.2: Input the low-resolution image into the super-resolution network to obtain a high-resolution image through super-resolution, and obtain a high-definition lip-sync video through rendering combination.

[0060] In an embodiment, after steps 1 to 4, dynamic three-dimensional reconstruction of a person and training of a super-resolution network are realized, and a trained model is trained on an NVIDIA A100 GPU for 12 hours.

[0061] In the experimental results, the lip-sync video generated by the present application can restore the facial details and lip movement of the person, and can generate more realistic and clear videos compared to previous methods. The visualized experimental results are shown in the accompanying drawings. Figure 2 For the visualized results of the high-definition rendering of the person of the present application, it can be seen that the facial details of the person are clearly visible, especially the teeth, mouth, hair and other texture complex regions of the person; Figure 3 For the comparison results of the present application method and previous methods, including RAD-NeRF and ER-NeRF, in order to facilitate display, the facial details of the person are enlarged. The mouth movement of the person generated by the method of the present application is the most obvious, the details are the most abundant, and there is no blur and distortion. In summary, the experimental results show the effectiveness and superiority of the present application.

Claims

1. A method for generating a high-definition lip-sync video based on super-resolution, characterized in that: The method comprises the following steps, Step 1: preprocessing video data; Step 1.1: extracting original video frames of the video, reducing the resolution of the original video frames to form low-resolution video frames, and performing semantic segmentation on the low-resolution video frames; Step 1.2: extracting pixel coordinates of a face key region of the low-resolution video frames by using face key points; Step 1.3: estimating the camera pose of the low-resolution video frames by using a 3DMM and a face tracking method; Step 1.4: extracting audio features of the video by using a language recognition model; Step 2: performing rendering training on the low-resolution lip-sync video by using a neural radiance field network; Step 2.1: inputting the low-resolution video frames as a training set and the audio features into the neural radiance field network to form an audio-driven character model after coding; Step 2.2: inputting new audio into the audio-driven character model to obtain a low-resolution lip-sync video after rendering training; Step 3: performing super-resolution on the low-resolution lip-sync video frame by frame to obtain high-resolution video frames, and combining the high-resolution video frames to form a high-resolution video; Step 3.1: inputting the low-resolution image into a feature decoder, performing super-resolution on the low-resolution image by using a depth map and an audio feature map to obtain a high-resolution image; Step 3.2: extracting image features of the low-resolution image by using a convolutional network; Step 3.3: decoding the image features by using a decoder of the neural network to obtain a decoded image, and sampling the decoded image by using a sampler with residual nested dense blocks for image super-resolution; Step 3.4: adding a feature fusion block to the decoder, defining the feature fusion block as comprising a dense block and a conditional information fusion block; the feature fusion block fuses the image features and prior condition features by using convolution; the dense block is defined as comprising five convolutional layers and an activation function for input of the image features; the conditional information fusion block is used for input of the conditional features; the conditional information fusion block is embedded between the dense blocks and connected in residual; Step 4: performing end-to-end joint training of the super-resolution network and the neural radiance field; Step 4.1: training the super-resolution network by using an adversarial loss, and performing supervision by using an L1 loss; Step 4.2: measuring the similarity between a super-resolved image block and a ground truth image block in a feature space by using a pre-trained VGG network; Step 5: rendering the lip-sync video; Step 5.1: inputting the audio into the neural radiance field to render a low-resolution image; Step 5.2: inputting the low-resolution image into the super-resolution network to obtain a high-resolution image by super-resolution, and obtaining a high-definition lip-sync video by rendering combination.

Images