Super-resolution-based high-definition mouth broadcast video generation method

Abstract

The invention relates to a super-resolution-based high-definition mouth broadcast video generation method, and belongs to the technical field of computer vision. The method comprises the following steps: 1, preprocessing video data; 2, performing rendering training on the low-resolution oral broadcast video by using a neural radiation field network; 3, super-dividing the low-resolution oral playing video according to frames to obtain high-resolution video frames, and combining the high-resolution video frames to form a high-resolution video; 4, performing end-to-end joint training on the super-division network and the neural radiation field; 5, rendering the mouth playing video; compared with the prior art, the method has the advantages that the super-resolution technology is adopted, the high-definition oral playing video can be generated, the cost for obtaining the high-definition oral playing video in the fields of video production and game creation is greatly reduced, and the video production efficiency is improved.

Description

Technical Field

[0001] The invention relates to a method for generating high-definition spoken video based on super-resolution, and belongs to the technical field of computer vision. Background Art

[0002] The task of spoken video generation is to use artificial intelligence technology to make the characters in the video make lip movements consistent with the pronunciation of a given audio. It has a wide range of applications in the fields of virtual reality, video, and game production. Many application scenarios of spoken video generation tasks, such as high-definition commercial film and television production and high-definition game production, have high requirements for the resolution and clarity of images and videos. Previous methods, such as AD-NeRF, RAD-NeRF, and ER-NeRF, can obtain high-quality rendered videos by directly training the model using higher-resolution images. However, this strategy not only brings more computational costs, but also does not significantly improve the effect of generating videos.

[0003] Another technology that can improve video quality is super-resolution technology, which is a technology that uses software algorithms to increase the resolution of images or videos. When only low-resolution input images are available, super-resolution technology can output high-resolution images that are consistent with low-resolution images in content, but visually clearer, more detailed, and more realistic. With the development of deep learning technology, convolutional neural networks, generative adversarial networks, and deep reinforcement learning technologies have been widely used in super-resolution reconstruction. However, the training of these methods for super-resolution is very complex and requires a large amount of data sets to train the super-resolution network. For 3D video rendering tasks, previous super-resolution networks do not fully utilize the prior information in rendering for this task. Therefore, directly using previous super-resolution methods is very costly and ineffective.

[0004] Therefore, how to improve the high definition and high resolution of spoken video has become an urgent problem to be solved. Summary of the invention

[0005] The purpose of the present invention is to improve the technical problems of high definition and high resolution of spoken video, and to propose a method for generating high-definition spoken video based on super resolution.

[0006] The working principle of the present invention is as follows: First, the present invention trains a model that uses neural radiation fields to perform three-dimensional reconstruction of characters and audio-driven, and renders low-resolution oral broadcast videos. Then, a lightweight conditional super-resolution method is used to render high-resolution videos of dynamic character models. This method uses a densely connected residual network, and combines the depth prior conditions and audio prior conditions obtained in the volume rendering process of this article. The features of the image and the features of the prior conditions are fused in the network, and the network is guided to achieve fine restoration of the high-frequency details and mouth shape of the character during the super-resolution process, and finally achieve efficient generation of high-resolution clear oral broadcast videos. Finally, the neural radiation field and the super-resolution network are jointly trained to obtain a super-resolution network with good super-resolution capabilities. Input any audio, the neural radiation field can simulate the mouth shape movement of the character, and then the results of the neural radiation field are passed through the super-resolution network to render high-resolution and high-definition oral broadcast videos.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] A method for generating high-definition oral video based on super-resolution of the present invention comprises the following steps:

[0009] Step 1: Preprocess video data;

[0010] Step 1.1: extract the original video frame of the video, reduce the resolution of the original video frame to form a low-resolution video frame, and perform semantic segmentation on the low-resolution video frame;

[0011] Step 1.2: Extract the pixel coordinates of the facial key areas of the low-resolution video frame using facial key points;

[0012] Step 1.3: Estimate the camera pose of the low-resolution video frame using 3DMM and face tracking.

[0013] Step 1.4: Use the language recognition model to extract the audio features of the video;

[0014] Step 2: Use the neural radiance field network to render low-resolution spoken video;

[0015] Step 2.1: Use low-resolution video frames as training sets, encode audio features, and input them into the neural radiance field network to form an audio-driven character model;

[0016] Step 2.2: Input new audio to the audio-driven character model to obtain a low-resolution spoken video after rendering training;

[0017] Step 3: Super-resolution the low-resolution spoken video frame by frame to obtain high-resolution video frames, which are then combined to form a high-resolution video.

[0018] Step 3.1: Input the low-resolution image into the feature decoder, and use the depth map and audio feature map to super-resolve the low-resolution image to obtain a high-resolution image.

[0019] Step 3.2: Use a convolutional network to extract image features from the low-resolution image;

[0020] Step 3.3: Decode the image features using a decoder of a neural network feature operation to obtain a decoded image; sample the decoded image using a sampler with RRDB (residual nested dense block) for image super-resolution;

[0021] Step 3.4: Add a feature fusion block to the decoder, define the feature fusion block as including a dense block and a conditional information fusion block; the feature fusion block uses convolution to fuse image features and prior conditional features; define the dense block as including 5 convolutional layers and Leaky ReLu (activation function) for image feature input; use the conditional information fusion block for conditional feature input; and perform residual connection after embedding the conditional information fusion block between dense blocks;

[0022] Step 4: Perform end-to-end joint training of the super-resolution network and the neural radiation field;

[0023] Step 4.1: Use adversarial loss to train the super-resolution network and use L1 loss for supervision;

[0024] Step 4.2: Use a pre-trained VGG network to measure the similarity between the super-resolved image patches and the true image patches in the feature space;

[0025] Step 5: Render the spoken video;

[0026] Step 5.1: Input the audio to the neural radiance field and render a low-resolution image.

[0027] Step 5.2: Input the low-resolution image into the super-resolution network, obtain the high-resolution image through super-resolution, and obtain the high-definition spoken video through rendering combination.

[0028] Beneficial effects:

[0029] Compared with the prior art, the present invention uses super-resolution technology to generate high-definition voice-over videos, which greatly reduces the cost of obtaining high-definition voice-over videos in the fields of video production and game creation, and improves the efficiency of video production. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a super-resolution network flow chart of the present invention;

[0031] Figure 2 It is the super-resolution visualization result of the method of the present invention;

[0032] Figure 3 The figure is a comparison result between the method of the present invention and the previous method. DETAILED DESCRIPTION

[0033] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the protection scope of the present invention.

[0034] Example

[0035] like Figure 1 As shown, a method for generating high-definition oral video based on super-resolution of the present invention is specifically implemented in the following steps:

[0036] Step 1: Preprocess video data;

[0037] Step 1.1: extract the original video frame of the video, reduce the resolution of the original video frame to form a low-resolution video frame, and perform semantic segmentation on the low-resolution video frame;

[0038] In an embodiment, the present invention trains a neural network using a 3-5 minute monocular camera video as training data, and the video records the action and audio of a person speaking at a fixed perspective. The video frame rate is 25fps, and the video frame is split into 4500 images. The video frame is down-scaled to a lower resolution for 3D reconstruction. For the down-scaled low-resolution image, semantic segmentation is performed to extract the head and torso of the person, and the original high-resolution video frame is also retained for training the super-resolution network.

[0039] Step 1.2: Extract the pixel coordinates of the facial key areas of the low-resolution video frame using facial key points;

[0040] Step 1.3: Estimate the camera pose of the low-resolution video frame using 3DMM and face tracking.

[0041] Step 1.4: Use the language recognition model to extract the audio features of the video;

[0042] Step 2: Use the neural radiance field network to render low-resolution spoken video;

[0043] Step 2.1: Use low-resolution video frames as training sets, encode audio features, and input them into the neural radiance field network to form an audio-driven character model;

[0044] Step 2.2: Input new audio to the audio-driven character model to obtain a low-resolution spoken video after rendering training;

[0045] In the embodiment, 4500 video frames are used for training to achieve dynamic 3D reconstruction of the characters. A new audio is input to the trained model, and the model can render the corresponding low-resolution spoken video.

[0046] Step 3: Super-resolution the low-resolution spoken video frame by frame to obtain high-resolution video frames, which are then combined to form a high-resolution video.

[0047] Step 3.1: Input the low-resolution image into the feature decoder, and use the depth map and audio feature map to super-resolve the low-resolution image to obtain a high-resolution image.

[0048] In an embodiment, the low-resolution image rendered by the model in step 2 is super-resolved to obtain a high-resolution image, which is combined into a high-resolution video. The feature decoder takes the low-resolution image as input and uses the depth map and audio feature map as conditions to perform image super-resolution to obtain a high-resolution image.

[0049] Step 3.2: Use a convolutional network to extract image features from the low-resolution image;

[0050] Step 3.3: Decode the image features using a decoder of a neural network feature operation to obtain a decoded image; sample the decoded image using a sampler with RRDB (residual nested dense block) for image super-resolution;

[0051] Step 3.4: Add a feature fusion block to the decoder, define the feature fusion block as including a dense block and a conditional information fusion block; the feature fusion block uses convolution to fuse image features and prior conditional features; define the dense block as including 5 convolutional layers and Leaky ReLu (activation function) for image feature input; use the conditional information fusion block for conditional feature input; and perform residual connection after embedding the conditional information fusion block between dense blocks;

[0052] In the embodiment, the depth map features and the audio feature map features are input into the conditional information fusion block to perform conditional information fusion; the present invention uses a convolution to extract the features of the two respectively, and then splices the channels of the two and sends them into the conditional information fusion block to achieve the fusion of image features and conditional features; there are two conditions used: one is the facial geometric features of the person, that is, the depth map, and the other is the audio features, that is, the audio feature map.

[0053] Step 4: Perform end-to-end joint training of the super-resolution network and the neural radiation field;

[0054] Step 4.1: Use adversarial loss to train the super-resolution network and use L1 loss for supervision;

[0055] Step 4.2: Use a pre-trained VGG network to measure the similarity between the super-resolved image patches and the true image patches in the feature space;

[0056] In an embodiment, the present invention uses a low-resolution facial image rendered by a neural radiation field as input, and uses a depth map and an audio feature map obtained during volume rendering of the neural radiation field as prior condition input. It is implemented through a learnable discriminator that determines the difference in data distribution between the predicted image and the training image, and is trained together with the super-resolution network.

[0057] Step 5: Render the spoken video;

[0058] Step 5.1: Input the audio to the neural radiance field and render a low-resolution image.

[0059] Step 5.2: Input the low-resolution image into the super-resolution network, obtain the high-resolution image through super-resolution, and obtain the high-definition spoken video through rendering combination.

[0060] In the embodiment, after the above steps 1 to 4, dynamic 3D reconstruction and super-resolution network training of the character are achieved, and the trained model is trained on a NVIDIA A100 GPU for 12 hours.

[0061] In the experimental results, the oral video generated by the present invention can restore the facial details and lip movements of the characters, and can generate more realistic and clear videos compared with the previous methods. The visualized experimental results are shown in the attached figure. Figure 2 This is the visualization result of the high-definition rendering of the character of the present invention, from which it can be seen that the facial details of the character are clearly visible, especially the areas with complex textures such as the teeth, mouth, and hair of the character; Figure 3 The figure shows the comparison between the method of the present invention and the previous methods, including RAD-NeRF and ER-NeRF. For the convenience of display, the facial details of the characters are magnified. The mouth movements of the characters generated by the method of the present invention are the most obvious, the details are the richest, and there is no blur and distortion. In summary, the experimental results show the effectiveness and superiority of the present invention.

Claims

1. A method for generating high-definition spoken video based on super-resolution, characterized by: The following steps are included: Step 1: Preprocess video data; Step 1.1: extract the original video frame of the video, reduce the resolution of the original video frame to form a low-resolution video frame, and perform semantic segmentation on the low-resolution video frame; Step 1.2: Extract the pixel coordinates of the facial key areas of the low-resolution video frame using facial key points; Step 1.3: Estimate the camera pose of the low-resolution video frame using 3DMM and face tracking. Step 1.4: Use the language recognition model to extract the audio features of the video; Step 2: Use the neural radiance field network to render low-resolution spoken video; Step 2.1: Use low-resolution video frames as training sets, encode audio features, and input them into the neural radiance field network to form an audio-driven character model; Step 2.2: Input new audio to the audio-driven character model to obtain a low-resolution spoken video after rendering training; Step 3: Super-resolution the low-resolution spoken video frame by frame to obtain high-resolution video frames, and then combine them to form a high-resolution video; Step 4: Perform end-to-end joint training of the super-resolution network and the neural radiation field; Step 4.1: Use adversarial loss to train the super-resolution network and use L1 loss for supervision; Step 4.2: Use a pre-trained VGG network to measure the similarity between the super-resolved image patches and the true image patches in the feature space; Step 5: Render the spoken video; Step 5.1: Input the audio into the neural radiation field to render a low-resolution image; Step 5.2: Input the low-resolution image into the super-resolution network, obtain the high-resolution image through super-resolution, and obtain the high-definition spoken video through rendering combination.

2. The method for generating high-definition spoken video based on super-resolution according to claim 1, characterized in that: Step 3 is implemented as follows: Step 3.1: Input the low-resolution image into the feature decoder, and use the depth map and audio feature map to super-resolve the low-resolution image to obtain a high-resolution image; Step 3.2: Use a convolutional network to extract image features from the low-resolution image; Step 3.3: Decode the image features using a decoder of a neural network feature operation to obtain a decoded image; sample the decoded image using a sampler with a residual nested dense block for image super-resolution; Step 3.4: Add a feature fusion block to the decoder, and define the feature fusion block as including a dense block and a conditional information fusion block; the feature fusion block uses convolution to fuse the image features and the prior conditional features; The dense block is defined as containing 5 convolutional layers and activation functions for the input of image features; The conditional information fusion block is used for the input of conditional features; the conditional information fusion block is embedded between dense blocks and residual connections are performed.

Images