Real-time oral broadcast video generation method based on neural radiation field

Abstract

The invention relates to a real-time oral broadcast video generation method based on a neural radiation field, and belongs to the technical field of computer vision. The method comprises the following steps: 1, preprocessing video data; 2, volume density calculation required by volume rendering is carried out on the three-dimensional multi-resolution Hash grid through a spatial feature coding method, and then coding features of the three-dimensional multi-resolution grid are obtained; 3, encoding the audio features; 4, encoding the eye features for representing the state of the eyes of the person in the video; 5, combining the dynamic oral broadcast videos; compared with the prior art, the method and the device have the advantages that character reconstruction can be completed in shorter training time, and audio-driven oral video rendering can be efficiently realized at a real-time rendering speed. The rendering quality of the method is superior to that of a previous method, clearer human face details can be reproduced, and the lip sound consistency of a rendered video is also superior to that of the previous method.

Description

Technical Field

[0001] The present invention relates to a method for generating real-time spoken video based on a neural radiation field, and belongs to the technical field of computer vision. Background Art

[0002] The task of spoken video generation aims to synthesize realistic and dynamic character head models that can be driven by the input audio signal. When a new audio signal is input, the face model can make corresponding movements, especially the mouth. The movement of the lips will correspond to the input speech pronunciation movement, so that the character looks like he is speaking the input speech synchronously. This task has a wide range of application value in many fields of human-computer interaction. For example, in applications such as virtual assistants, education and training, and online customer service, spoken video generation can provide a more vivid and intuitive way of presenting content. By synthesizing facial expressions and mouth movements synchronized with speech, the system can provide a more friendly and interactive user experience. In these interactive scenarios, real-time is usually emphasized, and the spoken video needs to be generated in real time during the interaction process to ensure that the interaction is carried out without delay.

[0003] Before the rise of deep learning, the task of generating spoken video relied on parametric 3D models and blend shape-based methods, in which the shape and appearance of an object are defined by a set of parameters that control the object's geometry, material properties, texture, and other features. By adjusting these parameters, the appearance of the object can be changed without changing the model's topology. Blend shape-based methods create a series of so-called blend shapes to represent different expression states. Each blend shape represents the deformation of a face under a specific expression, such as smiling, frowning, closing eyes, etc. By adjusting and combining these blend shapes, a variety of different expressions can be synthesized, making the synthesized avatars more vivid and with rich expression changes. These methods require a lot of manual work when creating and adjusting facial models, and are usually costly and inefficient.

[0004] With the emergence of deep learning, data-driven methods have become more common. These methods usually involve training neural networks on a large number of face images or video datasets to capture the complex relationship between facial expressions, lip movements, and audio features. Some works are based on generative adversarial networks, relying on 2D information to assist in generating spoken-word portraits by estimating the relationship between 2D key points and input audio, but these methods also have the disadvantages of unstable and complex training.

[0005] After Neural Radiance Fields (NeRF) became popular, some NeRF-based methods such as AD-NeRF, RAD-NeRF, and ER-NeRF also attempted to render spoken-word videos based on NeRF, but their rendered videos often have problems such as blurriness, distortion, and inaccurate character lip movements.

[0006] Therefore, how to improve the efficiency and video realism of existing methods for generating spoken-word videos has become an urgent problem to be solved. Summary of the invention

[0007] The purpose of the present invention is to solve the technical problems that the existing methods for generating spoken-word videos are inefficient and the generated videos are not realistic enough, and to propose a real-time spoken-word video generation method based on neural radiation fields.

[0008] The workflow of the present invention is as follows: first, on the framework of a multi-resolution hash grid, an efficient volume density calculation method is adopted to directly calculate the volume density value required for volume rendering by decoupling density information from grid features, thereby reducing the amount of calculation required in the MLP network, and storing higher-dimensional grid features in the multi-resolution grid to record more 3D geometric information, helping the model to perform more detailed reconstruction. Then, an audio feature fusion method is adopted to enable the model to learn the relationship between audio features and mouth movements through efficient encoding of audio features. In addition, in order to enable the characters in the video rendered by the model to have realistic blinking movements, a unique eye feature encoding method is designed, so that the model has a clear perception of the opening and closing state of the eyes, thereby reproducing the natural blinking movements of the characters during rendering. Finally, the obtained features are sent to a lightweight MLP network to predict the colors required for volume rendering, and then the volume rendering is completed to obtain images and videos rendered for the characters.

[0009] The objective of the present invention is achieved through the following technical solutions:

[0010] A method for generating a real-time oral video based on a neural radiation field of the present invention comprises the following steps:

[0011] Step 1: Preprocess the video data;

[0012] Step 1.1: Extract video frames from the video, perform semantic segmentation on each frame, extract the face and torso of the person in the image, preferably use 25fps as the video frame rate and split the video frame into 4500 images;

[0013] Step 1.2: After extracting facial key points from each frame of image, perform facial key point detection and extract the pixel coordinates of the eyes and mouth in the facial key areas;

[0014] Step 1.3: Estimate the camera pose of each frame image, and use 3DMM and face tracking methods to estimate the camera pose of each frame;

[0015] Step 1.4: Extract audio features of the video, use the language recognition model to extract audio features, and assign corresponding audio features to each frame of audio;

[0016] Step 2: Calculate the volume density required for volume rendering on the three-dimensional multi-resolution hash grid through a spatial feature encoding method, and then obtain the encoding features of the three-dimensional multi-resolution grid;

[0017] Step 2.1: Use a multi-resolution voxel grid to encode spatial features for volume rendering; define a three-dimensional spatial position coordinate as x, then the grid encoding feature is expressed as shown in formula (1);

[0018]

[0019] Among them, f is the encoded grid feature, is the grid encoder, It is a feature concatenation operation, which means concatenating the encoded features f at each resolution level. i Concatenate them to get the final encoding feature f.

[0020] Step 2.2: Calculate the volume density using the grid coded features. Perform a weighted average on the first dimension of the grid coded features from the resolution level, and assign weights according to the resolution. The volume density is obtained as shown in formula (2), and then the coded features of the three-dimensional multi-resolution grid are obtained.

[0021]

[0022] Among them, f d,0 ,f d,1 ,f d,2 and f d,3 They represent the density features from the first four grids with the highest resolution, and λ0, λ1, λ2 and λ3 are their corresponding weights.

[0023] Step 3: Encode audio features;

[0024] Step 3.1: The initial audio features are converted into a smooth 64-dimensional audio feature through a one-dimensional convolutional network with a self-attention mechanism;

[0025] Step 3.2: Encode the audio 64-dimensional smooth audio features and propose a method to fuse audio features and spatial geometric information.

[0026] Step 3.2.1: The 64-dimensional smoothed audio features and the encoded features of the three-dimensional multi-resolution grid in step 2 are used as input and then reduced in dimension through an MLP (multi-layer perceptron), and a two-dimensional audio feature vector is output;

[0027] Step 3.2.2: Encode the two-dimensional audio feature vector using a two-dimensional multi-resolution hash grid to obtain the encoded features of the audio, which are used to control the character's mouth movements;

[0028] Step 4: Encode the eye features to represent the state of the eyes of the characters in the video;

[0029] Step 4.1: extract eye features from the eye image and crop two eye image blocks from the left eye and right eye regions of each frame of the eye image, and input the two eye image blocks into the pre-trained ResNet encoder to obtain the eye features of the current frame;

[0030] Step 4.2: Obtain eye signals through eye features;

[0031] Step 4.2.1: Perform semantic segmentation on the eye feature data in the preprocessing stage to generate an eye mask, and calculate the percentage p of the eye area in the eye image;

[0032] Step 4.3: Assign an eye signal to each frame of eye image using the method shown in formula (3), and make the eye signal tend to 0 or 1 to represent the closed state and open state of the eye respectively;

[0033]

[0034] Among them, p max and p min They represent the maximum and minimum values ​​of the image eye proportion p in all training frames respectively, and then the eye feature is multiplied by this signal amount e to be used as the eye feature that will eventually be input into the model;

[0035] Step 5: Combine the dynamic oral video;

[0036] Step 5.1: Input the encoding features of the three-dimensional multi-resolution grid described in step 2, the audio features described in step 3, and the eye features described in step 4 into a 3-layer MLP to calculate the color parameter c;

[0037] Step 5.2: Use the color parameter c to perform volume rendering by pixel rendering color;

[0038] Step 5.3: Combine the volume rendered image and the audio into a dynamic spoken video.

[0039] Beneficial effects:

[0040] Compared with the prior art, the present invention can complete the reconstruction of characters with a shorter training time and efficiently realize audio-driven oral video rendering at the speed of real-time rendering. It is not only superior to previous methods in rendering quality, but also can reproduce clearer facial details of characters, and is also superior to previous methods in lip-sound consistency of rendered videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the present invention;

[0042] Figure 2 The visualization result of the multi-view rendering of the present invention;

[0043] Figure 3 The figure is a comparison result between the method of the present invention and the previous method. DETAILED DESCRIPTION

[0044] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the protection scope of the present invention.

[0045] Example

[0046] like Figure 1 As shown, a method for generating a real-time oral video based on a neural radiation field of the present invention is specifically implemented in the following steps:

[0047] Step 1: Preprocess the video data;

[0048] Step 1.1: Extract video frames from the video, perform semantic segmentation on each frame, extract the face and torso of the person in the image, preferably use 25fps as the video frame rate and split the video frame into 4500 images;

[0049] Step 1.2: After extracting facial key points from each frame of image, perform facial key point detection and extract the pixel coordinates of the eyes and mouth in the facial key areas;

[0050] Step 1.3: Estimate the camera pose of each frame image, and use 3DMM and face tracking methods to estimate the camera pose of each frame;

[0051] Step 1.4: Extract audio features of the video, use the language recognition model to extract audio features, and assign corresponding audio features to each frame of audio;

[0052] In the embodiment, the present invention takes a 3-minute monocular camera video as training data, in which the video records the action and audio of a person speaking at a fixed viewing angle as an example.

[0053] Step 2: Calculate the volume density required for volume rendering on the three-dimensional multi-resolution hash grid through a spatial feature encoding method, and then obtain the encoding features of the three-dimensional multi-resolution grid;

[0054] Step 2.1: Use a multi-resolution voxel grid to encode spatial features for volume rendering; define a three-dimensional spatial position coordinate as x, then the grid encoding feature is expressed as shown in formula (1);

[0055]

[0056] Among them, f is the encoded grid feature, is the grid encoder, It is a feature concatenation operation, which means concatenating the encoded features f at each resolution level. i Concatenate them to get the final encoding feature f.

[0057] In an embodiment, given a position x=(1,1,1), this position will be encoded into a [4,16]-dimensional vector through a grid encoder.

[0058] Step 2.2: Calculate the volume density using the grid coded features. Perform a weighted average on the first dimension of the grid coded features from the resolution level, and assign weights according to the resolution. The volume density is obtained as shown in formula (2), and then the coded features of the three-dimensional multi-resolution grid are obtained.

[0059]

[0060] Among them, f d,0 ,f d,1 ,f d,2 and f d,3 They represent density features from the first four highest resolution grids, and λ0, λ1, λ2 and λ3 are their corresponding weights. In the embodiment, λ0, λ1, λ2 and λ3 are 1, 0.8, 0.6 and 0.5 respectively, and σ is calculated to be a value of 0-1.

[0061] Step 3: Encode audio features;

[0062] Step 3.1: The initial audio features are converted into a smooth 64-dimensional audio feature through a one-dimensional convolutional network with a self-attention mechanism;

[0063] Step 3.2: Encode the audio 64-dimensional smooth audio features and propose a method to fuse audio features and spatial geometric information.

[0064] Step 3.2.1: The 64-dimensional smoothed audio features and the encoded features of the three-dimensional multi-resolution grid in step 2 are used as input and then reduced in dimension through an MLP (multi-layer perceptron), and a two-dimensional audio feature vector is output;

[0065] Step 3.2.2: Encode the two-dimensional audio feature vector using a two-dimensional multi-resolution hash grid to obtain the encoded features of the audio, which are used to control the character's mouth movements;

[0066] In the embodiment, after the above steps, a [2,32]-dimensional feature vector is obtained.

[0067] Step 4: Encode the eye features to represent the state of the eyes of the characters in the video;

[0068] Step 4.1: extract eye features from the eye image and crop two eye image blocks from the left eye and right eye regions of each frame of the eye image, and input the two eye image blocks into the pre-trained ResNet encoder to obtain the eye features of the current frame;

[0069] Step 4.2: Obtain eye signals through eye features;

[0070] Step 4.2.1: Perform semantic segmentation on the eye feature data in the preprocessing stage to generate an eye mask, and calculate the percentage p of the eye area in the eye image;

[0071] In an embodiment, the eye signal p is a value in the range of 0-0.05.

[0072] Step 4.3: Assign an eye signal to each frame of eye image using the method shown in formula (3), and make the eye signal tend to 0 or 1 to represent the closed state and open state of the eye respectively;

[0073]

[0074] Among them, p max and p min They represent the maximum and minimum values ​​of the image eye proportion p in all training frames respectively, and then the eye feature is multiplied by this signal amount e to be used as the eye feature that will eventually be input into the model;

[0075] In the embodiment, the eye feature signal is finally converted into a value between 0-1.

[0076] Step 5: Combine the dynamic oral video;

[0077] Step 5.1: Input the encoding features of the three-dimensional multi-resolution grid described in step 2, the audio features described in step 3, and the eye features described in step 4 into a 3-layer MLP to calculate the color parameter c;

[0078] Step 5.2: Use the color parameter c to perform volume rendering by pixel rendering color;

[0079] Step 5.3: Combine the volume rendered image and the audio into a dynamic spoken video.

[0080] In the embodiment, the RGB color of a pixel position can be obtained through calculation, which is a three-dimensional vector, and the value of each dimension is between 0-255;

[0081] In the experimental results, the oral video generated by the present invention can restore the facial details and lip movements of the characters, and can generate more realistic and clear videos compared with the previous methods. The visualized experimental results are shown in the attached figure. Figure 2 This is a clear visualization result of the multi-view rendering of the character of the present invention, from which it can be seen that the facial details of the character are clearly visible, and the geometric shape of the character is consistent under multiple viewpoints; Figure 3 This is the comparison result between the method of the present invention and the previous method. The top of the image is the word pronounced by the character. GT is the character image extracted from the real video. AD-NeRF, RAD-NeRF and ER-NeRF are the previous methods. For the convenience of display, the character's mouth is enlarged. The blue rectangle is the error of the mouth pronunciation movement, and the red box indicates obvious blur or distortion. The character's mouth movement generated by the method of the present invention is most consistent with the real image, and there is no blur or distortion. In summary, the experimental results show the effectiveness and superiority of the present invention.

Claims

1. A method for generating real-time oral video based on neural radiation field, characterized in that: The following steps are included: Step 1: Preprocess the video data; Step 2: Calculate the volume density required for volume rendering on the three-dimensional multi-resolution hash grid through a spatial feature encoding method, and then obtain the encoding features of the three-dimensional multi-resolution grid; Step 3: Encode audio features; Step 3.1: The initial audio features are converted into a smooth 64-dimensional audio feature through a one-dimensional convolutional network with a self-attention mechanism; Step 3.2: Encode the audio 64-dimensional smooth audio features and propose a method to fuse the audio features and spatial geometric information; Step 3.2.1: The 64-dimensional smoothed audio features and the encoded features of the three-dimensional multi-resolution grid in step 2 are used as input and then reduced in dimension through an MLP to output a two-dimensional audio feature vector; Step 3.2.2: Encode the two-dimensional audio feature vector using a two-dimensional multi-resolution hash grid to obtain the encoded features of the audio, which are used to control the character's mouth movements; Step 4: Encode the eye features to represent the state of the eyes of the characters in the video; Step 4.1: extract eye features from the eye image and crop two eye image blocks from the left eye and right eye regions of each frame of the eye image, and input the two eye image blocks into the pre-trained ResNet encoder to obtain the eye features of the current frame; Step 4.2: Obtain eye signals through eye features; Step 4.2.1: Perform semantic segmentation on the eye feature data in the preprocessing stage to generate an eye mask, and calculate the percentage p of the eye area in the eye image; Step 4.3: Assign an eye signal to each frame of eye image using the method shown in formula (3), and make the eye signal tend to 0 or 1 to represent the closed state and open state of the eye respectively; Among them, p max and p min They represent the maximum and minimum values ​​of the image eye proportion p in all training frames respectively, and then the eye feature is multiplied by this signal amount e to be used as the eye feature that will eventually be input into the model; Step 5: Combine the dynamic spoken video.

2. A method for generating real-time oral video based on neural radiation field according to claim 1, characterized in that: Step 1 is implemented as follows: Step 1.1: Extract video frames from the video, perform semantic segmentation on each frame, extract the face and torso of the person in the image, preferably use 25fps as the video frame rate and split the video frame into 4500 images; Step 1.2: After extracting facial key points from each frame of image, perform facial key point detection and extract the pixel coordinates of the eyes and mouth in the facial key areas; Step 1.3: Estimate the camera pose of each frame image, and use 3DMM and face tracking methods to estimate the camera pose of each frame; Step 1.4: Extract audio features of the video, use the language recognition model to extract audio features, and assign corresponding audio features to each frame of audio.

3. The method for generating real-time oral video based on neural radiation field according to claim 1, characterized in that: Step 2 is implemented as follows: Step 2.1: Use a multi-resolution voxel grid to encode spatial features for volume rendering; define a three-dimensional spatial position coordinate as x, then the grid encoding feature is expressed as shown in formula (1); Among them, f is the encoded grid feature, is the grid encoder, It is a feature concatenation operation, which means concatenating the encoded features f at each resolution level. i Concatenate to obtain the final encoding feature f; Step 2.2: Calculate the volume density using the grid coded features. Perform a weighted average on the first dimension of the grid coded features from the resolution level, and assign weights according to the resolution. The volume density is obtained as shown in formula (2), and then the coded features of the three-dimensional multi-resolution grid are obtained. Among them, f d,0 ,f d,1 ,f d,2 and f d,3 They represent the density features from the first four grids with the highest resolution, and λ0, λ1, λ2 and λ3 are their corresponding weights.

4. A method for generating real-time oral video based on neural radiation field according to claim 1, characterized in that: Step 5 is implemented as follows: Step 5.1: Input the encoding features of the three-dimensional multi-resolution grid described in step 2, the audio features described in step 3, and the eye features described in step 4 into a 3-layer MLP to calculate the color parameter c; Step 5.2: Use the color parameter c to perform volume rendering by pixel rendering color; Step 5.3: Combine the volume rendered image and the audio into a dynamic spoken video.

Images