High-quality face voice driving method based on neural radiation field

Abstract

The invention provides a high-quality face voice driving method based on a neural radiation field, and the method comprises the following steps: carrying out the feature extraction on video-synchronized voice information through a text-based voice recognition model, and obtaining the extracted voice features; segmenting an initial face speaking video set frame by frame; estimating attitude information of each frame of face by using a pre-trained three-dimensional face reconstruction model; learning a neural radiation field model for a target picture by using a multi-layer perceptron; and taking the voice features as condition information, and generating a picture under the current visual angle and voice condition by using a neural rendering mode. A face speaking model trained based on the neural radiation field has the capability of implicitly representing three-dimensional face displacement, including rigid and non-rigid motion. The neural rendering supports sampling setting of different ray angles and different densities, so that the generated face speaking video has the characteristics of high quality and stability.

Description

technical field [0001] The invention relates to the technical field of face image processing, in particular to a high-quality face voice driving method based on a neural radiation field. Background technique [0002] With the development of technology in the field of image processing in recent years, artificial intelligence-based digital humans are in great demand in applications such as remote video conferencing, virtual character generation, and animation video creation. How to construct realistic and high-quality virtual characters has become a widely concerned issue. Among them, the use of arbitrary input speech signals to drive the target face and generate natural speaking video sequences is a core application. [0003] In the past, there were mainly three methods for high-quality face-speech driving: manual modeling, by pre-modeling a series of mouth shapes of the target face, and then manually decomposing the speech signal into corresponding action sequences, so as t...

AI Technical Summary

Problems solved by technology

Although the method still uses the generative confrontation network as the intermediate mapping, but with the help of the editable 3D face model, the result is relatively stable; however, in terms of maintaining the details of the original target face such as lighting, wrinkles and background fusion, it reaches Less than realistic effect

Images