Voice-driven face animation generation method and device, equipment and storage medium

Abstract

The invention relates to the technical field of computer vision, in particular to a voice-driven face animation generation method, device and equipment and a storage medium, and the method comprises the steps: extracting image features corresponding to any query view angle based on any query view angle and a plurality of reference images; extracting initial audio features of the audio frame by frame, and performing time sequence filtering on the audio features to obtain audio features meeting an inter-frame smoothing condition; and driving a dynamic human face nerve radiation field by using the image features and the audio features, and obtaining a generated image of the current frame after voxel rendering. Therefore, the problem of low generalization of an existing voice-driven face animation synthesis method is solved, a dynamic face is modeled more accurately by proposing a dynamic face radiation field based on few-sample learning, few-sample learning is realized through a reference image mechanism, and model generalization is improved.

Description

technical field [0001] The present application relates to the technical field of computer vision, and in particular, to a voice-driven facial animation generation method, apparatus, device, and storage medium. Background technique [0002] Voice-driven face animation synthesis uses a speech audio as a driving signal to control the mouth shape, and generates a target face video that matches the given audio. This emerging technology has a wide range of application scenarios, such as movie dubbing, video conferencing, online education, and virtual avatars. Although a large amount of related research has emerged recently, how to generate natural and realistic voice-driven facial animation videos still presents considerable challenges. [0003] At present, voice-driven facial animation synthesis methods can be roughly divided into 2D-based methods and 3D-based methods. Among them, 2D-based methods usually rely on Generative Adversarial Networks (GAN), however, due to the lack o...

AI Technical Summary

Problems solved by technology

[0006] This application provides a voice-driven face animation generation method, device, equipment and storage medium to solve the low generalization problem of the existing voice-driven face animation synthesis method, by proposing a dynamic face animation based on few-sample learning Radiation field to more accurately model dynamic faces, and realize few-sample learning through the reference image mechanism to improve model generalization

Images