Voice-driven 3D virtual human expression voice and picture synchronization method and system based on deep learning

Abstract

The invention relates to a voice-driven 3D virtual human expression voice and picture synchronization method and system based on deep learning. The method comprises the following steps: extracting a logarithm amplitude spectrum in a voice signal as a voice signal characteristic; inputting the voice signal characteristic into a trained parameter prediction model which outputs an expression parameter value, wherein the parameter prediction model is a neural network model obtained by training a natural label pair relationship between a voice signal and an image signal in the video data; filteringthe expression parameter value output by the parameter prediction model; and performing image rendering of a 3D figure model by using the filtered expression parameter value to realize 3D virtual figure expression sound and picture synchronization. The system comprises a video analysis module, a parameter extraction module, a voice synthesis module, a voice signal processing module, a parameter prediction module, a parameter filtering module and a rendering module. According to the invention, the mouth lip effect of a virtual person is improved by learning a large amount of face video data such that the mouth lip effect is more natural and human-like.

Description

technical field [0001] The invention relates to the fields of computer graphics, computer vision, speech recognition, speech synthesis, etc., and specifically relates to a method of using a deep neural network to fit the relationship between speech and 3D model Blend Shape values, and to realize the synchronization of speech-driven 3D virtual human expression, sound and picture methods and systems. Background technique [0002] At present, there are several types of voice-driven methods for generating virtual human facial animations: [0003] (1) Speech generates the vertex coordinates of a 3D model with a fixed topology through the neural network, and these vertex coordinates can show facial animation on the DI4D PRO system. [0004] (2) Speech drives the avatar through the confrontation network to generate different 2D images, which are reflections of different angles of a 3D model. [0005] (3) Speech is split by phonemes, and each phoneme corresponds to an animation cl...

AI Technical Summary

Problems solved by technology

[0008] In order to overcome the problems that existing avatars do not have high naturalness of synchronizing facial expressions, audio and video, real-time interaction ability and learning ability to improve synchronous effects of facial expressions, audio and video, the present invention provides a self-supervised learning of synchronizing audio and video of virtual human method, by learning a large amount of face video data, to improve the effect of virtual human lips, making it more natural and human-like

Images