Method and system for realizing video and audio driven face animation by combining modal particle characteristics

Abstract

The invention discloses a method and system for realizing video and audio driven face animation by combining modal particle characteristics, and the method comprises the steps: constructing a voice feature matrix through the extraction of voice features, carrying out the sampling of the feature matrix through the multilayer convolution operation of a modal particle enhancement training network, and mapping the feature matrix to an intermediate variable of a low-dimensional space; converting the input voice into characters, identifying a mood word from the character content, constructing a onehot vector, and splicing the onehot vector with an intermediate variable to obtain an intermediate variable containing mood word features; carrying out the convolution through a modal particle enhancement training network, and then obtaining an expression AU parameter of a current frame through mapping, wherein the parameter is used for being fitted with an AU parameter generated by video tracking and voice prediction algorithms to serve as a driving parameter of a face model, so expression enhancement is realized. According to the invention, the three-dimensional Avatar model in the virtual scene can be jointly driven by inputting the video content of the face of the user and the audio content of the sound production of the user, and on the basis of real-time driving, the whole and local face animations are enabled to obtain more realistic and vivid expression effects.

Description

technical field [0001] The present invention relates to a technology in the field of computer graphics, in particular to a method and system for realizing video-audio-driven facial animation combined with features of modal particles. Background technique [0002] Existing implementations of facial expression animation include traditional interaction modeling and key frame animation methods, motion capture methods based on facial marker tracking, driving methods based on video stream images, and driving methods based on audio prediction. Among them, interactive modeling and key frame animation methods are widely used in games, 3D animation and other fields, and are the mainstream methods for producing high-precision 3D facial animation. This method has the advantages of high precision, mature technology, and is suitable for assembly line production, but it requires long-term settings and adjustments by modelers and animators, which is time-consuming and labor-intensive, and t...

AI Technical Summary

Problems solved by technology

[0004] The present invention requires a high degree of production cost and time period in the prior art, and the single-based video stream driving method and the audio driving method have their own disadvantages, and neither of them takes into account the defects of the characteristics of the modal particle, and proposes a method of combining the characteristics of the modal particle The face animation method and system driven by video and audio, by inputting the video content of the user's face and the audio content of the user's voice, can jointly drive the three-dimensional Avatar model in the virtual scene, and make the overall and partial facial animation on the basis of real-time driving more realistic and vivid performance

Images