Method and system for realizing video and audio driven face animation combined with modal particle features

Abstract

A method and system for video-audio-driven face animation combined with modal particle features, constructing a speech feature matrix by extracting speech features, using modal particles to enhance the multi-layer convolution operation of the training network to sample the feature matrix and map it to a low-dimensional space intermediate variable; convert the input speech into text, identify the modal particle from the text content and construct a one-hot vector, and splice with the intermediate variable to obtain an intermediate variable containing the characteristics of the modal particle; After the product is mapped to the expression AU parameters of the current frame, it is used to fit the AU parameters generated by the video tracking and voice prediction algorithms and then used as the driving parameters of the face model to achieve expression enhancement. In the present invention, by inputting the video content of the user's face and the audio content of the user's voice, the three-dimensional Avatar model in the virtual scene can be jointly driven, and on the basis of real-time driving, the overall and partial facial animation can be more realistic and vivid. Effect.

Description

technical field [0001] The present invention relates to a technology in the field of computer graphics, in particular to a method and system for realizing video-audio-driven facial animation combined with features of modal particles. Background technique [0002] Existing implementations of facial expression animation include traditional interaction modeling and key frame animation methods, motion capture methods based on facial marker tracking, driving methods based on video stream images, and driving methods based on audio prediction. Among them, interactive modeling and key frame animation methods are widely used in games, 3D animation and other fields, and are the mainstream methods for producing high-precision 3D facial animation. This method has the advantages of high precision, mature technology, and is suitable for assembly line production, but it requires long-term settings and adjustments by modelers and animators, which is time-consuming and labor-intensive, and t...

AI Technical Summary

Problems solved by technology

[0004] The present invention requires a high degree of production cost and time period in the prior art, and the single-based video stream driving method and the audio driving method have their own disadvantages, and neither of them takes into account the defects of the characteristics of the modal particle, and proposes a method of combining the characteristics of the modal particle The face animation method and system driven by video and audio, by inputting the video content of the user's face and the audio content of the user's voice, can jointly drive the three-dimensional Avatar model in the virtual scene, and make the overall and partial facial animation on the basis of real-time driving more realistic and vivid performance

Images