Audio-driven face animation generation method and system fused with emotion coding

Abstract

The invention belongs to the field of artificial intelligence, and relates to an audio-driven face animation generation method and system fused with emotion coding, and the method comprises the steps: firstly carrying out the preprocessing of a collected audio signal, extracting MFCC features, inputting the features into a voice recognition module, further extracting audio features, inputting the MFCC features into a voice emotion recognition module, obtaining an emotion category and performing one-hot coding, then connecting audio features and one-hot coding vectors of emotions, inputting the connected vectors into an expression recognition module to obtain an expression coefficient based on a 3DMM model, and finally inputting the expression coefficient and a face template into a face animation generation module to obtain a 3D face animation with expressions. The method is small in calculation amount, stable in training, simple in process and low in cost, the movie making period and cost can be greatly reduced, the emotional state conveyed by voice is fully considered, emotional codes are input into the network, the generated face animation is more vivid, and better experience can be brought to a user.

Description

technical field [0001] The invention belongs to the field of artificial intelligence, and in particular relates to an audio-driven face animation generation method and system incorporating emotion coding. Background technique [0002] In recent years, with the continuous development of artificial intelligence, cross-modal learning and modeling techniques have attracted more and more attention in interdisciplinary research such as computer vision, computer graphics, and multimedia. Visual and auditory modalities are two important sensory channels in human-to-human or human-computer interaction. There is a strong correlation between audio and facial animation, that is, many facial movements are directly caused by speech production. Therefore, understanding the correlation between speech and facial movements can provide additional assistance in analyzing human behavior. Audio-driven facial animation technology has a wide range of application scenarios, such as virtual anchors...

AI Technical Summary

Problems solved by technology

However, on the one hand, modal particles cannot fully reflect the real emotional state of the speaker; on the other hand, if the sentence does not contain modal particles, the method cannot judge the emotional state

In addition, this method needs to extract AU parameters from the video at the same time and weight them with the AU parameters of audio prediction, so the complexity of the algorithm is high

Images