A facial motion capture method and system based on deep learning

Abstract

The present invention discloses a facial motion capture method and system based on deep learning, comprising the following steps: S1: using a depth camera to collect video data and corresponding depth data of a human face, and constructing a data set; S2: constructing a facial motion recognition network, Use the data set to train the facial action recognition network; S3: input any video sequence into the trained facial action recognition network to predict the mixed shape coefficient; S4: apply the predicted mixed shape coefficient to any avatar, drive The facial movements of the avatar. The system includes a video acquisition module, a network training module, a facial movement prediction module, and an avatar animation display module. The algorithm of the present invention has a high operation speed, and only depth information is used for training during training. In the prediction stage, only the video taken by a single camera can be input to complete motion capture, without additional depth acquisition equipment, and facial motion capture can be performed in real time .

Description

technical field [0001] The present invention relates to the technical fields of computer vision and computer graphics, in particular to a deep learning-based facial motion capture method and system. Background technique [0002] Facial motion capture is a part of motion capture technology, which refers to the process of using mechanical devices, cameras and other equipment to record human facial expressions and movements, and convert them into a series of parameter data. Compared with artificially produced animated character expressions, the characters generated by capturing the facial movements of real people will be more realistic, and can greatly reduce the cost of artificial modeling. Nowadays, motion capture technology has become an indispensable production tool in the fields of film and television animation production, game development, and virtual reality. [0003] The current mainstream methods can be divided into: based on two-dimensional data and based on three-di...

AI Technical Summary

Problems solved by technology

The former uses an optical lens to understand human facial expressions and movements through algorithms, such as Faceware’s helmet-mounted single-camera facial motion capture system. The advantages of this method are low cost, easy access, and easy to use. The disadvantage is that the capture accuracy is comparable to other methods Relatively low; the latter obtains two-dimensional data through an optical lens, and at the same time obtains depth information through additional means or devices, such as multi-eye cameras, structured light, etc. For example, Apple’s Animoji installs an infrared camera next to the front camera to collect depth information , this method has fast processing speed and high precision, but requires additional depth acquisition equipment

Images