Facial motion capture method and system based on deep learning

Abstract

The invention discloses a facial motion capture method and system based on deep learning, and the method comprises the following steps: S1, collecting video data of a human face and corresponding depth data through a depth camera, and constructing a data set; S2, constructing a facial action recognition network, and performing facial action recognition network training by using the data set; S3, inputting any video sequence into the trained facial action recognition network, and predicting a mixed shape coefficient; and S4, applying the predicted mixed shape coefficient to any virtual image, and driving the face action of the virtual image. The system comprises a video acquisition module, a network training module, a facial action prediction module and a virtual image animation display module. According to the method, the algorithm operation speed is high, the depth information is only used for training in the training process, motion capture can be completed only by inputting a video shot by a single camera in the prediction stage, extra depth collection equipment is not needed, and face motion capture can be conducted in real time.

Description

technical field [0001] The present invention relates to the technical fields of computer vision and computer graphics, and in particular, to a method and system for capturing facial motion based on deep learning. Background technique [0002] Facial motion capture is a part of motion capture technology, which refers to the process of using mechanical devices, cameras and other equipment to record human facial expressions and movements and convert them into a series of parameter data. Compared with human-made animated character expressions, the characters generated by capturing real-life facial movements will be more realistic, and the cost of manual modeling can be greatly reduced. Nowadays, motion capture technology has become an indispensable production tool in the fields of film and television animation production, game development, and virtual reality. [0003] Now the mainstream methods can be divided into: based on two-dimensional data and based on three-dimensional d...

AI Technical Summary

Problems solved by technology

The former uses an optical lens to understand human facial expressions and movements through algorithms, such as Faceware’s helmet-mounted single-camera facial motion capture system. The advantages of this method are low cost, easy access, and easy to use. The disadvantage is that the capture accuracy is comparable to other methods Relatively low; the latter obtains two-dimensional data through an optical lens, and at the same time obtains depth information through additional means or devices, such as multi-eye cameras, structured light, etc. For example, Apple’s Animoji installs an infrared camera next to the front camera to collect depth information , this method has fast processing speed and high precision, but requires additional depth acquisition equipment

Images