Facial action unit recognition method and device based on joint learning and optical flow estimation

Abstract

The invention discloses a facial action unit recognition method and device based on joint learning and optical flow estimation. The method comprises the following steps: firstly extracting an original image pair needed by model training from video data to form a training data set, then preprocessing the original image pair to obtain an amplified image pair, constructing a convolutional neural network module I to extract multi-scale regional features of the amplified image pair, constructing a convolutional neural network module II to extract static global features of the amplified image pair, constructing a convolutional neural network module III to extract optical flow features of the amplified image pair, and finally, constructing a convolutional neural network module IV, fusing the static global features and the optical flow features, and carrying out facial action unit recognition. An end-to-end deep learning framework is adopted to jointly learn action unit recognition and optical flow estimation, action unit recognition is promoted by utilizing the relevance between tasks, the motion condition of facial muscles in a two-dimensional image can be effectively recognized, and unified facial action unit recognition system construction is achieved.

Description

technical field [0001] The invention relates to a facial action unit recognition method and device based on joint learning and optical flow estimation, belonging to computer vision technology. Background technique [0002] Optical flow estimation is a fundamental research task in computer vision, which is the bridge and link connecting images and videos. The core idea is to estimate the pixel-by-pixel correspondence given two frames of images before and after. This can also be approximately understood as a projected motion field of a 3D object on a 2D image plane. Optical flow plays an important role in behavior understanding, video processing, motion prediction, multi-view 3D reconstruction, autonomous driving, and real-time localization and mapping (SLAM). [0003] In order to study human facial expressions in more detail, Ekman, a famous American emotional psychologist, proposed the Facial Action Coding System (FACS) for the first time in 1978, and made important improv...

AI Technical Summary

Problems solved by technology

[0007] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a facial action unit recognition method and device based on joint learning and optical flow estimation, which automatically extracts ...

Images