Micro-expression recognition method based on graph deep learning and 3D facial reconstruction

The integration of graph deep learning and three-dimensional face reconstruction enhances micro-expression recognition by extracting detailed facial muscle movements through a multi-stream network, addressing the limitations of two-dimensional methods in capturing subtle features.

CN114333002BActive Publication Date: 2024-12-03NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111610586.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-27
Publication Date
2024-12-03
Estimated Expiration
2041-12-27

AI Technical Summary

Technical Problem

Existing micro-expression recognition methods face challenges in accurately capturing and recognizing subtle facial muscle movements due to their short duration and reliance on two-dimensional spatiotemporal features, which are difficult to extract effectively using single deep learning networks.

Method used

A micro-expression recognition method utilizing graph deep learning and three-dimensional face reconstruction, incorporating a multi-stream OGC-FL network model that includes a graph feature learning module, optical flow feature learning module, and three-dimensional detail reconstruction module to extract sparse spatial and dense image information, enhancing feature extraction and recognition accuracy.

Benefits of technology

The method improves the recognition accuracy of spontaneous micro-expressions by leveraging three-dimensional reconstruction to supplement two-dimensional feature information, providing robust and detailed feature extraction through a multi-stream network structure.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114333002B_ABST
    Figure CN114333002B_ABST
Patent Text Reader

Abstract

The present invention is a micro-expression recognition method based on graph deep learning and three-dimensional reconstruction of human face, comprising the following steps: constructing a graph feature learning module, performing graph feature analysis to obtain a one-dimensional feature vector; constructing an optical flow feature learning module, obtaining a one-dimensional feature vector by extracting optical flow features; constructing a three-dimensional detail reconstruction module to obtain a one-dimensional feature vector; constructing a multi-stream OGC-FL network model structure, and obtaining a micro-expression recognition classification result by multi-stream fusion. Compared with a single strategy, the multi-strategy generation of optical flow features in the present invention can screen out the most favorable generation strategy for the micro-expression recognition task; the multi-stream OGC-FL network model structure of the present invention finds the consistency of facial key point information and dense image information in recognizing micro-expressions, the sparse spatial information of the key points can judge the general state of micro-expressions through GFL, and the dense image information highlights the subtle muscle movements of the face, extracting more detailed information for MER.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more specifically to a micro-expression recognition method based on graph deep learning and three-dimensional reconstruction of human faces. Background Art

[0002] Micro-expression recognition has the potential to discover a person's true emotions. Micro-expressions often only last for a very short period of time, staying on the face for about 1 / 25 to 1 / 3 of a second, and only local muscle movements occur, so it is very difficult to observe and recognize them correctly. In addition, micro-expressions are a spontaneous facial feature that is difficult to forge or suppress. Since micro-expression recognition can reveal a person's true feelings, it can be widely used in many fields that require accurate recognition of human emotions, such as national defense security, criminal investigation, psychopathology, social communication, etc.

[0003] In recent years, the potential benefits of micro-expression recognition in the information society have become increasingly greater. Researchers have noticed the research value of micro-expression recognition and have publicly published a variety of micro-expression datasets (such as SMIC, CASMEⅡ, SAMM). Micro-expression recognition has gradually become a research hotspot. Micro-expression recognition methods can be roughly divided into two parts: traditional machine learning and deep learning models. Some traditional machine learning methods focus on using hand-crafted spatiotemporal features to identify subtle facial muscle movements. They are all used to capture the subtle features of facial micro-expressions. In some cases, these spatiotemporal features can even better recognize micro-expressions than well-trained experts. Deep learning is a hot research topic. Inspired by the great success of deep learning in object detection and face recognition, researchers have begun to try to use neural networks to automatically generate reliable micro-expression feature descriptors. Traditional CNN networks (such as AlexNet, VGGNet, ResNet) are composed of a certain order of convolutional layers with fixed-size filters. These extension methods of convolutional neural networks have achieved gratifying results in the field of micro-expression recognition. With the maturity of GCN, relational modeling and graph deep learning for visual tasks have received increasing attention. Using the sparse information of designed facial expressions for expression recognition in graph deep learning can make the recognition results more robust. Extracting the subtle and short-term muscle movement features of micro-expressions on a two-dimensional image is a difficult task for a single deep learning network. From the existing feature extraction methods, micro-expression recognition mainly relies on some hand-crafted two-dimensional spatiotemporal feature descriptions to explain the spatiotemporal variation trend of facial expressions. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a micro-expression recognition method based on graph deep learning and three-dimensional reconstruction of the face, and explores the supplementary description of the lost content in the two-dimensional image feature information of micro-expressions by three-dimensional face reconstruction, which can improve the recognition accuracy of spontaneous micro-expressions, and develops a multi-stream OGC-FL network model structure of spatiotemporal features. The sparse spatial information of key points can be used to judge the approximate state of micro-expressions through GFL, while the dense image information highlights the subtle muscle movements of the face, thereby extracting more detailed information for MER.

[0005] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0006] The present invention is a micro-expression recognition method based on graph deep learning and three-dimensional reconstruction of human face, and the method comprises the following steps:

[0007] Step 1: Construct a graph feature learning module (hereinafter referred to as GFL) module. Based on the 3D reconstruction of the face, the 3D facial key point coordinates are obtained from the 2D face image, and the face key point graph structure is constructed. The adjacency matrix of the adaptive graph convolutional network is constructed using the face key point graph structure. The face key point graph structure is input into the adaptive graph convolutional network, and the graph feature analysis is performed to obtain a one-dimensional feature vector.

[0008] Step 2: Construct an optical flow feature learning module (hereinafter referred to as OFL) module. The first is optical flow feature extraction. The OFL module selects vertex frames from the micro-expression video sequence, and performs optical flow estimation on them and the initial frame in the video sequence to obtain horizontal optical flow features and vertical optical flow features. It receives horizontal and vertical optical flow information in a dual-channel manner, and connects the output results into a vector after passing through two convolutional neural networks with small convolution kernels respectively. After passing through a fully connected layer, a one-dimensional feature vector is finally obtained. The convolutional neural network with small convolution kernels is an improved form of the Inception network module.

[0009] Step 3: Construct a 3D detail reconstruction module (hereinafter referred to as DCL) module. Generate the facial normal vector features of the micro-expression start frame and vertex frame respectively through the detail expression capture and animation technology, and establish the time-space connection between the start frame and vertex frame through the absolute value difference of the normal vector.

[0010] I d =|I A -I O |

[0011] Among them, I A Represents the vertex frame normal vector matrix, I O Represents the starting frame normal vector matrix, I d represents the space-time contact matrix;

[0012] The connection between time and space is input into the MobileNet network as spatiotemporal features for feature extraction and analysis to obtain a one-dimensional feature vector.

[0013] Step 4: Add the feature vector obtained by the graph feature learning module in step 1, the feature vector obtained by the optical flow feature learning module in step 2, and the feature vector obtained by the three-dimensional detail reconstruction module in step 3 in sequence, obtain the score through the fully connected layer and the softmax layer, and finally form a multi-stream OGC-FL network model structure to obtain the micro-expression recognition and classification results.

[0014] A further improvement of the present invention is that the convolutional neural network with a small convolution kernel consists of three branches, the first branch is composed of two 3×3 convolution kernels connected in sequence with a 1x1 convolution kernel, the second branch is composed of two 1×1 and 3×3 convolution kernels, and the third branch is a single 1×1 convolution kernel. The third branch uses a Maxpool layer to downsample the original optical flow features.

[0015] The further improvement of the present invention is that the face key point graph structure in step 1 refers to the three-dimensional coordinate information of 68 facial key points and their adjacency relationship. The present invention defines an effective face key point graph structure, which conforms to the typical characteristics of facial expression muscle movement and can better characterize the spatial characteristics of micro-expressions.

[0016] The beneficial effects of the present invention are as follows: the method of the present invention extends graph deep learning to the field of micro-expression recognition, and the adjacency matrix of the adapted GFL network structure is constructed on the basis of the facial key points generated by face alignment; on the other hand, the present invention uses multiple strategies to generate optical flow images, and compared with a single strategy, the multi-strategy generation of optical flow features can screen out the generation strategy that is most beneficial to the micro-expression recognition task; the present invention selects the intermediate frame and the starting frame to allow the optical flow field to have a larger instantaneous change rate, which is helpful to highlight the muscle movement characteristics of different categories of micro-expressions; the present invention uses a convolutional neural network module with a small convolution kernel to construct an OFL module, divides the horizontal optical flow and the vertical optical flow into two paths of input convolutional neural network modules with small convolution kernels, and forms a multi-stream OGC-FL network model structure together with GFL and DCL, finds the consistency of facial key point information and dense image information in recognizing micro-expressions, the sparse spatial information of the key points can judge the general state of the micro-expression through GFL, and the dense image information highlights the subtle muscle movements of the face, so as to extract more detailed information for MER. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is the face key points and adjacency matrix diagram of the present invention.

[0018] Figure 2 It is the TVL1 horizontal optical flow and vertical optical flow diagram of the present invention.

[0019] Figure 3 It is a face time-space difference map between the start frame and the vertex frame of the present invention.

[0020] Figure 4 It is a structural diagram of the multi-stream OGC-FL network model of the present invention.

[0021] Figure 5 It is a neural network module of the small convolution kernel of the present invention.

[0022] Figure 6 The OFL module of the present invention and the dual-flow network structure therein.

[0023] Figure 7 It is the structural diagram of the DCL module of the present invention. DETAILED DESCRIPTION

[0024] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.

[0025] like Figure 1-7 As shown, the present invention is a micro-expression recognition method based on graph deep learning and face 3D reconstruction, comprising the following steps:

[0026] Step 1: Construct a graph feature learning (GFL) module. Graph deep learning technology has made great breakthroughs in the field of action recognition recently, and this invention extends it to the field of expression recognition. Figure 1 As shown, the present invention constructs a facial graph structure adapted to facial feature extraction and an adjacency matrix of GFL based on the coordinates of 68 key points obtained by face reconstruction. The facial key point information is input into the GFL module, and the graph feature analysis is performed to obtain sparse spatial features.

[0027] Step 2: Construct an optical flow feature learning (hereinafter referred to as OFL) module. Optical flow feature extraction: select vertex frames that can highlight micro-expression features from the micro-expression video sequence, and perform optical flow estimation on them and the initial frame showing a calm face in the video sequence. The basic expression of optical flow is as follows:

[0028] f x u+f y v+f t =0

[0029] in,

[0030] According to the different optical flow estimation strategies, a variety of optical flow algorithms are evaluated, and the TV-L1 optical flow method is selected, and its optimization equation is:

[0031]

[0032] The spatiotemporal expression information is enriched in the optical flow features. OFL can be used to reconstruct and refine the motion information in the optical flow and improve the effectiveness of the feature information for the recognition results. Figure 2 As shown in the figure, the OFL two-stream network better displays the deep spatiotemporal information of micro-expressions through horizontal and vertical optical flow features, and has better robust recognition ability than the OFL single-stream network.

[0033] Step 3: Construct a 3D detail reconstruction (hereinafter referred to as DCL) module, such as Figure 3 As shown, we use the facial normal vector matrix generated by the detailed expression capture technology to establish the time-space connection between the starting frame and the vertex frame.

[0034] I d =|I A -I O |

[0035] Among them, I A Represents the vertex frame normal vector matrix, I O Represents the starting frame normal vector matrix, I d Represents the space-time contact matrix.

[0036] The connection between time and space is input into the MobileNet network as spatiotemporal features for feature extraction and analysis, and together with the GFL module and OFL module, a multi-stream network structure is formed.

[0037] Step 4: Through multi-stream fusion, the micro-expression recognition and classification results are finally obtained.

[0038] like Figure 4 As shown, the method proposed in the present invention consists of four modules: optical flow feature extraction based on OFL, facial geometric feature extraction based on GFL, three-dimensional reconstruction of the face, and multi-stream fusion. Among them, three-dimensional reconstruction of the face is used to generate dense facial geometric features. Optical flow feature extraction is used to extract the horizontal optical flow component and the vertical optical flow component of micro-expressions. Optical flow feature extraction based on OFL can be used to reconstruct and refine the motion information in the optical flow, and improve the effectiveness of feature information for recognition results. Facial geometric feature extraction based on GFL is used to extract sparse spatial information contained in facial key points. The three-dimensional reconstruction module of the face generates a high-precision facial expansion image after the original face image passes through the detailed expression capture process, and generates detailed expression expansion images of the micro-expression starting frame and the vertex frame respectively, obtains the time-space connection between the two, and inputs the generated picture into the convolutional neural network to obtain the feature vector of the output layer.

[0039] like Figure 5As shown, the optical flow feature extraction module is composed of a convolutional neural network module with a small convolution kernel, which is an improved form of the Inception network module. The present invention uses 1×1, 3×3 convolution layers to reduce the complexity of calculation. The convolutional neural network with a small convolution kernel is composed of three branches. The first branch connects two 3×3 convolution kernels in sequence with a 1x1 convolution kernel, the second branch is composed of two 1×1 and 3×3 convolution kernels, and the third branch is a single 1×1 convolution kernel. The third branch uses the Maxpool layer to downsample the original optical flow features, so that more texture information of micro-expressions can be retained, and features with better classification recognition are selected to obtain better results. The horizontal and vertical optical flow information is received in a dual-channel manner. After passing through two convolutional neural network modules respectively, the output results are connected into a vector.

[0040] The face geometric feature extraction module of GFL contains 10 graph convolution units, each of which consists of a graph convolution structure and a residual structure. The convolution kernel size of the graph convolution structure is 12 for the first three units, 24 for the middle four units, and 48 for the last three units; the residual structure is composed of residual blocks, which can deepen the network layer and prevent gradient explosion while retaining effective information as much as possible. In order to make GFL suitable for micro-expression recognition tasks, a global average pooling layer and a fully connected layer are added after the 10 graph convolution units. The output of the fully connected layer is a 256-dimensional vector. Therefore, the output of the geometric feature extraction module is a 256-dimensional expression category vector.

[0041] The feature vector of the DCL module is added to the facial geometric features of GFL and the optical flow features of OFL to obtain the final feature result of the multi-stream network, and the micro-expression recognition result is obtained through the final softmax layer.

[0042] The entire training process adopted the SGD optimization method and the cross entropy loss function. The initial learning rate was set to 0.1, which automatically decreased tenfold at 50, 100, and 150 epochs after 200 epochs of training.

[0043] The present invention develops an end-to-end spatiotemporal feature multi-stream OGC-FL network model structure, aiming to improve the recognition accuracy of spontaneous micro-expressions. The network extracts sparse spatial features through a topological graph structure composed of key points of the face. The optical flow of the micro-expression video sequence is used to extract the temporal features in the sequence. The combination of the two can further improve the feature extraction quality of the spatiotemporal features of facial expressions in the micro-expression sequence. In addition to learning the above-mentioned spatiotemporal features, the network reconstructs the three-dimensional face and amplifies the spatiotemporal differences of fine texture features, ultimately forming a multi-stream structure that can extract spatiotemporal features. We conducted a large number of experiments on the CASME2, ​​SMIC and SAMM micro-expression joint databases. The results show that the hybrid neural network structure proposed in the present invention is effective for micro-expression recognition in laboratory scenarios, with good recognition rate and robustness.

[0044] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A micro-expression recognition method based on graph deep learning and 3D facial reconstruction. Features: The micro-expression recognition method comprises the following steps: Step 1: Construct a graph feature learning module: Based on the 3D reconstruction of the face, the 3D facial key point coordinates are obtained from the 2D face image, and the face key point graph structure is constructed. The adjacency matrix of the adaptive graph convolutional network is constructed using the face key point graph structure. The face key point graph structure is input into the adaptive graph convolutional network, and the graph feature analysis is performed to obtain a one-dimensional feature vector. Step 2: Construct an optical flow feature learning module and obtain a one-dimensional feature vector through optical flow feature extraction; Step 3: Construct a 3D detail reconstruction module to obtain a one-dimensional feature vector; Step 4: Construct a multi-stream OGC-FL network model structure and obtain micro-expression recognition and classification results through multi-stream fusion. in, The step 3 is specifically as follows: generating the facial normal vector features of the micro-expression start frame and the vertex frame respectively through the detail expression capture and animation technology, establishing the time-space connection between the start frame and the vertex frame through the normal vector absolute value difference, I d =|I A -I O | Among them, I A Represents the vertex frame normal vector matrix, I O Represents the start frame normal vector matrix, I d represents the space-time contact matrix; The connection between time and space is input into the MobileNet network as the time and space features for feature extraction and analysis to obtain a one-dimensional feature vector. The step 4 is specifically as follows: the feature vector obtained by the graph feature learning module in step 1, the feature vector obtained by the optical flow feature learning module in step 2, and the feature vector obtained by the three-dimensional detail reconstruction module in step 3 are added in sequence, scores are obtained through a fully connected layer and a softmax layer, and finally a multi-stream OGC-FL network model structure is formed to obtain a micro-expression recognition classification result.

2. According to the method for micro-expression recognition based on graph deep learning and face 3D reconstruction according to claim 1, Features: The steps of constructing the optical flow feature learning module specifically include: Step 2-1: Optical flow feature extraction: The optical flow feature learning module selects vertex frames from the micro-expression video sequence; Step 2-2: Perform optical flow estimation on the vertex frame and the initial frame in the video sequence to obtain a two-stream network structure including horizontal optical flow features and vertical optical flow features; Step 2-3: The dual-stream network structure receives horizontal and vertical optical flow information in a dual-channel manner, and after passing through two convolutional neural networks with small convolution kernels respectively, the output results are connected into a vector; Step 2-4: After passing through a fully connected layer, a one-dimensional feature vector is finally obtained.

3. According to the micro-expression recognition method based on graph deep learning and face 3D reconstruction as claimed in claim 2, Features: The convolutional neural network with a small convolution kernel in step 2-3 consists of three branches. The first branch connects two 3×3 convolution kernels in sequence with a 1x1 convolution kernel. The second branch consists of two 1×1 and 3×3 convolution kernels. The third branch is a single 1×1 convolution kernel. The third branch uses a Maxpool layer to downsample the original optical flow features.

4. According to the method for micro-expression recognition based on graph deep learning and face 3D reconstruction as claimed in claim 1, Features: The facial key point graph structure in step 1 refers to the three-dimensional coordinate information of 68 facial key points and their adjacency relationships.