A real-world facial expression recognition method based on 3D facial feature reconstruction and graph deep learning

Through three-dimensional facial feature reconstruction and deep learning technology, facial appearance and geometric features are extracted, and information is fused using channel attention mechanism, the accuracy and robustness of facial expression recognition in real environments are solved, achieving higher recognition Accuracy and stability.

CN113989911BActive Publication Date: 2024-11-08NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111483036.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-07
Publication Date
2024-11-08
Estimated Expiration
2041-12-07

AI Technical Summary

Technical Problem

The existing facial expression recognition methods are difficult to effectively identify in real environments due to factors such as lighting, occlusion and posture changes, and the recognition accuracy and robustness are insufficient.

Method used

Three-dimensional facial feature reconstruction technology is used to generate facial appearance features and geometric features, and feature information is extracted separately through convolutional neural networks and graph neural networks, combined with channel attention mechanisms to build an end-to-end 3DF-RLN model for expression recognition.

Benefits of technology

The accuracy and robustness of facial expression recognition in real environments is improved, and the classification accuracy is 2% higher than that of the state-of-the-art methods on the RAF-DB and AffectNet datasets, which significantly improves the current situation of facial expression recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113989911B_ABST
    Figure CN113989911B_ABST
Patent Text Reader

Abstract

A real-world facial expression recognition method based on 3D facial feature reconstruction and graph deep learning is proposed. Through the establishment of a 3D dense face reconstruction network and an end-to-end trainable 3D facial feature reconstruction and learning network model 3DF‑RLN, facial appearance and facial geometric features are reconstructed from a single 2D face image, effectively representing facial expression information. The facial topology map based on facial key points is obtained from the facial geometric features, which can reflect the correlation between facial geometric features and has great significance for facial expression recognition and related research. The CNN network effectively extracts the expression information contained in the facial appearance features, and the GCN network effectively extracts the information contained in the facial geometric features. The fusion recognition module composed of channel attention and softmax effectively fuses the complementary information contained in the facial appearance features and the facial geometric features, thereby improving the accuracy of expression recognition. In general, this method improves the accuracy of facial expression recognition and improves the effect of facial expression recognition in real environments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a real-environment facial expression recognition method based on three-dimensional facial feature reconstruction and graph deep learning. Background Art

[0002] In the prior art, facial expression recognition, as one of the important branches in the field of computer vision, has been successfully applied to telemedicine, fatigue driving monitoring, and many human-computer interaction systems. Many existing expression recognition methods have achieved remarkable success on expression databases in laboratory environments. However, real-world facial expression images are still subject to the influence of arbitrary lighting, occlusion, and posture changes, which poses a huge challenge to existing facial expression recognition methods.

[0003] In recent years, with the advancement of face 3D reconstruction technology, the conversion from a single 2D facial image to the corresponding 3D facial data has been achieved while ensuring speed, accuracy and stability. Existing technologies can not only provide facial appearance features rich in texture information, but also generate reliable and high-precision facial geometric features. Facial appearance features can be saved as smooth 2D images. Researchers have achieved great success in the field of image-based recognition using convolutional neural networks. Facial geometric features refer to a fixed number of facial key point sequences, which have the advantages of small data volume and less susceptible to interference compared to 2D image sequences. Facial key point sequences have been proven to effectively represent changes in facial expressions. At present, graph neural networks have been successfully applied in behavior recognition based on skeleton sequences. Facial key points have similar spatial features and data forms to human skeleton sequences, and graph neural networks can be used to extract facial expression information contained in facial key points. Facial appearance features and facial geometric features are features of two different modalities, which respectively reflect the dense representation and sparse representation of facial images, and can complement each other to a certain extent. In addition, studying appropriate facial topology encoding methods can more effectively apply graph neural networks to the field of expression recognition and bring new solutions to facial expression recognition in real environments. Summary of the invention

[0004] The purpose of the present invention is to provide a real-world facial expression recognition method based on three-dimensional facial feature reconstruction and graph deep learning. The method generates facial appearance features and facial geometric features through three-dimensional dense face reconstruction technology, and uses convolutional neural networks and graph neural networks to extract the expression information contained in the two features respectively, and then fuses the two feature information through a channel attention mechanism, thereby improving the recognition rate and robustness of real-world facial expression recognition. In addition, the facial geometric features represented by the three-dimensional coordinate information of several facial key points can obtain a facial topology map based on the facial key points through learning, and the topology map can reflect the correlation between facial geometric features.

[0005] A real-world facial expression recognition method based on three-dimensional facial feature reconstruction and graph deep learning comprises the following steps:

[0006] Step 1: Input the real-world facial expression image into a three-dimensional dense face reconstruction network, and output facial appearance features and facial geometric features; the facial appearance features are the reconstructed appearance features obtained by frontalizing the multi-pose face image; the facial geometric features are the three-dimensional coordinate information of several facial key points, which are then learned to obtain a facial topology map based on the facial key points;

[0007] Step 2: Build an end-to-end trainable 3D facial feature reconstruction and learning network model 3DF-RLN. 3DF-RLN includes an appearance channel based on a convolutional neural network, a geometric channel based on a graph neural network, and a fusion recognition module.

[0008] Step 3: Use the facial appearance features and facial geometry features obtained in step 1 to train the 3DF-RLN model;

[0009] Step 4: When identifying the expression category of a new face image, the corresponding facial appearance features and facial geometric features obtained in step 1 are first input into the 3DF-RLN network model to finally obtain the expression recognition result of the face image.

[0010] Furthermore, the three-dimensional dense face reconstruction network in step 1 uses a convolutional neural network to solve posture, shape, and expression parameters to render an average three-dimensional face model, and obtains facial appearance features and facial geometric features in the process; wherein the posture, shape, and expression parameters are 12-dimensional, 40-dimensional, and 10-dimensional vectors, respectively.

[0011] Furthermore, the facial appearance features and facial geometric features in step 1; wherein the facial appearance features are posture adaptive features obtained in the process of three-dimensional face feature reconstruction, which can be regarded as quasi-frontal features; the facial geometric features refer to the three-dimensional coordinate information of 68 facial key points.

[0012] Furthermore, the appearance channel and geometric channel in step 2; the appearance channel is composed of a convolutional neural network, which is composed of 4 residual convolution blocks, 4 channel attention blocks, and 4 spatial attention blocks stacked, and the connection order is that each residual convolution block is followed by a channel attention block and a spatial attention block; the geometric channel is composed of a graph neural network, which is composed of 7 graph residual blocks, 7 channel attention blocks, and 7 spatial attention blocks stacked, and the connection order is that each graph residual block is followed by a channel attention block and a spatial attention block; wherein, in order to adapt to the expression recognition task, 2 fully connected layers, 1 normalization layer and 1 activation layer are added at the end of the two channels, and the output dimension is 256.

[0013] Furthermore, the fusion recognition module in step 2; the fusion recognition module consists of two parts: a channel attention block and a softmax, wherein the channel attention block is used to assign appropriate weights to the outputs of the appearance channel and the geometric channel, and the softmax is used to output the expression recognition results; the channel attention block structure in the fusion recognition module is consistent with the channel attention block structure used in the appearance channel or the geometric channel.

[0014] Furthermore, the structure of the convolutional neural network used in the three-dimensional dense face reconstruction network in step 1 is consistent with the convolutional neural network used in the appearance channel, but the final output dimension is 62.

[0015] Furthermore, the residual convolution block contains 2 convolution layers, 2 normalization layers, 2 activation layers, and 1 residual connection; the graph residual block contains 3 convolution layers, 1 normalization layer, 1 activation layer, and 1 residual connection; the internal connection order of the residual convolution block and the graph residual block is that each convolution layer is followed by a normalization layer and an activation layer, and finally followed by a residual connection; the channel attention block contains 2 convolution layers, 1 average pooling layer, 1 maximum pooling layer, and 2 fully connected layers, and the internal connection order is one convolution layer followed by a maximum pooling layer and a fully connected layer, and the other convolution layer is followed by an average pooling layer and a fully connected layer, and then the outputs of the two fully connected layers are added; the spatial attention block contains 1 convolution layer and 1 sigmoid layer, and the internal connection order is the convolution layer followed by the sigmoid layer.

[0016] The beneficial effects of the present invention are as follows: 3DF-RLN can reconstruct facial appearance and facial geometric features from a single 2D face image, and these two heterogeneous features effectively characterize facial expression information. Facial geometric features can also be learned to obtain a facial topology map based on facial key points, which can reflect the correlation between facial geometric features and is of great significance to facial expression recognition and related research. Convolutional neural networks can effectively extract expression information contained in facial appearance features, and graph neural networks can effectively extract information contained in facial geometric features. The fusion recognition module composed of channel attention and softmax can effectively fuse the complementary information contained in facial appearance features and facial geometric features, thereby improving the accuracy of expression recognition. In general, the 3DF-RLN proposed in the present invention improves the accuracy of facial expression recognition, and the classification accuracy obtained on the two real-environment expression data sets RAF-DB and AffectNet is 2% higher than the current state-of-the-art level; the current status of real-environment facial expression recognition is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 4 is a flow chart of a facial expression recognition method in an embodiment of the present invention.

[0018] Figure 2 Schematic diagram of facial appearance features in an embodiment of the present invention.

[0019] Figure 3 Schematic diagram of facial geometric features in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0021] Step 1: Input the real-world facial expression image into the 3D dense face reconstruction network and output facial appearance features and facial geometric features. Facial appearance features are as follows: Figure 2 As shown, the present invention saves it as a smooth two-dimensional image, and the facial geometric features are as follows Figure 3 As shown, it is composed of the three-dimensional coordinate information of 68 facial key points; among them, the facial appearance feature is the reconstructed appearance feature obtained by frontalizing the multi-pose face image. In order to facilitate feature extraction, the present invention processes it into a smooth two-dimensional image; the facial geometric feature is the three-dimensional coordinate information of several facial key points, and the feature can be used to obtain a facial topology map based on the facial key points through learning;

[0022] Step 2: Build an end-to-end trainable 3D facial feature reconstruction and learning network (hereinafter referred to as 3DF-RLN). 3DF-RLN consists of appearance channel, geometry channel and fusion recognition modules.

[0023] Step 3: Use the facial appearance features and facial geometry features obtained in step 1 to train the 3DF-RLN model.

[0024] Step 4: When identifying the expression category of a new face image, the corresponding facial appearance features and facial geometric features obtained in step 1 are first input into the 3DF-RLN network model to finally obtain the expression recognition result of the face image.

[0025] like Figure 1 As shown, the method proposed in the present invention consists of four modules: three-dimensional dense face reconstruction, appearance channel based on convolutional neural network, geometric channel based on graph neural network, and fusion recognition. Among them, three-dimensional dense face reconstruction uses convolutional neural network to solve posture, shape, and expression parameters to render an average three-dimensional face model, and obtains facial appearance features and facial geometric features in the process. The appearance channel based on convolutional neural network is used to extract the expression information contained in the facial appearance features. The geometric channel based on graph neural network is used to extract the expression information contained in the facial geometric features. In order to enhance the feature extraction capabilities of convolutional neural network and graph neural network, the present invention introduces channel and spatial attention mechanisms into the network. The facial geometric features based on the key points of the face are input into the graph convolutional network as learnable parameters, and a facial topology map can be learned, which can reflect the correlation between facial geometric features. The fusion recognition module uses the channel attention mechanism to effectively fuse the expression category vectors output by the appearance channel and the geometric channel, thereby strengthening the extraction and integration of the two heterogeneous features. Finally, softmax is input to obtain the expression recognition result.

[0026] The input of the appearance channel is the facial appearance features processed into a two-dimensional image, which can be regarded as the quasi-frontal features of the real-world facial expression image, which restores the lost information in the real-world facial expression image while retaining rich facial detail features. The appearance channel based on the convolutional neural network is composed of 4 residual convolution blocks. In order to improve the spatial feature extraction capability of the model, the present invention introduces channel and spatial attention blocks in each residual convolution block. Among them, the residual convolution block includes 2 convolution layers, 2 normalization layers, 2 activation layers, and 1 residual connection; the channel attention layer includes 2 convolution layers, 1 average pooling layer, 1 maximum pooling layer, and 2 fully connected layers; the spatial attention layer includes 1 convolution layer and 1 sigmoid layer. The channel attention block takes the average pooling and the maximum pooling of the output features of the convolution layer along the channel dimension and adds them together. After passing through the nonlinear layer, the attention score is obtained and multiplied by the original feature; the spatial attention block takes the average and the maximum value of the output features of the convolution layer along the spatial dimension and splices them. After passing through the nonlinear layer, the attention score is obtained and multiplied by the original feature; the residual connection can deepen the number of network layers and prevent gradient explosion while retaining effective information as much as possible. In order to make the convolutional neural network suitable for expression recognition tasks, the present invention adds two fully connected layers after the network, and adds a normalization layer and an activation layer between the two fully connected layers. The output of the convolutional neural network is a 1000-dimensional vector, in which the input and output of the first fully connected layer added are (1000, 512), and the input and output of the second fully connected layer are (512, 256). Therefore, the output of the appearance channel is a 256-dimensional expression category vector.

[0027] The input features of the geometric channel are the facial geometric features based on the facial key points. The present invention uses the adjacency matrix reflecting the facial geometric features as a trainable parameter and updates it iteratively during the training process. At the same time, the present invention introduces a similarity matrix, which is obtained by multiplying the facial key point features by its transpose. During the training process, adding the trainable adjacency matrix features to the similarity matrix can more accurately optimize the adjacency relationship between the facial geometric features.

[0028] The geometric channel based on graph neural network consists of 7 graph residual blocks, each of which has channel and spatial attention blocks added to it. Among them, the graph residual block contains 3 convolutional layers, 1 normalization layer, 1 activation layer, and 1 residual connection; the channel and spatial attention block structures are the same as those in the convolutional neural network. In order to make the graph neural network suitable for expression recognition tasks, 2 fully connected layers are added after the 7 graph convolution units, and 1 normalization layer and 1 activation layer are added between the 2 fully connected layers. The output of the convolutional neural network is a 512-dimensional vector, where the input and output of the first fully connected layer added are (512, 256), and the input and output of the second fully connected layer are (256, 256). Therefore, the output of the geometric channel is a 256-dimensional expression category vector.

[0029] The fusion recognition module consists of two parts: channel attention and softmax. The expression category vectors extracted from the appearance channel and the geometric channel are spliced ​​and connected to the channel attention mechanism. The specific method of the channel attention mechanism is to multiply the attention score obtained after the spliced ​​features pass through the nonlinear layer to the original features to strengthen the extraction and integration of the spatiotemporal features of the two networks. Finally, the expression recognition result is obtained through softmax. The classification accuracy of the 3DF-RLN model on the two real-world expression datasets RAF-DB and AffectNet is 2% higher than the current most advanced level, which also verifies the effectiveness of the method proposed in the present invention.

[0030] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by ordinary technicians in this field based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.

Claims

1. A real-world facial expression recognition method based on three-dimensional facial feature reconstruction and graph deep learning, characterized in that: The method comprises the following steps: Step 1: Input the real-world facial expression image into a three-dimensional dense face reconstruction network, and output facial appearance features and facial geometric features; the facial appearance features are the reconstructed appearance features obtained by frontalizing the multi-pose face image; the facial geometric features are the three-dimensional coordinate information of several facial key points, which are then learned to obtain a facial topology map based on the facial key points; Step 2: Build an end-to-end trainable 3D facial feature reconstruction and learning network model 3DF-RLN. 3DF-RLN includes an appearance channel based on a convolutional neural network, a geometric channel based on a graph neural network, and a fusion recognition module. The appearance channel and geometric channel in step 2; the appearance channel is composed of a convolutional neural network, which is composed of 4 residual convolution blocks, 4 channel attention blocks, and 4 space attention blocks stacked together, and the connection order is that each residual convolution block is followed by a channel attention block and a space attention block; the geometric channel is composed of a graph neural network, which is composed of 7 graph residual blocks, 7 channel attention blocks, and 7 space attention blocks stacked together, and the connection order is that each graph residual block is followed by a channel attention block and a space attention block; wherein, in order to adapt to the expression recognition task, 2 fully connected layers, 1 normalization layer, and 1 activation layer are added to the end of the two channels, and the output dimension is 256; The fusion recognition module in step 2; the fusion recognition module consists of a channel attention block and a softmax, wherein the channel attention block is used to assign appropriate weights to the outputs of the appearance channel and the geometric channel, and the softmax is used to output the expression recognition result; the channel attention block structure in the fusion recognition module is consistent with the channel attention block structure used in the appearance channel or the geometric channel; Step 3: Use the facial appearance features and facial geometry features obtained in step 1 to train the 3DF-RLN model; Step 4: When identifying the expression category of a new face image, the corresponding facial appearance features and facial geometric features obtained in step 1 are first input into the 3DF-RLN network model to finally obtain the expression recognition result of the face image.

2. A real environment facial expression recognition method based on three-dimensional face feature reconstruction and graph deep learning according to claim 1, characterized in that: The 3D dense face reconstruction network in step 1 uses a convolutional neural network to solve the posture, shape, and expression parameters to render an average 3D face model, and obtains facial appearance features and facial geometric features in the process. in The posture, shape, and expression parameters are 12-dimensional, 40-dimensional, and 10-dimensional vectors, respectively.

3. A real environment facial expression recognition method based on three-dimensional face feature reconstruction and graph deep learning according to claim 1, characterized in that: The facial appearance features and facial geometric features in step 1; wherein the facial appearance features are posture adaptive features obtained in the process of three-dimensional facial feature reconstruction, which are regarded as quasi-frontal features; the facial geometric features refer to the three-dimensional coordinate information of 68 facial key points.

4. A real environment facial expression recognition method based on three-dimensional face feature reconstruction and graph deep learning according to claim 2, characterized in that: The structure of the convolutional neural network used in the 3D dense face reconstruction network is consistent with the convolutional neural network used in the appearance channel, but the final output dimension is 62.

5. The real-world facial expression recognition method based on three-dimensional facial feature reconstruction and graph deep learning according to claim 3, characterized in that: The residual convolution block contains 2 convolution layers, 2 normalization layers, 2 activation layers, and 1 residual connection; the graph residual block contains 3 convolution layers, 1 normalization layer, 1 activation layer, and 1 residual connection; the internal connection order of the residual convolution block and the graph residual block is that each convolution layer is followed by a normalization layer and an activation layer, and finally followed by a residual connection; the channel attention block contains 2 convolution layers, 1 average pooling layer, 1 maximum pooling layer, and 2 fully connected layers. The internal connection order is one convolution layer followed by a maximum pooling layer and a fully connected layer, and the other convolution layer is followed by an average pooling layer and a fully connected layer, and then the outputs of the two fully connected layers are added; the spatial attention block contains 1 convolution layer and 1 sigmoid layer, and the internal connection order is a convolution layer followed by a sigmoid layer.