A facial expression recognition method and system based on global self-attention mechanism and geometric deep learning

Abstract

The application provides a facial expression recognition method and system based on a global self-attention mechanism and geometric deep learning, and the method comprises the following steps: pre-processing a real environment facial expression image, segmenting and extracting key points from the same image respectively to obtain an equal division image block and a facial key point topology map; constructing an end-to-end trainable neural network model AGT based on a global self-attention mechanism; using the facial expression image block and the facial key point topology obtained in the foregoing steps as input to train the AGT; and when recognizing the expression category of a new facial image, inputting the facial expression image block and the facial key point topology obtained in the foregoing steps into the AGT network model to obtain the expression category of the image. The method and system improve the recognition rate and robustness of real environment facial expression recognition, and furthermore, the expression information is extracted from facial topologies of different scales, so that the feature extraction capability of the network is further improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a method and system for facial expression recognition in real-world environments based on a global self-attention mechanism and geometric deep learning. Background Technology

[0002] Facial expression analysis has become an important area of ​​research in computer vision and artificial intelligence, with potential applications in fields such as distance education, safe driving, and service robots, where emotional human-computer interaction is required. Existing expression recognition methods have achieved extremely high accuracy rates on laboratory datasets. However, real-world expression data contains a large amount of interference, such as complex lighting conditions, uncertain poses, and occlusions, which significantly impacts the accuracy of expression recognition.

[0003] Existing deep learning-based facial expression recognition algorithms primarily rely on convolutional operations to extract mid-to-deep features from the input. However, convolutional kernels have limited receptive fields, limiting their ability to extract global features and regional relationships. They also lack the ability to adaptively learn the importance of different parts, leading to a significant drop in recognition accuracy when faced with classification tasks involving images with substantial interference. In recent years, attention mechanisms have begun to be applied in image recognition, allowing models to focus more on key and discriminative parts. However, in current facial expression recognition algorithms outside of laboratory settings, most attention mechanisms are implemented using convolutional neural networks, which cannot operate on non-Euclidean input data.

[0004] With the development of facial landmark detection technology, 68 facial landmarks detected from a single two-dimensional facial expression image have been proven to effectively reflect facial expression changes. This facial expression topology composed of landmarks is a heterogeneous data from traditional facial expression images, and can play a complementary role in terms of features to a certain extent. Summary of the Invention

[0005] The main objective of this invention is to provide a method and system for facial expression recognition in real-world environments based on a global self-attention mechanism and geometric deep learning. This method and system improve the recognition rate and robustness of facial expression recognition in real-world environments. Furthermore, extracting expression information from facial topology at different scales can further enhance the feature extraction capability of the network.

[0006] To achieve the above objectives, this invention provides a facial expression recognition method based on a global self-attention mechanism and geometric deep learning, comprising the following steps:

[0007] Step 1: Preprocess the facial expression images of real-world faces. Segment and extract key points from the same image to obtain equally divided image blocks and facial key point topology maps.

[0008] Step 2: Construct an end-to-end trainable neural network model AGT based on a global self-attention mechanism. AGT includes an ITP facial appearance feature extraction network, a GTP facial structure information extraction network, and a fusion decision module.

[0009] Step 3: Using the facial expression image patches and facial keypoint topology obtained in Step 1 as input, train the AGT; and

[0010] Step 4: When identifying the expression category of a new face image, the corresponding facial expression image block and facial key point topology are first obtained through Step 1, and then input into the AGT network model to finally obtain the expression category of the image.

[0011] A further improvement of the present invention is that, in step 1, the image patch is used as the input of the facial appearance feature extraction network ITP, the facial key point topology map is used as the input of the facial structural information extraction network GTP, and the image patch where the key point is located is serialized as the feature of the key point.

[0012] A further improvement of this invention is that ITP is a network model based on a global self-attention mechanism, used to extract facial facial features, while GTP is a multi-scale self-attention model, used to extract facial structural features. ITP and GTP adaptively learn the relationships between the various parts of the input through a self-attention mechanism, thereby identifying the most discriminative regions in the two heterogeneous facial expression data respectively. ITP is composed of one Dropout layer, 12 global self-attention layers, one normalization layer, and one fully connected layer stacked together. GTP is composed of one normalization layer, seven multi-scale self-attention layers, one pooling layer, and one fully connected layer stacked together.

[0013] A further improvement of the present invention is that the multi-scale self-attention layer used in GTP can apply the self-attention mechanism to facial expression topology at different scales, and can divide facial key points into the following regions: eyebrows (1-12), nose (13-21), left eye (22-27), right eye (28-33) and mouth (34-53); self-attention is calculated for each region separately, and then fused with the self-attention calculation results of the whole face facial key points.

[0014] A further improvement of the present invention is the fusion decision module in step 2; the fusion decision module consists of a fully connected layer, a softmax layer and a distillation module, wherein the fully connected layer is used to fuse the features extracted by GTP and ITP, the softmax layer is used to output the expression recognition result, and the distillation module is used to train the AGT network model with a distillation mechanism.

[0015] A further improvement of this invention is that the distillation module in AGT can train the model more effectively when the sample size is limited. The teacher model used for the distillation mechanism is RegNet-16GF based on convolution. The original image is input into the teacher model, and the teacher model prediction result is output. Finally, the cross-entropy loss function is calculated between the teacher prediction model and the AGT distillation result, and the cross-entropy loss function is calculated between the AGT prediction result and the real expression label. The two are added together to form the final loss function result.

[0016] A further improvement of the present invention is that, in step 4, after obtaining the corresponding facial expression image blocks and facial key point topology in step 1, the images are input into the AGT network model to finally obtain the probability value of the image being predicted as each expression. If the expression category corresponding to the highest probability is the same as the true category, it means that the prediction is correct. Finally, the classification accuracy of the expression image is obtained, that is, the ratio of the number of correct predictions to the total number of predictions.

[0017] To achieve the above-mentioned objectives, the present invention also provides a facial expression recognition system based on a global self-attention mechanism and geometric deep learning, for performing the aforementioned method.

[0018] The beneficial effects of this invention are as follows: By applying global self-attention and multi-scale self-attention mechanisms, the relationships between different parts of the input are effectively learned, thereby identifying the most discriminative part of the input and reducing the impact of a large amount of interference in unrestricted environments on the accuracy of facial expression recognition. The global self-attention mechanism is used to extract facial features by uniformly segmenting and serializing the image, and then modeling the relationships between the different parts. The multi-scale self-attention mechanism can extract facial features from facial geometry and topology, possessing the ability to learn across multiple nodes, thus complementing facial expression feature extraction. Fully connected layers and softmax layers can effectively fuse complementary facial features learned from ITP and GTP. The distillation mechanism introduces a convolution-based teacher model, endowing the AGT with the ability to learn inductive bias, improving the model's feature extraction capability and training efficiency. In summary, the AGT proposed in this invention improves the accuracy of facial expression recognition in non-laboratory environments, achieving high accuracy on the RAF-DB and FER-PLUS real-world facial expression image datasets, effectively improving the current state of facial expression recognition in real-world environments. Attached Figure Description

[0019] Figure 1 This is the overall flowchart of the present invention.

[0020] Figure 2 This is a schematic diagram of facial topology and multi-scale segmentation in this invention.

[0021] Figure 3 This is a flowchart of the distillation mechanism in this invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] It should be emphasized that, in describing this invention, various formulas and constraints are distinguished by consistent reference numerals, but it is not excluded that different reference numerals may be used to identify the same formulas and / or constraints. The purpose of this arrangement is to more clearly illustrate the features of this invention.

[0024] like Figure 1 As shown, this invention provides a real-world facial expression recognition method based on a global self-attention mechanism and geometric deep learning, which mainly includes the following steps:

[0025] Step 1: Preprocess the facial expression images of real-world faces, segment and extract key points from the same image to obtain equally divided image blocks and facial key point topology maps;

[0026] Step 2: Construct an end-to-end trainable neural network model AGT based on the global self-attention mechanism. AGT includes the facial appearance feature extraction network ITP, the facial structural information extraction network GTP, and the fusion decision module.

[0027] Step 3: Use the facial expression image patches and facial key point topology obtained in Step 1 as input to train the AGT;

[0028] Step 4: When identifying the expression category of a new face image, the corresponding facial expression image block and facial key point topology are first obtained through Step 1, and then input into the AGT network model to finally obtain the expression category of the image.

[0029] The method of the present invention will be described in detail below with reference to the accompanying drawings.

[0030] Step 1: Preprocess the facial expression images of real-world faces, segmenting and extracting key points from the same image to obtain equally divided image blocks and facial key point topology maps.

[0031] In step 1, image patches serve as input to the ITP (Intuitive Techniques for Facial Appearance) network, and the keypoint topology map serves as input to the GTP (Geometry for Facial Structure) network. The image patches containing keypoint locations are serialized and used as the features of the keypoints. The AGT input image is 224×224 pixels. After equidistant segmentation and serialization, the actual input x∈R196×768 is obtained. To facilitate feature extraction and distillation mechanisms, two equal-dimensional vectors, xclass and xdistill, are concatenated onto the actual input x to aggregate the feature extraction results from each image patch. Furthermore, to enable the network to learn the positional information in the input, a positional encoding vector xpos∈R198×768 is added, resulting in the final input x∈R198×768.

[0032] like Figure 2 As shown, since facial expressions are mainly concentrated around the facial features, this invention removes 15 key points from the facial contour and uses only 53 key points concentrated around the facial features to form the facial expression topology. Based on the position of each key point in the input image, the corresponding sequenced image block is selected as the node's feature. Similarly, to facilitate the implementation of feature extraction and distillation mechanisms, two equal-dimensional nodes, nclass and ndistill, are added to the facial expression topology to aggregate the feature extraction results of each facial expression key point.

[0033] Step 2: Construct an end-to-end trainable neural network model AGT based on the global self-attention mechanism. AGT includes the facial appearance feature extraction network ITP, the facial structural information extraction network GTP, and the fusion decision module.

[0034] Both ITP and GTP are network models based on a global self-attention mechanism. Through this mechanism, they adaptively learn the relationships between different parts of the input, thereby identifying the most discriminative regions in two heterogeneous facial expression datasets. In step 2, ITP is a network model based on a global self-attention mechanism used to extract facial appearance features; GTP is a multi-scale self-attention model used to extract facial structural features. ITP consists of one Dropout layer, twelve global self-attention layers, one normalization layer, and one fully connected layer stacked together; GTP consists of one normalization layer, seven multi-scale self-attention layers, one pooling layer, and one fully connected layer stacked together.

[0035] The multi-scale self-attention layer used in GTP can apply the self-attention mechanism to facial expression topology at different scales. That is, the facial key points are divided into the following regions: eyebrows (1-12), nose (13-21), left eye (22-27), right eye (28-33) and mouth (34-53); self-attention is calculated for each region separately, and then fused with the self-attention calculation results of the whole face facial key points.

[0036] The fusion decision module consists of a fully connected layer, a softmax layer, and a distillation module. The fully connected layer is used to fuse the features extracted by GTP and ITP, the softmax layer is used to output the expression recognition results, and the distillation module is used to train the AGT network model using the distillation mechanism.

[0037] Step 3: Input the paired sequential facial expression image blocks and facial expression key point topology obtained in Step 1 into ITP and GTP respectively for feature extraction. Then, the learned facial appearance features and facial structural information are concatenated and fused. After passing through the softmax layer, the prediction result of AGT is output.

[0038] In step 3, the xdistill and ndistill used for the distillation mechanism are concatenated and fused, and the AGT distillation results are output after passing through a softmax layer. For example... Figure 3 As shown, in this invention, the distillation module in AGT can train the model more effectively with limited samples. The distillation module used in this invention imparts inductive bias to AGT by transferring prior knowledge from a convolution-based teacher model; the teacher model used for the distillation mechanism is a convolution-based RegNet-16GF. The original image is input into the teacher model, and the teacher model's prediction result is output. Finally, a cross-entropy loss function is calculated between the teacher prediction model and the AGT distillation result, and a cross-entropy loss function is calculated between the AGT prediction result and the real expression label; the two are added together to form the final loss function result.

[0039] Step 4: When identifying the expression category of a new face image, the corresponding facial expression image block and facial key point topology are first obtained through Step 1, and then input into the AGT network model to finally obtain the expression category of the image.

[0040] In step 4, the corresponding facial expression image blocks and facial keypoint topology obtained in step 1 are input into the AGT network model to obtain the probability value of the image being predicted as each expression. If the expression category corresponding to the highest probability is the same as the true category, the prediction is correct. Finally, the classification accuracy of the expression image is obtained, which is the ratio of the number of correct predictions to the total number of predictions.

[0041] This invention enables simultaneous self-attention computation and feature fusion classification of two heterogeneous inputs: appearance image and geometric topology. While improving the overall feature extraction capability of the network model, a distillation mechanism is introduced, thereby enhancing the model's recognition accuracy and training efficiency.

[0042] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A facial expression recognition method based on global self-attention mechanism and geometric deep learning, characterized in that, The method comprises the following steps: Step 1, preprocessing the real environment facial expression image, segmenting and extracting key points of the same image respectively to obtain equally divided image blocks and facial key point topology; Step 2, constructing an end-to-end trainable neural network model AGT based on a global self-attention mechanism, wherein the AGT comprises a facial appearance feature extraction network ITP, a facial structural information extraction network GTP and a fusion decision module; Step 3, using the facial expression image blocks and facial key point topology obtained in step 1 as input to train the AGT; And Step 4, when identifying the expression category of a new facial image, first obtaining the corresponding facial expression image blocks and facial key point topology through step 1, and then inputting them into the AGT network model to finally obtain the expression category of the image; In step 1, the image blocks are input into the facial appearance feature extraction network ITP, the facial key point topology is input into the facial structural information extraction network GTP, and the image blocks where the key points are located are serialized as the features of the key points; The ITP is a network model based on a global self-attention mechanism, which is used to extract facial appearance features, and the GTP is a network model based on a multi-scale self-attention mechanism, which is used to extract facial structural features; ITP and GTP adaptively learn the relationship between each part of the input through the self-attention mechanism, and then identify the most discriminative regions in the two heterogeneous facial expression data respectively; the multi-scale self-attention layer used in GTP can apply the self-attention mechanism to different scales of facial expression topology, divide the facial key points into regions, calculate the self-attention for each region separately, and then fuse the calculation results of the full-face facial key point self-attention.

2. The facial expression recognition method based on global self-attention mechanism and geometric deep learning according to claim 1, characterized in that: The ITP is stacked by 1 Dropout layer, 12 self-attention layers, 1 normalization layer and 1 fully connected layer; the GTP is stacked by 1 normalization layer, 7 multi-scale self-attention layers, 1 pooling layer and 1 fully connected layer.

3. The method of claim 2, wherein the method is based on a global self-attention mechanism and geometric deep learning. The facial key points are divided into the following regions: eyebrows (1-12), nose (13-21), left eye (22-27), right eye (28-33) and mouth (34-53).

4. The facial expression recognition method based on global self-attention mechanism and geometric deep learning according to claim 3, characterized in that: The fusion decision module in step 2; the fusion decision module is composed of a fully connected layer, a softmax layer and a distillation module, wherein the fully connected layer is used to fuse the features extracted by GTP and ITP, the softmax layer is used to output the expression recognition result, and the distillation module is used to train the AGT network model in the distillation mechanism.

5. The method of claim 4, wherein the method is based on a global self-attention mechanism and geometric deep learning. The distillation module in AGT can train the model under the condition of limited samples, the teacher model for the distillation mechanism selects RegNet-16GF based on convolution; input the original image into the teacher model to output the prediction result of the teacher model; finally, calculate the cross-entropy loss function of the teacher prediction model and the AGT distillation result, and calculate the cross-entropy loss function of the AGT prediction result and the real expression label, and add them to form the final loss function result.

6. The method of claim 5, wherein the method is based on a global self-attention mechanism and geometric deep learning. In step 4, the corresponding facial expression image block and facial key point topology obtained in step 1 are input into the AGT network model, and finally the probability value of the image prediction for each expression is obtained. If the expression category corresponding to the maximum probability is the same as the true category, it means that the prediction is correct, and finally the classification accuracy of the expression image is obtained, that is, the ratio of the correct prediction number to the total prediction number.

7. A facial expression recognition system based on global self-attention mechanism and geometric deep learning, characterized in that: A computer program element comprising computer readable instructions for performing the method according to any one of claims 1-6 when executed by a computer. A computer program element comprising computer readable instructions for performing the method according to any one of claims 1-6 when executed by a computer.

Images