A facial expression recognition method based on facial key points and deep neural network
By combining facial key points and deep neural networks with the CGNet model and utilizing spatial and channel attention mechanisms, the real-time performance and accuracy issues of facial expression recognition in existing technologies have been resolved, achieving more efficient micro-expression recognition.
Patent Information
- Application Number
- CN202210318271.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-29
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-03-29
AI Technical Summary
Existing facial expression recognition methods have shortcomings in real-time performance and extraction of facial key point information, making it difficult for models to quickly recognize micro-expressions. Furthermore, the lack of preprocessing of facial images affects the recognition accuracy.
We employ the CGNet model based on facial key points and deep neural networks, combining image features and key point features. We extract features through spatial attention and channel attention mechanisms, construct a classification and recognition network with convolutional layers and graph convolutional layers, and fuse image and location features to improve recognition accuracy.
It improves the accuracy and speed of facial expression recognition, meets the real-time requirements of practical application scenarios, and enhances the ability to accurately recognize micro-expressions.
Smart Images

Figure CN114663953B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a facial expression recognition method based on facial key points and a deep neural network, and is applicable to the technical field of facial expression recognition in computer vision. Background Art
[0002] Intelligent recognition of facial expressions has always been an important scientific research direction and a hot topic. Facial expression analysis has a wide range of application scenarios, has a significant effect on improving the quality of human life, and is of great research value. Specific application areas include but are not limited to: in public areas of society, using video analysis technology to achieve multi-target tracking and expression analysis, timely identify potential dangers in public environments, and strengthen public safety management and control; identify and analyze the facial expressions of motor vehicle drivers to determine whether the driver is fatigued or drunk driving, and reduce the incidence of traffic accidents; with the assistance of high-definition high-speed camera equipment, analyze the micro-expressions of suspects and provide effective assistance to the police in handling cases; provide life assistance for the disabled to help them understand the emotional state of others and promote the convenience of communication.
[0003] With the development of technology, people are increasingly trying to use machine vision and image processing technology to achieve automated facial expression recognition. In recent years, deep learning algorithms have been widely used in various fields such as natural language processing, data mining, and image processing due to their powerful learning and adaptability. Some methods based on convolutional neural networks have also been introduced into the field of facial expression analysis, including AlexNet, ResNet-18, LeNet, etc. Thanks to the local connection and weight sharing mechanism of convolutional neural networks, the parameters and calculation amount of facial image processing are greatly reduced, and the accuracy of expression recognition is significantly improved compared with traditional methods. Although the facial expression analysis method based on deep learning has achieved good results, there are still some problems with the current method, including: blindly stacking convolutional layers will make it difficult for the model to complete the extraction and analysis of the face in a very short time, and cannot meet the real-time requirements in actual use scenarios; lack of preprocessing of facial images, usually after extracting the position of the face in the whole picture, it is simply cut and used as the input of the deep neural network. The network needs to analyze the factors affecting the expression from the whole image by itself, but ignores the effective extraction of facial key point information, which limits the accurate recognition of micro-expressions.
[0004] Therefore, it is of great significance to carry out algorithm research based on facial key point features. Summary of the invention
[0005] The present invention proposes a facial expression recognition method based on facial key points and deep neural network, which is named CGNet. CGNet in the present invention uses image features and key point feature information at the same time to comprehensively and accurately recognize expressions. At the same time, the spatial attention mechanism and channel attention mechanism are used to improve the efficiency of feature extraction. The trained CGNet network model has good recognition performance. Compared with the current algorithm FAN (Frame Attention Network), CGNet has a higher recognition rate, which shows that the present invention effectively improves the accuracy of facial expression recognition.
[0006] In order to achieve the above object, the present invention provides the following technical solutions:
[0007] A method for facial expression recognition based on facial key points and deep neural network, the method comprising the following steps:
[0008] Step S1: construct a training set, a validation set and a test set based on a public facial expression data set; extract several key point feature information of the face through preprocessing, and divide the eye, nose and mouth areas to obtain multi-channel image data and key point coordinate data respectively;
[0009] Step S2: Construct a classification recognition network CGNet based on convolutional layers and graph convolutional layers; use the constructed training set to supervise the CGNet network model until CGNet converges to the optimal performance; during the training process, use the validation set to reflect the training process;
[0010] Step S3: Test the converged CGNet network model on the constructed test set and evaluate the network performance of CGNet based on the test results.
[0011] Preferably, each piece of data in the facial expression data set in step S1 is given in the form of a data pair, including an RGB image to be classified as input data and a marked expression category as a true value.
[0012] Preferably, the preprocessing in step S1 includes: first extracting the face and the corresponding N key point positions in the input RGB face image to be classified to obtain N×2 coordinate information; then dividing the eye, nose and mouth areas of the face according to the key point positions, and changing the image pixel size of the RGB face image to be classified and the three sub-images obtained after the division to 224×224 by bilinear interpolation, and performing data enhancement processing, the enhancement means include: random rotation, random cutting, random modification of the saturation and hue of the RGB image, and finally standardizing the RGB image.
[0013] Preferably, the classification recognition network CGNet comprises:
[0014] a) RGB image feature extraction module: Its input is the face image and the three sub-images of the eye, nose and mouth regions obtained after the face image is divided. After obtaining the image, facial features are first extracted based on the ResNet18 network, and then the feature information obtained by ResNet18 is extracted in layers. The spatial attention mechanism is used to enhance the model's perception of local information, and finally 4 image features are output;
[0015] b) Key point position feature extraction module: Its input is N×2 coordinate information. After obtaining the coordinate information, it first performs feature extraction based on the graph convolution layer. Graph convolution represents a node by calculating the aggregation of the adjacent nodes of the current node, so that some nodes with unclear features can be filtered out, and the importance information of different key points can be obtained, and finally 1-way position feature is output;
[0016] c) Feature fusion module: The feature fusion module adopts a channel attention mechanism. By inputting the above-mentioned 4 image features and 1 position feature, the channel attention is used to calculate the weight of each feature, and the weighted sum is performed with the corresponding feature value, thereby producing the effect of fusing the image feature with the position feature.
[0017] Preferably, the graph convolution layer in the key point position feature extraction module is specifically: the graph convolution layer in the neural network is represented by the following nonlinear function:
[0018] H l+1 =f(H l ,A) (1)
[0019] For the above formula, H 0 =X is the input of the first layer, X∈R N*D , N is the number of nodes in the graph, D is the dimension of each node feature vector, and A is the adjacency matrix; each layer of graph convolution is expressed as:
[0020]
[0021] Where σ(·) is the ReLU activation function, W l is the weight parameter matrix of the lth layer.
[0022] The beneficial effects of the present invention are:
[0023] The CGNet network model proposed in the present invention uses the fusion of face image information and face key point features to perform facial expression recognition, strengthens the representation of facial expression features, improves the representation ability, and thus improves the detection progress. The gain is specifically reflected in: in terms of average classification accuracy, the present invention can obtain a higher accuracy rate than the existing method. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 This is a flowchart of the facial expression recognition method based on facial key points and deep neural network in Example 1.
[0025] Figure 2 This is a flow chart of expression classification network training and prediction in Example 1.
[0026] Figure 3 This is a structural diagram of the expression classification network model in Example 1.
[0027] Figure 4 This is an illustration of the ResNet18 network in Example 1.
[0028] Figure 5 This is a diagram showing the effect of facial expression recognition in Example 1.
[0029] Figure 6 It is the recognition accuracy value in Example 1. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0031] Example 1
[0032] See also Figure 1-Figure 6 ,This implementation provides a facial expression recognition method based on facial key points and deep neural network,the method trains an expression classification model through a deep learning method, and the model can output the predicted expression type based on the input human face RGB image.
[0033] Specifically, see Figure 1 , this method specifically includes:
[0034] Step S1: Obtain a facial expression data set, extract 68 key point feature information of the face through preprocessing, and divide the eye, nose, and mouth areas to obtain multi-channel image data and key point coordinate data respectively;
[0035] More specifically, step S1 includes: obtaining a data set for facial expression recognition. Each piece of data in the data set is given in the form of a data pair, including an RGB image to be classified as input data and an annotated expression category as a true value, wherein the expression category is represented by a number from 0 to 6 (0-anger, 1-contempt, 2-nausea, 3-fear, 4-happiness, 5-sadness, 6-surprise).
[0036] Each piece of data in the dataset needs to be preprocessed before being input into the model. First, the face in the input image and the corresponding 68 key point positions are extracted through the face detection module to obtain 68×2 coordinate information. Then, the eye, nose, and mouth areas of the face are divided according to the key point positions, and the image pixel size of the original image and the three sub-images is changed to 224×224 through bilinear interpolation. Data enhancement processing is performed, and the enhancement methods include: random rotation, random cutting, random modification of RGB image saturation, hue and other attributes, and finally the RGB image is standardized.
[0037] Step S2: training the expression classification network, extracting features of the image data and key point feature information using the convolution layer and graph convolution layer respectively, and outputting the classification results;
[0038] More specifically, in this embodiment, step S2 includes: splitting the data set into a training set containing 654 images and a test set containing 327 images, using the training set to train the expression classification network based on the Pytorch framework, fixing the model network parameters when each network parameter in the network model reaches the convergence standard, and obtaining the trained expression classification network, and finally using the trained expression network to predict the test set to obtain the expression classification result.
[0039] It should be noted that the expression classification method provided in this embodiment is not limited to the Pytorch framework. As long as the data set X can be trained and the loss function converges after several iterations (on the order of times) during the training process and finally the facial expressions can be accurately classified according to the RGB images, it can be used.
[0040] It should be noted that, in this example, the expression classification network described in step S2 includes the following main components:
[0041] a) RGB image feature extraction module: The input of the RGB image feature extraction module is a face image and three sub-images of the eye, nose and mouth regions obtained after the face image is divided. After obtaining the image, facial features are first extracted based on the ResNet18 network, and then the feature information obtained by ResNet18 is layered. The model's perception of local information is enhanced through the spatial attention mechanism, and finally 4 image features are output.
[0042] b) Key point position feature extraction module: The input of the key point position feature extraction module is the 68×2 coordinate information described in step S1. After obtaining the coordinate information, feature extraction is first performed based on the graph convolution layer. The graph convolution represents a node by calculating the aggregation of the adjacent nodes of the current node, thereby filtering out some nodes with unclear features, obtaining the importance information of different key points, and finally outputting 1 position feature.
[0043] c) Feature fusion module: The feature fusion module adopts a channel attention mechanism. By inputting the above five features, it distinguishes the importance of each feature under the current expression and performs weighted summation with the corresponding feature value, thereby producing the effect of fusing image features with position features.
[0044] Step S3: Calculate the average classification accuracy based on the classification results and the true labels;
[0045] More specifically, in this embodiment, the classification accuracy is calculated based on the number of correctly classified samples and the total number of samples in the test set using the prediction result described in step S2 and the true value described in step S1.
[0046] It should be noted that the indicators for measuring the effect of expression recognition are not limited to the classification accuracy. Any indicator that can show the difference between the obtained expression classification results and the real expression labels can be used as a measurement indicator.
[0047] like Figure 3 As shown in the figure, the model is the overall structure diagram of the expression classification network, which mainly includes three modules: RGB image feature extraction module, key point position feature extraction module, and feature fusion module.
[0048] The input of the RGB image feature extraction module is the face image and three sub-images of the eye, nose and mouth areas obtained after the face image is divided. After obtaining the image, facial features are first extracted based on the ResNet18 network. After the image is input into the network, it is first downsampled through layer1 of ResNet, and then enters layer2-layer5 for multi-scale feature extraction. Then, the feature information of layer2-layer5 is layered and taken out, and sent to the spatial attention mechanism (Spatial Attention) to enhance the model's perception of local information, and finally output 4 image features.
[0049] The input of the key point position feature extraction module is the 68×2 coordinate information described in step S1. After obtaining the coordinate information, feature extraction is first performed based on the graph convolutional network. The graph convolutional network consists of 4 graph convolutional layers (GCN). Each graph convolutional layer represents a node by calculating the aggregation of the adjacent nodes of the current node, so that some nodes with unclear features can be filtered out, and the importance information of different key points can be obtained, and finally 1 position feature is output.
[0050] The feature fusion module adopts the channel attention mechanism. By inputting the above five features, it distinguishes the importance of each feature under the current expression and performs weighted summation with the corresponding feature value, thereby producing the effect of fusing image features with position features.
[0051] like Figure 5 As shown in FIG. 1 , it is a partial effect diagram of facial expression recognition using the expression classification network. Obviously, from a qualitative point of view, compared with the comparison algorithm, the classification result output by the method proposed in the present invention is better and more accurate. Figure 6 As shown, from a quantitative point of view, the classification accuracy of the method proposed in the present invention is higher than that of other algorithms, so the performance is better.
[0052] The matters not described in detail in the present invention are all known technologies to those skilled in the art.
[0053] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A facial expression recognition method based on facial key points and deep neural network, It is characterized in that The method comprises the following steps: Step S1: construct a training set, a validation set and a test set based on a public facial expression data set; extract several key point feature information of the face through preprocessing, and divide the eye, nose and mouth areas to obtain multi-channel image data and key point coordinate data respectively; Step S2: Construct a classification recognition network CGNet based on convolutional layers and graph convolutional layers; use the constructed training set to supervise the CGNet network model until CGNet converges to the optimal performance; during the training process, use the validation set to reflect the training process; Step S3: Test the converged CGNet network model on the constructed test set, and evaluate the network performance of CGNet based on the test results; The classification recognition network CGNet includes: a) RGB image feature extraction module: Its input is the face image and the three sub-images of the eye, nose and mouth regions obtained after the face image is divided. After obtaining the image, facial features are first extracted based on the ResNet18 network, and then the feature information obtained by ResNet18 is extracted in layers. The spatial attention mechanism is used to enhance the model's perception of local information, and finally 4 image features are output; b) Key point position feature extraction module: Its input is N×2 coordinate information. After obtaining the coordinate information, it first extracts features based on the graph convolution layer. The graph convolution represents a node by calculating the aggregation of the neighboring nodes of the current node, filtering out some nodes with unclear features, obtaining the importance information of different key points, and finally outputting 1-way position features; c) Feature fusion module: The feature fusion module adopts a channel attention mechanism, by inputting the above-mentioned 4-way image features and 1-way position features, using channel attention to calculate the weight of each feature, and performing weighted summation with the corresponding feature value, thereby producing an effect of fusing image features with position features; The graph convolution layer in the key point position feature extraction module is specifically: the graph convolution layer in the neural network is represented by the following nonlinear function: H l+1 =f(H l ,A) (1) For the above formula, H 0 =X is the input of the first layer, X∈R N*D , N is the number of nodes in the graph, D is the dimension of each node feature vector, and A is the adjacency matrix; each layer of graph convolution is expressed as: Where σ(·) is the ReLU activation function, W l is the weight parameter matrix of the lth layer.
2. According to claim 1, a facial expression recognition method based on facial key points and deep neural network, It is characterized in that Each piece of data in the facial expression data set in step S1 is given in the form of a data pair, including an RGB image to be classified as input data and a marked expression category as a true value.
3. According to claim 1, a facial expression recognition method based on facial key points and deep neural network, It is characterized in that The preprocessing in step S1 includes: firstly extracting the human face and the corresponding N key point positions in the input RGB human face image to be classified to obtain N×2 coordinate information; then dividing the eye, nose and mouth areas of the human face according to the key point positions, and changing the image pixel size of the RGB human face image to be classified and the three sub-images obtained after the division to 224×224 by bilinear interpolation, and performing data enhancement processing, wherein the enhancement means include: random rotation, random cutting, random modification of the saturation and hue of the RGB image, and finally standardizing the RGB image.
