A facial expression recognition method based on an improved residual neural network
By improving the residual neural network, combining residual feature extraction and the Inception module, and using the Mish activation function and learning rate decay mechanism, the problems of insufficient feature extraction and complex deep learning training in traditional methods are solved, and high-precision facial expression recognition is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIV
- Filing Date
- 2022-12-01
- Publication Date
- 2026-08-04
AI Technical Summary
In existing technologies, the features generated and used by traditional facial expression recognition algorithms are shallow features, which cannot obtain deeper semantic and depth features from the original image. Furthermore, deep learning networks suffer from gradient vanishing and gradient exploding problems during training, resulting in insufficient feature extraction.
An improved residual neural network is adopted, including a residual feature extraction module, an Inception feature extraction module, and a classification module. The Mish activation function and cross-entropy loss function are used. By constructing a 20-layer residual neural network and combining a learning rate decay mechanism, network degradation and overfitting are prevented, thereby improving the richness of feature extraction and recognition accuracy.
It improves the accuracy and robustness of facial expression recognition, prevents network degradation and overfitting, and achieves high-precision expression recognition results, with an accuracy of 96.37% on the CK+ dataset and 93.38% on the KDEF dataset.
Smart Images
Figure CN115937937B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image recognition technology, and in particular relates to a facial expression recognition method based on an improved residual neural network. Background Technology
[0002] With the rapid development of artificial intelligence technology, facial expression recognition has become an important research topic in the field of computer image processing. It has broad application prospects in human-computer interaction, safe driving, and online education. For example, in safe driving, cameras can acquire drivers' facial expression information in real time to determine whether their driving status is good and avoid accidents. In education, students' classroom status can be assessed by recognizing their facial expressions in class, and teaching methods can be adjusted to achieve better teaching results.
[0003] Facial expression recognition mainly consists of three parts: preprocessing, feature extraction, and classification. The accuracy of the algorithm primarily depends on the feature extraction method. Facial expression feature extraction methods are mainly divided into traditional feature extraction methods and deep learning-based methods. Feature extraction plays a decisive role in the accuracy of facial expression recognition. There are many traditional feature extraction methods, such as Local Binary Pattern Transform (LBP), Scale Invariant Feature Transform (SIFT), and Gabor wavelet transform. While traditional facial expression recognition methods can achieve good results, the features generated and used by traditional facial recognition algorithms are shallow features, unable to obtain deeper semantic and depth features from the original image. Furthermore, to achieve good recognition results, these traditional facial recognition algorithms must be combined with manually generated features, which often introduce unexpected human factors and errors during feature extraction and recognition. The emergence of deep learning has effectively solved these problems.
[0004] In recent years, with the development of deep learning, deep convolutional neural networks (DCNNs) have made breakthroughs in fields such as image classification and recognition. While features can be better fitted with increasing network depth, training becomes more complex due to problems like vanishing and exploding gradients. Furthermore, many network models still suffer from insufficient feature extraction. Summary of the Invention
[0005] The purpose of this invention is to provide a facial expression recognition method based on an improved residual neural network to solve the above-mentioned technical problems.
[0006] To address the aforementioned technical problems, the present invention provides a specific technical solution for a facial expression recognition method based on an improved residual neural network, as follows:
[0007] A facial expression recognition method based on an improved residual neural network includes the following steps:
[0008] Step 1: Data preprocessing;
[0009] Step 2: Construct a facial expression recognition model based on an improved residual neural network;
[0010] Step 3: Iteratively train the face recognition model;
[0011] Step 4: Obtain facial expression recognition results based on the trained facial expression recognition model.
[0012] Further, step 1 includes the following steps:
[0013] Step 1.1: Obtain the face image dataset, use the Haar algorithm to identify the face regions in the data, then crop the face parts, adjust the image size to 48*48, and divide it into training and test sets in an 8:2 ratio;
[0014] Step 1.2: The obtained facial expression datasets are the CK+ expression dataset and the KDEF dataset. The CK+ expression dataset includes 593 video sequences from 123 subjects, of which the last frame of 327 video sequences is labeled with expression tags. The last three images from the labeled image sequence of each person are selected, totaling 981 images, containing 7 emotions: anger, contempt, disgust, fear, happiness, sadness, and surprise. The KDEF dataset contains 4900 images, of which 980 frontal facial images are selected as the experimental dataset, containing 7 emotions: anger, neutral, disgust, fear, happiness, sadness, and surprise.
[0015] Furthermore, the model in step 2 includes a residual feature extraction module, an Inception feature extraction module, and a classification module.
[0016] Furthermore, the residual feature extraction module includes one convolutional layer, seven residual modules one and two residual modules two, wherein residual module one includes two convolutional layers and two batch processing layers; residual module two includes three convolutional layers and three batch processing layers.
[0017] The residual feature extraction module uses the Mish activation function, as shown below:
[0018] f(x) = x·tanh(ln(1+e) x ))
[0019] The skip-layer connection uses a feature addition method, as shown in the following formula:
[0020]
[0021] Where 1≤h≤H, 1≤w≤W, 1≤c≤C, and point (h,w,c) represent the pixel values of the coordinates corresponding to the feature map. Indicate feature a, Indicate feature b, This indicates the characteristics after addition.
[0022] Furthermore, the Inception feature extraction module consists of nine Inception modules, and the feature fusion method uses feature concatenation, as shown in the following formula:
[0023]
[0024]
[0025] Where 1≤h≤H, 1≤w≤W, 1≤c≤C, and point (h,w,c) represent the pixel values of the coordinates corresponding to the feature map. Indicate feature a, Indicate feature b, and This represents the features after feature assembly.
[0026] Furthermore, the classification module uses the Softmax activation function and the cross-entropy loss function, as shown below:
[0027]
[0028]
[0029] In the Softmax formula: i represents the number of output nodes; x i is the output value of the i-th node, K represents the label of the output node (from 1 to K), and K is the number of output nodes, i.e., the number of classification categories. The multi-class output values are converted into a probability distribution within the range [0,1] using the Softmax function. In the loss function formula, It is the predicted value generated after the Softmax function, y i It is the actual value.
[0030] Furthermore, step 3 includes the following specific steps:
[0031] The input network data size is 48x48x1, the output size is 7x1, the convolutional kernel size of the convolutional layers is 3x3, the convolutional kernel size of the connection layers is 1, and the dimension is set to 7. The batch size is set to 32, the number of iterations is set to 200, and the initial learning rate is set to 0.9×10. -3 The learning rate decreases as the number of iterations increases, and the decay formula is shown below:
[0032] lry =lr x ×dr
[0033] lr y The learning rate after decay, lr x dr represents the learning rate before decay, and dr represents the decay coefficient.
[0034] The network is trained using a training set, and the hyperparameters are adjusted based on the learning performance on the test set to obtain the optimal model.
[0035] Furthermore, step 4 includes the following specific steps:
[0036] The processed test set is used as input samples, and the samples are input into a trained near-optimal network for facial expression recognition.
[0037] The facial expression recognition method based on an improved residual neural network of the present invention has the following advantages:
[0038] 1. Based on the concept of residuals, this invention designs two residual modules and constructs a 20-layer residual neural network to prevent the occurrence of network degradation problems.
[0039] 2. This invention introduces the Inception module and designs a facial expression recognition model based on an improved residual neural network, which solves the problem of insufficient feature extraction.
[0040] 3. This invention uses the newer Mish activation function to replace the commonly used ReLU activation function, which further improves the robustness of the model and the accuracy of facial expression recognition.
[0041] 4. The network constructed in this invention employs a learning rate decay mechanism to prevent overfitting. Attached Figure Description
[0042] Figure 1 This is a flowchart illustrating the implementation of the method of the present invention;
[0043] Figure 2 This is a network structure diagram of the method of the present invention;
[0044] Figure 3 This is a schematic diagram of the residual module design in the method of the present invention;
[0045] Figure 4 The figure shows the experimental results of the method of the present invention on the CK+ dataset;
[0046] Figure 5 The figure shows the experimental results of the method of this invention on the KDEF dataset. Detailed Implementation
[0047] To better understand the purpose, structure, and function of this invention, the following detailed description of a facial expression recognition method based on an improved residual neural network, in conjunction with the accompanying drawings, is provided.
[0048] like Figure 1 As shown, a facial expression recognition method based on an improved residual neural network according to the present invention includes the following process:
[0049] To obtain a dataset of facial images, the Haar algorithm was used to identify the facial regions in the data to avoid the influence of the image background on the recognition results. Then, the facial parts were cropped, the image size was adjusted to 48*48, and the dataset was divided into training and test sets in an 8:2 ratio.
[0050] The obtained facial expression datasets were the CK+ expression dataset and the KDEF dataset. The CK+ expression dataset included 593 video sequences from 123 subjects, of which 327 video sequences had the last frame labeled with an expression. In the experiment, the last three images from the labeled image sequence of each person were selected, totaling 981 images, containing 7 emotions: anger, contempt, disgust, fear, happiness, sadness, and surprise.
[0051] The KDEF dataset contains 4900 images, of which 980 are frontal images used as the experimental dataset. It includes seven emotions: anger, neutrality, disgust, fear, happiness, sadness, and surprise.
[0052] like Figure 2 As shown, a facial expression recognition model based on an improved residual neural network is constructed based on the residual module and the Inception module designed. It includes a 20-layer residual neural network, mainly consisting of a residual feature extraction module (10 layers), an Inception feature extraction module (9 layers), and a classification module (1 layer).
[0053] like Figure 3 As shown, the residual feature extraction module includes one convolutional layer, seven residual module one modules, and two residual module two modules. Residual module one includes two convolutional layers and two batch processing layers; residual module two includes three convolutional layers and three batch processing layers.
[0054] The residual feature extraction module uses the Mish activation function, as shown below. The non-monotonicity of Mish ensures that smaller negative inputs remain negative outputs, improving expressive power and gradient flow.
[0055] f(x) = x·tanh(ln(1+e) x ))
[0056] The skip-layer connection uses a feature addition method, as shown in the following formula:
[0057]
[0058] Where 1≤h≤H, 1≤w≤W, 1≤c≤C, and point (h,w,c) represent the pixel values of the coordinates corresponding to the feature map. Indicate feature a, Indicate feature b, This indicates the characteristics after addition.
[0059] The Inception feature extraction module consists of nine Inception modules. The feature fusion method used is feature concatenation, as shown in the formula below:
[0060]
[0061]
[0062] Where 1≤h≤H, 1≤w≤W, 1≤c≤C, and point (h,w,c) represent the pixel values of the coordinates corresponding to the feature map. Indicate feature a, Indicate feature b, and This represents the features after feature assembly.
[0063] The classification module consists of one fully connected layer, using the Softmax activation function and the cross-entropy loss function, as shown below:
[0064]
[0065]
[0066] In the Softmax formula: i represents the number of output nodes; x i This is the output value of the i-th node. K represents the label of the output node, from 1 to K. K is the number of output nodes, i.e., the number of classification categories. Multi-class output values can be converted into a probability distribution in the range [0,1] using the Softmax function. In the loss function formula, It is the predicted value generated after the Softmax function, y i It is the actual value.
[0067] The parameter settings for the residual module and the Inception module are shown in Table 1 and Table 2. In Table 1, 1_1 to 1_7 represent residual module 1, and 2_1 and 2_2 represent residual module 2.
[0068] Table 1 Residual Module Parameter Settings
[0069]
[0070] Table 2 Inception Module Parameter Settings
[0071]
[0072]
[0073] The input data size to the network is 48x48x1, and the output size is 7x1. The convolutional kernel size of the convolutional layers is 3x3, the convolutional kernel size of the connection layers is 1, and the dimension is set to 7. The batch size is set to 32, the number of iterations is set to 200, and the initial learning rate is set to 0.9×10⁻⁶. -3 The learning rate decreases as the number of iterations increases, and the decay formula is shown below:
[0074] lr y =lr x ×dr
[0075] lr y The learning rate after decay, lr x dr represents the learning rate before decay, and dr represents the decay coefficient. The decay coefficient settings are shown in Table 3.
[0076] Table 3 Learning rate decay parameters for different iteration numbers
[0077]
[0078]
[0079] The pre-designed network is trained using the training set, and the hyperparameters in the network are adjusted based on the learning performance on the test set to obtain the optimal model.
[0080] The processed test set is used as input samples, and these samples are fed into a pre-trained, near-optimal network for facial expression recognition. Figure 4 , Figure 5 As shown in the results, the facial expression recognition network model based on the improved residual neural network has an accuracy of 96.37% on the CK+ dataset and 93.38% on the KDEF dataset, which is higher than other recognition methods. Table 4 compares the recognition rates of seven methods on the CK+ and KDEF datasets.
[0081] Table 4 shows the accuracy of existing methods and the method of this invention on the CK+ and KDEF datasets.
[0082]
[0083] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.
Claims
1. A facial expression recognition method based on an improved residual neural network, characterized in that, Includes the following steps: Step 1: Data preprocessing; Step 1.1: Obtain the face image dataset, use the Haar algorithm to identify the face regions in the data, then crop the face parts, adjust the image size to 48*48, and divide it into training and test sets in an 8:2 ratio; Step 1.2: The obtained facial expression datasets are the CK+ expression dataset and the KDEF dataset. The CK+ expression dataset includes 593 video sequences from 123 subjects, of which the last frame of 327 video sequences is labeled with expression tags. The last three images from the labeled image sequence of each person are selected, totaling 981 images, containing 7 emotions: anger, contempt, disgust, fear, happiness, sadness, and surprise. The KDEF dataset contains 4900 images, of which 980 frontal facial images are selected as the experimental dataset, containing 7 emotions: anger, neutral, disgust, fear, happiness, sadness, and surprise. Step 2: Construct a facial expression recognition model based on an improved residual neural network; the model includes a residual feature extraction module, an Inception feature extraction module, and a classification module; the residual feature extraction module includes 1 convolutional layer, 7 residual module 1s, and 2 residual module 2s, wherein residual module 1 includes 2 convolutional layers and 2 batch processing layers; residual module 2 includes 3 convolutional layers and 3 batch processing layers; The residual feature extraction module uses the Mish activation function, as shown below: The skip-layer connection uses a feature addition method, as shown in the following formula: in , , , point This represents the pixel value corresponding to the coordinates of the feature map. Indicate feature a, Indicate feature b, Indicates the characteristics after addition; The Inception feature extraction module consists of 9 Inception modules, and the feature fusion uses a feature concatenation method, as shown in the following formula: in , , , point This represents the pixel value corresponding to the coordinates of the feature map. Indicate feature a, Indicate feature b, This represents the features after feature aggregation; The classification module uses the Softmax activation function and the cross-entropy loss function, as shown below: In the Softmax formula: Indicates the number of output nodes; It is the first The output value of each node, The label of the output node, i.e., from 1 to... , This refers to the number of output nodes, i.e., the number of classification categories. The multi-class output values are converted into a probability distribution within the range [0,1] using the Softmax function. In the loss function formula... These are the predicted values generated after the Softmax function. It is the actual value; Step 3: Iteratively train the face recognition model; Step 3 includes the following specific steps: The input network data size is 48x48x1, the output size is 7x1, the convolutional kernel size of the convolutional layers is 3x3, the convolutional kernel size of the connection layers is 1, and the dimension is set to 7. The batch size is set to 32, the number of iterations is set to 200, and the initial learning rate is set to... The learning rate decreases as the number of iterations increases, and the decay formula is shown below: in This represents the decayed learning rate. This represents the learning rate before decay. Indicates the attenuation coefficient; The network is trained using a training set, and the hyperparameters are adjusted based on the learning performance on the test set to obtain the optimal model. Step 4: Obtain facial expression recognition results based on the trained facial expression recognition model.
2. The facial expression recognition method based on an improved residual neural network according to claim 1, characterized in that, Step 4 includes the following specific steps: The processed test set is used as input samples, and the samples are input into a trained near-optimal network for facial expression recognition.