Face expression recognition method based on inceptionv3 network

Abstract

The application discloses a face expression recognition method based on an InceptionV3 network, relates to the technical field of emotion recognition, and comprises the following steps: collecting a face image, improving based on the InceptionV3 network to obtain a face expression recognition network model, wherein the face expression recognition network model comprises two convolution layers, two Inception modules with different structures, two pooling layers and two full connection layers combined with Dropout; inputting the face image into the recognition network model to obtain a face feature classification graph; the InceptionV3 network structure is simplified, the Dropout packet loss rate method is adopted to randomly disconnect the connection between neurons, the overfitting phenomenon in the training process is avoided, the double full connection structure is adopted for classification output, the main features are retained while the training duration is effectively reduced, and therefore the recognition precision of the recognition model is improved.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition technology, specifically to a facial expression recognition method based on the InceptionV3 network. Background Technology

[0002] Emotion is a state encompassing a person's thoughts, feelings, and behaviors, reflecting the quality of their psychological state. Facial expressions often directly reflect a person's most intuitive and authentic state. As early as ancient Chinese society, human emotions were categorized into seven basic forms: joy, anger, sorrow, happiness, love, hate, and fear. Psychologists Ekman and Friesen also proposed six basic human emotions in 1971: anger, sadness, happiness, disgust, and fear. Currently, automatic facial emotion recognition using image processing and machine learning methods has become an important research area in computer vision.

[0003] Research on facial emotion recognition mainly falls into two categories: traditional methods and deep learning-based approaches. Traditional methods extract emotion-related features from faces using either a holistic or local approach, then classify these features using methods such as Bayesian networks. Deep learning methods, such as convolutional neural networks (CNNs) and generative adversarial networks (GANs), have become increasingly popular for facial emotion recognition in recent years. Chen et al. proposed a deep neural network that integrates temporal and spatial features to analyze and understand facial expressions in video sequences, improving emotion recognition performance. They replaced the shallow feature extraction module with a continuously stacked small convolutional kernel to form an acceleration module. Wang et al. proposed a convolutional neural network based on the Inception concept, using 1×1 kernels to effectively classify facial emotions on small datasets, achieving excellent results on the CK+ and JAFFE datasets. Zhang et al. improved the GoogLeNet network by optimizing pruning, accelerating training speed while increasing the accuracy of facial expression recognition, achieving over 80% accuracy on the JAFFE, CK+, and Cohn-Kanada datasets. Cheng Weiyue et al. proposed a deep convolutional neural network algorithm that integrates global and local features. Two improved convolutional neural network branches extract global and local features respectively, and the weighted fused features are used for classification. Zhang Wei et al. proposed a facial expression recognition network based on an attention mechanism, combining multi-scale feature extraction with spatial attention. Channel- and spatially weighted feature maps are input into subsequent networks for further feature extraction and classification, improving recognition accuracy. Zhang et al. constructed an end-to-end deep model based on generative adversarial networks (GANs) for simultaneous facial expression recognition and facial image synthesis. Expanding the dataset improved the model's generalization ability and recognition accuracy. Delphine et al. proposed an expression recognition algorithm for partially occluded faces, using a jump-connected autoencoder method based on facial texture or geometry to reconstruct the occluded portion of the face in the optical flow domain.

[0004] The main goal of facial expression recognition is to improve classification accuracy. However, higher accuracy usually corresponds to deeper and wider network structures, which leads to a sharp increase in the number of parameters and computation, and results in excessively long training time. Summary of the Invention

[0005] To address the shortcomings of existing facial expression recognition networks, which have excessively deep and wide structures, resulting in a sharp increase in the number of parameters and computational load, this invention provides a facial expression recognition method based on the InceptionV3 network. By improving upon the InceptionV3 network, a facial expression recognition network model is obtained to recognize facial expressions, thereby solving the problem of the excessively deep and wide structures in existing facial expression recognition technologies, which lead to a sharp increase in the number of parameters and computational load.

[0006] A facial expression recognition method based on the InceptionV3 network includes the following steps:

[0007] Collect facial image data;

[0008] An improved facial expression recognition network model is obtained based on the InceptionV3 network. The facial expression recognition network model includes two convolutional layers, two Inception modules with different structures, namely 3×Inception structure and 5×Inception structure, two pooling layers, and two fully connected layers combined with Dropout.

[0009] Facial image data is input into the facial expression recognition network model. Features are extracted from the facial images through convolutional layers and pooling layers to obtain feature maps. The Dropout packet loss rate function is used to randomly disconnect the connections between neurons in the facial expression recognition network model at a packet loss rate of 25% after the pooling layer operation.

[0010] The feature map is extracted sequentially using a 3×Inception structure and an asymmetric 5×Inception structure.

[0011] The extracted features are classified using two fully connected layers with a 25% packet loss rate, and a face feature classification map is output.

[0012] Based on a facial emotion database, the Softmax function is used to classify facial features and expressions.

[0013] Furthermore, it also includes preprocessing the acquired facial images, which includes the following steps:

[0014] The Haar feature extraction classifier is used to extract the face region;

[0015] The extracted face images were filtered and enhanced using FFA-NET.

[0016] After filtering, the face images are cropped and cleaned.

[0017] Gaussian noise is used to augment the processed image data.

[0018] Furthermore, the two convolutional layers respectively select a 3×3 convolutional kernel with 32 channels and a 3×3 convolutional kernel with 64 channels for convolution operations, and set the padding value of both convolutional kernels to 2 and the stride value to 1.

[0019] Furthermore, when performing convolution operations in the convolutional layer, the ReLU activation function is used to increase the nonlinear features of the facial expression recognition network model.

[0020] Furthermore, it also includes using the BatchNormalization regularization method after the convolutional layer operation is completed, and extracting facial features by executing multiple convolutional kernels and pooling windows in parallel.

[0021] Furthermore, the 3×Inception structure uses a 1×1 convolution kernel and two consecutive 3×3 convolution kernels for convolution operations.

[0022] Furthermore, the 5×Inception structure uses 1×3 and 3×1 asymmetric convolution kernels for convolution operations.

[0023] Furthermore, the facial emotion database includes expressions such as anger, annoyance, fear, happiness, neutrality, sadness, and surprise.

[0024] Furthermore, the facial expression recognition model is trained using a dynamic learning rate decay method based on the loss value. The training process includes: decaying the learning rate by judging the decay of the loss value every five iterations; when the loss value is on an upward trend, the learning rate is decayed to 5% of the original learning rate, otherwise the original learning rate is maintained to train the model; when the learning rate decays to 0, the learning rate decay is stopped.

[0025] Furthermore, it also includes processing redundant information in the facial expression recognition network model using 2×2 max pooling before performing feature classification on the extracted features through two fully connected layers with a 25% packet loss rate.

[0026] This invention provides a facial expression recognition method based on the InceptionV3 network, which has the following characteristics:

[0027] Beneficial effects:

[0028] This invention proposes a facial emotion recognition method based on a simplified InceptionV3 network structure. The InceptionV3 network structure is simplified, resulting in fewer layers and a smaller parameter scale. The Dropout method is used to randomly disconnect connections between neurons, avoiding overfitting during training and reducing the network's parameter scale. Simultaneously, a nonlinear double fully connected structure is employed for classification output, preserving key features while effectively reducing training time. This improves the recognition accuracy of the facial expression recognition network model, leading to higher classification accuracy in facial expression recognition. Attached Figure Description

[0029] Figure 1 This is a diagram of the InceptionV3 network structure in an embodiment of the present invention;

[0030] Figure 2This is a structural diagram of the facial expression recognition network model in an embodiment of the present invention;

[0031] Figure 3 This is a structural diagram of the 3×Inception in an embodiment of the present invention;

[0032] Figure 4 This is a structural diagram of the 5×Inception in an embodiment of the present invention;

[0033] Figure 5 This is a sample image of a portion of the dataset in an embodiment of the present invention;

[0034] Figure 6 This is a graph showing the loss value and learning rate of the validation set in an embodiment of the present invention. Detailed Implementation

[0035] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0036] The core idea of ​​the Inception module is to combine different convolutional layers in parallel. The matrices processed by different convolutional layers are then concatenated along the depth dimension to form a deeper matrix. The Inception module can be repeatedly stacked to form larger networks, efficiently expanding the network's depth and width, improving the accuracy of deep learning networks while preventing overfitting. The advantage of the Inception module is that it first reduces the dimensionality of large matrices while aggregating visual information at different scales, facilitating feature extraction from different scales. Based on this, the InceptionV1, V2, and V3 network structures were subsequently proposed, with the InceptionV3 network structure as follows: Figure 1 As shown.

[0037] Table 1 shows the network structure parameters of InceptionV3. This network first extracts initial image features using three 3×3 convolutional layers, then uses 3×3 pooling layers to extract main features and reduce image size. Next, it uses 3×Inception, 5×Inception, and 2×Inception structures to further extract image features. The extracted images are then pooled into a 1×1×2048 structure using an 8×8 pooling layer, and then classified using a linear activation function and a softmax function. This network structure extracts facial features through multi-layer convolutional cascaded and parallel structures, thereby achieving emotion recognition in facial images. This network has a large number of layers and parameters, requiring a significant amount of training time. On a 299×299 face image, the training time for a single image is approximately 250ms, with a recognition accuracy of 76.6%.

[0038] Table 1 InceptionV3 Network Structure Parameters

[0039]

[0040] Compared to traditional convolutional neural networks, InceptionV3 improves the utilization of internal computational resources, enabling the network to increase in width and depth without changing the computational budget, thereby further improving the network's accuracy. However, increasing the network width and depth brings with it problems such as increasing network complexity, more parameters, and higher hardware requirements. In addition, overfitting due to the small sample size is also inevitable.

[0041] This invention proposes a lightweight facial expression recognition network based on the InceptionV3 network, classifying and recognizing seven basic emotions: "anger," "annoyance," "fear," "happiness," "neutrality," "sadness," and "surprise." The main differences between the classification network used in this invention and the InceptionV3 network are fourfold: First, the InceptionV3 network structure is simplified, with fewer layers and a smaller parameter scale; second, the Dropout method is used to randomly disconnect connections between neurons, further avoiding overfitting during training and reducing the network's parameter scale; third, a nonlinear double fully connected structure is used for classification output, preserving key features while effectively reducing training time; fourth, compared to the cosine annealing algorithm and the method of dynamically decaying the learning rate based on the validation set loss value, the latter is chosen to train the algorithm of this invention, thereby improving model accuracy and controlling the number of training iterations. The facial expression recognition network structure proposed in this invention is as follows: Figure 2 As shown.

[0042] The network model of this invention consists of two convolutional layers, two Inception modules with different structures, namely 3×Inception and 5×Inception, two pooling layers, and two fully connected layers combined with Dropout. Before using the 3×Inception structure, compared to the InceptionV3 network, the number of convolutional layers in this invention is reduced from six to two, and one 2×Inception module is eliminated. In terms of parameter count, the total number of parameters in this invention is 70% of that in the InceptionV3 model. The network structure parameters are shown in Table 2, with bolded annotations indicating parameter modifications.

[0043] Table 2 Lightweight InceptionV3 Network Structure Parameters

[0044]

[0045] The specific process for facial emotion recognition using this model is as follows:

[0046] Input a 48×48 grayscale image of a face.

[0047] Step 1: Perform initial feature extraction using dual-channel 3×3 convs (different channels).

[0048] Step 2: 2×2 MaxPooling and 25% packet loss rate, while retaining the main features and reducing parameters.

[0049] Step 3: 3×Inception further extracts facial features.

[0050] Step 4: 5×Inception extracts facial expression and emotion features again.

[0051] Step 5: 2×2 MaxPooling preserves key features.

[0052] Step 6: Fully connected layers Fc1 and Fc2, and classify using a 25% packet loss rate.

[0053] Output the classification results using Softmax.

[0054] In Step 1, a 3×3 convolutional kernel with 32 channels is selected for convolution, with a padding value of 2 and a stride of 1, to initially extract facial image features. The extracted facial feature image is then convolved with a 3×3 convolutional kernel with 64 channels, again with a padding value of 2 and a stride of 1. During the convolution operation, the ReLU activation function is used, primarily to add non-linear features. Batch Normalization is added after the convolution operation to prevent gradient vanishing. In Step 2, 2×2 max pooling is used to remove redundant features and extract the maximum features. After pooling, to accelerate training, the Dropout function is used, temporarily discarding some nodes from the network with a 25% probability after the pooling layer operation. Through this stage of feature extraction, some unimportant features have been removed, facilitating further feature extraction in the next stage, the Inception module.

[0055] In Step 3, a 3×Inception structure is used to further extract features from the image. The 1×1 convolutional kernel is primarily used for dimensionality reduction and adding non-linear features. While using a 5×5 convolutional kernel would result in a larger receptive field, it would also come with a significant increase in parameters. Experiments show that replacing the 5×5 kernel with two consecutive 3×3 kernels does not affect the receptive field size and reduces the number of parameters. After each convolutional kernel, we use BatchNormalization to accelerate training, extracting facial features by executing multiple convolutional kernels and pooling windows in parallel. The 3×Inception structure used in the experiments is described in [link to experiment]. Figure 3 .

[0056] After Step 3 processing, the feature map size is 12×12. When the feature map size is between 12 and 20, Step 4 uses another asymmetric 5×Inception structure. Figure 4 Further feature extraction is performed on the image, where n is 3. This module also uses 1×1 convolution kernels for dimensionality reduction and increasing nonlinearity, but unlike 1×1 kernels, it uses 1×3 and 3×1 asymmetric convolution kernels for convolution operations. Using asymmetric convolution kernels not only reduces parameters, speeds up computation, and reduces overfitting, but also increases the feature diversity of the model. When the feature map is of medium size (12–20), asymmetric convolution has a better effect on feature extraction.

[0057] In Step 5, 2×2 max pooling is used to process redundant information in the model. In Step 6, a dual fully connected structure combined with a 25% packet loss rate is used for feature classification. Finally, Softmax is used to complete the recognition and classification of seven types of facial emotions.

[0058] This invention's experiments are based on the TensorFlow library in Python 3, and the network is built using the Keras framework. The AutoDL server is used, and the hardware platform uses a 7-core Intel(R) Xeon(R) CPU E5-2680v4 @ 2.40GHz and an RTX3060 GPU.

[0059] RAF-DB is a large-scale facial expression database containing 29,762 real-world images, divided into two subsets: a single-label subset and a two-label subset. The single-label subset includes seven basic emotions, while the two-label subset includes twelve complex emotions. This invention uses a calibrated set from this database, comprising seven basic emotions: anger, annoyance, fear, happiness, sadness, surprise, and neutrality. This set consists of 15,339 color images (100×100 pixels each), with 12,271 images in the training set and 3,068 images in the test set.

[0060] The JAFFE database consists of 213 grayscale images (256×256 pixels) featuring six basic facial expressions and one neutral expression from ten Japanese female models. Each image has an average semantic rating of the six expressions from 60 Japanese viewers. While the database has a relatively small sample size, the data was collected in a laboratory setting with consistent lighting and brightness, making it suitable for training simple convolutional neural networks.

[0061] The CK+ dataset, released in 2010, is an extension of the CK dataset. Compared to the CK dataset, the CK+ dataset increases the number of subjects and sequences. The target expression for each sequence was FACS encoded, and the emotion labels were modified and validated. The CK+ dataset includes 981 grayscale images of 48×48 pixels, with a larger sample size than the JAFFE dataset, and contains seven label classes: happy, bored, angry, fearful, neutral, and sad.

[0062] Samples from RAF-DB, JAFFE, and CK+ datasets are shown below. Figure 5 As shown, the RAF-DB dataset is large in scale, but due to factors such as complex collection environments and multiple collection angles, it can affect model performance to some extent. Therefore, this invention combines the RAF-DB dataset with the JAFFE and CK+ datasets to construct samples from multiple angles and environments, effectively avoiding the problem of uneven distribution of the RAF-DB dataset, and also providing some assistance in model tuning.

[0063] Data preprocessing mainly consists of four steps. The first step uses a Haar feature extraction classifier to extract face regions. The second step uses FFA-NET to filter and enhance the images. The third step further ensures the quality of the dataset through image cropping and data cleaning. The fourth step uses Gaussian noise to balance the dataset.

[0064] Haar Feature Extraction Classifier: Since the face image size in the JAFFE dataset is 256×256, which is inconsistent with the size of the network input image, and the dataset images contain factors that affect facial feature extraction, such as hair and background, the Haar feature detection classifier is used in the data preprocessing stage to extract the face region. Then, the extracted image is cropped to 48×48, so that only the face region is retained in the images in the JAFFE dataset, and the face image size is consistent with the size of the network input image.

[0065] FFA-NET Image Enhancement: After face region extraction, some data samples suffer from insufficient clarity. Therefore, this invention employs the Feature Fusion Attention Network (FFA-NET) proposed by Qin et al. to enhance the sample images through filtering. The FA feature attention module can flexibly process different feature regions and assign different weights to each channel and pixel feature of the image. These weights are combined with local residual learning to generate the enhanced filtered image. Compared to images that have only undergone grayscale processing, the enhanced filtered image has a more uniform grayscale distribution and a better enhancement effect. Simultaneously, images without captured faces are manually removed and labeled.

[0066] Gaussian noise is used to balance the dataset: Due to the uneven distribution of the dataset, for example, the number of fear class samples in the RAF-DB dataset is only 6% of the number of happy class samples. Simply expanding the data of the class with fewer samples can easily lead to overfitting of the network during training. Therefore, noise is used to augment the data of the class with fewer samples in the experiment. Compared with the traditional method of copying the dataset, adding noise can effectively avoid overfitting while expanding the data. Common noises include Gaussian noise and Poisson noise. This invention mainly uses Gaussian noise to expand the dataset to a certain extent, thereby achieving a balance of sample numbers between classes. Specifically, Gaussian noise is randomly generated with a mean of 0 and a variance of 5, and the data of the class with fewer samples is expanded by two to three times.

[0067] Dataset Splitting: After preprocessing, the RAF-DB, JAFFE, and CK+ datasets were split into three parts: training, validation, and test sets. The JAFFE and CK+ datasets were randomly split, with a training to test set ratio of 4:1. 10% of the face images from the original training sets of the RAF-DB, JAFFE, and CK+ datasets were randomly selected as the validation set, while the remaining 90% were used as the training set.

[0068] Network training: The training parameters during the training process are shown in Table 3. Through training, the average accuracy of the training set reached 99.59%, the average accuracy of the validation set reached 94.62%, and the average training time per iteration was less than 1 second.

[0069] Since the JAFFE and CK+ datasets were collected in the laboratory, the environment is less complex, the brightness difference of the face images is small, and the difference in the number of samples between classes is not large, making them suitable for testing. Therefore, this invention uses 5-fold cross-validation and the JAFFE and CK+ datasets to test and analyze the network model of this invention.

[0070] Table 3 shows the training parameters.

[0071]

[0072] The training employs a method of dynamically decaying the learning rate based on the loss value. This method ensures the model remains in the learning phase, avoiding increased training time due to invalid iterations, thereby further optimizing the model and improving its performance. The learning rate is decayed based on the decline of the loss value every five iterations. If the loss value shows an upward trend, the learning rate is decayed to 5% of its original value; otherwise, the original learning rate is maintained and training continues. When the learning rate reaches zero, decay stops. During each iteration, a callback function checks the change in the loss value, dynamically adjusting the learning rate accordingly. Specifically, the initial learning rate is set to 0.001. Figure 6 This is a visualization of the learning rate dynamically changing with the validation set loss value during the experiment. Through experimental comparison, using a decaying learning rate can improve accuracy by about 2% compared to using a fixed learning rate.

[0073] Test results on the CK+ dataset: The model of this invention was used to test the CK+ dataset. Table 4 shows the confusion matrix of the test results on the CK+ dataset.

[0074] Table 4 shows the confusion matrix of the test results on the CK+ dataset.

[0075]

[0076] The model of this invention achieved an average precision of 91.82% on the CK+ dataset. As shown in the table, the precision was as high as 100% for "happy," and relatively high for "bored," "neutral," and "surprised," at 90.43%, 90.54%, and 97.6% respectively. The precision for "fear" and "sadness" was lower, at 74.72% and 75.19% respectively. This is mainly because the features in "fear" and "sadness" exhibit a tendency towards "anger," and the number of "fear" samples is too small, leading to an imbalanced dataset and thus lower precision.

[0077] JAFFE Dataset Test Results: The model of this invention was used to test the JAFFE dataset. Table 5 shows the confusion matrix of the JAFFE dataset test results.

[0078] Table 5. Confusion Matrix of JAFFE Dataset Test Results

[0079]

[0080] The model of this invention achieved an average accuracy of 77.89% on the JAFFE dataset, which is somewhat lower than that on the CK+ dataset. As shown in the figure, the accuracy was high for "happy" and "surprised," reaching 93.33% and 95% respectively; for "bored," the accuracy was 68.67%; for "neutral," "sad," and "angry," the accuracy was 78.67%, 62.67%, and 70.00% respectively; and for "fear," the accuracy was low at 62.00%. This is mainly because some data in the "fear" category exhibited characteristics of other categories such as "sad" and "surprised," thus causing a decrease in the recognition rate.

[0081] Observations revealed a significant decrease in accuracy in the JAFFE dataset compared to the CK+ dataset, particularly in categories such as "angry," "sad," and "annoyed." Several factors contribute to this: First, the facial expression data in the JAFFE dataset was collected earlier, leading to similarities between different classes. Second, the facial expression features in the JAFFE dataset are not clearly defined; the same facial expression may exhibit characteristics of multiple classes, making it difficult to distinguish visually. Third, the JAFFE dataset suffers from an imbalanced data distribution. Although Gaussian noise balancing was applied during data preprocessing to mitigate the imbalance to some extent, it still resulted in overfitting and low generalization ability.

[0082] This experiment will compare the present invention with references to [1] Wang XH, Liang YC, Ma XC. A Deep Learning Algorithm for Face Expression Classification Based on Inception [J]. Optical Technique, 2020, 46(03): 347-353, [2] Zhang HL, Bai XY. Facial Expression Recognition Method using Optimized Pruning GoogLeNet [J]. Computer Engineering and Applications, 2021, 57(19): 179-188, and [3] Szegedy C, Vanhoucke V, Ioffe S, et al. Rethinking the Inception Architecture for Computer Vision [C] / / Proceedingsoft The IEEE Conference on Computer Vision and Pattern Recognition. 2016: 2818-2826, and the reference [4] Du L, Hu H. Weighted patch-based manifold regularization dictionary pair training model for facial expression recognition using iterative optimization classification strategy [J]. Computer Vision and Image Understanding, 2019, 186: 13-24, compared the training time and accuracy. The red mark is the optimal value of the comparison index, and the purple mark is the second best value of the index. Among them, the reference [3] proposed the InceptionV3 model.

[0083] The experimental results show that the main contribution of this invention is to significantly improve time efficiency. Compared with the present invention and the InceptionV3 model (reference [3]), the time efficiency of the present invention model is significantly improved, as shown in Table 6.

[0084] Table 6 Comparison of training time (ms) for different algorithms

[0085]

[0086] As can be seen from Table 6, the training time of the model of the present invention is less than 16ms. Compared with the InceptionV3 model (reference [3]), the present invention has a significant improvement in training time.

[0087] Table 7. Classification accuracy (%) of different algorithms on the CK+ dataset

[0088]

[0089] As shown in Table 7, on the CK+ dataset, compared with the InceptionV3 model, this invention achieved improvements in "happy" and "neutral" and had a higher average accuracy than the traditional model. Compared with references [1], [2] and [3], this invention had the highest average accuracy, especially in "angry", "bored", "happy", "neutral" and "surprised". However, the accuracy in "sad" and "fearful" was lower than that of other references.

[0090] As shown in Table 8, compared to the InceptionV3 model, the model of this invention achieves improved performance in recognizing the "surprise" class on the JAFFE dataset, and its average accuracy is superior to that of the InceptionV3 model. However, the comparison results on the JAFFE dataset are significantly different from those on the CK+ dataset. This is because the data features of some classes in the JAFFE dataset are not obvious, and the emotion cannot be discerned by the naked eye. There are cases where there is a large discrepancy between the label and the data features, requiring relabeling and cleaning of the dataset to further improve the accuracy of the test.

[0091] Through comparative experiments, the model proposed in this invention significantly improves training efficiency compared to other models, which is the main contribution of this invention. Furthermore, comparative experiments revealed that the recognition accuracy of this invention's model on the CK+ database is significantly better than other comparative models. However, due to class imbalance and unclear class boundaries in the JAFFE database, the improvement in recognition accuracy of this invention's model on the JAFFE database is relatively small.

[0092] Table 8. Classification accuracy (%) of different algorithms on the JAFFE dataset

[0093]

[0094] This invention proposes a lightweight network model for recognizing emotions in facial images. Experimental results demonstrate that this model significantly improves time efficiency compared to the InceptionV3 network and classic facial emotion recognition methods, and also shows substantial improvements in classification accuracy on the CK+ dataset, particularly for "happy" and "neutral" emotions. On the JAFFE dataset, the average classification accuracy of this invention is higher than that of the comparative literature. Furthermore, the proposed model avoids overfitting while reducing computational cost. Future work will employ a strong feature-preserving model to further reduce training time and improve both time efficiency and classification accuracy. Simultaneously, a facial recognition algorithm will be used to identify and crop faces, further removing irrelevant information from the dataset, thereby optimizing the dataset and striving for even higher classification accuracy.

[0095] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A facial expression recognition method based on the InceptionV3 network, characterized in that, Includes the following steps: Collect facial image data; An improved facial expression recognition network model is obtained based on the Inception V3 network. This model includes two convolutional layers, two Inception modules with different structures (3×Inception and 5×Inception), two pooling layers, and two fully connected layers combined with Dropout. The two convolutional layers use 3×3 kernels with 32 channels and 3×3 kernels with 64 channels respectively for convolution operations, with padding of 2 and stride of 1 for both kernels. During convolution operations, the ReLU activation function is used to enhance the non-linear characteristics of the facial expression recognition network model. The 3×Inception structure uses a 1×1 convolutional kernel and two consecutive 3×3 convolutional kernels for convolution operations; the 5×Inception structure uses 1×3 and 3×1 asymmetric convolutional kernels for convolution operations; it also includes training the facial expression recognition model using a dynamic learning rate decay method based on the loss value. The training process includes: decaying the learning rate by judging the decay of the loss value every five iterations; when the loss value is increasing, the learning rate is decayed to 5% of the original learning rate; otherwise, the original learning rate is maintained while training the model; when the learning rate decays to 0, the learning rate decay is stopped. Facial image data is input into a facial expression recognition network model. Features are extracted from the facial images through convolutional layers and pooling layers to obtain feature maps. The Dropout packet loss rate function is used to randomly disconnect the connections between neurons in the facial expression recognition network model at a packet loss rate of 25% after the pooling layer operation. After the convolutional layer operation, the Batch Normalization regularization method is used to extract facial features by executing multiple convolutional kernels and pooling windows in parallel. The feature map is extracted sequentially using a 3×Inception structure and an asymmetric 5×Inception structure. The extracted features are classified using two fully connected layers with a packet loss rate of 25%, and a face feature classification map is output. Based on a facial emotion database, the Softmax function is used to classify facial features and expressions.

2. The facial expression recognition method based on the InceptionV3 network according to claim 1, characterized in that, After acquiring face image data, the process also includes preprocessing the acquired face image data, which includes the following steps: extracting face regions using a Haar feature extraction classifier; filtering and enhancing the extracted face images using FFA-NET; cropping and cleaning the face images after filtering; and enhancing the processed image data using Gaussian noise.

3. The facial expression recognition method based on the InceptionV3 network according to claim 1, characterized in that, The facial emotion database includes seven categories of emotions: anger, annoyance, fear, happiness, neutrality, sadness, and surprise.

4. The facial expression recognition method based on the InceptionV3 network according to claim 1, characterized in that, It also includes using 2×2 max pooling to process redundant information in the facial expression recognition network model before performing feature classification on the extracted features through two fully connected layers with a 25% packet loss rate.

Images