A lightweight convolutional network facial emotion recognition method with auxiliary classifier

By adopting a lightweight convolutional network with auxiliary classifier in facial emotion recognition, combining deep separable convolution and residual modules, the SENet channel attention mechanism is introduced, and the model parameter surge and overfitting in the existing technology is solved. The problem of this achieves efficient and accurate facial emotions recognition.

CN116580436BActive Publication Date: 2024-11-01CHANGCHUN UNIV OF SCI & TECH
View PDF 11 Cites 0 Cited by

Patent Information

Application Number
CN202310504418.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-05-08
Publication Date
2024-11-01
Estimated Expiration
2043-05-08

AI Technical Summary

Technical Problem

The existing facial emotion recognition technology has a surge in model parameters due to the small sample size of the data set, which is large in calculations, making it difficult to ensure real-time performance. In addition, when the data set is too small, it is easy to cause overfitting and reduce model performance.

Method used

Using a lightweight convolutional network with auxiliary classifier, the parameters can be reduced through deep separation of convolution modules and residual modules, the SENet channel attention mechanism is introduced, and adaptive weighted fusion is combined with the main classifier to reduce the risk of overfitting. Improve model efficiency and accuracy.

Benefits of technology

It is realized that the model depth is increased without increasing the model parameters, the possibility of overfitting and gradient explosion is reduced, the recognition rate and accuracy of the model are improved, and the robustness and generalization ability of the model are enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116580436B_ABST
    Figure CN116580436B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of artificial intelligence technology, and in particular, is a lightweight convolutional network facial emotion recognition method with an auxiliary classifier, which specifically includes the following steps: S101, constructing a lightweight convolutional neural network model with an auxiliary classifier; S102, preparing a face image data set, and dividing it into a training set and a test set in proportion; S103, inputting the face image data in the training set into the lightweight convolutional neural network model with an auxiliary classifier for training; S104, testing the lightweight convolutional neural network model with an auxiliary classifier in the test set, and optimizing the parameters of the convolutional neural network model. The present invention adopts depthwise separable convolution, utilizes pointwise convolution in depthwise separable convolution to increase the model depth while playing a dimensionality reduction role, and also introduces a residual network to further reduce the possibility of model overfitting and gradient explosion.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a lightweight convolutional network facial emotion recognition method with an auxiliary classifier. Background Art

[0002] Emotions are a stress response of humans to the external environment and a comprehensive manifestation of multiple sensory organs. They play a vital role in our daily lives and have therefore been widely studied by scholars from all walks of life. Facial expressions, speech, and body movements are some of the external activities affected by emotions. Changes in brain activity, heart rate, blood pressure, respiratory rate, body temperature, and skin conductance are all examples of internal emotional influences.

[0003] However, facial expressions are one of the most powerful, natural and common signals for humans to convey their emotional state and intentions. Currently, most research on facial emotion recognition uses convolutional neural networks for emotion recognition. Due to the small number of samples in the facial expression dataset, people generally improve recognition performance by increasing network complexity, which often leads to a surge in model parameters and an increase in device calculations, and cannot guarantee the real-time performance of emotion recognition. When the dataset is small, the model is too complex, which is more likely to cause overfitting and the performance of the model will also decline. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] In view of the shortcomings of the prior art, the present invention provides a lightweight convolutional network facial emotion recognition method with an auxiliary classifier, which solves the problems raised in the above background technology.

[0006] (II) Technical solution

[0007] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical scheme: a lightweight convolutional network facial emotion recognition method with an auxiliary classifier, the method specifically comprises the following steps:

[0008] S101. Build a lightweight convolutional neural network model with auxiliary classifier;

[0009] S102, prepare a face image dataset, and divide it into a training set and a test set in proportion;

[0010] S103, inputting the facial image data in the training set into the lightweight convolutional neural network model with auxiliary classifier for training;

[0011] S104, testing the lightweight convolutional neural network model with the auxiliary classifier in the test set, and optimizing the parameters of the convolutional neural network model;

[0012] S105: inputting the facial image data to be recognized into the optimized convolutional neural network model to obtain the prediction result of emotion classification.

[0013] Furthermore, the lightweight convolutional neural network model with an auxiliary classifier constructed in step S101 includes an underlying convolution module, a depthwise separable convolution module, a residual module, an auxiliary classifier, and a main classifier;

[0014] The bottom convolution module extracts shallow features such as texture of face image data through multiple ordinary convolutions and batch normalization;

[0015] The depthwise separable convolution module extracts the deep features of the facial image data by performing depthwise convolution and pointwise convolution operations on the shallow feature data obtained;

[0016] Residual module, the shallow feature data obtained is processed by ordinary convolution operation and batch normalization to output the residual feature map;

[0017] The auxiliary classifier inputs the deep fusion features and passes through the ordinary convolution layer and SENet channel attention module, adaptive average pooling layer and flattening layer, and finally uses the Softmax activation function to predict and classify facial expressions;

[0018] The main classifier, after the deep features are extracted from the input, performs facial expression prediction and classification after adaptive average pooling.

[0019] Furthermore, the depthwise separable convolution module performs channel-by-channel convolution operations through depthwise separable convolutions, and applies the results to all channels to increase the depth of the model and achieve parameter sharing, thereby reducing the parameters of the model and making the model lightweight.

[0020] Furthermore, the auxiliary classifier introduces the SENet channel attention mechanism to enhance or suppress different channels by constructing the importance of each feature channel, thereby improving the recognition accuracy of the auxiliary classifier.

[0021] Furthermore, the residual module network utilizes the gradient signal in the residual module to pass the output of one layer as input to another layer of the architecture through a "skip connection", reducing the possibility of model overfitting and gradient explosion.

[0022] Furthermore, the model is trained using the fer2013 facial expression dataset, which is divided into a training set and a test set in a ratio of 8:2.

[0023] (III) Beneficial effects

[0024] Compared with the prior art, the present invention provides a lightweight convolutional network facial emotion recognition method with an auxiliary classifier, which has the following beneficial effects:

[0025] The present invention adopts depthwise separable convolution and utilizes point-by-point convolution in depthwise separable convolution to increase the model depth while reducing dimensionality. It also introduces residual network to further reduce the possibility of model overfitting and gradient explosion.

[0026] The present invention realizes parameter sharing and reduces network parameters by adopting depthwise separable convolution. The depthwise separable convolution layer decomposes the standard convolution layer into two operations: depthwise convolution and pointwise convolution. Different from the standard convolution layer, the depthwise separable convolution only performs a convolution operation of a 1-dimensional kernel on each channel, and then applies the result to all channels. While reducing the number of parameters, the depth of the model is increased and the efficiency of the model is improved to a certain extent. On the basis of adopting depthwise separable convolution, the model adopts adaptive average pooling before the main classifier to compress the features at all spatial positions into a single feature value, effectively reducing the number of parameters that need to be learned, making the model more lightweight, and improving the recognition rate of the model.

[0027] The present invention makes full use of the features extracted from the middle layer, and uses the extracted overall facial features that can identify facial emotions such as frowning, smiling or anger for auxiliary classification, and introduces the SENet attention mechanism into the auxiliary classifier to improve the prediction accuracy of the auxiliary classifier.

[0028] The present invention further increases the depth of the model without increasing model parameters by adopting depthwise separable convolution. By using the features extracted from the middle layer, an auxiliary classifier is added for auxiliary classification, and the classification results of the auxiliary classifier are adaptively weighted and fused with the classification results of the main classifier to improve the accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flowchart of a lightweight convolutional network facial emotion recognition method with an auxiliary classifier;

[0030] Figure 2 A schematic diagram of the structure of a lightweight convolutional network facial emotion recognition method with an auxiliary classifier;

[0031] Figure 3 Schematic diagram of the specific composition of all depth-separable convolution modules and residual modules of the present invention;

[0032] Figure 4 This is a schematic diagram of the specific composition of all auxiliary classifiers of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Example 1

[0035] like Figure 1-4 As shown, an embodiment of the present invention proposes a lightweight convolutional network facial emotion recognition method with an auxiliary classifier, and the method specifically includes the following steps:

[0036] S101. Construct a lightweight convolutional neural network model with an auxiliary classifier; construct a bottom-level convolution module for shallow feature extraction; a deep separable convolution module that uses successive deep convolutions and point-by-point convolutions to extract deep features, increase model depth, and reduce model parameters; a residual module that performs ordinary convolution and batch normalization to output residual feature maps to reduce the possibility of model overfitting; an auxiliary classifier that uses the overall facial features extracted from the middle layer that can identify facial emotions such as frowning, smiling, or anger to perform auxiliary classification, and introduces the SENet channel attention mechanism to enhance or suppress different channels by constructing the importance of each feature channel, thereby improving the recognition accuracy of the auxiliary classifier; the main classifier, after extracting deep features, performs predictive classification through adaptive average pooling.

[0037] S102, prepare a face image dataset, and divide it into a training set and a test set according to a ratio; the entire model is trained using the fer2013 face expression dataset, which is divided into a training set and a test set according to a ratio of 8:2.

[0038] S103, input the face image data in the training set into the lightweight convolutional neural network model with auxiliary classifier for training; preprocess the training set divided in S102, flip the image at multiple angles, scale the image and translate the image, and input all the images as data sample images into the network model constructed in S101 for training; the model is trained by stochastic gradient descent, and the initial training parameters of the model are learning rate lr=0.01, momentum moment=0.9, nesterov=True, NAG algorithm is used to improve convergence, weight decay weight_decay=0.0001, learning rate scaling factor factor=0.5, patience=2, when the detection index no longer changes, wait for two batches to start adjusting the learning rate, the loss function is the multi-classification logarithmic loss function, and the evaluation index is accuracy.

[0039] S104, testing the lightweight convolutional neural network model with auxiliary classifier in the test set, and optimizing the parameters of the convolutional neural network model; inputting the test set divided in S102 as data sample images into the trained network model in S103, fine-tuning the parameters of the model, and finally achieving the best facial expression recognition effect of the model, and saving the model parameters.

[0040] S105. Input the facial image data to be identified into the optimized convolutional neural network model to obtain the prediction result of emotion classification; pre-process the facial image data to be identified, perform grayscale conversion on the facial image, and correct the length and width of the facial image after the grayscale conversion to adjust the length and width of the facial image to a preset size; segment the facial image according to the preset facial part rules, and mark the segmented parts; and copy the multi-angle flipped image and size-changed image of the facial area and input all the images as data sample images into the trained network model in S104 to perform facial expression prediction and classification.

[0041] Example 2

[0042] like Figure 1 As shown, a lightweight convolutional network facial emotion recognition method with an auxiliary classifier includes the following operations:

[0043] S101, build a lightweight convolutional neural network model with auxiliary classifiers, such as Figure 2 As shown in the figure, a schematic diagram of the structure of a lightweight convolutional neural network facial emotion recognition system with an auxiliary classifier is shown, which specifically includes an underlying convolution module, a depthwise separable convolution module, a residual module, an auxiliary classifier, and a main classifier.

[0044] The underlying convolution module mainly includes two rounds of convolution layers with a convolution kernel of 3x3, a batch normalization layer and a Relu activation function layer for feature extraction. The texture features of the image can be obtained by performing shallow feature extraction through the underlying convolution module.

[0045] like Figure 3 As shown in the figure, the deep separable convolution module is composed of a deep separable convolution layer 1, a batch normalization layer 1, a deep separable layer 2, a batch normalization layer 2, and a maximum pooling layer 1, wherein the deep separable convolution layer uses a 3×3 convolution kernel for feature extraction and padding is used to ensure that the size of the image before and after convolution is consistent; the activation function uses the Relu function for nonlinearization, and the residual module is mainly composed of an ordinary convolution layer 1, a batch normalization layer 3, an ordinary convolution layer 2, and a batch normalization layer 4, wherein the ordinary convolution layer uses a 3×3 convolution kernel for feature extraction. Finally, the deep features extracted by the residual module and the deep separable convolution module are linearly fused.

[0046] like Figure 4 As shown in the figure, the auxiliary classifier is composed of two convolution blocks, SENet channel attention module, adaptive average pooling layer, flattening layer, softmax activation function layer, and finally outputs the prediction result. The two convolution blocks are composed of ordinary convolution with 3×3 convolution kernel and batch normalization layer.

[0047] The main classifier is mainly composed of a convolution layer with a convolution kernel of 3×3, an adaptive average pooling layer and a softmax activation function layer. The adaptive average pooling layer compresses the features at all spatial positions into a single feature value, effectively reducing the number of parameters that need to be learned, making the model more lightweight, and improving the recognition rate of the model. The softmax activation function is used to predict and classify facial expressions.

[0048] Finally, the facial expression prediction results of the auxiliary classifier and the facial expression prediction results of the main classifier are adaptively weighted averaged to better utilize the advantages of each classifier and improve the quality of the fusion results. Adaptive weighted fusion can effectively reduce the problem of model overfitting or underfitting, reduce the risk of the entire system and improve the robustness. The advantages of each classifier complement each other and improve the generalization ability of the entire system.

[0049] S102. Prepare a facial image dataset and divide it into a training set and a test set in proportion. The network model uses the fer2013 facial expression dataset for model training. The fer2013 facial expression dataset consists of 35,886 facial expression pictures, each of which is composed of a 48×48 grayscale image of a fixed size. There are 7 expressions in total. We divide the dataset into a training set and a test set in a ratio of 8:2.

[0050] S103, input the face image data in the training set into the lightweight convolutional neural network model with auxiliary classifier for training; perform multi-angle flipping, scaling and translation preprocessing on the images in the training set, and input all the preprocessed face image data into the model for training. The preprocessing operation can increase the number of training sets, and multi-angle flipping, translation, scaling and other operations can improve the feature extraction ability of the network. The model is trained by stochastic gradient descent. The initial training parameters of the model are learning rate lr=0.01, momentum moment=0.9, nesterov=True, and NAG algorithm is used to improve convergence. Weight decay weight_decay=0.0001, learning rate scaling factor factor=0.5, patience=2. When the detection index no longer changes, wait for two batches to start adjusting the learning rate. L2 regularization is applied in model training, and the regularization coefficient is 0.01, that is, the sum of the squares of all weight parameters is multiplied by 0.01 to penalize, so as to help control the complexity of the model and improve its generalization performance. The loss function of the model is the multi-classification logarithmic loss function, and the evaluation indicator is accuracy.

[0051] S104. Test the lightweight convolutional neural network model with auxiliary classifier in the test set and optimize the parameters of the convolutional neural network model; input the test set described in S102 into the trained network model in S103, test the training effect of the model, fine-tune the model according to the accuracy of the model's facial expression prediction and the loss using the logarithmic loss function to obtain the best prediction effect, and save the model parameters.

[0052] S105. Input the face image data to be tested into the optimized lightweight convolutional neural network model with auxiliary classifier to obtain the prediction result of emotion classification; first collect the face data to be identified, and use the Haar cascade classifier to detect the face information from the input image. The Haar cascade classifier has performed grayscale conversion on the face image when detecting the face, and converted it into a grayscale image; correct the length and width of the face image after grayscale conversion, so that the length and width of the face image are adjusted to a preset size; segment the face image according to the preset face part rule, and copy the face area and input all images as data sample images into the trained network model in S104 to predict facial expressions.

[0053] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A lightweight convolutional network facial emotion recognition method with an auxiliary classifier, characterized by: The method specifically comprises the following steps: S101. Build a lightweight convolutional neural network model with auxiliary classifier; The lightweight convolutional neural network model with an auxiliary classifier constructed in step S101 includes an underlying convolution module, a depthwise separable convolution module, a residual module, an auxiliary classifier, and a main classifier; The bottom convolution module extracts shallow features such as texture of face image data through multiple ordinary convolutions and batch normalization; The depthwise separable convolution module extracts the deep features of the facial image data by performing depthwise convolution and pointwise convolution operations on the shallow feature data obtained; Residual module, the shallow feature data obtained is processed by ordinary convolution operation and batch normalization to output the residual feature map; The depthwise separable convolution module is composed of a depthwise separable convolution layer 1, a batch normalization layer 1, a depthwise separable layer 2, a batch normalization layer 2, and a maximum pooling layer 1. The depthwise separable convolution layer is The convolution kernel is used for feature extraction, and padding is used to ensure that the size of the image before and after convolution is consistent; the activation function uses the Relu function for nonlinearization. The residual module is mainly composed of ordinary convolution layer 1, batch normalization layer 3, ordinary convolution layer 2 and batch normalization layer 4. Among them, the ordinary convolution layer uses The convolution kernel is used for feature extraction, and finally the deep features extracted by the residual module and the depthwise separable convolution module are linearly fused; the feature extraction and feature fusion are performed three times through the depthwise separable convolution module and the residual module in sequence to obtain the deep fusion feature; based on the deep fusion feature, the feature extraction and feature fusion are performed again through the depthwise separable convolution module and the residual module to obtain the final deep fusion feature; The auxiliary classifier inputs the deep fusion features and passes through the ordinary convolution layer and SENet channel attention module, adaptive average pooling layer and flattening layer, and finally uses the Softmax activation function to predict and classify facial expressions; The main classifier performs facial expression prediction and classification on the final deep fusion features through adaptive average pooling; Adaptively weighted average the facial expression prediction results of the auxiliary classifier and the facial expression prediction results of the main classifier; The depthwise separable convolution module performs channel-by-channel convolution operations through depthwise separable convolutions, and applies the results to all channels to increase the depth of the model and achieve parameter sharing, thereby reducing the parameters of the model and making the model lightweight; The auxiliary classifier introduces the SENet channel attention mechanism to enhance or suppress different channels by constructing the importance of each feature channel, thereby improving the recognition accuracy of the auxiliary classifier; The residual module uses the gradient signal in the residual module to pass the output of one layer as input to another layer of the architecture through a "skip connection", thereby reducing the possibility of model overfitting and gradient explosion; The model is trained using the fer2013 facial expression dataset, which is divided into a training set and a test set in a ratio of 8:2; S102, prepare a face image dataset, and divide it into a training set and a test set in proportion; S103, inputting the facial image data in the training set into the lightweight convolutional neural network model with auxiliary classifier for training; S104, testing the lightweight convolutional neural network model with the auxiliary classifier in the test set, and optimizing the parameters of the convolutional neural network model; S105: inputting the facial image data to be recognized into the optimized convolutional neural network model to obtain the prediction result of emotion classification.

Citation Information

Patent Citations

  • Automatic image annotation method based on semi-supervised learning

    CN107644235A

  • Pre-activated residual depth separable convolutional network facial expression recognition method and device and carrier

    CN112668486A

  • Feature fusion and decision fusion mixed multi-modal emotion recognition method

    CN112800875A

  • Intelligent auxiliary guiding system based on voice signal processing

    CN113689885A

  • Expression label correction and identification method based on separable residual attention network

    CN114038037A