A face expression recognition method based on a spatial attention network

Abstract

The application relates to a face expression recognition method based on a spatial attention network, and aims to solve the problem that a global mean pooling layer in an existing spatial attention (SA) module inhibits some potential feature information, resulting in low face expression recognition accuracy.A specific process of the face expression recognition method based on the spatial attention network is as follows: step one, face image preprocessing, obtaining a face expression image dataset; the dataset comprises a training set and a test set; step two, performing feature extraction on the face expression image dataset obtained in the step one; step three, establishing a spatial attention module, inputting the features extracted in the step two into the spatial attention module to obtain a trained network model; and step four, inputting the face expression image test set obtained in the step one into the trained network model to output a classification result.The application is used in the technical field of image recognition processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition processing, in particular to a facial expression recognition method based on a spatial attention network. BACKGROUND

[0002] Facial expression recognition is a hot issue in the field of computer vision and plays an important role in human daily communication. It has important application value in the fields of human-computer interaction, fatigue detection, auxiliary medical treatment, case detection, etc. With the rapid development of artificial intelligence technology, deep learning has achieved great success in image recognition, classification tasks and other fields, and has made great progress in facial expression recognition. In the traditional expression data set under ideal conditions, the recognition rate has made great progress, and the accuracy rate in CK+ and JAFFE data sets has reached nearly 100%. But in the expression data sets published in recent years, most of the faces are affected by light, occlusion and multi-pose under natural conditions. Among them, multi-pose often cannot be avoided, because the camera for picture collection cannot always face the face, and the face collected each time cannot always maintain the same pose, which is the main problem affecting the current facial expression recognition. In order to solve this problem, many researchers introduce attention mechanism to obtain the target area of focus, so as to suppress other useless information. Among them, the most widely used is CBAM (Convolutional Block Attention Module), which respectively introduces a spatial attention module and a channel attention module. In the spatial attention SA (Spatial Attention) module of the prior art, global mean pooling and global maximum pooling are respectively performed, and dimension reduction is performed through a convolutional layer. Among them, the global mean pooling layer takes the mean value of each sub-feature point in the channel dimension, however, there are some small values in the output of the same position sub-feature points in different channels, which will bring smaller output mean value, thereby suppressing some potential distinguishing facial expression features. SUMMARY

[0003] The purpose of the present application is to solve the problem that the global mean pooling layer in the existing spatial attention SA module suppresses some potential feature information, resulting in low facial expression recognition accuracy, and a facial expression recognition method based on a spatial attention network is proposed.

[0004] The specific process of a facial expression recognition method based on a spatial attention network is as follows:

[0005] Step one, face image preprocessing, obtaining facial expression image data set; including training set and test set;

[0006] Step two, feature extraction is performed on the facial expression image data set obtained in step one, and the extracted features are denoted as X;

[0007] Step three, establish a spatial attention module, input the features X extracted in step two to the spatial attention module to obtain a trained network model;

[0008] Step four, input the facial expression image test set obtained in step one into the trained network model to output a classification result.

[0009] The beneficial effects of the present application are:

[0010] (1) The present application proposes a facial expression recognition method based on an improved spatial attention MSA (Modified Spatial Attention) network, the improvement lies in using sub-pixel convolution to up-sample part of the channels, and then down-sampling through the maximum pooling layer, so as to filter out the smaller sub-feature point output values in part of the channels. Then, global maximum pooling and global mean pooling are performed on these channels respectively, and dimension reduction is performed through the convolution layer. Compared with the existing spatial attention module, the present application filters out the smaller sub-feature points at the same position in part of the channels, so that the feature map output by the global mean pooling can better measure the weight of each sub-feature, thereby obtaining more effective information. The improved spatial attention module proposed in the present application has good anti-interference and robustness for occlusion and multi-pose changes in facial expression recognition, and improves the facial expression recognition accuracy.

[0011] (2) The present application can be inserted into any network as the existing spatial attention module.

[0012] (3) To verify the efficiency of the present application, experiments are carried out on the existing public data set, and the recognition accuracy of 89.28% is obtained on the RAF-DB data set, and the recognition accuracy of 89.00% is obtained on the FERPlus data set, which is higher than that of other complex methods. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 It is a basic flow chart of the facial expression recognition method based on the spatial attention network of the present application;

[0014] Figure 2 It is a schematic diagram of the spatial attention module in the CBAM method;

[0015] Figure 3 It is a schematic diagram of the spatial attention module proposed in the present application;

[0016] Figure 4The schematic diagram of sub-pixel convolution up-sampling is that sub-feature points at the same position in part of channels are gathered together, and the maximum pooling layer is down-sampled (is to part of channels, for example, the square of the up-sampling factor r in the application is 16, that is, 512 / 16 = 32, that is, there are 32 channels (each channel can be regarded as a group) after up-sampling, and the maximum value of the sub-feature points in each channel (group) is obtained through maximum pooling). DETAILED DESCRIPTION

[0017] Specific implementation one: the specific process of the face expression recognition method based on the spatial attention network in the embodiment is as follows:

[0018] Step one, face image preprocessing, obtaining a face expression image dataset; including a training set and a test set;

[0019] Step two, feature extraction is performed on the face expression image dataset obtained in step one, and the extracted features are denoted as X;

[0020] Step three, a spatial attention module is established, the features X extracted in step two are input into the spatial attention module, and a trained network model is obtained;

[0021] Step four, input the face expression image test set obtained in step one into the trained network model, and output the classification result.

[0022] Specific implementation two: the difference between the embodiment and the specific implementation one is that the face image preprocessing in step one obtains a face expression image dataset; the specific process is as follows:

[0023] Step one, face image cropping and scaling; the process is as follows:

[0024] The face image in the face database image dataset (the face database image dataset contains background) is cropped by using a face detection algorithm (the face detection algorithm includes Dlib MTCNN Retinaface, MTCNN) to obtain a face image without background, and the face image is enlarged to 224x224 size, thereby obtaining a cropped and scaled face expression image dataset;

[0025] Step two, the training set image data after face image cropping and scaling in step one is enhanced; the process is as follows:

[0026] The training set face expression image after face image cropping and scaling in step one is subjected to image data enhancement through normalization processing, center random cropping, horizontal flipping, and random occlusion, thereby obtaining an enhanced face expression image dataset.

[0027] The other steps and parameters are the same as those in the specific implementation one.

[0028] Specific implementation three: the difference between this embodiment and specific implementation one or two is that the feature extraction is performed on the facial expression image data set obtained in step one in step two, and the extracted feature is denoted as X; the specific process is as follows:

[0029] The facial expression image data set obtained in step one is input into the ResNet18 residual network for feature extraction, and the extracted feature is denoted as X.

[0030] The size of the feature X is 512x7x7, 512 is the channel number of the feature X, the first 7 is the height of the feature X, and the second 7 is the width of the feature X.

[0031] The other steps and parameters are the same as those in specific implementation one or two.

[0032] Specific implementation four: the difference between this embodiment and one of specific implementation one to three is that a spatial attention network model is established in step three, the feature X extracted in step two is input into the spatial attention network model, and a classification result is output; the specific process is as follows:

[0033] The spatial attention mechanism in the prior art CBAM method is as shown in Figure 2 , global mean pooling and global maximum pooling are performed on the channel dimension respectively, (wherein the global mean pooling is to obtain the mean value of each sub-feature point in all channels, and the global maximum pooling is to obtain the maximum value of each sub-feature point in all channels) to obtain two 1x H x W feature maps, and the two feature maps are spliced together in the channel dimension. Secondly, dimension reduction is performed through a 7x7 convolution kernel, and a spatial feature map is generated through a Sigmoid function. Finally, point multiplication is performed with the input feature to obtain the finally generated feature map. The improved spatial attention module proposed in the present application is as shown in Figure 3 . The present application adds sub-pixel convolution and maximum pooling layer before global maximum pooling and global mean pooling, filters out small values of sub-feature points in some channels, and obtains more distinguishable features. Figure 2 、 Figure 3 , wherein C represents the number of image channels, H represents the image height, and W represents the image width.

[0034] The attention mechanism proposed specifically includes the following contents:

[0035] Step three one, sub-pixel convolution upsampling is performed on the feature X extracted in step two, and then maximum pooling layer downsampling is performed, as shown in Figure 4 , the specific process is as follows:

[0036] Step three one, each sub-feature point at the same position in part of the channel dimension of the feature X extracted in step two is gathered together and mapped to the same feature map, (considering that the sub-pixel convolution upsampling factor is r2 , and experiments show that r=4 can obtain higher accuracy) to obtain a feature size of 32x(4x7)x(4x7), 32 is the number of feature channels, the first 4x7 is the feature height, and the second 4x7 is the feature width;

[0037] Step three one two, the feature map obtained by mapping is down-sampled through a max-pooling layer, the window size of the max-pooling layer is 4x4, and the step size of the max-pooling layer is 4, so that a feature map M with the same size as the feature X extracted in step two is obtained, the feature map size is 32x7x7, 32 is the number of feature map channels, the first 7 is the height of the feature map, and the second 7 is the width of the feature map;

[0038] M = Maxpool2d(pixelshuffle(r 2 ))

[0039] Step three two, the feature map M is respectively subjected to global average pooling and global maximum pooling, and the process is as follows:

[0040] Step three one two, the feature map M is subjected to global average pooling to obtain a feature map F1, the size of the feature map F1 is 1x7x7, 1 is the number of channels of the feature map F1, the first 7 is the height of the feature map F1, and the second 7 is the width of the feature map F1; F1 = AvgPool(M);

[0041] Step three two two, the feature map M is subjected to global maximum pooling to obtain a feature map F2, the size of the feature map F2 is 1x7x7, 1 is the number of channels of the feature map F2, the first 7 is the height of the feature map F2, and the second 7 is the width of the feature map F2; F2 = MaxPool(M);

[0042] Step three, the feature map F1 and the feature map F2 are spliced in the channel dimension to obtain F, and F is input into a convolution layer with a convolution kernel size of 7x7 for dimension reduction, and the output feature map size is 1x7x7, 1 is the number of feature channels, the first 7 is the height of the feature map, and the second 7 is the width of the feature map;

[0043] Then the feature map with a size of 1x7x7 is sequentially subjected to BN and Sigmoid function to obtain a feature map N;

[0044] F = [F1:F2]

[0045] N = σ(BN(conv(F)))

[0046] where Pixelshuffle represents sub-pixel convolution up-sampling, r 2wherein, represents up-sampling factor; represents Sigmoid function, Maxpool represents max-pooling layer, [F1:F2] represents splicing feature map F1 and feature map F2 in channel dimension, and conv represents convolution layer;

[0047] Step three, the obtained feature map N is multiplied with the input X element by element to obtain a feature map Y, the feature map Y has a size of 512x7x7, 512 is the channel number of the feature map Y, the first 7 is the height of the feature map Y, and the second 7 is the width of the feature map Y;

[0048] Y = X * N

[0049] wherein, * represents element-by-element multiplication;

[0050] Step three, the obtained feature map N is multiplied with the input X element by element to obtain a feature map Y, the feature map Y has a size of 512x7x7, 512 is the channel number of the feature map Y, the first 7 is the height of the feature map Y, and the second 7 is the width of the feature map Y;

[0051] The other steps and parameters are the same as one of the first to third embodiments.

[0052] The fifth embodiment is different from one of the first to fourth embodiments in that, in the step three, the feature map Y is sent into a global mean pooling layer to obtain a feature map with a size of 512x1x1, the feature map with the size of 512x1x1 is flattened and sent into a fully connected layer for dimension reduction, and finally sent into a Softmax layer to obtain a classification result y.

[0053] wherein, the fully connected layer is:

[0054] FC = (512, num_class)

[0055] The expression of the Softmax function is:

[0056]

[0057] wherein, num_class represents the number of expression categories, m represents the output of the previous layer, c is the number of output neurons of the upper layer (if the expression category is 8, the output is (512, 8) and there are eight neurons in total), y i represents the probability of outputting a certain expression category, and the maximum probability value represents the expression category of the image; m i is the value of the i-th neuron, m jThe value of the jth neuron is denoted as i, which represents the number of expression categories, respectively, happy, normal, sad, surprise, fear, anger, disgust, contempt, and a total of eight expression categories.

[0058] The other steps and parameters are the same as one of the first four embodiments.

[0059] The beneficial effects of the present application are verified by the following examples:

[0060] Example 1:

[0061] The specific steps of the face expression recognition method based on the improved spatial attention network are as follows:

[0062] Step 1, face image preprocessing, specifically including:

[0063] 1) Face image cropping and scaling; Most of the face database data set contains background information, the face part in the picture is detected by using a multi-task convolutional neural network (MTCNN), and the face picture is aligned, and then the face image is cropped, in order to meet the input size of the picture in the Resnet network, the picture is enlarged to 224x224, and the processed face expression data set is obtained;

[0064] 2) Image data enhancement; In order to make the network model have better generalization ability and avoid overfitting, the face image in the training set is enhanced, which includes normalization processing, center random cropping, horizontal flipping and random occlusion.

[0065] Step 2, the data set preprocessed in step 1 is sent into the ResNet18 residual network for feature extraction.

[0066] In the network, ResNet18 specifically includes two modules:

[0067] The first module is composed of a convolutional layer and a batch normalization layer, the convolutional layer is composed of 64 7x7 size convolutional kernels, the step is 2 and the zero padding is 3, the window size of the maximum pooling layer is 3x3, and the step is 2. The input size of the convolutional layer is 3x224x224, and the output size is 64x112x112; the input size of the pooling layer is 64x112x112, and the output size is 64x56x56;

[0068] The second module includes four residual modules with the same structure, which are connected in sequence as follows: convolution layer-batch normalization-Relu activation function-convolution layer-batch normalization, wherein the input size is 64x56x56, and the output size of the fourth residual module is 512x7x7, denoted as X. (remove the GAP layer and the fully connected layer Fc in the Resnet18 network). The Resnet18 module is pre-trained on the Ms-Celeb-1M dataset.

[0069] Step 3: The features obtained in step 2 are input into the improved spatial attention module to obtain a trained network model.

[0070] Since the same position sub-feature points in different channels may have some small values, the feature map output by the global average pooling layer will suppress some distinctive facial expression features. Therefore, a sub-pixel convolution and a maximum pooling layer are added before the global maximum pooling and the global average pooling to filter out small values of sub-feature points in some channels and obtain more distinctive features.

[0071] The improved spatial attention mechanism specifically includes the following steps:

[0072] 1) The features extracted in step 2 are first subjected to sub-pixel convolution upsampling, which gathers each sub-feature point at the same position in the channel dimension together and maps it to the same feature map, (here r is 4) to obtain a feature with a size of 32x(4x7)x(4x7). Then, the feature is downsampled by a maximum pooling layer with a window size of 4x4 and a step size of 4, so that a feature map M with the same input size is obtained, and the feature map has a size of 32x7x7.

[0073] 2) The feature map M is subjected to global average pooling to obtain a feature map F1, and the feature map M is subjected to global maximum pooling to obtain a feature map F2. Both feature maps have a size of 1x7x7.

[0074] 3) The feature map F1 and the feature map F2 are spliced in the channel dimension to obtain F, which is input to a convolution kernel with a size of 7x7 for dimension reduction, and the output size is 1x7x7; then the feature map is sequentially subjected to BN and a Sigmoid function to obtain a feature map N.

[0075] F1=AvgPool(M)

[0076] F2=MaxPool(M)

[0077] M=Maxpool2d(pixelshuffle(r 2 ))

[0078] F=[F1:F2]

[0079] N = σ(BN(conv(F)))

[0080] wherein Pixelshuffle represents sub-pixel convolution up-sampling; wherein σ represents a Sigmoid function;

[0081] 4) The obtained feature map N is multiplied with the input X element by element to obtain a feature map Y, and the feature map output size of Y is 512x7x7.

[0082] Y = X * N

[0083] wherein * represents element by element multiplication;

[0084] Step 4, specifically comprising the following steps:

[0085] Y is sent into a global mean pooling layer to obtain a feature map with an output size of 512x1x1, which is then flattened and sent into a fully connected layer, and finally sent into a Softmax layer to obtain an output y.

[0086] The fully connected layer is:

[0087] FC = (512, num_class)

[0088] The expression of the Softmax function is:

[0089]

[0090] wherein num_class represents the number of expression categories, m represents the output of the previous layer, and c is the number of output neurons of the upper layer (if the expression category is 8, the output is (512, 8) with eight neurons in total), y i represents the probability of a certain expression category of the output, and the maximum probability value represents the expression category of the image. Wherein i represents the number of expression categories, which are happy, neutral, sad, surprised, frightened, angry, disgusted, and contemptuous, a total of eight expression categories.

[0091] In the present application, the test results of the present method are compared with other advanced methods, see Table 1, Table 2. The influence of different up-sampling factors r on the experimental results is shown in Table 3. The dataset RAF-DB (Shan Li, Weihong Deng, and Jun Ping Du, “Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild,” In CVPR, pp. 2852-2861, 2017) contains 30,000 face images annotated with basic or compound expressions by 40 well-trained human encoders. In our experiment, only images containing 6 basic expressions (happy, surprised, sad, angry, disgusted, and fearful) and neutral expression were used, a total of 7 classes. 12271 images for training and 3068 images for testing were obtained.

[0092] The dataset FERPlus (Emad Barsoum, Cha Zhang, Cristian Canton Ferrer, and Zhengyou Zhang, “Training deep networks for facial expression recognition with crowd-sourced label distribution,” In ACM ICMI, pp. 279-283, 2016) is an extension of FER2013 in the ICML 2013 challenge. It is a large-scale dataset collected in Google Chrome. It contains 28,709 training images, 3,589 validation images and 3,589 test images, all of which are adjusted to a size of 48x48. There are more contempt expressions in this dataset, a total of 8 categories. In this experiment, the training images and the validation images are collectively used as the training set.

[0093] Table 1

[0094] Method Accuracy DDA[1] 86.90％ PSR[2] 88.98％ KTN[3] 88.07％ ResNet18+SA 88.79％ ResNet18+MSA 89.28％

[0095] wherein, SA is the original spatial attention module ( Figure 2 ), and MSA is the spatial attention module of the present application ( Figure 3 );

[0096] Table 1 is the comparison of the face expression recognition accuracy of the method proposed in the present application and other methods on the RAF-DB dataset.

[0097] Table 2

[0098] Method Accuracy SENet[4] 88.80％ RAN[5] 88.55％ ADC-Net[6] 88.90％ ReSNet18+SA 88.75％ ResNet18+MSA 89.0O％

[0099] Table 2 shows the results of the facial expression recognition accuracy of the proposed method and other methods on the FERPlus dataset.

[0100] Table 3

[0101] r RAF-DB FERPlUS 1 88.79％ 88.75％ 2 88.95％ 88.59％ 4 89.28％ 89.00％ 8 89.05％ 88.52％ 16 88.56％ 88.43％

[0102] Table 3 shows the influence of different up-sampling factors r on the experimental results. As can be seen from the table, some small values of sub-feature points are filtered out, thereby increasing the mean value of each sub-feature point, and more distinctive features can be obtained, thereby improving the accuracy of facial expression recognition. When r = 4, the accuracy of the two datasets reaches the highest, but then decreases with the increase of r. When r = 16, the accuracy decreases sharply, because the mean value of each sub-feature point is close to the maximum value of the sub-feature point at this time, which will lead to all sub-feature points having a high weight, thereby extracting a large amount of useless information, such as occlusions, hair, etc., which will seriously affect the network performance.

[0103] [1] Farzaneh AH, Qi X (2020) Discriminant Distribution-Agnostic Loss for Facial Expression Recognition in the Wild. In: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 406-407. IEEE

[0104] [2] Vo TH, Lee GS, Yang HJ, Kim SH (2020) Pyramid with super resolution for in-the-wild facial expression recognition. IEEE Access 8: 131988-132001

[0105] [3] Li H, Wang N, Ding X, Yang X, Gao X (2021) Adaptively learning facial expression representation via CF labels and distillation. IEEE Trans Image Process 30: 2016-2028

[0106] [4] Albanie S, Nagrani A, Vedaldi A, Zisserman A (2018) Emotion recognition in speech using cross-modal transfer in the wild. In: Proceedings of the 26th ACM international conference on Multimedia, pp 292-30

[0107] [5] Wang K, Peng X, Yang J, Meng D, Qiao Y (2020) Region attention networks for pose and occlusion robust facial expression recognition. IEEE Trans Image Process 29:4057-4069

[0108] [6] Xia HY, Li C, Tan Y, Li L, Song S (2021) Destruction and reconstruction learning for facial expression recognition. IEEE Multimedia 28(2):20-28

[0109] The present application can also have other various embodiments, and those skilled in the art can make various corresponding changes and modifications according to the present application without departing from the spirit and essence of the present application, but these corresponding changes and modifications should all belong to the protection scope of the claims attached to the present application.

Claims

1. A face expression recognition method based on a spatial attention network, characterized in that: The specific process is: Step one, face image preprocessing, obtaining face expression image dataset; including training set and test set; Step two, feature extraction is performed on the face expression image dataset obtained in step one, and the extracted features are denoted as X; Step three, a spatial attention module is established, the features X extracted in step two are input into the spatial attention module, and a trained network model is obtained; Step four, input the face expression image test set obtained in step one into the trained network model, and output the classification result; The spatial attention network model is established in step three, the features X extracted in step two are input into the spatial attention network model, and the classification result is output; The specific process is: Step three one, the features X extracted in step two are subjected to sub-pixel convolution upsampling, and then subjected to downsampling through a max pooling layer, and the specific process is: Step three one, each sub-feature point at the same position in the feature X extracted in step two is gathered together in the dimension of part of the channel, and is mapped to the same feature map, to obtain a feature with a size of 32x(4x7)x(4x7), 32 being the number of feature channels, the first 4x7 being the feature height, and the second 4x7 being the feature width; Step three one, the mapped feature map is subjected to downsampling through a max pooling layer, the max pooling layer window size is 4x4, and the max pooling layer step size is 4, so that a feature map M with the same size as the feature X extracted in step two is obtained, the feature map size is 32x7x7, 32 being the number of feature map channels, the first 7 being the feature map height, and the second 7 being the feature map width; M = Maxpool2d(pixelshuffle(r 2 )) Step three two, the feature map M is subjected to global average pooling and global maximum pooling respectively, and the process is: Step three two, the feature map M is subjected to global average pooling, to obtain a feature map F1, the feature map F1 size is 1x7x7, 1 being the number of feature map F1 channels, the first 7 being the feature map F1 height, and the second 7 being the feature map F1 width; F1=AvgPool(M); Step three two, the feature map M is subjected to global maximum pooling, to obtain a feature map F2, the feature map F2 size is 1x7x7, 1 being the number of feature map F2 channels, the first 7 being the feature map F2 height, and the second 7 being the feature map F2 width; F2=MaxPool(M); Step three, the feature map F1 and the feature map F2 are spliced in the channel dimension to obtain F, F is input into a convolution layer with a convolution kernel size of 7x7 for dimension reduction, and a feature map with a size of 1x7x7 is output, 1 being the number of feature map channels, the first 7 being the feature map height, and the second 7 being the feature map width; Then the feature map with a size of 1x7x7 is sequentially subjected to BN and Sigmoid function to obtain a feature map N; F=[F1:F2] N=σ(BN(conv(F))) where Pixelshuffle denotes sub-pixel convolution up-sampling, r 2 denotes an up-sampling factor; σ denotes a Sigmoid function, Maxpool denotes a max-pooling layer, [F1:F2] denotes concatenating feature map F1 and feature map F2 in the channel dimension, and conv denotes a convolution layer; Step three four, the obtained feature map N is multiplied element by element with the input X to obtain a feature map Y, the feature map Y size is 512x7x7, 512 being the number of feature map Y channels, the first 7 being the feature map Y height, and the second 7 being the feature map Y width; Y=X*N Wherein, * represents element by element multiplication; Step three five, the feature map Y is sent into the global mean pooling layer, the feature map size is 512x1x1, then the feature map with the size of 512x1x1 is flattened and sent into the fully connected layer for dimension reduction, finally the Softmax layer is sent to obtain the classification result y. 2.The method of claim 1, wherein: The face image preprocessing in the step one obtains the face expression image dataset; the specific process is: Step one, face image cropping and scaling; the process is: The face detection algorithm is used to crop the images in the face library dataset to obtain face images without background, and the face images are enlarged to 224x224 to obtain the face expression image dataset after cropping and scaling; Step two, the training set image data after face image cropping and scaling in step one is enhanced; The process is: The face expression image after face image cropping and scaling in step one is normalized, center randomly cropped, horizontally flipped, and randomly occluded to obtain the enhanced face expression image dataset. 3.The method of claim 2, wherein: The face expression image dataset obtained in step one is extracted in step two, and the extracted feature is denoted as X; the specific process is: The face expression image dataset obtained in step one is input into the ResNet18 residual network for feature extraction, and the extracted feature is denoted as X. The size of the feature X is 512x7x7, 512 is the channel number of the feature X, the first 7 is the height of the feature X, and the second 7 is the width of the feature X.

4. The method of claim 3, wherein the method is based on a spatial attention network. In step three five, the feature map Y is sent into the global mean pooling layer, the feature map size is 512x1x1, then the feature map with the size of 512x1x1 is flattened and sent into the fully connected layer for dimension reduction, finally the Softmax layer is sent to obtain the classification result y; the specific process is: The fully connected layer is: FC=(512, num_class) The expression of the Softmax function is: wherein num_class represents the number of expression categories, m represents the output of the previous layer, c is the number of output neurons of the upper layer, y i represents the probability of outputting a certain expression category, and the maximum probability represents the expression category of the image; m i is the value of the i-th neuron, m j is the value of the j-th neuron, i represents the number of expression categories, and respectively represents happy, neutral, sad, surprised, frightened, angry, disgusted, and contemptuous, a total of eight expression categories.

Images