Facial expression recognition method based on multi-level and multi-scale attention mechanism
The facial expression recognition method employs a multi-level and multi-scale attention mechanism with a restricted center loss to address the challenges of distinguishing similar expressions, enhancing feature extraction and classification accuracy.
Patent Information
- Application Number
- GB2024003110
- Authority / Receiving Office
- GB · GB
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-07-21
- Filing Date
- 2024-03-04
- Publication Date
- 2025-04-09
AI Technical Summary
Conventional facial expression recognition methods struggle to accurately distinguish between similar facial expressions due to high similarity within categories, facial occlusion, and changes in gestures, while relying solely on cross-entropy loss fails to account for differences between predicted and non-correct labels.
A facial expression recognition method utilizing a multi-level attention module to assign weights to features based on importance, a multi-scale attention module for fusion, and a restricted center loss to enhance feature differentiation, incorporating a feature pre-extraction module, multi-level and multi-scale attention modules, and a cross-entropy loss to optimize the model.
The method improves the robustness and accuracy of facial expression recognition by focusing on important details and contextual information, effectively distinguishing between different categories and preventing large loss values, resulting in enhanced classification performance.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
[0001] The present disclosure relates to the field of facial expression recognition, and in particular to a facial expression recognition method based on a multi-level and multi-scale attention mechanism. BACKGROUND
[0002] Facial expression is one of the simplest and most natural expressions in interpersonal communication, which vividly expresses human emotions and plays an important role in daily life. Facial expression recognition is an important research direction in artificial intelligence. In the past several decades, facial expression recognition has been widely used in fields such as human-computer interaction, digital entertainment, health care, and driving detection. As a result, more persons pay attention to researches on facial expression recognition.
[0003] Facial expression recognition aims to accurately classify emotions expressed in an image into one of basic emotions, namely, happiness, surprise, sadness, hate, fear, anger, and neutral. In a conventional facial expression recognition method, a facial expression image is mainly recognized based on a manual feature or a shallow feature. Due to high similarity of between-category facial expressions, facial occlusion, and a change in a gesture, the conventional method is difficult to correctly recognize a category which the image belongs to. In recent years, many advanced methods based on an attention mechanism are provided for performing facial expression recognition. However, the methods still have the following problems:
[0004] 1 Because expressions of different emotions may be significantly similar, expressions of a same emotion may be significantly different. The images may only be distinguished from each other based on detailed features such as a degree of a wrinkle on a forehead, a size of an eye, and a closing degree of a comer of a mouth. However, in an existing method, a model is difficult to concern the important detailed features.
[0005] 2. At present, most facial expression recognition models use only a cross-entropy loss as a final loss function. However, the cross-entropy loss concerns only a difference between a predicted score and a correct label, ignoring a difference between a predicted score and another non-correct label. SUMMARY
[0006] In view of the above problems, the present disclosure provides a facial expression recognition method based on a multi-level and multi-scale attention mechanism. Firstly, a multi-level attention module is designed. The multi-level attention module is used to assign different weights to features extracted by a feature pre-extraction module at different stages based on importance. Secondly, a multi-scale attention module is designed. The multi-scale attention module is used to perform multi-scale fusion on a deep feature, to improve semantic information and a representation capability of the deep feature; finally, a restricted center loss is designed. The restricted center loss is used to increase a distance between a same category of features, and decrease a distance between different categories of features, to improve differentiation between the different categories of features. Limiting conditions are added, to limit an intra-category distance and a between-category distance, to prevent an excessively large loss value.
[0007] The technical solution provided in the present disclosure to resolve the technical problems is as follows:
[0008] step 1: dividing a plurality of public datasets into a training set, a test set, and a validation set, where the training set, the test set, and the validation set each include a plurality of images of seven basic expressions;
[0009] step 2: constructing a facial expression recognition model, where the facial expression recognition model includes a feature pre-extraction module, a multi-level attention module, and a multi-scale attention module;
[0010] step 3: detecting and performing face alignment on images that are in the training set, the test set, and the validation set and that are input by the facial expression recognition model, and cropping, to a fixed size, the images that are in the training set, the test set, and the validation set and that are input by the facial expression recognition model; and randomly extracting 10% images from the images that are in the training set and input by the facial expression recognition model, and performing image enhancement on the 10% images, to prevent the facial expression recognition model from overfitting, where image enhancement includes horizontal flip, random crop, random removal, and random rotation;
[0011] step 4: inputting the images that are in the training set and that are obtained after the face alignment and cropping in step 3 into the facial expression recognition model, training and optimizing a learnable parameter in the facial expression recognition model based on a cross-entropy loss and a restricted center loss until accuracy of the facial expression recognition model is unable to be improved, where a specific training process includes the following sub-steps:
[0012] step 4.1: inputting the images in step 3 into the feature pre-extraction module, where the feature pre-extraction module includes a total of n basic blocks, and a feature map Xt output by each basic block in the feature pre-extraction module is taken as a feature output by the feature pre-extraction module at each stage;
[0013] step 4.2: assigning, by a weight assignment module in the multi-level attention module, different weights to feature maps output by first n-1 basic blocks based on different importance, where in the weight assignment module, firstly, channel information of the feature maps Xt output by the first n-1 basic blocks is aggregated by the average pooling layer, to obtain an average pooling channel vector q^9 \ and secondly, dimensionality reduction is performed on the channel vector q^9 by a fully connected layer and an activation layer, to obtain the final weight ah where a calculation formula is as follows:
[0014] q™9 = AvgPool(Xi\
[0015] at = al(&U($fV0)w^w^ where
[0016] AvgPool represents an average pooling operation; of and aj respectively represent a ReLU activation function and a Sigmoid activation function, and and w- represent parameters of the fully connected layer, with a subscript representing corresponding to an ith basic block and a superscript representing which full connection or activation layer; and q“va and cq respectively represent the average pooling channel vector and a weight corresponding to a feature output by the ith basic block;
[0017] step 4.3: performing a flattening operation (expanding a multi-dimensional feature map into a one-dimensional feature map), and mapping, by a fully connected layer, the feature maps output by the first n-1 basic blocks to a same dimension, to obtain a feature map Zt, where a calculation formula is as follows:
[0018] Zi = f^X^Wt, where
[0019] f( * ) represents a flattening operation, wt parameter of the fully connected layer, Zt represents a vector obtained after a feature map output by the ith basic block is remodeled;
[0020] step 4.4: multiply a, and Zt and then conduct addition, to obtain an output Koutlof the multi-level attention module;
[0021] n represents a quantity of basic blocks, where in the facial expression recognition model of the method, the feature pre-extraction module includes four basic blocks;
[0022] step 4.5: because SE Block has an excellent channel aggregation capability, connecting an average pooling layer, the fully connected layer, a Sigmoid activation layer, and a rectified linear unit (ReLU) activation layer in sequence, to obtain SEBlock, and performing weighting on an output Xn of an nth basic block in the feature pre-extraction module on a channel dimension, where in the feature pre-extraction module, firstly, channel information of Xn output by the first n basic blocks is aggregated by the average pooling layer, to obtain the average pooling channel vector q^fl \ and secondly, dimensionality uplift and dimensionality reduction are performed in sequence on q1^9 by the fully connected layer and the activation layer, to obtain a vector IV of which a size is consistent with a size of q^9', and finally, multiplying W by Xn, to obtain S, and implementing weighting of Xn on the channel dimension;
[0023] step 4.6: performing, by a multi-scale fusion module in the multi-scale attention module, multi-scale fusion on the feature map S, where in the multi-scale fusion module, firstly, the feature map S passes through three null convolutional layers, a normalization layer, and the activation layer separately, to obtain a corresponding feature map Dj; and secondly, the feature map Dj is added and fused, to obtain the feature map D, where a calculation formula is as follows:
[0024] Dj = a^BNjQijQS;^),
[0025] D = Dj, where
[0026] aj represents the ReLU activation function, BNj( * ) represents a normalization operation, ’J) represents a null convolution operation, and j represents an expansion ratio;
[0027] step 4.7: performing the flattening operation, and mapping, by the fully connected layer, the feature map D to a dimension the same as Youtl, to obtain a final output Yout2 of the multi-scale attention module;
[0028] step 4.8: adding Youtl to Yout2, to obtain a final feature output Y, where a calculation formula is as follows:
[0029] Y = (\Y0Utl + p2Y0Ut2, where
[0030] and p2 represent hyper-parameters, representing contribution of an output of the multi-level attention module and an output of the multi-scale attention module to a final output feature of the facial expression recognition model; and
[0031] step 4.9: constructing a loss function, calculating a loss, and optimizing the facial expression recognition model, where to reduce a difference between a predicted score and a real label, a cross-entropy loss Lce is used in the present disclosure, where a calculation formula is as follows:
[0032] L„ = where
[0033] m represents a quantity of samples in each small batch mini-batch, k represents a quantity of categories, represents a label of a 5"’ category corresponding to a h"' sample, and Y^ represents a predicted score of the ^th category corresponding to an output of the hth sample, where
[0034] to decrease a distance between a same category of features, increase a distance between different categories of features, and limit a size of a loss value, a restricted center loss Llc is designed in the present disclosure, where a calculation formula is as follows:
[0035] Lic = ^111^1) + / 2^=1^=1---z..........7................, where
[0036] Yh represents the predicted score of a category corresponding to an output of the 11th sample, ch represents a center vector of the category corresponding to the hA sample, || H2 represents a square of an L2 paradigm, c9 represents a center vector corresponding to the ^th category, yt, y2, mj.and m2 represent hyper-parameters,y1 and y2 are used to adjust a magnitude of Lk, is used to restrict and limit an intra-category distance, m2 is used to restrict and limit a between-category distance, and <5 represents an extremely small value and is used to prevent a denominator from being 0; and
[0037] adding the loss function above to obtain a total of loss Ltotai; and
[0038] step 5: performing inference on the images that are in the test set and the validation set, and inputting the images that are in the training set and the test set and that are obtained after the face alignment and cropping in step 3 into the facial expression recognition model, to obtain a corresponding classification result, and performing facial expression recognition, where the result is one of the seven basic expressions, namely, hate, happiness, anger, fear, surprise, sadness, and neutral.
[0039] A network architecture in the present disclosure includes the feature pre-extraction module, the multi-level attention module, and the multi-scale attention module. The feature pre-extraction module uses a multi-stage feature extraction manner, so that the feature extracted by the facial expression recognition model not only includes low-level spatial information in a shallow network (the first n-1 basic blocks), but also includes high-level semantic information in a deep network (the nth basic block). Therefore, robustness of the feature extracted by the facial expression recognition model is improved. The multi-level attention module acts on a shallow block of the feature pre-extraction module. The multi-level attention module is used to assign different weights to features extracted by the shallow block of the feature pre-extraction module based on importance, to enable the facial expression recognition model to pay attention to important details. The multi-scale attention module acts on a deepest basic block of the feature pre-extraction module, to help the facial expression recognition model to better fuse multi-scale contextual information, thereby improving an expressive capability of the feature output by the facial expression recognition model. The restricted center loss is designed in the present disclosure. Through the restricted center loss, a center distance between different categories is increased while being penalized based on a center loss function. Therefore, an excessively large loss value is effectively prevented, the facial expression recognition model is better optimized, and facial expressions are accurately recognized. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] FIG. 1 is a flowchart of a facial expression recognition method based on a multi-level and multi-scale attention mechanism according to the present disclosure;
[0041] FIG. 2 is a schematic diagram of an overall network architecture according to an embodiment of the present disclosure; and
[0042] FIG. 3 is a diagram of attentional visualization of a multi-scale attention module and a multi-level attention module in a facial expression recognition method based on a multi-level and multi-scale attention mechanism on an original image. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0043] The present disclosure provides a facial expression recognition method based on a multi-level and multi-scale attention mechanism. As shown in FIG. 1, an overall network architecture includes a feature pre-extraction module, a multi-level attention module, and a multi-scale attention module. The feature pre-extraction module extracts an initial feature of an input image. Study shows that in a convolutional network, if a convolution is deeper, a receptive field is larger, and learned semantic features are richer. If a convolution is shallower, a receptive field is smaller, and learned geometrical features (such as a shape, a contour, and an edge) are richer. Therefore, the feature pre-extraction module extracts the feature in stages, to facilitate a model to obtain more comprehensive information. Then, the multi-level attention module further extracts a shallow feature from the feature pre-extraction module, assigns a different weight based on importance of the extracted feature, multiplies the obtained feature by a corresponding weight, to obtain a final output of the multi-level attention module. In addition, the multi-scale attention block performs multi-scale fusion on a feature output by a deepest feature of the feature pre-extraction module, to improve semantic information and a representation capacity of the deeper feature, and adds and fuses the output feature to and with the final output of the multi-level attention module, to obtain a final output of the model. Finally, a cross-entropy loss and a restricted center loss are used to calculate losses for an original label, a center vector of each category, and the final output of the model. This step is only used in a training process to optimize an overall parameter of a network, to enable the model to predict an accurate classification score in a subsequent inference.
[0044] The present disclosure is described in more detail below with reference to the accompanying drawings and specific implementations.
[0045] Specifically, the present disclosure provides a facial expression recognition method based on a multi-level and multi-scale attention mechanism. As shown in FIG. 1, the method mainly includes the following steps.
[0046] Step 1: Divide four public datasets, namely, Affect-Net, RAF-DB, FERPlus, and SFEW2.0, into a training set, a test set, and a validation set, where the training set, the test set, and the validation set each include images of seven basic expressions.
[0047] The RAF-DB is a dataset of a real world, and includes 30,000 facial expression images of thousands of people of different races and different ages. An occlusion range, a change in a gesture, and brightness of the image are out of control. Each image is independently annotated by about 40 annotators. The dataset divided into seven basic labels, including a total of 15,339 images (of which 12,271 images are used as the training set and 3,068 images are used as the test set). The FERPlus dataset is extended from a FER2013 dataset. The FER2013 dataset includes a large-scale real data collected by the Google search engine. In the FER2013 dataset, all the images have been cropped to 48 x 48 after face alignment is performed. However, because there are many incorrectly labeled images in the FER2013 dataset, the FERPlus dataset expands and relabels the FER2013 dataset. The FERPlus dataset includes a total of 35,887 images (of which 28,709 images are taken as the training set, 3,589 images are taken as the test set, and 3,589 images are taken as the validation set). The Affect-Net dataset is a largest dataset in all public datasets for facial expression recognition. The Affect-Net dataset is collected from three major search engines, including a total of 287,401 images (of which 28,3901 images are used as the training set and 3,500 images are used as the test set). The SFEW2.0 dataset is extracted from an AFEW dataset. The SFEW2.0 dataset covers images of unconstrained facial expressions, different changes in a gesture, ranges of different ages, and different facial resolutions, including a total of 1,766 images (of which 958 images are used as the training set, 372 images are used as the test set, and 436 images are used as the validation set).
[0048] Step 2: As shown in FIG. 2, construct a facial expression recognition model, where the facial expression recognition model includes a feature pre-extraction module, a multi-level attention module, and a multi-scale attention module.
[0049] Step 3: Detect and perform face alignment on images that are in the training set, the test set, and the validation set and that are input by the facial expression recognition model, and crop, to a fixed size, the images that are in the training set, the test set, and the validation set and that are input by the facial expression recognition model; and randomly extract 10% images from the images that are in the training set and input by the facial expression recognition model, and perform image enhancement on the 10% images. Image enhancement operations such as horizontal flip, random crop, random removal, and random rotation are used to prevent the facial expression recognition model from overfitting.
[0050] Step 4: Input the images that are in the training set and that are obtained after the face alignment and cropping in step 3 into the facial expression recognition model, and train and optimize a learnable parameter in the facial expression recognition model based on a cross-entropy loss and a restricted center loss, until accuracy of the facial expression recognition model is unable to be improved, where a specific training process includes the following sub-steps:
[0051] Step 4.1: Input the images obtained after the face alignment and cropping in step 3 into the feature pre-extraction module, where the feature pre-extraction module includes a ResNet-50. The ResNet-50 includes four basic blocks. Each basic block includes 3, 4, 6, and 3 Bottlenecks. Each Bottleneck includes one 3x3 convolution and two 1X1 convolutions. When an image is input into the feature pre-extraction module, a spatial scale of a feature map is halved and a quantity of channels is doubled while the image passes through each basic block. A feature map Xt output by each basic block in the feature pre-extraction module is used as a feature output by the feature pre-extraction module at each stage, where a calculation formula is as follows:
[0052] Xi = r(X;i; 0,), where
[0053] r(-) represents an operation convolution and an activation operation, 0( represents a parameter in each basic block in the feature pre-extraction module, and i represents which basic block, starting from 1.
[0054] Step 4.2: A weight assignment module assigns, based on different importance, different weights to feature maps output by first three basic blocks of the feature pre-extraction module. The module includes an average pooling layer, a fully connected layer, and an activation layer. In the facial expression recognition model, firstly, channel information of the feature maps Xt output by the first n-1 basic blocks is aggregated by the average pooling layer, to obtain an average pooling channel vector q""9. Secondly, dimensionality reduction is performed twice on the channel vector q®vg by the fully connected layer and the activation layer, to obtain a final weight , where a calculation formula is as follows:
[0055] q"Vd = AvgPool^Xi),
[0056] oq = aj((a)((ci"l'9)w°))wl)^ where
[0057] a represents a ReLU activation function, AvgPool represents an average pooling operation, of and of respectively represent the ReLU activation function and a Sigmoid activation function, uf and w- represent parameters of the fully connected layer, with a subscript representing corresponding to a Ith first basic block and a superscript representing which full connection or activation layer, and q""9 and at respectively represent the average pooling channel vector and a weight corresponding to the feature output by the ith basic block.
[0058] Step 4.3: Perform a flattening operation (expanding a multi-dimensional feature map into a one-dimensional feature map), and mapping, by a fully connected layer, the feature maps Xt output by the first n-1 basic blocks to a same dimension, to obtain a feature map Zt, where a calculation formula is as follows:
[0059] Zi = f (Xi)wi, where
[0060] f( * ) represents the flattening operation, wt represents the parameter of the fully connected layer, and Z, represents a vector obtained after a feature map output by the i‘h basic block.
[0061] Step 4.4: Multiply cq by a corresponding ZL and then conduct addition, to obtain an output Koutlof the multi-level attention module, where a calculation formula is as follows:
[0062] Youtl = ^=1 at Zb where
[0063] n represents a quantity of basic blocks. In the facial expression recognition model in the method, the feature pre-extraction module includes four basic blocks.
[0064] Step 4.5: Because SEBlock has an excellent channel aggregation capability, in the method, SEBlock is used to perform weighting on an output Xn of a fourth basic block of the feature pre-extraction module on a channel dimension. SEBlock is mainly formed by connecting an average pooling layer, a fully connected layer, a Sigmoid activation layer, and a rectified linear unit (ReLU) activation layer in sequence. In the feature pre-extraction module, the average pooling layer is used to aggregate channel information of an output Xn of an n,h basic block, to obtain the average pooling channel vector . In addition, dimensionality uplift is performed on q™9 by the fully connected layer and the ReLU activation layer, and dimensionality reduction is performed by the fully connected layer and the Sigmoid activation layer, to obtain a vector W of which a size is consistent with a size of q^9 ■ Finally, W is multiplied by Xn, to obtain S. This implements weighting on Xn on the channel dimension, where a calculation formula is as follows:
[0065] C" = AvgPool^,
[0066] W =
[0067] 5 = WXn,
[0068] Step 4.6: A multi-scale fusion module performs multi-scale fusion on the feature map S. The module includes three 3x3 null convolutional layers with different expansion ratios, a normalization layer, and an activation layer. In the multi-scale fusion module, the feature map S separately passes through the three different null convolutional layers, the normalization layer, and the activation layer. The expansion ratios of the three null convolutional layers are set to 1, 2, and 3, to obtain a corresponding feature map Dj. Then, the feature map D, is added and fused, to obtain the feature map D, where a calculation formula is as follows:
[0069] D, =
[0070] D =
[0071] (Jj represents the ReLU activation function, BNj( * ) represents a normalization operation, [ijQ-; j) represents a null convolution operation, and j represents an expansion ratio;
[0072] Step 4.7: Perform the flattening operation, and map, by the fully connected layer, the feature map D to a dimension the same as Youtl, to obtain a final output Yout2 of the multi-scale attention module, where a calculation formula is as follows:
[0073] Yout2 = f(D)wn,
[0074] Step 4.8: Add Youtl to Yout2, to obtain a final feature output Y, where a calculation formula is as follows:
[0075] Y = p1Y0Utl+p2Y0Ut2,
[0076] P^ and P2 represent hyper-parameters, representing contribution of an output of the multi-level attention module and an output of the multi-scale attention module to the final output feature of the facial expression recognition model.
[0077] Step 4.9: Construct a loss function, calculate a loss, and optimize the facial expression recognition model. To reduce the difference between a predicted score and a real label, a cross-entropy loss Lce is used in the present disclosure, where a calculation formula is as follows:
[0078] Lce = log (¾ where
[0079] m represents a quantity of samples in each small batch mini-batch, k represents a quantity of categories, b9h represents a label of a gth category corresponding to a h"' sample, and Y^ represents a predicted score of the ^th category corresponding to an output of the htb sample.
[0080] Because a conventional center loss considers only decrease in the distance between the same category of features, without considering increase the distance between different categories of features. In addition, a large loss value is easy to occur. Therefore, a restricted center loss LLc is provided in the present disclosure. Through the restricted center loss, a center distance between different categories is increased while being penalized based on a center loss. Limiting conditions are increased to limit an intra-category distance and a between-category distance, to prevent an excessively large loss value, where a calculation formula is as follows:
[0081] Llc = Y1^=1 max{HYh- ch\\22,a)1} + Y2Ylkg=1 Zk 1=1---- j max where
[0082] Yh represents the predicted score of a category corresponding the output of the h‘h sample, ch represents a center vector of the category corresponding to the sample, cg represents a center vector corresponding to a g& category, and ||-||| represents a square of an L2 paradigm. y15 y2, aq, and m2 represent hyper-parameters,Yi and Y2 are used to adjust a magnitude of Lk, a)1 is used to restrict and limit an intra-category distance, m2 is used to restrict and limit a between-category distance, and 8 represents an extremely small value and is used to prevent denominator from being 0.
[0083] The loss functions above are integrated together to obtain a total loss Ltotal, where a calculation formula is as follows:
[0084] Ltotai — Lce + Lic.
[0085] Step 5: Perform inference on the images that are in the test set and the validation set, and input the images that are in the training set and the test set into the facial expression recognition model, to obtain a corresponding classification result, where the result is one of the seven basic expressions, namely, hate, happiness, anger, fear, surprise, sadness, and neutral. 10
[0086] To prove effectiveness of the method, in the method, a heat map is used to visualize a region concerned by the facial expression recognition model. As shown in FIG. 3, a first row represents anger, a second row represents fear, a first column represents an original image, and a second column and a third column respectively represent attention visualizations of final outputs of the multi-scale attention module and the multi-level attention module relative original images. The figure clearly shows that the multi-scale attention module pays more attention to a global feature, while the multi-level attention module pays more attention to an important detail feature. Accuracy of facial expression recognition is effectively improved through mutual cooperation of the two modules.
[0087] Table 1 Results of comparative experiment between the method and existing advanced methods for RAF-DB dataset
[0088] Existing methods Accuracy (%) RAN 86.9 Ad-Corre 86.96 DACL 87.78 SCN 87.03 RUL 88.98 MViT 88.62 EAC 90.35 The present disclosure 90.42
[0089] Table 2 Results of comparative experiment between the method and existing advanced methods for Affect-Net dataset
[0090] Existing methods Accuracy (%) Distilled student 65.4 Ad-Corre 63.36 DACL 65.20 Emotion-GCN 66.46 DDA-Loss 62.34 MViT 64.57 EAC 65.32 The present disclosure 65.91
[0091] Table 3 Results of comparative experiment between the method and existing advanced ii methods for FERPlus dataset
[0092] Existing methods Accuracy (%) RAN 89.16 SCN 88.01 RUL 88.75 EAC 89.64 KTN 90.49 DMUE 88.64 The present disclosure 90.05
[0093] To prove excellence of the method, in the present disclosure, comparative experiments between the method and other most advanced methods are performed on the three datasets, namely, the RAF-DB, the Affect-Net, and the FERPlus. Results of the experiments are shown in Table 1, Table 2, and Table 3. It may be learned from the results of the experiments that the method has high recognition accuracy.
Claims
1. A facial expression recognition method based on a multi-level and multi-scale attention mechanism, comprising:step 1: dividing a plurality of public datasets into a training set, a test set, and a validation set;step 2: constructing a facial expression recognition model, wherein the facial expression recognition model comprises a feature pre-extraction module, a multi-level attention module, and a multi-scale attention module;step 3: detecting and performing face alignment on images that are in the training set, the test set, and the validation set and that are input by the facial expression recognition model, and cropping, to a fixed size, the images that are in the training set, the test set, and the validation set and that are input by the facial expression recognition model; andrandomly extracting 10% images from the images that are in the training set and input by the facial expression recognition model, and performing image enhancement on the 10% images;step 4: inputting the images that are in the training set and that are obtained after the face alignment and cropping in step 3 into the facial expression recognition model, training and optimizing a parameter in the facial expression recognition model based on a cross-entropy loss and a restricted center loss until accuracy of the facial expression recognition model is unable to be improved; andstep 5: inputting the images that are in the training set and the test set and that are obtained after the face alignment and cropping in step 3 into the facial expression recognition model, to obtain a corresponding classification result, and performing facial expression recognition.
2. The facial expression recognition method based on a multi-level and multi-scale attention mechanism according to claim 1, wherein in step 1, the training set, the test set, and the validation set each comprise a plurality of images of seven basic expressions; andthe seven basic expressions are hate, happiness, anger, fear, surprise, sadness, and neutral.
3. The facial expression recognition method based on a multi-level and multi-scale attention mechanism according to claim 1, wherein in step 3, the image enhancement comprises horizontal flip, random crop, random removal, and random rotation.
4. The facial expression recognition method based on a multi-level and multi-scale attention mechanism according to claim 1, wherein step 4 specifically comprises:step 4.1: inputting the images obtained after the face alignment and cropping in step 3 into thefeature pre-extraction module, wherein the feature pre-extraction module includes n basic blocks, and a feature map Xt output by each basic block in the feature pre-extraction module is taken as a feature output by the feature pre-extraction module at each stage;step 4.2: assigning, by a weight assignment module in the multi-level attention module, different weights a, to feature maps output by first n-1 basic blocks based on different importance;step 4.3: performing a flattening operation of expanding a multi-dimensional feature map into a one-dimensional feature map, and mapping, by a fully connected layer, the feature maps Xt output by the first n-1 basic blocks to a same dimension, to obtain a feature mapstep 4.4: multiplying cq and ZL and then conducting addition, to obtain a final output Kouti of the multi-level attention module;step 4.5: connecting an average pooling layer, the fully connected layer, a Sigmoid activation layer, and a rectified linear unit (ReLU) activation layer in sequence, to obtain SEBlock, and performing weighting on an output Xn of an nth basic block in the feature pre-extraction module on a channel dimension, to obtain a feature map S;step 4.6: performing, by a multi-scale fusion module in the multi-scale attention module, multi-scale fusion on the feature map 5, to obtain a feature map D;step 4.7: performing the flattening operation, and mapping, by the fully connected layer, the feature map D to a dimension the same as Youtl, to obtain a final output Yout2 of the multi-scale attention module;step 4.8: adding Youtl to Yout2, to obtain a final feature output Y, wherein a calculation formula is as follows:= PiYouti + [WotM, whereinand / 32 are hyper-parameters; andstep 4.9: constructing a loss function, calculating a loss, and optimizing the facial expression recognition model.
5. The facial expression recognition method based on a multi-level and multi-scale attention mechanism according to claim 4, wherein in the weight assignment module in step 4.2, firstly, channel information of the feature maps Xt output by the first n-1 basic blocks is aggregated by the average pooling layer, to obtain an average pooling channel vector q™9-, andsecondly, dimensionality reduction is performed on the average pooling channel vector q1™9 by the fully connected layer and the ReLU activation layer, to obtain the weight at.
6. The facial expression recognition method based on a multi-level and multi-scale attentionmechanism according to claim 5, wherein in the SE Block in step 4.5, firstly, channel information of the output Xn of the nth basic block is aggregated by the average pooling layer, to obtain the average pooling channel vector q^9',secondly, dimensionality uplift and dimensionality reduction are performed in sequence on q™9 by the fully connected layer and the activation layer, to obtain a vector W of which a size is consistent with a size of q^9', andfinally, W is multiplied by Xn, to obtain the feature map S.
7. The facial expression recognition method based on a multi-level and multi-scale attention mechanism according to claim 6, wherein in the multi-scale fusion module in step 4.6, firstly, the feature map S passes through three null convolutional layers, a normalization layer, and the activation layer separately, to obtain a corresponding feature map Dh andsecondly, the feature map Dj is added and fused, to obtain the feature map D.
8. The facial expression recognition method based on a multi-level and multi-scale attention mechanism according to claim 7, wherein the loss function for calculating a loss in step 4.9 is:(1) a cross-entropy loss Lce, wherein a calculation formula is as follows:= log (Y9h), whereinm represents a quantity of samples in each small batch, k represents a quantity of categories, b9 represents a label of a ,9 th category corresponding to a hth sample, and Y^ represents a predicted score of the gth category corresponding to an output of the hth sample;(2) a restricted center loss Lic, wherein a calculation formula is as follows:Lie = Ki “ Mi "1} + Y2lkQ=1li=i---n. „ 1 ,„2—whereinM j maz|||c«— c ||2, aM+SYh represents the predicted score of a category corresponding to an output of the hth sample, ch represents a center vector of the category corresponding to the hth sample, IHI2 represents a square of an L2 paradigm, c9 represents a center vector corresponding to the gih category, jq, y2, ay,and a>2 represent hyper-parameters, Yi and Y2 are used to adjust a magnitude of Llc, a)1 is used to restrict and limit an intra-category distance, a>2 is used to restrict and limit a between-category distance, and 6 represents an extremely small value; and(3) the cross-entropy loss Lce is added to the restricted center loss Ltc, to obtain a total loss Ltotal-16