Facial expression recognition method and computer readable medium

By adopting a hybrid visual Transformer model of multi-scale feature fusion and block removal in facial expression recognition, combined with the improved attention module and PatchDropping module, the problem of insufficient recognition accuracy of CNN method in resource-constrained environments is solved, and the CNN method is realized Efficient facial expression recognition, especially in the RAF-DB and FER2013 datasets.

CN118711233BActive Publication Date: 2024-11-22JILIN ZIWEN TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410818719.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-24
Publication Date
2024-11-22
Estimated Expiration
2044-06-24

AI Technical Summary

Technical Problem

The existing CNN-based facial expression recognition method relies heavily on local spatial characteristics, lacks global information, and is difficult to balance the computational complexity and recognition accuracy, especially in resource-constrained environments.

Method used

Using a hybrid visual Transformer model based on multi-scale feature fusion and block removal, the improved efficient multi-scale attention module and PatchDropping module combines MobileNet V2 and MobileViTblock modules to capture multi-scale expression features and simulate the non-uniformity of human visual attention. distributed.

Benefits of technology

The feature extraction capability and recognition accuracy were significantly improved, and the calculation cost was reduced. The model size was only 3.64MB, and the accuracy rates on the RAF-DB and FER2013 data sets reached 86.51% and 69.84%, respectively.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118711233B_ABST
    Figure CN118711233B_ABST
Patent Text Reader

Abstract

The present invention discloses a facial expression recognition method and a computer-readable medium, which belongs to the field of facial expression recognition technology. The present invention captures multi-scale expression features through an improved efficient multi-scale attention module, realizes richer feature integration, enhances the network's perception of key expression areas, and improves feature extraction capabilities. At the same time, in order to further improve the performance of the model, the present invention designs a PatchDropping module, which aims to imitate the human visual system's attention allocation mechanism for local features, guide the network to focus on the most distinguishing features, reduce the impact of irrelevant features, and intuitively reduce the computational cost.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of facial expression recognition, and in particular relates to a facial expression recognition method and a computer-readable medium. Background Art

[0002] Facial expressions contain rich emotional information, and expressions are an important way for humans to express emotions and conduct social activities. Facial expression recognition plays a vital role in multiple human-computer interaction systems such as intelligent question answering, classroom teaching, intelligent medical care, criminal lie detection, and fatigue detection. It is the most direct and effective way for computers to obtain human emotions and has become an indispensable part of human-computer interaction. Therefore, people have invested a lot of energy in promoting the development of facial expression recognition (FER).

[0003] The application of convolutional neural networks (CNNs) has made great progress in the field of facial expression recognition (FER). However, due to problems such as occlusion, illumination, and head posture changes, facial expression recognition in real environments is extremely challenging. At the same time, pure CNN-based methods rely heavily on spatial local characteristics, lack global information, and have difficulty balancing the relationship between computational complexity and recognition accuracy. Deep network models significantly improve the depth of feature extraction and the accuracy of expression recognition by increasing the number of layers and parameters, but this improvement comes at the cost of significantly increasing computing resource consumption and model complexity. In practical applications, especially in resource-constrained environments such as mobile devices or embedded systems, this method of increasing recognition accuracy by increasing network depth is not practical. CNN-based models are still insufficient in terms of FER. Therefore, it is of great significance to conduct research on algorithms based on visual Transformers. Summary of the invention

[0004] The purpose of the present invention is to propose a facial expression recognition method and computer-readable medium based on a hybrid visual Transformer with multi-scale feature fusion and block removal, in view of the fact that facial expression recognition based on a pure CNN method is heavily dependent on spatial local characteristics, lacks global information and is difficult to balance the relationship between computational complexity and recognition accuracy.

[0005] To achieve the above purpose, the technical solution provided by the present invention is as follows:

[0006] In a first aspect, a method for recognizing facial expressions is provided, comprising the following steps:

[0007] Step S1: construct a training set, a validation set, and a test set based on a public facial expression dataset;

[0008] Step S2: preprocessing the constructed training set, validation set and test set to obtain face images that meet the model requirements;

[0009] Each piece of data in the constructed training set, validation set, and test set is given in the form of a data pair, including an RGB face image to be recognized as input data and the labeled expression category as the true value.

[0010] The preprocessing process includes face image extraction and enhancement operations performed on the extracted face images. The enhancement operations include flipping, rotation, scaling, pixel translation, random cropping, adding noise pixels, adjusting brightness and distorting image operations.

[0011] Step S3, constructing an expression recognition network model based on a hybrid visual Transformer with multi-scale feature fusion and block removal. The model network: the initial layer is a 3×3 standard convolution layer, followed by a feature extraction network, and finally the final recognition prediction is obtained through a global pooling layer and a fully connected layer;

[0012] Step S4, using the constructed training set to supervise the expression recognition network model based on the hybrid visual Transformer with multi-scale feature fusion and block removal until the network converges to the optimal performance;

[0013] Step S5, testing the converged expression recognition network model based on the hybrid vision Transformer with multi-scale feature fusion and block removal on the constructed test set, and evaluating the performance of the network model according to the recognition accuracy and the recognition confusion matrix;

[0014] In step S3, the feature extraction network is provided with a first MobileNet V2 module, a second MobileNet V2 module, a third MobileNet V2 module, a fourth MobileNet V2 module, a first MobileViTblock module, a fifth MobileNetV2 module, a second MobileViTblock module, a sixth MobileNetV2 module and a third MobileViT block module in sequence according to the data processing order, and the network structures of the first MobileNet V2 module, the second MobileNetV2 module, the third MobileNetV2 module, the fourth MobileNetV2 module, the fifth MobileNet V2 module and the sixth MobileNetV2 module are consistent, and the network structures of the first MobileViT block module, the second MobileViTblock module and the third MobileViTblock module are consistent; it should be understood that the terms "first", "second" and "third" are only used for descriptive purposes, and the features defined as "first", "second", "third", "fourth", "fifth" and "sixth" do not indicate any order, quantity or importance, but are only used to distinguish different components.

[0015] The processing process of the feature map in the MobileNetV2 module is as follows: the feature map input into the MobileNetV2 module is first expanded by point-by-point convolution; then, each channel is processed independently using depth-wise separable convolution to encode spatial features; then, one-dimensional convolution is applied to reduce the dimension of the feature map; after that, the feature map is input into the improved efficient multi-scale attention module to enhance the spatial semantic features; finally, the optimized features are input into the inverted residual module, and the low-level features are retained through shortcut connections. The improved efficient multi-scale attention module is at the tail of the existing efficient multi-scale attention module EMA, and a value representing the overall level of the channel is obtained by calculating the spatial average value of the feature map on each channel; next, for each pixel position in each channel, the square of the difference between its value and the corresponding channel mean is calculated to measure the degree of deviation of each pixel value from the channel mean;

[0016] The processing process of the feature map in the MobileViTblock module is as follows: the feature map in the MobileViTblock module is input, and the local features of the feature map are first extracted through a 3×3 depth-separable convolution, and then the number of channels of the feature map is adjusted from C to d through a 1×1 convolution. Then, the global features are extracted through the Transformer module with the Unfold-TPD-Fold mechanism and the number of channels of the feature map is adjusted back to C using a 1×1 convolution. Finally, the shortcut residual branch is used to splice the feature map with the original input map along the channel direction and these features are fused through a 3×3 convolution to obtain the final global feature output;

[0017] The process of extracting global features through the Transformer module with the Unfold-TPD-Fold mechanism is as follows: first, the feature map is expanded into multiple local windows through the Unfold operation, each window contains local spatial information; then, the expanded local window is used as input, and the global feature is modeled through TPD; finally, the processed feature map is folded back to the original spatial dimension through the Fold operation; TPD is a PatchDropping module embedded in the existing Transformer, and the feature map input into TPD is first normalized and then enters the multi-head self-attention mechanism, which calculates self-attention through multiple attention heads in parallel, and each head has independent parameters; the specific input is mapped to the query Query, key Key and value Value vector respectively through three linear transformations, and the linear change process is Q=XW Q , K = XW K 、V=XW V, where X represents the input feature map, Q, K and V represent the query, key and value vectors respectively, and W Q , W K and W V They are the weight matrices corresponding to the query, key, and value vectors respectively; the dot product between the query and the key is used to calculate the attention weight, which is then normalized by the softmax function; this weight is applied to the value vector to get the output of each head; the outputs of all heads are concatenated together and the final output of the multi-head self-attention is obtained through a linear transformation. The linear transformation process is:

[0018] MultiHead(Q,K,V)=Concat(head1,...,head i ,...,head h )Wo, its

[0019] middle, head i =Attention(QW Qi ,KW Ki ,VW Vi ), Wo is the linear transformation matrix, and the final output is obtained; then the patch is randomly discarded through the PatchDropping module, and then the residual connection is applied; the data after the PatchDropping module is input into the MLP through the Norm layer for nonlinear transformation and enhancement, and then the residual connection is applied again.

[0020] In a second aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed, the computer performs the steps of the above-mentioned facial expression recognition method.

[0021] In summary, compared with the prior art, the present invention captures multi-scale expression features through an improved efficient multi-scale attention module, achieves richer feature integration, enhances the network's perception of key expression areas, and improves feature extraction capabilities. At the same time, in order to further improve the performance of the model, the present invention designs a PatchDropping module. The core idea of ​​the PatchDropping module is to simulate the non-uniform distribution characteristics of human visual attention, and to force the model to learn more robust and discriminative feature representations by randomly discarding certain local areas (i.e., patches) in the feature map. Specifically, this design aims to improve model performance in the following ways: first, by randomly dropping patches, the model is encouraged to explore different areas in the feature map, thereby enhancing sensitivity to local features; second, the model is exposed to different combinations of patches during training, which helps to learn more generalized feature representations and reduce dependence on specific features; third, PatchDropping encourages the model to focus on both large-scale and small-scale features, because randomly dropping patches may cause the model to extract more information from the remaining patches, thereby promoting multi-scale feature learning; finally, the synergy with the multi-head self-attention mechanism (MSA) enables the model to maintain its understanding of the entire feature map and strong expression recognition capabilities even when local information is lost, aiming to imitate the human visual system's attention allocation mechanism for local features, guide the network to focus on the most discriminative features, reduce the influence of irrelevant features, and intuitively reduce the computational cost. After a large number of experiments, it is shown that the method of the present invention is significantly better than other methods, and with a model size of only 3.64MB, the accuracy rate is as high as 86.51% in the RAF-DB dataset and close to 70% in the FER2013 dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of a method for recognizing facial expressions in an embodiment of the present invention;

[0023] Figure 2 This is a structural diagram of the expression recognition network model in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the following is a detailed description of the present invention in combination with the embodiments of the present invention. Figure 1 and Figure 2 , the technical solution in the present invention is clearly and completely described. Obviously, the present invention is not limited by the following embodiments, and the specific implementation method can be determined according to the technical solution of the present invention and the actual situation. In order to avoid confusing the essence of the present invention, the known methods, processes, and procedures are not described in detail.

[0025] This embodiment proposes a lightweight and efficient method for facial expression recognition. The network model in the present invention uses a method combining local features and global features to extract features from facial expression images, and embeds an improved attention mechanism in the network model to focus on the detailed features of the expression. The network model in the present invention has good recognition performance after training, and has a higher recognition rate compared with the existing algorithm. The experimental results illustrate the accuracy of the facial expression recognition of the present invention.

[0026] MobileNet V2 is a lightweight convolutional neural network suitable for mobile and embedded devices with high computational efficiency and performance. The design of MobileNet V2 helps to extract low-level features of the input image, such as edges, textures, etc. These features are very important for high-level feature learning in subsequent stages. The feature map first passes through three serially connected MobileNet V2 modules, and the output is a series of feature maps processed by convolution and activation functions. Each module gradually increases the level of abstraction of the features. The next three stages involve alternating arrangements of MobileNet V2 modules and MobileViT block modules. MobileViT block maps the input features into a higher-level semantic space through a self-attention mechanism and a multi-layer perceptron (MLP). This transformation helps the model learn more complex and abstract image features. The purpose of alternating the MobileNet V2 module and the MobileViT block module is to gradually introduce the advantageous features of ViT into the entire network. The MobileNet V2 module provides the initial low-level and mid-level features, while the MobileViT block module enhances the relationship modeling capability between features through the self-attention mechanism, which helps to capture the global image semantics more effectively. The improved efficient multi-scale attention module is at the tail of the existing efficient multi-scale attention module EMA. By calculating the spatial average of the feature map in each channel, a value representing the overall level of the channel is obtained. Next, for each pixel position in each channel, the square of the difference between its value and the corresponding channel mean is calculated to measure the deviation of each pixel value from the channel mean.

[0027] The overall architecture of the model proposed in this invention is as follows Figure 2As shown, the input face image is first downsampled by 3×3 convolution to obtain an image pixel size of 128×128. Then the image enters the MobileNet V2 network structure for preliminary extraction of facial expression features. The specific process of preliminary extraction of facial expression features in the MobileNet V2 network structure is as follows: First, the input feature map is expanded by point-by-point convolution (1×1 convolution) to expand the number of channels from a low-dimensional feature map to a high-dimensional feature map, and then normalized by the BN layer to make its mean close to 0 and its variance close to 1, accelerating the convergence of the model. Then the ReLU6 activation function is passed. Next, the high-dimensional feature map is subjected to a depth-separable convolution (3×3 convolution), and convolution operations are performed independently on each channel to capture spatial information, keeping the number of channels unchanged, and also passing through the BN layer and the ReLU6 activation function. Subsequently, the high-dimensional feature map is compressed back to a low dimension by point-by-point convolution (1×1 convolution). Finally, the improved efficient multi-scale attention module proposed in the present invention is used. In this process, if the number of channels of the input and output is the same, a skip connection is performed, and the input feature map is directly added to the output feature map element by element to achieve residual connection. After passing through four MobileNetV2 network structures in succession, the pixel size of the face image becomes 32×32. Then the image enters the MobileViTblock network structure for deep facial expression feature extraction. In the MobileViTblock network structure, the feature map is first passed through a depthwise separable convolution layer (DwiseConv 3x3) with a convolution kernel size of 3×3 for local feature modeling, and then the number of channels is further adjusted through a convolution layer with a convolution kernel size of 1×1. Then, the global feature modeling is performed through the Unfold->TPD->Fold structure. In this process, the feature map is first expanded into multiple local windows through the Unfold operation, each window contains local spatial information. Then, these expanded local windows are used as input and passed through multiple Transformer layers for global feature modeling. Finally, the processed feature map is folded back to the original spatial dimension through the Fold operation. Then, a convolution layer with a convolution kernel size of 1×1 is used to adjust the number of channels back to the original size. Next, the shortcut branch is used to concat with the original input feature map (concatenated along the channel direction). Finally, a convolution layer with a convolution kernel size of 3×3 is used to fuse the features and obtain the output feature map. The feature map then passes through a network structure consisting of two MobileNet V2 network structures and a MobileViTblock network structure, and finally passes through a global pooling layer and a linear layer for the final classification output.

[0028] The facial expression recognition method comprises the following steps:

[0029] Step S1: construct a training set, a validation set and a test set based on a public facial expression dataset; each piece of data in the constructed training set, validation set and test set is given in the form of a data pair, including an RGB face image to be recognized as input data and annotated expression categories as true values.

[0030] Step S2: Each piece of data in the constructed training set, validation set, and test set needs to be preprocessed before being input into the model. First, the face contour in the input RGB face image to be recognized is extracted, and the pixel size of the obtained image is changed to 224×224;

[0031] Step S3: Construct a facial expression recognition network model based on a hybrid visual Transformer with multi-scale feature fusion and block removal; use the constructed training set to supervise the network model until the network converges to the optimal performance, and finally use the converged network model to predict the test set to obtain the expression recognition result.

[0032] In this example,

[0033] ① The facial expression recognition network model based on the hybrid vision Transformer of multi-scale feature fusion and block removal includes:

[0034] a) Local feature extraction: Its input is the preprocessed face image. First, the dimension of the input tensor is expanded through point-by-point convolution, and then each channel is processed independently using deep convolution to encode spatial features. Next, one-dimensional convolution is applied again to reduce the dimension of the feature map to reduce the computational burden. Subsequently, the feature map is input into the improved efficient multi-scale attention module (i.e., improved EMA) to enhance the distribution of spatial semantic features. Finally, the optimized features are input into the inverted residual module, which retains low-level features and promotes information flow through shortcut connections.

[0035] b) Global feature extraction: First, the feature map is used to extract local features of the image through a 3×3 convolution, and then the number of channels of the feature map is adjusted from C to d through a 1×1 convolution. Then, the global features are extracted through the Transformer module with the "Unfold-TPD-Fold" mechanism and the number of channels of the feature map is adjusted back to C using a 1×1 convolution. Finally, the feature map is concatenated with the original input map along the channel direction with the help of the shortcut residual branch, and these features are fused through a 3×3 convolution to obtain the final global feature output.

[0036] ② After the face image undergoes a 3×3 standard convolution, the pixel size becomes 128×128, and then enters the feature extraction network and undergoes three MV2 processes to obtain a facial expression feature map.

[0037] ③ Extract local features from the facial expression feature map obtained in step ② to obtain a facial expression local feature matrix. At this time, the pixel size of the feature map is 64×64.

[0038] ④ The global features of the obtained facial expression features are further extracted, and the facial expression features of pixel size 8×8 are obtained after three times of networks composed of MV2 and MobileViTblock alternately.

[0039] ⑤Finally, the final recognition prediction is obtained through the global pooling layer and the fully connected layer.

[0040] Step S4: Test the converged network model on the constructed test set, calculate the average recognition accuracy based on the recognition results and true labels, and evaluate the network performance.

[0041] In order to verify that the facial expression recognition method based on the hybrid visual Transformer of multi-scale feature fusion and block removal proposed by the present invention can achieve significant recognition effect, the recognition accuracy of the model is tested using the RAF-DB dataset and the FER2013 dataset, and the recognition accuracy is 86.51% and 69.84% respectively, and the model size is only 3.64MB. This verifies the effectiveness of the facial expression recognition method based on the hybrid visual Transformer of multi-scale feature fusion and block removal proposed by the present invention.

Claims

1. A method for recognizing facial expressions, characterized in that: The following steps are involved: Step S1: construct a training set, a validation set, and a test set based on a public facial expression dataset; Step S2: preprocessing the constructed training set, validation set and test set to obtain face images that meet the model requirements; Step S3, constructing an expression recognition network model based on a hybrid visual Transformer with multi-scale feature fusion and block removal. The model network: the initial layer is a 3×3 standard convolution layer, followed by a feature extraction network, and finally the final recognition prediction is obtained through a global pooling layer and a fully connected layer; Step S4, using the constructed training set to supervise the expression recognition network model based on the hybrid visual Transformer with multi-scale feature fusion and block removal until the network converges to the optimal performance; Step S5, testing the converged expression recognition network model based on the hybrid vision Transformer with multi-scale feature fusion and block removal on the constructed test set, and evaluating the performance of the network model according to the recognition accuracy and the recognition confusion matrix; In step S3, the feature extraction network is provided with a first MobileNet V2 module, a second MobileNet V2 module, a third MobileNet V2 module, a fourth MobileNet V2 module, a first MobileViTblock module, a fifth MobileNetV2 module, a second MobileViTblock module, a sixth MobileNetV2 module and a third MobileViT block module in sequence according to a data processing order, and the network structures of the first MobileNet V2 module, the second MobileNetV2 module, the third MobileNetV2 module, the fourth MobileNetV2 module, the fifth MobileNet V2 module and the sixth MobileNetV2 module are consistent, and the network structures of the first MobileViT block module, the second MobileViTblock module and the third MobileViTblock module are consistent; The processing process of feature maps in the MobileNetV2 module is as follows: the feature map input to the MobileNetV2 module is first expanded by point-by-point convolution; then, each channel is processed independently using depth-wise separable convolution to encode spatial features; Then, a one-dimensional convolution is applied to reduce the dimension of the feature map; after that, the feature map is input into the improved efficient multi-scale attention module to enhance the spatial semantic features; finally, the optimized features are input into the inverted residual module, and the low-level features are retained through shortcut connections. The improved efficient multi-scale attention module is at the tail of the existing efficient multi-scale attention module EMA, and a value representing the overall level of the channel is obtained by calculating the spatial average value of the feature map in each channel; next, for each pixel position in each channel, the square of the difference between its value and the corresponding channel mean is calculated to measure the degree of deviation of each pixel value from the channel mean; The processing process of the feature map in the MobileViTblock module is as follows: the feature map in the MobileViTblock module is input, and the local features of the feature map are first extracted through a 3×3 depth-separable convolution, and then the number of channels of the feature map is adjusted from C to d through a 1×1 convolution. Then, the global features are extracted through the Transformer module with the Unfold-TPD-Fold mechanism and the number of channels of the feature map is adjusted back to C using a 1×1 convolution. Finally, the shortcut residual branch is used to splice the feature map with the original input map along the channel direction and these features are fused through a 3×3 convolution to obtain the final global feature output; The process of extracting global features through the Transformer module with the Unfold-TPD-Fold mechanism is as follows: first, the feature map is expanded into multiple local windows through the Unfold operation, and each window contains local spatial information; then, the expanded local windows are used as input to model the global features through TPD; finally, the processed feature map is folded back to the original spatial dimension through the Fold operation; TPD embeds the PatchDropping module in the existing Transformer. The feature map is input into TPD and first normalized before entering the multi-head self-attention mechanism. This mechanism calculates self-attention in parallel through multiple attention heads, and each head has independent parameters. The specific input is mapped to the query Query, key Key and value vector through three linear transformations. The linear change process is Q = XW Q , K = XW K 、V=XW V , where X represents the input feature map, Q, K and V represent the query, key and value vectors respectively, and W Q , W K and W V are the weight matrices corresponding to the query, key, and value vectors respectively; the dot product between the query and the key is used to calculate the attention weight, which is then normalized by the softmax function; this weight is applied to the value vector to obtain the output of each head; the outputs of all heads are concatenated together and the final output of the multi-head self-attention is obtained through a linear transformation. The linear transformation process is: MultiHead(Q,K,V)=Concat(head1,...,head i ,...,head h ) I, among them, Wo is the linear transformation matrix, and the final output is obtained; then the patch is randomly discarded through the Patch Dropping module, and then the residual connection is applied; the data after the PatchDropping module is input into the MLP through the Norm layer for nonlinear transformation and enhancement, and then the residual connection is applied again.

2. The facial expression recognition method according to claim 1, characterized in that: Each piece of data in the constructed training set, validation set, and test set is given in the form of a data pair, including an RGB face image to be recognized as input data and the labeled expression category as the true value.

3. The facial expression recognition method according to claim 1, characterized in that: The preprocessing process includes face image extraction and enhancement operations performed on the extracted face images. The enhancement operations include flipping, rotation, scaling, pixel translation, random cropping, adding noise pixels, adjusting brightness and distorting image operations.

4. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed, the computer executes the steps of the facial expression recognition method described in any one of claims 1-3.