Facial expression recognition method based on multi-level feature extraction and fusion in natural environment

Abstract

The invention provides a facial expression recognition method based on multi-level feature extraction and fusion in a natural environment. The method is characterized by comprising the following steps: (1) a feature extraction module: carrying out convolution calculation by using three different convolution kernel scales and dense blocks with DenseNet as a backbone network to generate facial expression feature maps (Feature Maps) at a low level, a middle level and a high level; different dense blocks are connected through transition layers; (2) a feature fusion module which performs global attention and local attention processing on the processed high-level feature map and low-level feature map after element addition calculation so as to generate a fused facial expression feature map; and (3) optimizing a strategy: adopting a strategy of combining a smooth label (Label Smoothing) and L2 regularization to avoid over-fitting.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer artificial intelligence, and particularly relates to a multi-level feature extraction and fusion method for facial expression recognition in a natural environment. BACKGROUND

[0002] Human emotional and emotional states can be externally manifested through facial expressions, which play an important role in daily social activities. With the development of technology, facial expression recognition (FER) technology has been applied to many fields such as mental health assessment and human-computer interaction systems. As one of the hotspots in the field of computer vision, facial expression recognition has attracted widespread attention from researchers.

[0003] However, facial expression recognition faces many challenges. In terms of image acquisition, factors such as angle, scale, and occlusion can affect the final recognition result. In addition, in practical applications, facial expression images often have uneven lighting and low quality interference factors, which also have a negative impact on the recognition result. Therefore, how to design a facial expression recognition model with better performance and stronger generalization ability has become the focus of FER research.

[0004] In recent years, with the widespread application of deep learning methods in computer vision tasks, facial expression recognition methods based on deep learning have gradually developed. These methods use deep learning models to automatically learn features from raw data, replacing traditional manual feature extraction methods, and can achieve better performance on different tasks and datasets. Although deep learning-based methods have achieved significant success in laboratory-collected datasets, they still face many challenges in recognizing unconstrained facial expressions in outdoor environments, as facial expression images are easily affected by interference factors such as weak light conditions and blurred images.

[0005] As the models based on the conventional convolutional neural network (CNN) gradually show weakness in the competition of expression recognition, some researchers intend to study other methods to perform FER. Most of the methods are based on the directions of data enhancement, intra-category common feature, attention mechanism, label noise, uncertainty, etc. to perform FER. [1] proposes a deep attention center loss (DACL) method which can estimate the attention weight related to the feature importance and realize the intra-class compactness and inter-class separation. [2] proposes a self-supervised joint learning method to realize robust and diverse facial representation. [3] proposes a student-teacher integration (STO) method for FER based on online distillation, which is different from the existing methods, and the method designs a random subnetwork to replace the multi-branch structure component, and it is worth noting that STO achieves considerable performance without introducing any additional model parameters. [4] proposes a directional attention pseudo simulation network to realize high-precision FER by fusing global and local facial information.

[0006] However, they ignore the influence of different feature extraction methods on the final FER. In the feature extraction stage, a lot of low-level feature information is lost, and pure high-level abstract features are not enough to recognize challenging facial expressions, for example, when the facial expressions are too similar. In addition, these methods rarely explore the influence of the features extracted from different convolution kernel sizes on the final FER. SUMMARY

[0007] In order to solve the above problems, the embodiment of the application provides a multi-level feature extraction and fusion method for facial expression recognition in a natural environment.

[0008] The embodiment of the application provides a feature extraction module, a feature enhancement module and an optimization strategy for facial expression recognition in a natural non-restricted environment, which comprises:

[0009] The feature extraction module is mainly composed of DenseNet dense blocks with three different convolution kernel scales (3x3, 5x5 and 7x7), and the purpose is to extract as many multi-level facial expression features (low-level to high-level features) as possible. The over-layer connection between each sub-block is used to reduce the feature dimension.

[0010] The DenseNet backbone network of the dense block comprises: batch normalization (Batch Normalization), ReLu activation and 1x1 convolution operation.

[0011] In addition, one dense block contains different numbers of dense connection layers, and the feature map size of each layer is the same, and the layers are connected in a dense connection manner. For each dense connection layer, its output contains all previous layers, which can be expressed in a mathematical formula as follows:

[0012]

[0013] where C is a connection operation; is the output of the lth dense connection layer; respectively represent the outputs of the 0th, 1st,..., l-1th layers.

[0014] The transition includes batch normalization, ReLu activation, 1x1 convolution and 2x2 pooling.

[0015] The feature fusion module has global attention and local attention operations, and the high-level feature map is calculated by 3x3 convolution, 2x scale upsampling and element addition of the low-level feature map. The data matrix is used as the input of the feature fusion module. After global attention and local attention processing, element multiplication calculation is performed with the input, and the output of the two data matrices is calculated by element addition to generate the fused facial expression feature map.

[0016] The input of the feature fusion module can be expressed in a mathematical formula as follows:

[0017] F = U p (H (3×3) (H Map ))+L Map (2)

[0018] where H (3×3) represents a convolution operation with a 3x3 convolution kernel; U p is an upsampling operation, and we use an upsampling operation with a scale factor of 2 in the method. Then, we use global average pooling and 1x1 convolution to generate global fusion and local fusion attention, respectively. The complete process can be summarized in the following formula:

[0019]

[0020] F out =GA(F)×F+LA(F)×F (6)

[0021] Where GA(F) and LA(F) represent the global fusion and local fusion weights, respectively. out This represents the output of the Feature Fusion Module (FFM). These represent two convolutional blocks with a kernel size of 3x3, four convolutional blocks with a kernel size of 3x3, and one convolutional block with a kernel size of 1x1, respectively. σ is a sigmoid function. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is an architecture diagram of the network framework in an embodiment of the present invention;

[0024] Figure 2 This is an overview diagram of the feature fusion module in an embodiment of the present invention;

[0025] Figure 3 This is an overview diagram of the dense layer in an embodiment of the present invention;

[0026] Figure 4 This is an overview diagram of the transition layer in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] For the feature extraction module, three different convolutional kernel scales (3x3, 5x5, 7x7) and dense blocks with DenseNet as the backbone network are used.

[0029] like Figure 1 The L shown represents the number of layers in each dense block, which are 24, 16, and 12 respectively.

[0030] like Figure 3 The dense layer shown consists of two convolutional layers with batch normalization (BN) and ReLU operations. K is the size of the convolutional kernel in each dense layer, which depends on the different dense blocks; K is 3, 5, and 7 respectively.

[0031] As Figure 4 The transition layer is composed of a 1x1 convolution and a 2x2 pooling operation.

[0032] The input of the feature fusion module is H (3×3) represents a convolution operation with a 3x3 convolution kernel; U p is an up-sampling operation, and we use an up-sampling operation with a scale factor of 2 in the method

[0033] The global attention (Global Attention) includes: 2 layers of convolution layers with a convolution kernel size of 3x3, 4 layers of convolution layers with a convolution kernel size of 3x3, an average pooling layer (Average Pooling) and a Sigmoid activation function.

[0034] The local attention (Local Attention) includes: 2 layers of convolution layers with a convolution kernel size of 3x3, 4 layers of convolution layers with a convolution kernel size of 3x3, 1 layer of convolution layers with a convolution kernel size of 1x1 and a Sigmoid activation function.

[0035] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit it; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

[0036] References

[0037] [1] Farzaneh A H, Qi X. Facial expression recognition in the wild via deep attentive center loss [C] / / Proceedings of the IEEE / CVF winter conference on applications of computer vision. 2021: 2402-2411.

[0038] [2] Shome D, Kar T. FedAffect: Few-shot federated learning for facial expression recognition [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 4168-4175.

[0039] [3] Zheng Z, Rasmussen C, Peng X. Student-Teacher Oneness: A Storage-efficient approach that improves facial expression recognition [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 4077-4086.

[0040] [4] Wang Z, Zeng F, Liu S, et al. OAENet: Oriented attention ensemble for accurate facial expression recognition [J]. Pattern Recognition, 2021, 112: 107694.

Claims

1. A multi-stage feature extraction and fusion method for facial expression recognition in a natural environment, characterized in that, The application relates to a face expression recognition method based on a multi-level feature fusion network. The feature extraction module uses three different convolution kernel scales (3x3, 5x5 and 7x7) and dense blocks of a DenseNet backbone network to learn to extract low, medium and high level face expression feature maps in the spatial dimension and the channel dimension. The feature fusion module has global attention and local attention operations, and the high-level feature map is input into the feature fusion module after 3x3 convolution, 2-scale up sampling and element addition calculation of the low-level feature map, and then the input is subjected to element multiplication calculation after global attention and local attention processing, and the output data matrix is subjected to element addition calculation to generate the fused face expression feature map. The optimization strategy adds label smoothing and L2 regularization in the loss function to optimize the parameters of the convolution layer, the pooling layer and the full connection layer. The DenseNet backbone network of the dense block comprises batch normalization (Batch Normalization), ReLu activation and 1x1 convolution operation.

2. The method according to claim 1, wherein the multi-level feature extraction and fusion based facial expression recognition in natural environment is characterized in that, The transitioner (Transition) comprises batch normalization (Batch Normalization), ReLu activation, 1x1 convolution and 2x2 pooling.

3. The method according to claim 1, wherein the multi-level feature extraction and fusion based facial expression recognition in natural environment is characterized in that, The global attention (Global Attention) operation comprises two layers of 3x3 convolution layers, four layers of 3x3 convolution layers, an average pooling layer (AveragePooling) and a Sigmoid activation function.

4. The method according to claim 1, wherein the multi-level feature extraction and fusion based facial expression recognition in natural environment is characterized in that, The local attention (Local Attention) operation comprises two layers of 3x3 convolution layers, four layers of 3x3 convolution layers, one layer of 1x1 convolution layer and a Sigmoid activation function.

5. The method for facial expression recognition based on multi-level feature extraction and fusion in natural environment according to claim 1, characterized in that, ​ 6. The method for facial expression recognition based on multi-level feature extraction and fusion in natural environment according to claim 1, characterized in that, ​ 7. The method for facial expression recognition based on multi-level feature extraction and fusion in natural environment according to claim 1, characterized in that, ​ 8. The method for facial expression recognition based on multi-level feature extraction and fusion in natural environment according to claim 1, characterized in that, ​ 9. The feature fusion module based on claim 1, which is a Feature Fusion Module (FEM) based on Global Attention and Local Attention operation, a new method to improve the representation ability of feature maps, which enhances the feature representation of feature maps in channel and spatial dimensions respectively, and can be embedded into most convolutional neural networks to improve their performance.

Images