Facial expression recognition method and device based on deep capsule network

Abstract

This invention relates to a method and apparatus for facial expression recognition based on deep capsule networks. The method includes: fusing features extracted from an original image of a face and a Local Binary Pattern (LBP) image of the original image; weighting the fused features into features extracted from the original image based on an attention mechanism; and inputting the weighted features into a capsule network to obtain a facial expression classification. The technical solution provided by this invention achieves higher accuracy in facial expression classification than currently available facial expression recognition networks, demonstrating superiority and robustness in facial expression recognition.

Description

Technical Field

[0001] The present invention relates to the field of facial expression recognition, and in particular to a facial expression recognition method and device based on a deep capsule network. Background Art

[0002] Facial expressions are one of the most direct signals that people use to express their inner emotions in daily communication. Facial expression recognition (FER) is of great significance in fields related to facial expressions, such as autonomous driving, human-computer interaction, and medical treatment, and has gradually become an increasingly important research direction.

[0003] Traditional hand-designed features and shallow learning features can no longer adapt well to the interference of various practical factors, such as lighting conversion, different head postures and facial occlusion. With the in-depth learning and development of various image classification tasks, convolutional neural networks (CNNs) have gradually been used to deal with the diversity of FER and have made some progress, but there are still shortcomings such as long training time and low recognition rate under complex backgrounds. Summary of the invention

[0004] In order to solve the problem that convolutional neural networks are not capable of expressing facial expressions well, resulting in low recognition accuracy, the present invention provides a facial expression recognition method and device based on a deep capsule network.

[0005] According to a first aspect of an embodiment of the present invention, a method for facial expression recognition based on a deep capsule network is provided, comprising:

[0006] fusing features extracted from an original image of the face and a local binary pattern (LBP) image of the original image;

[0007] Based on an attention mechanism, weighting the fused features to the features extracted from the original image;

[0008] The weighted features are input into the capsule network to obtain the facial expression classification.

[0009] Further, the feature fusion extracted from the original facial image and the local binary pattern LBP image of the original image specifically includes:

[0010] The features of the original facial image are extracted through the first convolutional neural network and pooled to obtain an input feature map;

[0011] Extracting features of the LBP image of the original image through the first convolutional neural network and performing pooling processing to obtain a first LBP feature map;

[0012] The input feature map and the first LBP feature map are fused through a second convolutional neural network to obtain a second LBP feature map;

[0013] Wherein, the first convolutional neural network and the second convolutional neural network are both composed of three groups of convolution kernels, normalization layers and activation functions.

[0014] Furthermore, based on the attention mechanism, the fused features are weighted to the features extracted from the original image, specifically including:

[0015] The second LBP feature map is processed by an activation function and then multiplied by the input feature map.

[0016] Further, the processing of the second LBP feature map through an activation function specifically includes:

[0017] The second LBP feature is processed according to the following formula:

[0018] F m =σ(Wf m +b)

[0019] In the formula, f m is the second LBP feature map, W and b are the weight and bias of the convolutional layer respectively, and σ represents the activation function Softmax. The purpose is to assign different weights within (0,1) to the second LBP feature map so that the network can distinguish the importance of different features.

[0020] Furthermore, the expression of the loss function L of the capsule network is as follows:

[0021] L=L c +L e

[0022] Among them, L c represents the loss of category c, and its expression is as follows:

[0023] L c =T c max(0,m + -||v c ||) 2 +λ(1-T c )max(0,||v c ||-m - ) 2

[0024] Where, T c Represents the judgment index, if category c exists, it is 1, otherwise it is 0;

[0025]

[0026] In the formula, α represents a threshold, θ represents the maximum domain of the cosine function in the formula, and m represents L e The maximum value of L r For the reconstruction loss, the mean square loss function is used, L r The expression is as follows:

[0027] L r =(x r -x) 2

[0028] In the formula, x r and x represent the reconstructed image and original image output by the capsule network, respectively.

[0029] According to a second aspect of an embodiment of the present invention, a facial expression recognition device based on a deep capsule network is provided, comprising:

[0030] A feature fusion module, used for fusing features extracted from an original facial image and a local binary pattern (LBP) image of the original image;

[0031] An attention module, for weighting the fused features to the features extracted from the original image based on an attention mechanism;

[0032] The capsule network module is used to input the weighted features into the capsule network to obtain facial expression classification.

[0033] Furthermore, the feature fusion module specifically includes:

[0034] A first feature extraction unit, used for extracting features of an original facial image through a first convolutional neural network and performing pooling processing to obtain an input feature map;

[0035] A second feature extraction unit, configured to extract features of the LBP image of the original image through the first convolutional neural network and perform pooling processing to obtain a first LBP feature map;

[0036] A feature fusion unit, used for fusing the input feature map and the first LBP feature map through a second convolutional neural network to obtain a second LBP feature map;

[0037] Wherein, the first convolutional neural network and the second convolutional neural network are both composed of three groups of convolution kernels, normalization layers and activation functions.

[0038] Furthermore, the attention module is specifically used to multiply the second LBP feature map with the input feature map after being processed by an activation function.

[0039] According to a third aspect of an embodiment of the present invention, a terminal device is provided, including:

[0040] Processor; and

[0041] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.

[0042] According to a fourth aspect of an embodiment of the present invention, there is provided a non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to execute the method as described above.

[0043] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:

[0044] The embodiment of the present invention proposes a new deep capsule network (Fusion Deep Capusle-Network, FDC) model that is a fusion of a multi-layer convolutional network and a capsule network. First, LBP is used in a shallow network to extract image texture features and combined with an attention mechanism to capture important information about facial features. Secondly, the reconstruction loss function of the capsule network is improved to solve the problem that the reconstruction loss in the capsule network loss function tends to have a certain inhibitory effect on the overall loss, so that the network converges faster and achieves better performance. Finally. Tests were conducted on three public benchmark datasets KDEF, RaFD, and CK+. The experimental results show that the classification accuracy of the proposed network is better than the currently disclosed facial expression recognition network, with the highest accuracy rates reaching 92.54%, 97.93%, and 98.57%, respectively, proving that the proposed deep capsule network (FDC) model has certain superiority and robustness for facial expression recognition.

[0045] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present invention.

[0047] Figure 1 is a flowchart of a method for facial expression recognition based on a deep capsule network according to an exemplary embodiment of the present invention;

[0048] Figure 2 It is a feature fusion structure diagram;

[0049] Figure 3 This is the structure diagram of the LBP attention module;

[0050] Figure 4 It is the global structure of the feature fusion module;

[0051] Figure 5 It is the structure diagram of capsule network;

[0052] Figure 6 Examples of facial expressions on the CK+ dataset, KDEF dataset, and RaFD dataset under experimental conditions;

[0053] Figure 7 A diagram of the FDC network structure provided by an embodiment of the present invention;

[0054] Figure 8 The performance of the FDC network and other networks provided by the embodiments of the present invention on the data set;

[0055] Fig. 9 (a)(b)(c) are the confusion matrices of KDEF, RaFD and CK+ datasets respectively;

[0056] Fig.10 It is the recognition accuracy of various expressions in the dataset under ablation test. DETAILED DESCRIPTION

[0057] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0058] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms of "a", "said" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in the embodiments of the present invention refers to and includes any or all possible combinations of one or more associated listed items.

[0059] It should be understood that although the terms "first", "second", "third", etc. may be used to describe various information in the present invention, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0060] The technical solutions of the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0061] Figure 1 It is a flowchart of a facial expression recognition method based on a deep capsule network according to an exemplary embodiment of the present invention.

[0062] See also Figure 1 , the method comprising:

[0063] 110. Fusing features extracted from an original facial image and a local binary pattern (LBP) image of the original image;

[0064] Specifically, local binary patterns (LBP) reflect the basic information of facial expressions and are helpful for face recognition. More specifically, subtle changes can be reflected by features extracted with LBP. In addition, LBP can achieve rotation invariance and grayscale invariance, is suitable for extracting texture features of different scales, and can solve problems such as displacement imbalance, rotation angle, and lighting conditions in face images. In addition, LBP features can reflect subtle facial changes in skin texture.

[0065] LBP is defined as a window, where the pixel at the center of the window is set as the threshold, and then compared with the grayscale values ​​of the adjacent 8 pixels. If the surrounding pixel value is greater than the central pixel value, the pixel value position flag is 1, otherwise, the flag is 0. In this way, an 8-bit binary number is generated in the 3x3 neighborhood. Through the above steps, the LBP value of the central pixel of the window is obtained, which reflects the texture information of the area. Its definition is as follows:

[0066]

[0067] Where: p is the number of pixels, (x c ,y c ) is the coordinate of the pixel center, and the function of S(x) is as follows:

[0068]

[0069] Optionally, in this embodiment, step 110 specifically includes:

[0070] 1101. Extract features of the original facial image through a first convolutional neural network and perform pooling processing to obtain an input feature map;

[0071] 1102. Extract features of the LBP image of the original image through the first convolutional neural network and perform pooling processing to obtain a first LBP feature map;

[0072] 1103. Fusing the input feature map and the first LBP feature map through a second convolutional neural network to obtain a second LBP feature map;

[0073] Wherein, the first convolutional neural network and the second convolutional neural network are both composed of three groups of convolution kernels, normalization layers and activation functions.

[0074] Specifically, in this step, the first convolutional neural network and the second convolutional neural network adopt a multi-layer convolutional network structure, the main function of which is to fully extract image features. The convolutional network structure uses a pure convolutional layer as the backbone to extract features. In order to prevent the network from being too complicated, a small-size convolutional filter (3x3) is used for all layers. The input original image and the LBP image are subjected to feature extraction, and in order to make the features usable, the extracted features are then feature fused.

[0075] like Figure 2 As shown, the fusion structure F of the image is shown In ∈ C×H×W and F LBP ∈ C×H×W The feature maps are generated by the input original image and the input LBP image through the F1 module. C, H and W represent the number of channels, the height and width of the feature map respectively. In order to make full use of its feature information, F LBP and F In The features are combined as shown in formula (3):

[0076] F cat =[F In :F LBP ] (3)

[0077] In the formula, F cat is the feature map generated after fusion, indicating F cat ∈ 2C×H×W .

[0078] 120. Based on an attention mechanism, weighting the fused features to the features extracted from the original image;

[0079] Specifically, for classification tasks, it is necessary to extract image features and classify images into different category features based on the differences between these features. However, only important features are beneficial for classification, and different features have different importance for network propagation. In this step, based on the attention mechanism, it works by increasing the weights of useful features and making the network pay more attention to these features that are critical for expression recognition. In this way, the network can recognize different expressions more efficiently. The attention mechanism has been proven to be useful in pixel-level visual classification tasks and can be used to detect the degree of attention to features in different regions.

[0080] Optionally, in this embodiment, step 120 specifically includes:

[0081] The second LBP feature map is processed by an activation function and then multiplied by the input feature map.

[0082] Specifically, Figure 3 and Figure 4 As shown in Figure 1, this step is implemented through two branches, one branch is the branch of the backbone network extracting features, which is used to process the LBP feature map (LBPfeatures), and the other is the attention branch, which is used to process the input feature map (inputfeatures), as shown in formula (4):

[0083] F=F p F m (4)

[0084] In the formula, F p The feature map for extracting features from the backbone network, i.e., the input feature map, F m is the feature map of the attention module, namely the LBP feature map. The LBP feature map generation is expressed by formula (5):

[0085] F m =σ(Wf m +b) (5)

[0086] In the formula, f m is the second LBP feature map, W and b are the weight and bias of the convolutional layer respectively, and σ represents the activation function Softmax. The purpose is to assign different weights within (0,1) to the second LBP feature map so that the network can distinguish the importance of different features.

[0087] 130. The weighted features are input into the capsule network to obtain the facial expression classification.

[0088] Specifically, CNN networks are prone to losing some information, such as location information. However, the relative relationship between each part of the facial region is crucial for recognition. Therefore, the dynamic routing mechanism between capsules can solve these problems of CNN. In this step, capsule networks are used to encode and further obtain face image information. Figure 5 shown.

[0089] The capsule network is divided into an encoding structure and a decoding structure. In the encoding structure, the dynamic routing mechanism is performed in the PrimaryCaps layer and the DigitCaps layer. The DigitCaps layer mainly outputs 7 1*16 vectors, 7 represents the classification category, and the 1*16 vector represents the 16 features of the category. The decoding structure consists of three fully connected layers. The first two fully connected layers are nonlinear activation functions with ReLU, which can increase the nonlinearity of the network without being affected by the gradient vanishing problem. The last layer has a Sigmoid function.

[0090] The classification of the capsule network is done by squashing the nonlinear function after the DigitCaps layer to get the predicted probability of each class. Finally, the image category with the highest output is reconstructed and the reconstruction loss is calculated.

[0091] In general, the FDC network (Fusion Deep Capusle-Network) provided by the present invention is based on the basic CNN module and capsule network, and designs a model for facial expression recognition with a multi-layer convolutional network, attention mechanism and capsule network. The model is an end-to-end structure, and the network consists of four parts, namely, the feature extraction module F1 of the facial image, the feature fusion module F2, the attention module F3 and the capsule network module F4. Figure 7 As shown in the figure, the model starts with two independent CNN processing modules: one for processing the original image and the other for processing the LBP feature map. In order to prevent the network from being too complex and the number of parameters from being too large, the feature extraction module uses 3x3 convolution to apply to all layers. After integrating LBP attention, the network layer is 7x7x512, which enters the capsule network for final classification.

[0092] The face extraction module is divided into four modules, F1 to F4. Each module consists of multiple convolutional layers, normalization layers, and activation function layers. In order to extract deeper image information, each module consists of three groups of convolution kernels, normalization layers, and activation functions. Among them, the F1 and F4 modules are different from the F2 and F3 modules because they are connected to the pooling layer. Figure 6 As shown in the figure, the features extracted from the original image and the LBP image are fused through F1, and then the fused features are used as the attention module. The fused features are also weighted into the features extracted from the original image through F3 as the attention module. The attention module makes the network pay more attention to useful information by increasing the weight of useful features. After the network is extracted and weighted by F3 features and weighted by attention weights, it is passed through the pooling layer and F4 to obtain the final 7x7x512 output features and sent to the capsule network. The capsule network module converts the received features into a 7x16 dimensional vector, where 7 is the classification category, and each category is a 1x16 matrix containing multiple information of the image. Finally, the probability of each class is output through the vector model to achieve the classification effect.

[0093] Optionally, in this embodiment, the expression of the loss function L of the capsule network is as follows:

[0094] L=L c +L e

[0095] Among them, L c represents the loss of category c, and its expression is as follows:

[0096] L c =T c max(0,m + -||v c ||) 2 +λ(1-T c )max(0,||v c ||-m - ) 2

[0097] Where, T c Represents the judgment index, if category c exists, it is 1, otherwise it is 0;

[0098]

[0099] In the formula, α represents a threshold, θ represents the maximum domain of the cosine function in the formula, and m represents L e The maximum value of L r For the reconstruction loss, the mean square loss function is used, L r The expression is as follows:

[0100] L r =(x r -x) 2

[0101] In the formula, x r and x represent the reconstructed image and original image output by the capsule network, respectively.

[0102] Specifically, after the encoding layer of the capsule network, there are three fully connected layers for image reconstruction, which is the decoding stage. In order to minimize the reconstruction loss, the image can be better reconstructed from the encoding layer. The true label is used as the reconstruction target during training. The loss function of image reconstruction is constructed by calculating the Euclidean distance between the output pixel of the FC Sigmoid layer and the original image pixel. The coupling coefficient c ij Through dynamic routing updates, other convolutional parameters of the entire network and W in the capsule ij Need to be updated according to the loss function.

[0103] The loss function (L) includes the marginal loss (L c) and reconstruction loss (L r ). The expression of the loss function is defined in equations (6) and (7):

[0104] L c =T c max(0,m + -||v c ||) 2 +λ(1-T c )max(0,||v c ||-m - ) 2 (6)

[0105] L=L c +L r (7)

[0106] In the formula, c represents the classification category, L c represents the loss of category c, T c Represents the judgment index. If category c exists, it is 1, otherwise it is 0. r From formula (8), we can get:

[0107] L r =(x r -x) 2 (8)

[0108] In the formula, x r and x represent the reconstructed image and the original image respectively. r For the reconstruction loss, the mean square loss function is used. However, in the early stage of training, due to L r The loss value is too large, thus suppressing L c Therefore, the embodiment of the present invention introduces L e Loss to solve this problem, L e From formula (9), the overall loss is (10):

[0109]

[0110] L=L c +L e (10)

[0111] In the formula, α represents a threshold, θ represents the maximum domain of the cosine function in the formula. m represents L e The maximum value of is 0.0005 in the experiment. Use equation (13) to scale the reconstruction loss, thus avoiding the control of marginal loss during training. When α is set to 0.001, the experimental effect is better. Because the cosine function is in the interval It is monotonically decreasing and its value range is controlled in the range of (0,1), so the value of θ is selected as 1.57. In the early stage of training, the value of reconstruction loss is large and is within the range of θ. Due to the characteristics of the cosine function, the final value of reconstruction loss will become smaller. Therefore, when the reconstruction loss is too large during training, it will not affect the marginal loss.

[0112] Corresponding to the above method embodiment, the embodiment of the present invention also provides a facial expression recognition device based on a deep capsule network, including:

[0113] A feature fusion module, used for fusing features extracted from an original facial image and a local binary pattern (LBP) image of the original image;

[0114] An attention module, for weighting the fused features to the features extracted from the original image based on an attention mechanism;

[0115] The capsule network module is used to input the weighted features into the capsule network to obtain facial expression classification.

[0116] Optionally, in this embodiment, the feature fusion module specifically includes:

[0117] A first feature extraction unit, used for extracting features of an original facial image through a first convolutional neural network and performing pooling processing to obtain an input feature map;

[0118] A second feature extraction unit, configured to extract features of the LBP image of the original image through the first convolutional neural network and perform pooling processing to obtain a first LBP feature map;

[0119] A feature fusion unit, used for fusing the input feature map and the first LBP feature map through a second convolutional neural network to obtain a second LBP feature map;

[0120] Wherein, the first convolutional neural network and the second convolutional neural network are both composed of three groups of convolution kernels, normalization layers and activation functions.

[0121] Optionally, in this embodiment, the attention module is specifically used to multiply the second LBP feature map with the input feature map after processing it through an activation function.

[0122] Regarding the device in the above embodiment, the specific manner in which each module and unit performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0123] An embodiment of the present invention further provides a terminal device, including:

[0124] Processor; and

[0125] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.

[0126] An embodiment of the present invention further provides a non-transitory machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as described above.

[0127] In order to verify the effectiveness and generalization of the model, experiments and evaluations are conducted on the public datasets CK+, KEDF and RaFD. Figure 7 shown.

[0128] Since the number of images in the database is small, the embodiment of the present invention adopts the cross-validation method for evaluation. Before presenting the results, the three data sets are briefly introduced.

[0129] 1. CK+ Dataset

[0130] The Cohn-Kanade v2 database (CK+) dataset is a sequence-based dataset. It contains 593 image sequences of 123 objects. In the experiment, only the last frame is selected to construct the set of 7 emotions, and the neutral class is constructed from the first frame of each sequence. There are 45 samples in the anger class; disgust, 59; fear, 25; happiness, 69; neutral, 45; sadness, 28; surprise, 82.

[0131] 2. KDEF Dataset

[0132] The Karolinska Directed Emotional Faces (KDEF) dataset is a commonly used dataset for evaluating facial expressions. It contains 70 people, 50% male and 50% female, 7 different expressions, and a total of 980 images. In the KDEF dataset, each expression is divided into three angles: front, 45° side, and full side, which also brings certain difficulties to classification.

[0133] 3. RaFD Dataset

[0134] The Radboud Faces Database (RaFD) dataset

[0135] The team consists of 67 people and has a total of 8 emotion labels. In addition to the seven basic expressions, there is also a label for contempt. In addition, complex emotions are not symmetrical and only occur on one side of the face.

[0136] 4.2 Experimental parameters

[0137] The experimental environment is to use Nvidia RTX1080ti GPU for training under Ubuntu18.04 system. The code is written based on torch1.60, and in the model training, the Adam algorithm is used to optimize the network, and the initial learning rate is 0.0001. Since the number of data sets is not large, in order to prevent overfitting, the embodiment of the present invention adopts a 7-fold cross-validation method. Among them, CK+ adopts a data enhancement strategy to expand the number of data sets, rotate, flip and scale the image, and the rotation range is -20° to 20°. In network training, the number of CK+ iterations is 60 times, and the number of KDEF and RaFD iterations is 100 times.

[0138] 4.3 Experimental Results

[0139] In order to study the effectiveness of the proposed method, experiments were conducted with popular convolutional networks. The dataset used the same random seed and the same optimization algorithm to ensure the identity of the experiments. Table 1 shows the accuracy of the proposed method tested on KDEF, RaFD and CK+. The extracted algorithm was compared with popular networks such as VGG16, VGG19, ResNet18, ResNet50, ResNet101, GoogleNet, and Densenet121, indicating that the accuracy of the networks in the embodiments of the present invention has been improved to a certain extent.

[0140] Table 1 Comparison of the accuracy of the proposed method on the dataset with the popular network

[0141]

[0142]

[0143] From Table 1, it can be found that on the KDEF dataset, the popular deep learning networks with better performance are ResNet50, GoogleNet and Densenet121, and the performance on the KDEF dataset is 90.62%, 90.52% and 90.41% respectively. The algorithm of the embodiment of the present invention has an effect of 92.86% on this dataset, which is significantly improved compared with the other networks, and is 2.24%, 2.34% and 2.45% higher than ResNet50, GoogleNet and Densenet121 respectively; on the RaFD dataset, the popular deep learning networks with better performance are ResNet18, GoogleNet and Densenet121, and the performance on the RaFD dataset is 95.15%, 94.73% and 95.09% respectively. The algorithm of the embodiment of the present invention performs 97.93% on the dataset, which is significantly improved compared to other networks, and is 2.78%, 3.20% and 2.84% higher than ResNet18, GoogleNet and Densenet121 respectively; on the CK+ dataset, the popular deep learning networks with better performance are ResNet18 and ResNet50, which perform 95.31% and 96.09% on the CK+ dataset respectively. The algorithm of the embodiment of the present invention performs 98.57% on the dataset, which is significantly improved compared to other networks, and is 3.26% and 2.48% higher than ResNet18 and ResNet50 respectively.

[0144] In order to prove the effectiveness and advancement of the FDC model algorithm adopted in the embodiment of the present invention, it is compared with several methods adopted in the prior art, as shown in Table 2.

[0145] Table 2 Comparison of the accuracy of the proposed method on the dataset with the literature

[0146]

[0147] As can be seen from Table 2, the fusion improvement of traditional methods (such as OPD-GQMBP) still has a great improvement in facial expression recognition. It can be seen that the OPD-GQMBP method can achieve 90.20%, 97.20% and 96.48% accuracy in KDEF, RaFD and CK+ datasets respectively. VGG+LSTM can achieve an accuracy of 97.20% in CK+. The algorithm proposed in the embodiment of the present invention can improve 2.66%, 0.73% and 2.09% on the three datasets respectively compared with OPD-GQMBP, and improve 1.37% on the CK+ dataset compared with VGG+LSTM.

[0148] like Figure 8As shown in the figure, after comparison, it can be found that the proposed model has good robustness and generalization performance. The KDEF and RaFD datasets contain facial expressions from multiple angles. From the performance of the results, it is also proved that the capsule network under the attention mechanism has a strong representation ability for multiple facial features. At the same time, it also shows that the network has better performance than other networks and has certain advantages.

[0149] 4. Confusion Matrix

[0150] The confusion matrix is ​​divided according to the classification of each label in the test set. Through the confusion matrix, it is possible to analyze which emotion recognition is more difficult in the data set. In addition, confusion matrix analysis can identify which emotions affect other emotions. Fig. 9 The confusion matrix image generation effect of the datasets KDEF, RaFD and CK+ in the test set model of the embodiment of the present invention is demonstrated, and the recognition accuracy of each expression is analyzed.

[0151] Depend on Fig. 9 It can be seen that in the KDEF dataset, happy and natural expressions have the highest recognition rates, while fear and surprise have the highest misclassification rates, with fear having a 12% probability of being mistaken for surprise. In the RaFD dataset, the recognition rate of expressions is generally high, with the accuracy of natural expressions being 91%, and 6% being considered contemptuous expressions. Because the facial movements of contemptuous and natural expressions in this dataset are not large, confusion is easy to occur. In the CK+ dataset, because the contemptuous expression dataset has only 54 pictures, although its accuracy can reach 92% with data enhancement, there is still a 5% and 3% probability of being recognized as angry and sad, which affects the average accuracy.

[0152] from Fig. 9 From the results of the confusion matrices of the three data sets, the algorithm proposed in the embodiment of the present invention shows good performance in these three data sets, and the network has good classification performance, indicating the effectiveness of the network of the embodiment of the present invention in facial expression recognition.

[0153] 5. Ablation experiment

[0154] In order to prove the effectiveness of the network, the embodiment of the present invention conducted an ablation experiment, and conducted experiments with capsule network (CPN), CNN+CPN and CNN+CPN+attention mechanism (Att) and compared them with the model of the embodiment of the present invention. The model consists of three parts, namely feature extraction module, attention module, and capsule network module. In order to figure out how each module affects the performance of the network, an ablation experiment was conducted on the network. Because the capsule network is necessary for classification in the end in our network, the capsule network module is retained, and the feature extraction module, attention module and improved loss part are removed respectively, and then different experiments are conducted. The results are shown in Tables 3 and Fig.10 shown.

[0155] Table 3 Ablation experiments of the proposed method

[0156]

[0157] As can be seen from Table 3, after comparing the proposed method with the pure capsule network module, the non-attention module and the improved loss module, the accuracy has been improved to a certain extent. On the KDEF dataset, compared with the other three experiments, it has increased by 6.37%, 1.58% and 0.32% respectively. On the RaFD dataset, compared with the other three experiments, it has increased by 8.52%, 0.96% and 0.27% respectively. On the CK+ dataset, compared with the other three experiments, it has increased by 7.25%, 0.92% and 0.71%. Fig.10 It can be seen from the results that the improved loss function is effective for the model of the embodiment of the present invention. From the experimental results, it can be concluded that the attention module and the improved loss can effectively improve the performance of the network and show good model performance in facial expression recognition.

[0158] Through the above experimental process, the following conclusions can be drawn:

[0159] 1) The FDC network in the embodiment of the present invention combines the convolutional neural network and the capsule network, integrates the attention mechanism of LBP, and improves the reconstruction loss function of the capsule network.

[0160] 2) Capsule neural networks can capture the relative relationship between features of different local regions and represent features more effectively. It may be beneficial to some similar image-based recognition tasks. The attention mechanism of LBP enables the network to pay more attention to the important texture features of facial expressions and suppress background interference. The improved reconstruction loss makes up for the fact that the reconstruction loss is too large when starting training and affects the overall loss, thereby improving network performance.

[0161] 3) The proposed algorithm achieved accuracies of 92.54%, 97.93% and 98.57% on the KDEF, RaFD and CK+ datasets, respectively, demonstrating its effectiveness on these datasets and outperforming the current advanced facial expression recognition algorithms.

[0162] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used in the embodiments of the present invention is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed in the embodiments of the present invention.

Claims

1. A facial expression recognition method based on a deep capsule network, characterized in that: include: fusing features extracted from an original image of the face and a local binary pattern (LBP) image of the original image; Based on an attention mechanism, weighting the fused features to the features extracted from the original image; The weighted features are input into the capsule network to obtain the facial expression classification; The feature fusion extracted from the original facial image and the local binary pattern LBP image of the original image specifically includes: The features of the original facial image are extracted through the first convolutional neural network and pooled to obtain an input feature map; Extracting features of the LBP image of the original image through the first convolutional neural network and performing pooling processing to obtain a first LBP feature map; The input feature map and the first LBP feature map are fused through a second convolutional neural network to obtain a second LBP feature map; Wherein, the first convolutional neural network and the second convolutional neural network are both composed of three groups of convolution kernels, normalization layers and activation functions; Based on the attention mechanism, the fused features are weighted to the features extracted from the original image, specifically including: The second LBP feature map is processed by an activation function and then multiplied by the input feature map.

2. The method according to claim 1, characterized in that The processing of the second LBP feature map through an activation function specifically includes: The second LBP feature is processed according to the following formula: F m =σ(Wf m +b) In the formula, f m is the second LBP feature map, W and b are the weight and bias of the convolutional layer respectively, and σ represents the activation function Softmax. The purpose is to assign different weights within (0,1) to the second LBP feature map so that the network can distinguish the importance of different features.

3. The method according to claim 1, characterized in that The expression of the loss function L of the capsule network is as follows: L=L c +L e Among them, L c represents the loss of category c, and its expression is as follows: L c =T c max(0,m + -||v c ||) 2 +λ(1-T c )max(0,||v c ||-m - ) 2 Where, T c Represents the judgment index, if category c exists, it is 1, otherwise it is 0; In the formula, α represents a threshold, θ represents the maximum domain of the cosine function in the formula, and m represents L e The maximum value of L r For the reconstruction loss, the mean square loss function is used, L r The expression is as follows: L r =(x r -x) 2 In the formula, x r and x represent the reconstructed image and original image output by the capsule network, respectively.

4. A facial expression recognition device based on a deep capsule network, characterized in that: include: A feature fusion module, used for fusing features extracted from an original facial image and a local binary pattern (LBP) image of the original image; An attention module, for weighting the fused features to the features extracted from the original image based on an attention mechanism; The capsule network module is used to input the weighted features into the capsule network to obtain facial expression classification; The feature fusion module specifically includes: A first feature extraction unit, used for extracting features of an original facial image through a first convolutional neural network and performing pooling processing to obtain an input feature map; A second feature extraction unit, configured to extract features of the LBP image of the original image through the first convolutional neural network and perform pooling processing to obtain a first LBP feature map; A feature fusion unit, used for fusing the input feature map and the first LBP feature map through a second convolutional neural network to obtain a second LBP feature map; Wherein, the first convolutional neural network and the second convolutional neural network are both composed of three groups of convolution kernels, normalization layers and activation functions; The attention module is specifically used to multiply the second LBP feature map with the input feature map after processing it through an activation function.

5. A terminal device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 3.

6. A non-transitory machine-readable storage medium having executable code stored thereon, characterized in that: When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method as claimed in any one of claims 1 to 3.

Images