A face emotion recognition method based on high-order recursive attention weighting network

Abstract

The application discloses a face emotion recognition method based on a high-order recursive attention weighting network, which is a HRAM-CNN network model, and learning and training of face emotion recognition are performed, and then an image to be recognized is input into the trained HRAM-CNN network model to perform emotion judgment on each image; the HRAM-CNN network model is a high-order recursive attention mechanism HRAM which is put into a convolution layer of a convolution network model CNN. The application effectively combines the advantages of VisionTransformer and CNN, provides a high-order recursive attention mechanism combining the self-attention mechanism and the channel attention mechanism, and proposes that a large 7*7 kernel is used to construct the attention mechanism, so that adjacent information is effectively captured, and the feature extraction capability of the convolution process is further enhanced; the trained network can perform emotion judgment on each verification image.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of face recognition, and relates to a face emotion recognition method based on a high-order recursive attention weighting network. BACKGROUND

[0002] Face emotion is a carrier of human inner activity, and the change of face emotion can reflect the change of human psychology. It plays an indispensable role in daily life. Human expresses emotion mainly through language, tone, body movement and face emotion. In 1974, Albert Mechrabian, a famous psychologist, found through research that in the process of interpersonal communication, language and tone, tone can transmit 7% and 38% of information respectively, and face emotion can transmit 55% of information. Therefore, face emotion contains a large amount of effective information and can convey the true emotion of human heart, and the information conveyed is more accurate than language expression.

[0003] With the rapid development of machine vision and artificial intelligence, the popularity of computers and other intelligent devices, and the perfection of related theoretical knowledge system, face emotion recognition, as a frontier research field across psychology, machine vision, artificial intelligence and other disciplines, has attracted the attention of many experts and scholars, universities and Internet technology companies. In this era of Internet of Things and intelligent interaction, face emotion analysis has become another important bridge for human-machine interaction. Using face emotion recognition technology can break the bottleneck of previous human-computer interaction, making human-computer interaction more friendly, intelligent and convenient. Therefore, further research on it has very important academic research significance and application value. Face emotion recognition technology has been widely used in many fields, such as intelligent transportation, medical treatment, human-computer interaction, criminal investigation, online education and other fields.

[0004] The rise of machine learning provides a more powerful tool for face emotion recognition research. A typical face emotion recognition system includes face emotion dataset, feature extraction, training and verification. Machine learning algorithms are mainly divided into two categories:

[0005] 1) Traditional machine learning

[0006] Traditional machine learning first manually determines the corresponding "facial features" as the features of machine learning, so as to classify and identify the object, which can use support vector machine and random forest algorithm. Xu Wen-hui et al. adopt k-neighbor method to input the extracted features into support vector machine for face emotion recognition, which realizes good classification effect. However, support vector machine algorithm needs to solve support vector by means of quadratic programming, which involves the calculation of m-order matrix, resulting in that support vector machine algorithm is difficult to implement for large-scale training samples. Moreover, the classical support vector machine algorithm only gives the algorithm of binary classification, and the solving effect is poor for multi-classification problem; Wu Shili et al. adopt random forest model to identify emotion, and the result has good accuracy. However, random forest algorithm cannot control the internal operation of the model, resulting in that the model can only be tried between different parameters and random seeds.

[0007] 2) Deep learning

[0008] Deep learning is a paradigm of machine learning, which optimizes data analysis and shortens modeling process. Deep learning unifies various different feature extraction algorithms of traditional machine learning by neural network, and automatically learns high-dimensional abstraction of data through neural network to find important features required for classification problem. Yann LeCun et al. proposed a convolutional neural network structure LeNet, which uses back propagation algorithm to train convolutional neural network to extract features and complete identification task; Alex Krizhevsky et al. proposed AlexNet model, which enriches output features by deepening network layers based on LeNet; Visual Geometry Group proposed VGG model based on the network idea of AlexNet, which uses three 3*3 convolution kernels to replace the 7*7 convolution kernel in AlexNet, and uses two 3*3 convolution kernels to replace the 5*5 convolution kernel in AlexNet, so as to improve network depth and neural network effect under the condition of ensuring the same receptive field; Li S et al. proposed a convolutional neural network DLP-CNN which preserves deep locality, which further enhances the discriminability of deep features by reducing intra-class variation and increasing inter-class difference; Shi C et al. proposed a multi-branch cross-connection convolutional neural network MBCC-CNN, which makes the data flow between networks more smooth by adding a shortcut cross-connection in the convolution output layer, and improves the feature extraction ability of each receiving domain.

[0009] However, the convolutional neural network in the above documents lacks large receptive field to capture long-term dependency, so that the convolutional network is not sufficient for image feature extraction, and the accuracy of face emotion classification and identification is not high. SUMMARY

[0010] The technical problem solved by the present application is to provide a face emotion recognition method based on a high-order recursive attention weighting network, which can extract effective features from image data and be used for emotion classification, and has a good recognition accuracy.

[0011] The present application is realized by the following technical solutions:

[0012] The face emotion recognition method based on the high-order recursive attention weighting network is to construct an HRAM-CNN network model, and learn and train the HRAM-CNN network model for face emotion recognition, then input an image to be recognized into the trained HRAM-CNN network model, and judge the emotion of each image;

[0013] The HRAM-CNN network model is to put a high-order recursive attention mechanism HRAM into a convolution layer of a convolutional neural network model CNN, wherein an input layer of the CNN is responsible for inputting face emotion image data, a convolution layer is responsible for extracting image data features, a pooling layer is responsible for dimension reduction on the extracted data features, a full connection layer is to integrate the reduced features, and an output layer is responsible for image classification and recognition; HRAM is added after multiple Blocks of the CNN convolution layer;

[0014] The HRAM is constructed in a three-order recursive form by introducing high-order interaction after a first-order spatial interaction basic operation; let X be R HW×C be input features, and the basic operation of the HRAM is as shown in formula 1:

[0015]

[0016] wherein, is a linear projection layer for performing channel mixing, f represents a deep convolution, H represents the height of a feature matrix, W represents the width of the feature matrix, C represents the depth of the feature matrix, Avgpool represents average pooling, Linear represents full connection, P0 represents a projection feature, and q0 represents a neighbor feature thereof;

[0017]

[0018] Ω i is a local window centered on i, and omega represents a convolution weight of f.

[0019] The operation performed by the HRAM on an image feature with an input size of HxWxC is:

[0020] 1) the image feature is passed through a convolution Conv1*1 to increase the channel dimension from C to 2C;

[0021] 2) Then divide the channel dimension into two parts of C / 4 and 7C / 4, wherein the first part is firstly subjected to a global average pooling Adaptive AvgPool, two layers of full connection FC, and then a Scale operation is performed to weight the normalized weight obtained in the foregoing to the features of each channel to obtain the feature map of this part; the second part is firstly subjected to a depth convolution DW Conv, and then the image feature map obtained after the depth convolution is split into three parts of C / 4, C / 2 and C in the channel dimension;

[0022] 3) The feature map obtained in the first part is spliced with the C / 4 feature map obtained by splitting the second part to obtain a feature map with a channel dimension of C / 2;

[0023] 4) The feature map with a channel dimension of C / 2 is subjected to a Conv1*1 convolution to realize cross-channel information interaction and integration; then subjected to a global average pooling Adaptive AvgPool, two layers of full connection FC, and finally a Scale operation is performed to weight the normalized weight obtained in the foregoing to the features of each channel to obtain the feature map of this part;

[0024] 5) The feature map obtained in step 4) is spliced with the C / 2 feature map obtained by splitting the second part to obtain a feature map with a channel dimension of C;

[0025] 6) The feature map with a dimension of C is subjected to a Conv1*1 convolution to realize cross-channel information interaction and integration; then subjected to a global average pooling Adaptive AvgPool, two layers of full connection FC, and finally a Scale operation is performed to weight the normalized weight obtained in the foregoing to the features of each channel to obtain the feature map of this part;

[0026] 7) The feature map obtained in step 6) is spliced with the C feature map obtained by splitting the second part to obtain a feature map with a channel dimension of 2C;

[0027] 8) The feature map with a dimension of 2C is further subjected to a convolution Conv1*1, so that the channel dimension of the input image is reduced from 2C to C to obtain an output image feature map.

[0028] The HRAM-CNN network model comprises a feature extractor and a classifier, wherein the feature extraction processing of the input image by the feature extractor comprises:

[0029] 1) The approximate position feature of the image is extracted by the first Block;

[0030] 2) The contour feature of the face is extracted by the second Block;

[0031] 3) An HRAM is added after the third Block to extract the facial features of the face, so that the extracted facial features are more subtle;

[0032] 4) The fourth Block adds an HRAM to extract detailed information features including eye tilt angle, micro-closed degree, mouth opening angle and size;

[0033] 5) The fifth Block adds an HRAM to extract implicit features of the image;

[0034] The classifier processes the extracted features as follows: first, the image feature matrix is pulled into one dimension through two fully connected layers, and then the final output emotion type is obtained through the Softmax function in the output layer.

[0035] The training learning of the HRAM-CNN network model includes the following operations:

[0036] 1) Taking the cross-entropy loss function as the objective function LOSS, Loss = -log(P i ), P i is the output probability of the ith emotion signal, and the loss of all data after each iteration is defined as LOSS, I is the category, J is the number of each category, i is the ith data, and j is the jth data;

[0037] 2) Select the back propagation to adjust the network parameters, and obtain excellent network parameters after multiple iterations;

[0038] First, the forward propagation of the network is realized, and the output value of each node of each layer is obtained, secondly, the loss function of the network is determined, the residual error of the output and the sample value is calculated in the output layer, and the residual error of each node of each layer is calculated respectively, and finally the partial derivative of the loss function to the weight and bias is calculated respectively, and the weight and bias are updated according to the gradient descent method;

[0039] 3) Using ADAM optimizer to accelerate the loss convergence speed;

[0040] 4) Through multiple network training, an excellent network model and the parameters of the network model are obtained.

[0041] Compared with the prior art, the present application has the following beneficial technical effects:

[0042] The HRAM-CNN network model provided by the application is a fusion of HRAM and CNN, wherein the HRAM is a high-order recursive attention mechanism that fuses self-attention mechanism and channel attention mechanism, and the fusion of HRAM and VGG16 can realize long-distance dependence modeling to increase the global receptive field and enrich the output features. The HRAM-CNN network model analyzes the advantages of the Vision Transformer from the new perspective of high-order spatial attention, effectively combines the advantages of the Vision Transformer and the CNN, and combines the high-order recursive attention mechanism of the self-attention mechanism and the channel attention mechanism, and proposes to use a large 7*7 kernel to construct the attention mechanism, which can effectively capture adjacent information and further enhance the feature extraction capability of the convolution process. The face emotion verification set is subjected to the trained network, and emotion judgment can be performed on each verification image. Through the data provided in the CK+ data set, the model of the application has a lower LOSS value than the traditional convolutional neural network at the same prediction accuracy.

[0043] The model HRAM-CNN of the application and the VGG16 (which is a classic CNN model) without adding HRAM are also trained with a batch size of 128 and an epoch of 50 times, and the LOSS values of the model of the application on the training set and the test set are both less than those of the VGG16 model without adding HRAM, the LOSS values of the model of the application are 0.14 and 0.04 respectively, and the LOSS values of the VGG16 model without adding HRAM are 0.55 and 0.36 respectively, and the results show that the model of the application has better prediction accuracy.

[0044] The model HRAM-CNN of the application and the Vision transformer model are also trained with a batch size of 128 and an epoch of 50 times, and the LOSS values of the model of the application on the training set and the test set are both less than those of the Vision transformer model, the LOSS values of the model of the application are 0.14 and 0.04 respectively, and the LOSS values of the Vision transformer model are 0.59 and 0.46 respectively, and the results show that the model of the application has better prediction accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 HRAM is a high-order recursive attention model;

[0046] Figure 2 HRAM-CNN is a network model of the application;

[0047] Figure 3 The design flowchart of the application;

[0048] Figure 4 The experimental flowchart of the present application is shown in the following table:

[0049] Figure 5 The LOSS value and ACC value trend chart of the VGG16 network model through 50 iterations of training and testing data is shown in the following table: Figure 5 It is shown that the convolutional network model gradually converges in the training process.

[0050] Figure 6 The LOSS value and ACC value trend chart of the Vision transformer model through 50 iterations of training and testing data is shown in the following table:

[0051] Figure 7 The LOSS value and ACC value trend chart of the HRAM-CNN model of the present application through 50 iterations of training and testing data is shown in the following table. DETAILED DESCRIPTION

[0052] The present application will be further described in detail below in conjunction with examples, which are an explanation of the present application rather than a limitation.

[0053] The present application proposes a face emotion recognition method based on a high-order recursive attention weighting network to extract effective features from data and use them for emotion classification, thereby promoting human-computer interaction.

[0054] The face emotion recognition method based on the high-order recursive attention weighting network proposed by the present application is to build an HRAM-CNN network model and learn and train it for face emotion recognition, and then input the image to be recognized into the trained HRAM-CNN network model to judge the emotion of each image.

[0055] The HRAM-CNN network model is to put the high-order recursive attention mechanism HRAM into the convolutional layer of the convolutional network model CNN, wherein the input layer of the CNN is responsible for inputting face emotion image data, the convolutional layer is responsible for extracting image data features, the pooling layer is responsible for dimension reduction of the extracted data features, the fully connected layer is to integrate the reduced features, and the output layer is responsible for image classification and recognition; HRAM is added after multiple Blocks of the CNN convolutional layer.

[0056] The following will be described in detail.

[0057] Construction of high-order recursive attention mechanism HRAM

[0058] The HRAM model is as follows: Figure 1The shown is an efficient operation module for realizing long-range and high-order spatial interaction, which is constructed by standard convolution, linear projection, element splicing and element multiplication, has input adaptive spatial mixing function similar to self-attention, mainly relies on correct modeling of spatial interaction in visual data, thereby improves the modeling capability of the model.

[0059] Let X be in R HW×C As input features, the basic operation of HRAM proposed by the present application is shown in formula 1:

[0060]

[0061] Wherein is a linear projection layer for performing channel mixing, f represents a deep convolution (DWConv), H represents the height of a feature matrix, W represents the width of the feature matrix, C represents the depth of the feature matrix, Avgpool represents average pooling, Linear represents full connection, P0 represents a projection feature, and q0 represents a neighbor feature thereof;

[0062]

[0063] Wherein Ω i is a local window centered on i, and ω represents a convolution weight of f.

[0064] Formula (1) explicitly introduces the interaction between adjacent features and by element splicing. The interaction in the basic operation of HRAM is first-order, because each only interacts with its neighbor feature once. Therefore, after realizing the efficient first-order spatial interaction basic operation, a three-order recursive form is constructed by introducing high-order interaction.

[0065] Specifically, when HRAM is executed:

[0066] Firstly, P0 and q0 are obtained by using

[0067]

[0068] Then, the basic operation of HRAM is executed by high-order recursion:

[0069] P​k+1 = f k (q k ) + g k (P k ) / a k = 0, 1, 2 (4)

[0070] The output is scaled by 1 / a to stabilize training, {f k} is a set of deep convolutional layers, {g k} is used to match the dimensions in different orders:

[0071]

[0072] Finally, the output of the third recursive step q n is given to the projection layer to obtain the result of HRAM.

[0073] As can be seen from equation (4), the order of interaction of P k increases by 1 at each step, and HRAM realizes three-order spatial interaction. In a deep network model, due to nonlinearity, there is complex high-order interaction between two spatial positions, and the introduction of explicit high-order spatial interaction in architectural design helps to improve the modeling ability of the model, thereby improving the efficiency of facial emotion recognition. Moreover, in HRAM, only one f is needed to perform deep convolution on the concatenation of features , without calculating convolution in each recursive step as in equation (4), which can further simplify the implementation and improve the efficiency on GPU.

[0074] Since traditional CNN usually uses 3*3 convolution of the entire network, in order to enable HRAM to capture long-range spatial interaction, the present application adopts deep convolution with a convolution kernel size of 7*7; at the same time, in order to ensure that high-order interaction does not introduce too much computational overhead, the present application sets the channel dimension of each order to:

[0075]

[0076] This design shows that the present application performs interaction in a coarse-to-fine manner, with fewer channels calculated for lower orders. Therefore, the HRAM of the present application can better simulate more complex spatial interactions (helping to improve the modeling ability of the model, thereby improving the efficiency of facial emotion recognition), and can be used as a plug-and-play module to improve convolution-based models.

[0077] Referring to Figure 1 , the following will take the operation of HRAM on image features with an input size of HxWxC as an example for description:

[0078] 1) First, the image feature map with the size of HxWxC is passed through a convolution Conv1*1 to increase the channel dimension from C to 2C;

[0079] 2) Then, the channel dimension is divided into two parts of C / 4 and 7C / 4. The first part is first passed through a global average pooling AdaptiveAvgPool, then through two fully connected FCs, and finally a Scale operation is performed to weight the normalized weights obtained in the previous step to the features of each channel to finally obtain the feature map of this part; the second part is first passed through a depth convolution DW Conv, and then the image feature map obtained after the depth convolution is split into C / 4, C / 2, C in the channel dimension.

[0080] 3) The feature map obtained in the first part is spliced (Concat) with the C / 4 feature map obtained by splitting the second part to obtain a feature map with a channel dimension of C / 2.

[0081] 4) First, a Conv1*1 convolution is performed to realize cross-channel information interaction and integration. Then, a global average pooling AdaptiveAvgPool is performed, followed by two fully connected FCs, and finally a Scale operation is performed to weight the normalized weights obtained in the previous step to the features of each channel to finally obtain the feature map of this part;

[0082] 5) The feature map obtained in step 4) is spliced (Concat) with the C / 2 feature map obtained by splitting the second part to obtain a feature map with a channel dimension of C.

[0083] 6) First, a Conv1*1 convolution is performed to realize cross-channel information interaction and integration. Then, a global average pooling AdaptiveAvgPool is performed, followed by two fully connected FCs, and finally a Scale operation is performed to weight the normalized weights obtained in the previous step to the features of each channel to finally obtain the feature map of this part;

[0084] 7) The feature map obtained in step (6) is spliced (Concat) with the C feature map obtained by splitting the second part to obtain a feature map with a channel dimension of 2C.

[0085] 8) Finally, a convolution Conv1*1 is performed to reduce the channel dimension of the input image from 2C to C to obtain the output image feature map.

[0086] Construction of network model HRAM-CNN

[0087] The convolutional network model is composed of an input layer, a convolutional layer, a pooling layer, a full connection layer and an output layer, the input layer is responsible for inputting facial emotion data, the convolutional layer is responsible for extracting data features, the pooling layer is responsible for dimension reduction of the extracted data features, the full connection layer is used for integrating the reduced features, and the output layer is responsible for image classification.

[0088] Considering the timeliness and accuracy of the image classification task, the VGG16 network model with high speed and multiple convolutional layers is used to extract facial image features.

[0089] The VGG16 is a 16-layer network architecture composed of 13 convolutional layers and 3 full connection layers. The 13 convolutional layers are divided into five blocks, and each block is a combination of multiple Conv-BN-ReLU. The batch normalization layer (Batch Normal, BN) is used to speed up the model learning and convergence speed, and prevent the problems of gradient explosion, disappearance and overfitting. The ReLU is an activation function, which aims to increase the non-linear factor and improve the expression ability of the model.

[0090] The VGG16 is a 16-layer network architecture composed of 13 convolutional layers and 3 full connection layers. The 13 convolutional layers are divided into five blocks, and each block is a combination of multiple Conv-BN-ReLU. The batch normalization layer (Batch Normal, BN) is used to speed up the model learning and convergence speed, and prevent the problems of gradient explosion, disappearance and overfitting. The ReLU is an activation function, which aims to increase the non-linear factor and improve the expression ability of the model.

[0091] The seven different emotion types of the final output prediction are obtained by the Softmax function in the output layer, which are anger, contempt, disgust, fear, happiness, sadness and surprise. The network training graph and verification graph come from the data in the CK+ dataset, which is composed of 593 expression image sequences of 123 participants.

[0092] The parameters of the network are set, including the size of the convolution kernel, the step of the convolution kernel, the step of the pooling layer and the number of convolution kernels of the three full connection layers.

[0093] The size of the convolution kernel is set to 3*3, the step of the convolution kernel is 1, the step of the pooling layer is 2, the number of convolution kernels of the first layer full connection is 512, the number of convolution kernels of the second layer is 256, and the number of convolution kernels of the third layer is 7.

[0094] The Softmax function is used in the multi-classification process, which maps the outputs of multiple neurons to the interval (0, 1), and the values can be regarded as probabilities and the sum of the probabilities is 1, which corresponds to the probabilities of belonging to the seven types of facial emotion types.

[0095] As shown in Figure 2 , the HRAM-CNN network model includes a feature extractor and a classifier, wherein the feature extraction processing of the feature extractor on the input image includes:

[0096] 1) extracting the approximate position features of the image through the first Block;

[0097] 2) extracting the contour features of the face through the second Block;

[0098] 3) adding an HRAM after the third Block to extract the facial features of the face, so that the extracted facial features are more subtle;

[0099] 4) adding an HRAM through the fourth Block to extract detailed information features including eye tilt angle, micro-closed degree, mouth opening angle and size;

[0100] 5) adding an HRAM to the fifth Block to extract the implicit features of the image;

[0101] The classifier performs the following processing on the extracted features: first, the image feature matrix is stretched into one dimension through two fully connected layers, and then the final output emotion type is obtained through the Softmax function in the output layer.

[0102] The learning of the HRAM-CNN network model includes the following operations:

[0103] (1) Define the objective function LOSS, and the present application selects the cross entropy loss function (Cross Entropy Loss): Loss = -log(P i ), P i is the output probability of the ith facial signal, and the loss of all data after each iteration is defined as LOSS, I is the class, J is the number of each class, i is the ith data, and j is the jth data.

[0104] The LOSS value is the sum of the loss functions of all data after each iteration, which can be used to represent the performance of the model, and the smaller the LOSS value is, the higher the prediction rate of the current model on the input data is, that is, the more accurate the prediction is.

[0105] (2) Select back propagation to adjust the network parameters, and the most commonly used method is gradient descent method, and excellent network parameters are obtained through multiple iterations.

[0106] Gradient descent is a method often used in neural network training, gradient mathematically represents the fastest direction of rise, and negative gradient represents the fastest direction of decline, through which the convergence of neural network can be reached fastest.

[0107] First, the forward propagation of the network is realized, and the output value of each node of each layer is obtained, secondly, the loss function of the network is determined, the residual error of the output and the sample value is calculated in the output layer, and the residual error of each node of each layer is calculated respectively, and finally the partial derivative of the loss function with respect to the weight and bias is calculated respectively, and the weight and bias are updated according to the gradient descent method.

[0108] (3) using ADAM optimizer to accelerate the loss convergence speed.

[0109] (4) through multiple network training, an excellent network model and the parameters of the network model are obtained.

[0110] The following gives an embodiment of the HRAM-CNN network model for automatic recognition simulation experiment of facial emotion.

[0111] The automatic recognition simulation of the HRAM-CNN network model for facial emotion includes the following operations:

[0112] (1) by inputting CK+ data set, reading image data and corresponding seven labels of anger, contempt, disgust, fear, joy, sadness and surprise.

[0113] (2) preprocessing (resize) operation is performed on the input image: in order to enhance the detectability of image related information, simplify the data to the maximum, the input image is preprocessed, so as to improve the reliability of feature extraction, image segmentation, matching and recognition. Since the image is input in the form of vector matrix, the input dimension entering the full connection layer must be fixed due to the limitation of network structure, so it is necessary to adjust the size of the input image and fix it to the same size. Considering the configuration of the machine, the larger the image, the higher the configuration required for operation processing.

[0114] The size of the input image based on the VGG16 model is generally 224*224*3, so all the images are resized to 224*224*3 in the present application. Since the image size of the data set used in the present application is large, in order to reduce the image, the INTER_AREA pixel area related resampling method is used. It fully considers the surrounding pixels of the image, and effectively prevents the generation of ripples.

[0115] (3) the CK+ data set is divided, 80% is used for training and learning, and 20% is used for verification.

[0116] (4) the image and label of the data set are converted into torch format.

[0117] (5) Initialize hyperparameters: INIT_LR = 0.0005, EPOCHS = 50, Batch Size = 128.

[0118] (6) Put the training pictures in the CK+ dataset into the HRAM-CNN, train according to step three, complete the forward propagation of the model, update the parameters of each layer in the model according to the error between the output and the target, complete one iteration training, complete 50 iterations of training on all training data, record the LOSS value of the training set and the test set in each iteration training, and obtain the trained model. The trained network is used as the network for automatic recognition of facial emotions.

[0119] (7) Put the verification pictures in the CK+ dataset into the trained network one by one for verification, judge the emotions of each image, and compare the output and the label to calculate the prediction accuracy.

[0120] Specifically, randomly select a happy picture for prediction, convert the picture into a size of (224, 224) and convert it into a tensor tensor format, load the trained network model, put the input image into the trained model for prediction, and the predicted result is happy.

[0121] Figure 7 Compared with Figure 5 , Figure 6 , it is shown that the LOSS value of the HRAM-CNN model of the application is less than that of the VGG16 network model and the Vision transformer model after 50 iterations, which reflects the superiority of the model of the application.

[0122] The above embodiments are preferred examples of implementing the application, and the application is not limited to the above embodiments. Any non-essential addition, replacement made by a person skilled in the art according to the technical features of the technical solution of the application belongs to the protection scope of the application.​

Claims

1. A face emotion recognition method based on a high-order recursive attention weighting network, characterized in that, The HRAM-CNN network model is constructed and learned and trained for face emotion recognition, and then the image to be recognized is input into the trained HRAM-CNN network model to judge the emotion of each image. The HRAM-CNN network model is constructed and learned and trained for face emotion recognition, and then the image to be recognized is input into the trained HRAM-CNN network model to judge the emotion of each image. The HRAM is constructed in a third-order recursive form by introducing high-order interactions after a first-order spatial interaction basic operation; let X ∈ R HW×C For input features, the basic operation of the HRAM is shown in Equation 1: wherein, is a linear projection layer that performs channel mixing, f denotes a depth convolution, H denotes a height of a feature matrix, W denotes a width of a feature matrix, C denotes a depth of a feature matrix, Avgpool denotes an average pooling, Linear denotes a full connection, P0 denotes a projected feature, and q0 denotes its neighbor features; Ω i is a local window centered at i, and ω denotes the convolution weights for f. 2.The face emotion recognition method based on the high-order recurrent attention weighting network according to claim 1, wherein, The HRAM-CNN network model is constructed and learned and trained for face emotion recognition, and then the image to be recognized is input into the trained HRAM-CNN network model to judge the emotion of each image. First, use to obtain a set of projected features P0and The HRAM in execution: P k+1 = f k (q k )+ f k (P k ) / α k=0,1,2 (4) The output is scaled by 1 / a to stabilize training, {f k} is a set of deep convolutional layers, {g k} to match dimensions in different orders: Finally, the output of the third recursive step q n is given to the projection layer to obtain the result of the HRAM. 3.The face emotion recognition method based on the high-order recurrent attention weighting network according to claim 1 or 2, characterized in that, Then, the basic operation of HRAM is executed recursively: The operation of the HRAM on the image feature with the input size of HxWxC is: 1) The image feature is passed through a convolution Conv1*1 to increase the channel dimension from C to 2C; 2) Then the channel dimension is divided into C / 4 and 7C / 4 two parts, wherein the first part is first subjected to a global average pooling Adaptive AvgPool, two layers of full connection FC, and then a Scale operation is performed to weight the normalized weight obtained in the foregoing to the feature of each channel, to obtain the feature map of this part; the second part is first subjected to a depth convolution DW Conv, and then the image feature map obtained after the depth convolution is split into C / 4, C / 2, C three parts in the channel dimension; 3) The feature map obtained in the first part is spliced with the C / 4 feature map obtained by splitting the second part to obtain a feature map with a channel dimension of C / 2; 4) The feature map with a channel dimension of C / 2 is subjected to a Conv1*1 convolution to realize cross-channel information interaction and integration; then subjected to a global average pooling Adaptive AvgPool, two layers of full connection FC, and finally a Scale operation is performed to weight the normalized weight obtained in the foregoing to the feature of each channel, to obtain the feature map of this part; 5) The feature map obtained in step 4) is spliced with the C / 2 feature map obtained by splitting the second part to obtain a feature map with a channel dimension of C; 6) The feature map with a channel dimension of C is subjected to a Conv1*1 convolution to realize cross-channel information interaction and integration; then subjected to a global average pooling Adaptive AvgPool, two layers of full connection FC, and finally a Scale operation is performed to weight the normalized weight obtained in the foregoing to the feature of each channel, to obtain the feature map of this part; 7) The feature map obtained in step (6) is spliced with the C feature map obtained by splitting the second part to obtain a feature map with a channel dimension of 2C; 8) The feature map with a dimension of 2C is subjected to a convolution Conv1*1 again, to reduce the channel dimension of the input image from 2C to C, to obtain the output image feature map. 4.The face emotion recognition method based on the high-order recurrent attention weighting network according to claim 1, wherein, The convolutional neural network model CNN is a VGG16 model, and the convolutional layers thereof are divided into five blocks, each of which is a combination of multiple Conv-BN-ReLU; HRAM is added to the last three blocks of the VGG16; The output layer outputs the emotion type through a Softmax function, including anger, contempt, disgust, fear, happiness, sadness, and surprise. The Softmax function is used in the classification process, and it maps the outputs of multiple neurons to the interval (0, 1). These values can be regarded as probabilities, and the sum of the probabilities is 1, corresponding to the probabilities of belonging to the seven types of facial emotions. 5.The face emotion recognition method based on the high-order recurrent attention weighting network according to claim 4, wherein, The HRAM-CNN network model includes a feature extractor and a classifier, wherein the feature extraction process of the feature extractor on the input image includes: 1) extracting the approximate position features of the image through the first block; 2) extracting the contour features of the face through the second block; 3) adding an HRAM after the third block to extract the facial features of the face, so that the extracted facial features are more subtle; 4) adding an HRAM through the fourth block to extract detailed information features including the eye tilt angle, the degree of micro-closure, the mouth opening angle, and the size; 5) adding an HRAM to the fifth block to extract the implicit features of the image; The classifier processes the extracted features as follows: first, it expands the image feature matrix to one dimension through two fully connected layers, and then it obtains the final output emotion type through the Softmax function in the output layer. 6.The face emotion recognition method based on the high-order recurrent attention weighting network according to claim 1, wherein, The training and learning of the HRAM-CNN network model include the following operations: 1) Take the cross-entropy loss function as the objective function LOSS, Loss = -log(P i ), P i is the output probability of the i-th emotional signal, define the loss of all data after each iteration as LOSS, I is the category, J is the number of each category, i is the i-th category data, and j is the j-th data; 2) Selecting back propagation to adjust the network parameters, and obtaining excellent network parameters through multiple iterations; First, realize the forward propagation of the network and obtain the output value of each node in each layer, secondly, determine the loss function of the network, calculate the residual error between the output and the sample value in the output layer, and calculate the residual error of each node in each layer respectively, finally, calculate the partial derivative of the loss function with respect to the weight and bias, and update the weight and bias according to the gradient descent method; 3) Using the ADAM optimizer to accelerate the loss convergence speed; 4) Through multiple network training, an excellent network model and the parameters of the network model are obtained.

Images