A facial emotion recognition method based on attention mechanism

Abstract

The present invention discloses a facial emotion recognition method based on an attention mechanism, which first collects images and divides the images into a training sample set and a test sample set; then reads the original emotion annotation value of each sample image in the training sample set, and sends each sample image in the training sample set into a multi-task convolutional neural network to obtain output features; the output features obtained after preprocessing are input into a residual attention network to obtain an attention output feature map; the output feature map is respectively sent to a transformer Transformer and a non-local neural network for learning, thereby obtaining output features; a classifier is trained using a two-stage multi-task learning network, so that the deep features of the test samples can be used by the classifier to generate emotional state values ​​in a dimensional space. The present invention solves the problem that the existing facial emotion recognition method based on the attention mechanism lacks simultaneous modeling and learning of the dependency relationship between feature maps and the spatial position dependency relationship of features within the feature map.

Description

Technical Field

[0001] The present invention belongs to the technical field of pattern recognition, and in particular relates to a facial emotion recognition method based on an attention mechanism. Background Art

[0002] In recent years, with the continuous development of society and the urgent need for fast and effective automatic emotion recognition in various aspects, biometric recognition technology has developed rapidly in the past decade. In real life, for normal social communication and needs, people need to perform necessary emotion recognition to judge each other's emotions and make correct behavioral responses.

[0003] As early as 1981, Aaron Sloman proposed research on artificial intelligence emotions. In 1985, Marvin Minsky, one of the founders of artificial intelligence, raised the issue of computers and emotions. In 1995, Rosalind W. Picard of the MIT Media Lab formally proposed the concept of affective computing, and in 1997, she defined "affective computing" in "Affective Computing" as "computing related to, triggered by, or able to influence emotional factors."

[0004] As one of the important branches in the field of recognition, emotion recognition has been the focus of research by many scholars in recent years. For static face images, in traditional recognition methods, researchers usually extract emotional features manually (for example, 68 feature points based on Dlib), and then send them to a pre-designed classifier for classification (SVM, decision tree, random forest, etc.) to obtain the final emotion prediction result. However, in actual operation, this method requires the use of rich experience to manually select appropriate emotional features, and it often takes a lot of time and effort. Moreover, different features contribute differently to the final result, and traditional methods do not distinguish them well.

[0005] In recent years, with the continuous improvement of computer capabilities, deep learning has been re-proposed. Due to its powerful feature learning ability and high performance, it has gradually replaced traditional machine learning and become the mainstream method in the field of recognition. Among the current sentiment analysis methods based on deep learning, most methods will combine the attention mechanism to screen effective features, but lack the ability to model the learning ability of the dependency relationship between feature graphs, and the consideration of the dependency relationship between spatial position features in feature graphs. In addition, traditional methods train different models for different tasks, without taking advantage of the similarities between related tasks. Summary of the invention

[0006] The purpose of the present invention is to provide a facial emotion recognition method based on an attention mechanism, which solves the problem that the existing facial emotion recognition methods based on the attention mechanism lack the simultaneous modeling and learning of the dependency relationship between feature maps and the spatial position dependency relationship of features within the feature maps.

[0007] The technical solution adopted by the present invention is a facial emotion recognition method based on an attention mechanism, which is specifically implemented according to the following steps:

[0008] Step 1: Collect images and divide them into training sample sets x train And the test sample set x test ;

[0009] Step 2: Read the training sample set x train The original emotional annotation value of each sample image in the training sample set x train Each sample image in is sent to the multi-task convolutional neural network MTCNN, and face alignment is completed based on face detection and 5 key feature points to obtain the output feature x input =[x1, x2, ..., x n ];

[0010] Step 3: The output feature x obtained after preprocessing input Input into the residual attention network, and then extract the features M of different receptive fields through the trunk branch i，c (x), the mask branch learns the attention weight T i，c (x); Finally, the output of the trunk branch and the output of the mask branch are dot-producted to obtain the attention output feature map H i，c (x);

[0011] Step 4: Output the attention feature map H obtained in step 3 i，c (x) are sent to the transformer and non-local neural network for learning, and the dependency relationship Z between the transformer output feature maps trans , the dependency Z between spatial positions in the output feature map of the non-local neural network non-local ;

[0012] Step 5: The dependency Z between feature graphs trans and the dependency Z between the spatial positions in the feature map non-local Fusion is performed to obtain the output feature Z fusion ;

[0013] Step 6: Use a two-stage multi-task learning network to train a classifier and transform the output feature Z obtained in step 5 fusion Send it to the classifier to get the emotional state values: Arousal and Valence.

[0014] The present invention is also characterized in that:

[0015] Step 1 is implemented according to the following steps:

[0016] For the training sample set x train , the sample image is a tensor x of n×h×w dimensions train =[(h1,w1),(h2,w2),...,(h n , w n )], where n represents the total number of samples in the training sample set, h represents the length of each sample image, w represents the width of each sample image, and the original sentiment annotation value of the sample is an n×2-dimensional vector y a，v =[(a1, v1), (a2, v2),..., (a n , v n )], where (a n , v n ) represent the training sample set x train Arousal and Valence labels of the nth sample image in ;

[0017] For the test sample set x test , the sample image is a tensor x of m×h×w dimensions test =[(h1,w1),(h2,w2),...,(h m , w m )], where m represents the total number of samples in the test sample set, h represents the length of each sample image, w represents the width of each sample image, and the original sentiment annotation value of the sample is an m×2-dimensional vector y a，u =[(a1, v1), (a2, v2),..., (a m , v m )], where (a m , v m ) represent the test sample set x test Arousal and Valence labels of the mth sample image in .

[0018] Step 2 is implemented as follows:

[0019] Step 2.1: Read the training sample set x train Each sample image (h l , w l ), where l = 1, 2, ..., n, h l represents the length of the lth sample image, w lIndicates the width of the lth sample image. Then the input sample image is transformed into different scales to obtain a series of images of different sizes to construct image features. where x k Represents the features of the kth image;

[0020] Step 2.2, then the image features obtained in step 2.1 The multi-task convolutional neural network MTCNN is composed of three sub-networks, namely Proposal Network, RefineNetwork and Output Network. The Proposal Network consists of a four-layer convolutional neural network, the RefineNetwork consists of a three-layer convolutional neural network and two fully connected layers, and the Output Network consists of a four-layer convolutional neural network and two fully connected layers. Then the output feature x of MTCNN is obtained. input =[x1, x2, ..., x n ], where x n Represents the features of the nth image.

[0021] Step 3 is implemented as follows:

[0022] Step 3.1: Initialize the overall network architecture parameters, i.e. weights and biases, including all convolutional layers, pooling layers, and fully connected layers in the network;

[0023] Step 3.2: Output feature x input Send them to the trunk branch and mask branch respectively to get the output result M of the trunk branch i，c (x) and the output of the mask branch T i，c (x), the specific process is described as follows:

[0024] Main branch: output feature x input After two convolutional layers with a convolution kernel size of 3×3, and batch normalization BN and ReLU function activation are performed on the output results of each convolutional layer, the output result M of the trunk branch is obtained. i，c (x), the output result M i，c (x) is the output feature x input Characteristics of different receptive fields;

[0025] Assume that the output of the convolutional layer at layer l is z l , and then the final output o is obtained through batch normalization BN and ReLU activation function l , the specific calculation formula is as follows:

[0026] o l =ReLU(BN(zl ))=ReLU(BN(Wo l-1 +b)) (1)

[0027] W and b represent weight value and bias respectively, l = 1, 2;

[0028] Among them, BN is batch normalization, which is a method of normalizing the probability distribution of data. The calculation formula is as follows:

[0029]

[0030] In formula (2), x i Represents the output feature x input Any single sample image feature in x BN is the output feature after batch normalization, σ l is the standard deviation image of the l-th layer image features, μ l is the mean image of the image features of the lth layer;

[0031] σ in formula (2) l and μ l The definition is as follows:

[0032]

[0033]

[0034] In formulas (3) and (4), x k Image feature x input Any single sample image feature in , k represents the number of samples in each small batch, k ≥ 1;

[0035] Mask branch: output feature x input After two pooling operations and two upsampling operations, the sigmoid function is used to adjust the output to the range of 0 to 1, and the output result T of the mask branch is obtained. i，c (x), the output result is T i，c (x) represents feature M i，c The attention weight of (x);

[0036] Step 3.3: Output the trunk branch to M i，c (x) and the mask branch output T i，c (x) performs dot product and introduces the residual mechanism in this process to obtain the residual attention feature H i，c (x), the calculation formula is:

[0037] H i，c (x) = (1 + M i，c (x))*T i，c (x) (5)

[0038] Among them, H i，c (x) is the residual attention feature, M i，c (x) is the output feature of the trunk branch, T i，c (x) is the output feature of the mask branch.

[0039] Step 4 is implemented according to the following steps:

[0040] Step 4.1: First, the residual attention feature H obtained in step 3 i，c (x) is sent to Transformer to learn the dependency relationship Z between feature maps trans ;

[0041] Since the residual attention feature H i，c (x) has z channels, so the residual attention feature H i，c (x) is expressed as w×h×z, where w represents the length of the residual attention feature, h represents the width of the residual attention feature, and z represents the number of channels of the residual attention feature. Then the image feature H i，c The dimension of (x) becomes z×wh, which is convenient for feeding into Transformer;

[0042] Transformer is composed of multiple self-attention networks. Each self-attention network first calculates the query vector Q and the key-value pair vector KV through the weight matrix W and the input feature X. The calculation formula is:

[0043] Q=XW Q , K = XW K , V = XW V (6)

[0044] Where W Q represents the weight matrix of the query vector Q, W K represents the weight matrix of the key vector K, W V A weight matrix representing the value vector V.

[0045] Then the dot product model QK T Get the attention scoring function S(Q, K), normalize the score to between 0 and 1 through the softmax function, and get the attention weight value α:

[0046] α=softmax(QK T ) (7)

[0047] The attention weight value α is required, K T Represents the transposed vector of the key vector K.

[0048] Finally, the attention weight value α is weighted and summed with the key vector V to obtain the output feature. The calculation formula is:

[0049] Attention(Q,K,V)=softmax(QK T )V (8)

[0050] Where Attention(Q, K, V) represents the required output features, softmax is the activation function, Q is the query vector, K T represents the transposed vector of the key vector K, and V is the value vector.

[0051] In order to obtain multiple self-attention networks, according to the calculation steps shown in (8), h query vectors Q = [q1, q2, ..., q h ] are calculated with the input feature X, where h represents h self-attention networks, q h represents the query vector of the h-th self-attention network, and then merges the outputs of the h self-attention networks. This network composed of multiple self-attention networks is called a multi-head attention network, and its calculation formula is as follows:

[0052]

[0053] Among them, att((K, V), Q) is the output feature of the multi-head attention network, att((K, V), q h ) represents the output feature of the h-th self-attention network, It represents point-by-point addition of vectors;

[0054] Finally, a fully connected layer is used to complete the output and obtain the dependency relationship Z between the feature maps. trans ;

[0055] Step 4.2: Then use the residual attention feature H obtained in step 3 i，c (x) is fed into the local neural network to learn the dependency Z between spatial positions in the feature map non-local ,

[0056] For the residual attention feature H i，c (x) performs a 1×1 convolution and converts H i，c (x) is mapped to three different vector spaces, using θ, g represents these three spaces, and the weight matrix W of the corresponding vector space is obtained θ , W g , and then calculate the correlation score between two different pixel positions in the feature map space. The calculation formula is:

[0057]

[0058] Where f(x i , x j ) represents the correlation score, x i is the i-th pixel in the feature map, x j is the jth pixel in the feature map, W θ and is the vector space θ and The weight matrix of , the subscript T represents the transpose of the matrix;

[0059] x i 、x j The correlation score between two points and the characteristic representation function g(x j ) weighted sum to obtain the position dependency y for position i i , the calculation formula is:

[0060]

[0061] Among them, y i is the output calculated for position i, x is the input feature, C(x) is the normalization factor, g(x j ) is the input representation for computing position j;

[0062] According to the calculation steps shown in (11), the dependency y of all positions in the feature map is calculated i , and combined together to obtain the dependency Z between spatial positions in the feature map non-local .

[0063] Step 6 is implemented according to the following steps:

[0064] Step 6.1: In the first stage of multi-task learning, two branches are used to learn the output feature Z obtained in step 5 respectively. fusion Classification of Z class and dimension representation Z dim , where each branch consists of a fully connected layer, and then the learned classification representation and dimension representation of the two types of output features are cascaded into a feature Z mtl-1 ;

[0065] Step 6.2: In the second stage of multi-task learning, use a linear regressor to train feature Z mtl-1 Perform linear regression and get the output values: Arousal and Valence;

[0066] Among them, the loss function used by the linear regressor is Tukey's bi-weighted loss function, and the specific definition is as follows:

[0067]

[0068] In formula (14), Loss represents the loss value, c = 4.685 is a hyperparameter, and y i represents the true label value, Represents the predicted label value.

[0069] The beneficial effect of the present invention is that the present invention is a facial emotion recognition method based on the attention mechanism, which solves the problem that the existing facial emotion recognition method based on the attention mechanism lacks simultaneous modeling and learning of the dependency relationship between feature maps and the spatial position dependency relationship of features within the feature map. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is an overall network model architecture of a facial emotion recognition method based on an attention mechanism of the present invention.

[0071] Figure 2 It is the residual attention network model structure.

[0072] Figure 3 It is the Transformer model structure.

[0073] Figure 4 It is the process of converting the residual attention network output feature map into a sequence.

[0074] Figure 5 It is a multi-head attention network.

[0075] Figure 6 It is a non-local neural network module.

[0076] Figure 7 It is an emotional state value prediction module that integrates deep representation. DETAILED DESCRIPTION

[0077] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0078] The facial emotion recognition method based on the attention mechanism of the present invention is combined with Figure 1 to Figure 7 , follow the steps below:

[0079] Step 1: Collect images and divide them into training sample sets x train And the test sample set x test ;

[0080] Step 1 is implemented according to the following steps:

[0081] Each training sample and test sample has its corresponding original sentiment marker value.

[0082] For the training sample set x train , the sample image is a tensor x of n×h×w dimensionstrain =[(h1, W1), (h2, w2),..., (h n , w n )], where n represents the total number of samples in the training sample set, h represents the length of each sample image, w represents the width of each sample image, and the original sentiment annotation value of the sample is an n×2-dimensional vector y a，v =[(a1,v1)(a2,v2),...,(a n , v n )], where (a n , v n ) represent the training sample set x train Arousal and Valence labels of the nth sample image in ;

[0083] For the test sample set x test , the sample image is a tensor x of m×h×W dimensions test =[(h1,w1),(h2,w2),...,(h m , w m )], where m represents the total number of samples in the test sample set, h represents the length of each sample image, w represents the width of each sample image, and the original sentiment annotation value of the sample is an m×2-dimensional vector y a，v =[(a1, v1), (a2, v2),..., (a m , v m )], where (a m , v m ) represent the test sample set x test Arousal and Valence labels of the mth sample image in .

[0084] Step 2: Read the training sample set x train The original emotional annotation value of each sample image in the training sample set x train Each sample image in is sent to the multi-task convolutional neural network MTCNN, and face alignment is completed based on face detection and 5 key feature points to obtain the output feature x input =[x1, x2, ..., x n ];

[0085] Step 2 is implemented as follows:

[0086] Step 2.1: Read the training sample set x train Each sample image (h l , w l ), where l = 1, 2, ..., n, h l represents the length of the lth sample image, w lIndicates the width of the lth sample image. Then the input sample image is transformed into different scales to obtain a series of images of different sizes to construct image features. where x k Represents the features of the kth image;

[0087] Step 2.2, then the image features obtained in step 2.1 The multi-task convolutional neural network MTCNN is composed of three sub-networks, namely Proposal Network, RefineNetwork and Output Network. The Proposal Network consists of a four-layer convolutional neural network, the RefineNetwork consists of a three-layer convolutional neural network and two fully connected layers, and the Output Network consists of a four-layer convolutional neural network and two fully connected layers. Then the output feature x of MTCNN is obtained. input =[x1, x2, ..., x n ], where x n Represents the features of the nth image.

[0088] Step 3: The output feature x obtained after preprocessing input Input into the residual attention network, and then extract the features M of different receptive fields through the trunk branch i，c (x), the mask branch learns the attention weight T i，c (x); Finally, the output of the trunk branch and the output of the mask branch are dot-producted to obtain the attention output feature map H i，c (x); network structure such as Figure 2 The network architecture parameters are shown in Table 1.

[0089] Table 1 Residual attention network model parameters

[0090]

[0091] Step 3 is implemented as follows:

[0092] Step 3.1: Initialize the overall network architecture parameters, i.e. weights and biases, including all convolutional layers, pooling layers, and fully connected layers in the network;

[0093] Step 3.2: Output feature x input Send them to the trunk branch and mask branch respectively to get the output result M of the trunk branch i，c (x) and the output of the mask branch T i，c (x), the specific process is described as follows:

[0094] Main branch: output feature xinput After two convolutional layers with a convolution kernel size of 3×3, and batch normalization BN and ReLU function activation are performed on the output results of each convolutional layer, the output result M of the trunk branch is obtained. i，c (x), the output result M i，c (x) is the output feature x input Characteristics of different receptive fields;

[0095] Assume that the output of the convolutional layer at layer l is z l , and then the final output o is obtained through batch normalization BN and ReLU activation function l , the specific calculation formula is as follows:

[0096] o l =ReLU(BN(z l ))=ReLU(BN(Wo l-1 +b)) (1)

[0097] W and b represent weight value and bias respectively, l = 1, 2;

[0098] Among them, BN is batch normalization, which is a method of normalizing the probability distribution of data. The calculation formula is as follows:

[0099]

[0100] In formula (2), x i Represents the output feature x input Any single sample image feature in x BN is the output feature after batch normalization, σ l is the standard deviation image of the l-th layer image features, μ l is the mean image of the image features of the lth layer;

[0101] σ in formula (2) l and μ l The definition is as follows:

[0102]

[0103]

[0104] In formulas (3) and (4), x k Image feature x input Any single sample image feature in , k represents the number of samples in each small batch, k ≥ 1;

[0105] Mask branch: output feature x inputAfter two pooling operations and two upsampling operations, the sigmoid function is used to adjust the output to the range of 0 to 1, and the output result T of the mask branch is obtained. i，c (x), the output result is T i，c (x) represents feature M i，c The attention weight of (x);

[0106] Step 3.3: Output the trunk branch to M i，c (x) and the mask branch output T i，c (x) performs dot product and introduces the residual mechanism in this process to obtain the residual attention feature H i，c (x), the calculation formula is:

[0107] H i，c (x) = (1 + M i，c (x))*T i，c (x) (5)

[0108] Among them, H i，c (x) is the residual attention feature, M i，c (x) is the output feature of the trunk branch, T i，c (x) is the output feature of the mask branch.

[0109] Step 4: Output the attention feature map H obtained in step 3 i，c (x) are sent to the transformer and non-local neural network for learning, and the dependency relationship Z between the transformer output feature maps trans , the dependency Z between spatial positions in the output feature map of the non-local neural network non-local ;

[0110] Step 4 is implemented according to the following steps:

[0111] Step 4.1: First, the residual attention feature H obtained in step 3 i，c (x) is sent to Transformer to learn the dependency relationship Z between feature maps trans , the network structure is as follows Figure 3 shown.

[0112] Since the residual attention feature H i，c (x) has z channels, so the residual attention feature H i，c (x) is expressed as w×h×z, where w represents the length of the residual attention feature, h represents the width of the residual attention feature, and z represents the number of channels of the residual attention feature. Then the image feature H i，c The dimension of (x) becomes z×wh, which is convenient for feeding into Transformer. The conversion process is as follows: Figure 4 shown.

[0113] Transformer is composed of multiple self-attention networks. Each self-attention network first calculates the query vector Q and the key-value pair vector KV through the weight matrix W and the input feature X. The calculation formula is:

[0114] Q=XW Q , K = XW K , V = XW V (6)

[0115] Where W Q represents the weight matrix of the query vector Q, W K represents the weight matrix of the key vector K, W V A weight matrix representing the value vector V.

[0116] Then the dot product model QK T Get the attention scoring function S(Q, K), normalize the score to between 0 and 1 through the softmax function, and get the attention weight value α:

[0117] α=softmax(QK T ) (7)

[0118] The attention weight value α is required, K T Represents the transposed vector of the key vector K.

[0119] Finally, the attention weight value α is weighted and summed with the key vector V to obtain the output feature. The calculation formula is:

[0120] Attention(Q,K,V)=softmax(QK T )V (8)

[0121] Where Attention(Q, K, V) represents the required output features, softmax is the activation function, Q is the query vector, K T represents the transposed vector of the key vector K, and V is the value vector.

[0122] In order to obtain multiple self-attention networks, according to the calculation steps shown in (8), h query vectors Q = [q1, q2, ..., q h ] are calculated with the input feature X, where h represents h self-attention networks, q h Represents the query vector of the h-th self-attention network, and then merges the outputs of the h self-attention networks. This network composed of multiple self-attention networks is called a multi-head attention network. The network structure is as follows Figure 5 The calculation formula is as follows:

[0123]

[0124] Where att((K, V), Q) is the output feature of the multi-head attention network, att((K, V), q h ) represents the output feature of the h-th self-attention network, It represents point-by-point addition of vectors;

[0125] Finally, a fully connected layer is used to complete the output and obtain the dependency relationship Z between the feature maps. trans ;

[0126] Step 4.2: Then use the residual attention feature H obtained in step 3 i，c (x) is fed into the local neural network to learn the dependency Z between spatial positions in the feature map non-local , the network structure is as follows Figure 6 shown.

[0127] For the residual attention feature H i，c (x) performs a 1×1 convolution and converts H i，c (x) is mapped to three different vector spaces, using θ, g represents these three spaces, and the weight matrix W of the corresponding vector space is obtained θ , W g , and then calculate the correlation score between two different pixel positions in the feature map space. The calculation formula is:

[0128]

[0129] Where f(x i , x j ) represents the correlation score, x i is the i-th pixel in the feature map, x j is the jth pixel in the feature map, W θ and is the vector space θ and The weight matrix of , the subscript T represents the transpose of the matrix;

[0130] x i 、x j The correlation score between two points and the characteristic representation function g(x j ) weighted sum to obtain the position dependency y for position i i , the calculation formula is:

[0131]

[0132] Among them, y iis the output calculated for position i, x is the input feature, C(x) is the normalization factor, g(x j ) is the input representation for computing position j;

[0133] According to the calculation steps shown in (11), the dependency y of all positions in the feature map is calculated i , and combined together to obtain the dependency Z between spatial positions in the feature map non-local .

[0134] Step 5: The dependency Z between feature graphs trans and the dependency Z between the spatial positions in the feature map non-local Fusion is performed to obtain the output feature Z fusion ;

[0135] Step 6: Use a two-stage multi-task learning network to train a classifier and transform the output feature Z obtained in step 5 fusion Send it to the classifier to get the emotional state values: Arousal and Valence. The multi-task learning structure diagram is as follows Figure 7 shown.

[0136] Step 6 is implemented according to the following steps:

[0137] Step 6.1: In the first stage of multi-task learning, two branches are used to learn the output feature Z obtained in step 5 respectively. fusion Classification of Z class and dimension representation Z dim , where each branch consists of a fully connected layer, and then the learned classification representation and dimension representation of the two types of output features are cascaded into a feature Z mtl-1 ;

[0138] Step 6.2: In the second stage of multi-task learning, use a linear regressor to train feature Z mtl-1 Perform linear regression and get the output values: Arousal and Valence;

[0139] Among them, the loss function used by the linear regressor is Tukey's bi-weighted loss function, and the specific definition is as follows:

[0140]

[0141] In formula (14), Loss represents the loss value, c = 4.685 is a hyperparameter, and y i represents the true label value, Represents the predicted label value.

[0142] Example

[0143] The experiments of the present invention are conducted based on the AffectNet dataset, and then the performance of the present invention is evaluated and analyzed.

[0144] The experimental results are compared as follows:

[0145] The root mean square error (RMSE) and consistency correlation coefficient (CCC) are calculated for the label values ​​predicted by the model and the original label values, and then compared with the existing methods. The performance of different emotion recognition models is compared, and the results are shown in Table 2:

[0146] Table 2 Performance comparison of different network models

[0147]

[0148] From the data in the table, it can be seen that the facial expression recognition method based on the attention mechanism designed by the present invention is superior to the traditional method. This verifies the effectiveness of the present invention in modeling the dependencies between the output feature channels of the residual attention network and the dependencies between the spatial positions in the output feature map.

Claims

1. A facial emotion recognition method based on attention mechanism, characterized in that: Follow the steps below to implement it: Step 1: Collect images and divide them into training sample sets x train And the test sample set x test ; Step 2: Read the training sample set x train The original emotional annotation value of each sample image in the training sample set x train Each sample image in is sent to the multi-task convolutional neural network MTCNN, and face alignment is completed based on face detection and 5 key feature points to obtain the output feature x input =[x1, x2, ..., x n ]; Step 3: The output feature x obtained after preprocessing input Input into the residual attention network, and then extract the features M of different receptive fields through the trunk branch i，c (x), the mask branch learns the attention weight T i，c (x); Finally, the output of the trunk branch and the output of the mask branch are dot-producted to obtain the attention output feature map H i，c (x); Step 4: Output the attention feature map H obtained in step 3 i，c (x) are sent to the transformer and non-local neural network for learning, and the dependency relationship Z between the transformer output feature maps trans , the dependency Z between spatial positions in the output feature map of the non-local neural network non-local ; Step 5: The dependency Z between feature maps trans and the dependency Z between the spatial positions in the feature map non-local Fusion is performed to obtain the output feature Z fusion ; Step 6: Use a two-stage multi-task learning network to train a classifier and transform the output feature Z obtained in step 5 fusion Send it to the classifier to get the emotional state values: Arousal and Valence.

2. The facial emotion recognition method based on the attention mechanism according to claim 1, characterized in that: The step 1 is specifically implemented according to the following steps: For the training sample set x train , the sample image is a tensor x of n×h×w dimensions train =[(h1, W1), (h2, w2),..., (h n , w n )], where n represents the total number of samples in the training sample set, h represents the length of each sample image, w represents the width of each sample image, and the original sentiment annotation value of the sample is an n×2-dimensional vector y a，v =[(a1, v1), (a2, v2),..., (a n , v n )], where (a n , v n ) represent the training sample set x train Arousal and Valence labels of the nth sample image in ; For the test sample set x test , the sample image is a tensor x of m×h×W dimensions test =[(h1,w1),(h2,w2),...,(h m , w m )], where m represents the total number of samples in the test sample set, h represents the length of each sample image, w represents the width of each sample image, and the original sentiment annotation value of the sample is an m×2-dimensional vector y a，v =[(a1, v1), (a2, v2),..., (a m , v m )], where (a m , v m ) represent the test sample set x test Arousal and Valence labels of the mth sample image in .

3. The facial emotion recognition method based on attention mechanism according to claim 2, characterized in that: The step 2 is specifically implemented according to the following steps: Step 2.1: Read the training sample set x train Each sample image (h l , w l ), where l = 1, 2, ..., n, h l represents the length of the lth sample image, w l Indicates the width of the lth sample image; Then the input sample image is transformed into different scales to obtain a series of images of different sizes to construct image features where x k Represents the features of the kth image; Step 2.2, then the image features obtained in step 2.1 The multi-task convolutional neural network MTCNN is composed of three sub-networks, namely Proposal Network, Refine Network and Output Network. The Proposal Network consists of a four-layer convolutional neural network, the Refine Network consists of a three-layer convolutional neural network and two fully connected layers, and the Output Network consists of a four-layer convolutional neural network and two fully connected layers. Then the output feature x of MTCNN is obtained. input =[x1, x2, ..., x n ], where x n Represents the features of the nth image.

4. The facial emotion recognition method based on attention mechanism according to claim 3, characterized in that: The step 3 is specifically implemented according to the following steps: Step 3.1: Initialize the overall network architecture parameters, i.e. weights and biases, including all convolutional layers, pooling layers, and fully connected layers in the network; Step 3.2: Output feature x input Send them to the trunk branch and mask branch respectively to get the output result M of the trunk branch i，c (x) and the output of the mask branch T i，c (x), the specific process is described as follows: Main branch: output feature x input After two convolutional layers with a convolution kernel size of 3×3, and batch normalization BN and ReLU function activation are performed on the output results of each convolutional layer, the output result M of the trunk branch is obtained. i，c (x), the output result M i，c (x) is the output feature x input Characteristics of different receptive fields; Assume that the output of the convolutional layer at layer l is z l , and then the final output o is obtained through batch normalization BN and ReLU activation function l , the specific calculation formula is as follows: o l =ReLU(BN(z l ))=ReLU(BN(Wo l-1 +b)) (1) W and b represent weight value and bias respectively, l = 1, 2; Among them, BN is batch normalization, which is a method of normalizing the probability distribution of data. The calculation formula is as follows: In formula (2), x i Represents the output feature x input Any single sample image feature in x BN is the output feature after batch normalization, σ l is the standard deviation image of the l-th layer image features, μ l is the mean image of the image features of the lth layer; σ in formula (2) l and μ l The definition is as follows: In formulas (3) and (4), x k Image feature x input Any single sample image feature in , k represents the number of samples in each small batch, k ≥ 1; Mask branch: output feature x input After two pooling operations and two upsampling operations, the sigmoid function is used to adjust the output to the range of 0 to 1, and the output result T of the mask branch is obtained. i，c (x), the output result is T i，c (x) represents feature M i，c The attention weight of (x); Step 3.3: Output the trunk branch to M i，c (x) and the mask branch output T i，c (x) performs dot product and introduces the residual mechanism in this process to obtain the residual attention feature H i，c (x), the calculation formula is: H i，c (x)=(1+M i，c (x))*T i，c (x) (5) Among them, H i，c (x) is the residual attention feature, M i，c (x) is the output feature of the trunk branch, T i，c (x) is the output feature of the mask branch.

5. The facial emotion recognition method based on attention mechanism according to claim 4, characterized in that: The step 4 is specifically implemented according to the following steps: Step 4.1: First, the residual attention feature H obtained in step 3 i，c (x) is sent to Transformer to learn the dependency relationship Z between feature maps trans ; Since the residual attention feature H i，c (x) has z channels, so the residual attention feature H i，c (x) is expressed as w×h×z, where w represents the length of the residual attention feature, h represents the width of the residual attention feature, and z represents the number of channels of the residual attention feature. Then the image feature H i，c The dimension of (x) becomes z×wh, which is convenient for feeding into Transformer; Transformer is composed of multiple self-attention networks. Each self-attention network first calculates the query vector Q and the key-value pair vector KV through the weight matrix W and the input feature X. The calculation formula is: Q=XW Q ,K=XW K ,V=XW V (6) Where W Q represents the weight matrix of the query vector Q, W K represents the weight matrix of the key vector K, W V The weight matrix representing the value vector V; Then the dot product model QK T Get the attention scoring function S(Q, K), normalize the score to between 0 and 1 through the softmax function, and get the attention weight value α: α=softmax(QK T ) (7) The attention weight value α is required, K T represents the transposed vector of the key vector K; Finally, the attention weight value α is weighted and summed with the key vector V to obtain the output feature. The calculation formula is: Attention(Q,K,V)=softmax(QK T )V (8) Where Attention(Q, K, V) represents the required output features, softmax is the activation function, Q is the query vector, K T represents the transposed vector of the key vector K, and V is the value vector; In order to obtain multiple self-attention networks, according to the calculation steps shown in (8), h query vectors Q = [q1, q2, ..., q h ] are calculated with the input feature X, where h represents h self-attention networks, q h represents the query vector of the h-th self-attention network, and then merges the outputs of the h self-attention networks. This network composed of multiple self-attention networks is called a multi-head attention network, and its calculation formula is as follows: Where att((K, V), Q) is the output feature of the multi-head attention network, att((K, V), q h ) represents the output feature of the h-th self-attention network, It represents point-by-point addition of vectors; Finally, a fully connected layer is used to complete the output and obtain the dependency relationship Z between the feature maps. trans ; Step 4.2: Then use the residual attention feature H obtained in step 3 i，c (x) is fed into the local neural network to learn the dependency Z between spatial positions in the feature map non-local , For the residual attention feature H i，c (x) performs a 1×1 convolution and converts H i，c (x) is mapped to three different vector spaces, using θ, g represents these three spaces, and the weight matrix W of the corresponding vector space is obtained θ , W g , and then calculate the correlation score between two different pixel positions in the feature map space. The calculation formula is: Where f(x i , x j ) represents the correlation score, x i is the i-th pixel in the feature map, x j is the jth pixel in the feature map, W θ and is the vector space θ and The weight matrix of , the subscript T represents the transpose of the matrix; x i 、x j The correlation score between two points and the characteristic representation function g(x j ) weighted sum to obtain the position dependency y for position i i , the calculation formula is: Among them, y i is the output calculated for position i, x is the input feature, C(x) is the normalization factor, g(x j ) is the input representation for computing position j; According to the calculation steps shown in (11), the dependency y of all positions in the feature map is calculated i , and combined together to obtain the dependency Z between spatial positions in the feature map non-local .

6. The facial emotion recognition method based on attention mechanism according to claim 5, characterized in that: The step 6 is specifically implemented according to the following steps: Step 6.1: In the first stage of multi-task learning, two branches are used to learn the output feature Z obtained in step 5 respectively. fusion Classification of Z class and dimension representation Z dim , where each branch consists of a fully connected layer, and then the learned classification representation and dimension representation of the two types of output features are cascaded into a feature Z mtl-1 ; Step 6.2: In the second stage of multi-task learning, use a linear regressor to train feature Z mtl-1 Perform linear regression and get the output values: Arousal and Valence; Among them, the loss function used by the linear regressor is Tukey's bi-weighted loss function, and the specific definition is as follows: In formula (14), Loss represents the loss value, c = 4.685 is a hyperparameter, and y i represents the true label value, Represents the predicted label value.

Images