A semi-supervised facial emotion recognition method

Through the semi-supervised learning method, the CNN-LSTM regressor and triple loss of similarity learning are used to generate smooth pseudo-labels, solving the problem of insufficient tag data in face emotion recognition in deep learning, and achieving efficient and accurate faces Emotion recognition, especially on continuous domain datasets.

CN114529973BActive Publication Date: 2024-11-05WUXI GONGYU FUTURE EDUCATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210160626.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-22
Publication Date
2024-11-05
Estimated Expiration
2042-02-22

AI Technical Summary

Technical Problem

In the prior art, in facial emotion recognition, deep learning methods require a large amount of label data, which causes a lot of manpower and material resources to be consumed during the training process. The data of discrete domain data sets is limited, making it difficult to express rich human emotions. The tag acquisition cost of continuous domain data sets is high. Training is difficult.

Method used

Using semi-supervised learning method, a smooth pseudo-label is generated through a CNN-LSTM regressor combined with fine-tuning MixUp and time sets to generate smooth pseudo-labels, and triple-loss based on similarity learning is constructed, and a weighted update is used for supervised loss, semi-supervised loss and similarity triple-loss weighted update. Network parameters, obtain semi-supervised face emotion recognition model.

Benefits of technology

Efficient and accurate facial emotion recognition is achieved in continuous domain data sets, and excellent recognition rate is achieved in discrete domain data sets. By generating reliable pseudo-labels, it alleviates the performance loss caused by insufficient label data, and improves the recognition rate of mood changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114529973B_ABST
    Figure CN114529973B_ABST
Patent Text Reader

Abstract

The present invention discloses a semi-supervised facial emotion recognition method, including obtaining basic image data samples; predicting label data, calculating the distance between the predicted value and the label value as the supervised loss; generating smooth pseudo labels from unlabeled data, and using the distance between the smooth pseudo labels and the predicted value as the semi-supervised loss; constructing a triplet loss based on similarity learning, including clustering the deep features of the image data, constructing triples, calculating the similarity triplet loss, constructing a complete loss function, and updating network parameters according to gradient descent; obtaining a semi-supervised facial emotion recognition model, and performing emotion analysis on the current facial image data. The present invention has accurate and efficient recognition in continuous domain data sets, and at the same time achieves excellent recognition rate in discrete domain data sets by adjusting the end; the present invention can learn the similarity between facial images through the triplet loss function based on similarity learning, thereby improving the recognition rate of emotional changes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of computer technology, and in particular relates to a semi-supervised facial emotion recognition method. Background Art

[0002] Accurate recognition of human emotions can help artificial intelligence achieve human-computer interaction. At present, the mainstream methods of emotion recognition include feature-based methods and deep learning-based methods. The feature extraction method is mainly based on oriented gradient histograms and local binary patterns, but these features are not suitable for the nonlinear structure of data and have low accuracy. Using deep learning technology, emotion recognition based on facial expressions has been significantly improved. Deep learning technology requires a large amount of labeled data to train neural networks, which will result in a lot of manpower and material resources being consumed during the training process. Currently, common data sets include discrete domain data sets and continuous domain data sets. Among them, discrete domain data sets contain limited data and it is difficult to express rich human emotions, while the cost of obtaining labels for continuous domain data sets is high and training is difficult. Summary of the invention

[0003] The purpose of the present invention is to provide a semi-supervised facial emotion recognition method (Semi-supervised Emotion Recognition, referred to as Semi Emotion), which can simply and efficiently recognize a variety of facial emotions.

[0004] The semi-supervised facial emotion recognition method provided by the present invention comprises the following steps:

[0005] S1. Obtain basic image data samples, including labeled data and unlabeled data;

[0006] S2. Use CNN-LSTM (Convolutional Neural Network-Long Short-Term Memory Network) regressor to predict the label data and calculate the distance between the predicted value and the label value as the supervision loss;

[0007] Through the CNN-LSTM regressor, we use fine-tuning MixUp and temporal integration to generate smooth pseudo-labels for unlabeled data, and use the distance between the smoothed pseudo-labels and the predicted values ​​as the semi-supervised loss;

[0008] S3. Constructing a triplet loss based on similarity learning, including clustering the deep features of the image data, constructing triplets based on the clustering results, calculating the similarity triplet loss by the similarity between the triplets, and updating the network parameters according to the gradient descent;

[0009] S4. The complete loss function of the CNN-LSTM regressor is constructed by weighting the supervised loss, semi-supervised loss and similarity triplet loss. The network parameters are updated according to the loss value and gradient descent to obtain the semi-supervised facial emotion recognition model, and the emotion analysis is performed on the current facial image data.

[0010] The CNN-LSTM regressor includes, in each frame, using the S3D face detection algorithm to detect the face area of ​​the image; using the ResNet-based CNN encoder to extract deep features; before introducing the LSTM, adjusting the length of the extracted features to 2048 through the fully connected layer; after introducing the LSTM, outputting the excitation and the valence respectively by two different fully connected layers; the LSTM includes a forget gate, an input gate and an output gate.

[0011] The step S2 includes performing data expansion on the unlabeled data and then taking the average of the predictions of the expanded data; using fine-tuning MixUp to mix the labeled data with the unlabeled data to obtain the target data; the network predicts the target data and calculates the loss function.

[0012] The step S3 comprises the following steps:

[0013] A1. Clustering the deep features of the image:

[0014] A2. Calculate the weighted item through the sample features extracted by CNN after one iteration, and add the weighted item to the cluster center for update;

[0015] A3. Given a batch of samples, construct triplets for the data in the batch of samples, obtain the activation matrix of the data in the clustering space, and calculate the similarity and difference losses between the data; obtain the final similarity triplet loss through the similarity and difference losses between the data.

[0016] The step A1 includes clustering the deep features of the initial labeled data for a given basic image data sample to obtain an initial cluster center; during the training process, the unlabeled data and the labeled data are simultaneously input into the CNN-LSTM regressor, and the cluster center with the smallest distance from the basic image data sample is calculated. At the same time, the cluster category of the basic image data sample is determined by the cluster center with the smallest distance therefrom.

[0017] The step A2 includes calculating the weighted item based on the sample features extracted by CNN after one iteration, including the product of the sum of the initial clustering depth features and the reciprocal of the cluster center with the smallest distance from the basic image data sample, and the sum of the products of the sample features extracted by CNN after one iteration and the reciprocal of the cluster center with the smallest distance from the basic image data sample after one iteration, and iterating until a predetermined number of iteration terminations; the distance between samples in the clustering space after the iteration is the Euclidean norm of the feature vectors of two different basic image data samples.

[0018] The step A3 includes constructing a triplet given a batch of samples, wherein the triplet includes the label data in each batch, the data with the farthest distance from the same class of the label data in each batch in the cluster space, and the data with the closest distance from the different class of the label data in each batch in the cluster space; the encoding of the activation mode of the activation map by the label data in each batch is the product of the reshaped matrix of the activation channel of the label data in each batch in the CNN and its transpose; the encoding of the activation mode of the activation map by the label data in each batch is normalized row by row to obtain the activation matrix of the label data in each batch; and the activation matrix of the data with the farthest distance from the same class of the label data in each batch in the cluster space and the activation matrix of the data with the closest distance from the different class of the label data in each batch in the cluster space are solved at the same time;

[0019] The similarity between data is defined as the difference between activation matrices, and the difference between activation matrices is solved, including: solving the difference between the activation matrix of the labeled data in each batch and the activation matrix of the data farthest from the same type of labeled data in each batch in the cluster space;

[0020] Calculate the difference loss, including: solving the difference loss of the activation matrix of the labeled data in each batch and the activation matrix of the data closest to the heterogeneous data in each batch in the cluster space;

[0021] The similarity triplet loss is obtained as: the difference between the activation matrix of the labeled data in each batch and the activation matrix of the data farthest from the same class of the labeled data in each batch in the cluster space, minus the difference loss between the activation matrix of the labeled data in each batch and the activation matrix of the data closest to the heterogeneous class of the labeled data in each batch in the cluster space.

[0022] The step S2 comprises:

[0023] For unlabeled data, the CNN-LSTM regressor is made to learn from smoothed labels and data augmentation is first performed on the unlabeled data: u represents unlabeled data; augment(·) represents an augmentation operation; represents the expanded unlabeled data; k represents the number of data expansion; for the expanded unlabeled data Take the mean;

[0024]

[0025] Where q represents the predicted distribution after taking the mean; k represents the number of data expansions; i represents the count variable of the given basic image data sample; P model represents the deep network model; u i represents a certain picture; θ represents the network parameters;

[0026] Mix the labeled data (x1, p) and the unlabeled data (u1, q), where x1 represents the labeled image, p represents the label of the labeled image, u1 represents the unlabeled image, and q represents the predicted distribution obtained by taking the mean after expanding the unlabeled image data, and output the target data x′ represents the data obtained by mixing labeled images and unlabeled images; The distribution of the label image and the predicted distribution obtained by taking the average after the label image data is expanded; the target data As learning objectives:

[0027] x′=λ′x1+(1-λ′)u1

[0028]

[0029] Where α represents a hyperparameter; λ′=max(λ, 1-λ), λ~Bata(α, α), λ represents a value randomly taken from the Beta distribution, λ′ represents the maximum value between λ and 1-λ; max(·) represents the maximum value; Beta(·) represents the Beta distribution;

[0030] For target data The prediction value P′ is obtained by prediction. In order to obtain information from different angles, the CNN-LSTM regressors with different training periods are used to vote for the pseudo labels:

[0031]

[0032] Among them, P represents the final pseudo label; P n-model (y|x′; θ) represents the CNN-LSTM regressor before n iterations; the semi-supervised loss L is calculated using the mean square error between the predicted value P′ and the final pseudo-label P. u :

[0033] L u =||P′-P||2

[0034] Among them, ||·||2 represents the mean square error;

[0035] Use cross entropy to calculate the supervision loss L x :

[0036]

[0037] Among them, P nodel (y|x; θ) represents the prediction of the CNN-LSTM regressor for the labeled data; H(·) represents the cross entropy loss function.

[0038] In step A1, the clustering process includes for a given i-th basic image data sample y i , the eigenvector is x i ={x1, x2, ..., x 2048}; After clustering the deep features of the initial N label data, k initial cluster centers C = {C1, C2, ..., C k , ..., C K}; Among them, C k The length of the feature vector is 2048. During the training process, the unlabeled data and the labeled data are simultaneously input into the CNN-LSTM regressor to calculate the i-th basic image data sample y i The cluster center C with the smallest distance k The distance d ik , while the i-th basic image data sample y i The cluster category is determined by the cluster center C with the smallest distance to it. k Decide:

[0039]

[0040] C k = arg min k∈{1,2,...,K} d ik

[0041] Among them, C k Represents the i-th basic image data sample y i The cluster center with the smallest distance; arg min(·) represents the minimum distance; k represents the number of initial cluster centers; d ik Represents the i-th basic image data sample y i The eigenvector x i The distance to the kth cluster center; represents the Euclidean norm;

[0042] The step A2 comprises: calculating a weighted term according to the sample features extracted by CNN after one iteration:

[0043]

[0044] in, It means that after one iteration, it is the same as the i-th basic image data sample y i The cluster center with the smallest distance; X represents the initial clustering depth feature; X (1) Represents the sample features extracted by CNN after one iteration, and iterates until the predetermined number of iterations ends; C k Represents the i-th basic image data sample y i The cluster center with the smallest distance;

[0045] Calculate the distance between samples in the clustering space after iteration;

[0046]

[0047] in, Represents the i-th basic image data sample y after one iteration i The cluster center C with the smallest distance k The distance of x i Represents the i-th basic image data sample y i The eigenvector of j Represents the jth basic image data sample y j The eigenvector of represents the Euclidean norm;

[0048] The step A3 includes constructing a triple (N1, N2, N3) given a batch of samples N, where N1 represents the label data in each batch; N2 represents the data with the farthest distance from the same class of the label data N1 in each batch in the clustering space; N3 represents the data with the closest distance from the different class of the label data N1 in each batch in the clustering space; let is the activation channel of the lth layer in CNN for the label data N1 in each batch; R represents tensor space; C represents the number of channels; h represents height; w represents width; let As the activation channel of the lth layer in CNN for the label data N1 in each batch The reshaped matrix, Find:

[0049]

[0050] in, Indicates the encoding of the activation mode of the activation map by the i-th row and j-th column of the label data N1 in each batch; the encoding of the activation mode of the activation map by the i-th row and j-th column of the label data N1 in each batch Normalize row by row to obtain the activation matrix of label data N1 in each batch Solve the activation matrix of the data N2 that is farthest from the same type of labeled data N1 in each batch in the cluster space And the activation matrix of the data N3 that is closest to the heterogeneous data N1 in each batch in the cluster space

[0051] set up It is the activation channel of the lth layer in CNN of the data N2 that is farthest from the same type of label data N1 in each batch in the cluster space; R represents the tensor space; c represents the number of channels; h represents the height; w represents the width; let As the activation channel of the lth layer in CNN, the data N2 in the cluster space that is farthest from the same type of labeled data N1 in each batch is The reshaped matrix, Find:

[0052]

[0053] in, Indicates the encoding of the activation pattern of the activation map by the i-th row and j-th column of the data N2 in the cluster space that is the farthest from the same type of the labeled data N1 in each batch; Encoding of the activation pattern of the activation map by the i-th row and j-th column of the data N2 in the cluster space that is the farthest from the same type of the labeled data N1 in each batch Normalize row by row to obtain the activation matrix of data N2 in the cluster space that is farthest from the same type of label data N1 in each batch

[0054] set up It is the activation channel of the lth layer in CNN of the data N3 that is closest to the heterogeneous data N1 in each batch in the cluster space; R represents the tensor space; c represents the number of channels; h represents the height; w represents the width; let As the activation channel of the lth layer in CNN, the data N3 in the cluster space that is closest to the heterogeneous data N1 in each batch is The reshaped matrix, Find:

[0055]

[0056] in, Indicates the encoding of the activation pattern of the activation map by the i-th row and j-th column of the data N3 in the cluster space that is closest to the labeled data N1 in each batch; Encoding of the activation pattern of the activation map by the i-th row and j-th column of the data N3 in the cluster space that is closest to the labeled data N1 in each batch Normalize row by row to obtain the activation matrix of the data N3 in the cluster space that is closest to the heterogeneous data N1 in each batch.

[0057] Define the similarity between data as the difference between activation matrices, and solve the difference between activation matrices, including: solving the activation matrix of label data N1 in each batch And the activation matrix of the data N2 that is farthest from the same type of labeled data N1 in each batch in the cluster space The difference between si (N1, N2):

[0058]

[0059] Among them, B represents a triple; N1 represents the label data in each batch; N2 represents the data in the clustering space that is farthest from the same type of label data N1 in each batch; represents the Frobenius norm;

[0060] Calculate the difference loss, including: solve the activation matrix of label data N1 in each batch And the activation matrix of the data N3 that is closest to the heterogeneous data N1 in each batch in the cluster space The difference loss is:

[0061]

[0062] Among them, B represents a triple; N1 represents the label data in each batch; N3 represents the data in the cluster space that is closest to the label data N1 in each batch; represents the Frobenius norm;

[0063] The similarity triplet loss is obtained as:

[0064] L cs =L si (N1, N2)-L di (N1, N3)

[0065] Among them, L cs represents the similarity triplet loss; L si (N1, N2) represents the activation matrix of label data N1 in each batch And the activation matrix of the data N2 that is farthest from the same type of labeled data N1 in each batch in the cluster space The difference between di (N1, N3) represents the activation matrix of label data N1 in each batch And the activation matrix of the data N3 that is closest to the heterogeneous data N1 in each batch in the cluster space difference loss.

[0066] The step S4 includes that the CNN-LSTM regressor loss function is L=L x +L u +ω u L cs , where L x represents the supervision loss; L u represents the semi-supervised loss; ω u represents a hyperparameter; L cs represents the similarity triplet loss.

[0067] The semi-supervised facial emotion recognition method provided by the present invention can accurately and efficiently recognize emotions in continuous domain data sets, and can also achieve excellent recognition rates in discrete domain data sets by adjusting the end points. The present invention can learn the similarities between facial images through a triplet loss function based on similarity learning, thereby improving the recognition rate of emotional changes. The present invention generates reliable pseudo-labels for unlabeled data, and uses pseudo-label data to train the network, thereby alleviating the performance loss caused by insufficient labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a schematic diagram of the process of the present invention.

[0069] Figure 2 Schematic diagram of the CNN-LSTM regressor detection process of the method of the present invention.

[0070] Figure 3 Schematic diagram of the convolutional neural network process of the method of the present invention. DETAILED DESCRIPTION

[0071] like Figure 1 The semi-supervised facial emotion recognition method provided by the present invention comprises the following steps:

[0072] S1. Obtain basic image data samples, including labeled data and unlabeled data;

[0073] S2. Use CNN-LSTM regressor (convolutional neural network-long short-term memory network) to predict the label data and calculate the distance between the predicted value and the label value as the supervision loss;

[0074] Through the CNN-LSTM regressor, fine-tuning Mixup and temporal integration are used to generate smooth pseudo labels for unlabeled data, and the distance between the smoothed pseudo labels and the predicted values ​​is used as the semi-supervised loss;

[0075] S3. Constructing a triplet loss based on similarity learning, including clustering the deep features of the image data, constructing triplets based on the clustering results, calculating the similarity triplet loss by the similarity between the triplets, and updating the network parameters according to the gradient descent;

[0076] S4. The complete loss function of the CNN-LSTM regressor is constructed by weighting the supervised loss, semi-supervised loss and similarity triplet loss. The network parameters are updated according to the loss value and gradient descent to obtain the semi-supervised facial emotion recognition model, and the emotion analysis is performed on the current facial image data.

[0077] like Figure 2 The CNN-LSTM regressor detection flow diagram of the method of the present invention is shown in FIG. The CNN-LSTM regressor includes detecting the face area of ​​the image using the S3D face detection algorithm in each frame; extracting deep features using a CNN encoder based on ResNet; ResNet combines the characteristics of several convolutional layers to produce dense connections, and at the same time, ResNet can retain the underlying features of the image while preventing the gradient from disappearing. Before the introduction of LSTM, the length of the extracted features is adjusted to 2048 by the fully connected layer. After the introduction of LSTM, the two fully connected layers output the excitation and valence respectively. LSTM includes a forget gate, an input gate, and an output gate; the forget gate is used to discard useless information, the input gate can selectively save information, and the output gate is used to determine the output information. This structure allows LSTM to use the knowledge of the previous node to predict the current node. LSTM is selected because although emotions are output by only a single image, considering the changes in emotions in the continuous domain, continuous time signals become key information.

[0078] Because the AFEW-VA dataset extracts emotional information from a limited number of objects, it is difficult to learn sufficient feature diversity from the AFEW-VA dataset. In order to avoid overfitting, the ImageNet dataset is selected to pre-train the network. In addition, the pre-trained network is fine-tuned on the FER2013 dataset to obtain parameters suitable for facial emotion recognition. During the fine-tuning process, the parameters of the first half of the convolutional layer are frozen, and the parameters of the second half of the convolutional layer are learned through back propagation.

[0079] like Figure 3 The convolutional neural network process diagram of the method of the present invention is shown in FIG. 1 . The step S2 includes:

[0080] For unlabeled data, the CNN-LSTM regressor is made to learn from smoothed labels and data augmentation is first performed on the unlabeled data: u represents unlabeled data; augment(·) represents an augmentation operation; represents the expanded unlabeled data; k represents the number of data expansions; in order to correct the prediction, the expanded unlabeled data Take the mean;

[0081]

[0082] Where q represents the predicted distribution after taking the mean; k represents the number of data expansions; i represents the count variable of the given basic image data sample; P model represents the deep network model; u i represents a certain picture; θ represents the network parameters;

[0083] Mix the labeled data (x1, p) and the unlabeled data (u1, q), where x1 represents the labeled image, p represents the label of the labeled image, u1 represents the unlabeled image, and g represents the predicted distribution obtained by taking the mean after expanding the unlabeled image data, and outputs the target data x′ represents the data obtained by mixing labeled images and unlabeled images; The distribution of the label image and the predicted distribution obtained by taking the average after the label image data is expanded; the target data As learning objectives:

[0084] x′=λ′x1+(1-λ′)u1

[0085]

[0086] Where α represents a hyperparameter; λ′=max(λ, 1-λ), λ~Bata(α, α); λ represents a value randomly taken from the Beta distribution, λ′ represents the maximum value between λ and 1-λ, which is used as the balance coefficient; max(·) represents the maximum value; Beta(·) represents the Beta distribution;

[0087] For target data The prediction value P′ is obtained by prediction. In order to obtain information from different angles, the CNN-LSTM regressors with different training periods are used to vote for the pseudo labels:

[0088]

[0089] Among them, P represents the final pseudo label; P n-model(y|x′; θ) represents the CNN-LSTM regressor before n iterations; the semi-supervised loss L is calculated using the mean square error between the predicted value P′ and the final pseudo-label P. u :

[0090] L u =||P′-P||2

[0091] Among them, ||·||2 represents the mean square error;

[0092] Use cross entropy to calculate the supervision loss L x :

[0093]

[0094] Among them, P model (y|x; θ) represents the prediction of the CNN-LSTM regressor for the labeled data; H(·) represents the cross entropy loss function.

[0095] The step S3 comprises the following steps:

[0096] A1. Clustering the deep features of the image:

[0097] In step A1, the clustering process includes selecting better triples for similarity learning. This method uses an adjusted clustering method to cluster deep features. For a given i-th basic image data sample y i , the eigenvector is x i ={x1, x2, ..., x 2048}; After clustering the deep features of the initial N label data, k initial cluster centers C = {C1, C2, ..., C k , ..., C K}; Among them, C k The length of the feature vector is 2048. During the training process, the unlabeled data and the labeled data are simultaneously input into the CNN-LSTM regressor to calculate the i-th basic image data sample y i The cluster center C with the smallest distance k The distance d ik , while the i-th basic image data sample y i The cluster category is determined by the cluster center C with the smallest distance to it. k Decide:

[0098]

[0099] C k = arg min k∈{1,2,...,K} d ik

[0100] Among them, C k Represents the i-th basic image data sample y i The cluster center with the smallest distance; arg min(·) represents the minimum distance; k represents the number of initial cluster centers; d ik Represents the i-th basic image data sample y i The eigenvector x i The distance to the kth cluster center; represents the Euclidean norm;

[0101] A2. Calculate the weighted item based on the sample features extracted by CNN after one iteration, and add the weighted item to the cluster center for update; Calculate the weighted item based on the sample features extracted by CNN after one iteration:

[0102]

[0103] in, It means that after one iteration, it is the same as the i-th basic image data sample y i The cluster center with the smallest distance; X represents the initial clustering depth feature; X (1) represents the sample features extracted by CNN after one iteration, and the iteration continues until the predetermined number of iterations ends. In this embodiment, one iteration represents one round of deep neural network training, and the number of iterations is set before the network training starts; C k Represents the i-th basic image data sample y i The cluster center with the smallest distance;

[0104] Calculate the distance between samples in the clustering space after iteration;

[0105]

[0106] in, Represents the i-th basic image data sample y after one iteration i The cluster center C with the smallest distance k The distance of x i Represents the i-th basic image data sample y i The eigenvector of j Represents the jth basic image data sample y j The eigenvector of represents the Euclidean norm;

[0107] A3. Given a batch of samples, construct triplets for the data in the batch samples, obtain the activation matrix of the data in the cluster space, and calculate the similarity and difference losses between the data; obtain the final similarity triplet loss through the similarity and difference losses between the data;

[0108] The step A3 includes constructing a triple (N1, N2, N3) given a batch of samples N, where N1 represents the label data in each batch; N2 represents the data with the farthest distance from the same class of the label data N1 in each batch in the clustering space; N3 represents the data with the closest distance from the different class of the label data N1 in each batch in the clustering space; let is the activation channel of the lth layer in CNN for the label data N1 in each batch; R represents the tensor space; c represents the number of channels; h represents the height; w represents the width; let As the activation channel of the lth layer in CNN for the label data N1 in each batch The reshaped matrix, Find:

[0109]

[0110] in, Indicates the encoding of the activation mode of the activation map by the i-th row and j-th column of the label data N1 in each batch; the encoding of the activation mode of the activation map by the i-th row and j-th column of the label data N1 in each batch Normalize row by row to obtain the activation matrix of label data N1 in each batch Similarly, the activation matrix of the data N2 that is farthest from the same type of labeled data N1 in each batch can be solved in the cluster space: And the activation matrix of the data N3 that is closest to the heterogeneous data N1 in each batch in the cluster space Right now:

[0111] set up It is the activation channel of the lth layer in CNN of the data N2 that is farthest from the same type of label data N1 in each batch in the cluster space; R represents the tensor space; c represents the number of channels; h represents the height; w represents the width; let As the activation channel of the lth layer in CNN, the data N2 in the cluster space that is farthest from the same type of labeled data N1 in each batch is The reshaped matrix, Find:

[0112]

[0113] in, Indicates the encoding of the activation pattern of the activation map by the i-th row and j-th column of the data N2 in the cluster space that is the farthest from the same type of the labeled data N1 in each batch; Encoding of the activation pattern of the activation map by the i-th row and j-th column of the data N2 in the cluster space that is the farthest from the same type of the labeled data N1 in each batch Normalize row by row to obtain the activation matrix of data N2 in the cluster space that is farthest from the same type of label data N1 in each batch

[0114] set up It is the activation channel of the lth layer in CNN of the data N3 that is closest to the heterogeneous data N1 in each batch in the cluster space; R represents the tensor space; c represents the number of channels; h represents the height; w represents the width; let As the activation channel of the lth layer in CNN, the data N3 in the cluster space that is closest to the heterogeneous data N1 in each batch is The reshaped matrix, Find:

[0115]

[0116] in, Indicates the encoding of the activation pattern of the activation map by the i-th row and j-th column of the data N3 in the cluster space that is closest to the labeled data N1 in each batch; Encoding of the activation pattern of the activation map by the i-th row and j-th column of the data N3 in the cluster space that is closest to the labeled data N1 in each batch Normalize row by row to obtain the activation matrix of the data N3 in the cluster space that is closest to the heterogeneous data N1 in each batch.

[0117] Define the similarity between data as the difference between activation matrices, and solve the difference between activation matrices, including: solving the activation matrix of label data N1 in each batch And the activation matrix of the data N2 that is farthest from the same type of labeled data N1 in each batch in the cluster space The difference between si (N1, N2):

[0118]

[0119] Among them, B represents a triple; N1 represents the label data in each batch; N2 represents the data in the clustering space that is farthest from the same type of label data N1 in each batch; represents the Frobenius norm;

[0120] Calculate the difference loss, including: solve the activation matrix of label data N1 in each batch And the activation matrix of the data N3 that is closest to the heterogeneous data N1 in each batch in the cluster space The difference loss L di (N1, N3):

[0121]

[0122] Among them, B represents a triple; N1 represents the label data in each batch; N3 represents the data in the cluster space that is closest to the label data N1 in each batch; represents the Frobenius norm;

[0123] The similarity triplet loss is obtained as:

[0124] L cs =L si (N1, N2)-L di (N1, N3)

[0125] Among them, L cs represents the similarity triplet loss; L si (N1, N2) represents the activation matrix of label data N1 in each batch And the activation matrix of the data N2 that is farthest from the same type of labeled data N1 in each batch in the cluster space The difference between di (N1, N3) represents the activation matrix of label data N1 in each batch And the activation matrix of the data N3 that is closest to the heterogeneous data N1 in each batch in the cluster space difference loss.

[0126] The step S4 includes that the CNN-LSTM regressor loss function is L=L x +L u +ω u L cs , where L x represents the supervision loss; L u represents the semi-supervised loss; ω u represents a hyperparameter, which serves as a balance coefficient; L cs represents the similarity triplet loss.

[0127] In this embodiment, this method (Semi Emotion) is experimented on the AFEW-VA continuous domain emotion dataset. The AFEW-VA dataset consists of 600 videos selected from movies and TV series, and each frame is annotated with a label. Valence and Arousal are used as labels of the continuous domain dataset, and their sizes are in the range of [-10,10]. For the convenience of the experiment, we adjust the label value size of the AFEW-VA dataset to [-1,1]. The K-fold cross-validation method is used to evaluate the performance of Semi Emotion. First, the entire dataset is randomly divided into 5 subsets. Then, one subset is used as a test set, and the remaining subsets are used as training sets. 5 pairs of test sets and training sets are generated from different subsets. Finally, the average value of the 5 experiments is used as the final result, and the root mean square error is used as the indicator for experimental evaluation. The experiment uses the TensorFlow framework to implement Semi Emotion, and the SGD optimizer is adopted. The number of data augmentation k is set to 5, the hyperparameter α is set to 0.5, and ω u The initial setting is 1, and it decays to 0.5 after 50 iterations. The learning rate is set to 0.001, and it is reduced by 1 / 8 after every 20 iterations. The maximum number of iterations is set to 200. In order to avoid overfitting and obtain network parameters with good diversity, the network is first pre-trained using the large-scale dataset ImageNet. Then, in order to make the parameters suitable for face images, the pre-trained network is fine-tuned on the discrete domain emotion dataset FER2013 (the parameters of the first half of the network layers are frozen, and the parameters of the second half of the network layers are learned by back propagation. The fine-tuning is only iterated 50 times on the FER2013 dataset). After fine-tuning, the network iteration officially begins. Finally, the experimental results are compared with Choi's method. Table 1 shows the comparison of results on the AFEW-VA dataset, and Table 2 shows the comparison of cross-validation results on the AFEW-VA dataset.

[0128] Table 1

[0129]

[0130] Table 2

[0131]

[0132] As shown in Tables 1 and 2, the experimental results prove that this method, Semi Emotion, has a good recognition effect for continuous domain emotion recognition, and has smaller fluctuations in multiple experiments and is more stable in the emotion recognition process. A new triplet loss function is introduced to help the model learn the similarity of data. By enhancing the perception of intermediate layer features, this new triplet loss function can help the model better encode facial images for emotion recognition.

Claims

1. A semi-supervised facial emotion recognition method, characterized in that The steps include: S1. Obtain basic image data samples, including labeled data and unlabeled data; S2. Use CNN-LSTM regressor to predict the label data and calculate the distance between the predicted value and the label value as the supervision loss; Through the CNN-LSTM regressor, we use fine-tuning MixUp and temporal integration to generate smooth pseudo-labels for unlabeled data, and use the distance between the smoothed pseudo-labels and the predicted values ​​as the semi-supervised loss; S3. Constructing a triplet loss based on similarity learning, including clustering the deep features of the image data, constructing triples based on the clustering results, calculating the similarity triplet loss by the similarity between the triplets, and updating the network parameters according to the gradient descent; including the following steps: A1. Clustering the deep features of the image: A2. Calculate the weighted item through the sample features extracted by CNN after one iteration, and add the weighted item to the cluster center for update; A3. Given a batch of samples, construct triplets for the data in the batch of samples, obtain the activation matrix of the data in the clustering space, and calculate the similarity and difference losses between the data; obtain the final similarity triplet loss through the similarity and difference losses between the data; In a specific implementation, given a batch of samples, a triple is constructed, the triple includes the label data in each batch, the data with the farthest distance from the same class of the label data in each batch in the clustering space, and the data with the closest distance from the different class of the label data in each batch in the clustering space; the activation mode of the activation map encoded by the label data in each batch is the product of the reshaped matrix of the activation channel of the label data in each batch in the CNN and its transpose; The encoding of the activation pattern of the activation map by the label data in each batch is normalized row by row to obtain the activation matrix of the label data in each batch; At the same time, solve the activation matrix of the data farthest from the same class of labeled data in each batch in the cluster space, and the activation matrix of the data closest to the heterogeneous class of labeled data in each batch in the cluster space; The similarity between data is defined as the difference between activation matrices, and the difference between activation matrices is solved, including: solving the difference between the activation matrix of the labeled data in each batch and the activation matrix of the data farthest from the same type of labeled data in each batch in the cluster space; Calculate the difference loss, including: solving the difference loss of the activation matrix of the labeled data in each batch and the activation matrix of the data closest to the heterogeneous data in each batch in the cluster space; The similarity triplet loss is obtained as follows: the difference between the activation matrix of the labeled data in each batch and the activation matrix of the data farthest from the same class of the labeled data in each batch in the cluster space, minus the difference loss between the activation matrix of the labeled data in each batch and the activation matrix of the data closest to the heterogeneous class of the labeled data in each batch in the cluster space; S4. The complete loss function of the CNN-LSTM regressor is constructed by weighting the supervised loss, semi-supervised loss and similarity triplet loss. The network parameters are updated according to the loss value and gradient descent to obtain the semi-supervised facial emotion recognition model, and the emotion analysis is performed on the current facial image data.

2. The semi-supervised facial emotion recognition method according to claim 1, characterized in that The CNN-LSTM regressor includes, in each frame, using the S3D face detection algorithm to detect the face area of ​​the image; using the ResNet-based CNN encoder to extract deep features; before introducing the LSTM, adjusting the length of the extracted features to 2048 through the fully connected layer; after introducing the LSTM, outputting the excitation and the valence respectively by two different fully connected layers; the LSTM includes a forget gate, an input gate and an output gate.

3. The semi-supervised facial emotion recognition method according to claim 2 is characterized in that The step S2 includes performing data expansion on the unlabeled data and then taking the average of the predictions of the expanded data; using fine-tuning MixUp to mix the labeled data with the unlabeled data to obtain the target data; the network predicts the target data and calculates the loss function.

4. The semi-supervised facial emotion recognition method according to claim 1, characterized in that The step A1 comprises clustering the depth features of the initial label data to obtain an initial cluster center for a given basic image data sample; During the training process, the unlabeled data and labeled data are simultaneously input into the CNN-LSTM regressor to calculate the cluster center with the smallest distance to the basic image data sample. At the same time, the cluster category of the basic image data sample is determined by the cluster center with the smallest distance to it.

5. The semi-supervised facial emotion recognition method according to claim 4, characterized in that The step A2 includes calculating the weighted item based on the sample features extracted by CNN after one iteration, including the product of the sum of the initial clustering depth features and the reciprocal of the cluster center with the smallest distance from the basic image data sample, and the sum of the products of the sample features extracted by CNN after one iteration and the reciprocal of the cluster center with the smallest distance from the basic image data sample after one iteration, and iterating until a predetermined number of iteration terminations; the distance between samples in the clustering space after the iteration is the Euclidean norm of the feature vectors of two different basic image data samples.

6. The semi-supervised facial emotion recognition method according to claim 1, characterized in that The step S2 comprises: For unlabeled data, the CNN-LSTM regressor is made to learn from smoothed labels and data augmentation is first performed on the unlabeled data: u represents unlabeled data; augment(·) represents an augmentation operation; represents the expanded unlabeled data; k represents the number of data expansions; for the expanded unlabeled data Take the mean; Where q represents the predicted distribution after taking the mean; k represents the number of data expansions; i represents the count variable of the given basic image data sample; P model represents the deep network model; u i represents a certain picture; θ represents the network parameters; Mix the labeled data (x1, p) and the unlabeled data (u1, q), where x1 represents the labeled image, p represents the label of the labeled image, u1 represents the unlabeled image, and q represents the predicted distribution obtained by taking the mean after expanding the unlabeled image data, and output the target data x′ represents the data obtained by mixing labeled images and unlabeled images; The distribution of the label image and the predicted distribution obtained by taking the average after the label image data is expanded; the target data As learning objectives: x′=λ′x1+(1-λ′)u1 Where α represents a hyperparameter; λ′=max(λ, 1-λ), λ~Bata(α, α), λ represents a value randomly taken from the Beta distribution, λ′ represents the maximum value between λ and 1-λ; max(·) represents the maximum value; Beta(·) represents the Beta distribution; For target data The prediction value P′ is obtained by prediction. In order to obtain information from different angles, the CNN-LSTM regressors with different training periods are used to vote for the pseudo labels: Among them, P represents the final pseudo label; P n-model (y|x′; θ) represents the CNN-LSTM regressor n iterations ago; the semi-supervised loss L is calculated using the mean square error between the predicted value P′ and the final pseudo-label P. u : L u =||P′-P||2 Among them, ||·||2 represents the mean square error; Use cross entropy to calculate the supervision loss L x : Among them, P model (y|x; θ) represents the prediction of the CNN-LSTM regressor for the labeled data; H(·) represents the cross entropy loss function.