A micro-expression recognition method based on feature decoupling

By constructing feature decoupling networks and adversarial learning methods, separating facial expression features and identity features, the problems of interference with identity information and data acquisition difficulties in micro-expression recognition are solved, and the recognition accuracy and robustness are improved.

CN114937298BActive Publication Date: 2024-11-08NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210522618.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-13
Publication Date
2024-11-08
Estimated Expiration
2042-05-13

AI Technical Summary

Technical Problem

Micro-expression recognition has interference caused by strong correlation between facial expression characteristics and identity information, which leads to inaccurate recognition. It is difficult to collect micro-expression data and high labeling costs, making it easy to overfit the model.

Method used

Design a micro-expression recognition method based on feature decoupling. By constructing a common expression-identity feature decoupling network and a micro-expression-identity feature decoupling network, using transfer learning and adversarial learning methods to separate facial expression features and identity features. Reduce identity information interference and improve the accuracy of micro-expression recognition through adversarial network models.

Benefits of technology

It effectively eliminates identity information interference, improves the accuracy of micro-expression recognition, reduces the risk of model overfitting, and uses rich ordinary emoticon data to assist micro-expression recognition, improving the recognition effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114937298B_ABST
    Figure CN114937298B_ABST
Patent Text Reader

Abstract

The present invention discloses a micro-expression recognition method based on feature decoupling, which comprises the following steps: constructing and training a normal expression-identity feature decoupling network, using the trained normal expression-identity feature decoupling network as the initial model of the micro-expression-identity feature decoupling network, fine-tuning the model using a human face micro-expression image sample, and obtaining a micro-expression-identity feature decoupling network; constructing an adversarial network model including the normal expression-identity feature decoupling network and the micro-expression-identity feature decoupling network, using a human face normal expression image sample and a human face micro-expression image sample of the same expression category to train the adversarial network model through an adversarial learning method; using the micro-expression-identity feature decoupling network in the trained adversarial network model as the final micro-expression recognition model. The present invention can enable the model to exclude the interference of human face identity features, pay more attention to micro-expression features, and thus improve the accuracy of micro-expression recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of micro-expression recognition and artificial intelligence, and in particular to a micro-expression recognition method based on feature decoupling. Background Art

[0002] Facial expressions are one of the main ways for people to express their emotions, and they are also an important way for robots to understand human emotions intelligently. Micro-expressions are often produced when people want to suppress their emotions, which can neither be faked nor suppressed. Since micro-expressions are produced spontaneously in an unconscious state, they are difficult to conceal or disguise, and are usually directly related to real emotions. Therefore, micro-expressions are more reliable in sentiment analysis and have broad application prospects. For example, in the field of national security, some well-trained terrorists and other dangerous people may easily pass the polygraph test, but through micro-expressions, their true emotions under the false surface can often be discovered, and due to this characteristic of micro-expressions, it also has a good application in the judicial system and medical clinics.

[0003] Due to the interference of many identity information irrelevant to facial expression features, micro-expression recognition is not accurate enough. Therefore, removing the facial identity features that are strongly associated with facial identity information and making the model pay more attention to facial expression features will improve the accuracy of micro-expressions. The method based on adversarial learning feature decoupling utilizes the adversarial relationship between different tasks during the multi-task network training process. When the features of a task cannot be used as useful features for micro-expression classification, it can be considered that the extracted features are irrelevant to the task, that is, the purpose of decoupling is achieved. The key is to dig out the features that determine the differences between similar image categories and achieve decoupling from irrelevant features.

[0004] At the same time, due to the short duration of micro-expressions and the weak amplitude of facial muscle changes, the collection of micro-expression data is difficult and the annotation cost is very high. In addition, due to the limitations of the existing micro-expression data, the direct use of deep learning methods in micro-expression recognition is prone to model overfitting. The establishment of ordinary expression databases has been relatively complete, and each database contains a large number of well-annotated training samples. Therefore, using rich ordinary expression data to assist micro-expression recognition will improve the accuracy of micro-expression recognition. In order to achieve better performance in micro-expression recognition, the transfer learning method is introduced. Ordinary expression samples of different identities are used to train an ordinary expression-identity feature decoupling network as the initial model of the micro-expression-identity feature decoupling network. The knowledge in the source model is used to strengthen the learning in the target task, thereby improving the accuracy of micro-expression recognition.

[0005] The Chinese patent application "A method for micro-expression recognition based on 3D convolutional neural network" (patent application number CN201610954555.9, publication number CN106570474A) extracts the grayscale channel feature map, horizontal gradient channel feature map, vertical gradient channel feature map, horizontal optical flow channel feature map, and vertical optical flow channel feature map for each frame image in the micro-expression video sequence to obtain a feature map group corresponding to the micro-expression video sequence to be identified, and then inputs it into the 3D convolutional neural network to further extract features and classify. This method processes every frame image of the micro-expression video sequence, which is extremely computationally intensive and requires a large number of training samples. In the training process of the model, if the samples are insufficient, it is easy to have overfitting problems. Summary of the invention

[0006] The purpose of the present invention is to provide a micro-expression recognition method based on feature decoupling, which eliminates the interference of facial identity features related to identity information, pays more attention to facial expression features, and thus improves the accuracy of micro-expression recognition.

[0007] To achieve the above functions, the present invention designs a micro-expression recognition method based on feature decoupling. For the facial micro-expression image to be recognized, the following steps S1 to S4 are performed to complete the recognition of facial micro-expressions in the facial micro-expression image to be recognized:

[0008] S1: Collecting different human face common expression image samples, taking the pre-processed human face common expression image samples as input, based on the feature extraction module, the expression reconstruction module, and the expression classification module, taking the expression category corresponding to the human face common expression image samples as output, constructing a common expression-identity feature decoupling network that separates the facial expression features and facial identity features of each human face common expression image sample, and using different human face common expression image samples, combined with preset corresponding expression categories, training the common expression-identity feature decoupling network to obtain a trained common expression-identity feature decoupling network;

[0009] S2: Collect different facial micro-expression image samples, take the pre-processed facial micro-expression image samples as input, take the expression category corresponding to each facial micro-expression image sample as output, take the trained common expression-identity feature decoupling network as the initial model, adopt the transfer learning method, construct the micro-expression-identity feature decoupling network, and use different facial micro-expression image samples, combined with the preset corresponding expression category, to train the micro-expression-identity feature decoupling network, and obtain the trained micro-expression-identity feature decoupling network;

[0010] S3: Based on the common expression-identity feature decoupling network obtained in step S1 and the micro-expression-identity feature decoupling network obtained in step S2, and in combination with the discriminator, an adversarial learning method is adopted to obtain an adversarial network model, and common expression image samples and micro-expression image samples of the same expression category are used to train the adversarial network model to obtain a trained adversarial network model;

[0011] S4: Using the micro-expression-identity feature decoupling network in the adversarial network model obtained in step S3 as a micro-expression recognition model, for the face image to be recognized, the recognition of facial micro-expressions in the face image to be recognized is completed.

[0012] As a preferred technical solution of the present invention: the common expression-identity feature decoupling network in step S1 includes a feature extraction module, an expression reconstruction module, and an expression classification module. The specific structure of the common expression-identity feature decoupling network is as follows:

[0013] The feature extraction module includes a dual-branch feature extractor, which includes an identity feature extractor and an expression feature extractor, wherein the identity feature extractor extracts the face identity features of the face common expression image samples based on a convolutional neural network and an identity feature extractor loss function, and the expression feature extractor extracts the face expression features of the face common expression image samples based on a convolutional neural network and an expression feature extractor loss function;

[0014] The expression reconstruction module fuses the facial identity features of a human face common expression image sample with the facial expression features of another human face common expression image sample, encodes the fused features based on an embedding layer, and enlarges the size of the features through upsampling and convolutional neural network to generate an expression migration image;

[0015] The expression classification module includes a discriminator for evaluating the expression transfer image effect and outputting common expression categories. The discriminator is composed of a convolutional neural network and a fully connected layer. The discriminator includes two branches, namely a face identity discriminator branch and a face expression discriminator branch.

[0016] As a preferred technical solution of the present invention: in step S1, the pre-processed common expression image samples of each face are used as input, and based on the feature extraction module, the expression reconstruction module, and the expression classification module, the expression category corresponding to the common expression image samples of the face is used as output, and the specific steps of constructing a common expression-identity feature decoupling network that separates the facial expression features and the facial identity features of each common expression image sample of the face are as follows:

[0017] S11: Feature extraction: taking two different common facial expression image samples I from each of the preprocessed common facial expression image samples e and Ii As input, a dual-branch feature extractor is used to extract common facial expression image samples I and i Face identity features Samples of common facial expression images I e facial expression features

[0018] S12: Expression reconstruction: Reconstructing facial identity features and facial expression features Combined, reconstruct the common facial expression image, and use the reconstructed common facial expression image as the expression transfer image, where the reconstruction loss L rec As follows:

[0019]

[0020] S13: Expression classification: The expression migration image obtained in step S12 is input into the discriminator of the expression classification module. The cross entropy function L used by the discriminator is c As follows:

[0021]

[0022] Where y is the expression category label, L is the total number of expression category labels, Facial expression features for prediction Distribution of expression categories;

[0023] S14: introducing paired image consistency constraints for the common expression-identity feature decoupling network, and adjusting the common expression-identity feature decoupling network parameters through paired image consistency loss;

[0024] S15: Use the common expression image samples of each face to train the common expression-identity feature decoupling network to obtain a trained common expression-identity feature decoupling network.

[0025] As a preferred technical solution of the present invention: in step S2, the pre-processed facial micro-expression image samples are used as input, the expression categories corresponding to the facial micro-expression image samples are used as output, and the trained common expression-identity feature decoupling network is used as the initial model. The specific steps of constructing the micro-expression-identity feature decoupling network are as follows:

[0026] S21: fixing the parameters of the normal expression-identity feature decoupling network obtained in step S1, and using the normal expression-identity feature decoupling network trained in step S1 as the initial model of the micro-expression-identity feature decoupling network. The structure of the micro-expression-identity feature decoupling network is the same as that of the normal expression-identity feature decoupling network, and both include a feature extraction module, an expression reconstruction module, and an expression classification module.

[0027] S22: Based on the transfer learning method, the initial model of the micro-expression-identity feature decoupling network is adjusted using pre-processed samples of different facial micro-expression images to obtain the micro-expression-identity feature decoupling network.

[0028] As a preferred technical solution of the present invention: in step S3, based on the normal expression-identity feature decoupling network obtained in step S1 and the micro-expression-identity feature decoupling network obtained in step S2, and in combination with the discriminator, the specific steps of using the adversarial learning method to obtain the adversarial network model are as follows:

[0029] S31: Take a human face common expression image sample y belonging to the same expression category c , and a facial micro-expression image sample x c They are respectively input into the normal expression-identity feature decoupling network obtained in step S1 and the micro-expression-identity feature decoupling network obtained in step S2 to obtain the normal expression image samples y c The corresponding common facial expression features And facial micro-expression image samples x c Corresponding facial micro-expression features

[0030] S32: Fix the parameters of the normal expression-identity feature decoupling network trained in step S1 and the parameters of the micro-expression-identity feature decoupling network trained in step S2, input the normal expression features of the face output by the normal expression-identity feature decoupling network and the micro-expression features of the face output by the micro-expression-identity feature decoupling network into the discriminator for adversarial learning, wherein the loss function L of the discriminator is D As follows:

[0031]

[0032] In the formula, Common facial expression features for prediction The distribution of expression categories, To predict facial micro-expression features Distribution of expression categories;

[0033] The adversarial loss L of the micro-expression-identity feature decoupling network adv As follows:

[0034]

[0035] S33: facial micro-expression image sample x c Input into the micro-expression-identity feature decoupling network, the expression classification module of the micro-expression-identity feature decoupling network classifies the facial micro-expression image sample x cThe cross entropy L for classification 1 The loss is as follows:

[0036]

[0037] Where y is the expression category label, L is the total number of expression category labels, To predict facial micro-expression features Distribution of expression categories;

[0038] S34: Based on the regularization method, the adversarial network model constructed for the ordinary expression-identity feature decoupling network, the micro-expression-identity feature decoupling network, and the discriminator introduces the loss unequal regularization L LIR As follows:

[0039] L LIR =max*L 2 -L 1 ,0+

[0040] Where, L 2 is a sample of a common facial expression image y c Cross entropy loss for classification;

[0041] S35: Calculate the overall loss function L of the adversarial network model SUM As follows:

[0042] L SUM =L 1 +m 1 L adv +m 2 L LIR

[0043] In the formula, m 1 、m 2 is a hyperparameter;

[0044] S36: Use the facial ordinary expression image samples and facial micro-expression image samples belonging to the same expression category to train the adversarial network model to obtain a trained adversarial network model.

[0045] As a preferred technical solution of the present invention: the convolutional neural network of the feature extraction module includes at least one convolutional layer and a pooling layer, and the convolutional layer includes m 1 k 1 ×k 1 The convolution kernel, where m 1 ∈*64,128,256,512+,k 1 ∈*3,5,7+, the pooling layer includes k 2 ×k 2 The pooling kernel of k 2 ∈*1,2,3+.

[0046] The present invention also provides a system for micro-expression recognition method based on feature decoupling, which is characterized by comprising:

[0047] one or more processors;

[0048] A memory storing operable instructions, wherein when the instructions are executed by the one or more processors, the one or more processors perform operations to perform the following steps S1 to S4 for the facial micro-expression image to be identified, thereby completing the recognition of facial micro-expressions in the facial micro-expression image to be identified:

[0049] S1: Collecting different human face common expression image samples, taking the pre-processed human face common expression image samples as input, based on the feature extraction module, the expression reconstruction module, and the expression classification module, taking the expression category corresponding to the human face common expression image samples as output, constructing a common expression-identity feature decoupling network that separates the facial expression features and facial identity features of each human face common expression image sample, and using different human face common expression image samples, combined with preset corresponding expression categories, training the common expression-identity feature decoupling network to obtain a trained common expression-identity feature decoupling network;

[0050] S2: Collect different facial micro-expression image samples, take the pre-processed facial micro-expression image samples as input, take the expression category corresponding to each facial micro-expression image sample as output, take the trained common expression-identity feature decoupling network as the initial model, adopt the transfer learning method, construct the micro-expression-identity feature decoupling network, and use different facial micro-expression image samples, combined with the preset corresponding expression category, to train the micro-expression-identity feature decoupling network, and obtain the trained micro-expression-identity feature decoupling network;

[0051] S3: Based on the common expression-identity feature decoupling network obtained in step S1 and the micro-expression-identity feature decoupling network obtained in step S2, and in combination with the discriminator, an adversarial learning method is adopted to obtain an adversarial network model, and common expression image samples and micro-expression image samples of the same expression category are used to train the adversarial network model to obtain a trained adversarial network model;

[0052] S4: Using the micro-expression-identity feature decoupling network in the adversarial network model obtained in step S3 as a micro-expression recognition model, for the face image to be recognized, the recognition of facial micro-expressions in the face image to be recognized is completed.

[0053] The present invention also provides a computer-readable medium for storing software, characterized in that the computer-readable medium includes instructions that can be executed by one or more computers, and when the instructions are executed by the one or more computers, they perform the operation of a micro-expression recognition method based on feature decoupling.

[0054] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0055] (1) The present invention constructs a general expression-identity feature decoupling network that separates facial expression features from facial identity features. The general expression-identity feature decoupling network eliminates the interference of facial identity features that are strongly associated with identity information, pays more attention to facial expression features, and solves the problem of inaccurate micro-expression recognition due to the interference of many identity information that are irrelevant to facial expression features;

[0056] (2) The present invention uses a transfer learning method to use the trained common expression-identity feature decoupling network as the initial model of the micro-expression-identity feature decoupling network, thereby solving the problem of limited micro-expression data and using the knowledge in the source model to strengthen the learning in the target task, thereby improving the accuracy of micro-expression recognition;

[0057] (3) The present invention aims at the fact that micro-expressions have a short duration and weak changes in facial muscles. In addition, due to the limitation of existing micro-expression data, the direct use of deep learning methods in micro-expression recognition is prone to model overfitting and other problems. The invention introduces a transfer learning method to assist micro-expression recognition by utilizing abundant ordinary expression data, thereby improving the accuracy of micro-expression recognition.

[0058] (4) The present invention constructs an adversarial network model including a normal expression-identity feature decoupling network and a micro-expression-identity feature decoupling network. The adversarial learning method is used in the feature space to assist the training of the micro-expression recognition model, and the similar features of micro-expressions and normal expressions are effectively learned. In the label space, loss inequality regularization is used to calibrate the classification loss of the normal expression-identity feature decoupling network and the micro-expression-identity feature decoupling network, which can obtain better recognition effect than the method based on manual features. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of a micro-expression recognition method based on feature decoupling provided according to an embodiment of the present invention;

[0060] Figure 2 is a diagram of a human face common expression-identity feature decoupling network structure provided according to an embodiment of the present invention;

[0061] Figure 3 is a diagram of a facial micro-expression-identity feature decoupling network structure provided according to an embodiment of the present invention;

[0062] Figure 4 It is a structural diagram of an adversarial network model provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0064] Reference Figure 1 The present invention provides a method for micro-expression recognition based on feature decoupling. For a facial micro-expression image to be recognized, the following steps S1 to S4 are performed to complete the recognition of facial micro-expressions in the facial micro-expression image to be recognized:

[0065] S1: Collect different common expression image samples of faces, take the preprocessed common expression image samples of faces as input, and construct a common expression-identity feature decoupling network that separates the facial expression features and facial identity features of each common expression image sample based on the feature extraction module, expression reconstruction module, and expression classification module, and take the expression category corresponding to the common expression image sample of faces as output. Then, use different common expression image samples of faces, combined with the preset corresponding expression category, to train the common expression-identity feature decoupling network, and obtain the trained common expression-identity feature decoupling network. The structure diagram of the common expression-identity feature decoupling network of faces is shown in FIG. Figure 2 ;

[0066] Among them, the common expression-identity feature decoupling network includes a feature extraction module, an expression reconstruction module, and an expression classification module. The specific structure of the common expression-identity feature decoupling network is as follows:

[0067] The feature extraction module includes a dual-branch feature extractor, which includes an identity feature extractor and an expression feature extractor, wherein the identity feature extractor extracts the face identity features of the face common expression image samples based on a convolutional neural network and an identity feature extractor loss function, and the expression feature extractor extracts the face expression features of the face common expression image samples based on a convolutional neural network and an expression feature extractor loss function;

[0068] The convolutional neural network of the feature extraction module contains at least one convolutional layer and a pooling layer. The convolutional layer includes m 1 k 1 ×k 1 The convolution kernel, where m 1 ∈*64,128,256,512+,k 1 ∈*3,5,7+, the pooling layer includes k 2 ×k 2 The pooling kernel of k 2∈*1,2,3+.

[0069] In one embodiment, the human face common expression image samples are first preprocessed, passed through a convolution layer with a convolution kernel size of 7×7, and then a layer of maximum pooling is performed. After downsampling, the output of the previous layer is convolved with four 3×3 convolution kernels, followed by three dimensional upscaling and downsampling, and finally average pooling before entering the fully connected layer.

[0070] The expression reconstruction module fuses the facial identity features of a human face common expression image sample with the facial expression features of another human face common expression image sample, encodes the fused features based on an embedding layer, and enlarges the size of the features through upsampling and convolutional neural network to generate an expression migration image; wherein the introduction of the embedding layer enables the expression reconstruction module to process data from different spaces, so that the expression reconstruction module can modify the expression but keep other facial expressions, such as hair, glasses or posture unchanged;

[0071] The expression classification module is used to calculate the probability that the input common facial expression samples belong to each expression category label, and includes a discriminator for evaluating the expression transfer image effect and outputting the common expression category. The discriminator is composed of a convolutional neural network and a fully connected layer. The discriminator includes two branches, namely a face identity discriminator branch and a face expression discriminator branch. The two branches of the discriminator are used to evaluate the effect of the synthesized expression transfer image. If the effect of the synthesized expression transfer image can deceive the discriminator, it means that the features learned by the common expression-identity feature decoupling network meet the preset standards; the fully connected layer of the face expression feature extractor branch is used to discriminate the features learned by the feature extraction module.

[0072] The specific steps of step S1 are as follows:

[0073] S11: Feature extraction: taking two different common facial expression image samples I from each of the preprocessed common facial expression image samples e and I i As input, a dual-branch feature extractor is used to extract common facial expression image samples I and i Face identity features Samples of common facial expression images I e facial expression features

[0074] S12: Expression reconstruction: Reconstructing facial identity features and facial expression features Combined, reconstruct the common facial expression image, and use the reconstructed common facial expression image as the expression transfer image, where the reconstruction loss L rec As follows:

[0075]

[0076] S13: Expression classification: The expression migration image obtained in step S12 is input into the discriminator of the expression classification module. The cross entropy function L used by the discriminator is c As follows:

[0077]

[0078] Where y is the expression category label, L is the total number of expression category labels, Facial expression features for prediction Distribution of expression categories;

[0079] S14: introducing paired image consistency constraints for the common expression-identity feature decoupling network, and adjusting the common expression-identity feature decoupling network parameters through paired image consistency loss;

[0080] S15: Use the common expression image samples of each face to train the common expression-identity feature decoupling network to obtain a trained common expression-identity feature decoupling network.

[0081] S2: Collect different facial micro-expression image samples, take the pre-processed facial micro-expression image samples as input, take the expression category corresponding to each facial micro-expression image sample as output, take the trained common expression-identity feature decoupling network as the initial model, adopt the transfer learning method, construct the micro-expression-identity feature decoupling network, and use different facial micro-expression image samples, combined with the preset corresponding expression category, to train the micro-expression-identity feature decoupling network to obtain the trained micro-expression-identity feature decoupling network. The structure diagram of the facial micro-expression-identity feature decoupling network is shown in Fig. Figure 3 ;

[0082] In one embodiment, the preprocessing of each human face normal expression image sample and each human face micro-expression image sample includes face detection, cropping, and face alignment processing.

[0083] The specific steps of step S2 are as follows:

[0084] S21: fixing the parameters of the normal expression-identity feature decoupling network obtained in step S1, and using the normal expression-identity feature decoupling network trained in step S1 as the initial model of the micro-expression-identity feature decoupling network. The structure of the micro-expression-identity feature decoupling network is the same as that of the normal expression-identity feature decoupling network, and both include a feature extraction module, an expression reconstruction module, and an expression classification module.

[0085] S22: Since micro-expressions last for a short time and the amplitude of facial muscle changes is weak, and due to the limitation of existing micro-expression data, the direct use of deep learning methods in micro-expression recognition has poor robustness. Therefore, using rich ordinary expression data to assist micro-expression recognition will improve the accuracy of micro-expression recognition. Therefore, based on the transfer learning method, the present invention uses pre-processed samples of different facial micro-expression images to adjust the initial model of the micro-expression-identity feature decoupling network to obtain the micro-expression-identity feature decoupling network.

[0086] S3: Based on the common expression-identity feature decoupling network obtained in step S1 and the micro-expression-identity feature decoupling network obtained in step S2, and in combination with the discriminator, an adversarial learning method is adopted to obtain an adversarial network model, and common expression image samples and micro-expression image samples of the same expression category are used to train the adversarial network model to obtain a trained adversarial network model;

[0087] The specific steps of step S3 are as follows:

[0088] S31: Take a human face common expression image sample y belonging to the same expression category c , and a facial micro-expression image sample x c They are respectively input into the normal expression-identity feature decoupling network obtained in step S1 and the micro-expression-identity feature decoupling network obtained in step S2 to obtain the normal expression image samples y c The corresponding common facial expression features And facial micro-expression image samples x c Corresponding facial micro-expression features

[0089] S32: fixing the parameters of the normal expression-identity feature decoupling network trained in step S1 and the parameters of the micro-expression-identity feature decoupling network trained in step S2, inputting the normal expression features of the face output by the normal expression-identity feature decoupling network and the micro-expression features of the face output by the micro-expression-identity feature decoupling network into the discriminator, and performing adversarial learning;

[0090] The discriminator is used to distinguish between common facial expression features and micro-expression features. The micro-expression-identity feature decoupling network generates micro-expression features that are difficult for the discriminator to distinguish. The discriminator distinguishes these two features and conducts adversarial learning.

[0091] The loss function of the discriminator is L D As follows:

[0092]

[0093] In the formula, Common facial expression features for prediction The distribution of expression categories, To predict facial micro-expression features Distribution of expression categories;

[0094] Using minimization Instead of minimizing It can avoid the gradient vanishing problem, so the adversarial loss L of the micro-expression-identity feature decoupling network is adv As follows:

[0095]

[0096] S33: facial micro-expression image sample x c Input into the micro-expression-identity feature decoupling network, the classification loss is used to improve the accuracy of micro-expression recognition. The expression classification module of the micro-expression-identity feature decoupling network is used to classify the facial micro-expression image sample x c The cross entropy L for classification 1 The loss is as follows:

[0097]

[0098] Where y is the expression category label, L is the total number of expression category labels, To predict facial micro-expression features Distribution of expression categories;

[0099] S34: For the adversarial network model constructed by the common expression-identity feature decoupling network, micro-expression-identity feature decoupling network, and discriminator, since the common expression image samples and micro-expression image samples of the same expression category can produce similar outputs, a regularization method is introduced, called loss unequal regularization. Loss unequal regularization assumes that secondary features have more information than primary features, and common expression image samples of the face have more information than micro-expression image samples of the face. The loss unequal regularization L is introduced. LIR As follows:

[0100] L LIR =max*L 2 -L 1 ,0+

[0101] Where, L 2 is a sample of a common facial expression image y c Cross entropy loss for classification;

[0102] S35: Calculate the overall loss function L of the adversarial network model SUM As follows:

[0103] L SUM =L1 +m 1 L adv +m 2 L LIR

[0104] In the formula, m 1 、m 2 is a hyperparameter used to control the loss;

[0105] S36: Use the facial ordinary expression image samples and facial micro-expression image samples belonging to the same expression category to train the adversarial network model to obtain a trained adversarial network model.

[0106] S4: Using the micro-expression-identity feature decoupling network in the adversarial network model obtained in step S3 as a micro-expression recognition model, for the face image to be recognized, complete the recognition of facial micro-expressions in the face image to be recognized.

[0107] The embodiment of the present invention further provides a system for a micro-expression recognition method based on feature decoupling, characterized in that it includes:

[0108] one or more processors;

[0109] A memory storing operable instructions, wherein when the instructions are executed by the one or more processors, the one or more processors perform operations to perform the following steps S1 to S4 for the facial micro-expression image to be identified, thereby completing the recognition of facial micro-expressions in the facial micro-expression image to be identified:

[0110] S1: Collecting different human face common expression image samples, taking the pre-processed human face common expression image samples as input, based on the feature extraction module, the expression reconstruction module, and the expression classification module, taking the expression category corresponding to the human face common expression image samples as output, constructing a common expression-identity feature decoupling network that separates the facial expression features and facial identity features of each human face common expression image sample, and using different human face common expression image samples, combined with preset corresponding expression categories, training the common expression-identity feature decoupling network to obtain a trained common expression-identity feature decoupling network;

[0111] S2: Collect different facial micro-expression image samples, take the pre-processed facial micro-expression image samples as input, take the expression category corresponding to each facial micro-expression image sample as output, take the trained common expression-identity feature decoupling network as the initial model, adopt the transfer learning method, construct the micro-expression-identity feature decoupling network, and use different facial micro-expression image samples, combined with the preset corresponding expression category, to train the micro-expression-identity feature decoupling network, and obtain the trained micro-expression-identity feature decoupling network;

[0112] S3: Based on the normal expression-identity feature decoupling network obtained in step S1 and the micro-expression-identity feature decoupling network obtained in step S2, and in combination with the discriminator, an adversarial learning method is used to obtain an adversarial network model, and the normal expression image samples and micro-expression image samples of the same expression category are used to train the adversarial network model to obtain a trained adversarial network model. The adversarial network model structure diagram is shown in FIG. Figure 4 ;

[0113] S4: Using the micro-expression-identity feature decoupling network in the adversarial network model obtained in step S3 as a micro-expression recognition model, for the face image to be recognized, the recognition of facial micro-expressions in the face image to be recognized is completed.

[0114] An embodiment of the present invention also provides a computer-readable medium storing software, characterized in that the computer-readable medium includes instructions that can be executed by one or more computers, and when the instructions are executed by the one or more computers, the operations of the micro-expression recognition method based on feature decoupling are performed.

[0115] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A micro-expression recognition method based on feature decoupling, characterized in that: For the facial micro-expression image to be identified, the following steps S1 to S4 are performed to complete the recognition of facial micro-expressions in the facial micro-expression image to be identified: S1: Collecting different human face common expression image samples, taking the pre-processed human face common expression image samples as input, based on the feature extraction module, the expression reconstruction module, and the expression classification module, taking the expression category corresponding to the human face common expression image samples as output, constructing a common expression-identity feature decoupling network that separates the facial expression features and facial identity features of each human face common expression image sample, and using different human face common expression image samples, combined with preset corresponding expression categories, training the common expression-identity feature decoupling network to obtain a trained common expression-identity feature decoupling network; S2: Collect different facial micro-expression image samples, take the pre-processed facial micro-expression image samples as input, take the expression category corresponding to each facial micro-expression image sample as output, take the trained common expression-identity feature decoupling network as the initial model, adopt the transfer learning method, construct the micro-expression-identity feature decoupling network, and use different facial micro-expression image samples, combined with the preset corresponding expression category, to train the micro-expression-identity feature decoupling network, and obtain the trained micro-expression-identity feature decoupling network; S3: Based on the common expression-identity feature decoupling network obtained in step S1 and the micro-expression-identity feature decoupling network obtained in step S2, and in combination with the discriminator, an adversarial learning method is adopted to obtain an adversarial network model, and common expression image samples and micro-expression image samples of the same expression category are used to train the adversarial network model to obtain a trained adversarial network model; The specific steps of step S3 are as follows: S31: Take a human face common expression image sample y belonging to the same expression category c , and a facial micro-expression image sample x c They are respectively input into the normal expression-identity feature decoupling network obtained in step S1 and the micro-expression-identity feature decoupling network obtained in step S2 to obtain the normal expression image samples y c The corresponding common facial expression features And facial micro-expression image samples x c Corresponding facial micro-expression features S32: Fix the parameters of the normal expression-identity feature decoupling network trained in step S1 and the parameters of the micro-expression-identity feature decoupling network trained in step S2, input the normal expression features of the face output by the normal expression-identity feature decoupling network and the micro-expression features of the face output by the micro-expression-identity feature decoupling network into the discriminator for adversarial learning, wherein the loss function L of the discriminator is D As follows: In the formula, Common facial expression features for prediction The distribution of expression categories, To predict facial micro-expression features Distribution of expression categories; The adversarial loss L of the micro-expression-identity feature decoupling network adv As follows: S33: facial micro-expression image sample x c Input into the micro-expression-identity feature decoupling network, the expression classification module of the micro-expression-identity feature decoupling network classifies the facial micro-expression image sample x c The cross entropy L1 loss for classification is as follows: In the formula, y is the expression category label, L is the total number of expression category labels, To predict facial micro-expression features Distribution of expression categories; S34: Based on the regularization method, the adversarial network model constructed for the ordinary expression-identity feature decoupling network, the micro-expression-identity feature decoupling network, and the discriminator introduces the loss unequal regularization L LIR As follows: L LIR =max{L2-L1,0} Where L2 is the sample y of the common facial expression image c Cross entropy loss for classification; S35: Calculate the overall loss function L of the adversarial network model SUM As follows: L SUM =L1+m1L adv +m2L LIR In the formula, m1 and m2 are hyperparameters; S36: using the human face common expression image samples and human face micro-expression image samples belonging to the same expression category to train the adversarial network model to obtain a trained adversarial network model; S4: Using the micro-expression-identity feature decoupling network in the adversarial network model obtained in step S3 as a micro-expression recognition model, for the face image to be recognized, the recognition of facial micro-expressions in the face image to be recognized is completed.

2. The micro-expression recognition method based on feature decoupling according to claim 1, characterized in that: In step S1, the common expression-identity feature decoupling network includes a feature extraction module, an expression reconstruction module, and an expression classification module. The specific structure of the common expression-identity feature decoupling network is as follows: The feature extraction module includes a dual-branch feature extractor, which includes an identity feature extractor and an expression feature extractor, wherein the identity feature extractor extracts the face identity features of the face common expression image samples based on a convolutional neural network and an identity feature extractor loss function, and the expression feature extractor extracts the face expression features of the face common expression image samples based on a convolutional neural network and an expression feature extractor loss function; The expression reconstruction module fuses the facial identity features of a human face common expression image sample with the facial expression features of another human face common expression image sample, encodes the fused features based on an embedding layer, and enlarges the size of the features through upsampling and convolutional neural network to generate an expression migration image; The expression classification module includes a discriminator for evaluating the expression transfer image effect and outputting common expression categories. The discriminator is composed of a convolutional neural network and a fully connected layer. The discriminator includes two branches, namely a face identity discriminator branch and a face expression discriminator branch.

3. The micro-expression recognition method based on feature decoupling according to claim 2, characterized in that: In step S1, the preprocessed common expression image samples of each face are used as input, and based on the feature extraction module, the expression reconstruction module, and the expression classification module, the expression categories corresponding to the common expression image samples of the face are used as outputs to construct a common expression-identity feature decoupling network that separates the facial expression features and the facial identity features of each common expression image sample of the face as follows: S11: Feature extraction: taking two different common facial expression image samples I from each of the preprocessed common facial expression image samples e and I i As input, a dual-branch feature extractor is used to extract common facial expression image samples I and i Face identity features Samples of common facial expression images I e facial expression features S12: Expression reconstruction: Reconstructing facial identity features and facial expression features Combined, reconstruct the common facial expression image, and use the reconstructed common facial expression image as the expression transfer image, where the reconstruction loss L rec As follows: S13: Expression classification: The expression migration image obtained in step S12 is input into the discriminator of the expression classification module. The cross entropy function L used by the discriminator is c As follows: In the formula, y is the expression category label, L is the total number of expression category labels, Facial expression features for prediction Distribution of expression categories; S14: introducing paired image consistency constraints for the common expression-identity feature decoupling network, and adjusting the common expression-identity feature decoupling network parameters through paired image consistency loss; S15: Use the common expression image samples of each face to train the common expression-identity feature decoupling network to obtain a trained common expression-identity feature decoupling network.

4. The micro-expression recognition method based on feature decoupling according to claim 3, characterized in that: In step S2, the preprocessed facial micro-expression image samples are used as input, the expression categories corresponding to the facial micro-expression image samples are used as output, and the trained common expression-identity feature decoupling network is used as the initial model. The specific steps of constructing the micro-expression-identity feature decoupling network are as follows: S21: fixing the parameters of the normal expression-identity feature decoupling network obtained in step S1, and using the normal expression-identity feature decoupling network trained in step S1 as the initial model of the micro-expression-identity feature decoupling network. The structure of the micro-expression-identity feature decoupling network is the same as that of the normal expression-identity feature decoupling network, and both include a feature extraction module, an expression reconstruction module, and an expression classification module. S22: Based on the transfer learning method, the initial model of the micro-expression-identity feature decoupling network is adjusted using pre-processed samples of different facial micro-expression images to obtain the micro-expression-identity feature decoupling network.

5. The micro-expression recognition method based on feature decoupling according to claim 2, characterized in that: The convolutional neural network of the feature extraction module contains at least one convolutional layer and one pooling layer. The convolutional layer includes m1 k1×k1 convolution kernels, where m1∈{64,128,256,512}, k1∈{3,5,7}, and the pooling layer includes k2×k2 pooling kernels, where k2∈{1,2,3}.

6. A system for micro-expression recognition based on feature decoupling, characterized in that: include: one or more processors; A memory storing operable instructions, wherein when the instructions are executed by the one or more processors, the one or more processors perform operations to perform the following steps S1 to S4 for the facial micro-expression image to be identified, thereby completing the recognition of facial micro-expressions in the facial micro-expression image to be identified: S1: Collecting different human face common expression image samples, taking the pre-processed human face common expression image samples as input, based on the feature extraction module, the expression reconstruction module, and the expression classification module, taking the expression category corresponding to the human face common expression image samples as output, constructing a common expression-identity feature decoupling network that separates the facial expression features and facial identity features of each human face common expression image sample, and using different human face common expression image samples, combined with preset corresponding expression categories, training the common expression-identity feature decoupling network to obtain a trained common expression-identity feature decoupling network; S2: Collect different facial micro-expression image samples, take the pre-processed facial micro-expression image samples as input, take the expression category corresponding to each facial micro-expression image sample as output, take the trained common expression-identity feature decoupling network as the initial model, adopt the transfer learning method, construct the micro-expression-identity feature decoupling network, and use different facial micro-expression image samples, combined with the preset corresponding expression category, to train the micro-expression-identity feature decoupling network, and obtain the trained micro-expression-identity feature decoupling network; S3: Based on the common expression-identity feature decoupling network obtained in step S1 and the micro-expression-identity feature decoupling network obtained in step S2, and in combination with the discriminator, an adversarial learning method is adopted to obtain an adversarial network model, and common expression image samples and micro-expression image samples of the same expression category are used to train the adversarial network model to obtain a trained adversarial network model; The specific steps of step S3 are as follows: S31: A human face common expression image sample y belonging to the same expression category c , and a facial micro-expression image sample x c They are respectively input into the normal expression-identity feature decoupling network obtained in step S1 and the micro-expression-identity feature decoupling network obtained in step S2 to obtain the normal expression image samples y c The corresponding common facial expression features And facial micro-expression image samples x c Corresponding facial micro-expression features S32: Fix the parameters of the normal expression-identity feature decoupling network trained in step S1 and the parameters of the micro-expression-identity feature decoupling network trained in step S2, input the normal expression features of the face output by the normal expression-identity feature decoupling network and the micro-expression features of the face output by the micro-expression-identity feature decoupling network into the discriminator for adversarial learning, where the loss function L of the discriminator is D As follows: In the formula, Common facial expression features for prediction The distribution of expression categories, To predict facial micro-expression features Distribution of expression categories; The adversarial loss L of the micro-expression-identity feature decoupling network adv As follows: S33: facial micro-expression image sample x c Input into the micro-expression-identity feature decoupling network, the expression classification module of the micro-expression-identity feature decoupling network classifies the facial micro-expression image sample x c The cross entropy L1 loss for classification is as follows: In the formula, y is the expression category label, L is the total number of expression category labels, To predict facial micro-expression features Distribution of expression categories; S34: Based on the regularization method, the adversarial network model constructed for the ordinary expression-identity feature decoupling network, the micro-expression-identity feature decoupling network, and the discriminator introduces the loss unequal regularization L LIR As follows: L LIR =max{L2-L1,0} Where L2 is the sample y of the common facial expression image c Cross entropy loss for classification; S35: Calculate the overall loss function L of the adversarial network model SUM As follows: L SUM =L1+m1L adv +m2L LIR In the formula, m1 and m2 are hyperparameters; S36: using the human face common expression image samples and human face micro-expression image samples belonging to the same expression category to train the adversarial network model to obtain a trained adversarial network model; S4: Using the micro-expression-identity feature decoupling network in the adversarial network model obtained in step S3 as a micro-expression recognition model, for the face image to be recognized, the recognition of facial micro-expressions in the face image to be recognized is completed.

7. A computer-readable medium storing software, characterized in that: The readable medium includes instructions that can be executed by one or more computers, and when the instructions are executed by the one or more computers, the operations of the micro-expression recognition method based on feature decoupling as described in any one of claims 1-5 are performed.