A micro-expression recognition method based on noise label self-repairing

Abstract

The application discloses a micro-expression recognition method based on noise label self-repairing. The method is based on the optical flow method and shallow-MobileNet network to extract the deep features of micro-expression, suppresses the features irrelevant to the micro-expression movement, and overcomes the local problem existing in the micro-expression movement. An importance reweighting module is used to weight the importance of samples in two dimensions of local and global dimensions, so that uncertain micro-expression samples are expected to be assigned a lower importance weight, and the loss reweighting is applied to suppress the expression of uncertain samples, and the unconsciousness of the network model to uncertain label samples in the training process is overcome. Finally, the re-labeling module tries to identify the noise samples with wrong labels and modify the labels, so that the noise label problem existing in the micro-expression data set is overcome from the root of the data, the purity of the training data samples is improved, and the robustness of the model training is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to the emotion analysis of micro-expressions, specifically to a micro-expression recognition method based on noise label self-healing. Background Technology

[0002] Microexpressions are brief, involuntary facial expressions that typically appear when people consciously or unconsciously conceal their emotions. Therefore, microexpressions can be seen as a "leak" of one's true feelings, especially in high-risk situations. Research on microexpression recognition (MER) can improve our ability to perceive and be sensitive to subtle facial behaviors. It is an important topic in understanding human emotions and emotional phenomena, and microexpression analysis technology has significant research and application value in scenarios such as criminal investigation, clinical medicine, business negotiations, and public safety.

[0003] Traditional micro-expression recognition methods typically employ manual feature extraction, which can be categorized into texture-based feature extraction and geometric transformation-based feature extraction, depending on the processing algorithm. Texture-based features utilize apparent texture features as micro-expression features. Geometric transformation-based features represent facial features by locating the geometric shape of facial landmarks, thus forming a facial feature vector. Common methods include optical flow (OF) and main direction mean optical flow (MDMO), which utilize physical or machine models for micro-expression recognition. However, manually created features suffer from drawbacks such as large parameter counts, complex design, and insufficient resolution, resulting in typically low recognition accuracy in real-world applications.

[0004] In recent years, with the continuous development of deep learning technology, deep convolutional neural networks (CNNs) have achieved remarkable results in computer vision fields such as object detection, semantic segmentation, and image processing due to their powerful feature extraction capabilities and ability to represent complex problems. Features extracted in the context of deep learning can be categorized into features extracted based on CNNs, features extracted based on CNNs and Long Short-Term Memory (LSTM) network architectures, and features extracted based on CNNs and attention mechanisms. These features typically exhibit better performance due to the model's powerful feature learning capabilities. Although deep learning-based micro-expression recognition methods outperform traditional methods based on manual feature extraction, the micro-expressions themselves suffer from limitations such as small amplitude, short duration, and localized facial locations, making them difficult to capture and analyze. This also impacts the collection of micro-expression datasets. Currently used training datasets generally suffer from small data volumes and imbalanced class distributions, making models more prone to overfitting during training. Furthermore, although there are strict environmental requirements for the collection of micro-expression datasets, the subjective judgments of the subjects themselves and the subjective opinions of the annotators can bring some uncertain labeling and incorrect labeling problems to the dataset. The inclusion of noisy labels is not conducive to the robustness of model training and directly increases the difficulty of micro-expression recognition. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a micro-expression recognition method based on noise label self-repair. The method reweights the data according to the importance of the sample data, overcoming the impact of uncertain labels on model training and improving the accuracy and efficiency of micro-expression recognition.

[0006] A micro-expression recognition method based on noise label self-healing specifically includes the following steps:

[0007] Step 1: Feature extraction based on optical flow and Shallow-MobileNet network

[0008] Step 1-1: Optical Flow Feature Extraction

[0009] The TV-L1 optical flow model was used to extract optical flow features in the horizontal and vertical directions from the starting and peak frames of micro-expressions, and then optical strain optical flow features were calculated. Optical strain optical flow features can characterize the degree of facial deformation and are not easily affected by lighting conditions or facial occlusion conditions. Then, the optical flow features in the two directions were superimposed with the optical strain optical flow features to form an optical flow map, which effectively characterizes the local motion when facial micro-expressions occur.

[0010] Steps 1-2: Deep feature extraction based on Shallow-MobileNet

[0011] The Shallow-MobileNet model architecture uses the MobileNet-V3 model architecture as its backbone, including the Block layer from MobileNet-V3 and an average pooling layer for eliminating redundant features. The Shallow-MobileNet model analyzes and calculates the optical flow map obtained in step 1-1, transforming the low-level optical flow features into high-level features with discriminative rows, resulting in a deep feature vector representation. The deep features of each micro-expression are then sequentially input into the classifier model to obtain the corresponding category prediction value.

[0012] Step 2: Weight the data samples according to their importance.

[0013] Each optical flow map is treated as a data sample and input into the model in batches. An importance reweighting module is used to evaluate the importance of each data sample to determine its contribution to model training. Lower importance weights are assigned to micro-expression samples with uncertain expectations, and these weights are applied to the loss weighting to suppress the expression of uncertain samples. Importance weights include local importance weights and global importance weights.

[0014] Step 2-1: Calculate the local importance weights

[0015] Local importance weights are used to perceive the relative importance of samples within each batch. First, an initial class center feature vector is defined for each class and updated by minimizing the Center-Loss loss function. The local importance module calculates the Euclidean distance between each sample in the current batch and the updated class center feature vector using the Distance distance function, and assigns a local importance weight to each sample based on this distance value. Within the same batch, samples farther from the center feature vector have smaller local importance weights.

[0016] Step 2-2: Calculate the global importance weight.

[0017] Global importance weights are used to perceive the importance of each sample in the overall training data. The global importance weighting module consists of a linear fully connected layer and a sigmoid activation function, which adaptively learns the features of each sample data and assigns different global importance weights to the samples.

[0018] Steps 2-3: Weight fusion and loss reweighting

[0019] The local importance weights calculated in steps 2-1 and 2-2 are fused with the global importance weights. During fusion, a hyperparameter λ is set to assign different proportional coefficients to the two weights, thus obtaining the comprehensive importance weight of each sample considering both global and local dimensions. The fused comprehensive importance weight is then applied to reweight the class prediction values ​​output by the classifier model in step 1 to suppress the expressive power of uncertain samples. A hyperparameter γ is set, and based on the fused comprehensive importance weight, samples in each batch are divided into high-importance and low-importance groups. In the next step, samples in the low-importance group are then discriminated.

[0020] Step 3: Update noise labels

[0021] A relabeling module is established to discriminate the labels of samples in the low-importance group based on the grouping of the intra-class importance weighting module. When the difference between the maximum predicted class probability and the original label probability of a sample exceeds a set threshold β, and the distance between the maximum predicted class and the class center feature vector of the sample is less than the distance between the original label and the class center feature vector, the labels of the training samples are self-repaired by replacing the original labels of the samples with the maximum predicted class, thereby improving the purity of the data samples during the training process.

[0022] As a preferred approach, to avoid prematurely identifying and altering noise labels, a slow-start warm-up method is used to start the relabeling module. The relabeling module is then started only after the model has been trained for a fixed number of epochs and the importance reweighting module has reached a relatively stable state.

[0023] The present invention has the following beneficial effects:

[0024] 1. Lightweight networks based on Shallow-MobileNet feature fewer parameters, lower computational cost, and shorter inference time, making them better suited to the limited sample size of datasets in micro-expression recognition tasks. This reduces the dependence of network model training on the amount of data samples. While maintaining training speed, Shallow-MobileNet introduces a SE channel attention mechanism, allowing the model to focus more on information-rich parts of the feature map. This suppresses facial features unrelated to micro-expression movements, overcoming the locality problem inherent in micro-expression movements. This helps the model extract discriminative high-level features and obtain deep feature vector representations of the samples.

[0025] 2. The importance reweighting module innovatively captures the importance of sample data through evaluation, overcoming the unconsciousness of uncertain labeled samples during network model training by reweighting the loss. It introduces both local and global dimensions to evaluate sample importance, accurately and comprehensively determining the importance of samples, thereby suppressing the expressive power of uncertain samples during model training, improving the model's robustness, and providing a reference for the relabeling module.

[0026] 3. The relabeling module attempts to identify and modify the labels of mislabeled noise samples by setting strict correction conditions. This overcomes the problem of noisy labels in the micro-expression dataset from the root of the data, improves the purity of the training data samples, and enhances the robustness of model training. Attached Figure Description

[0027] Figure 1 This is an overall flowchart of the present invention;

[0028] Figure 2 The optical flow map is obtained by superimposing the horizontal and vertical optical flows extracted by the optical flow method with the strain optical flow calculated subsequently.

[0029] Figure 3 This is a diagram of the Shallow-MobileNet network model structure.

[0030] Figure 4 A visualization of deep features based on Center-Loss loss; Detailed Implementation

[0031] The present invention will be further explained below with reference to the accompanying drawings;

[0032] like Figure 1 As shown, a micro-expression recognition method based on noise label self-repair consists of three parts. The first part involves deep feature extraction based on optical flow and the Shallow-MobileNet network. The second part uses an importance reweighting module to weight the samples in each batch group in both local and global dimensions, and then divides them into low-importance and high-importance groups based on hyperparameters. The third part identifies noise in the samples of the low-importance group according to set conditions and performs self-repair of the noise labels. The specific steps are as follows:

[0033] Step 1: Feature extraction based on optical flow and Shallow-MobileNet network.

[0034] Step 1-1: Optical Flow Feature Extraction

[0035] First, optical flow features are extracted from the micro-expression data in the dataset. Figure 1The Onset frame represents a static frame where no micro-expression movement has occurred, while the Apex frame represents the frame where the amplitude of the micro-expression movement reaches its peak. Local motion information of facial micro-expressions in these two frames can be extracted using the TV-L1 optical flow method. The magnitude and direction of each pixel's motion are represented by a two-dimensional optical flow field. Strain patterns are used to measure the degree of deformation of an object under external force, effectively reflecting the region where micro-expression movements occur on the face. For sufficiently small facial pixel movements, it can represent the magnitude of deformation of facial muscle tissue, showing good performance in micro-expression recognition tasks. Given a two-dimensional optical flow vector, the optical strain feature can be derived to describe the facial motion pattern. By attaching the optical strain feature to the optical flow field, each micro-expression data point can be represented as a triple based on the optical flow feature representation. The superimposed three optical flow feature maps are shown below. Figure 2 As shown.

[0036] Steps 1-2: Deep feature extraction based on Shallow-MobileNet

[0037] The optical flow triplet features obtained in step 1-1 are normalized to 3×28×28 and then used as input to the Shallow-MobileNet network for deep feature extraction, aiming to obtain a high-level feature vector representation in the feature map. The network model structure of Shallow-MobileNet is as follows: Figure 3 As shown, firstly, two 3×3 convolutions are used to increase the dimensionality and reduce the width of the feature map. Then, a channel attention mechanism is used to learn the importance of each feature channel. Based on the importance, the necessary micro-expression motion features are focused on while suppressing some unnecessary facial features. Next, a 1×1 convolution is used to reduce the dimensionality of the features, retaining effective features and reducing computational cost. Average pooling is used to suppress noise and reduce redundant information. Finally, a higher-quality deep semantic feature with a dimension of 7×8×8 is obtained. Finally, the deep features of each micro-expression are sequentially input into the classifier model to obtain the corresponding class prediction value.

[0038] Step 2: Weight the data samples according to their importance.

[0039] The importance reweighting module determines the contribution of data samples to model training by evaluating their importance. It is expected that uncertain micro-expression samples will be assigned lower importance weights and applied to loss weighting to suppress the expression of uncertain samples. The weights are obtained through two levels: local importance weights and global importance weights.

[0040] Step 2-1, Local Importance Weights

[0041] Due to the complexity and unique characteristics of facial micro-expression data, samples of the same type of micro-expression data from different subjects exhibit significant intra-class variability. Therefore, the Center-Loss loss function is introduced to reduce intra-class distance. The Center-Loss loss function L... Center The calculation formula is:

[0042]

[0043] Where y i x is the category label of the i-th sample. i Let be the deep features of the i-th sample, which are flattened into a one-dimensional feature vector to represent the deep features of the current sample. Indicates that in the current batch, y i The in-class center vectors of all samples in a category, where N is the number of samples in a batch. This is achieved by minimizing the center loss L. Center For the in-class center vector The update pulls samples of the same class towards the corresponding center, thereby reducing the overall variation within the class. The deep feature visualization results using the Center-Loss loss function are shown below. Figure 4 As shown, the sample space of each category exhibits a circular pattern, clustering around the intra-class central feature vector, effectively reducing the intra-class distance between samples of the same class.

[0044] The quantification of local importance weight is determined by the relative importance among samples in the same batch. The Euclidean distance between each sample and the intra-class center vector of its class is used as the evaluation criterion. The sample farthest from its class center should have a relatively small importance weight. The function is used to calculate the Euclidean distance between the deep feature vector of a sample and the corresponding class center vector, as shown in Formula 2:

[0045]

[0046] Where n is the dimension of the feature vector of each sample.

[0047] The calculation of local importance weights requires inferring the relative importance among samples within the same batch. The sample closest to its intra-class center within the same batch is used as a reference. Then, the intra-class center distances of all samples are subtracted accordingly. Thus, the sample with the highest importance receives a value of zero, while samples with lower importance receive smaller negative values. Finally, the local importance weight of a sample can be obtained using a power function, as shown in Formula 3.

[0048]

[0049] in is the local importance weight value of the i-th sample, and k is the sample number in the current batch that is closest to its corresponding class center.

[0050] Step 2-2, Global Importance Weights

[0051] Global importance weights are derived by perceiving the overall importance of each training data sample, capturing the contribution of each sample to model training. The global importance weighting module consists of a fully connected (FC) layer and a sigmoid activation function. The FC layer transforms the deep feature vector extracted by the Shallow-MobileNet network into a scalar, the magnitude of which is typically positively correlated with the importance of the current sample.

[0052]

[0053] in W represents the global importance weight of the i-th sample. a These are the parameters of the fully connected layer, where σ represents the sigmoid function, used to map global importance weights to the (0,1) interval.

[0054] Steps 2-3: Weight fusion and loss reweighting

[0055] The final sample importance weights are obtained by fusing local importance weights based on intra-class center distance and adaptive global importance weights. A hyperparameter λ is introduced to evaluate the importance of each weight component to obtain the optimal fused result. This two-dimensional evaluation criterion improves the fairness and effectiveness of quantifying sample importance weights. The final comprehensive sample importance weight α... i for:

[0056]

[0057] In this embodiment, the value of the hyperparameter λ is set to 0.3.

[0058] After obtaining the final importance weights of the samples through weight fusion, the output of the final classifier is reweighted to ensure that the model learns meaningful and important sample data while suppressing the negative impact of uncertain samples on model learning. Finally, the Adam optimization algorithm is used for optimization, and the Logit-Weighted Cross-Entropy loss (WCE-Loss) loss function is used during backpropagation.

[0059]

[0060] Where C is the number of classes in the classifier, and W... j L is the parameter of the j-th classifier in the final fully connected layer of the model.WCE The value of is positively correlated with the overall importance weight. By combining the joint center loss and adding a hyperparameter δ to balance the two losses, the overall loss L of the model is:

[0061] L=δL center +L WCE (7)

[0062] The importance reweighting module then sorts the learned attention weights in descending order and divides them into high-importance and low-importance groups according to a certain ratio γ:

[0063] group H ={x i |1 <i≤M},group L ={x i |M <i≤N} (8)

[0064] Where group H and group L These are the high importance group with γ*N=M samples and the low importance group with NM samples, respectively, where γ=0.7.

[0065] Step 3: Update noise labels based on the relabeling module

[0066] In steps 2-3, the importance weights of the samples obtained in each batch were divided into low importance groups. L and high importance group H Since correcting correct samples to incorrect labels would increase the number of noisy labels, to reduce the risk of data label relabeling, we chose to only apply the labels to low-importance groups. L The sample labels in the data were corrected.

[0067] In addition, strict conditions are set for the determination of noise labels to ensure the accuracy of noise label correction. The relabeling module will only perform self-repair of the labels of data samples when the following two conditions are met simultaneously, thereby improving the purity of data samples during the training process:

[0068] (1) The difference between the softmax value of the class with the maximum predicted probability of the current sample output by the model and the original class of the sample exceeds the threshold β;

[0069] (2) The intra-class center distance corresponding to the largest predicted class of the sample is less than the intra-class center distance under the original class label.

[0070] Expressed as a formula:

[0071]

[0072] Where y′ is the corrected new label, max is the class number corresponding to the maximum predicted class of the sample output by the model, gt represents the original label number of the sample, and P max It is the probability of the largest predicted class, P. gt β is the probability of the original sample class, and β is a hyperparameter setting that sets the threshold for the difference between the two probabilities. As a preferred value, it is set to 0.7, which also indicates that it is only considered as one of the conditions to be met when the model's predicted value of the label is very different from the original class, in order to ensure the rigor of the noise label change.

[0073] To prevent the relabeling module from prematurely correcting the dataset, it will start slowly with a warm-up, only commencing operation after the importance reweighting module and the model have reached a certain stable state, ensuring the stability of the module's operation. Samples that meet the correction criteria will then be relabeled. The correction will occur in the PyTorch framework's DataLoader, without changing the original labels of the samples in the dataset. These corrected labels may receive higher importance weights in the next epoch, thus achieving self-correction of noisy labels and making the model training purer and more robust.

[0074] Micro-expression recognition was performed on the Full, SMIC, CASMEII, and SAMM datasets using our proposed method and existing techniques. The results are shown in Table 1. Our proposed method achieved the best performance compared to traditional methods (Bi-WOOF) and deep learning-based methods (AlexNet, OFAPexNet, CapsuleNet, RCN-A, STSTNet). Compared to the single method STSTNet, our proposed method improved the UF1 and UAR metrics by 3.78% and 1.41% respectively on the Full dataset. It also showed good results on the SMIC and CASMEII datasets, demonstrating the superiority of our proposed method and its ability to effectively improve the accuracy of micro-expression recognition.

[0075]

[0076] Table 1.

Claims

1. A micro-expression recognition method based on noise label self-healing, characterized in that: Specifically, the following steps are included: Step 1: Micro-expression feature extraction The TV-L1 optical flow model is used to extract optical flow features in the horizontal and vertical directions from the starting and peak frames of micro-expressions, and optical strain optical flow features are further calculated. Then, the three optical flow features are superimposed to form an optical flow map, which is then passed through the Block of the MobileNet-V3 model structure and an average pooling layer to obtain a deep feature vector representation. Finally, the deep features of each micro-expression are input into the classifier model to obtain the corresponding category prediction value. Step 2: Weight the data samples according to their importance. All deep features of micro-expressions are divided into equal batches, and each sample is weighted by importance within each batch: Step 2-1: Calculate the local importance weights For each category, an initial category center feature vector is defined and updated by minimizing the Center-Loss loss function. Then, the Euclidean distance between each sample in the current batch and the updated category center feature vector is calculated. The samples that are farther away from the center feature vector have smaller local importance weights. Step 2-2: Calculate the global importance weight. The features of each sample data are adaptively learned through a linear fully connected layer, different global importance weights are assigned to the samples, and the global importance weight values ​​are mapped to the range [0,1] through the sigmoid activation function; Steps 2-3: Weight fusion and loss reweighting Set the hyperparameter λ to assign different proportional coefficients to the local importance weights and global importance weights calculated in steps 2-1 and 2-2, and fuse them to obtain the comprehensive importance weight of each sample; The loss reweighting of the class prediction values ​​output by the classifier model in step 1 is performed using a comprehensive importance weight; Then, set the hyperparameter γ, and based on the integrated importance weight after fusion, divide the samples in each batch into high importance group and low importance group, and in the next step, discriminate the samples in the low importance group; Local importance weight The calculation formula is: Let be the deep features of the i-th sample. It is the sample label that is closest to the center of its corresponding class in the current batch. This represents the deep feature vector of the i-th sample and its corresponding class center vector. Euclidean distance between them: Global importance weight The calculation formula is: in, These are the parameters of the fully connected layer. This represents the sigmoid function, used to map global importance weights to the (0,1) interval; Overall Importance Weight for: Step 3: Update noise labels For samples in the low importance group, discrimination is performed. When the difference between the maximum predicted class probability and the original label probability of a sample exceeds a set threshold... When the distance between the maximum predicted class of a sample and the feature vector of the class center is less than the distance between the original label and the feature vector of the class center, the label of the sample is self-repaired by replacing the original label of the sample with the maximum predicted class, thereby improving the purity of the data samples during the training process.

2. The micro-expression recognition method based on noise label self-healing as described in claim 1, characterized in that: The deep feature extraction process in step 1 is as follows: First, two 3×3 convolutions are used to increase the dimensionality and reduce the width of the optical flow map respectively. Then, the channel attention mechanism is used to obtain the importance of each feature channel. After that, a 1×1 convolution is used to reduce the dimensionality of the feature. Average pooling is used to suppress noise and reduce redundant information. Finally, the deep features are obtained.

3. The micro-expression recognition method based on noise label self-healing as described in claim 1, characterized in that: Center-Loss loss function The calculation formula is: in It is the category label of the i-th sample. Let be the deep features of the i-th sample, which are flattened into a one-dimensional feature vector to represent the deep features of the current sample. Indicates that in the current batch, The intra-class center vector of all samples in the category. Given the number of samples in a batch; minimize the center loss. For the in-class center vector Update.

4. The micro-expression recognition method based on noise label self-healing as described in claim 1, characterized in that: Set the hyperparameter λ=0.

3.

5. The micro-expression recognition method based on noise label self-healing as described in claim 1, characterized in that: To avoid prematurely identifying and changing noise labels, the noise label update step is performed after the model has been trained for a fixed number of epochs.

6. The micro-expression recognition method based on noise label self-healing as described in claim 1 or 3, characterized in that: The model is optimized using the Adam optimization algorithm, and the WCE-Loss loss function is used during backpropagation. in, To take into account the importance weights, It is the category label of the i-th sample. Let be the deep features of the i-th sample. The number of samples in a batch. It is the number of classes in the classifier. These are the parameters of the j-th classifier in the final fully connected layer of the model. The value is positively correlated with the overall importance weight; joint center loss is used, and hyperparameters are added to balance the two losses. The overall loss L of the model is: 。

Images