Micro-expression recognition method based on meta-learning

Abstract

The present invention discloses a micro-expression recognition method based on meta-learning, including data preprocessing; micro-expression feature extraction and fusion; model pre-training; micro-expression recognition. The proposed method is based on meta-learning and is suitable for small sample learning. The fusion features of optical flow and frame difference are respectively input into the meta-learning model pre-trained with macro-expression and micro-expression, and the obtained feature vectors are concatenated and classified, thereby greatly improving the generalization performance of model recognition and the recognition effect of micro-expressions.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and relates to a method for micro-expression recognition using a meta-learning algorithm in deep learning. Background Art

[0002] Recently, deep learning has made considerable progress in many fields. However, due to its dependence on large-scale data, deep learning cannot solve many problems in the real world. This requires the ability to learn quickly. Humans have the ability to use past experience and knowledge to guide the learning of new knowledge through a small number of examples. On this basis, few-sample learning methods have received widespread attention.

[0003] Meta-learning is currently a key method to solve few-shot learning. It first extracts experience from learning a series of known tasks, and then uses the obtained experience to handle new tasks with a small number of samples.

[0004] Expressions are an intuitive reflection of human emotions and an important means of interpersonal communication. Therefore, expression recognition has always been one of the research focuses in the field of computer vision, and has achieved remarkable results in the past few decades. Micro-expressions, as a special kind of facial expression, are spontaneously generated when a person is unconscious. Compared with ordinary expressions, micro-expressions have the characteristics of short duration, low action intensity, difficulty in concealing or disguising, and are usually analyzed in video state. Since micro-expressions are difficult to conceal or disguise, they can better reflect people's true emotions, making micro-expressions more authentic and reliable in emotional and psychological analysis, and have potential utilization value and broad application prospects in communication and negotiation, psychological intervention, and teaching evaluation. Summary of the invention

[0005] The purpose of the present invention is to provide a micro-expression recognition method based on meta-learning, which solves the problem of small amount of micro-expression data and subtle changes in movements. The beneficial effect of the present invention is that it utilizes the characteristics of the meta-learning method and can achieve good generalization results for micro-expression recognition of a small number of samples.

[0006] The technical solution adopted by the present invention is carried out according to the following steps:

[0007] Step 1. Data preprocessing;

[0008] Step 2. Extract and fusion of micro-expression features;

[0009] Step 3. Model pre-training;

[0010] Step 4. Micro-expression recognition.

[0011] Furthermore, in Step 1., the micro-expression dataset is preprocessed. The micro-expression dataset contains several micro-expression video clips, and facial feature points are aligned using facial registration; facial registration extracts facial feature points, and the extracted feature points are mapped from the new image to the template.

[0012] Furthermore, in Step 2., the optical flow features and frame difference features between the starting frame and the vertex frame of the micro-expression sequence are calculated, and the two features are fused. The calculation process of the optical flow features is as follows:

[0013] The optical flow feature is calculated using the Gunnar Farneback algorithm, which extracts the dense optical flow between the start frame and the vertex frame, obtains the optical flow values ​​of the x and y components, calculates the vector modulus, and divides the optical flow value by its maximum value to make it range between 0 and 1. The normalized x component, y component, and vector modulus size constitute the three channels represented by the optical flow feature.

[0014] Furthermore, in Step 3., the feature extraction model of the framework is optimized. The feature extraction model uses the Resnet-18 model as the backbone network, and the loss is set to the cross entropy loss. The model parameters are optimized using the macro expression dataset and the micro expression dataset respectively to obtain two different models. After removing the fully connected layer, the network structure and the model parameters optimized on different datasets are used as two different feature extractors (classifiers).

[0015] Furthermore, in Step 4., the micro-expression dataset is divided according to the meta-learning definition, and its features are respectively input into the meta-fusion learning network pre-trained with the macro-expression dataset and the micro-expression dataset to obtain two different feature vectors, which are then concatenated for micro-expression classification.

[0016] The data division method according to the meta-learning definition is as follows:

[0017] The dataset is divided into training samples and test samples. Each part consists of tasks, and these tasks have their own training sets and test sets. In order to avoid confusion, the training set in the task is called the support set, and the test set is called the query set. N-way K-shot is a common experimental setting in few-shot learning. N-way means that there are N categories in the training data, and K-shot means that there are K labeled data under each category. The LOSO (Leave-one-subject-out) verification method is used, that is, all videos of a target are used as the verification set each time, and the model does not include the distribution of micro-expressions of the target. The single target in the verification set is regarded as a new class, and all other target sets except this target are regarded as base classes (training samples), and training is performed in sequence. The query set in the test sample is composed of new classes, and the support set is obtained from the base class.

[0018] The method of constructing the meta-fusion learning module is as follows:

[0019] The meta-fusion learning module solves the N-way K-shot classification problem of frame difference and optical flow fusion features. Based on the pre-trained feature extractors, the pre-processed micro-expression data features are divided into multiple tasks, and the support set is encoded by the classifier in each task to obtain the feature vector. In the meta-training stage, the average feature representation of each class is calculated by the feature average of all samples of the class:

[0020]

[0021] Among them, the average characteristic ω c as the centroid of class c, S as the support set, and S c Represents a small sample set of category c. The vector f obtained by calculating the encoding of the test set using cosine similarity θ (x) and the center of mass ω c The cosine distance between <f θ (x),ω c >. The test set samples are classified by the softmax probability of the cosine distance:

[0022]

[0023] The loss function for the N-way K-shot classification problem uses the cross entropy loss calculated from the predicted distribution and true distribution of the above test set:

[0024]

[0025] in represents the predicted distribution, C is the number of categories, Represents the true distribution of the i-th data sample. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flow chart of micro-expression recognition performed by the present invention;

[0027] Figure 2 The overall structure of the present invention;

[0028] Figure 3 It is the meta-baseline algorithm model. DETAILED DESCRIPTION

[0029] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] The present invention Figure 1As shown, micro-expression recognition is divided into four modules: data preprocessing, micro-expression feature extraction and fusion, model pre-training, and micro-expression recognition.

[0031] In the data preprocessing stage, the micro-expression dataset to be processed is preprocessed. The micro-expression dataset contains several micro-expression video clips, and facial feature points are aligned using facial registration; facial registration extracts facial feature points and maps the extracted feature points from the new image to the template.

[0032] In the micro-expression feature extraction and fusion stage, the optical flow features and frame difference features between the starting frame and the vertex frame of the micro-expression sequence are calculated, and the two features are fused. The optical flow feature is calculated using the Gunnar Farneback algorithm, and the dense optical flow between the starting frame and the vertex frame is extracted to obtain the optical flow values ​​of the x and y components and calculate the vector modulus. The optical flow value is divided by its maximum value to make it range between 0 and 1. The normalized x component, y component and vector modulus size constitute the three channels of the optical flow feature representation.

[0033] The feature extraction model of the framework is optimized in the model pre-training stage. The feature extraction model uses the Resnet-18 model as the backbone network and sets the loss to the cross entropy loss. The model parameters are optimized using the macro expression dataset and the micro expression dataset respectively to obtain two different models. After removing the fully connected layer, the network structure and the model parameters optimized on different datasets are used as two different feature extractors (classifiers).

[0034] In the micro-expression recognition stage, the micro-expression dataset is divided according to the meta-learning definition, and its features are respectively input into the meta-fusion learning network pre-trained with the macro-expression dataset and the micro-expression dataset to obtain two different feature vectors, which are then concatenated for micro-expression classification.

[0035] Figure 2 For the overall architecture of the present invention, before extracting the features of micro-expressions, the micro-expression video clips must first be preprocessed, including face alignment and face cropping. Then, the start frame and vertex frame obtained by preprocessing are used to calculate the optical flow and frame difference features, and the two feature combinations are respectively input into the meta-learning network that has been pre-trained for micro-expressions and macro-expressions, and the obtained feature vectors are spliced ​​to calculate the cosine distance for micro-expression classification.

[0036] Figure 3 The meta-baseline algorithm model in the present invention includes a pre-training module and a meta-learning module. The resnet18 network is used in the pre-training module, and the cross entropy loss is used for training on the macro expression data set and the micro expression data set respectively to obtain two different sets of model parameters, and then the network after removing the fully connected layer is used as a classifier.

[0037] Based on the pre-trained feature extractors, the pre-processed micro-expression data features are divided into multiple tasks, and the support set is encoded by the classifier in each task to obtain the feature vector. In the meta-training stage, the average feature representation of each class is calculated using the feature average of all samples of the class:

[0038]

[0039] Among them, the average characteristic ω c as the centroid of class c, S as the support set, and S c Represents a small sample set of category c. The vector f obtained by calculating the encoding of the test set using cosine similarity θ (x) and the center of mass ω c The cosine distance between <f θ (x),ω c >. The test set samples are classified by the softmax probability of the cosine distance:

[0040]

[0041] The loss function for the N-way K-shot classification problem uses the cross entropy loss calculated from the predicted distribution and true distribution of the above test set:

[0042]

[0043] in represents the predicted distribution, C is the number of categories, Represents the true distribution of the i-th data sample.

[0044] Technical effect:

[0045] In order to verify the effectiveness of the method proposed in this invention, we conducted verification on three public micro-expression datasets, namely, Spontaneous Micro-Expression Datebase (SMIC), the Chinese Academy of Sciences Micro-expression (CASME) dataset and CASME II dataset.

[0046] SMIC: SMIC was designed by Zhao Guoying's team at the Center for Machine Learning and Vision Research at the University of Oulu in Finland in 2012. It is the world's first public database of spontaneous micro-expressions. The image resolution is 640*480 pixels, and the facial area resolution is 190*230 pixels. SMIC contains 164 micro-expression videos of 16 subjects. The database divides micro-expressions into three categories: positive, negative, and surprise. Positive expressions are "happy" expressions, and negative expressions include "sadness", "anger", "fear", and "disgust". It should be noted that SMIC does not mark vertex frames and AU units.

[0047] CASME: This database contains 195 micro-expression videos of 35 subjects (13 females and 22 males). Fu Xiaolan's team summarized the expression induction method published by Ekman, using 17 short videos that can induce emotions such as "disgust", "depression", "surprise", and "tension", and asked the subjects to suppress their expressions. The entire process of micro-expressions was captured by a 60-frame-per-second camera. There are two image resolutions: 640*480 and 1280*720, and the facial expression resolution is 150*190. The obtained micro-expression samples were encoded in AUs, including three parts: onset, apex, and offset.

[0048] CASME II: The CASME II dataset currently contains 255 micro-expression video clips, which are shot with a high-speed camera at 200FPS. The facial resolution of the video clips can reach about 280*340 pixels, and the image resolution is 640*480 pixels. The CASME II dataset divides micro-expressions into 7 categories for annotation, namely happiness, disgust, surprise, repression, sadness, fear, and others. In addition, the CASME II dataset also annotates the onset, apex, and offset of micro-expression activities, among which Apex is helpful for micro-expression recognition. In addition to annotating emotions, the CASME II dataset also annotates the AU (facial activity unit) of each micro-expression, which can be used as the basis for classifying micro-expressions. In this method, only four categories, namely, surprise, repression, happiness, and disgust, are used.

[0049] For a fairer comparison, all experiments use LOSO (Leave-one-subject-out) cross-validation, where one subject's samples are used as the test set and the rest of the samples are used as the training set. Taking CASME II as an example, we collected micro-expression video sequences of 26 subjects, and each time we used the micro-expression video sequences of 25 subjects as our training set, and the rest of the subjects as the test set for the experiment.

[0050] The classification accuracy (acc) is the ratio of the number of correctly classified samples to the total number of samples in the experiment.

[0051] The proposed method is compared with the most advanced methods. Tables 1, 2, and 3 show the comparison results on the SMIC, CASME, and CASME II datasets, respectively.

[0052] Table 1: Comparison results on the SMIC dataset

[0053] method Accuracy LBP-SIP 0.4212 FHOFO 0.5122 FR 0.579 3DFCNN 0.5549 The present invention 0.5915

[0054] Table 2: Comparison results on the CASME dataset

[0055]

[0056]

[0057] Table 3: Comparison results on the CASME II dataset

[0058] method Accuracy DTCM 0.7206 The present invention 0.7933

[0059] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention are within the scope of the technical solution of the present invention.

Claims

1. A micro-expression recognition method based on meta-learning, characterized by Follow these steps: Step 1. Preprocess the micro-expression dataset, which contains several micro-expression video clips. Use facial registration to align facial feature points. Facial registration extracts facial feature points and maps the extracted feature points from the new image to the template. Step 2. Calculate the optical flow feature and frame difference feature between the starting frame and the vertex frame of the micro-expression sequence, and fuse the two features. The calculation process of the optical flow feature is as follows: The optical flow feature is calculated using the Gunnar Farneback algorithm. The dense optical flow between the starting frame and the vertex frame is extracted to obtain the optical flow values ​​of the x and y components and calculate the vector modulus. The optical flow value is divided by its maximum value to make it range between 0 and 1. The normalized x component, y component and vector modulus size constitute the three channels of the optical flow feature representation. Step 3. Optimize the feature extraction model of the framework. The feature extraction model uses the Resnet-18 model as the backbone network and sets the loss to the cross entropy loss. Use the macro expression dataset and micro expression dataset to optimize the model parameters and obtain two different models. After removing the fully connected layer, use the network structure and the model parameters optimized on different datasets as two different feature extractors. Step 4. The micro-expression dataset is divided according to the meta-learning definition, and its features are respectively input into the meta-fusion learning network pre-trained with the macro-expression dataset and the micro-expression dataset to obtain two different feature vectors, which are then concatenated for micro-expression classification. The data division method according to the meta-learning definition is as follows: The dataset is divided into training samples and test samples. Each part consists of tasks. These tasks have their own training sets and test sets. In order to avoid confusion, the training set in the task is called the support set, and the test set is called the query set. N-way K-shot is a common experimental setting in few-shot learning. N-way means that there are N categories in the training data, and K-shot means that there are K labeled data under each category. The LOSO verification method is used, that is, all videos of a target are used as the verification set each time, and the model does not contain the distribution of micro-expressions of the target. The single target in the verification set is regarded as a new class, and all other target sets except the target are regarded as base classes, and training is performed in sequence. The query set in the test sample is composed of the new class, and the support set is obtained from the base class. The method of constructing the meta-fusion learning module is as follows: The meta-fusion learning module solves the N-way K-shot classification problem of frame difference and optical flow fusion features. Based on the pre-trained feature extractors, the pre-processed micro-expression data features are divided into multiple tasks. In each task, the support set is encoded by the classifier to obtain the feature vector; in the meta-training stage, the average feature representation of each class is calculated using the feature average of all samples of the class: ； Among them, the average feature as the centroid of class c, S as the support set, Represents a small sample set of category c, and uses cosine similarity to calculate the vector encoded by the test set and centroid The cosine distance between , the test set samples are classified by the softmax probability of the cosine distance: ； The loss function for the N-way K-shot classification problem uses the cross entropy loss calculated from the predicted distribution and true distribution of the above test set: ； in represents the predicted distribution, C is the number of categories, Represents the true distribution of the i-th data sample.

Images