Micro-expression recognition method based on optical flow features

Abstract

The application discloses a micro-expression recognition method based on optical flow features, first selects micro-expression data sets: CASME, CASME II and CAS(ME) 2 , and respectively maps emotions of all video frame sequences in the three data sets into three categories of "Negative", "Positive" and "Surprise"; all video frame sequences of the selected data set are preprocessed to obtain video frame sequences with a resolution of 128*128; then, a TV-L1 energy functional is used to extract optical flow features of the video frame sequences, and the optical flow features are spliced by a channel superposition method to serve as inputs of a neural network RSCANet; finally, the neural network RSCANet is used to extract micro-expression features and obtain a classification result. The application solves the problem that it is difficult to extract subtle motion changes of a face in a video frame due to the characteristics of micro-expression in the prior art.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of digital image processing and recognition, and particularly relates to a micro-expression recognition method based on optical flow features. BACKGROUND

[0002] In recent years, micro-expression recognition has important research value in the fields of psychological counseling and confidence lie detection. However, due to the characteristics of micro-expression itself, such as small motion amplitude and short duration, it is difficult to recognize micro-expression manually. Even if the recognition accuracy of a psychological expert after professional training is only about 47%. Because relying on the human eye to recognize micro-expression is limited by professional training and a large amount of time cost, and the recognition accuracy is low, the wide promotion of micro-expression recognition has been seriously hindered. With the rapid development of computer vision and machine learning technology, more and more researchers apply machine learning algorithms to micro-expression recognition, which solves many difficulties existing in manual recognition and improves the recognition accuracy. However, due to the characteristics of micro-expression, so far, how to extract the subtle motion changes of the face in the video frame is still a key problem in this field. Therefore, micro-expression recognition is still in the rapid development stage, and the research on micro-expression recognition has become an important topic in the field of affective computing. SUMMARY

[0003] The purpose of the application is to provide a micro-expression recognition method based on optical flow features, which solves the problem of difficulty in extracting the subtle motion changes of the face in the video frame due to the characteristics of micro-expression in the prior art.

[0004] The technical solution adopted by the application is a micro-expression recognition method based on optical flow features, which is implemented according to the following steps:

[0005] Step 1, select micro-expression data sets: CASME, CASME II and CAS(ME) 2 , and map the emotions of all video frame sequences in the three data sets to the "Negative", "Positive" and "Surprise" three categories respectively;

[0006] Step 2, pre-process all video frame sequences of the selected data set to obtain video frame sequences with a resolution of 128x128;

[0007] Step 3, use TV-L1 energy functional to extract the optical flow features of the video frame sequences obtained in step 2, and splice the optical flow features through channel superposition as the input of the neural network RSCANet;

[0008] Step 4, use the neural network RSCANet to extract micro-expression features and obtain the classification result.

[0009] The application is also characterized in that,

[0010] Step 1 is specifically implemented according to the following steps:

[0011] In the CASME dataset: "Disgust", "Repression" and "Sadness" are mapped to "Negative", "Happiness" is mapped to "Positive", and "Surprise" remains unchanged; in the CASME II dataset: "Disgust" and "Repression" are mapped to "Negative", "Happiness" is mapped to "Positive", and "Surprise" remains unchanged; in the CAS(ME) 2 The emotion categories in the dataset remain unchanged as the original "Negative", "Positive" and "Surprise".

[0012] Step 2 is specifically implemented as follows:

[0013] The model provided by the dlib library is used to detect the 68-point key point information to realize face alignment and cropping operation, and the resolution of all video frame sequences is uniformly adjusted to 128x128. The starting frame and the peak frame with the largest micro-expression change of each video are selected as the input of step 3.

[0014] Step 3 is specifically implemented as follows:

[0015] Let u and v represent the horizontal component and the vertical component of the optical flow respectively:

[0016]

[0017] Where dx and dy represent the pixel changes along the x and y dimensions respectively, and dt represents the time change;

[0018] The optical strain ε is obtained by calculating the derivative of the optical flow, and is defined as:

[0019]

[0020] Where u = [u, v] T is the displacement vector, so ε is also expressed as:

[0021]

[0022] Where the diagonal strain components (ε xx ,ε yy ) are normal strain components, and (ε xy ,ε yx) is the shear strain component; then, the optical strain value of each pixel is calculated by taking the sum of the square of the normal strain component and the shear strain component, so that:

[0023]

[0024] Finally, the three optical flow features of u, v and |ε| are spliced together through channel stacking as the input of the subsequent neural network RSCANet.

[0025] Step 4 is specifically as follows:

[0026] Step 4.1, an RSCANet network is constructed, and the specific structure of the RSCANet network is as follows: the first layer is a two-dimensional convolution Conv1, which is used to extract the shallow features of the input obtained in step 3; two residual contraction modules RSBlock1 and RSBlock2 are connected after Conv1 to enhance the noise resistance of the neural network; a residual block ARBlock with a fusion channel attention mechanism is added after the two residual contraction modules, the residual block ARBlock is composed of two convolution layers Conv1 and Conv2, a maximum pooling layer MaxPool and a channel attention module CABlock, which is used to help the network further extract channels containing useful information; finally, a fully connected FC layer is used to complete the emotion classification of micro expressions;

[0027] Step 4.2, the soft thresholding formula is represented as:

[0028]

[0029] Where x represents the input and y represents the output.

[0030] Step 4.3, a residual block with a fusion channel attention mechanism is added after the two residual contraction modules to help the network extract channels containing useful information, and the channel attention module CABlock is represented as:

[0031] CABlock(x)=Sigmoid(MLP(AvgPool(x))+MLP(MaxPool(x))) (6)

[0032] Where the average pooling operation AvgPool considers each pixel as equally important and preserves higher-level information; the maximum pooling operation MaxPool is used to highlight the main features while eliminating redundancy; MLP represents a multilayer perceptron with one hidden layer; the Sigmoid function is used to normalize the output to between 0 and 1.

[0033] Step 4.4, a fully connected FC layer is used for emotion classification of micro expressions, and the performance evaluation index used is UFl and UAR, and UFl is represented as:

[0034]

[0035] wherein,

[0036]

[0037]

[0038] UAR represents:

[0039]

[0040] In the above formula (7) to formula (10), C is the number of divided emotion categories, i represents the i-th video frame sequence, j represents the j-th experiment, P i represents the accuracy of the i-th video frame sequence, R i represents the recall rate of the i-th video frame sequence, represents that the emotion of the i-th video frame sequence in the j-th experiment is actually "Positive" and is also predicted as "Positive", represents that the emotion of the i-th video frame sequence in the j-th experiment is actually "Positive" and is predicted as "Negative", represents that the emotion of the i-th video frame sequence in the j-th experiment is actually "Negative" and is predicted as "Positive";

[0041] The values of UAR and UFI represent the final classification performance, when UAR and UFI increase at the same time, it indicates that the performance of the whole micro-expression recognition is improved, when UAR and UFI are unchanged, it indicates that the performance of the whole micro-expression recognition does not change, when UAR or UFI decreases, it indicates that the performance of the whole micro-expression recognition is reduced.

[0042] The beneficial effects of the present application are that the micro-expression recognition method based on optical flow features proposes a new neural network RSCANet for micro-expression recognition research based on optical flow, which is composed of two parts: the first half uses two RSBlocks to enhance the anti-noise ability of the network; the second half uses ARBlock to extract important and useful information in the micro-expression video frame sequence, and the two are fused to improve the recognition performance. The leave-one-subject-out cross-validation strategy is adopted, and the RSCANet network model is tested in CASME, CASMEII and CAS(ME) 2 Three combined databases, which proves that the proposed method has certain advantages in recognition accuracy and model parameters compared with other several mainstream methods based on deep learning. At the same time, a series of ablation experiments are used to further ensure the effectiveness of the method. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is the algorithm framework flowchart used in the micro-expression recognition method based on optical flow features of the present application. DETAILED DESCRIPTION

[0044] The present application will be described in detail below in conjunction with the drawings and specific embodiments.

[0045] The micro-expression recognition method based on optical flow features of the present application, the flowchart is shown as Figure 1 The specific implementation is as follows:

[0046] Step 1, select micro-expression data sets: CASME, CASME II and CAS(ME) 2 , and map the emotions of all video frame sequences in the three data sets to the "Negative", "Positive" and "Surprise" three categories respectively;

[0047] Step 1 is implemented according to the following steps:

[0048] Select micro-expression data sets: CASME, CASME II and CAS(ME) 2 , and use video frame sequences containing rich micro-expressions in the three data sets for experiments. In order to alleviate the emotion category imbalance problem between the data sets used, the original emotions are mapped to the "Negative", "Positive" and "Surprise" three categories respectively. Specifically, in the CASME data set: "Disgust" (disgust), "Repression" (repression) and "Sadness" (sadness) are mapped to "Negative" (negative), "Happiness" (happiness) is mapped to "Positive" (positive), and "Surprise" (surprise) remains unchanged; in the CASME II data set: "Disgust" (disgust) and "Repression" (repression) are mapped to "Negative" (negative), "Happiness" (happiness) is mapped to "Positive" (positive), and "Surprise" (surprise) remains unchanged; in the CAS(ME) 2 data set, the emotion categories remain the original "Negative" (negative), "Positive" (positive) and "Surprise" (surprise) unchanged.

[0049] The data set division is shown in Table 1:

[0050] Table 1 Data set division

[0051] Emotion categories CASME CASME II CAS(ME) 2 ]]> Total number of categories Negative 83 88 21 192 Positive 7 32 6 45 Surprise 18 25 8 51 Total number of categories 108 145 35 288

[0052] Step 2, pre-process all video frame sequences of the selected data set to obtain video frame sequences with a resolution of 128x128;

[0053] Step 2 is as follows:

[0054] In order to minimize the influence of non-face regions in the video frame sequence on micro-expression recognition, a 68-point key point information detection model provided by the dlib library is used to realize face alignment and cropping operation, and the resolution of all video frame sequences is uniformly adjusted to 128x128. In addition, considering that the change between adjacent frame sequences of micro-expression is very small, in order to reduce redundancy, the starting frame and the peak frame with the largest micro-expression change of each video are selected as the input of step 3.

[0055] Step 3, TV-L1 energy functional is used to extract the optical flow features (optical flow horizontal component, optical flow vertical component and optical strain) of the video frame sequences obtained in step 2, and the optical flow features are spliced by channel superposition as the input of neural network RSCANet;

[0056] Step 3 is as follows:

[0057] Since the estimation of optical flow to the weak motion of the face region largely determines the accuracy of micro-expression recognition, TV-L1 energy functional with better noise robustness is used to extract the optical flow features of the single-channel video frame sequence obtained in step 2. Let u and v represent the optical flow horizontal component and the optical flow vertical component respectively:

[0058]

[0059] Where dx and dy represent the pixel changes along x and y dimensions respectively, and dt represents the time change;

[0060] Optical strain can approximate the intensity of facial deformation. The optical strain ε is obtained by calculating the derivative of the optical flow, and is defined as:

[0061]

[0062] Where u=[u,v] T is the displacement vector, so ε is also expressed as:

[0063]

[0064] Where the diagonal strain components (ε xx ,ε yy ) are normal strain components, (ε xy ,ε yx) is a shear strain component; then, the optical strain value of each pixel is calculated by taking the sum of squares of the normal strain component and the shear strain component, so as to:

[0065]

[0066] Finally, the three optical flow features of u, v and |ε| are spliced together through channel stacking as the input of the subsequent neural network RSCANet.

[0067] Step 4, using the neural network RSCANet to extract micro-expression features and obtain classification results.

[0068] Step 4 is specifically as follows:

[0069] Step 4.1, constructing the RSCANet network, the specific structure of the RSCANet network is: the first layer is a two-dimensional convolution Conv1, which is used to extract the shallow features of the input obtained in step 3; two residual contraction modules RSBlock1 and RSBlock2 are connected after Conv1 to enhance the noise resistance of the neural network; a residual block ARBlock with a fusion channel attention mechanism is added after the two residual contraction modules, the residual block ARBlock is composed of two convolution layers Conv1 and Conv2, a maximum pooling layer MaxPool and a channel attention module CABlock, which is used to help the network further extract channels containing useful information; finally, the emotion classification of micro-expression is completed by using a fully connected FC layer;

[0070] Step 4.2, in the micro-expression video frame sequence currently used, there will inevitably be some noise. Based on this, the residual contraction module is used to enhance the noise resistance of the network. The core of the residual contraction module is to add a soft threshold operation. The soft threshold function is used as a nonlinear layer, and the features whose absolute values are less than a certain threshold are directly set to 0, and the other features are shrunk to a certain extent in the direction of 0, so as to enhance the ability of the neural network to extract useful features by denoising. The soft threshold formula is expressed as:

[0071]

[0072] Wherein, x represents the input, and y represents the output;

[0073] Step 4.3, a residual block with a fusion channel attention mechanism is added after the two residual contraction modules to help the network extract channels containing useful information. The channel attention mechanism mainly focuses on what kind of features are meaningful, which can make the neural network focus on some channels with large weight values, so the residual block with a fusion channel attention mechanism is used to obtain better results. The channel attention module CABlock is expressed as:

[0074] CABlock(x) = Sigmoid(MLP(AvgPool(x)) + MLP(MaxPool(x))) (6)

[0075] wherein the average pooling operation AvgPool treats each pixel as equally important and preserves higher-level information; the max pooling operation MaxPool is used to highlight the main features while eliminating redundancy; MLP represents a multi-layer perceptron with one hidden layer; the Sigmoid function is used to normalize the output to between 0 and 1;

[0076] Step 4.4, the micro-expression emotion classification is performed by using a fully connected FC layer, and the performance evaluation indexes used are UFl and UAR, and UFl is expressed as:

[0077]

[0078] wherein,

[0079]

[0080]

[0081] UAR is expressed as:

[0082]

[0083] In the above formula (7) to formula (10), C is the number of divided emotion categories (C = 3), i represents the ith video frame sequence, j represents the jth experiment, P i represents the accuracy of the ith video frame sequence, R i represents the recall rate of the ith video frame sequence, represents that the emotion of the ith video frame sequence in the jth experiment is actually "Positive" and is also predicted as "Positive", represents that the emotion of the ith video frame sequence in the jth experiment is actually "Positive" and is predicted as "Negative", represents that the emotion of the ith video frame sequence in the jth experiment is actually "Negative" and is predicted as "Positive";

[0084] The values of UAR and UFl represent the final classification performance, when UAR and UFl increase at the same time, it indicates that the performance of the entire micro-expression recognition is improved; when UAR and UFl do not change, it indicates that the performance of the entire micro-expression recognition does not change; when UAR or UFl decreases, it indicates that the performance of the entire micro-expression recognition is reduced.

[0085] Experimental results comparison and verification: As can be seen from Table 2, compared with mainstream methods, the proposed RSCANet model has the highest UAR (0.7420) and UF1 (0.7218) performance on CASME, CASME II and CAS(ME) 2 The UAR (0.7420) and UF1 (0.7218) obtained on the combined database are the highest. Compared with the Micro-Attention model with the optimal UAR and UF1 in several other comparison methods, they are improved by 3.82% and 3.83% respectively. Moreover, the parameter quantity (params) of the model in this paper is also the least (71019) among several methods. Therefore, the method solves the key problem of difficulty in extracting subtle motion changes of human faces in video frames in the prior art of micro-expression recognition.

[0086] Table 2 Comparison of the method in this paper with mainstream methods

[0087]

[0088] Ablation experiment results verification: As shown in Table 3, in order to verify the effectiveness of the proposed RSCANet network, three groups of ablation experiments are designed using the leave-one-subject-out cross-validation method:

[0089] Experiment 1: The experiment is performed using the basic residual block (ResBlock) instead of RSBlock. It is found by comparison that using RSBlock can enhance the ability of the deep neural network to extract useful features from the input. The recognition result is obviously improved: UAR is increased by 6.44%, and UF1 is increased by 7.25%.

[0090] Experiment 2: Remove the CABlock in the third residual block and only use three RSBlocks for the experiment to prove that adding CABlock can indeed extract important and useful information in the input frame sequence, thereby improving the recognition performance: UAR is increased by 1.53%, and UF1 is increased by 2.23%.

[0091] Experiment 3: Remove the optical strain and only use the horizontal component of the optical flow and the vertical component of the optical flow for the experiment. The results show that adding optical strain can improve the recognition effect to a certain extent: UAR is increased by 4.76%, and UF1 is increased by 3.85%. The reason is that the optical strain can approximate the face deformation strength, provide more critical input features for the network, and the recognition effect is more accurate.

[0092] Table 3 Ablation experiment results

[0093]

Claims

1. A micro-expression recognition method based on optical flow features, characterized in that, Specifically, the following steps are implemented: Step 1, Selecting micro-expression datasets: CASME, CASME II and CAS(ME) 2 and map all video frame sequences in the three datasets into "Negative", "Positive" and "Surprise" categories, respectively; Step 2, pre-process all video frame sequences of the selected data set to obtain video frame sequences with a resolution of 128x128; Step 3, extract the optical flow features of the video frame sequences obtained in step 2 using the TV-L1 energy functional, and concatenate the optical flow features through channel stacking as the input of the neural network RSCANet; Step 4, extract micro-expression features using the neural network RSCANet and obtain the classification results; Step 4 is specifically as follows: Step 4.1, construct the RSCANet network, and the specific structure of the RSCANet network is as follows: the first layer is a two-dimensional convolution Conv1, which is used to extract the shallow features of the input obtained in step 3; Two residual shrinkage modules RSBlock1 and RSBlock2 are connected after Conv1 to enhance the noise resistance of the neural network; A residual block ARBlock that integrates channel attention mechanisms is added after the two residual shrinkage modules, which is composed of two convolution layers Conv1 and Conv2, a maximum pooling layer MaxPool, and a channel attention module CABlock, to help the network further extract channels containing useful information; finally, a fully connected FC layer is used to complete the emotion classification of micro-expression; Step 4.2, the soft thresholding formula is expressed as: Where x represents the input, and y represents the output; Step 4.3, a residual block that integrates channel attention mechanisms is added after the two residual shrinkage modules to help the network extract channels containing useful information, and the channel attention module CABlock is expressed as: Where the average pooling operation AvgPool considers each pixel as equally important and preserves higher-level information; the maximum pooling operation MaxPool is used to highlight the main features while eliminating redundancy; MLP represents a multilayer perceptron with one hidden layer; the Sigmoid function is used to normalize the output to between 0 and 1; Step 4.4, use a fully connected FC layer for emotion classification of micro-expression, and the performance evaluation indicators used are UF1 and UAR, UF1 is expressed as: Where, UAR is expressed as: In the above equations (7) to (10), C is the number of divided emotion categories, i represents the i-th video frame sequence, j represents the j-th experiment, P i represents the precision of the i-th video frame sequence, R i represents the recall of the i-th video frame sequence, represents that the emotion of the i-th video frame sequence in the j-th experiment is actually "Positive" and is also predicted as "Positive", represents that the emotion of the i-th video frame sequence in the j-th experiment is actually "Positive" and is predicted as "Negative", represents that the emotion of the i-th video frame sequence in the j-th experiment is actually "Negative" and is predicted as "Positive"; The values of UAR and UF1 represent the final classification performance, when UAR and UF1 increase at the same time, it means that the performance of the entire micro-expression recognition has improved; when UAR and UF1 remain unchanged, it means that the performance of the entire micro-expression recognition has not changed; when UAR or UF1 decreases, it means that the performance of the entire micro-expression recognition has decreased.

2. The micro-expression recognition method based on optical flow features according to claim 1, characterized in that, Step 1 is implemented according to the following steps: In CASME dataset: "Disgust", "Repression" and "Sadness" are mapped to "Negative", "Happiness" is mapped to "Positive", and "Surprise" remains unchanged; in CASME II dataset: "Disgust" and "Repression" are mapped to "Negative", "Happiness" is mapped to "Positive", and "Surprise" remains unchanged; in CAS(ME) 2 The emotion categories in the dataset remain unchanged as the original "Negative", "Positive" and "Surprise".

3. The micro-expression recognition method based on optical flow features according to claim 2, characterized in that, Step 2 is specifically as follows: Use the model provided by the dlib library to detect 68 key points to realize face alignment and cropping, and adjust the resolution of all video frame sequences to 128x128, select the starting frame and the peak frame with the largest micro-expression change of each video as the input of step 3.

4. The micro-expression recognition method based on optical flow features according to claim 3, characterized in that, Step 3 is specifically as follows: Let u and v represent the horizontal and vertical components of the optical flow respectively: Where dx and dy represent the pixel changes along the x and y dimensions, and dt represents the time change; Optical strain is obtained by calculating the derivative of the optical flow and is defined as: where u = [u, v] T is the displacement vector, so is also expressed as: where the diagonal strain components (ε xx , ε yy ) are normal strain components, (ε xy , ε yx ) are shear strain components; then, the optical strain value of each pixel is calculated by taking the square sum of the normal strain components and the shear strain components, so that: Finally, u, v, These three optical flow features are spliced together as the input of the subsequent neural network RSCANet through channel stacking.

Images