Micro-expression sequence recognition method based on double-flow vision transformer

Abstract

The application discloses a micro-expression sequence recognition method based on a double-flow VisionTransformer, first acquires a plurality of micro-expression images in continuous time, pre-processes micro-expression sequence images, and obtains pre-processed micro-expression sequence images and optical flow images; through a TS-ViT module, fusion features of each optical flow image and the corresponding micro-expression image are obtained; then through a time attention mechanism module, the spatio-temporal features of continuous frames are learned, and then feature fusion is performed to obtain a new feature vector sequence; the new feature vector sequence is input into a Bi-LSTM to learn the forward and backward dependency relationship in the time dimension of the micro-expression sequence, and finally, a micro-expression recognition result is output. The application solves the problem of network training underfitting caused by the small amount of micro-expression dataset.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision, and particularly relates to a micro-expression sequence recognition method based on a double-flow Vision Transformer. BACKGROUND

[0002] Micro-expression is an involuntary facial movement that reveals hidden feelings in high-risk situations, and has practical significance in medical treatment, national security, interrogation and many human-computer interaction systems. Early micro-expression recognition methods are mainly based on traditional appearance and geometric features. Unlike macro-expression, micro-expression has small motion amplitude and very short duration, only 0.04s-0.2s. Since micro-expression usually occurs in an unconscious state and is difficult to fake, micro-expression is very reliable in the field of emotion analysis; on the other hand, it is difficult to capture the changes of micro-expression by naked eyes, so the computer is needed to recognize micro-expression.

[0003] In recent years, with the development of deep learning and its successful application in macro-expression recognition, researchers have begun to use micro-expression recognition. Although micro-expression recognition is challenging due to limited micro-expression samples and low intensity, great progress has been made in micro-expression recognition through the design of effective shallow networks, the exploration of generative adversarial networks (GAN) and the like. At present, deep learning-based micro-expression recognition has received more and more attention and has achieved the most advanced performance.

[0004] Micro-expression recognition based on deep learning methods needs a huge dataset as a foundation, but the size of datasets such as SAMM, CASME, CASME II is too small, and some mainstream micro-expression discrimination methods only use the starting frame and the vertex frame of micro-expression, ignoring the time sequence features of micro-expression sequence, which is not sufficient to utilize the dataset and has low precision. SUMMARY

[0005] The application aims at the deficiencies of the prior art, and provides a micro-expression sequence recognition method based on a double-flow Vision Transformer.

[0006] Single-stream network is a basic model structure, and the single-stream network for micro-expression recognition only extracts features from a single image of the micro-expression sequence. However, there are slight motion changes between frames of the micro-expression sequence and limited samples of the dataset, and a single image cannot provide enough information. Therefore, dual-stream networks that learn features are very popular in expression recognition. In addition, micro-expressions are spontaneous and have different durations. Therefore, RNN can be considered for modeling data sequences, and VisionTransformer and cross-attention mechanisms can be used to extract and fuse features. However, due to the problem of gradient vanishing and explosion, it is difficult to train an RNN. Therefore, an improved version of RNN, long short-term memory (LSTM), can be used to solve the problem of vanishing gradients. LSTM is very suitable for processing time series with unknown durations. Bi-LSTM is a combination of forward LSTM and backward LSTM, which can learn the context relationship on the micro-expression time sequence.

[0007] The technical scheme adopted by the present application to solve its technical problems comprises the following steps:

[0008] Step 1, obtaining multiple micro-expression images in continuous time;

[0009] Step 2, preprocessing the micro-expression sequence images to obtain preprocessed micro-expression sequence images and optical flow images;

[0010] Step 3, sending each optical flow image and the corresponding micro-expression image obtained in step 2 into a TS-ViT module to obtain the fusion features of each optical flow image and the corresponding micro-expression image; the TS-ViT module comprises a Vision Transformer and a cross-attention mechanism module;

[0011] Step 4, learning the spatio-temporal features of the continuous frames through a temporal attention mechanism module, and then performing feature fusion to obtain a new feature vector sequence;

[0012] Step 5, inputting the new feature vector sequence into a Bi-LSTM to learn the forward and backward dependencies in the time dimension of the micro-expression sequence;

[0013] Step 6, mapping the feature vector space through linear transformation to the sample space through a fully connected layer, then mapping the output to the interval (0, 1) using the Softmax function, obtaining a vector composed of probability values of the class to which each group of micro-expression sequences belongs, and finally obtaining the micro-expression recognition result according to the probability vector value;

[0014] Further, the specific method of step 2 is as follows:

[0015] The face detection algorithm is used for face detection on the micro-expression sequence image, the face is found in the image and face cropping and alignment are performed. The aligned micro-expression sequence picture of each sample is respectively obtained N optical flow maps by TV-L1 optical flow method from the starting frame of the micro-expression sequence of the sample.

[0016] The extracted optical flow map and the corresponding image are scaled to generate a picture with a size of 224*224*3. Therefore, the optical flow map and the original image corresponding to each micro-expression sequence can be represented as X element of R 2*N*C*H*W , wherein N represents the number of pictures, C represents the number of channels, H and W represent the length and width respectively.

[0017] The present application has the following beneficial effects:

[0018] 1. The TS-ViT module is designed, the double-flow network structure is fully utilized for the micro-expression data set, the cross information of the micro-expression optical flow picture and the original picture is learned through the cross attention mechanism, and then the results of the double-flow network are averaged to obtain the final cross feature.

[0019] 2. In view of the problem of small amount of data of the micro-expression data set, the TS-ViT module of the present method is pre-trained first, so as to have the ability of recognizing basic features of the picture, then the pre-trained weight is applied to the micro-expression data set for fine tuning, and the problem of underfitting of the network caused by the small amount of data of the micro-expression data set is solved. DETAILED DESCRIPTION

[0020] Figure 1 It is a schematic diagram of the overall model structure of the embodiment of the present application;

[0021] Figure 2 It is a schematic diagram of the structure of the TS-ViT model of the embodiment of the present application;

[0022] Figure 3 It is a schematic diagram of the structure of the cross attention mechanism module of the embodiment of the present application;

[0023] Figure 4 It is a schematic diagram of the structure of the time attention mechanism module of the embodiment of the present application. DETAILED DESCRIPTION

[0024] The present application will be further described below in combination with the drawings.

[0025] As Figure 1 shown, a micro-expression sequence recognition method based on a double-flow Vision Transformer comprises the following steps:

[0026] Step 1: Select the mixed data set MEGC2019 as the training data set, which contains 109 positive samples, 250 negative samples and 83 surprise samples. Each sample contains a number of micro-expression sequence pictures.

[0027] Step 2: Preprocess the micro-expression sequence images to obtain preprocessed micro-expression sequence images and optical flow images.

[0028] Face detection algorithm (Dlib) is used for face detection on training samples, and face detection, face cropping and alignment are performed in the image. The aligned micro-expression sequence pictures of each sample are respectively obtained by TV-L1 optical flow method from the starting frame of the micro-expression sequence of the sample.

[0029] Optical flow is an approximation of local image motion, which has been proven to be helpful for motion representation. Optical flow-based face motion analysis provides a promising micro-expression recognition analysis method, including subtle movements, various head poses and face occlusions. Optical flow specifies the magnitude and direction of image pixel movement in a given image sequence with a two-dimensional vector field (horizontal and vertical optical flow). The TV-L1 optical flow method can maintain discontinuities in the flow field, thereby improving the robustness to noise, occlusion and illumination changes. The extracted optical flow map and corresponding image are scaled to produce a picture with a size of 224*224*3. Therefore, each set of micro-expression sequence corresponding to the optical flow map and the original image can be represented as X∈R 2*N*C*H*W , where N represents the number of pictures, C represents the number of channels, H and W represent the length and width respectively.

[0030] Step 3: Send each optical flow image and corresponding micro-expression image obtained in step 2 into the TS-ViT module to obtain the fusion features of each optical flow image and corresponding micro-expression image.

[0031] The TS-ViT module is used to extract fusion features Z∈R N*200 from X in step 2. The TS-ViT module introduces a cross-attention mechanism module on the basis of the Vision Transformer module. TS-ViT contains 2 basic Vision Transformer modules, corresponding to branch 1 (optical flow image branch) and branch 2 (corresponding micro-expression image branch). Each basic Vision Transformer module has 12 encoders, a hidden layer size of 768, a multi-layer perceptron (MLP) size of 3072, and a number of attention heads of 12. Figure 2The cross-attention mechanism module is introduced behind the shown Transformer Encoder layer, and the input of the cross-attention mechanism module is the output of the Transformer Encoder layer on the two branches.

[0032] First, each image is divided into 16x16 patches, and each patch is 14x14 pixels. As shown in Figure 2 , branch 1 represents the optical flow image branch, and branch 2 represents the corresponding micro-expression image branch. In order to more effectively fuse the features of the two branches, a cross-attention mechanism is introduced, as shown in Figure 3 , first use the class token of each branch as a proxy to exchange information with the patch tokens of the other branch, and then project back to the respective branch. Since the class token on each branch has learned abstract information in all patch tokens in its own branch, interacting with patch tokens on the other branch helps to contain information of different scales.

[0033] As shown in Figure 3 , for branch 1, first collect patch tokens from branch 2 and connect with the class token of branch 1 itself, and the calculation method is shown in formula (1):

[0034]

[0035] Where f 1 (*) is a size-aligned projection function, represents the class token of branch 1, represents the patch tokens of branch 2, and then and are combined to obtain x′ 1 . Finally, the cross-attention mechanism is performed between and x ′1 , where the class token is the only query because the information of the patch tokens is fused into the class token. Mathematically, the cross-attention mechanism (CA) is represented as formula (2):

[0036]

[0037]

[0038] Where W q ,W k ,W v ∈R D*(D / h)For learnable parameters, D and h are embedding dimension and head number. Since class token is only used in query, the computation and memory complexity of generating map(A) in cross attention mechanism is linear. Multiple heads are used in cross attention mechanism, which is denoted as MCA. The output z of final cross attention mechanism is calculated as formula (3): 1

[0039]

[0040]

[0041]

[0042] where f 1 (*) and g 1 (*) are dimension-aligned projection function and inverse projection function respectively, LN is layer normalization, z 1 contains and

[0043] The same process is performed on branch 2 to obtain z 2 , and finally the class token of z 1 and z 2 is extracted, and the weighted average is performed through the MLP head to finally obtain the feature Z∈R N*200 .

[0044] Step 4: Then learn the spatio-temporal features of continuous frames through the time attention mechanism module, and then perform feature fusion to obtain a new feature vector sequence, the specific method is as follows:

[0045] Calculate the corresponding maximum pooling F MAX and average pooling F AVG of the feature Z obtained in step 3, and then calculate the time attention weight using formula (4):

[0046] F Attention =σ(MLP(F MAX )+MLP(F AVG )) (4)

[0047] In formula (4), F Attention ∈R N*1 represents the time attention weight, and the structure diagram of the time attention mechanism module of the application is shown in Figure 4 . Sigma represents the activation function Sigmoid, and MLP is a multi-layer perceptron with an input dimension of 200 and an output dimension of 200. Then F Attention is substituted into formula (5) to obtain the fusion feature value ​

[0048]

[0049] Equation (5) in

[0050] Step 5: input the new feature vector sequence into Bi-LSTM to learn the forward and backward dependencies in the time dimension of the micro-expression sequence;

[0051] Bi-LSTM is used to learn the forward and backward dependencies in the time dimension of the micro-expression sequence. In order to increase the robustness of the model and alleviate the occurrence of overfitting, the dropout value in the Bi-LSTM function is set to 0.5, and the feature obtained in step 4 is input into the Bi-LSTM network to obtain the feature

[0052] Step 6: again Map the feature vector space through a fully connected layer to the sample space through linear transformation, then use the Softmax function to map the output to the interval (0, 1), and obtain the probability value vector of each group of micro-expression sequence belonging to the class, and finally get the micro-expression recognition result according to the probability vector value.

[0053] Step 7, the training steps of the micro-expression sequence recognition method based on the dual-flow Vision Transformer include:

[0054] s7.1, in order to accelerate the convergence effect of the network, the TS-ViT module of the method is pre-trained using the ImageNet image dataset to obtain the pre-training weight.

[0055] s7.2, set the number of iterations to 100, the learning rate to 0.001, and the batch size to 4.

[0056] s7.3, set the loss function as the cross-entropy loss function, which is defined as shown in equation 6.

[0057]

[0058] Where y i represents the true label of sample i, p i represents the probability value predicted by sample i, S represents the number of samples, and LOSS represents the difference between the true sample label and the predicted probability.

[0059] s7.4, use the preprocessed data of the mixed dataset MEGC2019 to train the parameters of the entire network, where the TS-ViT module uses the pre-training weight and does not participate in the training process.

[0060] The above description is further detailed in connection with specific / preferred embodiments of the present application, and should not be construed as limiting the specific implementation of the present application to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of substitutions or modifications can be made to the described embodiments, and these substitutions or modifications should be considered to fall within the protection scope of the present application.

[0061] The part of the present application not described in detail belongs to the technology known to those skilled in the art.

Claims

1. A method for micro-expression sequence recognition based on a dual-stream Vision Transformer, characterized in that, The method comprises the following steps: Step 1, obtaining a plurality of micro-expression images in continuous time; Step 2, pre-processing the micro-expression sequence images to obtain pre-processed micro-expression sequence images and optical flow images; Step 3, sending each optical flow image and the corresponding micro-expression image obtained in step 2 into a TS-ViT module to obtain the fusion features of each optical flow image and the corresponding micro-expression image; The TS-ViT module comprises a Vision Transformer and a cross-attention mechanism module; Step 4, learning the space-time features of the continuous frames through a time attention mechanism module, and then performing feature fusion to obtain a new feature vector sequence; Step 5, inputting the new feature vector sequence into a Bi-LSTM to learn the forward and backward dependencies in the time dimension of the micro-expression sequence; Step 6, mapping the feature vector space to the sample space through linear transformation by a fully connected layer, and then mapping the output to the interval (0, 1) by a Softmax function to obtain a vector composed of the probability values of the class to which each group of micro-expression sequences belongs, and finally obtaining the micro-expression recognition result according to the probability vector value.

2. The micro-expression sequence recognition method based on the dual-stream Vision Transformer according to claim 1, characterized in that, The specific method of step 2 is as follows: The micro-expression sequence images are detected by a face detection algorithm, and the face is found and cropped and aligned in the image; N optical flow images are obtained by TV-L1 optical flow method for each sample aligned micro-expression sequence picture and the starting frame of the micro-expression sequence of the sample; The extracted optical flow map and the corresponding image are scaled to produce a picture with a size of 224*224*3; therefore, each group of micro-expression sequence corresponds to an optical flow map and an original image represented as where N represents the number of pictures, C represents the number of channels, H and W represent the length and width, respectively.

3. The micro-expression sequence recognition method based on dual-stream Vision Transformer according to claim 2, characterized in that, The specific method of step 3 is as follows: extracting fusion features from X of step 2 using TS-ViT module ; the TS-ViT module introduces a cross-attention mechanism module on the basis of the Vision Transformer module, and the TS-ViT contains two basic Vision Transformer modules, corresponding to branch 1, i.e. the optical flow image branch, and branch 2, i.e. the micro-expression image branch. Each basic Vision Transformer module has 12 encoders, a hidden layer size of 768, a multi-layer perceptron (MLP) size of 3072, and 12 heads of multi-head attention. A cross-attention mechanism module is introduced after the Transformer Encoder layer, and the input of the cross-attention mechanism module is the output of the Transformer Encoder layer on the two branches. First, each image is divided into 16x16 patches, so that each patch is 14x14 pixels; In order to more effectively fuse the features of the two branches, a cross-attention mechanism is introduced, first using the class token of each branch as a proxy to exchange information with the patch tokens of the other branch, and then projecting back to the respective branches; Since the class token on each branch has learned abstract information in all patch tokens in its own branch, interacting with the patch tokens on the other branch helps to contain information of different scales; For branch l, first collect the patch tokens from branch 2 and connect them with the class token of branch l itself, and the calculation method is shown in formula (1): ， （1） wherein, is a size-aligned projection function, represents class tokens of branch 1, represents patch tokens of branch 2, and then and are combined to obtain ; finally, a cross-attention mechanism is performed between and , wherein the class token is the only query because the information of the patch tokens is fused into the class token; in mathematics, the cross-attention mechanism CA is represented as shown in equation (2): , , ， ， （2） wherein, , , are learnable parameters, D and h are embedding dimension and number of heads; since class token is only used in query, the computation and memory complexity of generating map(A) in cross-attention mechanism is linear; multiple heads are used in cross-attention mechanism and denoted as MCA; the output of final cross-attention mechanism is calculated as shown in equation (3): ， ， （3） wherein, and are a dimension-aligned projection function and an inverse projection function, respectively, is layer normalization, comprises and ; The same process is performed on branch 2 to obtain , and finally the class tokens of and are extracted, weighted averaged by the MLPHead, and finally the feature is obtained.

4. The micro-expression sequence recognition method based on dual-stream Vision Transformer according to claim 3, characterized in that, The specific method of step 4 is as follows: Compute the corresponding max-pooling of the feature Z obtained from step 3 and average-pooling Then compute the temporal attention weight using equation (4): （4） In formula (4), denotes the time attention weight, σ denotes the activation function Sigmoid, and MLP is a multi-layer perceptron with an input dimension of 200 and an output dimension of 200; then is substituted into formula (5) to obtain the fusion feature value : （5） In equation (5) .

5. The micro-expression sequence recognition method based on dual-stream Vision Transformer according to claim 4, characterized in that, The specific method of step 5 is as follows: Bi-LSTM is used to learn the forward and backward dependencies in the time dimension of micro-expression sequences. In order to increase the robustness of the model and alleviate the occurrence of overfitting, the dropout value in the Bi-LSTM function is set to 0.5, and the features obtained in step 4 are input into the Bi-LSTM network to obtain the features .​

Images