Micro-expression recognition method based on visual Transform

Abstract

The invention discloses a micro-expression recognition method based on a visual Transform, and the method comprises the steps: firstly carrying out the data preprocessing, and extracting a starting frame and a peak frame from a micro-expression segment composed of a plurality of frames; then, face regions in a start frame and a peak frame are extracted by using an MTCNN to remove noise and unnecessary background information; next, calculating according to two frames of face images obtained after face detection to obtain a single optical flow feature map; then the optical flow feature map is input into an advanced feature extractor composed of a double-layer normalized shift Patch marker, a Transform encoder and a feature selection module for fine-grained and discriminative feature extraction of micro-expressions; and finally, inputting the extracted features into a classification head, and outputting a classification result. According to the method, a visual Transform is used as a backbone network, multiple technologies capable of adapting to a micro-expression recognition task are designed on the basis, and micro-expression recognition with excellent performance is realized by using a neural network with an extremely small scale.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a micro-expression recognition method based on visual Transformer. Background Art

[0002] After a long evolution, humans have developed various ways to express emotions, including language, gestures, and expressions. Among them, conveying psychological feelings through facial expressions is a common and intuitive way, and is also considered a non-verbal medium for emotional communication. Facial expression refers to a form of facial activity in which several areas of human facial muscles are activated to express emotions. According to the scale and scale of the muscle movements involved, human facial expressions can be divided into macro-expressions and micro-expressions. Macro-expressions are spontaneous, usually lasting 1 / 2 to 4 seconds, and are easy to detect and perceive. Micro-expressions were first discovered by Haggard and Isaacs in 1966. Three years later, Ekman and Friesen also announced the discovery of micro-expressions when examining lie detector videos of psychiatric patients. It will appear when people try to hide or conceal their true emotions, and can reflect people's most real emotional state at the current moment, such as happiness, sadness, surprise, etc.

[0003] Micro-expression recognition belongs to micro-expression analysis, which is the task of classifying micro-expression images or clips into various emotion categories. However, compared with macro expressions, micro-expressions are very subtle and short-lived, usually lasting only 1 / 25 to 1 / 5 second (some say 1 / 25 to 1 / 3 second), and their expression on the face is extremely small, so it is very difficult to analyze and judge them only by the naked eye. Even for experts, the rapid and accurate recognition of micro-expressions is challenging. At the same time, emotional context and cultural background will also have an impact on micro-expression recognition.

[0004] Research on micro-expressions and their recognition is of great significance in the fields of mental health, public safety, business negotiations, human-computer interaction, etc. The work of micro-expression recognition can be traced back to 1966, and it has experienced nearly sixty years of development. Since the 1960s, with the advancement of machine learning and image processing technology, the use of computers for micro-expression recognition has become an important research direction. Early micro-expression recognition is usually based on image processing technology and feature engineering. The main steps include video acquisition, image preprocessing, feature extraction, feature selection and dimensionality reduction, classifier classification, and model training and evaluation. In 2011, Pfister et al. used a temporal interpolation model and the first comprehensive spontaneous micro-expression corpus to complete micro-expression recognition. In 2014, Wang et al. used a six-intersection local binary pattern volume descriptor for micro-expression recognition, providing a more compact and lightweight representation of micro-expressions. In 2015, Liu et al. proposed a simple and effective main direction average optical flow feature for micro-expression recognition. In 2017, Lu Guanming et al. proposed a feature selection algorithm based on the combination of Relief and local linear embedding manifold learning algorithm, which achieved the purpose of reducing the dimension of the extracted local binary pattern feature vector. These methods have laid the foundation for the use of machines for micro-expression recognition and provided inspiration for subsequent research work.

[0005] In recent years, with the advancement and popularization of deep learning methods, various methods based on deep learning have made breakthroughs in micro-expression recognition. These methods mainly include three data modalities: static images, dynamic image sequences, and their combinations. In 2019, Zhou et al. used Inception combined with optical flow features to conduct the micro-expression recognition challenge MEGC2019, and achieved significant performance improvements in the UAR and UF1 indicators. In the same year, Liong et al. designed a shallow three-stream 3D CNN that can extract discriminative high-level and detailed representations of micro-expressions. Xia et al. proposed a micro-expression recognition method based on a deep recursive convolutional network, which successfully captured the spatiotemporal variation information of micro-expression sequences. In 2020, Lai Zhenyi et al. proposed a CNN real-time micro-expression recognition algorithm combined with dilated convolution. In the same year, Chen et al. used spatial and channel attention modules to deal with the problem of micro-expression spatiotemporal feature extraction. These methods have greatly improved the effect of micro-expression recognition, proving the huge development opportunities of using deep learning technology to achieve micro-expression recognition, but there are also challenges and room for improvement in many aspects. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present invention provides a micro-expression recognition method based on visual Transformer. First, data preprocessing is performed to extract the start frame and peak frame from the micro-expression fragment composed of multiple frames; then the facial area in the start frame and peak frame is extracted using MTCNN to remove noise and unnecessary background information; next, a single optical flow feature map is calculated based on the two frames of facial images obtained after face detection; then the optical flow feature map is input into an advanced feature extractor composed of a double-layer normalized shift patch marker, a transformer encoder, and a feature selection module to extract fine-grained and discriminative features of micro-expressions; finally, the extracted features are input into a classification head to output the classification results. The present invention uses visual Transformer as the backbone network, and on this basis designs a number of technologies that can adapt to micro-expression recognition tasks, and uses a very small-scale neural network to achieve micro-expression recognition with superior performance.

[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0008] Step 1: Data preprocessing;

[0009] Extracting a start frame and a peak frame from a micro-expression segment consisting of multiple frames;

[0010] Step 2: Face detection;

[0011] Use MTCNN to extract the facial regions in the starting frame and the peak frame to remove noise and unnecessary background information;

[0012] Step 3: Optical flow feature calculation;

[0013] A single optical flow feature map is calculated based on the two frames of facial images obtained after face detection;

[0014] Step 4: Feature extraction;

[0015] The optical flow feature map is input into an advanced feature extractor consisting of a two-layer normalized shifted patch marker, a Transformer encoder, and a feature selection module to extract fine-grained and discriminative features of micro-expressions.

[0016] Step 5: Classification output;

[0017] The extracted features are input into the classification head and the classification results are output.

[0018] Furthermore, the MTCNN consists of three cascaded convolutional neural networks P-Net, R-Net and O-Net, each of which processes and filters the input image in turn to achieve face detection and alignment.

[0019] Furthermore, the step 3 is specifically as follows:

[0020] Step 3-1: After using MTCNN to detect the start frame and peak frame of the micro-expression segment, two facial images without background noise are obtained. The optical flow features generated by these two frames are calculated using the following formula:

[0021] V=(u(x,y),v(x,y))|x=1,2,…,X,y=1,2,…,Y. (5)

[0022] Where x and y represent the height and width of the frame, respectively, u(x, y) and v(x, y) are the horizontal and vertical components of the optical flow feature map V, V = [V x ,V y ],V∈R W×H×2 ;

[0023] Step 3-2: Use the first-order derivative of the optical flow field to calculate the change of the optical flow field, which is called optical strain;

[0024] The optical strain calculation formula is as follows:

[0025]

[0026] in, is the first-order partial derivative of V;

[0027] Step 3-3: Form a three-dimensional optical flow feature map and represent it as V m =[V x ,V y ,V z ],V m ∈R W×H×3 .

[0028] Furthermore, the step 4 is specifically as follows:

[0029] TransFG is introduced into micro-expression recognition to solve the problem of feature extraction;

[0030] Step 4-1: Design a double-layer normalized shift patch tokenization module;

[0031] Step 4-1-1: Shift the input micro-expression image by half a patch along the four diagonal directions of the upper left, upper right, lower left, and lower right; crop the shifted features to the same size as the input image and then concatenate them with the input; the concatenated features are divided into non-overlapping patches and flattened; obtain the visual token through a double-layer patch normalization module, which is expressed as:

[0032]

[0033] in, The i-th shifted image, represents learnable linear projection, LN(·) represents layer normalization; d S represents the hidden dimension of the Transformer encoder, N s represents the number of shifted images; P(.) represents the learnable linear projection;

[0034] Step 4-1-2: Concatenate the category token and the visual token, and then add the position embedding;

[0035] The category token is a token with representation information of the entire image, and the position embedding provides position information for the visual token; the formula for applying the module to the Patch embedding layer is as follows:

[0036]

[0037] in, Represents a category token, is a learnable position embedding, N is the number of embedding tokens;

[0038] Step 4-2: Local self-attention;

[0039] Step 4-2-1: The diagonal mask is defined as follows:

[0040]

[0041] in Represents each component of the mask similarity matrix;

[0042] Step 4-2-2: The local self-attention with diagonal mask and learnable temperature scaling is defined as:

[0043]

[0044] where τ is the learnable temperature scaling factor, E v Represents the linear projection weight of Value; R M (x) represents the diagonal mask attention weight.

[0045] The beneficial effects of the present invention are as follows:

[0046] The present invention uses visual Transformer as the backbone network to realize micro-expression recognition. On this basis, a variety of plug-and-play visual Transformer components specially optimized for micro-expression recognition tasks are designed, and micro-expression recognition with superior performance is achieved under the condition of using an extremely small-scale neural network (about 7M trainable parameters). BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is the overall workflow diagram of the micro-expression recognition algorithm MEFormer of the present invention;

[0048] Figure 2 It is a working flow diagram of the feature selection module in the advanced feature extractor of the present invention;

[0049] Figure 3 It is a workflow diagram of a double-layer normalized shifted patch marker in the advanced feature extractor of the present invention;

[0050] Figure 4 It is a working principle diagram of the local self-attention mechanism of the present invention. DETAILED DESCRIPTION

[0051] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0052] In view of the special characteristics of micro-expressions, such as brevity and subtlety, for micro-expression recognition using deep learning methods, deep neural networks are required to efficiently extract the discriminative features of different categories of micro-expressions, and the neural network needs to be cleverly designed to have a strong fine-grained feature capture capability. In addition, compared with convolutional neural networks, Transformer pays more attention to global features, and can achieve better classification results than convolutional neural networks through pre-training on many tasks, but at the same time, the cost is that it lacks local inductive bias compared to convolutional neural networks, and requires large-scale data support to obtain good micro-expression representation. However, due to the above reasons, spontaneous micro-expression data is difficult to collect and annotate, which leads to the generally small size of micro-expression data sets. On the premise of using existing data sets, how to make deep neural networks adapt to small-scale micro-expression data sets is also a problem that needs to be improved.

[0053] The purpose of this invention is to propose a new micro-expression recognition method based on visual Transformer to optimize the fine-grained feature extraction and small-scale data set adaptability problems existing in the above-mentioned micro-expression recognition field.

[0054] To achieve the above objectives, the present invention proposes a new micro-expression recognition method MEFormer. According to the work objectives, MEFormer can be divided into two main stages. In the first stage, MTCNN (Multi-task Cascaded Convolutional Networks, MTCNN) and the optical flow calculation module are used to obtain the optical flow feature map of the start frame and the peak frame in a micro-expression segment; in the second stage, the optical flow feature map is input into the advanced feature extractor to extract the fine-grained features of micro-expressions, and finally classified by the classification head. Specifically, the algorithm includes the following steps:

[0055] 1. Data preprocessing: Extract the start frame and peak frame from the micro-expression segment composed of multiple frames. During the training and testing process, the indexes of the peak frame and the start frame have been provided by the dataset. In the actual algorithm deployment application, due to the short-lived nature of micro-expressions, it is reasonable to approximate the start frame and peak frame with two adjacent frames in the video stream.

[0056] 2. Face detection: The facial regions in the starting frame and the peak frame are extracted using MTCNN to remove noise and unnecessary background information.

[0057] 3. Optical flow feature calculation: A single optical flow feature map is calculated based on the two frames of facial images obtained after face detection.

[0058] 4. Feature extraction: The optical flow feature map is input into an advanced feature extractor consisting of a two-layer normalized shifted patch marker, a Transformer encoder, and a feature selection module to extract fine-grained and discriminative features of micro-expressions.

[0059] 5. Classification output: Input the extracted features into the classification head and output the classification results.

[0060] Example:

[0061] The overall workflow of the present invention is as follows Figure 1 As shown. In MEFormer, the starting frame and peak frame of a micro-expression clip are first input into the face detection in MTCNN, and the two detected facial images are used to calculate their optical flow feature maps. Next, the optical flow feature maps are input into the advanced feature extractor with Vision Transformer as the backbone for fine-grained feature extraction. The advanced feature extractor consists of a two-layer normalized shift patch marker, a local self-attention Transformer encoder, and a feature selection module. During the training process, MEFormer introduces contrast loss to enhance the model's sensitivity to discriminative features. Finally, the features generated by the advanced feature extractor are sent to a perceptron consisting of a single fully connected layer to generate a prediction vector, and the prediction vector is classified by the Softmax classifier. The following is a detailed description of the algorithm's composition and implementation.

[0062] 1. Face detection;

[0063] To automatically identify micro-expressions, the first step is to detect the facial area of ​​a person from an image containing background information and noise. To this end, MTCNN is selected to implement the detection task. MTCNN is a model for face detection and alignment. It consists of three cascaded convolutional neural networks (P-Net, R-Net, and O-Net). Each network processes and filters the input image more accurately in turn, combining high accuracy and real-time performance. It can achieve efficient and accurate face detection and alignment, providing a reliable foundation for subsequent emotion recognition.

[0064] 2. Optical flow feature calculation;

[0065] Expressions come from facial muscle movements. Different forms of movement produce different types of expressions, such as happiness, sadness, surprise, etc. Therefore, it is a reasonable strategy to transform the classification of expressions into the classification of facial muscle movements, which requires extracting facial muscle movement features from micro-expression fragments. The present invention chooses to use optical flow features to realize the modeling of motion features. Optical flow features are a method for calculating pixel motion in an image sequence. They are used to analyze the motion of objects in an image over a period of time and have a wide range of applications in the fields of computer vision and image processing.

[0066] A micro-expression segment is generally composed of a start frame, a peak frame, an end frame, and a transition frame between the three. The peak frame is the frame with the strongest expression and contains the most typical micro-expression information. Specifically, after using MTCNN to detect the start frame and peak frame of the micro-expression segment, two facial images without background noise will be obtained. The start frame is plain, and the peak frame is the strongest expression. Calculating the optical flow features between them can maximize the typical features of a micro-expression, which is conducive to the model learning high-quality knowledge. Now we need to calculate the optical flow features generated by these two images. The calculation formula is as follows:

[0067] V=(u(x,y),v(x,y))|x=1,2,…,X,y=1,2,…,Y. (5)

[0068] Where x and y represent the height and width of the frame, respectively, u(x, y) and v(x, y) are the horizontal and vertical components of the optical flow feature map V, V = [V x ,V y ],V∈R W×H×2 .

[0069] Next, the first-order derivative of the optical flow field is used to calculate the change in the optical flow field, which is often referred to as optical strain. Optical strain provides an estimate of the degree of facial displacement, thereby providing a richer reference for the subtle movements that occur within a micro-expression cycle. Through this calculation of optical strain, complex facial movements can be captured and analyzed, which helps to accurately identify micro-expressions. The calculation formula is as follows:

[0070]

[0071] in, is the first-order partial derivative of V. Finally, a three-dimensional optical flow feature map is formed and represented as V m =[V x ,V y ,V z ],V m ∈R W×H×3 .

[0072] 3. Advanced feature extraction;

[0073] Compared with macro expressions, the optical flow feature map of micro expressions contains more fine-grained features, which require special strategies to extract these features. This paper introduces TransFG into micro expression recognition to solve the problem of feature extraction. TransFG is an improved Vision Transformer, which is used for fine-grained image recognition tasks and has achieved good results on benchmarks such as CUB-200-2011 and StanfordCars. It has a strong ability to capture fine-grained features.

[0074] Specifically, TransFG has two designs based on ViT:

[0075] Feature selection module: Figure 2 The working principle of the feature selection module is shown. Specifically, in order to make full use of the attention information, TransFG changes the input format of the last Transformer block in Vision Transformer and adds a feature selection module before the last Transformer block. Assuming that the number of self-attention heads is set to K, the intermediate features input to the last Transformer block can be expressed as And the attention weights of the L-1 Transformer blocks before the last Transformer block are expressed as:

[0076]

[0077] Due to the lack of embedded token identifiability, the original attention weights may not necessarily reflect the relative importance of the input tokens, especially for deeper layers in the model. To this end, the feature selection module integrates the attention weights of all layers before the last Transformer block. It recursively applies matrix multiplication to the original attention weights in all layers to obtain new attention weights:

[0078]

[0079] and the original attention weights a of a single Transformer block L-1 In comparison, a final It includes the propagation of micro-expression information from the input layer to the higher layer embedding, which is a better design for selecting the discriminant area. Then choose relative to a final The indices A1,A2,…,A of the maximum values ​​of the K different attention heads in K , from Z L-1 Finally, the selected token is concatenated with the classification token as the input feature of the last Transformer layer:

[0080]

[0081] By replacing the entire original input sequence with tokens corresponding to the informative regions and concatenating them with the classification tokens as the input to the last Transformer layer, the model not only retains the global information, but also forces the last Transformer layer to focus on the subtle differences between micro-expressions of different categories, discarding less discriminative areas such as the background or common features in the category.

[0082] Contrastive learning: Since the feature differences between micro-expression categories are small, simple cross entropy loss is not enough to fully supervise the learning of features. In addition to the cross entropy loss, a contrast loss L is added. con , the contrast loss is used to minimize the similarity of classification tokens corresponding to different labels and maximize the similarity of classification tokens of samples with the same category. In order to prevent the loss from being dominated by simple negative samples (different class samples with small similarity), a constant margin α is introduced in the contrast loss. Only negative sample pairs with similarity greater than α will contribute to the loss L. con Make a contribution. The contrast loss of a sample with a batch size of B is expressed as:

[0083]

[0084] Where Z i and Z jPreprocessed by L2 normalization, Sim(Z i ,Z j ) is Z i and Z j The total loss function is expressed as:

[0085]

[0086] in is the cross entropy loss function calculated using the predicted label y′ and the true label y.

[0087] In micro-expression recognition, in addition to the problem of feature extraction, due to the characteristics of micro-expressions themselves, building a large-scale micro-expression dataset is also a difficult task. Therefore, the current micro-expression datasets are generally small in size, which affects the performance of micro-expression recognition. So how to learn rich, high-quality knowledge from small-scale datasets has become a problem worth studying. In order to make MEFormer adapt to small-scale data for micro-expression recognition, inspired by Lee et al., the present invention has made a new design for TransFG, designed a double-layer normalized shift patch tokenization module for it, and improved the self-attention in Transformer. The following is a detailed introduction to these two designs:

[0088] Double-layer normalized shifted patch tokenization module: First, each input image is spatially shifted by half the block size in four diagonal directions (i.e., top left, top right, bottom left, and bottom right). Next, the shifted features are cropped to the same size as the input image and then concatenated with the input. Afterwards, the concatenated features are divided into non-overlapping patches and flattened. Next, visual tokens are obtained through a double-layer patch normalization module. The whole process is as follows: Figure 3 As shown, it can be expressed as:

[0089]

[0090] in, The i-th shifted image, represents a learnable linear projection, LN(·) represents layer normalization, and d S represents the hidden dimension of the Transformer encoder, N s Indicates the amount of image shifting.

[0091] Next, concatenate the category token with the visual token and then add the position embedding. The category token here is a token with representation information for the entire image, and the position embedding provides the position information for the visual token. If the class token is not used, only the position embedding is added to the output of the module. The formula for how to apply the module to the Patch embedding layer is as follows:

[0092]

[0093] in, Represents a category token, is a learnable positional embedding and N is the number of embedding tokens.

[0094] Local Self-Attention: Figure 4 The workflow of local self-attention is shown. Specifically, the diagonal mask assigns higher scores to inter-token relations by essentially excluding self-token relations from participating in the softmax operation. Specifically, the diagonal mask sets the diagonal components of the similarity matrix in the QKV calculation to -∞. This makes the self-attention more focused on other tokens instead of focusing on the self-token. The proposed diagonal mask is defined as follows:

[0095]

[0096] in Represents each component of the mask similarity matrix.

[0097] The second key point of local self-attention is learnable temperature scaling. The "temperature" of Softmax refers to a parameter in the Softmax function that controls the "smoothness" of the output probability distribution. In the Softmax function, for each element in the input vector, an exponential function is applied to calculate its exponential value, which is then divided by the sum of the exponential values ​​of all elements to get a probability distribution. This temperature parameter is usually expressed as a scalar value, and the "temperature" of the output is adjusted by dividing the output of the Softmax function by this parameter value. Higher temperatures result in smoother output distributions, while lower temperatures make the output distributions more spiky. In deep learning, by adjusting the temperature parameter of Softmax, the diversity and certainty of the model output can be controlled. In local self-attention, it allows MEFormer to determine the Softmax temperature itself during the learning process. Generally speaking, a lower Softmax temperature will make the attention score distribution sharper. Learnable temperature scaling makes the distribution of attention scores sharper by lowering the temperature, that is, the probability values ​​of some classes will be higher, while the probability values ​​of other classes will be closer to zero. This can enhance the certainty of the model, making the model more inclined to make exact predictions. For classification, this certainty can improve the accuracy of the model. The local self-attention with diagonal mask and learnable temperature scaling is defined as:

[0098]

[0099] where τ is a learnable temperature scaling factor.

[0100] To verify the effectiveness of MEFormer, this paper carried out micro-expression recognition experiments on four data sets. This paper introduces the experimental data sets, experimental environment, experimental methods, etc. in detail and analyzes the experimental results.

[0101] 1. Dataset;

[0102] The experiments of the present invention are based on CASME II, SAMM and SMIC datasets. At the same time, referring to previous work, in order to ensure consistency and comparability, CASME II, SAMM and SMIC are merged into a new composite dataset. The experiments of the present invention will be carried out on these four datasets. In order to unify the standards and achieve fair evaluation, according to the method proposed by Van et al., multiple emotion categories are divided into "positive", "negative" and "surprise". The specific emotion categories are divided as follows: "positive" emotion categories include "happy", "negative" emotion categories "sadness", "disgust", "contempt", "fear", "anger", and "surprise" emotion categories only include "surprise" categories. The basic information of the dataset is as follows:

[0103] The SAMM dataset contains 28 participants, 133 micro-expressions, and 147 long videos (including 343 macro-expressions). The dataset contains rich action unit encodings and provides comprehensive facial expression information. SAMM also provides the start, end, and vertex frame coordinates of micro-expressions. The resolution of the original samples in the dataset is 2040×1088 pixels, and the frame rate is set to 200 frames per second. In order to facilitate the experiment, the present invention cropped the face images in the original samples to obtain face images with a resolution of 28×28 pixels. The emotional categories in the images in SAMM are divided into "disgust", "fear", "contempt", "anger", "depression", "surprise", "happiness", and "other" categories. After being divided into three emotion categories, the numbers of "negative", "positive", and "surprise" are 92, 26, and 15, respectively.

[0104] The CASME II dataset consists of data from 24 subjects, with a total of 145 samples. All samples were captured using a laboratory camera with a frame rate set to 200 frames per second. The original size of the samples is 640×480 pixels. In the experiments of the present invention, the facial images were cropped and resized to a resolution of 28×28 pixels. The samples in CASME II are divided into the categories of "happy", "surprised", "disgusted", "sad", "fearful", "depressed", and "other". After merging into three emotion categories, the numbers of "negative", "positive", and "surprised" are 88, 32, and 25, respectively. The start, end, and vertex frames are annotated in CASME II.

[0105] The SMIC-HS dataset contains data from 16 subjects, with a total of 164 samples. All samples were captured using a laboratory camera with a frame rate of 100 frames per second. The original image size of the samples is 640×480 pixels. In order to focus on the facial region of interest and keep a consistent input size for the micro-expression recognition task, the facial images are cropped to a resolution of 28×28 pixels. The number of "negative", "positive", and "surprised" is 70, 51, and 43, respectively.

[0106] The Composite dataset was proposed by the 2nd Micro-Expression Grand Challenge (MEGC2019). The composite dataset merges samples from three spontaneous facial ME datasets: CASME II, SAMM, and SMIC-HS, in order to facilitate the evaluation of newly developed methods. Since the annotations in the three datasets are quite different, the composite dataset unifies the emotion labels in the three datasets. The emotion labels are re-annotated as "positive", "negative", and "surprised".

[0107] 2. Implementation details;

[0108] All experiments in this paper were performed using the PyTorch 2.0.0 framework on a Windows operating system, and training and evaluation were performed using a single Nvidia RTX 4090 GPU.

[0109] 3. Evaluation indicators;

[0110] In order to deal with the problem of uneven sample distribution in the micro-expression dataset, the present invention uses the unweighted average recall rate and the unweighted F1 score as the evaluation indicators of the micro-expression recognition experiment. The specific calculation process of the evaluation indicators is as follows:

[0111] Unweighted F1-score (UF1): is a metric commonly used to evaluate the performance of multi-class classification tasks with unbalanced class distribution. To calculate UF1, it is necessary to calculate the false positives (FP), true positives (TP), and false negatives (FN) for each class c in all folds of leave-one-subject-out (LOSO) cross-validation. Then, the F1 score for each class can be calculated using the following formula:

[0112]

[0113] Unweighted Average Recall (UAR): is a particularly useful metric for evaluating model effectiveness when the class ratio is unbalanced. To calculate UAR, first calculate the true positive examples (TP) for each emotion category c. The true positive examples represent the number of correctly classified samples in each category. In addition, the total number of samples in each category is calculated:

[0114]

[0115] Among them, TP c Indicates the number of true positive samples of a specific category, FP c Indicates the number of false positive samples of a specific category, FN c represents the number of false negative samples of a specific category, c represents the specific category, C represents the number of categories, and the present invention is 3.

[0116] 4. Comparative experiment;

[0117] To verify the performance of MEFormer, the present invention conducted comparative experiments and compared MEFormer with existing methods on SMIC, SAMM, CASMEII and Composite datasets.

[0118] As shown in Table 1, MEFormer generally achieved better results than existing methods on the four datasets. Compared with the most advanced HTNet, the UF1 and UAR scores on the composite dataset increased by up to 1.56% and 2.72%, the UF1 score on the SAMM dataset increased by up to 1.57%, the UF1 and UAR scores on the CASME II dataset increased by up to 2.76% and 2.75%, and the UF1 and UAR scores on the SMIC dataset increased by up to 1.18% and 3.1%, achieving good improvements. It is worth noting that HTNet performed better in the UAR indicator of the SAMM dataset. It is believed that this is due to the label distribution problem in the SAMM dataset. The relatively uneven label distribution leads to a loss in recall rate, which has become a problem worthy of future study. In general, based on the experimental results, it can be concluded that MEFormer effectively optimizes the adaptability of small-scale datasets and the problem of fine-grained feature extraction.

[0119] Table 1 Comparative experiments with existing methods in the field of micro-expression recognition based on deep learning

[0120]

[0121]

Claims

1. A micro-expression recognition method based on visual Transformer, characterized in that: The steps include: Step 1: Data preprocessing; Extracting a start frame and a peak frame from a micro-expression segment consisting of multiple frames; Step 2: Face detection; Use MTCNN to extract the facial regions in the starting frame and the peak frame to remove noise and unnecessary background information; Step 3: Optical flow feature calculation; A single optical flow feature map is calculated based on the two frames of facial images obtained after face detection; Step 4: Feature extraction; The optical flow feature map is input into an advanced feature extractor consisting of a two-layer normalized shifted patch marker, a Transformer encoder, and a feature selection module to extract fine-grained and discriminative features of micro-expressions. Step 5: Classification output; The extracted features are input into the classification head and the classification results are output.

2. The micro-expression recognition method based on visual Transformer according to claim 1, characterized in that: The MTCNN consists of three cascaded convolutional neural networks P-Net, R-Net and O-Net. Each network processes and filters the input image in turn to achieve face detection and alignment.

3. The micro-expression recognition method based on visual Transformer according to claim 2, characterized in that: The step 3 is specifically as follows: Step 3-1: After using MTCNN to detect the start frame and peak frame of the micro-expression segment, two facial images without background noise are obtained. The optical flow features generated by these two frames are calculated using the following formula: V=(u(x,y),v(x,y))|x=1,2,…,X,y=1,2,…,Y. (5) Where x and y represent the height and width of the frame, respectively, u(x, y) and v(x, y) are the horizontal and vertical components of the optical flow feature map V, V = [V x ,V y ],V∈R W×H×2 ; Step 3-2: Use the first-order derivative of the optical flow field to calculate the change of the optical flow field, which is called optical strain; The optical strain calculation formula is as follows: in, is the first-order partial derivative of V; Step 3-3: Form a three-dimensional optical flow feature map and represent it as V m =[V x ,V y ,V z ],V m ∈R W×H×3 .

4. The micro-expression recognition method based on visual Transformer according to claim 3, characterized in that: The step 4 is specifically as follows: TransFG is introduced into micro-expression recognition to solve the problem of feature extraction; Step 4-1: Design a double-layer normalized shift patch tokenization module; Step 4-1-1: Shift the input micro-expression image by half a patch along the four diagonal directions of the upper left, upper right, lower left, and lower right; crop the shifted features to the same size as the input image and then concatenate them with the input; the concatenated features are divided into non-overlapping patches and flattened; obtain the visual token through a double-layer patch normalization module, which is expressed as: in, The i-th shifted image, represents learnable linear projection, LN(·) represents layer normalization; d S represents the hidden dimension of the Transformer encoder, N s represents the number of shifted images; P(.) represents the learnable linear projection; Step 4-1-2: Concatenate the category token and the visual token, and then add the position embedding; The category token is a token with representation information of the entire image, and the position embedding provides position information for the visual token; the formula for applying the module to the Patch embedding layer is as follows: in, Represents a category token, is a learnable position embedding, N is the number of embedding tokens; Step 4-2: Local self-attention; Step 4-2-1: The diagonal mask is defined as follows: in Represents each component of the mask similarity matrix; Step 4-2-2: The local self-attention with diagonal mask and learnable temperature scaling is defined as: where τ is the learnable temperature scaling factor, E v Represents the linear projection weight of Value; R M (x) represents the diagonal mask attention weight.

5. A computer program, characterized in that The computer program enables a computer to execute the method according to any one of claims 1 to 4.

6. An electronic device, characterized in that: include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method as claimed in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

8. A chip, characterized in that: include: A processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes a method as claimed in any one of claims 1 to 4.

9. A computer program product, characterized in that The computer program product comprises a computer storage medium storing a computer program, wherein the computer program comprises instructions executable by at least one processor, and when the instructions are executed by the at least one processor, the method according to any one of claims 1 to 4 is implemented.

Images