A micro-expression recognition method based on visual transformer

Abstract

The application discloses a micro-expression recognition method based on visual Transformer, first, data preprocessing is carried out, and a starting frame and a peak frame are extracted from a micro-expression segment composed of multiple frames; then, a face region in the starting frame and the peak frame is extracted by using MTCNN, so as to remove noise and unnecessary background information; next, a single optical flow feature map is calculated according to two face images obtained after face detection; then, the optical flow feature map is input into an advanced feature extractor composed of a double-layer normalized shift patch marker, a Transformer encoder and a feature selection module, so as to extract fine-grained and discriminative features of micro-expression; finally, the extracted features are input into a classification head, and a classification result is output. The application takes visual Transformer as a backbone network, designs multiple technologies suitable for a micro-expression recognition task on the basis, and realizes micro-expression recognition with superior performance by using a neural network with a very small scale.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a micro-expression recognition method based on visual Transformer. Background Art

[0002] Through a long evolutionary history, humans have developed various ways to express emotions, including language, gestures, and facial expressions. Facial expressions are a universal and intuitive way to convey feelings, and are considered a non-verbal medium for emotional communication. Facial expressions refer to a form of facial movement in which several areas of the human facial muscles are activated to express emotion. Based on the scale and extent of the muscle movements involved, human facial expressions can be categorized as macro-expressions and micro-expressions. Macro-expressions are spontaneous, typically lasting between 1 / 2 and 4 seconds, and are easily detected and perceived. Micro-expressions were first discovered by Haggard and Isaacs in 1966, and three years later by Ekman and Friesen while examining lie detector videos of psychiatric patients. Micro-expressions appear when people attempt to hide or conceal their true emotions, and can reflect a person's most genuine emotional state at the moment, such as happiness, sadness, or surprise.

[0003] Microexpression recognition, a form of microexpression analysis, involves classifying microexpression images or clips into various emotion categories. However, compared to macroexpressions, microexpressions are extremely subtle and brief, typically lasting only 1 / 25 to 1 / 5 of a second (some say 1 / 25 to 1 / 3 of a second). Their presence on a face is incredibly subtle, making analysis and interpretation by the naked eye difficult. Even for experts, quickly and accurately identifying microexpressions is challenging. Furthermore, emotional context and cultural background can also influence microexpression recognition.

[0004] Research on micro-expressions and their recognition is of great significance in fields such as mental health, public safety, business negotiations, and human-computer interaction. Micro-expression recognition can be traced back to 1966, representing nearly six decades of development. Since the 1960s, with advances in machine learning and image processing, computer-based micro-expression recognition has become a significant research direction. Early micro-expression recognition was typically based on image processing techniques and feature engineering, with key steps including video acquisition, image preprocessing, feature extraction, feature selection and dimensionality reduction, classifier classification, and model training and evaluation. In 2011, Pfister et al. used a temporal interpolation model and the first comprehensive spontaneous micro-expression corpus to achieve micro-expression recognition. In 2014, Wang et al. used a six-intersection local binary pattern volume descriptor for micro-expression recognition, providing a more compact and lightweight representation of micro-expressions. In 2015, Liu et al. proposed a simple yet effective principal direction average optical flow feature for micro-expression recognition. In 2017, Lu Guanming et al. proposed a feature selection algorithm based on combining Relief with a local linear embedding manifold learning algorithm, achieving dimensionality reduction of the extracted local binary pattern feature vectors. These methods laid the foundation for machine-aided micro-expression recognition and provided inspiration for subsequent research.

[0005] In recent years, with the advancement and widespread adoption of deep learning methods, various deep learning-based approaches have achieved breakthroughs in micro-expression recognition. These approaches primarily encompass three data modalities: static images, dynamic image sequences, and combinations thereof. In 2019, Zhou et al. used Inception combined with optical flow features to compete in the Micro-Expression Recognition Challenge (MEGC2019), achieving significant performance improvements in the UAR and UF1 metrics. In the same year, Liong et al. designed a shallow three-stream 3D CNN capable of extracting discriminative high-level and detailed representations of micro-expressions. Xia et al. proposed a micro-expression recognition method based on a deep recurrent convolutional network, successfully capturing the spatiotemporal variations of micro-expression sequences. In 2020, Lai Zhenyi et al. proposed a real-time micro-expression recognition algorithm for CNNs combined with dilated convolutions. In the same year, Chen et al. addressed the problem of extracting spatiotemporal features of micro-expressions using spatial and channel-wise attention modules. These approaches have significantly improved micro-expression recognition, demonstrating the enormous development opportunities for deep learning-based micro-expression recognition. However, these approaches also present challenges and room for improvement. Summary of the Invention

[0006] To overcome the shortcomings of the existing technology, the present invention provides a micro-expression recognition method based on a visual Transformer. First, data preprocessing is performed to extract the starting frame and peak frame from a micro-expression segment consisting of multiple frames. Then, the facial region in the starting frame and peak frame is extracted using MTCNN to remove noise and unnecessary background information. Next, a single optical flow feature map is calculated based on the two facial images obtained after face detection. The optical flow feature map is then input into an advanced feature extractor consisting of a two-layer normalized shifted patch marker, a Transformer encoder, and a feature selection module to extract fine-grained and discriminative features of micro-expressions. Finally, the extracted features are input into a classification head to output the classification results. The present invention uses the visual Transformer as the backbone network and designs multiple technologies based on it that can adapt to the micro-expression recognition task. The method achieves excellent micro-expression recognition performance using an extremely small-scale neural network.

[0007] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0008] Step 1: Data preprocessing;

[0009] Extracting the starting frame and the peak frame from a micro-expression segment consisting of multiple frames;

[0010] Step 2: Face detection;

[0011] Use MTCNN to extract the facial regions in the starting frame and peak frame to remove noise and unnecessary background information;

[0012] Step 3: Optical flow feature calculation;

[0013] A single optical flow feature map is calculated based on the two frames of facial images obtained after face detection;

[0014] Step 4: Feature extraction;

[0015] The optical flow feature map is input into an advanced feature extractor consisting of a two-layer normalized shifted patch marker, a Transformer encoder, and a feature selection module to extract fine-grained and discriminative features of micro-expressions.

[0016] Step 5: Classification output;

[0017] The extracted features are input into the classification head and the classification results are output.

[0018] Furthermore, the MTCNN consists of three cascaded convolutional neural networks P-Net, R-Net and O-Net, each of which processes and filters the input image in turn to achieve face detection and alignment.

[0019] Furthermore, the step 3 is specifically as follows:

[0020] Step 3-1: After using MTCNN to detect the starting frame and peak frame of the micro-expression segment, two frames of facial images without background noise are obtained. The optical flow features generated by these two frames are calculated using the following formula:

[0021] V=(u(x,y),v(x,y))|x=1,2,…,X,y=1,2,…,Y. (5)

[0022] Where x and y represent the height and width of the frame, u(x,y) and v(x,y) are the horizontal and vertical components of the optical flow feature map V, V = [V x ,V y ],V∈R W×H×2 ;

[0023] Step 3-2: Use the first-order derivative of the optical flow field to calculate the change in the optical flow field, which is called optical strain;

[0024] The optical strain calculation formula is as follows:

[0025]

[0026] in, is the first-order partial derivative of V;

[0027] Step 3-3: Form a three-dimensional optical flow feature map and represent it as V m =[V x ,V y ,V z ],V m ∈R W×H×3 .

[0028] Furthermore, the step 4 is specifically as follows:

[0029] TransFG is introduced into micro-expression recognition to solve the problem of feature extraction;

[0030] Step 4-1: Design a two-layer normalized shift patch tokenization module;

[0031] Step 4-1-1: Shift the input micro-expression image by half a patch along its diagonal directions in the upper left, upper right, lower left, and lower right directions; crop the shifted features to the same size as the input image and then concatenate them with the input; the concatenated features are divided into non-overlapping patches and flattened; obtain visual tokens through a double-layer patch normalization module, which can be expressed as follows:

[0032]

[0033] in, The image of the i-th displacement, represents the learnable linear projection, LN(·) represents layer normalization; d S represents the hidden dimension of the Transformer encoder, N s represents the number of shifted images; P(.) represents the learnable linear projection;

[0034] Step 4-1-2: Concatenate the category token and the visual token, and then add the position embedding;

[0035] The category token is a token with representation information of the entire image, and the position embedding provides position information for the visual token. The formula for applying the module to the Patch embedding layer is as follows:

[0036]

[0037] in, Represents a category token, is a learnable position embedding, N is the number of embedding tokens;

[0038] Step 4-2: local self-attention;

[0039] Step 4-2-1: The diagonal mask is defined as follows:

[0040]

[0041] in Represents each component of the mask similarity matrix;

[0042] Step 4-2-2: Local self-attention with diagonal mask and learnable temperature scaling is defined as:

[0043]

[0044] where τ is the learnable temperature scaling factor, E v Represents the linear projection weight of Value; R M (x) represents the diagonal mask attention weight.

[0045] The beneficial effects of the present invention are as follows:

[0046] The present invention uses the visual Transformer as the backbone network to realize micro-expression recognition. On this basis, a variety of plug-and-play visual Transformer components are designed that are specially optimized for the micro-expression recognition task. The micro-expression recognition with superior performance is achieved under the condition of using an extremely small-scale neural network (about 7M trainable parameters). BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is the overall workflow diagram of the micro-expression recognition algorithm MEFormer of the present invention;

[0048] Figure 2 It is a workflow diagram of the feature selection module in the advanced feature extractor of the present invention;

[0049] Figure 3 This is a workflow diagram of the dual-layer normalized shifted patch marker in the advanced feature extractor of the present invention;

[0050] Figure 4 This is a diagram of the working principle of the local self-attention mechanism of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described below with reference to the accompanying drawings and examples.

[0052] Given the fleeting and subtle nature of micro-expressions, micro-expression recognition using deep learning methods requires deep neural networks to efficiently extract discriminative features for different micro-expression categories. This requires ingenious neural network design to equip the network with powerful, fine-grained feature capture capabilities. Furthermore, compared to convolutional neural networks, Transformers focus more on global features and, through pre-training, can achieve better classification results than convolutional neural networks on many tasks. However, this comes at the cost of a lack of local inductive bias compared to convolutional neural networks, requiring large amounts of data to effectively represent micro-expressions. However, due to these factors, spontaneous micro-expression data is difficult to collect and annotate, resulting in generally small datasets. While using existing datasets, adapting deep neural networks to small-scale micro-expression datasets is also an issue that needs improvement.

[0053] The purpose of this invention is to propose a new micro-expression recognition method based on visual Transformer to optimize the fine-grained feature extraction and small-scale dataset adaptability problems existing in the above-mentioned micro-expression recognition field.

[0054] To achieve the above objectives, this paper proposes a new micro-expression recognition method, MEFormer. Based on the work objectives, MEFormer can be divided into two main stages. The first stage uses MTCNN (Multi-task Cascaded Convolutional Networks, MTCNN) and an optical flow calculation module to obtain optical flow feature maps of the starting frame and peak frame of a micro-expression segment. The second stage inputs the optical flow feature maps into an advanced feature extractor to extract fine-grained features of micro-expressions, and finally performs classification through a classification head. Specifically, the algorithm includes the following steps:

[0055] 1. Data Preprocessing: Extract the start and peak frames from multi-frame micro-expression segments. During training and testing, the indices of the peak and start frames are provided by the dataset. In actual algorithm deployment and application, due to the transient nature of micro-expressions, it is reasonable to approximate the start and peak frames by two adjacent frames in the video stream.

[0056] 2. Face detection: Use MTCNN to extract the facial regions in the starting frame and peak frame to remove noise and unnecessary background information.

[0057] 3. Optical flow feature calculation: A single optical flow feature map is calculated based on the two frames of facial images obtained after face detection.

[0058] 4. Feature extraction: The optical flow feature map is input into an advanced feature extractor consisting of a two-layer normalized shift patch marker, a Transformer encoder, and a feature selection module to extract fine-grained and discriminative features of micro-expressions.

[0059] 5. Classification output: Input the extracted features into the classification head and output the classification results.

[0060] Example:

[0061] The overall workflow of the present invention is as follows Figure 1 As shown in Figure 2. In MEFormer, the starting frame and peak frame of a micro-expression clip are first fed into the face detection in MTCNN. The two detected facial images are used to calculate their optical flow feature maps. Next, the optical flow feature maps are fed into an advanced feature extractor based on the Vision Transformer backbone for fine-grained feature extraction. The advanced feature extractor consists of a two-layer normalized shifted patch marker, a local self-attention Transformer encoder, and a feature selection module. During training, MEFormer introduces a contrastive loss to enhance the model's sensitivity to discriminative features. Finally, the features generated by the advanced feature extractor are fed into a perceptron consisting of a single fully connected layer to generate a prediction vector, which is then classified by a Softmax classifier. The following is a detailed description of the algorithm's structure and implementation.

[0062] 1. Face detection;

[0063] To automatically recognize micro-expressions, the first step is to detect facial regions from images containing background information and noise. To achieve this, we chose MTCNN. MTCNN is a model for face detection and alignment. It consists of three cascaded convolutional neural networks (P-Net, R-Net, and O-Net). Each network sequentially processes and filters the input image with greater precision. Combining high accuracy with real-time performance, it enables efficient and accurate face detection and alignment, providing a reliable foundation for subsequent emotion recognition.

[0064] 2. Optical flow feature calculation;

[0065] Facial expressions are derived from facial muscle movements. Different forms of movement produce different types of expressions, such as happiness, sadness, and surprise. Therefore, it is a reasonable strategy to transform the classification of facial expressions into the classification of facial muscle movements. This requires extracting facial muscle movement features from micro-expression segments. This paper chooses to use optical flow features to implement motion feature modeling. Optical flow features (Optical Flow) are a method for calculating pixel motion in image sequences. They are used to analyze the movement of objects in images over a period of time and have wide applications in computer vision and image processing.

[0066] A micro-expression segment generally consists of a start frame, a peak frame, an end frame, and the transition frames between them. The peak frame is the frame with the strongest expression and contains the most typical micro-expression information. Specifically, using MTCNN to detect the start frame and peak frame of a micro-expression segment will produce two facial images without background noise. The start frame is plain, and the peak frame is the one with the strongest expression. Calculating the optical flow features between them can maximize the typical characteristics of a micro-expression, which helps the model learn high-quality knowledge. Now we need to calculate the optical flow features generated by these two images. The calculation formula is as follows:

[0067] V=(u(x,y),v(x,y))|x=1,2,…,X,y=1,2,…,Y. (5)

[0068] Where x and y represent the height and width of the frame, u(x,y) and v(x,y) are the horizontal and vertical components of the optical flow feature map V, V = [V x ,V y ],V∈R W×H×2 .

[0069] Next, the first-order derivative of the optical flow field is used to calculate the change in the optical flow field, which is often referred to as optical strain. Optical strain provides an estimate of the degree of facial displacement, thus providing a richer reference for the subtle movements that occur within a micro-expression cycle. This calculation of optical strain can capture and analyze complex facial movements, facilitating accurate micro-expression recognition. Its calculation formula is as follows:

[0070]

[0071] in, is the first-order partial derivative of V. Finally, a three-dimensional optical flow feature map is formed and expressed as V m =[V x ,V y ,V z ],V m ∈R W×H×3 .

[0072] 3. Advanced feature extraction;

[0073] Compared to macro-expressions, the optical flow feature maps of micro-expressions contain finer-grained features, requiring specialized strategies to extract these features. This paper introduces TransFG into micro-expression recognition to address this feature extraction issue. TransFG, an improved Vision Transformer, has been used for fine-grained image recognition tasks and has achieved excellent results on benchmarks such as CUB-200-2011 and StanfordCars, demonstrating its powerful ability to capture fine-grained features.

[0074] Specifically, TransFG has two designs based on ViT:

[0075] Feature selection module: Figure 2 The working principle of the feature selection module is shown. Specifically, in order to make full use of the attention information, TransFG changes the input format of the last Transformer block in the Vision Transformer and adds a feature selection module before the last Transformer block. Assuming that the number of self-attention heads is set to K, the intermediate features input to the last Transformer block can be expressed as And the attention weights of the L-1 Transformer blocks before the last Transformer block are expressed as:

[0076]

[0077] Due to the lack of embedded token identifiability, the original attention weights may not reflect the relative importance of the input tokens, especially for deeper layers in the model. To this end, the feature selection module integrates the attention weights of all layers before the last Transformer block. It recursively applies matrix multiplication to the original attention weights in all layers to obtain new attention weights:

[0078]

[0079] and the original attention weight a of a single Transformer block L-1 In comparison, a final This includes the propagation of micro-expression information from the input layer to higher layers, which is a better design for selecting the discriminant area. final The indices A1, A2, ..., A of the maximum values ​​of the K different attention heads in K , from Z L-1 Extract the token of the corresponding index. Finally, concatenate the selected token with the classification token as the input feature of the last Transformer layer:

[0080]

[0081] By replacing the entire original input sequence with tokens corresponding to the information region and concatenating them with the classification token as the input to the last Transformer layer, the model not only retains the global information, but also forces the last Transformer layer to focus on the subtle differences between micro-expressions of different categories, discarding less discriminative areas such as the background or common features in the category.

[0082] Contrastive learning: Since the feature differences between micro-expression categories are small, simple cross-entropy loss is not enough to fully supervise feature learning. In addition to the cross-entropy loss, contrast loss L is added. con , contrast loss is used to minimize the similarity of classification tokens corresponding to different labels and maximize the similarity of classification tokens of samples with the same category. In order to prevent the loss from being dominated by simple negative samples (different class samples with small similarity), a constant margin α is introduced in the contrast loss. Only negative sample pairs with similarity greater than α will contribute to the loss L con Contribute. The contrast loss of a sample with a batch size of B is expressed as:

[0083]

[0084] where Z i and Z jPreprocessing is performed by L2 normalization, Sim(Z i ,Z j ) is Z i and Z j The total loss function is expressed as:

[0085]

[0086] in is the cross entropy loss function calculated using the predicted label y′ and the true label y.

[0087] In micro-expression recognition, in addition to the challenge of feature extraction, constructing large-scale micro-expression datasets is also a challenging task due to the inherent characteristics of micro-expressions. Consequently, current micro-expression datasets are generally small, which impacts micro-expression recognition performance. Therefore, how to learn rich, high-quality knowledge from small datasets becomes a worthy research question. To adapt MEFormer to small-scale micro-expression recognition data, inspired by Lee et al., this paper redesigns TransFG, designing a two-layer normalized shifted patch tokenization module and improving the self-attention in the Transformer. The following describes these two designs in detail:

[0088] Double-layer normalized shifted patch tokenization module: First, each input image is spatially shifted by half the block size in four diagonal directions (i.e., top left, top right, bottom left, and bottom right). Next, the shifted features are cropped to the same size as the input image and then concatenated with the input. Afterwards, the concatenated features are divided into non-overlapping patches and flattened. Next, visual tokens are obtained through a double-layer patch normalization module. The whole process is as follows: Figure 3 As shown, it can be expressed as:

[0089]

[0090] in, The image of the i-th displacement, represents the learnable linear projection, LN(·) represents layer normalization, and d S represents the hidden dimension of the Transformer encoder, N s Indicates the amount of image shift.

[0091] Next, the category token is concatenated with the visual token, and then the position embedding is added. The category token here is a token with representation information of the entire image, and the position embedding provides position information for the visual token. If the category token is not used, only the position embedding is added to the output of the module. The formula for how to apply the module to the Patch Embedding layer is as follows:

[0092]

[0093] in, Represents a category token, is a learnable position embedding and N is the number of embedding tokens.

[0094] Local Self-Attention: Figure 4 The workflow of local self-attention is presented. Specifically, the diagonal mask assigns higher scores to inter-token relationships by essentially excluding self-token relationships from participating in the softmax operation. Specifically, the diagonal mask sets the diagonal components of the similarity matrix in the QKV calculation to -∞. This makes the self-attention focus more on other tokens instead of the self-token. The proposed diagonal mask is defined as follows:

[0095]

[0096] in Represents each component of the mask similarity matrix.

[0097] The second key feature of local self-attention is learnable temperature scaling. The "temperature" of Softmax refers to a parameter in the Softmax function that controls the smoothness of the output probability distribution. In the Softmax function, an exponential function is applied to each element in the input vector to calculate its exponential value, which is then divided by the sum of the exponential values ​​of all elements to produce a probability distribution. This temperature parameter, typically expressed as a scalar value, is used to adjust the output "temperature" of the Softmax function by dividing it. A higher temperature results in a smoother output distribution, while a lower temperature results in a sharper output distribution. In deep learning, adjusting the Softmax temperature parameter can control the diversity and certainty of the model's output. In local self-attention, it allows MEFormer to independently determine the Softmax temperature during the learning process. Generally speaking, a lower Softmax temperature results in a sharper distribution of attention scores. Learnable temperature scaling makes the distribution of attention scores sharper by lowering the temperature, meaning that the probability values ​​for some classes become higher while the probabilities for other classes approach zero. This enhances the model's certainty, making it more likely to make accurate predictions. For classification, this increased certainty can improve model accuracy. The local self-attention with diagonal mask and learnable temperature scaling is defined as:

[0098]

[0099] where τ is a learnable temperature scaling coefficient.

[0100] To verify the effectiveness of MEFormer, this paper conducted micro-expression recognition experiments on four datasets. This paper introduces the experimental datasets, experimental environment, experimental methods, etc. in detail and analyzes the experimental results.

[0101] 1. Dataset;

[0102] The experiments in this paper are based on the CASME II, SAMM, and SMIC datasets. Furthermore, referring to previous work, to ensure consistency and comparability, CASME II, SAMM, and SMIC were merged into a new composite dataset. The experiments in this paper will be conducted on these four datasets. To unify standards and achieve fair evaluation, the various emotion categories are divided into "positive," "negative," and "surprise" according to the method proposed by Van et al. The specific emotion categories are as follows: the "positive" emotion category includes "happiness," the "negative" emotion category includes "sadness," "disgust," "contempt," "fear," and "anger," and the "surprise" emotion category includes only the "surprise" category. Basic information about the dataset is as follows:

[0103] The SAMM dataset contains 28 participants, 133 micro-expressions and 147 long videos (including 343 macro-expressions). The dataset contains rich action unit encodings and provides comprehensive facial expression information. SAMM also provides the start, end and vertex frame coordinates of micro-expressions. The resolution of the original samples in the dataset is 2040×1088 pixels, and the frame rate is set to 200 frames per second. In order to facilitate the experiment, the present invention cropped the facial images in the original samples to obtain facial images with a resolution of 28×28 pixels. The emotional categories in the images in SAMM are divided into "disgust", "fear", "contempt", "anger", "depression", "surprise", "happiness" and "other" categories. After being divided into three emotion categories, the numbers of "negative", "positive" and "surprise" are 92, 26 and 15 respectively.

[0104] The CASME II dataset consists of data from 24 subjects, with a total of 145 samples. All samples were captured using a laboratory camera with a frame rate set to 200 frames per second. The original size of the sample is 640×480 pixels. In the experiments of the present invention, the facial images were cropped and resized to a resolution of 28×28 pixels. The samples in CASME II are divided into the categories of "happy", "surprised", "disgusted", "sad", "fearful", "depressed", and "other". After being merged into three emotion categories, the number of "negative", "positive", and "surprised" are 88, 32, and 25, respectively. The start, end, and apex frames are annotated in CASME II.

[0105] The SMIC-HS dataset contains data from 16 subjects, totaling 164 samples. All samples were captured using a laboratory camera at a frame rate of 100 frames per second. The original image size of the samples is 640 × 480 pixels. To focus on the facial region of interest and maintain a consistent input size for the micro-expression recognition task, the facial images are cropped to a resolution of 28 × 28 pixels. The number of "negative," "positive," and "surprised" expressions is 70, 51, and 43, respectively.

[0106] The Composite dataset was proposed by the Second Micro-Expression Grammar Challenge (MEGC2019). This composite dataset combines samples from three spontaneous facial ME datasets: CASME II, SAMM, and SMIC-HS. This is to facilitate the evaluation of newly developed methods. Due to significant differences in annotations between the three datasets, the Composite dataset unifies the emotion labels from the three datasets. The emotion labels are re-annotated as "positive," "negative," and "surprise."

[0107] 2. Implementation details;

[0108] All experiments in this paper were performed using the PyTorch 2.0.0 framework on a Windows operating system, and training and evaluation were performed using a single Nvidia RTX 4090 GPU.

[0109] 3. Evaluation indicators;

[0110] In order to address the problem of uneven sample distribution in the micro-expression dataset, this paper uses unweighted average recall and unweighted F1 score as evaluation indicators for the micro-expression recognition experiment. The specific calculation process of the evaluation indicators is as follows:

[0111] Unweighted F1-score (UF1): is a metric commonly used to evaluate the performance of multi-class classification tasks with an imbalanced class distribution. To calculate UF1, it is necessary to calculate the false positives (FP), true positives (TP), and false negatives (FN) for each class c across all folds of leave-one-subject-out (LOSO) cross-validation. Then, the F1 score for each class can be calculated using the following formula:

[0112]

[0113] Unweighted Average Recall (UAR): This is a particularly useful metric for evaluating model effectiveness when the class ratio is unbalanced. To calculate UAR, we first count the true positive examples (TP) for each emotion class c. The true positive examples represent the number of correctly classified examples in each class. In addition, we also count the total number of examples in each class:

[0114]

[0115] Among them, TP c Indicates the number of true positive samples of a specific category, FP c Indicates the number of false positive samples of a specific category, FN c represents the number of false negative samples of a specific category, c represents the specific category, and C represents the number of categories, which is 3 in the present invention.

[0116] 4. Comparative experiment;

[0117] To verify the performance of MEFormer, this paper conducted comparative experiments and compared MEFormer with existing methods on SMIC, SAMM, CASMEII and Composite datasets.

[0118] As shown in Table 1, MEFormer generally achieves better results than existing methods on all four datasets. Compared to the state-of-the-art HTNet, the UF1 and UAR scores on the composite dataset improve by up to 1.56% and 2.72%, the UF1 score on the SAMM dataset improves by up to 1.57%, the UF1 score on the CASME II dataset improves by up to 2.76% and 2.75%, and the UF1 and UAR scores on the SMIC dataset improve by up to 1.18% and 3.1%, achieving significant improvements. Notably, HTNet performs better on the UAR metric on the SAMM dataset. This is believed to be due to the uneven label distribution in the SAMM dataset, which leads to a loss in recall, a topic worthy of future research. Overall, the experimental results conclude that MEFormer effectively optimizes the adaptability to small datasets and the extraction of fine-grained features.

[0119] Table 1 Comparative experiments with existing methods in the field of micro-expression recognition based on deep learning

[0120]

[0121]

Claims

1. A micro-expression recognition method based on visual Transformer, characterized in that: The steps include: Step 1: Data preprocessing; Extracting the starting frame and the peak frame from a micro-expression segment consisting of multiple frames; Step 2: Face detection; Use MTCNN to extract the facial regions in the starting frame and peak frame to remove noise and unnecessary background information; Step 3: Optical flow feature calculation; A single optical flow feature map is calculated based on the two frames of facial images obtained after face detection; Step 4: Feature extraction; The optical flow feature map is input into an advanced feature extractor consisting of a two-layer normalized shifted patch marker, a Transformer encoder, and a feature selection module to extract fine-grained and discriminative features of micro-expressions. TransFG is introduced into micro-expression recognition to solve the problem of feature extraction; Step 4-1: Design a two-layer normalized shift patch tokenization module; Step 4-1-1: Shift the input micro-expression image by half a patch along its diagonal directions in the upper left, upper right, lower left, and lower right directions; crop the shifted features to the same size as the input image and then concatenate them with the input; the concatenated features are divided into non-overlapping patches and flattened; obtain visual tokens through a double-layer patch normalization module, which can be expressed as follows: in, The image of the i-th displacement, represents the learnable linear projection, LN(·) represents layer normalization; d S represents the hidden dimension of the Transformer encoder, N s represents the number of shifted images; P(.) represents the learnable linear projection; Step 4-1-2: Concatenate the category token and the visual token, and then add the position embedding; The category token is a token with representation information of the entire image, and the position embedding provides position information for the visual token. The formula for applying the module to the Patch embedding layer is as follows: in, Represents a category token, is a learnable position embedding, N is the number of embedding tokens; Step 4-2: local self-attention; Step 4-2-1: The diagonal mask is defined as follows: in Represents each component of the mask similarity matrix; Step 4-2-2: Local self-attention with diagonal mask and learnable temperature scaling is defined as: where τ is the learnable temperature scaling factor, E v Represents the linear projection weight of Value; R M (x) represents the diagonal mask attention weight; Step 5: Classification output; The extracted features are input into the classification head and the classification results are output.

2. A micro-expression recognition method based on visual Transformer according to claim 1, characterized in that: The MTCNN consists of three cascaded convolutional neural networks: P-Net, R-Net, and O-Net. Each network processes and filters the input image in turn to achieve face detection and alignment.

3. A micro-expression recognition method based on visual Transformer according to claim 2, characterized in that: The step 3 is specifically as follows: Step 3-1: After using MTCNN to detect the starting frame and peak frame of the micro-expression segment, two frames of facial images without background noise are obtained. The optical flow features generated by these two frames are calculated using the following formula: V=(u(x,y),v(x,y))|x=1,2,…,X,y=1,2,…,Y. (5) Where x and y represent the height and width of the frame, u(x,y) and v(x,y) are the horizontal and vertical components of the optical flow feature map V, V = [V x ,V y ],V∈R W×H×2 ; Step 3-2: Use the first-order derivative of the optical flow field to calculate the change in the optical flow field, which is called optical strain; The optical strain calculation formula is as follows: in, is the first-order partial derivative of V; Step 3-3: Form a three-dimensional optical flow feature map and represent it as V m =[V x ,V y ,V z ],V m ∈R W×H×3 .

4. An electronic device, characterized in that: include: processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 3.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

6. A chip, characterized in that: include: A processor, configured to call and run a computer program from a memory, so that a device equipped with the chip executes the method according to any one of claims 1 to 3.

7. A computer program product, characterized in that The computer program product comprises a computer storage medium storing a computer program, wherein the computer program comprises instructions executable by at least one processor, and when the instructions are executed by the at least one processor, the method according to any one of claims 1 to 3 is implemented.

Images