A micro-expression recognition method and system based on a double-flow gate Transformer

Abstract

The application discloses a micro-expression recognition method and system based on a double-flow gate Transformer, and the micro-expression recognition method constructs a micro-expression recognition model based on deep learning technology to recognize the micro-expression emotion category of a face image to be recognized.In the micro-expression recognition model, parallel static appearance flow and dynamic motion flow are constructed, wherein the static flow adopts a convolutional neural network to extract face texture features and embed a channel attention module to enhance the response of key areas, and the dynamic flow generates a motion residual image by performing difference operation on adjacent frames, and then utilizes a convolutional neural network to extract inter-frame displacement features and also embeds a channel attention module; subsequently, the two types of features are respectively input into independent time sequence Transformer encoders to output static time sequence feature vectors and dynamic time sequence feature vectors; finally, the two types of time sequence feature vectors are weightedly fused through a gate adaptive fusion module to output a micro-expression emotion category.

Description

Technical Field

[0001] This invention belongs to the field of emotion recognition technology, specifically relating to a micro-expression recognition method and system based on dual-stream gated Transformer. Background Technology

[0002] Microexpressions are involuntary facial muscle responses that occur when an individual suppresses their true emotions. They are characterized by short duration (0.04-0.5 seconds), small amplitude of movement, and difficulty in faking, making them valuable in fields such as public safety, psychological diagnosis and treatment, and human-computer interaction. However, microexpression signals are weak and easily affected by factors such as changes in lighting and head posture, making robust automatic recognition a technical challenge in the field of computer vision.

[0003] Early micro-expression recognition primarily employed hand-designed feature descriptors, such as LBP-TOP and optical flow methods. These methods have limited representational capabilities and poor robustness to complex environmental changes. With the development of deep learning, researchers have proposed architectures such as C3D networks and two-stream CNNs. C3D networks have a large number of parameters and insufficient ability to model micro-motion signals; two-stream CNNs use optical flow as a temporal input, resulting in long computation times and susceptibility to noise interference, limiting the model's real-time performance and generalization ability. In recent years, the Transformer architecture has been introduced into the field of micro-expression recognition due to its advantages in long-range dependency modeling.

[0004] However, the existing technologies still have the following drawbacks: (1) The signal-to-noise ratio of micro-expression signals is extremely low. It is difficult to capture instantaneous muscle deformation by simply relying on static texture features. Existing motion feature extraction methods (such as optical flow) have a large computational load and are sensitive to the environment, lacking efficient motion enhancement methods; (2) Most methods use a single network or simply splice and fuse static appearance and dynamic motion features, ignoring the semantic heterogeneity of the two types of features and failing to achieve adaptive collaboration; (3) Public datasets (such as CASMEII and SAMM) are small in scale and have extremely unbalanced category distribution, which makes deep model training prone to overfitting and has poor recognition performance for rare emotion categories; (4) Existing time series modeling mostly uses RNN or LSTM. In capturing the complete evolution of micro-expressions from the beginning to the peak and then to the decay, there are gradient vanishing or long-term information forgetting problems, making it difficult to fully model the long-range dependency relationship between frames. Summary of the Invention

[0005] This invention addresses the problems in existing micro-expression recognition technologies, such as weak and difficult-to-extract micro-expression signals, low efficiency in fusing static appearance and dynamic motion features, easy overfitting of models under small sample conditions, and insufficient modeling of time-dependent factors. It provides a micro-expression recognition method and system based on a dual-stream gated Transformer.

[0006] In a first aspect, the present invention provides a micro-expression recognition method based on a dual-stream gated Transformer, the method comprising:

[0007] Acquire multiple consecutive frames of facial images of the test subject, and select multiple key frame images as temporal image blocks to input into the micro-expression recognition model;

[0008] The micro-expression recognition model includes a static appearance flow branch, a dynamic motion flow branch, a static temporal Transformer encoder, a dynamic temporal Transformer encoder, and a gated adaptive fusion module. The static appearance flow branch processes temporal image blocks of each frame to obtain static texture deep features corresponding to each frame's temporal image block. The dynamic motion flow branch processes the motion residual map between temporal image blocks of adjacent frames to obtain inter-frame displacement deep features. The static temporal Transformer encoder processes a feature sequence composed of multiple static texture deep features to obtain a static temporal feature vector. The dynamic temporal Transformer encoder processes a feature sequence composed of multiple inter-frame displacement deep features to obtain a dynamic temporal feature vector. The gated adaptive fusion module fuses the static and dynamic temporal feature vectors and performs classification based on the fusion result to obtain the micro-expression emotion category output by the micro-expression recognition model.

[0009] Preferably, the method for selecting the temporal image block is as follows: calculate the inter-frame motion intensity, locate the peak frame with the largest motion intensity, and take the peak frame and the facial images of the preceding and following frames centered on the peak frame as the temporal image block.

[0010] Preferably, the motion residual map is obtained by performing pixel-level difference operations on temporal image blocks of two adjacent frames in sequence.

[0011] Preferably, in the static apparent flow branch, a first convolutional neural network is used to extract features from the temporal image block to obtain a first feature extraction vector; the first channel attention module is used to process the first feature extraction vector to output the deep features of the static texture.

[0012] Preferably, the first convolutional neural network is a ResNet18 network with the original fully connected layers removed.

[0013] Preferably, in the dynamic motion flow branch, the motion residual map between two adjacent temporal image blocks is obtained; the second convolutional neural network is used to extract features from the motion residual map to obtain a second feature extraction vector; the second channel attention module is used to process the second feature extraction vector to output the deep features of inter-frame displacement.

[0014] Preferably, the second convolutional neural network is an EfficientNetB0 network with the original classifier layer removed.

[0015] Preferably, the static temporal Transformer encoder and the dynamic temporal Transformer encoder have the same structure, both including a positional encoding layer, a stacked multi-layer Transformer encoder layer, and a global temporal average pooling layer connected in sequence; the positional encoding layer is used to construct a learnable parameter matrix with the same length and feature dimension as the input feature sequence, and the learnable parameter matrix is ​​added to the input feature sequence element by element; the Transformer encoder layer uses a multi-head self-attention sub-layer and a feedforward neural network sub-layer to process the input feature sequence.

[0016] Preferably, in the gated adaptive fusion module, the static temporal feature vector and the dynamic temporal feature vector are concatenated along the channel dimension to obtain a concatenated vector; the concatenated vector is processed by a gated sub-network, and the processing result is normalized by Softmax to obtain the fusion weights. and fusion weight According to fusion weight and fusion weight The static temporal feature vector and the dynamic temporal feature vector are weighted and fused to obtain a fused feature vector. The fused feature vector is then processed by layer normalization, fully connected layer and SoftMax activation function to obtain the micro-expression emotion category.

[0017] Preferably, the gated subnetwork includes a first fully connected layer, a GELU activation function, and a second fully connected layer connected in sequence.

[0018] Secondly, the present invention provides a micro-expression recognition system based on a dual-stream gated Transformer, which is used to perform the micro-expression recognition method described above. The micro-expression recognition system includes an image acquisition module, an image preprocessing module, and an image recognition module. The image acquisition module is used to acquire multiple consecutive frames of facial images to be recognized. The image preprocessing module is used to extract temporal image blocks from the multiple frames of facial images to be recognized. The image recognition module is used to output the emotion type corresponding to the facial images to be recognized based on the temporal image blocks.

[0019] Thirdly, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory stores the computer program; and the processor executes the aforementioned micro-expression recognition method.

[0020] Fourthly, the present invention provides a readable storage medium storing a computer program; when executed by a processor, the computer program is used to implement the aforementioned micro-expression recognition method.

[0021] The beneficial effects of this invention are:

[0022] 1. This invention uses inter-frame difference instead of traditional optical flow as motion feature input, which greatly reduces computational complexity. At the same time, combined with the EfficientNetB0 lightweight network and channel attention module, the frame sequence preprocessing time on the CASME II dataset is reduced by about 85% compared with the optical flow method, and the error accumulation problem of optical flow estimation in high noise environment is avoided.

[0023] 2. This invention uses a gated adaptive fusion module to replace the traditional feature splicing or fixed weighted fusion. It can dynamically adjust the weight ratio of static and dynamic features according to the sample quality. When motion blur occurs, the static flow weight is automatically increased, which enhances the model's anti-interference ability and brings an accuracy improvement of about 2.1% on the CASME II dataset and about 1.8% on the SAMM dataset.

[0024] 3. This invention independently embeds channel attention modules in both static and dynamic flows. Experiments show that in the CASMEII five-class classification task, the model accuracy increased from 68.5% to 72.0% after adding dual-channel attention, and in the SAMM three-class classification task, the accuracy increased from 73.2% to 76.0%, verifying the selective enhancement effect of channel attention on weak facial expression features.

[0025] 4. This invention employs independent temporal Transformer encoders to model static and dynamic feature sequences separately, achieving an F1 score improvement of approximately 4.6% on the CASME II dataset compared to LSTM-based methods. Through learnable positional encoding and multi-head self-attention mechanisms, it effectively captures the temporal dependencies of micro-expressions throughout their entire process, from initiation and peak to decay.

[0026] 5. This invention freezes the convolutional neural network during the model training phase and only updates the parameters of the channel attention module, the Transformer encoder, and the gated fusion module. Under the small sample conditions of CASME II (only 247 samples) and SAMM (only 158 samples), it effectively suppresses overfitting. The final test accuracy reaches 72.0% and 76.0% respectively, which is better than existing mainstream methods such as LBP-TOP (63.4%), AKMNet (67.1%), Later (70.7%) and SSSN (71.2%). Attached Figure Description

[0027] Figure 1 This is an overall flowchart of the present invention.

[0028] Figure 2This is a schematic diagram of the static apparent flow branch and the dynamic motion flow branch in this invention.

[0029] Figure 3 This is a schematic diagram of the gated adaptive fusion module in this invention.

[0030] Figure 4 This is a graph showing the changes in the accuracy of the training and validation sets as a function of the number of training rounds in this invention. Detailed Implementation

[0031] The present invention will be further described below with reference to the accompanying drawings.

[0032] A micro-expression recognition method and system based on a dual-stream gated Transformer is disclosed. The micro-expression recognition system includes an image acquisition module, an image preprocessing module, an image recognition module, and a training module. The image acquisition module acquires multiple consecutive frames of facial images. The image preprocessing module corrects head posture, crops core expression regions, and extracts temporal image patches from the multiple frames of facial images to be recognized. The image recognition module uses the micro-expression recognition model to output the corresponding emotion type based on the temporal image patches. The training module trains the micro-expression recognition model.

[0033] like Figure 1 As shown, the micro-expression recognition method includes the following steps:

[0034] Step 1: Building the dataset

[0035] This embodiment uses two mainstream micro-expression datasets, CASME II and SAMM, to construct the dataset of this invention. The CASME II dataset contains 247 micro-expression samples, covering five categories of emotions: happiness, surprise, disgust, repression, and others. The subjects are Asian, and accurate Apex frame annotations are provided. The SAMM dataset contains 158 micro-expression samples, and the subjects cover multiple ethnic groups. It is used to test the model's cross-ethnicity generalization ability.

[0036] Step 2: Data Preprocessing

[0037] Read multiple consecutive frames of facial images, obtain the emotion category label for each micro-expression sample, and then perform preprocessing operations on each sample. Detect the facial region in each frame using methods from OpenCV or Dlib libraries, locate key points, and perform affine transformations to correct head pose. Then, crop the core expression region (from the upper edge of the eyebrow to the lower edge of the chin) and scale it to a uniform size of 224×224 pixels. Calculate the sum of the absolute values ​​of the grayscale differences between frames, smooth it using a sliding window, locate the peak frame (Apex) with the highest motion intensity, and take 3 frames before and after the peak frame (Apex) to form a 7-frame temporal image block.

[0038] Step 3: Construct a micro-expression recognition model

[0039] The micro-expression recognition model comprises a static appearance flow branch, a dynamic motion flow branch, a static temporal Transformer encoder, a dynamic temporal Transformer encoder, and a gated adaptive fusion module. The static appearance flow branch processes temporal image blocks of each frame separately to obtain the corresponding static texture deep features for each frame's temporal image block. The dynamic motion flow branch processes temporal image blocks from adjacent frames together to obtain six inter-frame displacement deep features. The static temporal Transformer encoder processes the feature sequence composed of the seven static texture deep features output by the static appearance flow branch to obtain a static temporal feature vector; the dynamic temporal Transformer encoder processes the feature sequence composed of the six inter-frame displacement deep features output by the dynamic motion flow branch to obtain a dynamic temporal feature vector. The gated adaptive fusion module fuses the static and dynamic temporal feature vectors and further processes the fusion result to obtain the micro-expression emotion category output by the micro-expression recognition model.

[0040] like Figure 2 As shown, in the static appearance flow branch, a first convolutional neural network is used to extract features from temporal image blocks, resulting in a 512-dimensional first feature extraction vector. A first-channel attention module processes the first feature extraction vector, outputting a weighted 512-dimensional static texture deep feature. A linear projection layer is then used to linearly project the static texture deep feature, mapping the 512-dimensional static texture deep feature to a 32-dimensional static texture deep feature. In the first-channel attention module, global average pooling and global max pooling are performed on the first feature extraction vector, mapped by a shared multilayer perceptron (the number of hidden layer neurons is the number of channels divided by a reduction ratio of 16), and then summed. The sum is then activated by a sigmoid function to generate the first-channel weights. The first-channel weights are multiplied element-wise by the first feature extraction vector to obtain the static texture deep feature.

[0041] In this embodiment, the pre-trained ResNet18 network with its original fully connected layers removed is used as the first convolutional neural network.

[0042] In the dynamic motion flow branch, pixel-level difference operations are performed on temporal image blocks of adjacent frames sequentially to obtain 6 sets of motion residual maps. A second convolutional neural network is used to extract features from the motion residual maps, resulting in a 1280-dimensional second feature extraction vector. A second-channel attention module is used to process the second feature extraction vector, outputting a weighted 1280-dimensional inter-frame displacement deep feature. A linear projection layer is then used to linearly project the inter-frame displacement deep feature, mapping the 1280-dimensional inter-frame displacement deep feature to a 32-dimensional inter-frame displacement deep feature.

[0043] In this embodiment, the pre-trained EfficientNetB0 network is used as a second convolutional neural network after its original classifier layer is removed.

[0044] In this embodiment, the structure of the second channel attention module is the same as that of the first channel attention module, and the reduction ratio is 32.

[0045] The static and dynamic temporal Transformer encoders share the same structure, both consisting of a positional encoding layer, two stacked Transformer encoder layers, and a global temporal average pooling layer. The positional encoding layer initializes a learnable parameter matrix with the same length and feature dimension (32-dimensional) as the input sequence, adding it element-wise to the input feature sequence to inject inter-frame temporal order information. Each Transformer encoder layer contains a multi-head self-attention sub-layer and a feedforward neural network sub-layer, each followed by a Pre-LayerNorm structure and residual connections. The multi-head self-attention sub-layer has 8 heads, ensuring that 32 is divisible by 8. The hidden layer dimension of the feedforward neural network sub-layer is 128. The global temporal average pooling layer calculates the mean of the sequence features output from the Transformer encoder layer along the time dimension, obtaining a fixed-dimensional temporal feature vector (32-dimensional). The encoder output for the static stream is a static temporal feature vector. The encoder corresponding to the dynamic stream outputs a dynamic temporal feature vector. .

[0046] like Figure 3 As shown, in the gated adaptive fusion module, the static temporal feature vector is... and dynamic time series feature vectors The concatenation is performed along the channel dimension to obtain a 64-dimensional concatenated vector. A gated subnetwork consisting of two fully connected layers and a GELU activation function (the first fully connected layer maps the 64-dimensional vector to 32-dimensional vectors, and the second fully connected layer maps the 32-dimensional vector to 2-dimensional vectors) is then used to process the concatenated vector. The processing result is then normalized using Softmax to obtain the fusion weights. and fusion weight ( ); by fusion weight and fusion weight For static time series feature vectors and dynamic time series feature vectors Weighted fusion is performed to obtain a 32-dimensional fusion feature vector. ; sequentially fuse the feature vectors The micro-expression emotion categories and confidence scores are obtained by performing layer normalization, Dropout (dropout rate 0.2), fully connected layer (output dimension is the number of emotion categories C) and SoftMax activation function.

[0047] Step 4: Model Training

[0048] The dataset was randomly partitioned into training, validation, and test sets in an 8:1:1 ratio (CASME II uses global randomization, SAMM uses independent partitioning for each subject). Focal Loss was used as the objective function (focusing parameter γ = 2.0, balance factor dynamically calculated based on the number of samples in each class), and the AdamW optimizer was employed. The initial learning rate was 8 × 10⁻⁶ for CASME II. -4 SAMM is taken as 1×10 -3 The weight decay coefficient is 1×10 -4 During training, the pre-trained weights of ResNet18 and EfficientNetB0 are frozen. The training epochs are 40, and the batch size is 16. The ReduceLROnPlateau learning rate scheduling strategy is used; if the validation loss does not decrease for 5 consecutive epochs, the learning rate is multiplied by 0.5.

[0049] Input the multi-frame facial images (7-frame temporal image blocks) to be identified into the trained micro-expression recognition model, and output the micro-expression emotion category and the corresponding confidence score. Figure 4 The graph shows the accuracy of this invention on the CASME II validation set. The final test accuracy reached 72.0% (CASME II five-class classification) and 76.0% (SAMM three-class classification). The recognition comparison results of this invention with existing models are shown in Table 1.

[0050] Table 1. Comparison of recognition results between the present invention and other models.

[0051]

[0052] The dual-stream gated Transformer architecture designed in this invention extracts static texture features and inter-frame displacement features of micro-expressions in parallel through static appearance flow and dynamic motion flow. Channel attention modules are embedded in both flows to enhance the response of key emotion-related channels, effectively compensating for the insufficient ability of a single network to extract weak expression features. Independent temporal modeling and gated adaptive fusion of heterogeneous features from both flows allow the network to dynamically adjust the fusion weights of static and dynamic features based on sample quality. When the motion signal is disturbed by illumination or pose, the model automatically strengthens the static flow weights, and vice versa, significantly improving the model's robustness and generalization ability in complex environments. The temporal Transformer encoder establishes long-range inter-frame dependencies through a multi-head self-attention mechanism, fully capturing the dynamic evolution of micro-expressions from onset, peak, to decay, overcoming the gradient vanishing or long-term information forgetting problems inherent in traditional RNN / LSTM. Experimental results on the CASME II and SAMM public datasets show that the recognition accuracy of the present invention reaches 72.0% and 76.0% respectively, which is better than the existing mainstream methods, verifying the effectiveness and advancement of the present invention.

[0053] The above-described specific technical process is used to explain and illustrate the present invention, rather than to limit it. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A micro-expression recognition method based on a dual-stream gated Transformer, characterized in that: The method includes: Acquire multiple consecutive frames of facial images of the test subject, and select multiple key frame images as temporal image blocks to input into the micro-expression recognition model; The micro-expression recognition model includes a static appearance flow branch, a dynamic motion flow branch, a static temporal Transformer encoder, a dynamic temporal Transformer encoder, and a gated adaptive fusion module. The static appearance flow branch processes temporal image blocks of each frame to obtain static texture deep features corresponding to each frame's temporal image block. The dynamic motion flow branch processes the motion residual map between temporal image blocks of adjacent frames to obtain inter-frame displacement deep features. The static temporal Transformer encoder processes a feature sequence composed of multiple static texture deep features to obtain a static temporal feature vector. The dynamic temporal Transformer encoder processes a feature sequence composed of multiple inter-frame displacement deep features to obtain a dynamic temporal feature vector. The gated adaptive fusion module fuses the static and dynamic temporal feature vectors and performs classification based on the fusion result to obtain the micro-expression emotion category output by the micro-expression recognition model.

2. The micro-expression recognition method based on dual-stream gated Transformer according to claim 1, characterized in that: The method for selecting the temporal image block is as follows: calculate the inter-frame motion intensity, locate the peak frame with the largest motion intensity, and take the peak frame and the facial images of the previous and next frames centered on the peak frame as the temporal image block.

3. The micro-expression recognition method based on dual-stream gated Transformer according to claim 1, characterized in that: The motion residual map is obtained by performing pixel-level difference operations on temporal image blocks of two adjacent frames in sequence.

4. The micro-expression recognition method based on dual-stream gated Transformer according to claim 1, characterized in that: In the static appearance flow branch, a first convolutional neural network is used to extract features from temporal image blocks to obtain a first feature extraction vector; the first channel attention module is used to process the first feature extraction vector to output the deep features of static texture.

5. The micro-expression recognition method based on dual-stream gated Transformer according to claim 4, characterized in that: The first convolutional neural network uses a ResNet18 network with the original fully connected layers removed.

6. The micro-expression recognition method based on dual-stream gated Transformer according to claim 1, characterized in that: In the dynamic motion flow branch, the motion residual map between two adjacent temporal image blocks is obtained; the second convolutional neural network is used to extract features from the motion residual map to obtain the second feature extraction vector; the second channel attention module is used to process the second feature extraction vector to output the deep features of inter-frame displacement.

7. The micro-expression recognition method based on dual-stream gated Transformer according to claim 6, characterized in that: The second convolutional neural network uses the EfficientNetB0 network after removing the original classifier layer.

8. The micro-expression recognition method based on dual-stream gated Transformer according to claim 1, characterized in that: The static temporal Transformer encoder and the dynamic temporal Transformer encoder have the same structure, both including a positional encoding layer, stacked multi-layer Transformer encoder layers, and a global temporal average pooling layer connected in sequence. The positional encoding layer is used to construct a learnable parameter matrix with the same length and feature dimension as the input feature sequence, and the learnable parameter matrix is ​​added to the input feature sequence element by element. The Transformer encoder layer uses a multi-head self-attention sub-layer and a feedforward neural network sub-layer to process the input feature sequence.

9. The micro-expression recognition method based on dual-stream gated Transformer according to claim 1, characterized in that: In the gated adaptive fusion module, the static temporal feature vector and the dynamic temporal feature vector are concatenated along the channel dimension to obtain the concatenated vector. The concatenated vector is then processed by a gated sub-network, and the processing result is normalized using Softmax to obtain the fusion weights. and fusion weight According to fusion weight and fusion weight The static temporal feature vector and the dynamic temporal feature vector are weighted and fused to obtain a fused feature vector. The fused feature vector is then processed by layer normalization, fully connected layer and SoftMax activation function to obtain the micro-expression emotion category.

10. A micro-expression recognition system based on a dual-stream gated Transformer, characterized in that: This system is used to perform a micro-expression recognition method based on a dual-stream gated Transformer as described in claim 1. The micro-expression recognition system includes an image acquisition module, an image preprocessing module, and an image recognition module. The image acquisition module is used to acquire multiple consecutive frames of facial images to be recognized. The image preprocessing module is used to extract temporal image blocks from the multiple frames of facial images to be recognized. The image recognition module is used to output the emotion type corresponding to the facial images to be recognized based on the temporal image blocks.

Images