Micro-expression recognition method based on motion amplification and hierarchical Transform
By employing motion amplification and hierarchical Transformer methods, the problem of extracting weak, local, and transient features in micro-expression recognition is solved, achieving higher recognition accuracy and stronger generalization ability, applicable to various datasets and complex environments.
Patent Information
- Application Number
- CN202511657736.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-13
- Publication Date
- 2026-02-06
AI Technical Summary
Existing technologies struggle to capture subtle, localized, and transient dynamic features in micro-expression recognition, resulting in insufficient recognition accuracy.
We employ a motion magnification and hierarchical Transformer approach. By performing motion magnification on micro-expression video sequences, we calculate optical flow maps using the RAFT optical flow estimation method, extract motion relationship features of various facial parts through a hierarchical Transformer network, fuse features using a block aggregation layer, and finally classify micro-expressions using a classifier.
It significantly improves the accuracy and generalization ability of micro-expression recognition, can adapt to different ethnicities and collection environments, and is suitable for complex real-world application scenarios.
Smart Images
Figure CN121482845A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of micro-expression recognition, and particularly relates to a micro-expression recognition method based on motion amplification and hierarchical Transformer. BACKGROUND
[0002] Facial expression is an important carrier of human emotional expression and perception. Micro-expression, as a special facial expression, is a short and rapid facial muscle movement that is involuntarily revealed when an individual tries to suppress or hide the true emotion. Its duration is usually only 1 / 25 to 1 / 5 seconds, and the movement amplitude is extremely subtle. Since micro-expression is highly related to the true emotional state of an individual, it has important application value in the fields of clinical psychology, national security, judicial lie detection, etc.
[0003] However, the recognition and research of micro-expression face great challenges. The "short and weak" characteristics make it difficult for even trained human observers to reliably capture and identify it with the naked eye. In the field of computer vision, traditional micro-expression recognition methods usually follow a two-stage process of feature extraction and classification. Early methods mostly rely on hand-designed features (such as LBP-TOP, Optical Strain) to capture dynamic information in the spatial and temporal domain. Although these methods have made some progress, their feature representation ability is limited, and they lack sensitivity to subtle movements, making it difficult to generalize to different datasets and real-world scenarios.
[0004] Deep learning technology, especially convolutional neural networks (CNN), has been introduced into the field of micro-expression recognition and has shown potential superior to traditional methods. However, standard CNN models still have significant limitations when dealing with micro-expression:
[0005] It is difficult to capture subtle movements. Since the motion change between the starting frame and the peak frame of micro-expression is very small, CNN models have difficulty extracting discriminative dynamic features from it, resulting in the loss of key motion information. In addition, there is a lack of focus on local key regions. The essence of micro-expression is the coordinated movement of local facial muscle units (such as orbicularis oculi, zygomaticus major). Existing CNN models usually treat the global image equally, lacking an effective mechanism to focus on these local regions (such as the eye and the mouth) that are crucial to recognition, thus introducing a lot of irrelevant background noise. Existing models are weak in fusing multi-scale features and establishing long-distance spatial dependencies, and cannot fully model the complex motion relationships of different parts of the face during micro-expression occurrence.
[0006] In summary, the existing technology has obvious deficiencies in dealing with the subtlety, locality and transience of micro-expression, which limits the further improvement of micro-expression recognition accuracy. Therefore, the present application proposes a micro-expression recognition method based on motion amplification and hierarchical Transformer. SUMMARY
[0007] The purpose of the present application is to address the problem that the prior art has obvious deficiencies in dealing with the faintness, locality and transience of micro-expression.
[0008] The technical scheme of the present application: a micro-expression recognition method based on motion amplification and hierarchical Transformer, comprising the following steps:
[0009] S1, performing motion amplification processing on the starting frame and the peak frame in the micro-expression video sequence to obtain an amplified peak frame;
[0010] S2, calculating the optical flow map between the starting frame and the amplified peak frame based on the RAFT optical flow estimation method;
[0011] S3, inputting the optical flow map into the hierarchical Transformer network to extract the motion relationship features between each part of the face;
[0012] S4, fusing the multi-scale features output by the hierarchical Transformer network through a block aggregation layer to obtain enhanced feature representation;
[0013] S5, based on the enhanced feature representation, using a classifier to classify micro-expression.
[0014] Optionally, the motion amplification processing in step S1 is implemented through a deep learning model, and is specifically represented as:
[0015]
[0016] wherein, is the starting frame of the micro-expression, is the peak frame of the micro-expression, is the amplification ratio, represents the motion amplification technology using deep learning used, represents the micro-expression peak frame after amplification.
[0017] Optionally, the RAFT optical flow estimation method in step S2 comprises:
[0018] using a convolutional neural network to extract feature maps of the starting frame and the amplified peak frame respectively;
[0019] constructing a feature correlation matrix to calculate the similarity between pixels;
[0020] iteratively updating the optical flow field based on the gated recurrent unit to gradually optimize the optical flow estimation result;
[0021] output the optical flow map between the starting frame and the peak frame after amplification.
[0022] Optionally, the RAFT method adopts a multi-head self-attention mechanism in the optical flow estimation process to process multiple feature subspaces in parallel.
[0023] Optionally, the hierarchical Transformer network in step S3 includes multiple levels, and each level includes:
[0024] a local self-attention module for capturing subtle movements within a key facial region;
[0025] a cross-block feature interaction mechanism for facilitating information fusion between local and global features.
[0026] Optionally, the block aggregation layer in step S4 includes:
[0027] a convolution layer for feature transformation;
[0028] a layer normalization layer for stabilizing the training process;
[0029] a max-pooling layer for dimension reduction and significant feature preservation.
[0030] Optionally, the classifier used in step S5 is a multi-layer perceptron, and the loss function is cross-entropy loss.
[0031] Optionally, the motion amplification factor is in the range of 1-10.
[0032] Compared with the prior art, the present application has at least one of the following beneficial technical effects:
[0033] Through the motion amplification technology based on deep learning, the motion amplitude between the starting frame and the peak frame of micro-expression is nonlinearly increased, and the difficulty of feature extraction caused by too subtle motion is fundamentally solved.
[0034] The local attention mechanism is introduced into the hierarchical Transformer, so that the model can automatically focus on key facial action units such as the eye and the corner of the mouth, effectively filter irrelevant region interference, and extract more discriminative features.
[0035] The structure of hierarchical Transformer combined with block aggregation layer adaptively fuses multi-scale information from local subtle dynamics to global expression patterns, greatly enhancing the recognition robustness of different intensity micro-expressions.
[0036] The method is verified to be effective on mixed datasets containing multiple races and collection environments (CASME II, SMIC, SAMM), indicating that the method can adapt to individual differences and is more suitable for complex real application scenarios.
[0037] The microexpression recognition method based on motion amplification and hierarchical Transformer provided by the application enhances the weak motion signal of microexpression through motion amplification technology, accurately focuses on key facial regions such as eye and mouth corners by using local attention mechanism in hierarchical Transformer, and realizes multi-scale feature fusion by combining block aggregation, and finally verifies higher recognition accuracy and stronger generalization ability in multiple datasets. BRIEF DESCRIPTION OF DRAWINGS
[0038] Fig. 1 is a flowchart of the microexpression recognition method based on motion amplification and hierarchical Transformer of the application;
[0039] Fig. 2 is a schematic diagram of the motion amplification module of the application;
[0040] Fig. 3 is a schematic diagram of the RAFT optical flow calculation module;
[0041] Fig. 4 is an effect diagram of microexpression samples after different amplification coefficients. DETAILED DESCRIPTION
[0042] The embodiments of the application will be described in detail below with specific specific examples. Those skilled in the art can easily understand other advantages and effects of the application from the content disclosed in the specification. The application can also be implemented or applied by different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the application. It should be noted that the following examples and features in the examples can be combined with each other without conflict.
[0043] Example 1
[0044] As shown in Figure 1 The microexpression recognition method based on motion amplification and hierarchical Transformer provided by the application is described below.
[0045] Figure 1 The architecture provided by the application is shown to be composed of three main components, which are motion amplification module, RAFT optical flow calculation module and hierarchical Transformer and block aggregation module.
[0046] 1. Motion amplification processing Firstly, the motion amplification module amplifies the motion signal of the microexpression by using the motion vector field of the microexpression. and the peak frame Motion magnification is performed at an appropriate level. This process ensures the visibility of micro facial movements. The implementation is as follows: This process is implemented through a deep learning model, which can be represented as:
[0047]
[0048] where, is the magnification factor, represents the motion magnification technique using deep learning used. represents the micro-expression peak frame after magnification. In this way, for a micro-expression, a series of can be obtained due to .
[0049] As shown in Figure 2 , the entire motion magnification module consists of three parts:
[0050] Encoder: As a spatial domain feature extractor, it accurately captures the subtle motion features between the starting frame and the peak frame through multi-scale convolution operations.
[0051] Manipulator: A parameterized magnification mechanism is used to perform matrix operations on the extracted motion features and learnable magnification coefficients, achieving nonlinear enhancement of motion amplitude.
[0052] Decoder: The magnified motion information is fused into the original video sequence through a feature reconstruction network.
[0053] 2. RAFT optical flow calculation
[0054] Secondly, the RAFT optical flow calculation module uses deep learning technology to extract the optical flow information between the starting frame and the magnified frame. This process is represented as:
[0055]
[0056] where, is the starting frame of the micro-expression, represents the magnified micro-expression peak frame, is the RAFT module.
[0057] As shown in Figure 3 , the RAFT algorithm is based on deep learning, and its specific implementation steps are as follows:
[0058] Feature extraction: The first step of RAFT is to extract features from the starting frame and the peak frame of the micro-expression. In this process, features are extracted from the magnified ). This process is implemented through a deep convolutional neural network, generating two feature maps and :
[0059]
[0060] Similarity calculation: Calculate the similarity between pixels, which can be expressed as:
[0061]
[0062] where, is a hyperparameter that controls the degree of similarity smoothing.
[0063] Iterative update of optical flow field: RAFT estimates optical flow by iteratively updating the optical flow field using GRU recurrent units. The update process of the optical flow field is represented by the following formula:
[0064]
[0065] where, is the increment calculated by the recurrent network. The model gradually reduces the error of optical flow estimation through multiple iterations.
[0066] Loss function: RAFT uses a suitable loss function to guide the training process, usually using the endpoint error of optical flow:
[0067]
[0068] where, is the true optical flow, is the optical flow predicted by the model, is the total number of pixels.
[0069] In this embodiment, the RAFT algorithm is based on deep learning, which estimates optical flow by iteratively updating the optical flow field using GRU recurrent units. This simulates the optimization process in traditional methods. The RAFT network consists of three main layers: feature encoding layer, feature correlation layer and recurrent update operator. The feature encoding layer extracts features pixel by pixel, while the feature correlation layer generates 4D correlation information for all pixels. The convergence layer generates lower resolution correlation information. The recurrent update operator uses 4D correlation information under the principle of GPU, starting from the initial zero setting, iteratively updating the optical flow field.
[0070] 3. Hierarchical Transformer and block aggregation, finally, the motion relationship of each part of the face when micro-expression occurs is encoded through a hierarchical Transformer network. Transformer introduces a self-attention mechanism to capture detailed details and features.
[0071] Block aggregation process: The attention module combines smaller image patches into larger blocks through a block aggregation process. The Block aggregation process includes several steps: first, a 3x3 convolutional layer, followed by layer normalization, and finally a 3x3 max-pooling operation.
[0072] Local and global feature extraction: In low-level block aggregation, the exchange of information within local facial regions is emphasized to extract complex features that capture subtle facial dynamics. The present invention employs local attention for individual image patches, thereby significantly improving performance. By employing local attention and focusing on specific facial regions such as the eye and lip regions, the model of the present invention effectively captures key features. At the same time, high-level block aggregation allows global information exchange, thereby extracting coarse-grained features that capture broader facial expressions. The present invention effectively enhances the model's attention to key facial regions, enabling better recognition of subtle changes in microexpressions.
[0073] 4. Classification output
[0074] Finally, the feature maps generated by the block aggregation layer will be input to an MLP layer for microexpression classification. The difference between the predicted value and the true value is measured by the cross-entropy loss function, guiding the model training process.
[0075] Selection of magnification factor
[0076] As shown in Figure 4 , as the motion magnification factor increases step by step, the motion displacement of each region of the face (especially the eye, lip, and nose wing regions, which are key muscle activity regions) increases proportionally. This processing helps to enhance the difference features between microexpressions and neutral expressions. However, when the magnification factor exceeds 5, the image starts to have structural distortion and texture abnormalities, such as blurred facial contours, lost details, and increased noise interference. These artifacts reduce the quality of motion representation, which in turn affects the accuracy of subsequent feature extraction and classification. Therefore, in practical applications, factors such as the original intensity of microexpressions, image resolution, and lighting conditions should be considered, and the optimal magnification factor should be determined through experiments. It is recommended to control the magnification factor within 5 times to enhance motion information while maximizing the naturalness of facial structure and the reliability of features.
[0077] The application applies a deep learning-based motion amplification technique to the micro-expression between the starting frame and the peak frame. Through a parameterized amplification mechanism, the motion amplitude of facial muscles is nonlinearly enhanced, making subtle changes that are otherwise difficult to detect become apparent. The preprocessing operation directly amplifies the signal-to-noise ratio of the "input signal" faced by the subsequent feature extraction network, enabling the convolutional neural network and the optical flow calculation to capture more discriminative dynamic features, fundamentally solving the feature extraction difficulty problem caused by the small amplitude of micro-expression movements. In view of the characteristics that micro-expression is highly dependent on local facial muscle movements, the application introduces a local attention mechanism in the hierarchical Transformer. This mechanism forces the model to preferentially focus on key action unit regions such as the eye and mouth in the shallow network stage, rather than equally processing the entire face. This design can effectively capture the most relevant local subtle changes of micro-expression, while ignoring irrelevant information from other areas of the face, thereby extracting higher-purity and more discriminative features, directly improving the accuracy of classification.
[0078] It is worth noting that the hierarchical Transformer combined with the block aggregation layer structure adopted by the application can automatically learn and fuse multi-scale features from local to global. Low-level features focus on capturing detailed, local facial dynamics (such as muscle tremors). High-level features integrate information over a larger range through block aggregation to capture coarse-grained expression patterns. The multi-scale feature fusion mechanism enhances the model's robustness to micro-expressions of different performance intensity and range, enabling it to more comprehensively understand the spatiotemporal evolution process of micro-expressions.
[0079] The application is verified on three major mainstream datasets (CASME II, SMIC, and SAMM) containing different races, ages, and collection environments. Experimental results show that the recognition performance of the present method is significantly better than that of previous technologies. It is proved that the features extracted by the present method have strong generalization ability and are not dependent on a specific group or collection condition, making it more suitable for complex application scenarios in the real world. The present application integrates motion amplification, optical flow calculation, and feature learning and classification into a coherent process. Each module has its own role and works closely together: the motion amplification module prepares high-quality data for subsequent steps, the RAFT module provides optimal motion representation, and the hierarchical Transformer and block aggregation are responsible for efficient feature learning and fusion. This modular end-to-end design not only has superior performance, but also provides a solid technical foundation for the practical deployment and application of micro-expression recognition technology.
[0080] The above specific embodiments are only several optional embodiments of the present application. Based on the technical solutions of the present application and the related inspiration of the above embodiments, those skilled in the art can make various alternative improvements and combinations to the above specific embodiments.
Claims
1. A micro-expression recognition method based on motion magnification and hierarchical Transformer, characterized in that, Includes the following steps: S1. Perform motion magnification processing on the starting frame and peak frame in the micro-expression video sequence to obtain the magnified peak frame; S2. Based on the RAFT optical flow estimation method, calculate the optical flow map between the starting frame and the magnified peak frame; S3. Input the optical flow map into a hierarchical Transformer network to extract the motion relationship features between different parts of the face; S4. The multi-scale features output by the hierarchical Transformer network are fused through a block aggregation layer to obtain an enhanced feature representation; S5. Based on the enhanced feature representation, perform micro-expression classification using a classifier.
2. The micro-expression recognition method based on motion magnification and hierarchical Transformer according to claim 1, characterized in that, The motion amplification processing in step S1 is implemented through a deep learning model, specifically as follows: ， in, This is the starting frame of micro-expressions. These are the peak frames of micro-expressions. It's the magnification factor. This indicates the motion amplification technology using deep learning. This represents the magnified peak frames of micro-expressions.
3. The micro-expression recognition method based on motion magnification and hierarchical Transformer according to claim 1, characterized in that, The RAFT optical flow estimation method in step S2 includes: The feature maps of the initial frame and the magnified peak frame were extracted using a convolutional neural network. Construct a feature association matrix and calculate the similarity between pixels; The optical flow field is iteratively updated based on gated cyclic units to gradually optimize the optical flow estimation results; Output the optical flow map between the starting frame and the magnified peak frame.
4. The micro-expression recognition method based on motion magnification and hierarchical Transformer according to claim 3, characterized in that, The RAFT method employs a multi-head self-attention mechanism during optical flow estimation to process multiple feature subspaces in parallel.
5. The micro-expression recognition method based on motion magnification and hierarchical Transformer according to claim 1, characterized in that, The hierarchical Transformer network in step S3 includes multiple layers, each layer containing: Local self-attention module, used to capture subtle movements in key areas of the face; Cross-block feature interaction mechanism is used to promote the fusion of information between local and global features.
6. The micro-expression recognition method based on motion magnification and hierarchical Transformer according to claim 1, characterized in that, The block aggregation layer in step S4 includes: Convolutional layers are used for feature transformation; Layer normalization is used to stabilize the training process; Max pooling layers are used to reduce dimensionality while preserving salient features.
7. The micro-expression recognition method based on motion magnification and hierarchical Transformer according to claim 1, characterized in that, The classifier used in step S5 is a multilayer perceptron, and the loss function is cross-entropy loss.
8. The micro-expression recognition method based on motion magnification and hierarchical Transformer according to claim 2, characterized in that, The motion magnification The value range is 1 to 10.