Micro-expression emotion recognition method and system based on multi-modal time sequence decoupling

Abstract

The invention provides a micro-expression emotion recognition method and system based on multi-modal time sequence decoupling, and the method comprises the steps: collecting and preprocessing the face video data, and obtaining a standardized video clip; extracting global time sequence features and geometric features by adopting a hierarchical emotion dynamic feature extraction framework; establishing a generative adversarial decoupling network, and separating the global time sequence features into emotion features and noise features; splicing the emotional features with the geometric features to obtain enhanced features; a dual-path network architecture is combined with the enhanced features, emotion classification and emotion feature regression are carried out respectively, and an emotion prediction probability and an emotion feature regression vector are obtained; and based on the enhanced features, the emotion prediction probability and the emotion feature regression vector, obtaining an emotion state score, and based on the emotion state score, obtaining a micro-expression emotion recognition result. According to the method, more comprehensive and more objective quantitative analysis on the emotional state is provided, and a richer decision basis is provided for emotional understanding and human-computer interaction.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, computer vision and human-computer interaction, and particularly relates to a micro-expression emotion recognition method and system based on multi-modal time sequence decoupling. BACKGROUND

[0002] With the rapid development of human-computer interaction and emotion computing technology, the demand for objective, efficient and non-contact analysis of individual emotional states is increasing. As an unconscious physiological response of emotional expression, facial micro-expression contains rich internal emotional information and shows important application value in the field of emotion state recognition. However, the existing micro-expression analysis technology based on artificial intelligence still faces significant challenges in practical application: insufficient transient feature capture: traditional CNN models rely on single-frame image analysis mechanism and are difficult to effectively capture transient micro-expression (such as subtle mouth corner tremor and eyelid twitch) with extremely short duration (<0.5 seconds), resulting in low recognition rate of key emotional clues; poor robustness in complex scenes: centralized time sequence modeling methods (such as 3D-CNN) fail to effectively decouple core emotional features and environmental noise interference when processing actual scenes (such as light changes, head posture changes and individual differences), resulting in a significant decrease in recognition accuracy compared to ideal environments, limiting the universality and landing application of the technology; single analysis dimension and lack of quantitative indicators: existing commercial systems (such as Affectiva) mainly output discrete basic emotion classification results (such as happy, sad, angry, etc.), lack of fine-grained quantitative description of emotional response dynamic process (such as response delay and intensity change rate), and cannot meet the demand for in-depth and multi-dimensional analysis of emotional state. SUMMARY

[0003] To solve the problems existing in the prior art, the application provides a micro-expression emotion recognition method and system based on multi-modal time sequence decoupling, which provides a robust, accurate and quantifiable micro-expression emotion state analysis technology solution for emotion computing, human-computer interaction, behavior analysis and other scenes.

[0004] To achieve the above-mentioned purpose, the application provides the following solutions: The micro-expression emotion recognition method based on multi-modal time sequence decoupling comprises: Collecting and preprocessing face video data, obtaining standardized video segments through sliding window segmentation and key frame extraction; Using a hierarchical emotion dynamic feature extraction framework to extract global time sequence features and geometric features of each standardized video segment; Establishing a generative adversarial decoupling network to separate the global time sequence features into emotional features and noise features; Concatenating the emotional features and the geometric features to obtain concatenated features, and using channel attention and spatial attention to enhance the concatenated features to obtain enhanced features; A dual-path network architecture is used in conjunction with the enhanced features to perform emotion classification and sentiment feature regression, respectively, to obtain the emotion prediction probability and the sentiment feature regression vector. Based on the enhanced features, the predicted emotion probability, and the emotion feature regression vector, an emotion state score is obtained, and based on the emotion state score, a micro-expression emotion recognition result is obtained.

[0005] Preferred methods for obtaining standardized video clips include: The MTCNN algorithm is used to locate faces in the standardized video clips to obtain facial regions; Affine transformation is used to normalize the pixels of the facial region and perform brightness equalization processing to obtain a facial region image frame with enhanced details. Based on the duration of micro-expressions, sliding window parameters are established, and the optical flow energy of facial region image frames within each sliding window is calculated. Facial region image frames that meet the preset energy peak are used as key frames to obtain the standardized video clip.

[0006] Preferred methods for obtaining global temporal features include: A lightweight 3D-CNN network is used to encode features for each standardized video segment to obtain local spatiotemporal features; Based on the aforementioned local spatiotemporal features, calculate the global contrast features; The global contrast features are concatenated with the local spatiotemporal features, and the weights of the corresponding standardized video segments are calculated by introducing adaptive sensitive regions. Standardized video segments that meet the preset weight thresholds are retained to obtain a set of keyframes. The fragment weights in the keyframe set and the local spatiotemporal features are input into the Transformer encoder to obtain the global temporal features.

[0007] Preferably, the geometric features include eye features and mouth features; wherein, the eye features include blinking frequency and eyelid opening change rate; and the mouth features include lip corner upturn and lip asymmetry.

[0008] Preferably, the generative adversarial decoupling network includes: A generator, comprising an encoder and a decoder, is used to separate the global temporal features into sentiment features and noise features; the encoder employs a three-layer fully connected network, and the decoder has two parallel paths that output sentiment features and noise features respectively. A discriminator is used to guide the generator to separate the pure sentiment features using a loss function; the loss function includes sentiment classification loss and domain adversarial loss. A gradient reversal layer is used to update the parameters of the generator and discriminator using a gradient reversal strategy; the gradient reversal strategy includes forward propagation and backward propagation.

[0009] Preferably, the dual-path network architecture includes: An emotion classification path is used to classify emotions based on the enhanced features and obtain different emotion prediction probabilities. The emotional feature regression path is used to calculate the facial expression reaction delay, the reduction ratio of muscle movement amplitude, and the social avoidance index based on the standardized fragments, and to obtain the emotional feature regression vector.

[0010] This invention also provides a micro-expression emotion recognition system based on multimodal temporal decoupling, for implementing the method, comprising: The video acquisition module is used to acquire and preprocess facial video data, and obtain standardized video segments through sliding window segmentation and keyframe extraction. The feature extraction module is used to extract global temporal features and geometric features of each standardized video segment using a hierarchical emotion dynamic feature extraction framework. The feature decoupling module is used to establish a generative adversarial decoupling network to separate the global temporal features into sentiment features and noise features. The feature enhancement module is used to concatenate the emotional features with the geometric features to obtain concatenated features, and to enhance the concatenated features using channel attention and spatial attention to obtain enhanced features; The feature parsing module is used to combine the enhanced features with a dual-path network architecture to perform emotion classification and sentiment feature regression, respectively, to obtain the emotion prediction probability and the sentiment feature regression vector. The emotion recognition module is used to obtain an emotion state score based on the enhanced features, the emotion prediction probability, and the emotion feature regression vector, and to obtain a micro-expression emotion recognition result based on the emotion state score.

[0011] Preferably, the video acquisition module includes: The face localization unit is used to perform face localization on the standardized video segment using the MTCNN algorithm to obtain the face region; The affine transformation unit is used to perform pixel normalization and brightness equalization processing on the facial region using affine transformation to obtain a facial region image frame with enhanced details. The keyframe acquisition unit is used to establish sliding window parameters based on the duration of micro-expressions, calculate the optical flow energy of facial region image frames within each sliding window, and use facial region image frames that meet the preset energy peak as keyframes to obtain the standardized video segment.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: A cascaded processing architecture is proposed, consisting of sliding window segmentation → local 3D-CNN feature extraction → ASE attention weighting → Transformer temporal modeling. Through a four-level collaborative mechanism (sliding window (0.5±0.1s) covering the complete life cycle of micro-expressions, 3D-CNN extracting spatiotemporal features around the eyes / corner of the mouth, ASE module generating heatmaps to locate key response regions (such as the left corner of the mouth movement area), and Transformer modeling long-range dependencies), the transient micro-expressions are captured at the millisecond level, while a visualized decision tracing chain is established.

[0013] A generative adversarial decoupling network is employed to separate sentiment features (V) through a gradient inversion strategy. emo ) and environmental noise characteristics (V noise This method significantly improves the feature signal-to-noise ratio (SNR). Validated on the FDDB dataset, this method reduces recognition errors by 68% in scenarios with varying lighting and head rotation.

[0014] A dual-path architecture was designed: Path 1 outputs the basic emotion classification, and Path 2 generates dynamic behavioral parameters (response latency τ, muscle movement amplitude attenuation ratio R, and emotional state score E). An attention fusion mechanism was used to address the problem of inconsistent emotional expression (such as forced smiles), resulting in a 19.3% improvement in F1-score on the CMU-MOSEI dataset. Attached Figure Description

[0015] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the micro-expression emotion recognition method based on multimodal temporal decoupling according to an embodiment of the present invention; Figure 2 This is a flowchart of the data preprocessing process according to an embodiment of the present invention; Figure 3 This is a flowchart of the ASE module in an embodiment of the present invention; Figure 4 This is a heatmap comparing the intensity of emotional responses in an embodiment of the present invention. Figure 5 This is a structural diagram of the Transformer according to an embodiment of the present invention; Figure 6 This is a flowchart illustrating the layered MIL workflow of an embodiment of the present invention; Figure 7 An adversarial decoupling network structure diagram is generated for embodiments of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] Example 1 like Figure 1 As shown, the micro-expression emotion recognition method based on multimodal temporal decoupling includes: S1: Acquire and preprocess facial video data, and obtain standardized video segments through sliding window segmentation and keyframe extraction. The input device used in this embodiment supports an RGB camera with a resolution of at least 480p and a frame rate of at least 30fps. A microphone is optional. For data alignment, timestamp synchronization technology is used to ensure that the time error between video frames and audio (if present) is controlled within 10ms. Specifically, this can be achieved by using Rt=|t. video -t audio The timeframe is ≤10ms. For example... Figure 2 As shown.

[0020] A further implementation method for obtaining standardized video clips includes: The MTCNN algorithm is used to locate faces in standardized video clips and obtain facial regions. Specifically, the MTCNN algorithm is used to detect faces and output bounding box coordinates (x, y, w, h).

[0021] Affine transformation is used to normalize the pixels of the facial region and perform brightness equalization to obtain a facial region image frame with enhanced details. Specifically, affine transformation is applied to normalize the facial region to 224×224 pixels to ensure that the attitude deviation (including pitch angle and yaw angle) does not exceed 15°.

[0022] Finally, brightness equalization is performed, using adaptive histogram equalization (CLAHE) to enhance facial details. The calculation formula is as follows: Enhanced=CLAHE (Iraw,clip_limit=2.0,tile_grid_size=(8,8)) (1) Where clip_limit is the contrast limit threshold, and tile_grid_size is the grid size for image segmentation.

[0023] Based on the duration of micro-expressions, sliding window parameters are established. The optical flow energy of facial region image frames within each sliding window is calculated. Frames meeting a preset energy peak are selected as keyframes to obtain standardized video segments. Specifically, the sliding window segmentation and keyframe extraction process sets the sliding window parameters based on the duration characteristics of micro-expressions (0.2-0.5 seconds), where the window length is Tw = 0.5 ± 0.1 s (i.e., 15-20 frames, the specific number of frames depends on the frame rate), and the step size is Ts = 0.1 s (i.e., 3-6 frames). The frame selection strategy is to calculate the optical flow energy Eflow for each frame within the window and select the frame with the energy peak as the keyframe. The formula for calculating the optical flow energy is: Eflow=∑ (x,y) u(x,y) 2 +v (x,y) 2 (2) Where u and v are the horizontal and vertical components of the optical flow field at (x,y).

[0024] S2: The hierarchical emotional dynamic feature extraction framework does not directly adopt existing network structures. Instead, it addresses the transient and low-intensity characteristics of micro-expressions by making several key structural improvements and co-designs to the basic network model, specifically as follows: Improvements and adaptation of lightweight 3D-CNN: Micro-expression analysis requires processing video segments. Traditional 3D-CNNs have a large number of parameters and high computational cost, making them difficult to run in real-time on edge devices. Simultaneously, the subtle muscle movements of micro-expressions require the network to be highly sensitive to spatiotemporal changes; Innovative design of the adaptive sensitive region (ASE) module: Micro-expressions do not appear uniformly across all facial regions. Traditional average pooling or max pooling introduces a large amount of noise from non-emotionally relevant regions. This invention aims to adaptively focus on key facial regions of emotional response (such as the zygomaticus major muscle and the glabella); Temporal modeling optimization of the Transformer encoder: Micro-expressions contain complete temporal dynamics of start, peak, and end, and their duration is short, requiring the model to accurately capture long-range dependencies between frames. Figure 5 As shown.

[0025] A further implementation method involves using geometric features including eye features and mouth features. Eye features include blink frequency and eyelid opening change rate (Reye); mouth features include lip corner lift and lip asymmetry. Specifically, dynamic geometric features related to emotional response are calculated based on 68 facial key points, including eye features and mouth features. Eye features include blink frequency (fblink) and eyelid opening change rate (Reye); mouth features include lip corner lift (Smile_score) and lip asymmetry (Asym_lip). The formula for lip corner lift is defined as: (3) in y upper_lip , y lower_lip Here are the y-coordinates of the center points of the upper and lower lips. d interocular The distance between the eyes.

[0026] A further implementation method for obtaining global temporal features includes: A lightweight 3D-CNN network is used to encode features for each standardized video segment, obtaining local spatiotemporal features. Specifically, a lightweight 3D-CNN (ShuffleNetV2 architecture) is used to encode features for each video segment. This network structure contains three spatiotemporal convolutional blocks, each consisting of a 3D convolution (3×3×3 kernel), BatchNorm, and ReLU activation. The final output feature vector is... f i =CNN3D(Si), where f i ∈R 512 To optimize computational complexity, depthwise separable convolutions are used to reduce the number of parameters, enabling the model to meet the deployment requirements of edge devices. For example... Figure 6 As shown.

[0027] Based on local spatiotemporal features, global contrast features are calculated. These global contrast features are then concatenated with the local spatiotemporal features, and an adaptive sensitive region (ASE) module is introduced to calculate the weights of corresponding standardized video segments. Standardized video segments that meet a preset weight threshold are retained to obtain a keyframe set. Specifically, an adaptive sensitive region (ASE) module is introduced to calculate segment weights w. i First, calculate the global contrast features: (4) Then obtain the splicing feature h i =[ f i ; g i ]∈R 1024 And then through Calculate the weights (where W∈R) 1024×1 After that, keyframes are filtered, and the weight w is retained. i Fragments with a length greater than 0.7 form a keyframe set F. k .

[0028] The fragment weights and local spatiotemporal features in the keyframe set are input into the Transformer encoder to obtain global temporal features.

[0029] like Figure 3 As shown, the heatmap generated by the ASE module maps feature weight intensity (deep red: w) through the RGB color gamut. i >0.8, Blue: w i <0.3), which can intuitively show the salience of emotional responses in facial regions. Typical patterns include: high weight (deep red) in the corners of the mouth area is often associated with negative emotional reinforcement, while low weight (blue) around the eyes often occurs in a state of emotional inhibition.

[0030] like Figure 4 As shown, under the enhanced emotional state, the corner of the mouth region exhibits a consistently high-weighted response (>0.8, dark red), while under the neutral state, the response intensity in the same region significantly decreases (<0.4, blue). Algorithm validation (n=500) shows that the difference between high and low response states is significant (t=18.24, p<0.001, Cohen's d=1.52), and the correlation coefficient between response intensity and expert sentiment annotation reaches r=0.82±0.03.

[0031] Transformer temporal modeling process. Weighted features... Input a Transformer encoder, where the position encoding is: (5) In the formula, This represents the position number of the segment in the input sequence, and i represents the dimension index in the position encoding vector.

[0032] The formula for calculating multi-head attention is: , (6) Q, K, and V represent the query, key, and value matrix, respectively. , , This represents the trainable projective weight matrix corresponding to the i-th attention head. This represents the trainable linear projection weight matrix of the multi-head attention output layer.

[0033] The final output is the global time-series vector Vt = MeanPooling(Transformer ( )).

[0034] S3: Establish a generative adversarial decoupling network to separate global temporal features into sentiment features and noise features; for example... Figure 7 As shown. A further implementation method involves generating the adversarial decoupling network, which includes: The generator, consisting of an encoder and a decoder, separates global temporal features into sentiment features and noise features. The encoder employs a three-layer fully connected network with dimensions of 512→256→128 respectively. The decoder has two parallel paths, each outputting a sentiment feature V. emo With noise characteristics V noise .

[0035] The discriminator, using a loss function, guides the generator to separate pure sentiment features. The loss function includes sentiment classification loss and domain adversarial loss. Training the discriminator D involves two types of losses: one is the sentiment classification loss, formulated as follows: (7) In the formula, c represents the sentiment category index. This represents the actual label (one-hot encoded form) corresponding to this category.

[0036] Second, the domain adversarial loss, the formula is: (8) In the formula, D represents the discriminator network. This represents the mathematical expectation of the calculated sentiment feature vector. This represents the mathematical expectation of the noise eigenvector calculation.

[0037] The gradient reversal layer is used to update the parameters of the generator and discriminator using a gradient reversal strategy; the gradient reversal strategy includes forward propagation and backward propagation.

[0038] A gradient reversal strategy is used during gradient update, namely: Forward propagation: ; Backpropagation: (9) S4: Concatenate the sentiment features with the geometric features to obtain the concatenated features, and then enhance the concatenated features using channel attention and spatial attention to obtain the enhanced features; specifically, first concatenate the sentiment features with the geometric features to obtain V. fusion =[V emo Asym;f blink Subsequently, a Feature Enhancement Module (FEM) is applied, which includes channel attention and spatial attention. The formula for calculating channel attention is as follows: (10) in, Represents the Sigmoid activation function. , This represents the trainable weight matrix of the fully connected layer. , This represents the corresponding trainable bias vector.

[0039] The formula for calculating spatial attention is: (11) in, The kernel size represents the convolution kernel size. convolutional layers, This represents max pooling along the channel dimension. This represents average pooling along the channel dimension.

[0040] The final enhanced features are .

[0041] S5: A dual-path network architecture is adopted, combined with enhanced features, to perform emotion classification and sentiment feature regression respectively, and obtain the emotion prediction probability and sentiment feature regression vector.

[0042] A further implementation method involves a dual-path network architecture comprising: The emotion classification path is used to classify emotions based on enhanced features and obtain the prediction probabilities of different emotions. Specifically, its network structure contains two fully connected layers with dimensions of 514→256→6 respectively. The activation function is Softmax, and the final output is Pe=[Pangry,Phappy,...,Pneutral], where each element corresponds to the prediction probability of different basic emotions (such as anger, happiness, neutrality, etc.).

[0043] The emotion feature regression path is used to calculate the facial expression reaction delay, muscle movement amplitude attenuation ratio, and social avoidance index based on standardized segments, thereby obtaining the emotion feature regression vector. Specifically, it mainly calculates the facial expression reaction delay τ, and the specific steps are as follows: First, the stimulus initiation frame is detected based on the optical flow abrupt change point, satisfying the condition REflow > 3σ; then, the peak muscle response frame is determined, which must simultaneously satisfy: The first derivative is zero: d(zygomaticus) / dt = 0; The second derivative is negative: d 2 (zygomaticus) / dt 2 < 0; Amplitude exceeding 3 times the standard deviation of the baseline: A_peak > μ_baseline + 3σ (where μ_baseline and σ are the mean and standard deviation of the zygomaticus major muscle displacement at rest).

[0044] Finally, the time delay τ is calculated using the formula τ = t_peak - t_stimulus, where t_peak is the timestamp of the peak frame of the muscle response, and t_stimulus is the timestamp of the stimulus initiation frame.

[0045] The continuous behavior quantization path serves as path 2 in the dual-path network architecture. It is primarily used for emotional dynamic feature regression, specifically including facial expression reaction delay τ, muscle movement amplitude attenuation ratio R, and social avoidance index SAI. The facial expression reaction delay τ is obtained through MLP regression, using the Huber loss as the loss function; the formula for calculating the muscle movement amplitude attenuation ratio R is: R=100%×(1-A_current / A_baseline) (12) Note: R represents the instantaneous relative change in muscle movement amplitude between the resting baseline state and the current facial expression event. Its calculation does not rely on long-term physiological data and only reflects the muscle movement intensity characteristics of a single facial expression.

[0046] The Social Avoidance Index (SAI) is calculated based on blink frequency and fixation duration, using the following formula: SAI=λ1·(f_blink_current / f_blink_baseline-1) +λ2·(1-T_gaze_current / T_gaze_baseline) (13) Where f_blink_current represents the blink frequency within the current time window; f_blink_baseline represents the baseline blink frequency obtained from neutral facial expression segments of the subject; T_gaze_current represents the average single gaze duration within the current time window; T_gaze_baseline represents the baseline average single gaze duration obtained from neutral facial expression segments of the subject; λ1 and λ2 are weighting coefficients used to balance the contributions of blinking and gaze behavior to the avoidance index, and were set to λ1=0.6 and λ2=0.4 through experiments. SAI > 0 indicates a tendency towards social avoidance (such as more frequent blinking and shorter gaze duration), and the larger the value, the stronger the avoidance tendency.

[0047] The output of the emotion feature regression path is an emotion feature regression vector [τ, R, SAI] composed of facial expression reaction delay τ, muscle movement amplitude attenuation ratio R, and social avoidance index SAI. This vector is then fed into the emotion recognition module for the final emotion state score calculation.

[0048] S6: Based on enhanced features, emotion prediction probability, and emotion feature regression vector, obtain emotion state score, and based on the emotion state score, obtain micro-expression emotion recognition result.

[0049] Specifically, feature fusion is first performed to obtain: Next, the emotional state score is calculated using the following formula: (14) W4 represents the learnable weight matrix used to perform a linear transformation on the enhanced feature Venhanced, which maps the deep visual features to the fusion space. W5 represents the learnable weight matrix used to perform a linear transformation on the concatenated feature Concat(P_e, τ, R, SAI) (i.e., emotion prediction probability, facial expression reaction delay, muscle movement amplitude attenuation ratio, and social avoidance index), which maps multi-dimensional behavioral quantification indicators and classification probabilities to the same fusion space.

[0050] Finally, the emotional state score E is used to describe the state: low arousal / neutral (E<0.3), moderate arousal / specific emotional tendency (0.3≤E<0.7), and high arousal / strong emotional tendency (E≥0.7). In this process, the Social Avoidance Index (SAI) is a key quantitative indicator that directly participates in the score calculation. When the SAI value is high, it has a negative moderating effect on the emotional state score E, making the final identification result more inclined towards negative emotional states such as anxiety, tension, or avoidance, thus providing a more comprehensive and refined interpretation of the emotional state.

[0051] Uncertainty quantification process. Monte Carlo Dropout is used to estimate prediction uncertainty. Specifically, the forward propagation is repeated T times (e.g., T=20) to obtain the prediction distribution {S1, S2, ..., ST}; then the variance is calculated using the formula: (15) Finally, the confidence interval was determined to be... ±1.96Var (S), where This is the average of multiple prediction results.

[0052] This embodiment also includes edge deployment and performance optimization. The purpose of this section is to ensure that the system operates efficiently in resource-constrained environments.

[0053] The model compression and quantization stage primarily utilizes two methods to achieve model lightweighting: weight quantization and knowledge distillation. Weight quantization compresses 32-bit floating-point numbers into 8-bit integers, significantly reducing model storage requirements. Knowledge distillation uses the complete model as the teacher model to train a lightweight student model that inherits key knowledge. After processing, the number of model parameters is reduced from 5.2M to 1.8M, and the computational load is reduced by 60%, improving efficiency while maintaining performance. Hyperparameters such as initial learning rate, batch size, and number of iterations are set, and an optimizer suitable for multi-task learning is selected. Early stopping strategies and regularization methods are applied to reduce overfitting risk and improve model generalization ability. The preprocessed training set is input into the model, and network parameters are optimized through backpropagation. The training effect is monitored using the validation set, and hyperparameters are dynamically adjusted. The collaborative consistency of the dual-path output is optimized as a key focus.

[0054] Edge computing deployment: An NVIDIA Jetson Nano (4GB RAM) was selected as the hardware platform, which meets the operational requirements of lightweight models. To further optimize inference speed, TensorRT was used for acceleration, ultimately achieving an end-to-end latency of ≤200ms and power consumption of <10W, making it suitable for stable operation in resource-constrained edge scenarios (such as portable detection devices). See Table 1.

[0055] Table 1

[0056] The user interface design emphasizes real-time performance and readability. Real-time visualization functionality comprises three core modules: 1. Dynamic display of facial key points: achieving 30fps real-time rendering, with OpenGL acceleration ensuring display latency <100ms. The system includes three main features: 1. **Key Facial Action Units (AUs) Related to Sentiment Analysis:** Key areas such as frown lines (AU4) and downturned corners of the mouth (AU15) are highlighted. Abnormal micro-expressions lasting longer than 0.5 seconds trigger a flashing red warning box. 2. **Dynamic Curves Displayed on the Behavioral Dynamic Time-Sequence Monitoring Dashboard:** Reaction delay τ (0-1000ms range), muscle movement amplitude attenuation ratio R (0-100% range), emotional state score E (0-1.0 range), and multi-timescale switching (real-time mode, trend mode). 3. **Simultaneous Generation of PDF Reports:** Reports include keyframe screenshots, detailed explanations of risk levels, and targeted intervention suggestions, providing quantitative data for system optimization and user feedback. Specifically, the test set is input into the trained model to evaluate the consistency between emotional state recognition accuracy and behavioral quantitative indicators. The optimized model is deployed on edge devices to achieve localized processing of raw video and encrypted feature output, generating a visual report containing keyframe sequences, behavioral pattern analysis, and interaction optimization suggestions, providing real-time feedback for the human-computer interaction system.

[0057] In summary, this invention proposes an innovative micro-expression emotion recognition method. This method ingeniously integrates Hierarchical Multi-Instance Learning (HIL) with a cross-modal feature decoupling network. Through three-dimensional innovation of algorithm, data, and application, it achieves a breakthrough in emotion analysis technology in complex scenarios, significantly improving the model's recognition ability, generalization ability, accuracy, and robustness in complex scenarios. This method pioneers a hierarchical MIL framework (sliding window segmentation → local 3D-CNN → interpretable attention → Transformer temporal coding), solving the problem of low detection rate of transient micro-expressions in traditional single-frame CNNs. Through a generative adversarial decoupling network with a gradient inversion strategy, it can separate emotion-related features from scene noise features in facial dynamics, maintaining good performance even in low-quality videos. The dual-path recognition architecture can simultaneously output basic emotion classification and indicators such as facial expression response delay and muscle movement amplitude attenuation ratio, providing objective quantitative evidence for quantifying emotional states. In terms of application value, this method supports local deployment on edge devices with an end-to-end latency of ≤200ms, and the original video data does not leave the device, complying with international data privacy standards such as GDPR / CCPA. The visualization report generated by the analysis system provides key micro-expression timestamps and behavioral dynamic parameter curves (such as response delay τ, rate of change of motion R, and emotional state score E), assisting in the analysis of emotional response intensity. Compared with traditional manual evaluation methods (such as behavioral observation records), it can improve analysis efficiency by 8.3 times, making it suitable for large-scale group behavior analysis scenarios.

[0058] In terms of future prospects, this technical framework can be horizontally transferred to scenarios such as stress monitoring (e.g., through glabellar line frequency analysis) and fatigue early warning (eyelid tremor), requiring only adjustments to the supervision labels of the feature decoupling network; vertically, by combining wearable eye movement / heart rate data, a multimodal behavior perception system can be built to further improve screening accuracy; it can also be integrated with smart terminal devices (such as in-vehicle systems / AR glasses) to achieve seamless emotional interaction, demonstrating broad practical application potential. In summary, this invention provides an efficient and accurate solution for micro-expression analysis through a deep learning architecture that integrates video preprocessing, preliminary feature extraction strategies, and multi-dimensional feature fusion. It meets the real-time analysis needs of industrial applications and has broad application prospects in fields such as smart cockpits, virtual interaction, and safety education.

[0059] Example 2 This invention also provides a micro-expression emotion recognition system based on multimodal temporal decoupling, and a method for implementing this system, including: The video acquisition module is used to acquire and preprocess facial video data, and obtain standardized video segments through sliding window segmentation and keyframe extraction. The feature extraction module is used to extract global temporal features and geometric features of each standardized video segment using a hierarchical emotion dynamic feature extraction framework. The feature decoupling module is used to build a generative adversarial decoupling network to separate global temporal features into sentiment features and noise features; The feature enhancement module is used to concatenate emotional features with geometric features to obtain concatenated features, and then enhance the concatenated features using channel attention and spatial attention to obtain enhanced features. The feature parsing module is used to perform emotion classification and sentiment feature regression respectively by adopting a dual-path network architecture and combining enhanced features to obtain the emotion prediction probability and sentiment feature regression vector. The emotion recognition module is used to obtain an emotion state score based on enhanced features, emotion prediction probability, and emotion feature regression vector, and to obtain micro-expression emotion recognition results based on the emotion state score.

[0060] A further implementation method includes a video capture module comprising: The face localization unit is used to locate faces in standardized video clips using the MTCNN algorithm to obtain facial regions. The affine transformation unit is used to perform pixel normalization and brightness equalization on the facial region using affine transformation to obtain a facial region image frame with enhanced details. The keyframe acquisition unit is used to establish sliding window parameters based on the duration of micro-expressions, calculate the optical flow energy of facial region image frames within each sliding window, and use facial region image frames that meet the preset energy peak as keyframes to obtain standardized video clips.

[0061] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A micro-expression emotion recognition method based on multimodal temporal decoupling, characterized in that, include: Collect and preprocess facial video data, and obtain standardized video segments through sliding window segmentation and keyframe extraction; A hierarchical emotion dynamic feature extraction framework is adopted to extract the global temporal features and geometric features of each standardized video segment; A generative adversarial decoupling network is established to separate the global temporal features into sentiment features and noise features; The emotional features are concatenated with the geometric features to obtain concatenated features, and channel attention and spatial attention are used to enhance the concatenated features to obtain enhanced features; A dual-path network architecture is used in conjunction with the enhanced features to perform emotion classification and sentiment feature regression, respectively, to obtain the emotion prediction probability and the sentiment feature regression vector. Based on the enhanced features, the predicted emotion probability, and the emotion feature regression vector, an emotion state score is obtained, and based on the emotion state score, a micro-expression emotion recognition result is obtained.

2. The method according to claim 1, characterized in that, Methods for obtaining standardized video clips include: The MTCNN algorithm is used to locate faces in the standardized video clips to obtain facial regions; Affine transformation is used to normalize the pixels of the facial region and perform brightness equalization processing to obtain a facial region image frame with enhanced details. Based on the duration of micro-expressions, sliding window parameters are established, and the optical flow energy of facial region image frames within each sliding window is calculated. Facial region image frames that meet the preset energy peak are used as key frames to obtain the standardized video clip.

3. The method according to claim 1, characterized in that, Methods for obtaining global temporal features include: A lightweight 3D-CNN network is used to encode features for each standardized video segment to obtain local spatiotemporal features; Based on the aforementioned local spatiotemporal features, calculate the global contrast features; The global contrast features are concatenated with the local spatiotemporal features, and the weights of the corresponding standardized video segments are calculated by introducing adaptive sensitive regions. Standardized video segments that meet the preset weight thresholds are retained to obtain a set of keyframes. The fragment weights in the keyframe set and the local spatiotemporal features are input into the Transformer encoder to obtain the global temporal features.

4. The method according to claim 1, characterized in that, The geometric features include eye features and mouth features; wherein, the eye features include blinking frequency and eyelid opening change rate; the mouth features include lip corner upturn and lip asymmetry.

5. The method according to claim 1, characterized in that, The generative adversarial decoupling network includes: A generator, comprising an encoder and a decoder, is used to separate the global temporal features into sentiment features and noise features; the encoder employs a three-layer fully connected network, and the decoder has two parallel paths that output sentiment features and noise features respectively. A discriminator is used to guide the generator to separate the pure sentiment features using a loss function; the loss function includes sentiment classification loss and domain adversarial loss. A gradient reversal layer is used to update the parameters of the generator and discriminator using a gradient reversal strategy; the gradient reversal strategy includes forward propagation and backward propagation.

6. The method according to claim 1, characterized in that, The dual-path network architecture includes: An emotion classification path is used to classify emotions based on the enhanced features and obtain different emotion prediction probabilities. The emotional feature regression path is used to calculate the facial expression reaction delay, muscle movement amplitude attenuation ratio, and social avoidance index based on the standardized fragment, and outputs an emotional feature regression vector composed of the facial expression reaction delay, muscle movement amplitude attenuation ratio, and social avoidance index.

7. A micro-expression emotion recognition system based on multimodal temporal decoupling, used to implement the method described in any one of claims 1-6, characterized in that, include: The video acquisition module is used to acquire and preprocess facial video data, and obtain standardized video segments through sliding window segmentation and keyframe extraction. The feature extraction module is used to extract global temporal features and geometric features of each standardized video segment using a hierarchical emotion dynamic feature extraction framework. The feature decoupling module is used to establish a generative adversarial decoupling network to separate the global temporal features into sentiment features and noise features. The feature enhancement module is used to concatenate the emotional features with the geometric features to obtain concatenated features, and to enhance the concatenated features using channel attention and spatial attention to obtain enhanced features; The feature parsing module is used to combine the enhanced features with a dual-path network architecture to perform emotion classification and sentiment feature regression, respectively, to obtain the emotion prediction probability and the sentiment feature regression vector. The emotion recognition module is used to obtain an emotion state score based on the enhanced features, the emotion prediction probability, and the emotion feature regression vector, and to obtain a micro-expression emotion recognition result based on the emotion state score.

8. The system according to claim 7, characterized in that, The video acquisition module includes: The face localization unit is used to perform face localization on the standardized video segment using the MTCNN algorithm to obtain the face region; The affine transformation unit is used to perform pixel normalization and brightness equalization processing on the facial region using affine transformation to obtain a facial region image frame with enhanced details. The keyframe acquisition unit is used to establish sliding window parameters based on the duration of micro-expressions, calculate the optical flow energy of facial region image frames within each sliding window, and use facial region image frames that meet the preset energy peak as keyframes to obtain the standardized video segment.

Images