Micro-expression real-time emotion judgment method and system

Abstract

The present application relates to the field of computer vision and affective computing, and specifically provides a micro-expression real-time emotion judgment method and device, which has the following steps: S1, multi-mode data acquisition; S2, micro-expression feature extraction; S3, multi-modal fusion optimization; S4, real-time emotion tracking; S5, edge-cloud collaborative judgment.Compared with the prior art, the present application can dynamically compensate the algorithm to effectively improve the micro-expression detection rate, the edge computing greatly reduces the response time, the feature compression algorithm saves the bandwidth, and the present application supports the identification of multiple basic emotions and composite emotions, and the adaptive adjustment mechanism switches the analysis mode in different scenes such as medical treatment and security.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision and emotion computing, specifically providing a method and system for real-time emotion assessment of micro-expressions. Background Technology

[0002] Micro-expression analysis technology: Traditional micro-expression analysis mainly relies on high frame rate cameras (such as 1000fps or higher) to capture subtle facial changes, combined with hand-designed features (such as LBP-TOP, HOG) for recognition. Commonly used micro-expression coding systems (FACS) decompose facial muscle movements into 44 "action units" (AUs), and determine emotions by analyzing the combinations of AUs.

[0003] Multimodal fusion technology refers to the technique of integrating information from different sensors or data sources (such as vision, speech, text, physiological signals, etc.) and using machine learning or deep learning models for joint analysis to improve the accuracy and robustness of the system in recognition, classification, or decision-making tasks. In the field of micro-expression recognition, no research has yet been found that integrates facial movements, limbs, and gestures.

[0004] Traditional high frame rate cameras are susceptible to lighting interference, and handcrafted features (such as LBP-TOP) are insufficient for extracting instantaneous micro-expressions (0.04-0.5 seconds), resulting in low detection rates and high false positive rates. Single-modal emotion recognition (such as pure visual analysis) is easily affected by environmental interference and struggles to capture the correlation between emotions and physiological responses. Furthermore, multimodal fusion is inefficient, with significant differences in sampling frequencies between different modalities. Sliding window alignment introduces errors, making it unable to adapt to individual differences. This leads to high computational complexity, making real-time processing difficult and hindering the deployment of edge devices. Summary of the Invention

[0005] This invention addresses the shortcomings of the prior art by providing a highly practical method for real-time micro-expression emotion assessment.

[0006] A further technical objective of this invention is to provide a reasonably designed, safe, and applicable real-time micro-expression emotion analysis system.

[0007] The technical solution adopted by this invention to solve its technical problem is:

[0008] A method for real-time micro-expression emotion assessment includes the following steps:

[0009] S1, Multi-mode data acquisition;

[0010] S2. Micro-expression feature extraction;

[0011] S3, Multimodal fusion optimization;

[0012] S4, Real-time Emotion Tracking;

[0013] S5, edge-cloud collaborative analysis.

[0014] Furthermore, in step S1, an infrared camera array is used to collect facial micro-expression sequences, and a millimeter-wave radar is integrated to capture micro-motion tremor features. A wrist-worn PPG or GSR sensor is used to collect heart rate variability physiological signals.

[0015] Furthermore, step S2 includes:

[0016] S2-1. Image preprocessing: an inter-frame optical flow compensation algorithm is used, and an improved LBP-TOP algorithm is used to extract spatiotemporal texture features.

[0017] S2-2. The backbone network adopts MobileNetV3+SE attention module to realize real-time processing at the edge and introduces spatiotemporal Transformer encoder to establish long-range micro-expression dynamic association.

[0018] S2-3. Construct a facial muscle motion vector field using millimeter-wave radar data and fuse it with visual features at an early stage to improve the detection rate of micro-expressions.

[0019] Furthermore, step S3 includes:

[0020] S3-1, Data-level fusion: Establish a unified spatiotemporal coordinate system and align the time axis of visual or physiological data;

[0021] S3-2, Feature-level fusion: In the visual stream, 3D-ResNet18 is used to extract the spatiotemporal features of micro-expressions, and in the physiological stream, WaveNet is used to analyze the video characteristics of PPG signals. Weights are automatically learned through a cross-modal attention mechanism.

[0022] Furthermore, in step S4, a lightweight DeepSORT algorithm is used to predict the continuously matched emotion trajectories using Kalma filtering. Unmatched segments trigger a large cloud model for verification. When the user is detected wearing glasses or a mask, the weight of the gesture modality is automatically increased.

[0023] Furthermore, in step S5, intelligent task allocation is implemented, a lightweight detection model is run at the edge, and a multimodal Transformer large model is deployed in the cloud to support fine-grained emotion classification. A dynamic unloading strategy is implemented, and real-time evaluation is based on network QoS. Execution is automatically selected to be performed locally or in the cloud. Differential compression technology is used for key frame transmission. A continuous learning system is established to update the edge model through federated learning.

[0024] A real-time micro-expression emotion assessment system firstly acquires multi-modal data, including collecting facial micro-expression sequences from an infrared camera array, simultaneously integrating millimeter-wave radar to capture micro-movement tremor features, and configuring a wrist-worn PPG or GSR sensor to collect heart rate variability physiological signals.

[0025] Then, micro-expression features are extracted, multimodal fusion optimization is performed, and finally, real-time emotion tracking and edge-cloud collaborative analysis are conducted.

[0026] Furthermore, micro-expression feature extraction includes:

[0027] (1) Image preprocessing: inter-frame optical flow compensation algorithm is used, and spatiotemporal texture features are extracted using the improved LBP-TOP algorithm;

[0028] (2) The backbone network adopts MobileNetV3+SE attention module to realize real-time processing at the edge, and introduces spatiotemporal Transformer encoder to establish long-range micro-expression dynamic association.

[0029] (3) Construct a facial muscle motion vector field using millimeter-wave radar data and fuse it with visual features in the early stage to improve the detection rate of micro-expressions.

[0030] Furthermore, multimodal fusion optimization includes:

[0031] (1) Data-level fusion to establish a unified spatiotemporal coordinate system and align the time axis of visual or physiological data;

[0032] (2) Feature-level fusion: In the visual stream, the spatiotemporal features of micro-expressions are extracted by 3D-ResNet18, and in the physiological stream, the characteristics of PPG signal video are analyzed by WaveNet. Weights are automatically learned through cross-modal attention mechanism.

[0033] Furthermore, during real-time emotion tracking, a lightweight DeepSORT algorithm is used to predict continuously matching emotion trajectories using Kalma filtering. Unmatched segments trigger a large cloud model for verification. When the system detects that a user is wearing glasses or a mask, the weight of the gesture modality is automatically increased.

[0034] When conducting edge-cloud collaborative analysis, intelligent task allocation is implemented. A lightweight detection model runs on the edge, while a large multimodal Transformer model is deployed in the cloud to support fine-grained emotion classification. A dynamic offloading strategy is employed, and real-time evaluation is based on network QoS. Execution is automatically selected to be local or cloud-based, and differential compression technology is used for key frame transmission. A continuous learning system is established to update the edge model through federated learning.

[0035] Compared with existing technologies, the real-time micro-expression emotion judgment method and system of the present invention have the following outstanding advantages:

[0036] In this invention, multimodal fusion reduces the false positive rate. Compared with single-modal benchmark tests, the dynamic compensation algorithm effectively improves the micro-expression detection rate. Edge computing greatly reduces the response time, and the feature compression algorithm saves bandwidth. It also supports the recognition of multiple basic emotions and complex emotions, and the adaptive adjustment mechanism switches the analysis mode in different scenarios such as medical care and security. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating a real-time micro-expression emotion assessment method. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] The following is a preferred embodiment:

[0041] like Figure 1 As shown, the real-time micro-expression emotion judgment method in this embodiment has the following steps:

[0042] S1, Multi-mode data acquisition;

[0043] It employs a 1000fps high-speed infrared camera array to capture facial micro-expression sequences, and uses near-infrared imaging to solve the problem of ambient light interference; it also integrates millimeter-wave radar to capture micro-motion tremor characteristics (sampling rate 200Hz), and is equipped with a wrist-worn PPG / GSR sensor to collect physiological signals such as heart rate variability.

[0044] S2. Micro-expression feature extraction;

[0045] include:

[0046] S2-1. Image preprocessing: Inter-frame optical flow compensation algorithm is used to eliminate image blur caused by high-speed motion; improved LBP-TOP algorithm (fusion gradient direction information) is used to extract spatiotemporal texture features.

[0047] S2-2, Lightweight Detection Network: The backbone network adopts MobileNetV3+SE attention module to achieve real-time processing at the edge; a spatiotemporal Transformer encoder is introduced to establish long-range micro-expression dynamic association (processing 128 frames of temporal window). ③ Motion Feature Enhancement: Facial muscle motion vector field is constructed through millimeter-wave radar data; early fusion with visual features is performed to improve the micro-expression detection rate.

[0048] S3, Multimodal fusion optimization;

[0049] include:

[0050] S3-1, Data-level Fusion: Establish a unified spatiotemporal coordinate system and align the time axis of visual / physiological data.

[0051] S3-2, Feature-level fusion: In the visual stream, 3D-ResNet18 is used to extract the spatiotemporal features of micro-expressions, and in the physiological stream, WaveNet is used to analyze the video characteristics of PPG signals. Weights are automatically learned through a cross-modal attention mechanism.

[0052] S4, Real-time Emotion Tracking;

[0053] The multi-target tracking structure employs a lightweight DeepSORT algorithm (cosine distance + Mahalanobis distance joint matching). Kalma filtering is used to predict continuously matched emotion trajectories, and unmatched segments trigger a large cloud model for verification. When the system detects that the user is wearing glasses / masks, the weight of the gesture modality is automatically increased.

[0054] S5, edge-cloud collaborative analysis;

[0055] Intelligent task allocation: a lightweight detection model runs at the edge, while a large multimodal Transformer model is deployed in the cloud to support fine-grained emotion classification; dynamic offloading strategy: based on real-time network QoS evaluation (bandwidth / compression ratio), it automatically selects local or cloud execution, and differential compression technology is used for key frame transmission; a continuous learning system is established to update the edge model through federated learning and protect privacy data.

[0056] Based on the above method, the micro-expression real-time emotion judgment system in this embodiment firstly acquires multi-modal data, acquiring facial micro-expression sequences from an infrared camera array, and simultaneously integrating millimeter-wave radar to capture micro-movement tremor features, and configuring a wrist-worn PPG or GSR sensor to acquire heart rate variability physiological signals.

[0057] Then, micro-expression features are extracted, multimodal fusion optimization is performed, and finally, real-time emotion tracking and edge-cloud collaborative analysis are conducted.

[0058] Micro-expression feature extraction includes:

[0059] (1) Image preprocessing: inter-frame optical flow compensation algorithm is used, and spatiotemporal texture features are extracted using the improved LBP-TOP algorithm;

[0060] (2) The backbone network adopts MobileNetV3+SE attention module to realize real-time processing at the edge, and introduces spatiotemporal Transformer encoder to establish long-range micro-expression dynamic association.

[0061] (3) Construct a facial muscle motion vector field using millimeter-wave radar data and fuse it with visual features in the early stage to improve the detection rate of micro-expressions.

[0062] Multimodal fusion optimization includes:

[0063] (1) Data-level fusion to establish a unified spatiotemporal coordinate system and align the time axis of visual or physiological data;

[0064] (2) Feature-level fusion: In the visual stream, the spatiotemporal features of micro-expressions are extracted by 3D-ResNet18, and in the physiological stream, the characteristics of PPG signal video are analyzed by WaveNet. Weights are automatically learned through cross-modal attention mechanism.

[0065] In real-time emotion tracking, a lightweight DeepSORT algorithm is used. Kalma filtering is used to predict continuously matching emotion trajectories. Unmatched segments trigger a large cloud model for verification. When the user is detected wearing glasses or a mask, the weight of the gesture modality is automatically increased.

[0066] When conducting edge-cloud collaborative analysis, intelligent task allocation is implemented. A lightweight detection model runs on the edge, while a large multimodal Transformer model is deployed in the cloud to support fine-grained emotion classification. A dynamic offloading strategy is employed, and real-time evaluation is based on network QoS. Execution is automatically selected to be local or cloud-based, and differential compression technology is used for key frame transmission. A continuous learning system is established to update the edge model through federated learning.

[0067] The above-described specific embodiments are merely specific examples of the present invention. The patent protection scope of the present invention includes, but is not limited to, the above-described specific embodiments. Any technical solution that conforms to the above-described specific embodiments of the present invention and any appropriate changes or substitutions made by those skilled in the art should fall within the patent protection scope of the present invention.

[0068] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for real-time micro-expression emotion assessment, characterized in that, It has the following steps: S1, Multi-mode data acquisition; An infrared camera array collects facial micro-expression sequences, and a millimeter-wave radar is integrated to capture micro-motion tremor features. A wrist-worn PPG or GSR sensor is configured to collect heart rate variability physiological signals. S2. Micro-expression feature extraction; S3, Multimodal fusion optimization; include: S3-1, Data-level fusion: Establish a unified spatiotemporal coordinate system and align the time axis of visual or physiological data; S3-2, Feature-level fusion: In the visual stream, 3D-ResNet18 is used to extract the spatiotemporal features of micro-expressions, and in the physiological stream, WaveNet is used to analyze the video characteristics of PPG signals. Weights are automatically learned through a cross-modal attention mechanism. S4, Real-time Emotion Tracking; S5, edge-cloud collaborative analysis; Intelligent task allocation, lightweight detection model running at the edge, and multimodal Transformer large model deployed in the cloud to support fine-grained emotion classification, dynamic offloading strategy, real-time evaluation based on network QoS, automatic selection of local or cloud execution, differential compression technology for key frame transmission; and the establishment of a continuous learning system to update the edge model through federated learning.

2. The method for real-time micro-expression emotion assessment according to claim 1, characterized in that, Step S2 includes: S2-1. Image preprocessing: an inter-frame optical flow compensation algorithm is used, and an improved LBP-TOP algorithm is used to extract spatiotemporal texture features. S2-2. The backbone network adopts MobileNetV3+SE attention module to realize real-time processing at the edge and introduces spatiotemporal Transformer encoder to establish long-range micro-expression dynamic association. S2-3. Construct a facial muscle motion vector field using millimeter-wave radar data and fuse it with visual features at an early stage to improve the detection rate of micro-expressions.

3. The method for real-time micro-expression emotion assessment according to claim 2, characterized in that, In step S4, a lightweight DeepSORT algorithm is used to predict the continuously matched emotion trajectories using Kalmar filtering. Unmatched segments trigger a large cloud model for verification. When the user is detected wearing glasses or a mask, the weight of the gesture modality is automatically increased.

Images