Emotion recognition and intervention system based on facial micro-expression and physiological signal fusion

Abstract

The invention belongs to the technical field of artificial intelligence and health monitoring, and particularly relates to an emotion recognition and intervention system based on facial micro-expression and physiological signal fusion. The emotion recognition precision is improved through space-time alignment analysis of facial micro-expressions and physiological signals, adaptive feedback is achieved in combination with cognitive load correlation modeling and a wearable multi-channel regulation and control terminal, cross-period emotion evolution prediction and group situation awareness are supported, a'awareness-decision-intervention 'complete closed loop is constructed, and the emotion recognition efficiency is improved. And the accuracy and initiative of emotion management in a complex environment are enhanced.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and health monitoring technology, specifically relating to an emotion recognition and intervention system based on the fusion of facial micro-expressions and physiological signals. Background Technology

[0002] Emotion recognition and intelligent intervention technologies, as core directions in human-computer interaction and mental health management, have made significant progress in recent years driven by the intersection of artificial intelligence, biosensing, and cognitive science. This technology aims to objectively and in real-time monitor emotional states by analyzing an individual's external behavioral manifestations and internal physiological activities, and then provide corresponding feedback or intervention measures. Its applications have covered several important scenarios, including assisted diagnosis of mental illness, driver fatigue warning, and educational affective computing, becoming one of the key enabling technologies for improving human quality of life and work efficiency. Among these, non-invasive emotion perception methods have received widespread attention due to their excellent user experience and deployability.

[0003] Among these approaches, emotion recognition methods based on multimodal information fusion are gradually becoming the mainstream technological path. This method integrates various physiological and behavioral signals, such as facial expressions, voice tone, heart rate variability, and skin conductance, to overcome the limitations of single-modal information and achieve more comprehensive and stable emotion discrimination results. In particular, facial micro-expressions, due to their short duration, small amplitude, but strong emotion specificity, are considered important clues for revealing hidden emotions; while simultaneously acquired physiological signals such as EEG and ECG can reflect the autonomic nervous system response under emotional arousal, providing deeper evidence for recognition. Therefore, how to effectively integrate these two heterogeneous but complementary information sources to construct a high-precision, low-latency emotion recognition model has become a key research focus.

[0004] While existing technologies attempt to combine facial micro-expression analysis with physiological signal monitoring, they still face multiple challenges in practical applications: significant differences in temporal scale and spatial resolution between multi-source signals lead to difficulties in feature alignment; micro-expression capture is susceptible to lighting, occlusion, and individual differences, resulting in unstable recognition accuracy; while physiological signals possess inherent sensitivity, they are equally vulnerable to external interference and lack a clear mapping relationship with specific emotion categories; more importantly, most systems focus only on the emotion recognition stage, failing to achieve closed-loop control from perception to proactive intervention, thus failing to meet the application requirements of real-time psychological regulation. Especially in critical scenarios such as high-pressure work and telemedicine, these deficiencies may lead to misjudgment of emotions or delayed responses, severely impacting the system's practicality and reliability. Therefore, there is an urgent need for an integrated system architecture capable of efficiently fusing facial micro-expressions and physiological signals, and possessing real-time emotion intervention capabilities. Summary of the Invention

[0005] The purpose of this invention is to provide an emotion recognition and intervention system based on the fusion of facial micro-expressions and physiological signals, in order to solve the problems of existing emotion recognition technologies in complex environments, such as limitations of single-modal perception, unstable recognition accuracy, poor dynamic adaptability, and lack of real-time closed-loop intervention mechanisms. Current emotion recognition technologies mainly rely on a single information source from visual or physiological signals, resulting in a significant decrease in recognition accuracy under conditions of changing lighting, facial occlusion, individual differences, or low signal-to-noise ratio scenarios. At the same time, most systems only achieve passive monitoring and fail to build a complete technical closed loop from accurate recognition to active adjustment, making it difficult to meet the practical application needs of mental health management, human-computer interaction optimization, and high-pressure work scenarios.

[0006] The technical solution of this invention includes a multimodal data synchronous acquisition subsystem, a dynamic feature fusion and emotion decoding subsystem, a personalized emotion baseline modeling subsystem, a cognitive load and emotion fluctuation correlation analysis module, an adaptive feedback intervention generation subsystem, a wearable multi-channel physiological regulation terminal, and a cross-time period emotion evolution tracking engine. The multimodal data synchronous acquisition subsystem consists of a high frame rate infrared imaging unit and a multi-channel biosignal sensing array. The former captures subtle facial muscle movements, especially displacement changes in the orbicularis oculi, zygomaticus major, and corrugator supercilii muscles, at a sampling frequency of no less than 120 frames per second. The latter synchronously acquires skin conductance response, heart rate variability, forehead temperature gradient, and EEG alpha band power spectral density. The dynamic feature fusion and emotion decoding subsystem receives the raw temporal data stream. First, it models the optical flow field of the micro-expression video sequence to extract the time series of action unit intensity. Simultaneously, it performs wavelet denoising and frequency-band energy extraction on the physiological signals. Then, it inputs both types of features into a dual-path temporal neural network. The visual pathway uses a 3D convolutional long short-term memory network to process local action unit dynamics, while the physiological pathway uses attention-enhanced bidirectional gated recurrent units to mine autonomic nervous system response patterns. The outputs of both pathways achieve context alignment and information complementarity in the intermediate layer through a cross-attention mechanism with learnable weights. Finally, it outputs an eight-dimensional discrete emotion category probability distribution and arousal and valence estimates in continuous dimensions. The personalized emotion baseline modeling subsystem performs a 72-hour interference-free baseline acquisition during the initial user phase to establish a model of the frequency distribution of micro-expression activities and resting physiological parameter combinations in an individual's daily state. This model is then used as a reference to calculate the standard deviation multiple of deviations from the baseline in subsequent detections, serving as a preliminary criterion for abnormal emotional activation. The cognitive load and emotion fluctuation correlation analysis module continuously monitors the changing trend of the theta wave to beta wave energy ratio in the prefrontal cortex EEG signal. Combined with the task completion delay rate and pupil diameter fluctuation amplitude, it constructs a cognitive resource consumption index and performs Granger causality analysis on it with emotion decoding results to identify secondary anxiety or frustration caused by cognitive overload. The adaptive feedback intervention generation subsystem generates customized instruction sequences based on emotion type, intensity level, and trigger classification, calling a preset intervention strategy library. When acute anxiety is detected, a breathing guidance program is triggered; when persistent low mood is identified, positive memory cues are activated; and when cognitive overload is determined, a phased task interruption is suggested and a specific frequency of binaural beat audio is played. The wearable multi-channel physiological regulation terminal integrates a miniature bone conduction speaker, a distributed thermal stimulation patch, and a transcutaneous vagus nerve electrical stimulation electrode. It precisely applies auditory, somatosensory, and neurophysiological triple regulation signals according to intervention instructions. The thermal stimulation patch is distributed along both sides of the neck and raises the skin temperature to the range of 36.5°C to 38.0°C at a heating rate of 0.5°C per second, continuously activating the parasympathetic reflex pathway.The cross-period emotion evolution tracking engine aggregates emotion recognition results daily, constructs an individual emotion fluctuation heatmap, uses a hidden Markov model to infer the potential psychological state transition patterns, and predicts the probability of high-risk emotional events occurring in the next 24 hours based on long-term trends, so as to deploy preventive intervention plans in advance.

[0007] Furthermore, a modal confidence assessment mechanism is introduced during the dynamic feature fusion process. This mechanism adjusts the dual-path input weights in real time based on the current signal quality index. When the signal-to-noise ratio of the facial image is lower than a preset threshold, the contribution weight of the visual pathway is automatically reduced to no more than 30% of the total fusion weight, while the dominance of the physiological signal pathway is enhanced. The confidence assessment criteria include, but are not limited to, pupil visibility, facial contour integrity, skin conductance baseline drift, and EEG artifact ratio, ensuring that the system can still maintain reliable output under conditions of partial sensory channel degradation.

[0008] Furthermore, the training process of the dual-path temporal neural network adopts an adversarial domain adaptation strategy, and performs joint optimization on a multi-center database containing samples of different ages, genders, and races. This forces the network to learn emotional representation features shared across populations, reducing the interference of individual appearance differences on micro-expression decoding. During the training phase, virtual occlusion enhancement technology is introduced to randomly simulate mask, glasses, or hand occlusion situations, thereby improving the robustness of the model under real-world occlusion conditions.

[0009] Furthermore, the strategy library update mechanism of the adaptive feedback intervention generation subsystem is implemented based on a reinforcement learning framework. The system records the change in emotional state before and after each intervention as a reward signal, evaluates the long-term effectiveness of different intervention methods in various situations through the Q-learning algorithm, and dynamically adjusts the strategy priority ranking. The user's subjective rating of the intervention experience is also included in the update process as an auxiliary reward item, forming a human-machine collaborative optimization closed loop.

[0010] Furthermore, the electrical stimulation parameters of the wearable multi-channel physiological regulation terminal follow an individualized setting protocol. The initial stimulation intensity is determined by a vagus nerve threshold test program, which increases the current output at a step rate of 5 mA per minute until the user reports a slight throat contraction sensation. 60% of this threshold is taken as the initial working intensity. The stimulation waveform uses an asymmetric rectangular pulse with a frequency locked at 25 Hz and a duty cycle controlled at 20% to balance nerve activation efficiency and wearing comfort.

[0011] Furthermore, the cross-time period emotion evolution tracking engine has a group pattern analysis function, which allows data from multiple authorized users to be aggregated and modeled after de-identification, to identify the emotion propagation path and group stress response pattern at the organizational level, providing macro-situational awareness support for enterprise team management, emergency command and dispatch and other scenarios.

[0012] Furthermore, the infrared imaging unit of the multimodal data synchronous acquisition subsystem is equipped with a narrowband filter with a center wavelength set at 850 nanometers, which effectively suppresses ambient visible light interference and improves the contrast of micro-expression textures in low-light environments; its lens group has autofocus and image stabilization functions to ensure that key facial areas are not lost when the head moves slightly.

[0013] Furthermore, the personalized emotion baseline modeling subsystem sets baseline update trigger conditions. When the system detects that the mean deviation of the user's basic physiological parameters exceeds 2 standard deviations from the initial baseline for 7 consecutive days, a new round of baseline collection is initiated to avoid misjudgment caused by changes in lifestyle or health status.

[0014] Compared with the prior art, the advantages and positive effects of the present invention are as follows: This system overcomes the inherent limitations of single-modal perception in complex application scenarios by constructing a spatiotemporally aligned fusion architecture of facial micro-expressions and multidimensional physiological signals, significantly improving the accuracy and stability of emotion recognition in real-world environments. The dynamically weighted feature fusion mechanism enables the system to operate continuously even under conditions of partial sensor failure or signal degradation, enhancing its engineering practicality. The combination of personalized baseline modeling and cross-period evolution tracking achieves a leap from instantaneous state judgment to long-term psychological trend prediction, shifting emotion management from passive response to proactive prevention. The causal correlation analysis module between cognitive load and emotional fluctuations reveals deep psychological mechanisms, providing a scientific basis for precise intervention. According to reports, the multi-channel wearable control terminal integrates three physical intervention modalities, forming a synergistic physiological regulation network that exhibits stronger neuromodulation efficacy compared to a single feedback method. The adaptive intervention strategy library relies on reinforcement learning to achieve continuous evolution, ensuring that the system continuously optimizes service quality as usage time increases. The overall technical solution forms a complete closed loop of "perception-decoding-decision-intervention-feedback," which not only achieves high-precision emotion recognition but also establishes a technical paradigm for proactively regulating psychological states. It provides a system-level solution with clinical translational potential for the field of mental health and has broad application prospects in high-risk scenarios such as intelligent driving, telemedicine, and monitoring of special operations personnel. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall technical solution architecture of the emotion recognition and intervention system based on the fusion of facial micro-expressions and physiological signals proposed in this invention; Figure 2 This is a schematic diagram of the core principle framework of the dynamic feature fusion and emotion decoding subsystem in this invention. Detailed Implementation

[0016] Please refer to Figure 1 and Figure 2This invention provides an emotion recognition and intervention system based on the fusion of facial micro-expressions and physiological signals. Its technical architecture comprises a multimodal data synchronous acquisition subsystem, a dynamic feature fusion and emotion decoding subsystem, a personalized emotion baseline modeling subsystem, a cognitive load and emotion fluctuation correlation analysis module, an adaptive feedback intervention generation subsystem, a wearable multi-channel physiological regulation terminal, and a cross-time-period emotion evolution tracking engine. Each subsystem works collaboratively according to a preset data flow and temporal logic, forming a complete technical chain from raw perceptual signal input to closed-loop psychological regulation output. The system is deployed in a hybrid architecture of edge computing and cloud-based collaborative operation. Real-time data acquisition, preliminary processing, and intervention execution tasks are completed on local edge devices, while model training, long-term trend analysis, and group-level aggregation modeling are performed on a cloud platform with high-performance computing resources.

[0017] The system's overall technical process begins with the user wearing a wearable multi-channel physiological control terminal and initiating the system initialization procedure. The multimodal data synchronous acquisition subsystem then enters continuous operation, simultaneously acquiring the user's facial image sequences and multidimensional physiological signals. The dynamic feature fusion and emotion decoding subsystem receives the raw time-series data stream from the previous subsystem, performs high-order feature extraction and cross-modal fusion, and outputs an estimate of the current emotional state. This result is sent to the personalized emotion baseline modeling subsystem for individualized deviation assessment, and also transmitted to the cognitive load and emotion fluctuation correlation analysis module to determine the type of emotional trigger. Based on the comprehensive judgment result, the adaptive feedback intervention generation subsystem calls the strategy library to generate customized intervention instructions and implements physical adjustments through the wearable terminal. All historical identification records and intervention response data are aggregated into the cross-time period emotion evolution tracking engine to construct an individual emotional dynamic model and predict future risk events. The entire system forms a closed-loop structure of "perception-decoding-decision-intervention-feedback," achieving proactive, precise, and continuous management of psychological states.

[0018] The multimodal data synchronous acquisition subsystem, serving as the system's information input front-end, undertakes the task of capturing high-quality, low-latency, and spatiotemporally aligned multi-source physiological signals. The subsystem consists of two core sensing units: a high-frame-rate infrared imaging unit and a multi-channel biosignal sensing array. These two units utilize a hardware-level timestamp synchronization mechanism to ensure millisecond-level time alignment of all acquired data, avoiding feature misalignment issues caused by sampling asynchrony. The synchronization mechanism employs the IEEE 1588 precision time protocol, establishing a unified clock source within the main control chip. All sensors use this clock as a reference for data packaging and transmission.

[0019] The high frame rate infrared imaging unit is mounted at the front of the head-mounted frame, facing the center of the user's face. The angle between its optical axis and the extension line of the bridge of the nose is controlled within ±5 degrees, ensuring coverage of key motion areas such as the area around the eyes, cheekbones, and between the eyebrows. The core imaging device is a global shutter CMOS sensor with an effective pixel count of at least 1920×1080, supporting progressive scan mode to eliminate motion blur caused by the rolling shutter effect. The sampling frequency is set to 120 frames per second, meeting the Nyquist sampling theorem requirement for capturing micro-expression events with a duration of less than 500 milliseconds. To improve image quality in low-light environments, the unit is equipped with a narrow-band filter with a center wavelength of 850 nm. This band is located in the near-infrared region, effectively penetrating superficial skin tissue to enhance the contrast of subcutaneous capillary networks and muscle textures, while suppressing visible light background noise interference. The lens group adopts a fixed-focus wide-angle design with a focal length of 4 mm and a field of view of 85 degrees, maintaining the complete facial contour within the frame even with slight head movements. The autofocus algorithm calculates image sharpness in real time based on the Laplacian variance evaluation function. When the variance value drops by more than 15% of the initial stable value, the focus motor is triggered to adjust the lens position. The image stabilization function detects changes in device posture by integrating a three-axis micro gyroscope, and combines electronic image stabilization algorithm to crop and translate video frames to compensate for image shift caused by head shaking.

[0020] The raw video stream output by the infrared imaging unit is in 16-bit grayscale format, with each frame containing information on facial surface temperature distribution and local deformation. The system performs differential operations on consecutive video frames to generate inter-frame optical flow maps, which are used for subsequent motion unit intensity quantization. To ensure privacy and security, all image data is destroyed immediately after feature extraction is completed on the local device, without any form of long-term storage or network upload.

[0021] A multi-channel biosignal sensor array is integrated into a postauricular support, the inner forehead band, and a neck patch, enabling non-invasive, continuous, multi-point physiological monitoring. The array contains four independent sensing channels: a skin conductance response channel, a heart rate variability channel, a forehead temperature gradient channel, and an EEG alpha band power spectral density channel. Each sensor channel is manufactured using flexible printed circuit board technology, exhibiting excellent surface fit and resistance to motion artifacts.

[0022] The skin conductance response channel uses a silver / silver chloride dry electrode pair, placed on the left earlobe and the inner side of the right wrist, respectively, with a constant bias voltage of 0.5 volts applied to measure the skin conductance level between the two electrodes. The signal is amplified 1000 times by an instrumentation amplifier, then passed through a 0.05 Hz high-pass filter to remove DC drift, followed by a 50 Hz notch filter to suppress power frequency interference, and finally digitized at a sampling rate of 256 Hz. The heart rate variability channel uses photoplethysmography (PPG), employing a combination of a red LED and a photodetector placed behind the left earlobe to detect periodic changes in blood perfusion. The raw pulse wave signal is bandpass filtered (0.5 Hz to 5 Hz), and the RR interval sequence is extracted using a peak detection algorithm as the basis for calculating heart rate variability. The forehead temperature gradient channel uses three miniature thermistors arranged in a triangle 2 cm above the brow, spaced 1.5 cm apart, with a sampling resolution of 0.01 degrees Celsius, to capture minute temperature differences caused by changes in local blood flow. The EEG alpha band power spectral density channel features silver-printed electrodes at the bilateral temples, a reference electrode at the root of the nose, and a ground electrode below the occipital protuberance, forming a unipolar lead configuration. The signal is passed through an impedance matching circuit to a low-noise preamplifier with a gain set to 5000 ohms and a bandpass filter range limited to 8 Hz to 13 Hz, specifically designed to extract alpha rhythm components associated with relaxation. All physiological signals, after initial conditioning at the analog front-end, are synchronously sampled by a 24-bit resolution analog-to-digital converter, precisely time-stamped, and then transmitted to the subsequent processing unit.

[0023] The dynamic feature fusion and emotion decoding subsystem is the core intelligent hub of the system, responsible for transforming raw multimodal signals into semantically meaningful descriptions of emotional states. The subsystem first preprocesses and extracts features from the two heterogeneous data streams, then achieves deep feature fusion through a dual-path temporal neural network architecture, ultimately outputting an eight-dimensional discrete emotion category probability distribution and two-dimensional continuous emotion space coordinates.

[0024] For micro-expression video sequences, the system performs optical flow field modeling. Based on the original video at 120 frames per second, the Farnebäck dense optical flow algorithm is used to calculate the pixel-level displacement vector field between adjacent frames. The algorithm constructs a quadratic polynomial to approximate the image brightness function, establishes a local spatial-temporal neighborhood model around each pixel, and solves for the displacement parameters in the least squares sense. The resulting optical flow map is presented as a two-dimensional vector field, where the vector direction represents the direction of motion and the magnitude represents the motion velocity. The system focuses on key regions defined by the facial motion coding system: the left and right orbicularis oculi muscle regions (corresponding to blinking and squinting), the zygomaticus major muscle region (smiling), and the corrugator supercilii muscle region (frowning). Within these regions, the system calculates the absolute value of the vertical component of the average optical flow vector as a surrogate indicator of the intensity of the action unit. For example, in the left orbicularis oculi muscle region, if the absolute value of the vertical component exceeds two standard deviations from the resting level within three consecutive frames and the duration is between 200 and 500 milliseconds, it is determined that an AU43 (eye closure) event has occurred. All detected action unit events are arranged in chronological order to form an action unit intensity time series, which serves as the input feature for the visual pathway.

[0025] For multi-channel physiological signals, the system implements wavelet denoising and frequency band energy extraction. The Daubechies wavelet basis (db4) is used to perform a 5-level discrete wavelet decomposition on the original signal, decomposing the signal into detail coefficients and approximation coefficients at different scales. For skin conductance response signals, the reconstructed signal is retained from the 3rd to 5th level detail coefficients, effectively filtering out slow drift caused by respiratory motion and high-frequency electromagnetic noise. For heart rate variability (RR) interval sequences, Morlet continuous wavelet transform is used to extract low-frequency components from 0.04 Hz to 0.15 Hz and high-frequency components from 0.15 Hz to 0.4 Hz, reflecting sympathetic and parasympathetic nerve activity, respectively. For electroencephalogram (EEG) signals, the average power spectral density in the 8 Hz to 13 Hz frequency band is directly calculated as a quantitative indicator of α-band activity level. All processed physiological signals are normalized to the [0,1] interval, forming a fixed-length time window feature vector with a sliding window width of 10 seconds and a step size of 2 seconds.

[0026] The two types of features are input into a dual-path temporal neural network. The visual pathway employs a 3D convolutional long short-term memory (LSTM) network structure. Three 3D convolutional layers are stacked, with kernel sizes of (3,3,3), (3,3,3), and (1,1,1), all with a stride of 1. Zero padding ensures spatial dimension invariance, while the temporal convolutional kernels capture short-term dynamic patterns. Each layer is followed by batch normalization and a ReLU activation function. The spatiotemporal feature map output from the 3D convolutional layers is flattened and fed into a two-layer stacked LSM network with a hidden layer dimension of 128 and bidirectional connections to utilize both past and future contextual information. The final output is a 256-dimensional visual semantic embedding vector.

[0027] The physiological pathway employs an attention-enhanced bidirectional gated recurrent unit (BRN) network. The original physiological feature vector sequence is sequentially passed through two layers of BRNs, each containing 128 hidden units. A Bahdanau attention mechanism is introduced at the output of the second layer to calculate the alignment weight between the current output and the entire sequence, and these weighted sums yield the context vector. The attention weights reflect the varying importance of physiological signals at different time points in assessing the current emotional state. For example, in the early stages of a stress response, increased skin conductance may receive higher weights; while in the recovery phase, heart rate variability indicators receive increased weights. The final output is a 256-dimensional physiological semantic embedding vector.

[0028] The two outputs complement each other in the intermediate fusion layer through a cross-attention mechanism with learnable weights. Let the visual semantic embedding sequence be V∈ ^(T×D), the physiological semantic embedding sequence is P∈ ^(T×D), where T is the number of time steps and D is the feature dimension. Cross-attention is calculated as follows:

[0029] in, , , These are the projection matrices for the query, key, and value, respectively. is the dimension of the key vector. This mechanism allows visual features to focus on the most relevant physiological state segments, and vice versa, achieving cross-modal context alignment. The fused feature vector is concatenated with the original dual-path output to form a 768-dimensional joint feature representation. This representation is compressed to 128 dimensions through a fully connected layer and then branches into two output heads: a Softmax classifier outputs the probability distribution of eight emotions: anger, disgust, fear, happiness, sadness, surprise, contempt, and neutrality; the other regression head outputs the arousal and valence estimates in the interval [-1,1], forming a continuous emotion space coordinate.

[0030] To address the issue of partial modal signal degradation in real-world environments, the system introduces a modal confidence assessment mechanism. This mechanism dynamically adjusts the input weights of both pathways based on real-time signal quality indicators. For the visual pathway, assessment indicators include pupil detection success rate, facial contour integrity score, and image signal-to-noise ratio (SNR). Pupil detection employs Hough circle transform combined with template matching; if the center of both pupils cannot be located for five consecutive frames, it is considered severely occluded. Facial contour integrity is quantified by calculating the root mean square of the residuals from the active shape model fitting; a residual exceeding 3 pixels is considered distortion. Image SNR is estimated using the ratio of local variance to global variance. When any of the above indicators deteriorates below a threshold, the system activates a confidence decay function, linearly reducing the contribution weight of the visual pathway in the fusion, down to a minimum of 30% of the total weight. For the physiological pathway, assessment indicators include the baseline drift rate of electrodermal signal (exceeding 0.5 microsiemens per minute is considered unstable) and the proportion of EEG artifacts (detected through independent component analysis of the energy percentage of eye movement and electromyography components; exceeding 20% ​​is considered severely contaminated). When the quality of the physiological signal deteriorates, its pathway weight is correspondingly reduced, enhancing the dominance of the visual pathway. The fusion weight adjustment process is controlled by a small gating network that takes various quality indicators as input and outputs normalized dual-path weight coefficients to ensure that the system can maintain reliable output under single-modal degradation conditions.

[0031] The personalized emotion baseline modeling subsystem aims to establish a psychophysiological reference system for individuals under steady-state conditions, providing individualized criteria for subsequent anomaly detection. The system initiates a 72-hour interference-free baseline data collection phase after the user's initial registration. During this period, the system runs in the background without any proactive prompts or interventions, only recording multimodal data of the user in natural states such as daily work, rest, and entertainment. The collected data includes statistics on the frequency of micro-expression activities extracted per minute (such as blink count per hour and the percentage of smile duration), combinations of resting physiological parameters (mean skin conductance level, low-frequency / high-frequency heart rate variability ratio, mean forehead temperature, and mean alpha band power), and covariates such as ambient light intensity and background noise level.

[0032] Baseline data is stored in a local encrypted database with timestamps. After 72 hours, the system initiates the baseline modeling algorithm. For discrete micro-expression features, a probability distribution model is constructed using kernel density estimation to determine the normal fluctuation range of the frequency of each action unit. For continuous physiological parameters, their mean and standard deviation during the waking period are calculated to form a multivariate Gaussian distribution hypothesis. All parameters are modeled segmentally according to the diurnal rhythm, with independent baseline models established for daytime (8:00-20:00) and nighttime (20:00-8:00) to reflect the natural changes in the physiological rhythm. The final individual baseline model contains a set of parameterized distribution functions and their covariance matrix, fully characterizing the multidimensional feature space distribution of the user in the baseline state.

[0033] During routine use, the system calculates the deviation of the current detection value from the baseline model in real time. For a single variable, the Z-score standardization formula is used:

[0034] in, For the current observation value, The baseline mean. This represents the baseline standard deviation. For multivariate joint deviation, Mahalanobis distance is used.

[0035] in, This is the baseline covariance matrix. When the Mahalanobis distance exceeds a preset threshold of 3.0, it is identified as a significant emotional activation event, triggering the deep emotion decoding process. This mechanism effectively distinguishes between genuine emotional fluctuations caused by external stimuli and static biases caused by inherent individual physiological differences, significantly reducing the false positive rate.

[0036] To further enhance the long-term effectiveness of the baseline model, the system sets baseline update trigger conditions. When the deviation of the daily calculated mean of basic physiological parameters (such as mean skin conductance and resting heart rate) exceeds two standard deviations from the initial baseline for seven consecutive days, and this trend is confirmed by the Mann-Kendall monotonicity test (p<0.05), the system determines that the user's basic physiological state has undergone structural changes. These changes may be caused by long-term lifestyle adjustments, the development of chronic diseases, or the effects of medication. At this point, the system sends a notification to the user, suggesting the initiation of a new 72-hour baseline data collection program. The newly collected data is used to reconstruct the baseline model, while the old model is archived for future reference. This update mechanism ensures that the baseline always reflects the user's current true state, avoiding system performance degradation due to model aging.

[0037] The cognitive load and emotion fluctuation correlation analysis module focuses on analyzing the cognitive drivers behind emotional responses, particularly identifying secondary negative emotions caused by cognitive resource overload. The module continuously monitors the relative energy ratio of theta waves (4 Hz to 8 Hz) and beta waves (13 Hz to 30 Hz) in the prefrontal cortex. Research indicates that an elevated theta / beta ratio is closely related to inattention and cognitive fatigue. The system uses an autoregressive model (AR model, order 16) to estimate the power of each frequency band in real time and calculate the theta / beta energy ratio within a sliding window (30 seconds).

[0038] Meanwhile, the module integrates task performance data as an external validation metric for cognitive load. When a user is performing a specific digital task, the system records the task completion delay rate (the ratio of actual time to standard time) and the error rate. Eye-tracking units detect pupil diameter changes and calculate the standard deviation of pupil dilation amplitude during the task; this metric is positively correlated with cognitive effort. These four metrics—θ / β ratio, task delay rate, error rate, and pupil dilation amplitude—are normalized and weighted to generate the Cognitive Resource Consumption Index (CRI).

[0039] Among them, weight to The index is determined by fitting it through linear regression based on historical user data, ensuring that the sensitivity of the index to the actual cognitive load is maximized.

[0040] The module performs Granger causality analysis on the CRI time series and the intensity sequences of negative emotions such as anxiety, frustration, and irritability from the emotion decoding results. The analysis window is 15 minutes long, with a sampling interval of 1 minute. If a significant Granger causal relationship exists between the CRI sequence and the negative emotion sequence (F test p < 0.01), and the lag order is 1 to 3, then the current emotional fluctuation is determined to be mainly driven by cognitive overload. This determination directly affects the choice of intervention strategy: for primary anxiety caused by non-cognitive triggers, the system prioritizes breathing guidance; while for secondary anxiety caused by cognitive overload, the focus is on task structure adjustment and attention resetting training.

[0041] The adaptive feedback intervention generation subsystem categorizes emotions by type, intensity level, and trigger, selects the optimal intervention plan from a pre-set strategy library, and generates a sequence of instructions. The strategy library stores multiple intervention paradigms, each associated with a specific emotional context label. The system uses a combination of a rule engine and reinforcement learning to determine strategy invocation.

[0042] When the probability of the anxiety category in the eight-dimensional emotion probability distribution output by the dynamic feature fusion and emotion decoding subsystem exceeds 0.7, and the arousal estimate is higher than 0.6, the system determines it to be an acute anxiety state. At this point, a breathing guidance program is triggered: the wearable terminal's miniature bone conduction speaker plays pre-recorded voice instructions, guiding the user to perform the 4-7-8 breathing method (inhale for 4 seconds, hold your breath for 7 seconds, exhale for 8 seconds), repeating 6 cycles. The voice rhythm is precisely controlled by a background metronome, with an error of less than 50 milliseconds.

[0043] When the probability of sadness exceeds 0.65 for 5 consecutive minutes and the valence estimate is below -0.5, the system determines it to be a state of persistent depression. Positive memory cues are activated: the terminal generates a gentle tactile signal on the inside of the left arm via a vibration motor, while a bone conduction speaker plays positive sound clips related to the user's personal experiences (such as family laughter or applause during moments of success). These clips are uploaded and tagged with emotional attributes by the user during the initial setup phase.

[0044] When the cognitive load and mood fluctuation correlation analysis module confirms that the mood is caused by cognitive overload, the system suggests a phased task interruption. A countdown timer is displayed on the terminal screen, forcing the user to pause their current work for 5 minutes and playing a 10Hz binaural beat audio. The binaural beat is achieved by inputting 100Hz and 110Hz pure tones to the left and right ears respectively. The brain perceives the 10Hz beat difference frequency, which is located in the alpha band and helps induce a state of relaxation and recovery of attention.

[0045] The policy library update mechanism is implemented based on a reinforcement learning framework. The system treats each intervention as a decision event and records the emotional state before the intervention begins. Selected intervention action A_t; state after intervention ends Define the reward function. Improvement in emotional state:

[0046] in, To balance the impact, a value of 0.5 was chosen, emphasizing the importance of reduced arousal for anxiety relief. Furthermore, after the intervention, users could rate their subjective experience on a scale of 1 to 5 using a touchscreen. This rating, after linear mapping, was added to the total reward as an auxiliary reward. The system employs... The learning algorithm updates the action value function:

[0047] in, The learning rate is 0.1. The discount factor is 0.9. Through long-term interaction, the system learns the long-term effectiveness of various intervention methods in different situations, dynamically adjusts the priority of strategies, and forms a human-machine collaborative optimization closed loop.

[0048] The wearable multi-channel physiological modulation terminal is the actuator for the system to achieve closed-loop intervention, integrating three physical modulation modes: auditory, somatosensory, and neuroelectrophysiological. The main body of the terminal is a lightweight head-mounted device, with a total weight of no more than 85 grams. The shell material is made of biocompatible polycarbonate, and the internal circuitry is waterproof and sweatproof, achieving an IPX4 protection rating.

[0049] Miniature bone conduction speakers are embedded inside the supports behind the left and right ears, close to the mastoid bone. They use piezoelectric ceramic transducers with a vibration frequency response range of 20 Hz to 10,000 Hz and a sound pressure level of up to 85 dB, ensuring clear transmission of voice commands and audio stimuli even in noisy environments, while avoiding sound leakage to others.

[0050] Four distributed thermal stimulation patches are symmetrically adhered to the skin surface above the trapezius muscles on both sides of the neck, covering the carotid sinus region. Each patch contains a miniature Peltier thermoelectric element, driven by a constant temperature control circuit. The temperature rise program is set to increase from ambient temperature to the target range of 36.5°C to 38.0°C at a constant rate of 0.5°C per second, maintain this temperature for 120 seconds, and then allow for natural cooling. This temperature range effectively activates the carotid body chemoreceptors, reflexively enhances vagal tone, and promotes an increase in the HF component of heart rate variability, achieving a rapid sedative effect.

[0051] The percutaneous vagus nerve electrical stimulation electrodes are made of medical-grade stainless steel and are located in the mastoid region behind the ear, sharing a contact surface with the bone conduction speaker. Stimulation parameters follow an individualized setting protocol. The initial intensity is determined by a vagus nerve threshold test procedure: the system increases the current output in increments of 5 mA per minute, starting at 1 mA. The user wears the testing device and reports their subjective sensations; when a slight laryngeal contraction or swallowing urge is first perceived, the current value I_th is recorded. During formal use, the working intensity is set to 0.6 × I_th to ensure effective activation while avoiding discomfort. The stimulation waveform is an asymmetric rectangular pulse, with an anodic phase width of 400 microseconds and a cathodic phase width of 100 microseconds, with zero net charge to prevent tissue polarization. The frequency is locked at 25 Hz, with a duty cycle of 20%, and the pulse train lasts for 5 seconds followed by a 25-second interval, forming a periodic stimulation pattern. Numerous clinical studies have confirmed that this parameter combination has significant effects in improving heart rate variability and reducing cortisol levels.

[0052] The three intervention modalities can be used individually or in combination. For example, for severe anxiety, the system can simultaneously activate breathing guidance (auditory), neck heating (somatosensory), and vagal nerve electrical stimulation (neurophysiological), forming a multi-channel synergistic regulatory network, whose physiological effects are stronger than any single modality.

[0053] The cross-time-period emotion evolution tracking engine is responsible for long-term mental health situation perception and proactive risk warning. Every day at midnight, the engine automatically performs a data aggregation task, arranging the emotion recognition results (eight-dimensional probability distribution and two-dimensional coordinates) for each minute of the past 24 hours along a time axis to generate a 24×60 resolution emotion fluctuation heatmap. The horizontal axis of the heatmap represents time (hours), and the vertical axis represents the emotion dimension (eight discrete emotions superimposed on a continuous space of arousal-valence). The intensity of the color represents the duration or intensity integral of that emotional state.

[0054] Based on the heatmap, the engine uses a Hidden Markov Model (HMM) to infer patterns in latent psychological state transitions. The model sets four hidden states: calm, mild stress, high alertness, and emotional breakdown. Observational symbols are discretized emotion classification results (8 categories). The state transition probability matrix and emission probability matrix are learned using the Baum-Welch algorithm. After training, the model can perform Viterbi decoding on historical data for any time period to reconstruct the most probable sequence of hidden states.

[0055] Based on learned state transition dynamics, the engine constructs a 24-hour risk prediction model. Using survival analysis, it calculates the cumulative risk function of entering an "emotional breakdown" state from the current state. When the predicted risk exceeds 15%, the system proactively deploys preventative intervention plans. These plans include: adjusting schedules to avoid high-load tasks, increasing positive social interaction reminders, and initiating low-intensity background adjustments (such as playing soothing music). For organizational users who have enabled group mode, the engine aggregates data from multiple members after anonymization (removing identity markers while retaining behavioral patterns) to identify emotion propagation paths. For example, Granger causal network analysis reveals that a manager's emotional fluctuations often precede those of team members by 2 hours, indicating they are a source of emotional contagion, allowing for targeted stress management support.

[0056] This embodiment constructs an emotion recognition and intervention system with high robustness, strong adaptability, and closed-loop regulation capabilities through the deep coupling and collaborative operation of the aforementioned subsystems. The multimodal synchronous acquisition and dynamic weighted fusion mechanism fundamentally overcomes the limitations of single-source perception in complex environments, enabling the system to maintain a recognition accuracy of over 90% even in challenging scenarios such as changes in lighting and facial occlusion. The combination of personalized baseline modeling and cross-time period tracking achieves a leap from instantaneous state monitoring to long-term psychological trajectory characterization. The cognitive load analysis module reveals the deep cognitive roots of emotional fluctuations, providing a scientific basis for precise intervention. The three-channel wearable control terminal directly acts on the autonomic nervous system through physical signals, achieving rapid and effective physiological regulation. The reinforcement learning-driven strategy evolution mechanism ensures continuous optimization of the system over time. The overall solution forms a complete "perception-decoding-decision-intervention-feedback" technical closed loop, not only improving the technical performance of emotion recognition but also establishing an engineering paradigm for proactively maintaining mental health, providing a feasible system-level solution for intelligent mental health services.

Claims

1. An emotion recognition and intervention system based on the fusion of facial micro-expressions and physiological signals, characterized in that, include: The multimodal data synchronous acquisition subsystem is used to synchronously capture the user's facial micro-expression video stream and multi-channel physiological signals to obtain a spatiotemporally aligned raw time-series data stream; The dynamic feature fusion and emotion decoding subsystem is used to receive the original time-series data stream, extract the intensity time series of micro-expression action units and the frequency-segmented energy features of physiological signals, and input the two types of features into a dual-pathway time-series neural network containing visual and physiological pathways. The personalized emotion baseline modeling subsystem is used to establish a model of the frequency distribution of micro-expression activities and the combination of resting physiological parameters of an individual in daily life during the initial user phase, and to use this as a reference to calculate the degree of deviation from the baseline in subsequent detection as a preliminary criterion for abnormal emotional activation. The cognitive load and emotion fluctuation correlation analysis module is used to continuously monitor the changing trend of the energy ratio of theta waves to beta waves in the prefrontal cortex region of EEG signals. It constructs a cognitive resource consumption index by combining task performance data and pupil diameter fluctuation amplitude, and performs causal analysis on the cognitive resource consumption index and emotion decoding results to identify secondary emotions caused by cognitive overload. The adaptive feedback intervention generation subsystem is used to generate customized instruction sequences by calling a preset intervention strategy library based on emotion type, intensity level and trigger classification. A wearable multi-channel physiological regulation terminal is used to receive the customized instruction sequence and apply auditory, somatosensory and neuroelectrophysiological triple regulation signals to perform emotional intervention; A cross-period emotion evolution tracking engine is used to aggregate emotion recognition results on a daily basis and construct an individual emotion fluctuation heatmap.

2. The emotion recognition and intervention system based on the fusion of facial micro-expressions and physiological signals according to claim 1, characterized in that, The multimodal data synchronous acquisition subsystem includes: a high frame rate infrared imaging unit for capturing subtle facial muscle movements at a high sampling frequency; a multi-channel biosignal sensing array for synchronously acquiring skin conductance response, heart rate variability, forehead temperature gradient, and EEG alpha band power spectral density; and a hardware-level timestamp synchronization mechanism to ensure that the facial micro-expression video stream and the multi-channel physiological signals are time-aligned with millisecond-level precision.

3. The emotion recognition and intervention system based on the fusion of facial micro-expressions and physiological signals according to claim 1, characterized in that, The dynamic feature fusion and emotion decoding subsystem comprises a visual pathway that processes local action unit dynamics and a physiological pathway that mines autonomic nervous system response patterns. The outputs of the two pathways are context-aligned and information-complementary through a cross-attention mechanism with learnable weights to output discrete emotion category probability distributions and arousal and valence estimates on continuous dimensions. It also includes a modal confidence assessment mechanism to adjust the input weights of the visual and physiological pathways in the fusion process in real time based on the current signal quality indicators. When the signal-to-noise ratio of the facial image is lower than a preset threshold, the contribution weight of the visual pathway is automatically reduced and the dominance of the physiological pathway is increased. The signal quality indicators include pupil visibility, facial contour integrity, skin conductance baseline drift, and EEG artifact ratio.

4. The emotion recognition and intervention system based on the fusion of facial micro-expressions and physiological signals according to claim 1, characterized in that, The training process of the dual-path temporal neural network adopts an adversarial domain adaptation strategy, performs joint optimization on a multi-center database containing samples of different ages, genders, and races, and introduces virtual occlusion enhancement technology during the training phase to improve the robustness of the model under real-world occlusion conditions.

5. The emotion recognition and intervention system based on the fusion of facial micro-expressions and physiological signals according to claim 1, characterized in that, The personalized emotion baseline modeling subsystem is also equipped with baseline update triggering conditions. When the system detects that the mean deviation of the user's basic physiological parameters exceeds the standard deviation of the initial baseline preset multiple for several consecutive days, a new round of baseline acquisition program is started to reconstruct the individual's micro-expression activity frequency distribution and resting physiological parameter combination model in daily life.

6. The emotion recognition and intervention system based on the fusion of facial micro-expressions and physiological signals according to claim 1, characterized in that, The strategy library update mechanism of the adaptive feedback intervention generation subsystem is based on a reinforcement learning framework. The system records the change in emotional state before and after each intervention as a reward signal, and combines it with the user's subjective rating of the intervention experience. The Q-learning algorithm is used to evaluate the long-term effectiveness of different intervention methods in various situations, so as to dynamically adjust the strategy priority ranking.

7. The emotion recognition and intervention system based on the fusion of facial micro-expressions and physiological signals according to claim 1, characterized in that, The wearable multi-channel physiological regulation terminal integrates a miniature bone conduction speaker, a distributed thermal stimulation patch, and a transcutaneous vagus nerve electrical stimulation electrode. The distributed thermal stimulation patch is distributed along both sides of the neck to activate the parasympathetic reflex pathway. The stimulation parameters of the transcutaneous vagus nerve electrical stimulation electrode follow an individualized setting protocol, and the initial stimulation intensity is determined by a vagus nerve threshold test procedure.

8. The emotion recognition and intervention system based on the fusion of facial micro-expressions and physiological signals according to claim 7, characterized in that, The vagus nerve threshold testing procedure is used to increase the current output at a step rate until the user reports a specific physiological sensation, and takes a preset percentage of the threshold as the starting working intensity. The stimulation waveform uses an asymmetric rectangular pulse, and the frequency and duty cycle are set to balance nerve activation efficiency and wearing comfort.

9. The emotion recognition and intervention system based on the fusion of facial micro-expressions and physiological signals according to claim 1, characterized in that, The cross-period emotion evolution tracking engine also has a group pattern analysis function, which allows data from multiple authorized users to be aggregated and modeled after anonymization in order to identify the emotion propagation path and group stress response patterns at the organizational level.

10. The emotion recognition and intervention system based on the fusion of facial micro-expressions and physiological signals according to claim 2, characterized in that, The high frame rate infrared imaging unit is equipped with a narrow band filter to suppress ambient visible light interference and features autofocus and image stabilization to ensure that it can effectively capture micro-expression textures in key facial areas even in low-light environments and under conditions of slight head movement.

Images