Emotion recognition method and system based on multi-modal spatio-temporal feature fusion

Abstract

The application discloses a mood recognition method and system based on multi-modal spatio-temporal feature fusion, and belongs to the field of affective computing and pattern recognition. The method comprises the following steps: constructing an initial adjustment resource index of a subject based on conscientiousness and emotional intelligence score; synchronously collecting multi-modal physiological signals such as electroencephalogram, heart rate variability and galvanic skin response to generate a multi-source heterogeneous physiological dataset; extracting a multi-modal spatio-temporal fusion feature vector encoding instantaneous physiological activation and long-term evolution trend; calculating a real-time instantaneous adjustment resource loss rate positively correlated with task load and negatively correlated with adjustment efficiency, and correcting the effective loss rate through the initial adjustment resource index; and dynamically calculating the remaining adjustment resource stock, and triggering a stress overload warning when the decay trend of the remaining adjustment resource stock meets the overconsumption condition. The application can improve the accuracy and timeliness of psychological resource loss evaluation and early warning of physiological collapse.

Description

Technical Field

[0001] This application belongs to the field of affective computing and pattern recognition, specifically involving an emotion recognition method and system based on multimodal spatiotemporal feature fusion. Background Technology

[0002] With the continuous advancement of affective computing and health monitoring technologies, utilizing physiological feedback for emotional state assessment has become a key approach to improving the intelligence level of human-computer interaction and safeguarding mental health. By integrating multiple sensor devices, modern emotion recognition systems can capture the physiological state of the human body in real time, providing crucial data support for the quantitative management of mental and physical stress. Especially in high-intensity work environments, continuous monitoring of individual emotional fluctuations and physiological indicators helps to achieve early detection and pre-intervention of occupational burnout, psychological stress, and emotional disorders, which has significant industry value in ensuring individual safety and improving work efficiency.

[0003] Among these, emotion recognition methods based on multimodal physiological signals aim to construct more comprehensive and accurate emotion representation models by fusing heterogeneous data from multiple sources, such as electroencephalogram (EEG) signals, heart rate variability, and skin conductance responses. The core of this technology lies in capturing subtle physiological changes in the human body under different external stimuli using multi-channel sensors and leveraging advanced algorithms to extract and classify features from massive amounts of physiological data. With the improvement of computing power, the shift from single-time-dimensional features to in-depth spatiotemporal multi-dimensional feature mining, combined with dynamic reasoning based on individual psychological background information, has become a cutting-edge trend in the field of affective computing.

[0004] However, traditional emotion recognition technologies primarily rely on the static feature extraction of physiological signals, neglecting the heterogeneous influence of individual personality traits on stress regulation, leading to significant biases in assessing stress tolerance among different subjects. Furthermore, existing systems typically only identify the current instantaneous emotional state, failing to effectively distinguish between different types of stressors, such as challenging stress and obstructive stress. In addition, existing assessment models are mostly based on linear mapping logic, lacking in-depth quantitative analysis of the depletion of an individual's internal regulatory resources. This makes it impossible to accurately identify the precursors of physiological breakdown caused by regulatory resource exhaustion under the illusion of high performance, resulting in insufficient effectiveness of the early warning system. Summary of the Invention

[0005] The purpose of this invention is to provide an emotion recognition method and system based on multimodal spatiotemporal feature fusion, which can effectively solve the problems in the background art mentioned above.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] Firstly, emotion recognition methods based on multimodal spatiotemporal feature fusion include:

[0008] Acquire psychological trait data of the subjects and construct an initial regulatory resource index based on the psychological trait data to characterize the psychological energy baseline of the subjects before they are subjected to stress load;

[0009] The subject's multimodal physiological signals in the current task scenario are collected synchronously. The multimodal physiological signals include at least electroencephalogram (EEG) signals, heart rate variability signals, and skin conductance response signals. The multimodal physiological signals are then processed in a spatiotemporal synchronization manner to form a multi-source heterogeneous physiological dataset.

[0010] Feature extraction is performed on the multi-source heterogeneous physiological dataset to generate a multimodal spatiotemporal fusion feature vector, which encodes instantaneous physiological activation patterns and long-term psychological evolution trends.

[0011] Using the multimodal spatiotemporal fusion feature vector, the instantaneous adjustment resource loss rate, which is positively correlated with task load intensity and negatively correlated with adjustment efficiency, is calculated in real time. The instantaneous adjustment resource loss rate is then corrected by the initial adjustment resource index to obtain the effective loss rate corrected by the individual baseline.

[0012] Based on the effective loss rate, the initial adjustment resource index is cumulatively deducted, and the remaining adjustment resource stock is dynamically calculated.

[0013] Monitor the decay trend of the remaining regulatory resources, and trigger a pressure overload warning command at least when the decay trend meets the preset overconsumption condition.

[0014] Preferably, the step of acquiring the subject's psychological trait data and constructing an initial regulatory resource index based on the psychological trait data includes:

[0015] The subjects' conscientiousness scores were obtained, which were obtained by weighted summation of the scores of the self-discipline sub-dimension, the organization sub-dimension, and the achievement orientation sub-dimension.

[0016] The subject's emotional intelligence score is obtained by summing the scores of emotional perception, emotional utilization, and emotional understanding.

[0017] The due diligence dimension score and the emotional intelligence score are normalized respectively to generate a normalized due diligence score and a normalized emotional intelligence score.

[0018] The normalized due diligence score and the normalized emotional intelligence score are weighted and combined to generate the initial moderating resource index, wherein the weight coefficient of the normalized due diligence score is greater than the weight coefficient of the normalized emotional intelligence score.

[0019] Preferably, the method further includes:

[0020] The sympathetic activation index, parasympathetic inhibition index, effective loss rate, and external performance indicators are extracted from the multimodal spatiotemporal fusion feature vector.

[0021] When the following conditions are met simultaneously: the sympathetic nerve activation index is in a state of high sympathetic nerve activation, the effective attrition rate is within a preset benign range, and the external performance indicator is maintained at a preset high level, the stress currently faced by the subject is judged as challenging stress.

[0022] When the sympathetic nerve activation index is continuously higher than the threshold of the sympathetic nerve hyperactivation state for a preset duration, the effective attrition rate exceeds the preset warning limit, and the parasympathetic nerve inhibition index shows that the parasympathetic nerve activity is suppressed, the pressure currently faced by the subject is judged as obstructive pressure, and a warning response corresponding to the obstructive pressure is triggered.

[0023] Preferably, the method further includes, after determining that it is an obstructive pressure:

[0024] Track physiological recovery indicators during non-task periods, including the recovery time of skin conductance level from the activated state to the resting baseline, and the low-frequency to high-frequency power ratio of heart rate variability;

[0025] When the recovery time exceeds the preset recovery threshold and the ratio of low-frequency to high-frequency heart rate variability power continues to deviate from the preset balance range, it is determined that the subject has entered the regulatory resource depletion stage, and an early warning instruction for physiological collapse corresponding to the regulatory resource depletion stage is triggered.

[0026] Preferably, the real-time calculation of the instantaneous adjustment resource consumption rate, which is positively correlated with task load intensity and negatively correlated with adjustment efficiency, includes:

[0027] The relative energy proportion of the EEG β band in the multimodal spatiotemporal fusion feature vector is extracted as a proxy indicator of the amount of psychological effort invested.

[0028] The reciprocal of the heart rate recovery rate is extracted as a proxy indicator of regulation efficiency. The heart rate recovery rate is the speed at which the heart rate recovers from its peak value to near the resting heart rate baseline during the recovery time of the task interval.

[0029] The normalized weighted combination of the sympathetic-parasympathetic balance index, the average skin electrical conductivity, and the cortical activation index was extracted and used as the task load intensity.

[0030] The ratio of the task load intensity to the adjustment efficiency is calculated as the instantaneous adjustment resource loss rate.

[0031] Preferably, the effective loss rate corrected by individual baseline is obtained by correcting the instantaneous adjusted resource loss rate with the initial adjusted resource index, which is achieved in the following way:

[0032]

[0033] in, The effective loss rate, For the instantaneous adjustment of resource consumption rate, The initial adjustment resource index. This is an adjustable correction factor.

[0034] Preferably, when collecting the multimodal physiological signals, each sampling point is accompanied by a spatiotemporal anchor point identifier consisting of a high-precision globally unified timestamp and spatial location coordinates;

[0035] The method further includes: after calculating the effective loss rate, performing spatiotemporal context correction using the spatiotemporal anchor identifier, including:

[0036] When the spatial coordinates of the subject are detected to be within the preset work area during the preset late night period, it is determined to be an occupational overload risk scenario, and the effective attrition rate is multiplied by a scenario correction factor greater than 1 to obtain the corrected attrition rate.

[0037] Under the occupational overload risk scenario, the initial adjustment resource index is cumulatively deducted based on the correction loss rate, and the remaining adjustment resource stock is dynamically calculated.

[0038] Preferably, after triggering the pressure overload warning command, the method further includes implementing tiered intervention with a hysteresis control mechanism based on the degree of scarcity of the remaining regulatory resources, wherein:

[0039] Set corresponding trigger thresholds and recovery thresholds for mild, moderate, and severe resource loss levels, respectively.

[0040] When the remaining adjustment resource inventory is continuously lower than the trigger threshold of a certain higher level resource loss level for a first preset time, the resource loss level will be upgraded from the current level to the higher level resource loss level.

[0041] Only when the remaining adjustable resource inventory remains above the recovery threshold of the current resource loss level for a second preset duration under the current resource loss level will it be downgraded to a lower resource loss level, in order to avoid frequent switching of intervention levels.

[0042] Preferred options also include:

[0043] The system receives subjective feedback tags from subjects regarding the accuracy of the early warning, associates and stores these subjective feedback tags with multimodal physiological feature fragments prior to triggering the early warning, and forms incremental learning samples.

[0044] The long-term physiological change trend of the subjects is continuously recorded, including the decay slope, average loss rate and recovery efficiency of the initial regulatory resource index.

[0045] Once the accumulated incremental learning samples meet the preset conditions, the parameters used in calculating the effective loss rate and the initial adjustment resource index are incrementally updated using the incremental learning samples, so that the model can adapt to the individual differences and long-term psychological state changes of the subjects.

[0046] Secondly, an emotion recognition system based on multimodal spatiotemporal feature fusion, used to perform the above method, includes:

[0047] The resource quantification module is used to acquire the psychological trait data of the subjects and construct an initial regulatory resource index based on the psychological trait data to characterize the psychological energy baseline of the subjects before they bear stress load.

[0048] The signal acquisition module is used to synchronously acquire the multimodal physiological signals of the subject in the current task scenario, and to perform spatiotemporal synchronization processing on the multimodal physiological signals to form a multi-source heterogeneous physiological dataset. The signal acquisition module includes a multi-channel sensor group, an edge synchronization unit, and an indoor positioning module.

[0049] The feature fusion module is used to extract features from the multi-source heterogeneous physiological dataset and generate a multimodal spatiotemporal fusion feature vector. The multimodal spatiotemporal fusion feature vector encodes instantaneous physiological activation patterns and long-term psychological evolution trends.

[0050] The evaluation and calculation module is used to calculate the instantaneous adjustment resource loss rate in real time using the multimodal spatiotemporal fusion feature vector, obtain the effective loss rate by correcting the initial adjustment resource index, dynamically calculate the remaining adjustment resource stock by cumulatively deducting the initial adjustment resource index, and monitor the decay trend of the remaining adjustment resource stock.

[0051] The early warning output module is used to trigger a pressure overload early warning command at least when the attenuation trend meets a preset excessive consumption condition.

[0052] In summary, this application includes at least one of the following beneficial technical effects:

[0053] 1. By quantifying individual conscientiousness and emotional intelligence as an initial regulatory resource index and using it as a baseline for psychological energy in subsequent calculations, this method effectively addresses the problem of traditional techniques neglecting the heterogeneity of individual personality traits, thereby improving the accuracy and personalization of stress tolerance assessments for different subjects.

[0054] 2. By utilizing the autonomic nervous system activity state and regulatory resource depletion rate reflected by multimodal spatiotemporal fusion feature vectors, it is possible to effectively distinguish between different types of stressors, such as challenging stress and hindering stress, and proactively identify regulatory resource depletion and physiological collapse precursors before external performance declines, thereby improving the timeliness and effectiveness of early warning.

[0055] 3. By combining psychological trait data with the spatiotemporal fusion characteristics of multimodal physiological signals, we can achieve a shift from traditional linear mapping logic to in-depth quantitative analysis of individual intrinsic regulatory resources. This breaks through the bottleneck of existing technologies that make it difficult to accurately quantify resource consumption under the illusion of high performance, making the evaluation results more interpretable and of practical reference value. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the overall scheme of the emotion recognition method based on multimodal spatiotemporal feature fusion in this application;

[0057] Figure 2 This is a diagram illustrating the core principle framework of this application for extracting multimodal spatiotemporal fusion features based on spatiotemporal convolutional neural networks;

[0058] Figure 3 This is a flowchart of the real-time calculation of the regulatory resource depletion rate based on the dynamic analysis of the initial regulatory resource index, physiological activation level and psychological effort input in this application;

[0059] Figure 4 This application presents a flowchart for determining stress source attributes and outputting a physiological collapse warning based on the matching relationship between the rate of resource depletion and the multimodal spatiotemporal fusion feature vector.

[0060] Figure 5 This is a diagram showing the multi-level interaction between the multi-channel sensor array, the cloud processing platform, and the subject's terminal in this application. Detailed Implementation

[0061] The following will be combined with the appendix Figures 1 to 5 The technical solution of the present invention is clearly and completely described below. Obviously, the following embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0062] In this embodiment, an emotion recognition method based on multimodal spatiotemporal feature fusion, by fusing individual psychological trait data with real-time collected multimodal physiological signals, achieves accurate quantification of an individual's psychological resource state and early warning of collapse risk, specifically including the following steps:

[0063] In the above method, step S1 involves constructing an initial regulatory resource index and obtaining psychological trait data of the subjects.

[0064] Step S101: Obtain the subject's conscientiousness dimension score.

[0065] Conscientiousness scores are obtained through a pre-set personality assessment scale, which subdivides conscientiousness into three sub-dimensions: self-discipline, organization, and achievement orientation. Each sub-dimension contains several behavioral description items, and each item uses a 5- or 7-point Likert score.

[0066] Participants selected a level based on their level of compliance, and the numerical value corresponding to the selected level was used as the score for the corresponding item. The scores for each item within the sub-dimension were summed or averaged to obtain the raw score for self-discipline. Logical original division and achievement-oriented original scores .

[0067] The self-discipline sub-dimensional assesses a subject's ability to maintain task goals when faced with temptations or distractions, such as "When there is an entertainment temptation, I can still complete the scheduled work on time"; the organization sub-dimensional assesses a subject's degree of organization and planning of the work environment and task processes, such as "I am used to giving clear priorities to tasks"; and the achievement orientation sub-dimensional assesses a subject's intrinsic motivation to pursue high-standard results, such as "I always set work goals for myself that are higher than others' expectations".

[0068] The scores of the three sub-dimensions are weighted according to preset weights to generate the original total score for due diligence. In one example, the weights are configured as follows: self-discipline weight is 0.4, orderliness weight is 0.3, and achievement-oriented weight is 0.3, with the sum of the three always being 1. The weighting formula is:

[0069]

[0070] in, The original total score for due diligence is comprised of scores for the three sub-dimensions mentioned above.

[0071] It should be noted that the above weights can be slightly adjusted according to the characteristics of the occupational group to which the subjects belong. For example, the achievement-oriented weight can be appropriately increased for positions that rely heavily on creativity, but the sum of the three weights must be kept at 1 to ensure that the scores among individuals have a uniform dimension and are comparable.

[0072] Step S102: Obtain the subject's emotional intelligence score.

[0073] The emotional intelligence scale is used to obtain quantitative values ​​for three levels: emotional perception, emotional utilization, and emotional understanding. The emotional intelligence scale also uses a multi-level scoring system. Each level consists of several self-assessment questions, and the score for each question is the level value selected by the subject.

[0074] The emotional perception level quantifies the subject's sensitivity to their own physiological arousal and emotional changes, such as whether they can promptly detect an increased heart rate and muscle tension and identify them as a precursor to anxiety; the emotional utilization level assesses the subject's efficiency in converting emotional energy into task motivation, such as whether they can transform tension into a boost to maintain focus; and the emotional understanding level assesses the subject's cognitive depth regarding the causes and evolution of emotions, such as whether they can realize that anger stems from unmet expectations rather than the event itself.

[0075] The raw scores for the three levels are denoted as follows: , , ,in The score represents the level of emotional perception. The score represents the level of emotional utilization. The scores representing emotional understanding are summed directly from the three factors to obtain the raw total score for emotional intelligence. The calculation formula is:

[0076]

[0077] If considered as an equally weighted average, it is equivalent to direct summation after linear scaling and does not affect subsequent processing; the raw total score of emotional intelligence reflects the subject's comprehensive ability in emotional perception, application and understanding.

[0078] Step S103: Normalize the conscientiousness score and emotional intelligence score.

[0079] To eliminate the dimensional differences between the personality assessment scale and the emotional intelligence scale, the raw total score for conscientiousness was adjusted. and raw total score of emotional intelligence Perform min-max normalization separately, mapping to the standard numerical space between 0 and 1. The normalization formula is:

[0080]

[0081]

[0082] in, and These are the theoretical minimum and maximum values ​​of the total conscientiousness score on the personality assessment scale, respectively. and These are the theoretical minimum and maximum values ​​of the total score of the Emotional Intelligence Scale, respectively. If there are no theoretical extreme values ​​or the number of items in the personality assessment scale or the Emotional Intelligence Scale is not fixed, then the minimum and maximum values ​​actually observed in a large-scale norm sample are used as substitutes.

[0083] As a default implementation, when normative data is unavailable, theoretical extreme values ​​can be calculated based on the scale range of the personality assessment scale itself. That is, assuming each sub-dimensional consists of k items, with each item having a score range of 1 to 5, the original score range for a single sub-dimensional is k to 5k. After weighted summation... The minimum value is k, and the maximum value is 5k; for example, when k=5, , Similarly, the extreme value of the raw total score for emotional intelligence can be calculated based on the number of items and the rating levels. After this transformation, and All scores are between 0 and 1, with higher scores indicating stronger corresponding psychological traits.

[0084] Step S104: Generate the initial adjustment resource index through weighted mapping.

[0085] The normalized original total score of due diligence Compared with the raw total score of emotional intelligence Perform a linear combination to generate the initial adjustment resource index. The combination formula is:

[0086]

[0087] In the formula, The first weighting coefficient corresponding to due diligence. The second weighting coefficient corresponds to emotional intelligence, and the two satisfy the following relationship:

[0088] ,and ;

[0089] The aforementioned weighting relationship reflects the different positioning of the two psychological resources: conscientiousness reflects the stable kinetic energy of an individual's sustained investment in goal-oriented behavior and resistance to interference, belonging to the stock foundation of regulatory resources; emotional intelligence reflects the strategic use of emotional information during the regulatory process, playing a role in amplifying efficiency. The stock foundation carries a larger weight, which can make... More accurately characterizing an individual's psychological energy baseline before experiencing stress load.

[0090] In one example, it is configured as follows This embodiment allows for fine-tuning based on application scenarios. For example, for customer service positions that frequently deal with interpersonal stress, it can be... Adjusted to 0.4 Adjusted to 0.6, but must be maintained. Furthermore, the sum of the two is 1, to ensure that the baseline estimate does not overly depend on strategic factors.

[0091] In the above method, step S2 involves collecting multimodal spatiotemporal physiological signals. Physiological signals of three modalities—electroencephalogram (EEG), heart rate variability, and skin conductance—are acquired simultaneously through a multi-channel sensor array. Combined with spatiotemporal labeling and quality control methods, a high-quality multi-source heterogeneous dataset is formed that can be used for subsequent feature extraction.

[0092] Step S201: Configure the EEG acquisition unit and acquire EEG signals.

[0093] Multiple electrodes of the EEG acquisition unit were placed in preset positions on the forehead and top of the subject according to the international 10-20 system layout to capture electrical activity in the frontal lobe and central cortex, which are closely related to attention and emotion regulation.

[0094] The contact impedance of each electrode is detected by injecting a weak alternating current before acquisition. The EEG acquisition unit only starts formal recording of the corresponding channel when the contact impedance drops below 50kΩ; otherwise, it continuously prompts for electrode adjustment. The EEG signal is sampled at a rate of 256Hz and converted from analog to digital. The passband is set from 0.5Hz to 45Hz to preserve the complete information of the δ, θ, α, β and low γ frequency bands.

[0095] Step S202: Configure the photoplethysmography (PPG) acquisition unit and obtain heart rate variability indicators.

[0096] The photoplethysmography (PPG) acquisition unit is fixed above the radial artery in the wrist or at the fingertip of the subject. It integrates a light-emitting diode (LED) and a photodetector of a specific wavelength. The LED alternately emits green light with a wavelength of 525nm and infrared light with a wavelength of 940nm. The photodetector receives the changes in light intensity after reflection or transmission through the tissue and converts them into a voltage signal, thereby obtaining a continuous pulse waveform. The pulse waveform is digitized at a sampling rate of 64Hz.

[0097] Real-time peak detection is performed on the raw pulse waveform to extract the time intervals between consecutive heartbeats, forming a heartbeat interval sequence. in This represents the i-th heartbeat interval, measured in milliseconds (ms). The time-domain index of heart rate variability, i.e., the root mean square of the difference between adjacent heartbeat intervals, is calculated in real-time based on the heartbeat interval sequence. The calculation formula is:

[0098]

[0099] In the formula, N represents the total number of intracardiac intervals in the current sliding window. The length of the sliding window can be configured from 30s to 300s, with a typical value of 120 seconds. At the same time, frequency domain indicators such as the ratio of low-frequency power to high-frequency power are calculated to reflect the dynamic regulatory state of the autonomic nervous system.

[0100] Step S203: Configure the skin conduction acquisition unit and acquire skin conduction response signals.

[0101] The two electrodes of the skin conduction acquisition unit are fixed to the thenar eminence of the subject's palm or the fingertips of the index and middle fingers. A constant DC voltage of 0.5V is applied between the electrodes. By measuring the changes in the microcurrent flowing through the skin, the skin conduction response signal is obtained.

[0102] The skin conductance response signal was synchronously acquired at a sampling rate of 64 Hz, recording the baseline drift of the skin conductance tension level and the transient phase response peak caused by sweat gland activity. In subsequent analysis, the skin conductance level component and the skin conductance response component were separated from the skin conductance response signal. The former reflects the overall physiological arousal level, while the latter reflects the rapid response to specific stimuli.

[0103] Step S204: Perform spatiotemporal synchronization of multimodal signals and construct the dataset.

[0104] During signal acquisition, each sampling point generated by a sensor is accompanied by a spatiotemporal anchor identifier, which includes a high-precision globally unified timestamp and spatial location coordinates. The high-precision globally unified timestamp is generated by an edge synchronization unit connected to a network time protocol server, with a resolution of at least 1 ms. The spatial location coordinates are obtained through an indoor positioning module, outputting the subject's current two-dimensional plane coordinates or a preset spatial region label. Each sampling point is thus marked as a data tuple in the form of a timestamp of 1620000000.123 and spatial coordinates of x=3.2, y=5.8.

[0105] All sensor channels are driven by a unified synchronous clock for sampling. The sampling rate of the EEG channel is 256Hz, while the sampling rates of the heart rate variability and skin conductance channels are 64Hz. Each sampling moment of the low sampling rate channel is precisely aligned with the temporally adjacent sampling point in the EEG channel, thereby ensuring frame-level correspondence between different modalities in the temporal dimension. The resulting multi-source heterogeneous physiological dataset has a unified timestamp and spatial coordinates for each record, providing a foundation for subsequent spatiotemporal convolutional networks to extract spatial connectivity features.

[0106] Step S205: Perform data quality monitoring and anomaly handling.

[0107] Simultaneously with signal acquisition, the signal quality of each channel is continuously monitored. For the EEG channel, the signal-to-noise ratio (SNR) within a specified frequency band is calculated in real time. The SNR is defined as:

[0108]

[0109] in, The average power in the α band is 8–12 Hz. The average power in the 60–80Hz high-frequency band.

[0110] For heart rate variability channels, the signal-to-noise ratio (SNR) is assessed by the peak sharpness of the pulse waveform and the degree of baseline drift; for electrodermal (ED) channels, the SNR is examined by the ratio of the amplitude of the conductance signal fluctuation during the unstimulated period to the amplitude of the response during the stimulated period.

[0111] Each modality has a preset independent quality threshold; for example, the signal-to-noise ratio threshold for the EEG channel is set to 10 dB.

[0112] If the signal quality index of any channel remains below a preset threshold for more than 3 seconds, an adaptive compensation process is automatically triggered. The compensation prioritizes an adaptive noise cancellation algorithm: using the interfered signal as the main input, a signal from another channel that is functionally related to the interfered channel but has a different interference mode is selected as the reference input. Known adaptive filtering techniques are used to estimate and remove noise, outputting a clean signal. For example, when the EEG channel suffers severe EMG interference, the instantaneous frequency change envelope of the heart rate variability signal is extracted as a reference, and adaptive filtering techniques are used to reconstruct a clean waveform in the low-frequency band of the EEG.

[0113] If a channel experiences severe signal loss that cannot be recovered even after adaptive compensation (i.e., the effective data rate is below 30% for more than 5 consecutive seconds), the channel will not be hard-killed. Instead, the weight of the corresponding mode will be automatically reduced in the feature fusion layer of subsequent step S3. The weight reduction method is to multiply the feature vector of the corresponding mode by an attenuation factor α, where α takes the value of:

[0114]

[0115] in, This represents the average signal-to-noise ratio of the corresponding channel within the current sliding window. The signal-to-noise ratio thresholds preset for the corresponding channels ensure that the multimodal fusion results can maintain reasonable stability even under adverse conditions of severe degradation of single-mode signals.

[0116] Through step S2, the method of this embodiment completes cross-modal signal acquisition from subcutaneous electrophysiology to surface optics and conductivity, and converts the acquired signals into a multimodal synchronous dataset with strict spatiotemporal labeling and quality screening.

[0117] In the above method, step S3 extracts multimodal spatiotemporal fusion features. The multi-source heterogeneous physiological dataset output in step S2 is injected into a preset spatiotemporal convolutional neural network model. Through modality-specific temporal convolution, cross-channel spatial graph convolution, attention fusion, and hierarchical multi-scale feature integration, a fusion feature vector that simultaneously encodes instantaneous physiological activation patterns and long-term psychological evolution trends is finally generated.

[0118] Step S301: Construct parallel modality-specific feature encoding channels.

[0119] The spatiotemporal convolutional neural network model sets up three parallel feature processing channels at the input end, which perform proprietary encoding for three modalities: electroencephalography (EEG), heart rate variability, and skin conductance response, as detailed below:

[0120] (1) The first channel processes EEG signals. The first channel is stacked with 3 to 5 one-dimensional temporal convolutional layers. If higher accuracy is required, it can be extended to 5 layers.

[0121] The first convolutional layer has a kernel length of 16 sampling points, a stride of 2, and 32 output channels; the second convolutional layer has a kernel length of 2, a stride of 2, and 64 output channels; the third convolutional layer has a kernel length of 64, a stride of 2, and 128 output channels.

[0122] Each convolutional layer is followed by batch normalization and ReLU activation, where ReLU is defined as... The convolution kernel slides along the time axis at a sampling rate of 256Hz, directly acting on the original voltage amplitude sequence, responding layer by layer to the energy fluctuations of the 8-13Hz α band, the activation level of the 14-30Hz β band, and the rhythmic changes of the 4-8Hz θ band, forming a time-frequency response feature map covering multiple frequency bands.

[0123] (2) The second channel processes the heart rate variability signal, and the second channel outputs the signal in step S202. The interval sequence is used as input, and a convolutional window of length 30 interval points slides along the sequence, with the window sliding 5 interval points at a time.

[0124] The convolution window uses a dilated convolution structure, with the dilation factors of the three layers set sequentially. , , This allows the convolution kernel to capture long-range correlations between heartbeats on a scale spanning dozens of heartbeats; the output features directly reflect the dynamic regulatory state of the sympathetic and parasympathetic branches in the autonomic nervous system.

[0125] (3) The third channel processes the skin conductance signal. First, the original skin conductance signal is differentially processed by the peak detection preprocessing layer. The positive zero crossing point is extracted as the occurrence time of the instantaneous phase response peak. At the same time, the average amplitude and frequency of the peak within each 10s window are statistically analyzed. The average amplitude is in micro-Siemens.

[0126] This is followed by two one-dimensional convolutional layers with kernel lengths of 8 and 16, strides of 2, and output channels of 32 and 64, respectively. Each layer is followed by batch normalization and ReLU. These two layers identify the baseline drift trend on a 60–120 s timescale, as well as the instantaneous activation intensity and recovery time of the skin conductance response under sudden stimulation.

[0127] After each of the three channels is encoded independently, it outputs a modality-specific temporal feature matrix, with the row direction being the feature vector sequence in the time dimension and the column direction being the feature dimension extracted by each convolution kernel.

[0128] Step S302: Construct a sensor spatial topology map and extract spatial correlation features.

[0129] After obtaining the temporal features of each modality, a spatial convolutional layer is introduced, and a graph convolution operator is used to model the functional connectivity between different acquisition sites.

[0130] First, all sensor data collection points on the subject's body surface are abstracted into a graph structure. The node set V includes EEG electrode sites, PEP pulse wave acquisition points, and skin conduction acquisition points. The number of EEG electrode sites depends on the selected international 10-20 system configuration. A typical configuration is to select 8 EEG electrode sites, 1 PEP pulse wave acquisition point, and 2 skin conduction acquisition points, for a total of 11 nodes. Each node corresponds to a specific anatomical location.

[0131] Connection relationships and edge weights of edge set E It is determined by both physical distance and signal correlation.

[0132] Method 1 is based on the physical spatial distance between sensors, taking nodes With nodes Inter-Euclidean distance The reciprocal of the distance is used as the distance weight, i.e. , Units are in cm.

[0133] Method 2 is based on the degree of functional coupling between signals, taking the absolute value of the Pearson correlation coefficient between the signals of two nodes within the current processing time window. As a relevance weight, i.e. .

[0134] The final edge weight is the weighted sum of the two values; a default configuration is... If the physical distance between two nodes exceeds 30cm and the correlation coefficient is less than 0.3, the corresponding edge weight is reset to 0, and the nodes are considered to be without edge connection, thereby sparsifying the graph.

[0135] Two spatial convolutional layers are stacked, and feature aggregation is performed using the message-passing paradigm of graph convolutional networks. For the l-th layer, nodes... The feature update rule is as follows:

[0136]

[0137] in, For nodes In the feature vector of the l-th layer, the input layer dimension is aligned with the feature dimension of the output of each channel encoder, and is unified to the same dimension by zero padding. For nodes The set of neighboring nodes; The above are the mixed edge weights; and They are nodes With nodes The degree; Let be the learnable weight matrix of the l-th layer, with the output dimension of the first layer set to 128 and the output dimension of the second layer set to 128. This is the ReLU activation function.

[0138] After two layers of spatial convolution iteration, the feature vector of each node is integrated with the physiological indicators of its own collection site and the activity patterns of its spatial neighbors and functionally related nodes. The originally isolated single-point indicators are transformed into a spatial topological feature matrix that reflects the overall coordinated state of the organism.

[0139] Step S303: Perform cross-modal feature fusion using an attention mechanism.

[0140] After obtaining the feature matrices of each modality after spatiotemporal encoding, the feature fusion layer is entered, and a multi-head attention mechanism is used to adaptively assign weights to the features of the three modalities of EEG, heart rate variability and skin conductance response.

[0141] For the current time step, the feature matrices updated by the three modality space convolutions are first compressed into fixed-length feature vectors by global average pooling, denoted as . The dimensions of the three vectors are uniformly set to d=128. The three vectors are stacked along the modal dimension to form a vector of size... input matrix .

[0142] The multi-attention module is set to have 8 attention heads (h=8), and the feature dimensions of each head are... For the k-th attention head, the k-th attention head independently learns three sets of projection matrices: querying the projection matrix. Key projection matrix Sum projection matrix All sizes The calculation process for the kth attention head is as follows:

[0143]

[0144]

[0145]

[0146] in, For size The attention weight matrix, where each element represents the degree of attention one mode pays to another, and the denominator is... Scaling factor

[0147] The outputs of the eight attention heads are concatenated along the feature dimension and then multiplied by the output projection matrix. The size is Receive multi-head attention output .right Average pooling is performed along the modal dimension to obtain a 128-dimensional cross-modal fusion vector.

[0148] The spatial topological feature matrix generated in step S302 is compressed into a 128-dimensional vector using global average pooling. This vector is then concatenated with the aforementioned cross-modal fusion vector to form a 256-dimensional concatenated vector. This concatenated vector is mapped through a fully connected layer containing 256 neurons, employing the ReLU activation function, and outputting a 256-dimensional multimodal spatiotemporal fusion feature vector. .

[0149] Step S304: Perform hierarchical feature abstraction and semantic splicing of psychological traits.

[0150] After obtaining the spatiotemporal fusion features of a single time frame, a hierarchical fusion strategy is further introduced to capture cross-scale psychological state changes, ranging from instantaneous fluctuations at the second level to long-term evolutions at the hour level.

[0151] (1) At the underlying feature level, the system performs both Fast Fourier Transform and Wavelet Transform on the original signal segment output in step S2. The Fast Fourier Transform slides on the original signal with a window length of 4s, and slides for 1s each time to extract the power spectral density of each frequency band as time-frequency features. The wavelet transform uses Morlet wavelet as the mother wavelet, with the center frequency set to 1Hz and the scale parameter spanning from 1 to 64, corresponding to an analysis frequency range of approximately 0.5-32Hz, generating a multi-resolution energy distribution map on the time-frequency plane. The outputs of the two transforms are spliced ​​frame by frame on the time axis to jointly constitute the underlying multi-scale time-frequency features.

[0152] (2) Mid-level feature layer, the feature obtained in step S303 The vector sequence is input into a long short-term memory network with two layers. The number of hidden units in each layer is set to 128. The forget gate bias is initialized to 1 to enhance information retention. The time step unfolding length is 120 frames, corresponding to an observation window of about 2 minutes.

[0153] Long Short-Term Memory Network (LSTM) automatically learns the rising, maintaining, or falling trends of physiological states as the task progresses through the coordinated control of the forgetting gate, input gate, and output gate. It extracts the hidden state of the last time step as the mid-level temporal trend feature vector with a dimension of 128 high-level feature layers and directly appends the subject's psychological trait labels obtained in step S1 to the end of the mid-level temporal trend feature vector.

[0154] Psychological trait labels include normalized conscientiousness scores. And emotional intelligence score After concatenation, a 130-dimensional global semantic vector is constructed. The first two dimensions of the global semantic vector encode the baseline level of an individual's regulatory capacity, while the last 128 dimensions encode the current and recent physiological activation and recovery state. The downstream resource allocation calculation model can make a comprehensive judgment by simultaneously referring to individual characteristics and real-time load.

[0155] After step S3, the multi-source heterogeneous physiological data from EEG, heart rate variability and skin conductance response, along with individual psychological trait information, are transformed into a multimodal global semantic vector that combines temporal evolution trends, spatial coordination patterns and psychological background, as the direct input to the resource allocation calculation model in step S4.

[0156] In the above method, step S4 involves real-time calculation of the adjustment resource depletion rate. The multimodal global semantic vector output in step S3 is input into the resource allocation calculation model. The resource allocation calculation model is based on the psychological resource conservation theory, which regards an individual's psychological adjustment ability as a limited and consumable dynamic resource.

[0157] Step S401: Extract proxy indicators of psychological effort input.

[0158] The system extracts physiological activation features related to cognitive load from the multimodal spatiotemporal fusion feature vector as proxy variables for the amount of psychological effort input.

[0159] The relative energy proportion of the beta band in EEG was selected as the core indicator. Power spectra were calculated using short time windows of 4 seconds each on the original EEG signal, and the total power within the 14–30 Hz band was summed. Divide by the total power in the 4–30 Hz frequency band The resulting ratio is denoted as :

[0160]

[0161] in, For β band power, Power across the entire frequency band of 4–30Hz. A higher value indicates stronger cortical activation and greater psychological effort.

[0162] Step S402: Extract proxy indicators of regulation efficiency.

[0163] Regulation efficiency measures how quickly an individual can restore their physiological state to baseline levels during task breaks, with heart rate recovery speed selected as a proxy.

[0164] During continuous monitoring, the preset activity threshold was 1.5 times the individual's multimodal physiological activation level at rest. When the activation level remained below the activation threshold for 30 consecutive seconds, an interval was initiated, and the peak heart rate at the start of the interval was recorded. The unit is beats per minute, and the lowest average heart rate during the resting period of the day is taken as the resting baseline. .

[0165] The endpoint for recovery is set to Calculate heart rate from The time taken to reach the recovery endpoint, expressed in seconds. Regulation efficiency Take the reciprocal of the recovery time:

[0166]

[0167] If the heart rate does not drop to the recovery endpoint throughout the entire rest period, it indicates that no significant recovery has been observed. Take the preset lower limit value .

[0168] Step S403: Calculate the task load intensity.

[0169] The task load intensity L is comprehensively assessed based on multimodal physiological activation levels. The system extracts three indicators within a sliding window: the low-frequency to high-frequency power ratio from the heart rate variability frequency domain indicators. Mean skin conductivity level and EEG and Band power ratio After normalizing each of these three indicators to the 0-1 range, they are linearly combined with fixed weights:

[0170]

[0171] Weight values: Sympathetic-parasympathetic balance index weights Skin conductance level weighting Cortical activation index weights The sum of the three is 1. A higher L indicates stronger current physiological activation and a heavier task load.

[0172] Step S404: Calculate the instantaneous adjustment resource loss rate and perform initial resource correction.

[0173] The instantaneous adjustment resource consumption rate r is defined as the ratio of task load intensity L to adjustment efficiency. The ratio:

[0174]

[0175] This ratio quantitatively describes how quickly adjustment resources are consumed per unit load at the current efficiency; the unit of r is seconds.

[0176] To account for individual differences, the system utilizes the initial adjustment resource index from step S104. The effective loss rate after correction is:

[0177]

[0178] Correction coefficient The adjustable range is 0.3-0.5, with a typical value of 0.4. The larger the factor The smaller the value, the lower the effective rate of loss for individuals with abundant initial resources under the same load.

[0179] Step S405, Spatiotemporal context correction of loss rate.

[0180] The assessment calculation module includes a spatiotemporal contextual correlation submodule, which continuously analyzes the spatial anchor points and timestamps of the subjects. When it is detected that the subject's spatial location is still marked as an office area between 11:00 PM and 5:00 AM the next day, it is determined to be an occupational overload risk scenario. At this time, a scenario correction factor λ is automatically introduced to further amplify the effective attrition rate.

[0181]

[0182] When the late-night work condition is triggered, the default value of λ is 1.3, which can be configured as needed between 1.2 and 1.5. If the late-night work condition is not triggered, then λ=1, without additional amplification, making the model more sensitive to the resource consumption estimates of individuals who are still in a high-pressure environment outside of working hours.

[0183] Step S406: Accumulate and calculate the remaining adjustment resource stock and monitor excessive consumption.

[0184] The system adjusts the resource index from the initial stage. By subtracting the effective loss rate, which has been corrected for time and space, from the remaining adjustment resource stock at the current time t, we obtain the remaining adjustment resource stock at the current time t. The accumulation process uses discrete summation.

[0185]

[0186] Take a fixed step size of 1 second. Let n be the i-th time step, and n be the total number of steps from the start of monitoring to the current time. The above accumulation method reflects the non-linear consumption of resources over time, representing the corrected loss rate at corresponding moments.

[0187] Simultaneously, monitor the rate of decline of the remaining stock. Within a 60-second time window... Perform linear regression to obtain the slope:

[0188]

[0189] Warning and alert value .when Furthermore, if the above state persists for more than 120 seconds, it is determined to be excessive consumption, providing a basis for triggering an advanced warning in step S5.

[0190] Step S4 above provides a complete quantitative path from multidimensional physiological characteristics to instantaneous loss rate. Step S406 then uses the instantaneous loss rate to dynamically track the remaining resource inventory and its decay trend. The output... and It is directly used as a decision variable in step S5 to distinguish the nature of stressors, determine resource depletion, and trigger physiological collapse warnings.

[0191] In the above method, step S5 identifies the stressor attributes and outputs a physiological collapse warning. Using the regulation resource depletion rate and remaining resource stock output in step S4, combined with the key indicators reflecting the state of autonomic nervous activity in the multimodal spatiotemporal fusion feature vector, the nature of the stressor currently faced by the subject is determined, and an early warning of physiological collapse is triggered based on the degree of resource depletion. At the same time, differentiated intervention suggestions are output based on spatial location and resource stock level.

[0192] Step S501: Extract the multimodal features and indicators for pressure source determination.

[0193] The system analyzes four key decision indicators from the multimodal spatiotemporal fusion feature vector. The details are as follows:

[0194] The first item is the sympathetic activation index, which is calculated by weighting the increase in heart rate from the resting baseline with the peak density of skin conductance response. The weight for heart rate increase is set to 0.6, and the weight for skin conductance peak density is set to 0.4, with their sum being 1. Heart rate increase refers to the current heart rate relative to the daily resting baseline. The percentage increase refers to the number of phase response peaks with an amplitude exceeding 0.02 μS per minute. When the heart rate increase exceeds 15% of the resting value and at least 3 significant response peaks appear in the skin conductance per minute, it is considered a state of sympathetic hyperactivation.

[0195] The second item is the parasympathetic inhibition index, which extracts the high-frequency power of heart rate variability. The frequency band is 0.15–0.40 Hz. When When the level drops to below 40% of the individual's resting baseline, parasympathetic activity is considered suppressed.

[0196] The third item is the spatiotemporally corrected loss rate obtained in steps S404 and S405. .

[0197] The fourth item is the performance indicator that is transmitted in real time by the external task system through the standard interface. You can choose either task accuracy or output rate per unit time.

[0198] Step S502, Challenge Stress Assessment.

[0199] The current pressure is classified as challenging pressure when all three of the following conditions are met simultaneously. Specifically:

[0200] Condition 1: The sympathetic nerve activation index is in a high activation state, that is, the threshold condition described in step S501 is met.

[0201] Condition 2, loss rate Located within a preset benign range. The upper bound of the benign range is 1.2 times the average wear rate of the individual during historical moderate load periods, and the lower bound is 0.5 times the average wear rate. Moderate load periods refer to all sampling times in the past 7 days when the individual's task load intensity L is between 0.3 and 0.7.

[0202] Condition 3: External performance indicators remain at a preset high level, such as a task accuracy rate of no less than 80%.

[0203] Challenging stress represents a positive state in which an individual actively mobilizes their physical and mental resources when facing demanding tasks, and is usually accompanied by high performance.

[0204] Step S503, Determination of obstructive pressure.

[0205] When the sympathetic activation index remains above the high activation threshold for more than 10 minutes, and the rate of depletion... If the pressure exceeds the preset warning limit and the parasympathetic inhibition index shows that parasympathetic nerve activity is suppressed, the current pressure will still be judged as obstructive pressure even if the external performance indicators do not show a significant decline.

[0206] The upper limit of the attrition rate warning is set at 90% of the individual's historical maximum sustainable attrition rate. The maximum sustainable value is obtained through an incremental load test during the calibration phase. The test paradigm is as follows: the subject performs a working memory task with progressively increasing difficulty, increasing the difficulty level by one level every 5 minutes, for a total of 10 levels. The steady-state attrition rate at each level is recorded, and the attrition rate corresponding to the highest level that the subject can maintain for more than 5 minutes is taken as the maximum sustainable value.

[0207] Obstacles are a type of exhaustive stress, indicating that although the subjects are still barely maintaining their outward performance, their endogenous regulatory resources are being rapidly depleted.

[0208] Step S504: Adjust the determination of resource depletion stage.

[0209] After identifying it as obstructive stress, physiological recovery indicators during non-task periods were further tracked.

[0210] Record the time required for skin conductance to recover from the activated state to the resting state. The activated state is defined as a state in which the skin conductance level exceeds 50% of the resting baseline, and the recovery state is defined as a return to within 10% of the baseline. If the recovery time exceeds 15 minutes, and the low-frequency to high-frequency power ratio of heart rate variability is also recorded, the recovery is considered a failure. If the value continuously deviates from the equilibrium range, i.e., above 3.0 or below 0.6, the individual is determined to have entered the stage of depletion of regulatory resources.

[0211] Entering the stage of depleted regulatory resources means that even if the task is stopped, the physiological system will have difficulty returning to a stable baseline on its own, and the risk of collapse increases sharply.

[0212] Step S505 triggers early warning of physiological breakdown and spatially differentiated intervention.

[0213] When the remaining adjustment resource stock is calculated in step S406 When the resource level is below a preset threshold and the physiological recovery rate is below a preset recovery threshold, an early warning instruction for physiological collapse is triggered.

[0214] The preset resource threshold is taken as the initial adjustment resource index. The physiological recovery rate is defined as the ratio of the current heart rate recovery rate during the interval period to the individual's historical normal recovery rate, with a preset recovery rate threshold of 50%.

[0215] After the warning instruction is issued, differentiated intervention suggestions are output based on the subject's current spatial location coordinates. If the spatial location is determined to be an office area, the intervention suggestions include mandatory rest reminders, suggestions for workload reallocation, and guidance on cognitive dissociation. If the spatial location is determined to be a non-office area, the intervention suggestions shift to resetting the boundaries between work and home, such as prompting the subject to turn off office equipment, guiding the subject to deep breathing exercises, or mindfulness meditation.

[0216] Step S506: Dynamic prioritization of intervention recommendations.

[0217] The early warning output module divides the intervention into three levels based on the degree of scarcity of remaining adjustment resources.

[0218] Mild resource depletion, existing stock at Between 60% and 80%, output lightweight mindfulness breathing prompts.

[0219] Moderate resource depletion, existing stock Between 40% and 60%, the output environment switching suggestion prompts the subject to leave the current location and move around.

[0220] Severe resource depletion, stockpile below When the resource depletion characteristic described in step S504 is detected at 40% of the target, the warning output module interrupts the current task flow with the highest priority instruction, displays a red warning on the subject's mobile terminal, and automatically sends a status report containing the current resource status and physiological indicators to the preset health management personnel.

[0221] Step S507, hysteresis control with dynamic priority grading.

[0222] The early warning output module sets a 10-minute hysteresis interval between the three intervention levels described in step S506. When upgrading from a low level to a high level, the existing stock must remain below the trigger threshold of the corresponding high level for 10 minutes before switching can proceed; when downgrading from a high level to a low level, the existing stock must remain above the recovery threshold of the corresponding low level for 10 minutes before switching can proceed.

[0223] The recovery threshold is set at 1.05 times the trigger threshold, i.e., the mild recovery threshold is 84%, the moderate recovery threshold is 63%, and the severe recovery threshold is 42%, in order to avoid frequent switching of intervention levels due to short-term fluctuations in the stock around the threshold.

[0224] The above step S5 constitutes a complete decision-making link from the qualitative identification of stressors to the output of graded early warnings. By coupling the dynamic model of regulating resources with multimodal physiological characteristics and external performance, it can not only distinguish between positive challenges and negative consumption, but also capture the signs of physiological collapse before the performance appearance collapses, so as to realize graded intervention with scene perception capabilities, thereby improving the effectiveness and timeliness of early warnings.

[0225] In the above method, step S6 is self-learning optimization based on subject feedback.

[0226] To enable the model to continuously adapt to individual differences and long-term psychological state changes, a self-learning optimization step driven by subject feedback is introduced. This self-learning optimization step is independent of the aforementioned real-time monitoring loop and fine-tunes the model parameters through periodic offline or online incremental updates.

[0227] The system collects subjective feedback from subjects on the accuracy of the warnings periodically or after each warning is lifted via the subjects' mobile terminals. The feedback interface provides two selectable labels: "accurate" and "inaccurate." Subjects make their selections based on their own emotional state at the time and their judgment of the rationality of the warning. Each feedback data is associated with and stored with multimodal physiological feature segments within the 10-minute period prior to the warning, forming sample pairs for incremental learning.

[0228] Simultaneously, the long-term physiological changes of the subjects are continuously recorded, lasting for weeks or months. Macroscopic indicators such as attenuation slope, average loss rate, and changes in recovery efficiency serve as the basis for group comparison and individual correction.

[0229] After the trigger conditions of 200 cumulative new samples or 7 days since the last update are met, incremental learning algorithms are used to update key parameters. Possible incremental learning algorithms include online gradient descent or mini-batch incremental training based on replay mechanisms. The core idea is to mix new samples with a portion of retained typical historical samples and input them into the model, gradually adjusting the loss rate correction coefficient without retraining from scratch. The combined weights of each load index in step S403 And the mapping weights between conscientiousness and emotional intelligence in step S104. and .

[0230] To address the differences in temperament type and age structure among subject groups in different task domains, at the group level, automatic optimization is performed in small steps based on the overall trend of the recent warning feedback accuracy of the corresponding group subjects; at the individual level, for specific subjects whose feedback data frequently does not conform to the actual labels, the mapping weights corresponding to the specific subjects are gradually transferred to a ratio that better reflects the actual regulatory resource structure of the specific subjects.

[0231] Therefore, it can be seen that the intervention logic in steps S6 and S5 forms a closed-loop evolution mechanism. Step S5 uses the current model parameters to assess and intervene in the state of the subjects. The subjective feedback of the subjects on the intervention results and the long-term physiological data are then used by step S6 to fine-tune the model parameters, so that the model can generate more accurate judgments and more appropriate intervention suggestions in the next warning. This allows the method of this embodiment to get rid of the dependence on static experience thresholds after deployment and gradually approach the optimal warning parameter configuration of the individual subjects.

[0232] This embodiment also provides an emotion recognition system based on multimodal spatiotemporal feature fusion, which adopts a distributed architecture of front-end acquisition, cloud computing and terminal feedback, including a resource quantification module, a signal acquisition module, a feature fusion module, an evaluation calculation module and an early warning output module.

[0233] The resource quantification module is used to execute step S1, construct the initial adjustment resource index, and resides in the cloud processing platform in software form.

[0234] The signal acquisition module is used to perform step S2. It consists of a multi-channel sensor group deployed on the subject's body surface, an edge synchronization unit, and an indoor positioning module. Each sensor keeps its clock aligned with the edge synchronization unit via Bluetooth Low Energy or Wi-Fi protocol. The raw data is encapsulated and sent to the cloud processing platform in a structured data stream via a wireless transmission protocol.

[0235] The feature fusion module is used to execute step S3. It is deployed on a high-performance computing cluster in the cloud and uses a pre-trained spatiotemporal convolutional neural network model as the core computing engine to output multimodal global semantic vectors.

[0236] The evaluation calculation module is used to execute step S4. It is deployed in the cloud to realize the real-time calculation of the resource loss rate and the dynamic tracking of the remaining resource inventory. The embedded spatiotemporal context association submodule uses the subject's real-time spatial anchor point and timestamp to determine whether the subject is in a late-night office state and applies scene correction according to step S405.

[0237] The early warning output module is used to execute step S5. The front part is deployed in the cloud to complete the qualitative identification of stress sources, and the back part is a cross-cloud intervention instruction distributor. It performs graded intervention according to steps S505 and S506. The early warning information is delivered to the subject's associated mobile terminal or wearable display device in real time by means of vibration, text pop-up window or voice broadcast.

[0238] The above five modules constitute a complete business chain from establishing a psychological baseline, collecting physiological signals, fusing spatiotemporal features, dynamically calculating resources to stress identification and graded intervention. In the continuous operation, step S6 forms a control logic of assessment, intervention, feedback and optimization, gradually adapting to the individual differences of subjects with different professional backgrounds and age groups.

[0239] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0240] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment includes only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An emotion recognition method based on multimodal spatiotemporal feature fusion, characterized in that, include: Acquire psychological trait data of the subjects and construct an initial regulatory resource index based on the psychological trait data to characterize the psychological energy baseline of the subjects before they are subjected to stress load; The subject's multimodal physiological signals in the current task scenario are collected synchronously. The multimodal physiological signals include at least electroencephalogram (EEG) signals, heart rate variability signals, and skin conductance response signals. The multimodal physiological signals are then processed in a spatiotemporal synchronization manner to form a multi-source heterogeneous physiological dataset. Feature extraction is performed on the multi-source heterogeneous physiological dataset to generate a multimodal spatiotemporal fusion feature vector, which encodes instantaneous physiological activation patterns and long-term psychological evolution trends. Using the multimodal spatiotemporal fusion feature vector, the instantaneous adjustment resource loss rate, which is positively correlated with task load intensity and negatively correlated with adjustment efficiency, is calculated in real time. The instantaneous adjustment resource loss rate is then corrected by the initial adjustment resource index to obtain the effective loss rate corrected by the individual baseline. Based on the effective loss rate, the initial adjustment resource index is cumulatively deducted, and the remaining adjustment resource stock is dynamically calculated. Monitor the decay trend of the remaining regulatory resources, and trigger a pressure overload warning command at least when the decay trend meets the preset overconsumption condition.

2. The emotion recognition method based on multimodal spatiotemporal feature fusion according to claim 1, characterized in that, The acquisition of the subjects' psychological trait data and the construction of an initial moderating resource index based on the psychological trait data include: The subjects' conscientiousness scores were obtained, which were obtained by weighted summation of the scores of the self-discipline sub-dimension, the organization sub-dimension, and the achievement orientation sub-dimension. The subject's emotional intelligence score is obtained by summing the scores of emotional perception, emotional utilization, and emotional understanding. The due diligence dimension score and the emotional intelligence score are normalized respectively to generate a normalized due diligence score and a normalized emotional intelligence score. The normalized due diligence score and the normalized emotional intelligence score are weighted and combined to generate the initial moderating resource index, wherein the weight coefficient of the normalized due diligence score is greater than the weight coefficient of the normalized emotional intelligence score.

3. The emotion recognition method based on multimodal spatiotemporal feature fusion according to claim 1, characterized in that, The method further includes: The sympathetic activation index, parasympathetic inhibition index, effective loss rate, and external performance indicators are extracted from the multimodal spatiotemporal fusion feature vector. When the following conditions are met simultaneously: the sympathetic nerve activation index is in a state of high sympathetic nerve activation, the effective attrition rate is within a preset benign range, and the external performance indicator is maintained at a preset high level, the stress currently faced by the subject is judged as challenging stress. When the sympathetic nerve activation index is continuously higher than the threshold of the sympathetic nerve hyperactivation state for a preset duration, the effective attrition rate exceeds the preset warning limit, and the parasympathetic nerve inhibition index shows that the parasympathetic nerve activity is suppressed, the pressure currently faced by the subject is judged as obstructive pressure, and a warning response corresponding to the obstructive pressure is triggered.

4. The emotion recognition method based on multimodal spatiotemporal feature fusion according to claim 3, characterized in that, The method further includes, after determining that it is obstructive pressure: Track physiological recovery indicators during non-task periods, including the recovery time of skin conductance level from the activated state to the resting baseline, and the low-frequency to high-frequency power ratio of heart rate variability; When the recovery time exceeds the preset recovery threshold and the ratio of low-frequency to high-frequency heart rate variability power continues to deviate from the preset balance range, it is determined that the subject has entered the regulatory resource depletion stage, and an early warning instruction for physiological collapse corresponding to the regulatory resource depletion stage is triggered.

5. The emotion recognition method based on multimodal spatiotemporal feature fusion according to claim 1, characterized in that, The real-time calculation of the instantaneous adjustment resource consumption rate, which is positively correlated with task load intensity and negatively correlated with adjustment efficiency, includes: The relative energy proportion of the EEG β band in the multimodal spatiotemporal fusion feature vector is extracted as a proxy indicator of the amount of psychological effort invested. The reciprocal of the heart rate recovery rate is extracted as a proxy indicator of regulation efficiency. The heart rate recovery rate is the speed at which the heart rate recovers from its peak value to near the resting heart rate baseline during the recovery time of the task interval. The normalized weighted combination of the sympathetic-parasympathetic balance index, the average skin electrical conductivity, and the cortical activation index was extracted and used as the task load intensity. The ratio of the task load intensity to the adjustment efficiency is calculated as the instantaneous adjustment resource loss rate.

6. The emotion recognition method based on multimodal spatiotemporal feature fusion according to claim 1, characterized in that, The instantaneous adjusted resource loss rate is corrected by the initial adjusted resource index to obtain the effective loss rate corrected by the individual baseline, which is achieved in the following way: in, The effective loss rate, For the instantaneous adjustment of resource consumption rate, The initial adjustment resource index. This is an adjustable correction factor.

7. The emotion recognition method based on multimodal spatiotemporal feature fusion according to claim 1 or 6, characterized in that, When collecting the multimodal physiological signals, each sampling point is accompanied by a spatiotemporal anchor point identifier consisting of a high-precision globally unified timestamp and spatial location coordinates; The method further includes: after calculating the effective loss rate, performing spatiotemporal context correction using the spatiotemporal anchor identifier, including: When the spatial coordinates of the subject are detected to be within the preset work area during the preset late night period, it is determined to be an occupational overload risk scenario, and the effective attrition rate is multiplied by a scenario correction factor greater than 1 to obtain the corrected attrition rate. Under the occupational overload risk scenario, the initial adjustment resource index is cumulatively deducted based on the correction loss rate, and the remaining adjustment resource stock is dynamically calculated.

8. The emotion recognition method based on multimodal spatiotemporal feature fusion according to claim 1, characterized in that, After triggering the pressure overload warning command, the system also includes a tiered intervention with a hysteresis control mechanism, based on the degree of scarcity of the remaining regulatory resources, wherein: Set corresponding trigger thresholds and recovery thresholds for mild, moderate, and severe resource loss levels, respectively. When the remaining adjustment resource inventory is continuously lower than the trigger threshold of a certain higher level resource loss level for a first preset duration, the resource loss level will be upgraded from the current level to the higher level resource loss level. Only when the remaining adjustable resource inventory remains above the recovery threshold of the current resource loss level for a second preset duration under the current resource loss level will it be downgraded to a lower resource loss level, in order to avoid frequent switching of intervention levels.

9. The emotion recognition method based on multimodal spatiotemporal feature fusion according to claim 1, characterized in that, Also includes: The system receives subjective feedback tags from subjects regarding the accuracy of the early warning, associates and stores these subjective feedback tags with multimodal physiological feature fragments prior to triggering the early warning, and forms incremental learning samples. The long-term physiological change trend of the subjects is continuously recorded, including the decay slope, average loss rate and recovery efficiency of the initial regulatory resource index. Once the accumulated incremental learning samples meet the preset conditions, the parameters used in calculating the effective loss rate and the initial adjustment resource index are incrementally updated using the incremental learning samples, so that the model can adapt to the individual differences and long-term psychological state changes of the subjects.

10. An emotion recognition system based on multimodal spatiotemporal feature fusion, used to execute the emotion recognition method based on multimodal spatiotemporal feature fusion as described in any one of claims 1-9, characterized in that, include: The resource quantification module is used to acquire the psychological trait data of the subjects and construct an initial regulatory resource index based on the psychological trait data to characterize the psychological energy baseline of the subjects before they bear stress load. The signal acquisition module is used to synchronously acquire the multimodal physiological signals of the subject in the current task scenario, and to perform spatiotemporal synchronization processing on the multimodal physiological signals to form a multi-source heterogeneous physiological dataset. The signal acquisition module includes a multi-channel sensor group, an edge synchronization unit, and an indoor positioning module. The feature fusion module is used to extract features from the multi-source heterogeneous physiological dataset and generate a multimodal spatiotemporal fusion feature vector. The multimodal spatiotemporal fusion feature vector encodes instantaneous physiological activation patterns and long-term psychological evolution trends. The evaluation and calculation module is used to calculate the instantaneous adjustment resource loss rate in real time using the multimodal spatiotemporal fusion feature vector, obtain the effective loss rate by correcting the initial adjustment resource index, dynamically calculate the remaining adjustment resource stock by cumulatively deducting the initial adjustment resource index, and monitor the decay trend of the remaining adjustment resource stock. The early warning output module is used to trigger a pressure overload early warning command at least when the attenuation trend meets a preset excessive consumption condition.

Images