Emotional state dynamic monitoring system and method based on video image spatial-temporal characteristics

Abstract

The invention relates to an emotional state dynamic monitoring system and method, in particular to an emotional state dynamic monitoring system and method based on video image spatio-temporal characteristics, and solves the problems that an existing emotional state monitoring system and method are single in recognition dimension, poor in environmental adaptability, short in time sequence modeling and low in recognition efficiency. The emotion recognition precision is low, and the long-term evolution trend of the emotional state is difficult to effectively capture. The emotional state recognition module is adopted to recognize and quantify the emotional state to obtain the emotional intensity, then the time sequence modeling and trend analysis module is combined to obtain the emotional duration score and the emotional intensity trend, and the real emotional state can be comprehensively reflected; meanwhile, a spatial-temporal feature extraction module is adopted to extract spatial features and time sequence features of the face video data, and time sequence evolution characteristics of an emotional state are reflected through time sequence modeling; in addition, the weight of each frame of image in the face video data is adaptively adjusted by evaluating the environmental factors, so that the emotion recognition precision and stability can be ensured.

Description

Technical Field

[0001] This invention relates to a dynamic monitoring system and method for emotional states, specifically to a dynamic monitoring system and method for emotional states based on the spatiotemporal features of video images. Background Technology

[0002] With the increasing demands for operational safety in transportation, industrial production, and intelligent acceleration sectors, real-time and accurate monitoring of driver emotional states has gradually become a key technological requirement for ensuring operational safety and preventing accidents. Taking the transportation sector as an example, with the rapid growth of vehicle ownership and the development of intelligent driving technology, driver state monitoring technology has become an important means of improving road traffic safety. Statistics show that traffic accidents caused by abnormal driver emotions account for more than 30%, including road rage, fatigued driving, and inattention. Accurately identifying the driver's emotional state is of great significance for preventing traffic accidents and improving driving safety.

[0003] Existing emotion state monitoring systems suffer from the following main problems: First, they lack a single identification dimension. Most existing systems can only identify basic discrete emotion categories, such as joy, anger, sorrow, and fear, lacking quantitative analysis of multi-dimensional information such as emotion intensity, duration, and trends, thus failing to comprehensively reflect the true emotional state. Second, they have poor environmental adaptability. For example, driving environments are characterized by drastic changes in lighting, vibration interference, and multitasking, causing existing emotion state monitoring systems to experience a significant drop in accuracy in complex scenarios, limiting their practicality. Third, they lack temporal modeling. Existing emotion state monitoring systems are mainly based on single-frame images or simple feature fusion, ignoring the temporal evolution of emotional states and failing to capture the dynamic changes in emotions. Fourth, they lack sufficient risk assessment. Existing emotion state monitoring systems lack assessment mechanisms that correlate emotional states with behavioral risks, failing to provide effective support for safety warnings.

[0004] Furthermore, traditional emotion recognition methods primarily rely on static facial expression analysis, extracting facial key point features or using deep learning networks for classification. However, these methods face numerous challenges in driving environments. On one hand, drivers' facial expressions during driving are often subtle and less obvious than in everyday life, making it difficult for traditional methods to accurately capture these subtle changes. On the other hand, factors such as changes in lighting, head movements, and facial occlusion during vehicle movement can significantly affect recognition accuracy, rendering static image-based methods unreliable.

[0005] In recent years, some researchers have begun to explore the integration of multimodal information to improve the accuracy of emotion recognition, such as combining facial expressions and physiological signals. However, these methods often require additional sensor equipment, increasing system complexity and cost, and suffer from problems such as high invasiveness and low user acceptance in practical applications. Furthermore, existing multimodal information fusion methods mostly employ simple feature concatenation or weighted averaging, lacking in-depth modeling of the temporal relationships between different modalities, resulting in limited fusion effectiveness.

[0006] In terms of temporal modeling, although some researchers have begun to focus on the temporal information in video sequences, most methods are still limited to the analysis of short time series and cannot effectively capture the long-term evolution trend of emotional states. At the same time, they lack consideration for the special characteristics of the driving environment and have failed to design algorithm architectures that are adapted to driving scenarios. Summary of the Invention

[0007] The purpose of this invention is to address the technical problems of existing emotion state monitoring systems and methods, such as limited recognition dimensions, poor environmental adaptability, and lack of temporal modeling, which lead to low emotion recognition accuracy and difficulty in effectively capturing the long-term evolution trend of emotion states. The invention provides an emotion state dynamic monitoring system and method based on the spatiotemporal features of video images.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A dynamic monitoring system for emotional states based on the spatiotemporal features of video images is characterized by the following features: a video acquisition and preprocessing module, a spatiotemporal feature extraction module, and an adaptive fusion module connected in sequence, as well as an emotional state recognition module, a temporal modeling and trend analysis module, and a risk assessment and early warning module.

[0010] The video acquisition and preprocessing module is used to acquire facial video data of the monitored object and preprocess it, while the spatiotemporal feature extraction module is used to extract spatial features and temporal features respectively.

[0011] The output of the adaptive fusion module is connected to the input of the emotion state recognition module and the first input of the temporal modeling and trend analysis module, respectively, to dynamically adjust the fusion weights of spatial features and temporal features and fuse them to obtain spatiotemporal fusion features.

[0012] The output of the emotion state recognition module is connected to the second input of the time series modeling and trend analysis module and the first input of the risk assessment and early warning module, respectively. The emotion state recognition module is used to identify the emotion state based on the spatiotemporal fusion characteristics and quantify it to obtain the emotion intensity.

[0013] The output of the time series modeling and trend analysis module is connected to the second input of the risk assessment and early warning module. It is used to analyze the temporal evolution of emotional state based on the characteristics of emotional state and spatiotemporal fusion, and to obtain abnormal information, emotional duration score and emotional intensity trend.

[0014] The risk assessment and early warning module is used to assess risk based on emotional intensity, abnormal information, emotional duration score, and emotional intensity trend, and to issue early warnings to the monitored objects in combination with emotional state and abnormal information.

[0015] Furthermore, the spatiotemporal feature extraction module includes a spatial feature extraction unit and a temporal feature extraction unit;

[0016] The spatial feature extraction unit adopts a ResNet-50 network based on an attention mechanism. The ResNet-50 network based on an attention mechanism includes a convolutional layer, a Block 1 layer, a Block 2 layer, a Block 3 layer and a Block 4 layer connected in sequence, as well as a multi-scale feature fusion layer and a global pooling layer.

[0017] The input of the convolutional layer is connected to the output of the video acquisition and preprocessing module. The outputs of Block1, Block2, Block3, and Block4 are respectively connected to the input of the multi-scale feature fusion layer. The output of the multi-scale feature fusion layer is connected to the input of the global pooling layer. The output of the global pooling layer is connected to the first input of the adaptive fusion module.

[0018] The Block1, Block2, Block3, and Block4 layers are used to extract facial spatial features at different spatial scales, respectively. The multi-scale feature fusion layer is used to fuse facial spatial features at different spatial scales based on channel attention and spatial attention. The global pooling layer is used to perform global pooling on the fused facial spatial features to obtain spatial features.

[0019] The temporal feature extraction unit includes a bidirectional LSTM network, a self-attention network, a position coding network, a layer normalization network, and a temporal fusion network connected in sequence.

[0020] The input of the bidirectional LSTM network is connected to the output of the video acquisition and preprocessing module, and is used to extract facial temporal features at different time scales. The self-attention network, the position encoding network, and the layer normalization network are used to assign weights, encode positions, and normalize the feature components in the facial temporal features, respectively. The output of the temporal fusion network is connected to the second input of the adaptive fusion module, and is used to fuse facial temporal features at different time scales according to position encoding and weights to obtain temporal features.

[0021] Furthermore, the adaptive fusion module includes an environmental factor evaluation unit, a dynamic weight calculation unit, a quality gating unit, an adaptive spatiotemporal feature fusion unit, and a video feature fusion unit;

[0022] The input of the environmental factor evaluation unit is connected to the output of the global pooling layer, the first output is connected to the first input of the dynamic weight calculation unit, and the second output is connected to the input of the quality gating unit. The environmental factor evaluation unit is used to evaluate the environmental factors of the facial video data frame by frame according to the spatial features, obtain the environmental factor score, and obtain the overall quality of each frame of the facial video data based on it.

[0023] The second input of the dynamic weight calculation unit is connected to the output of the temporal fusion network, and the output is connected to the first input of the adaptive spatiotemporal feature fusion unit. It is used to calculate the fusion weight of spatial features and temporal features frame by frame based on environmental factor scores and temporal features.

[0024] The second and third inputs of the adaptive spatiotemporal feature fusion unit are connected to the outputs of the global pooling layer and the temporal fusion network, respectively, and the output is connected to the first input of the video feature fusion unit. The adaptive spatiotemporal feature fusion unit is used to fuse the spatial features and temporal features frame by frame according to the fusion weights of the spatial features and temporal features to obtain the fusion features of each frame of the facial video data.

[0025] The output of the quality gating unit is connected to the second input of the video feature fusion unit, and is used to adjust its weight according to the overall quality of each frame of the facial video data.

[0026] The output of the video feature fusion unit is connected to the input of the emotion state recognition module and the first input of the temporal modeling and trend analysis module, respectively, and is used to fuse the fusion features of all images in the facial video data according to the weights to obtain spatiotemporal fusion features.

[0027] Furthermore, the emotion state recognition module includes a multi-label classifier and an emotion intensity regression unit, and the time series modeling and trend analysis module includes a time series modeling unit and a trend prediction unit;

[0028] The input terminals of the multi-label classifier, the emotion intensity regression unit, and the trend prediction unit, as well as the first input terminal of the temporal modeling unit, are respectively connected to the output terminal of the video feature fusion unit. The output terminal of the multi-label classifier is respectively connected to the second input terminal of the temporal modeling unit and the first input terminal of the risk assessment and early warning module. The multi-label classifier is used to identify emotional states. The output terminal of the emotion intensity regression unit is connected to the first input terminal of the risk assessment and early warning module and is used to quantify the emotional state to obtain the emotion intensity.

[0029] The temporal modeling unit adopts a hidden Markov model, and its output is connected to the second input of the risk assessment and early warning module. The temporal modeling unit is used to identify abnormal information of emotional state and emotional duration score. Abnormal information includes abnormality identifier, abnormality type and abnormality score. The trend prediction unit adopts an ARIMA model, and its output is connected to the second input of the risk assessment and early warning module. The trend prediction unit is used to predict the trend of emotional intensity based on spatiotemporal fusion characteristics.

[0030] Furthermore, the risk assessment and early warning module includes a risk assessment unit, a risk level classification unit, and an intelligent decision-making early warning unit;

[0031] The first, second, and third input terminals of the risk assessment unit are respectively connected to the output terminals of the emotion intensity regression unit, the time series modeling unit, and the trend prediction unit, and the output terminal is connected to the first input terminal of the risk level classification unit. The risk assessment unit is used to assess risk based on emotion intensity, abnormality identification and abnormality score, emotion duration score, and emotion intensity trend to obtain a risk score.

[0032] The second input of the risk level classification unit is connected to the output of the time series modeling unit, and the output is connected to the first input of the intelligent decision-making early warning unit. The risk level classification unit is used to determine the early warning level based on the anomaly identifier, anomaly score, and risk score.

[0033] The second input terminal of the intelligent decision-making early warning unit is connected to the output terminal of the multi-label classifier, and the third input terminal is connected to the output terminal of the time series modeling unit. The intelligent decision-making early warning unit is used to provide personalized early warning schemes for the monitored objects according to the anomaly type, early warning level and emotional state, and to issue early warnings.

[0034] This invention also provides a method for dynamic monitoring of emotional states based on spatiotemporal features of video images. The method employs the aforementioned dynamic monitoring system for emotional states based on spatiotemporal features of video images, and its key feature is that it includes the following steps:

[0035] Step 1: The video acquisition and preprocessing module acquires facial video data of the monitored object, preprocesses it, and then sends it to the spatiotemporal feature extraction module;

[0036] Step 2: The spatiotemporal feature extraction module extracts the spatial and temporal features of the facial video data, and then sends them to the adaptive fusion module for fusion to obtain spatiotemporal fusion features, which are then sent to the emotion state recognition module and the temporal modeling and trend analysis module respectively.

[0037] Step 3: The emotion state recognition module identifies the emotion state based on the spatiotemporal fusion features, quantifies it to obtain the emotion intensity, and then sends the emotion state to the time series modeling and trend analysis module, and sends the emotion state and emotion intensity to the risk assessment and early warning module.

[0038] Step 4: The temporal modeling and trend analysis module identifies abnormal information and emotional duration based on spatiotemporal fusion characteristics and emotional state. It also predicts the emotional intensity trend based on spatiotemporal fusion characteristics, obtains an emotional duration score based on the emotional duration, and sends the abnormal information, emotional duration score, and emotional intensity trend to the risk assessment and early warning module.

[0039] Step 5: The risk assessment and early warning module assesses the risk based on the intensity of emotions, abnormal information, duration of emotions, and trend of emotional intensity. It then issues early warnings to the monitored subjects based on their emotional state and abnormal information. Finally, it returns to Step 1 to continue collecting facial video data of the monitored subjects until the dynamic monitoring of their emotional state is completed.

[0040] Further, in step 4, the abnormal information includes an anomaly identifier, an anomaly type, and an anomaly score; if the anomaly identifier indicates that no anomaly exists, then in step 5, the risk is assessed using the following formula:

[0041] Risk_Score = α•I + β•D + γ•T

[0042] Where Risk_Score is the risk score, I is the emotion intensity, D is the emotion duration score, T is the emotion intensity trend, and α, β, and γ are the weight parameters of I, D, and T, respectively.

[0043] If the anomaly is identified as present, then in step 5, the risk is assessed using the following formula:

[0044] Risk_Score = α•I + β•D + γ•T + η•K

[0045] Where K is the anomaly score, and η is the weighting parameter of K.

[0046] Furthermore, step 2 specifically involves:

[0047] Step 2.1: The spatial feature extraction unit of the spatiotemporal feature extraction module extracts facial spatial features from facial video data according to different spatial scales, and fuses them according to channel attention and spatial attention. Then, the fused facial spatial features are globally pooled to obtain spatial features, and sent to the environmental factor evaluation unit and the adaptive spatiotemporal feature fusion unit of the adaptive fusion module.

[0048] Step 2.2: The temporal feature extraction unit of the spatiotemporal feature extraction module extracts facial temporal features from facial video data according to different time scales, and performs weight allocation, position encoding and normalization on the feature components in the facial temporal features in sequence. Then, the facial temporal features at different time scales are fused according to position encoding and weight to obtain temporal features, and sent to the dynamic weight calculation unit and adaptive spatiotemporal feature fusion unit of the adaptive fusion module.

[0049] Step 2.3: The environmental factor evaluation unit of the adaptive fusion module evaluates the environmental factors of the facial video data frame by frame according to the spatial features, obtains the environmental factor score, and obtains the overall quality of each frame of the facial video data based on it. Then, the environmental factor score is sent to the dynamic weight calculation unit, and the overall quality of each frame of the facial video data is sent to the quality gating unit.

[0050] Step 2.4: The dynamic weight calculation unit obtains the fusion weights of spatial and temporal features of each frame of the facial video data based on the environmental factor score and temporal features, and sends them to the adaptive spatiotemporal feature fusion unit. The adaptive spatiotemporal feature fusion unit fuses the spatial and temporal features frame by frame to obtain the spatiotemporal features of each frame of the facial video data, and sends them to the video feature fusion unit.

[0051] The quality gating unit adjusts the weights of each frame of the facial video data based on the overall quality of the image and sends them to the video feature fusion unit.

[0052] Step 2.5: The video feature fusion unit fuses the spatiotemporal features of all images in the facial video data according to the weights to obtain spatiotemporal fusion features, and sends them to the emotion state recognition module and the temporal modeling and trend analysis module respectively.

[0053] Furthermore, in step 1, the specific method of preprocessing is as follows:

[0054] A. Image stabilization: A feature point matching-based stabilization algorithm is adopted. The Harris corner detection algorithm is used to detect stable feature points. Then, the Lucas-Kanade optical flow estimation algorithm is used to track the motion trajectory of the feature points. Finally, the RANSAC algorithm is used to estimate the geometric transformation relationship between each frame of the facial video data, and affine transformation is performed based on it. Motion compensation is then performed using interpolation methods.

[0055] B. Illumination Normalization: The multi-scale Retinex algorithm is used for illumination equalization, combined with the restricted adaptive histogram equalization algorithm to enhance local contrast, and then the brightness is adaptively adjusted through gamma correction.

[0056] C. Face detection and tracking: Multi-scale face detection is performed using the MTCNN network, and the facial motion trajectory is stably tracked using a Kalman filter, followed by bounding box optimization.

[0057] D. Key Region Extraction: Using the 68-point marking method, the eye, mouth, and eyebrow regions are accurately located based on the Dlib shape predictor. Then, the corresponding regions of interest are extracted and scale normalization and geometric correction are performed.

[0058] Furthermore, in step 2.3, the environmental factors include light intensity, head posture, facial occlusion, and image sharpness;

[0059] In step 3, the emotional state includes emotional category, emotional value, and emotional activation level.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] 1. The present invention provides a dynamic monitoring system for emotional state based on the spatiotemporal features of video images. It uses an emotional state recognition module to identify and quantify emotional state to obtain emotional intensity. Then, it combines a temporal modeling and trend analysis module to analyze the temporal evolution of emotional state to obtain the duration and intensity trend of emotion. This can obtain multi-dimensional information, comprehensively reflect the real emotional state, and improve the accuracy of emotion recognition.

[0062] 2. The emotional state dynamic monitoring system based on video image spatiotemporal features provided by the present invention uses a video acquisition and preprocessing module to acquire facial video data of the monitored object, and combines a spatiotemporal feature extraction module to extract spatial features and temporal features respectively. It can reflect the temporal evolution characteristics of emotional state through temporal modeling and effectively capture the long-term evolution trend of emotional state.

[0063] 3. The emotional state dynamic monitoring system based on spatiotemporal features of video images provided by the present invention uses a ResNet-50 network based on attention mechanism for spatial feature extraction unit. It enhances feature expression ability through multi-scale feature fusion and attention mechanism. The temporal feature extraction unit uses a bidirectional LSTM network and combines it with a self-attention network, which can capture emotional change patterns at different time scales, thereby further improving the accuracy of emotion recognition.

[0064] 4. The emotional state dynamic monitoring system based on the spatiotemporal features of video images provided by the present invention uses a dynamic weight calculation unit to calculate the weights of spatial features and temporal features in the adaptive fusion module. Combined with the quality gating unit, the weights are automatically reduced when the overall quality of the image is low, which can avoid noise interference and further improve the accuracy of emotion recognition.

[0065] 5. The dynamic monitoring method for emotional state based on the spatiotemporal features of video images provided by this invention analyzes the temporal relationship between consecutive frames in video image data to construct a dynamic evolution model of emotional state. It can capture subtle features such as micro-expression changes and emotional transition processes. By establishing feature representations at multiple time scales, it can simultaneously focus on short-term micro-expression changes and long-term emotional trend evolution, significantly improving the accuracy and robustness of emotion recognition.

[0066] 6. The present invention provides a dynamic monitoring method for emotional state based on the spatiotemporal features of video images. It evaluates the environmental factors of each frame of facial video data and adaptively adjusts the fusion weights of spatial features and temporal features, as well as the weights of each frame itself, to ensure recognition accuracy and stability under complex conditions such as changes in lighting, vibration interference, and head movement.

[0067] 7. The dynamic monitoring method for emotional state based on the spatiotemporal features of video images provided by this invention identifies emotional states, calculates emotional duration scores, predicts emotional intensity trends, and constructs a multi-dimensional feature vector of emotional states, providing a rich data foundation for subsequent risk assessment; at the same time, by combining discrete emotion recognition with continuous emotion dimension analysis, it can more accurately reflect the true emotional state.

[0068] 8. The present invention provides a dynamic monitoring method for emotional state based on the spatiotemporal features of video images, which uses the predicted trend of emotional intensity to assess risk, and achieves a technological breakthrough from passive identification to proactive early warning. Attached Figure Description

[0069] Figure 1 This is a system schematic diagram according to an embodiment of the present invention;

[0070] Figure 2 This is a schematic diagram of the overall structure of an embodiment of the present invention;

[0071] Figure 3 This is a schematic diagram of the spatial feature extraction unit in an embodiment of the present invention;

[0072] Figure 4 This is a schematic diagram of the temporal feature extraction unit in an embodiment of the present invention;

[0073] Figure 5 This is a schematic diagram of the adaptive fusion module in an embodiment of the present invention;

[0074] Figure 6 This is a schematic diagram of the risk assessment and early warning module in an embodiment of the present invention. Detailed Implementation

[0075] The following detailed description, with reference to the accompanying drawings and specific embodiments, uses the dynamic monitoring of a driver's emotional state as an example to further illustrate the present invention's proposed system and method for dynamic monitoring of emotional states based on the spatiotemporal features of video images. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0076] A dynamic monitoring system for emotional states based on the spatiotemporal features of video images, such as... Figure 1 As shown, it includes a video acquisition and preprocessing module, a spatiotemporal feature extraction module, and an adaptive fusion module connected in sequence, as well as an emotion state recognition module, a time series modeling and trend analysis module, and a risk assessment and early warning module.

[0077] The video acquisition and preprocessing module is used to acquire and preprocess the driver's facial video data. This module employs a high-resolution infrared camera with a resolution of at least 1920×1080 and a frame rate of 30fps, mounted directly in front of the driver on the dashboard or rearview mirror to ensure clear capture of the driver's face. The high-resolution infrared camera is equipped with autofocus and light compensation functions, enabling it to adapt to different lighting conditions and operate stably in both bright daylight and low-light environments.

[0078] like Figure 2 As shown, the spatiotemporal feature extraction module includes a spatial feature extraction unit and a temporal feature extraction unit, used to extract spatial and temporal features respectively. The adaptive fusion module includes an environmental factor assessment unit, a dynamic weight calculation unit, a quality gating unit, an adaptive spatiotemporal feature fusion unit, and a video feature fusion unit, used to dynamically adjust the fusion weights of spatial and temporal features and fuse them to obtain spatiotemporal fused features. The emotion state recognition module includes a multi-label classifier and an emotion intensity regression unit, used to identify and quantify emotion states based on spatiotemporal fusion features to obtain emotion intensity. The temporal modeling and trend analysis module includes a temporal modeling unit and a trend prediction unit, used to analyze the temporal evolution of emotion states based on emotion states and spatiotemporal fusion features, obtaining abnormal information, emotion duration scores, and emotion intensity trends. The risk assessment and early warning module includes a risk assessment unit, a risk level classification unit, and an intelligent decision-making early warning unit, used to assess driving risks based on emotion intensity, abnormal information, emotion duration scores, and emotion intensity trends, and to issue early warnings to monitored objects based on emotion states and abnormal information.

[0079] The spatiotemporal feature extraction module uses a deep learning network to extract spatial and temporal features of the face, including multi-dimensional information such as micro-expressions, eye movement patterns, and head posture. For example... Figure 3As shown, the spatial feature extraction unit employs a ResNet-50 network (Residual Network) based on an attention mechanism, which enhances feature representation capabilities. It extracts features at different network levels, combining shallow detail information with deep semantic information to form richer feature representations. The attention-based ResNet-50 network includes sequentially connected convolutional layers, Block 1, Block 2, Block 3, and Block 4, as well as a multi-scale feature fusion layer and a global pooling layer. The input of the convolutional layers is connected to the output of the video acquisition and preprocessing module. The outputs of Block 1, Block 2, Block 3, and Block 4 are connected to the input of the multi-scale feature fusion layer, which is connected to the input of the global pooling layer. The output of the global pooling layer is connected to the input of the environmental factor evaluation unit and the second input of the adaptive spatiotemporal feature fusion unit. Block 1, Block 2, Block 3, and Block 4 are used to extract facial spatial features at different spatial scales. The multi-scale feature fusion layer fuses facial spatial features at different spatial scales based on channel attention and spatial attention. Channel attention learns the importance weights of different feature channels through global average pooling and a multilayer perceptron, while spatial attention learns the importance distribution of spatial locations through convolutional operations, thereby highlighting key emotion-related regions. The global pooling layer performs global pooling on the fused facial spatial features to obtain spatial features.

[0080] like Figure 4As shown, the temporal feature extraction unit comprises a bidirectional LSTM (Long Short-Term Memory) network, a self-attention network, a positional encoding network, a layer normalization network, and a temporal fusion network connected in sequence. The input of the bidirectional LSTM network is connected to the output of the video acquisition and preprocessing module, used to extract facial temporal features at different time scales. The bidirectional LSTM network can simultaneously model forward and backward temporal dependencies, with time scales including time windows of 30 frames and 300 frames. A 30-frame time window is used to capture micro-expressions and rapid emotional changes, effectively detecting subtle changes in facial expressions; a 300-frame time window is used to analyze changes in emotional states, capturing the long-term evolution trend of emotional states. The self-attention network, positional encoding network, and layer normalization network are used to perform weight allocation, positional encoding, and normalization of the feature components in the facial temporal features, respectively. The output of the temporal fusion network is connected to the second input of the dynamic weight calculation unit and the third input of the adaptive spatiotemporal feature fusion unit, respectively. This allows for the fusion of facial temporal features at different time scales based on positional encoding and weights, resulting in temporal features. The temporal fusion network can systematically learn the weight of each time scale on emotional states, forming a comprehensive temporal representation. This enables the system to sensitively detect short-term micro-expression changes while stably analyzing long-term emotional trend evolution.

[0081] The adaptive fusion module can dynamically adjust the fusion weights of spatial and temporal features according to environmental conditions, achieving optimal fusion of multi-source features. For example... Figure 5As shown, the first output of the environmental factor evaluation unit is connected to the first input of the dynamic weight calculation unit, and the second output is connected to the input of the quality gating unit. The environmental factor evaluation unit is used to evaluate the environmental factors of the facial video data frame by frame based on spatial features, obtain an environmental factor score, and obtain the overall quality of each frame of the facial video data based on the score. The output of the dynamic weight calculation unit is connected to the first input of the adaptive spatiotemporal feature fusion unit, and is used to calculate the fusion weights of spatial and temporal features frame by frame using a multilayer perceptron based on the environmental factor score and temporal features. The dynamic weight calculation unit can learn the nonlinear mapping relationship between environmental factors and optimal fusion weights. The output of the adaptive spatiotemporal feature fusion unit is connected to the first input of the video feature fusion unit, and is used to fuse the spatial and temporal features frame by frame based on the fusion weights of the spatial and temporal features, to obtain the fusion features of each frame of the facial video data. The output of the quality gating unit is connected to the second input of the video feature fusion unit. It adjusts the weights of each frame in the facial video data based on the overall quality, automatically reducing weights when image quality is low to avoid noise interference. This effectively prevents low-quality data from negatively impacting overall performance and improves the system's robustness in complex scenarios. The output of the video feature fusion unit is connected to the inputs of the multi-label classifier, the emotion intensity regression unit, the trend prediction unit, and the first input of the temporal modeling unit, respectively. The video feature fusion unit fuses the fusion features of all images in the video image data according to their weights to obtain spatiotemporal fusion features.

[0082] The output of the multi-label classifier is connected to the second input of the temporal modeling unit and the second input of the intelligent decision-making and early warning unit, respectively. The multi-label classifier is used to identify emotional states. The output of the emotion intensity regression unit is connected to the first input of the risk assessment unit to quantify emotional states and obtain emotion intensity. The multi-label classifier adopts a multi-task learning architecture, simultaneously predicting emotional states in three dimensions: emotion category, emotional value, and emotional activation, thus providing a more comprehensive description of the driver's emotional state. Emotion categories include seven types: neutral, happy, angry, sad, fearful, surprised, and disgusted; emotional value is divided into three categories: positive, neutral, and negative; and emotional activation is divided into three levels: low, medium, and high. The emotion intensity regression unit uses a regression network to predict continuous intensity values ​​within the range of 0-1. The network structure includes multiple fully connected layers and a sigmoid activation function, enabling it to output accurate quantification results of emotion intensity. The emotional intensity I was divided into five levels: 0.0≤I≤0.2 indicates very low, 0.2<I≤0.4 indicates low, 0.4<I≤0.6 indicates moderate, 0.6<I≤0.8 indicates high, and 0.8<I≤1.0 indicates very high, providing detailed intensity information for subsequent driving risk assessment.

[0083] The temporal modeling unit employs a Hidden Markov Model (HMM). Its output is connected to the third input of the risk assessment unit, the second input of the risk level classification unit, and the third input of the intelligent decision-making and early warning unit. The temporal modeling unit identifies anomalous information about emotional states and scores the duration of emotions. Anomalous information includes anomaly identifiers, anomaly types, and anomaly scores. The HMM models the transition probabilities between different emotional states. By analyzing historical data, it learns normal emotional transition patterns and then compares the current emotional state with previous monitoring results to determine if abnormal emotional state patterns exist. This provides crucial input information for risk assessment and early warning. Abnormal emotional state patterns include rapid emotional fluctuations, prolonged negative emotions, and anomalous state transitions. The trend prediction unit employs an ARIMA (Autoregressive Integrated Moving Average) model. Its output is connected to the second input of the risk assessment unit. The trend prediction unit predicts the trend of emotional intensity based on spatiotemporal fusion characteristics. The ARIMA model establishes a predictive model by analyzing the autoregressive, differencing, and moving average characteristics of emotional states, enabling the system to issue early warnings before emotional problems worsen and to take appropriate preventative measures.

[0084] like Figure 6As shown, the output of the risk assessment unit is connected to the first input of the risk level classification unit. It assesses risk based on emotional intensity, anomaly identification and score, emotional duration score, and emotional intensity trend to obtain a risk score. The output of the risk level classification unit is connected to the first input of the intelligent decision-making early warning unit. The risk level classification unit determines the early warning level based on the anomaly identification, score, and risk score. The intelligent decision-making early warning unit provides personalized early warning plans for the monitored object based on the anomaly type, early warning level, and emotional state, and issues an early warning. In this embodiment, three different early warning levels are designed: low risk, medium risk, and high risk. Low risk corresponds to a risk score of 0.0 ≤ Risk_Score ≤ 0.3, indicating a normal emotional state, requiring a Level 1 response and routine monitoring without special intervention. Medium risk corresponds to a risk score of 0.3 < Risk_Score ≤ 0.7, indicating a minor anomaly detected, requiring a Level 2 response, issuing a mild reminder and suggesting appropriate attention. High risk corresponds to a risk score of 0.7 < Risk_Score ≤ 1.0, indicating a serious anomaly detected, requiring a Level 3 response and immediate intervention. If an anomaly is detected, the early warning level is automatically increased based on the anomaly score. For anxiety and anger, the warning system includes playing soothing music, adjusting the in-car temperature and light intensity; for fatigue, the warning system includes voice reminders to rest and adjusting the seat angle for comfort; for high-risk situations, the warning system includes activating the driver assistance mode, finding a safe place to park and rest, and even activating the emergency assistance function to contact relevant personnel.

[0085] This embodiment also provides a method for dynamic monitoring of emotional states based on spatiotemporal features of video images. Employing the aforementioned dynamic monitoring system for emotional states based on spatiotemporal features of video images, the method includes the following steps:

[0086] Step 1: The video acquisition and preprocessing module acquires driver facial video image data, preprocesses it to obtain video image data, and sends it to the spatiotemporal feature extraction module. The specific preprocessing method is as follows:

[0087] A. Image Stabilization: A feature point matching-based stabilization algorithm is adopted. Stable feature points are detected by Harris corner detection, and then the Lucas-Kanade optical flow estimation algorithm is used to track the motion trajectory of the feature points. Finally, the RANSAC (Random Sample Consensus) algorithm is used to estimate the geometric transformation relationship between each frame of the facial video data, and affine transformation is performed based on it. Motion compensation is then performed by interpolation method.

[0088] B. Illumination Normalization: The multi-scale Retinex algorithm (Retina and Cortex, an image enhancement algorithm based on the human visual system) is used for illumination equalization, and combined with the restricted adaptive histogram equalization algorithm to enhance local contrast. Then, the brightness is adaptively adjusted through gamma correction.

[0089] C. Face detection and tracking: Multi-task Cascaded Convolutional Network (MTCNN) is used for multi-scale face detection, and Kalman filter is used to stably track facial motion trajectory, followed by bounding box optimization.

[0090] D. Key Region Extraction Unit: Using the 68-point labeling method, the eye, mouth, and eyebrow regions are accurately located based on the Dlib shape predictor (the core module for facial key point detection in the Dlib open-source machine learning / computer vision library). Then, the corresponding regions of interest are extracted and scale normalization and geometric correction are performed.

[0091] Step 2: The spatiotemporal feature extraction module extracts the spatial and temporal features of the facial video data, and then sends them to the adaptive fusion module for fusion to obtain spatiotemporal fused features, which are then sent to the emotion state recognition module and the temporal modeling and trend analysis module, respectively. Specifically:

[0092] Step 2.1: The spatial feature extraction unit of the spatiotemporal feature extraction module extracts facial spatial features from facial video data according to different spatial scales, and fuses them according to channel attention and spatial attention. Then, the fused facial spatial features are globally pooled to obtain spatial features, which are then sent to the environmental factor evaluation unit and the adaptive spatiotemporal feature fusion unit of the adaptive fusion module.

[0093] Step 2.2: The temporal feature extraction unit of the spatiotemporal feature extraction module extracts facial temporal features from facial video data according to different time scales, and performs weight allocation, position encoding and normalization on the feature components in the facial temporal features in sequence. Then, the facial temporal features at different time scales are fused according to position encoding and weight to obtain temporal features, and sent to the dynamic weight calculation unit and adaptive spatiotemporal feature fusion unit of the adaptive fusion module.

[0094] Step 2.3: The environmental factor evaluation unit of the adaptive fusion module evaluates the environmental factors of the facial video data frame by frame based on spatial features, obtains an environmental factor score, and uses this score to determine the overall quality of each frame in the facial video data. The environmental factor score is then sent to the dynamic weight calculation unit, and the overall quality of each frame in the facial video data is sent to the quality gating unit. The environmental factors include illumination intensity, image sharpness, facial occlusion, and head pose. Illumination intensity is quantified by calculating the ratio of the variance to the mean of image brightness. Image sharpness is evaluated using the Laplacian operator to calculate image sharpness and assess its focus quality. Facial occlusion is evaluated by analyzing the visibility of key points and calculating the proportion of occluded key points. Head pose is evaluated by calculating Euler angles based on facial geometry to assess the degree of head deflection relative to the frontal position.

[0095] Step 2.4: The dynamic weight calculation unit obtains the fusion weights of spatial and temporal features of each frame of the facial video data based on the environmental factor score and temporal features, and sends them to the adaptive spatiotemporal feature fusion unit. The adaptive spatiotemporal feature fusion unit fuses the spatial and temporal features frame by frame to obtain the spatiotemporal features of each frame of the facial video data, and sends them to the video feature fusion unit.

[0096] The quality gating unit adjusts the weights of each frame of the facial video data based on the overall quality of the image and sends them to the video feature fusion unit.

[0097] Step 2.5: The video feature fusion unit fuses the spatiotemporal features of all images in the facial video data according to the weights to obtain spatiotemporal fusion features, and sends them to the emotion state recognition module and the temporal modeling and trend analysis module respectively.

[0098] Step 3: The emotion state recognition module identifies the emotion state based on spatiotemporal fusion features, quantifies it to obtain the emotion intensity, and then sends the emotion state to the time series modeling and trend analysis module, and sends the emotion state and emotion intensity to the risk assessment and early warning module. The emotion state includes emotion category, emotion value, and emotion activation level.

[0099] Step 4: The temporal modeling and trend analysis module identifies abnormal information and emotional duration based on spatiotemporal fusion characteristics and emotional state. It also predicts the emotional intensity trend based on spatiotemporal fusion characteristics, obtains an emotional duration score based on the emotional duration, and sends the abnormal information, emotional duration score, and emotional intensity trend to the risk assessment and early warning module.

[0100] Step 5: The risk assessment and early warning module assesses driving risk based on emotional intensity, abnormal information, emotional duration score, and emotional intensity trend. It then issues an early warning to the driver based on the emotional state and abnormal information. The system then returns to Step 1 to continue collecting facial video data from the driver until dynamic monitoring of the emotional state is complete. If an abnormality is flagged as non-existent, the risk is assessed using the following formula:

[0101] Risk_Score = α·I+β·D+γ·T

[0102] Where Risk_Score is the risk score, I is the emotion intensity, D is the emotion duration score, T is the emotion intensity trend, and α, β, and γ are the weight parameters of I, D, and T, respectively. In this embodiment, α, β, and γ are taken as 0.5, 0.3, and 0.2, respectively.

[0103] In other embodiments, a contextual factor C may be incorporated into the risk assessment. Contextual factor C is scored based on driving time, road conditions, weather, and vehicle speed. The above formula can be written as: Risk_Score = α•I+β•D+γ•T+δ•C, where δ is the weighting parameter of C. In this case, α, β, γ, and δ are taken as 0.4, 0.3, 0.2, and 0.1, respectively.

[0104] If an anomaly is identified, the anomaly score will be used as the primary factor in assessing risk, and the values ​​of α, β, and γ will be adjusted as shown in the following formula:

[0105] Risk_Score = α•I + β•D + γ•T + η•K

[0106] Where K is the anomaly score, and η is the weighting parameter of K.

[0107] The dynamic emotion state monitoring system and method based on the spatiotemporal features of video images provided in this embodiment significantly outperforms existing technologies in several key indicators. In terms of recognition accuracy, the basic emotion recognition accuracy reaches over 92%, a significant improvement compared to approximately 85% in existing technologies. The mean absolute error of emotion intensity prediction is controlled within 0.08, a substantial improvement compared to 0.15 in existing technologies. The temporal consistency index reaches over 0.91, indicating that the system can maintain good predictive stability over time series.

[0108] In terms of real-time performance, the system processing latency is controlled within 50ms, a significant improvement compared to the approximately 120ms of existing technologies, meeting the stringent requirements of real-time processing. Frame rate processing capability reaches over 30fps, enabling the processing of high-definition video streams without frame drops. Memory usage is controlled within 500MB, more efficient than the approximately 800MB of existing technologies, facilitating deployment in automotive environments.

[0109] In terms of environmental adaptability, the system supports a light range of 10-50,000 lux, covering various conditions from low light at night to strong sunlight. The head posture tolerance angle reaches ±45°, capable of handling normal head movements during driving. Facial occlusion tolerance reaches 30% of the facial area, handling partial occlusion situations such as wearing glasses or masks.

[0110] This invention proposes a dynamic monitoring system and method for emotional states based on the spatiotemporal features of video images. By deeply mining information in the temporal dimension, it achieves accurate modeling of the emotional state change process, capturing richer emotional change information compared to traditional static analysis methods. The adaptive fusion module can dynamically adjust the recognition strategy according to real-time environmental conditions, maintaining high-precision recognition even in complex scenarios such as changes in lighting, vibration interference, and head movements. This significantly improves recognition performance in complex environments and solves the problem of poor environmental adaptability in existing technologies. Multi-dimensional emotional quantitative assessment provides a more comprehensive description of emotional states, offering a rich data foundation for risk assessment. Furthermore, risk warning based on emotional evolution patterns shifts from passive identification to proactive prevention, possessing significant security value.

[0111] The present invention provides a dynamic emotional state monitoring system and method based on the spatiotemporal features of video images. It requires only a single camera to achieve full functionality, resulting in low hardware costs, simple installation, high operating efficiency, and the ability to meet real-time processing requirements. It is suitable for operation in resource-constrained in-vehicle environments. Furthermore, it exhibits strong adaptability to various complex environments, operating stably under different lighting, posture, and occlusion conditions. Its excellent scalability allows for seamless integration with existing in-vehicle systems, facilitating industrialization. In addition, the non-contact monitoring method does not affect normal operation, providing a good user experience. A multi-level early warning mechanism effectively reduces the risk of false alarms and missed alarms, improving the system's reliability. Real-time risk assessment can promptly identify potential safety issues, providing effective safety protection for the monitored objects.

Claims

1. A dynamic monitoring system for emotional states based on spatiotemporal features of video images, characterized in that: It includes a video acquisition and preprocessing module, a spatiotemporal feature extraction module, and an adaptive fusion module connected in sequence, as well as an emotion state recognition module, a time series modeling and trend analysis module, and a risk assessment and early warning module; The video acquisition and preprocessing module is used to acquire facial video data of the monitored object and preprocess it, while the spatiotemporal feature extraction module is used to extract spatial features and temporal features respectively. The output of the adaptive fusion module is connected to the input of the emotion state recognition module and the first input of the temporal modeling and trend analysis module, respectively, to dynamically adjust the fusion weights of spatial features and temporal features and fuse them to obtain spatiotemporal fusion features. The output of the emotion state recognition module is connected to the second input of the time series modeling and trend analysis module and the first input of the risk assessment and early warning module, respectively. The emotion state recognition module is used to identify the emotion state based on the spatiotemporal fusion characteristics and quantify it to obtain the emotion intensity. The output of the time series modeling and trend analysis module is connected to the second input of the risk assessment and early warning module. It is used to analyze the temporal evolution of emotional state based on the characteristics of emotional state and spatiotemporal fusion, and to obtain abnormal information, emotional duration score and emotional intensity trend. The risk assessment and early warning module is used to assess risk based on emotional intensity, abnormal information, emotional duration score, and emotional intensity trend, and to issue early warnings to the monitored objects in combination with emotional state and abnormal information.

2. The dynamic monitoring system for emotional state based on spatiotemporal features of video images according to claim 1, characterized in that: The spatiotemporal feature extraction module includes a spatial feature extraction unit and a temporal feature extraction unit; The spatial feature extraction unit adopts a ResNet-50 network based on an attention mechanism. The ResNet-50 network based on an attention mechanism includes a convolutional layer, a Block 1 layer, a Block 2 layer, a Block 3 layer and a Block 4 layer connected in sequence, as well as a multi-scale feature fusion layer and a global pooling layer. The input of the convolutional layer is connected to the output of the video acquisition and preprocessing module. The outputs of Block1, Block2, Block3, and Block4 are respectively connected to the input of the multi-scale feature fusion layer. The output of the multi-scale feature fusion layer is connected to the input of the global pooling layer. The output of the global pooling layer is connected to the first input of the adaptive fusion module. The Block1, Block2, Block3, and Block4 layers are used to extract facial spatial features at different spatial scales, respectively. The multi-scale feature fusion layer is used to fuse facial spatial features at different spatial scales based on channel attention and spatial attention. The global pooling layer is used to perform global pooling on the fused facial spatial features to obtain spatial features. The temporal feature extraction unit includes a bidirectional LSTM network, a self-attention network, a position coding network, a layer normalization network, and a temporal fusion network connected in sequence. The input of the bidirectional LSTM network is connected to the output of the video acquisition and preprocessing module, and is used to extract facial temporal features at different time scales. The self-attention network, the position encoding network, and the layer normalization network are used to assign weights, encode positions, and normalize the feature components in the facial temporal features, respectively. The output of the temporal fusion network is connected to the second input of the adaptive fusion module, and is used to fuse facial temporal features at different time scales according to position encoding and weights to obtain temporal features.

3. The dynamic monitoring system for emotional state based on spatiotemporal features of video images according to claim 2, characterized in that: The adaptive fusion module includes an environmental factor evaluation unit, a dynamic weight calculation unit, a quality gating unit, an adaptive spatiotemporal feature fusion unit, and a video feature fusion unit; The input of the environmental factor evaluation unit is connected to the output of the global pooling layer, the first output is connected to the first input of the dynamic weight calculation unit, and the second output is connected to the input of the quality gating unit. The environmental factor evaluation unit is used to evaluate the environmental factors of the facial video data frame by frame according to the spatial features, obtain the environmental factor score, and obtain the overall quality of each frame of the facial video data based on it. The second input of the dynamic weight calculation unit is connected to the output of the temporal fusion network, and the output is connected to the first input of the adaptive spatiotemporal feature fusion unit. It is used to calculate the fusion weight of spatial features and temporal features frame by frame based on environmental factor scores and temporal features. The second and third inputs of the adaptive spatiotemporal feature fusion unit are connected to the outputs of the global pooling layer and the temporal fusion network, respectively, and the output is connected to the first input of the video feature fusion unit. The adaptive spatiotemporal feature fusion unit is used to fuse the spatial features and temporal features frame by frame according to the fusion weights of the spatial features and temporal features to obtain the fusion features of each frame of the facial video data. The output of the quality gating unit is connected to the second input of the video feature fusion unit, and is used to adjust its weight according to the overall quality of each frame of the facial video data. The output of the video feature fusion unit is connected to the input of the emotion state recognition module and the first input of the temporal modeling and trend analysis module, respectively, and is used to fuse the fusion features of all images in the facial video data according to the weights to obtain spatiotemporal fusion features.

4. The dynamic monitoring system for emotional state based on spatiotemporal features of video images according to claim 3, characterized in that: The emotion state recognition module includes a multi-label classifier and an emotion intensity regression unit, and the time series modeling and trend analysis module includes a time series modeling unit and a trend prediction unit. The input terminals of the multi-label classifier, the emotion intensity regression unit, and the trend prediction unit, as well as the first input terminal of the temporal modeling unit, are respectively connected to the output terminal of the video feature fusion unit. The output terminal of the multi-label classifier is respectively connected to the second input terminal of the temporal modeling unit and the first input terminal of the risk assessment and early warning module. The multi-label classifier is used to identify emotional states. The output terminal of the emotion intensity regression unit is connected to the first input terminal of the risk assessment and early warning module and is used to quantify the emotional state to obtain the emotion intensity. The temporal modeling unit adopts a hidden Markov model, and its output is connected to the second input of the risk assessment and early warning module. The temporal modeling unit is used to identify abnormal information of emotional state and emotional duration score. Abnormal information includes abnormality identifier, abnormality type and abnormality score. The trend prediction unit adopts an ARIMA model, and its output is connected to the second input of the risk assessment and early warning module. The trend prediction unit is used to predict the trend of emotional intensity based on spatiotemporal fusion characteristics.

5. The dynamic monitoring system for emotional state based on spatiotemporal features of video images according to claim 4, characterized in that: The risk assessment and early warning module includes a risk assessment unit, a risk level classification unit, and an intelligent decision-making early warning unit; The first, second, and third input terminals of the risk assessment unit are respectively connected to the output terminals of the emotion intensity regression unit, the time series modeling unit, and the trend prediction unit, and the output terminal is connected to the first input terminal of the risk level classification unit. The risk assessment unit is used to assess risk based on emotion intensity, abnormality identification and abnormality score, emotion duration score, and emotion intensity trend to obtain a risk score. The second input of the risk level classification unit is connected to the output of the time series modeling unit, and the output is connected to the first input of the intelligent decision-making early warning unit. The risk level classification unit is used to determine the early warning level based on the anomaly identifier, anomaly score, and risk score. The second input terminal of the intelligent decision-making early warning unit is connected to the output terminal of the multi-label classifier, and the third input terminal is connected to the output terminal of the time series modeling unit. The intelligent decision-making early warning unit is used to provide personalized early warning schemes for the monitored objects according to the anomaly type, early warning level and emotional state, and to issue early warnings.

6. A method for dynamic monitoring of emotional states based on spatiotemporal features of video images, employing the dynamic monitoring system for emotional states based on spatiotemporal features of video images as described in any one of claims 1-5, characterized in that, Includes the following steps: Step 1: The video acquisition and preprocessing module acquires facial video data of the monitored object, preprocesses it, and then sends it to the spatiotemporal feature extraction module; Step 2: The spatiotemporal feature extraction module extracts the spatial and temporal features of the facial video data, and then sends them to the adaptive fusion module for fusion to obtain spatiotemporal fusion features, which are then sent to the emotion state recognition module and the temporal modeling and trend analysis module respectively. Step 3: The emotion state recognition module identifies the emotion state based on the spatiotemporal fusion features, quantifies it to obtain the emotion intensity, and then sends the emotion state to the time series modeling and trend analysis module, and sends the emotion state and emotion intensity to the risk assessment and early warning module. Step 4: The temporal modeling and trend analysis module identifies abnormal information and emotional duration based on spatiotemporal fusion characteristics and emotional state. It also predicts the emotional intensity trend based on spatiotemporal fusion characteristics, obtains an emotional duration score based on the emotional duration, and sends the abnormal information, emotional duration score, and emotional intensity trend to the risk assessment and early warning module. Step 5: The risk assessment and early warning module assesses the risk based on the intensity of emotions, abnormal information, duration of emotions, and trend of emotional intensity. It then issues early warnings to the monitored subjects based on their emotional state and abnormal information. Finally, it returns to Step 1 to continue collecting facial video data of the monitored subjects until the dynamic monitoring of their emotional state is completed.

7. The method for dynamic monitoring of emotional state based on spatiotemporal features of video images according to claim 6, characterized in that: In step 4, the anomaly information includes an anomaly identifier, anomaly type, and anomaly score; if the anomaly identifier indicates that no anomaly exists, then in step 5, the risk is assessed using the following formula: Risk_Score = α•I + β•D + γ•T Where Risk_Score is the risk score, I is the emotion intensity, D is the emotion duration score, T is the emotion intensity trend, and α, β, and γ are the weight parameters of I, D, and T, respectively. If the anomaly is identified as present, then in step 5, the risk is assessed using the following formula: Risk_Score = α•I + β•D + γ•T + η•K Where K is the anomaly score, and η is the weighting parameter of K.

8. The method for dynamic monitoring of emotional state based on spatiotemporal features of video images according to claim 6 or 7, characterized in that, Step 2 is as follows: Step 2.1: The spatial feature extraction unit of the spatiotemporal feature extraction module extracts facial spatial features from facial video data according to different spatial scales, and fuses them according to channel attention and spatial attention. Then, the fused facial spatial features are globally pooled to obtain spatial features, and sent to the environmental factor evaluation unit and the adaptive spatiotemporal feature fusion unit of the adaptive fusion module. Step 2.2: The temporal feature extraction unit of the spatiotemporal feature extraction module extracts facial temporal features from facial video data according to different time scales, and performs weight allocation, position encoding and normalization on the feature components in the facial temporal features in sequence. Then, the facial temporal features at different time scales are fused according to position encoding and weight to obtain temporal features, and sent to the dynamic weight calculation unit and adaptive spatiotemporal feature fusion unit of the adaptive fusion module. Step 2.3: The environmental factor evaluation unit of the adaptive fusion module evaluates the environmental factors of the facial video data frame by frame according to the spatial features, obtains the environmental factor score, and obtains the overall quality of each frame of the facial video data based on it. Then, the environmental factor score is sent to the dynamic weight calculation unit, and the overall quality of each frame of the facial video data is sent to the quality gating unit. Step 2.4: The dynamic weight calculation unit obtains the fusion weights of spatial and temporal features of each frame of the facial video data based on the environmental factor score and temporal features, and sends them to the adaptive spatiotemporal feature fusion unit. The adaptive spatiotemporal feature fusion unit fuses the spatial and temporal features frame by frame to obtain the spatiotemporal features of each frame of the facial video data, and sends them to the video feature fusion unit. The quality gating unit adjusts the weights of each frame of the facial video data based on the overall quality of the image and sends them to the video feature fusion unit. Step 2.5: The video feature fusion unit fuses the spatiotemporal features of all images in the facial video data according to the weights to obtain spatiotemporal fusion features, and sends them to the emotion state recognition module and the temporal modeling and trend analysis module respectively.

9. The method for dynamic monitoring of emotional state based on spatiotemporal features of video images according to claim 8, characterized in that, In step 1, the specific method of preprocessing is as follows: A. Image stabilization: A feature point matching-based stabilization algorithm is adopted. The Harris corner detection algorithm is used to detect stable feature points. Then, the Lucas-Kanade optical flow estimation algorithm is used to track the motion trajectory of the feature points. Finally, the RANSAC algorithm is used to estimate the geometric transformation relationship between each frame of the facial video data, and affine transformation is performed based on it. Motion compensation is then performed using interpolation methods. B. Illumination Normalization: The multi-scale Retinex algorithm is used for illumination equalization, combined with the restricted adaptive histogram equalization algorithm to enhance local contrast, and then the brightness is adaptively adjusted through gamma correction. C. Face detection and tracking: Multi-scale face detection is performed using the MTCNN network, and the facial motion trajectory is stably tracked using a Kalman filter, followed by bounding box optimization. D. Key Region Extraction: Using the 68-point marking method, the eye, mouth, and eyebrow regions are accurately located based on the Dlib shape predictor. Then, the corresponding regions of interest are extracted and scale normalization and geometric correction are performed.

10. The method for dynamic monitoring of emotional state based on spatiotemporal features of video images according to claim 9, characterized in that: In step 2.3, the environmental factors include light intensity, head posture, facial occlusion, and image sharpness; In step 3, the emotional state includes emotional category, emotional value, and emotional activation level.

Images