Driver mental health state monitoring method and system and computer storage medium

Abstract

The invention discloses a driver mental health state monitoring method and system and a computer storage medium, and the method comprises the steps: carrying out the extraction of emotional features according to synchronously collected multi-modal data, and obtaining a plurality of emotional state probability features and driving aggressive index features; respectively calculating an emotional stability index and a driving stability index of the driver based on the plurality of emotional state probability features and the driving aggressive index features; performing weighted fusion on the emotion stability index and the driving stability index to obtain a comprehensive stability index corresponding to the driver; determining a driving risk level by using a comparison result of the comprehensive stability index and a personal dynamic threshold matched with a driver so as to trigger an early warning strategy corresponding to the driving risk level; the personal baseline model is periodically incrementally updated to optimize the personal dynamic threshold.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of intelligent transportation technology, and in particular to a method, system, and computer storage medium for monitoring the mental health status of drivers. Background Technology

[0002] Commercial vehicles (including city buses, highway passenger buses, taxis, ride-hailing vehicles, ordinary trucks, and hazardous goods vehicles) are the lifeblood of the national economy, and their road traffic safety is a crucial component of public safety. Alarmingly, over 90% of traffic accidents stem from driver-related factors. Traditionally, industry and research have focused on explicit factors such as fatigued driving, drunk driving, and distracted driving. However, with the accelerating pace of society, the mental health of drivers—a long-neglected implicit factor—is increasingly becoming a "silent killer" causing major traffic accidents. Long-term work pressure, life pressure, traffic congestion, and poor driver-passenger interactions can easily lead to anxiety, depression, road rage, and other psychological problems in drivers, significantly reducing their psychological resilience—their ability to maintain cognitive function and operational stability in the face of stressful events. This decline in psychological resilience directly manifests as a loss of driving stability: when encountering sudden situations, drivers may overreact (e.g., sharp turns, sudden braking), react slowly (e.g., mental blanking, stiffness), or engage in aggressive driving behavior, greatly increasing the risk of accidents.

[0003] Currently, although the development of this technology is on the rise, it is far from meeting actual needs:

[0004] 1) From single-modal to multi-modal fusion: The technology is evolving from relying on a single camera or vehicle signal to fusing multi-source data such as visual, speech, and physiological signals in order to improve recognition reliability.

[0005] 2) From post-analysis to real-time intervention: Leveraging edge computing capabilities to achieve real-time processing and immediate early warning at the vehicle end is becoming the key to technology implementation.

[0006] 3) From general models to personalized adaptation: The academic community has begun to realize the importance of individual differences, but how to achieve efficient and accurate personalized modeling in engineering practice remains a huge challenge.

[0007] However, existing technological solutions have fundamental flaws in addressing the core issue of "driver mental health": most of them attempt to use a fixed, universal standard to measure all drivers, completely ignoring the huge differences in individual drivers' personalities, skills, experience, and driving styles, and failing to address the essential safety aspects of "driving stability" and "psychological resilience".

[0008] Application content

[0009] The purpose of this application is to provide a method, system, and computer storage medium for monitoring the mental health status of drivers, in order to solve the problems in the existing technology of monitoring the mental health status of drivers that fail to deeply explore the dynamic causal relationship between emotions and operations, ignore individual differences, and cannot quantify stability.

[0010] The embodiments of this application adopt the following technical solutions:

[0011] This application provides a method for monitoring the mental health status of drivers, the method comprising:

[0012] Based on the synchronously collected multimodal data, emotional features were extracted to obtain multiple emotional state probability features and driving aggression index features;

[0013] The driver's emotional stability index and driving stability index are calculated based on multiple emotional state probability features and driving aggression index features, respectively.

[0014] The emotional stability index and the driving stability index are weighted and fused to obtain a comprehensive stability index corresponding to the driver.

[0015] The driving risk level is determined by comparing the comprehensive stability index with the driver's personal dynamic threshold, thereby triggering a warning strategy corresponding to the driving risk level and performing graded warnings. The personal dynamic threshold is extracted from a pre-constructed personal baseline model matched with the driver.

[0016] The individual baseline model is periodically and incrementally updated to optimize the individual dynamic threshold.

[0017] This application embodiment also provides a driver mental health status monitoring system, the driver mental health status monitoring system comprising:

[0018] The extraction module extracts emotional features based on synchronously collected multimodal data, obtaining multiple emotional state probability features and driving aggression index features;

[0019] The calculation module calculates the driver's emotional stability index and driving stability index based on multiple emotional state probability features and the driving aggression index features, respectively.

[0020] The fusion module performs a weighted fusion of the emotional stability index and the driving stability index to obtain a comprehensive stability index corresponding to the driver.

[0021] The judgment module uses the comparison result between the comprehensive stability index and the personal dynamic threshold matched with the driver to determine the driving risk level, so as to trigger the early warning strategy corresponding to the driving risk level and perform graded early warning. The personal dynamic threshold is extracted from a pre-constructed personal baseline model matched with the driver.

[0022] The update module periodically and incrementally updates the individual baseline model to optimize the individual dynamic threshold.

[0023] This application also provides a computer storage medium, including a program for use in conjunction with an electronic device, the program being executed by a processor to complete the following steps:

[0024] Based on the synchronously collected multimodal data, emotional features were extracted to obtain multiple emotional state probability features and driving aggression index features;

[0025] The driver's emotional stability index and driving stability index are calculated based on multiple emotional state probability features and driving aggression index features, respectively.

[0026] The emotional stability index and the driving stability index are weighted and fused to obtain a comprehensive stability index corresponding to the driver.

[0027] The driving risk level is determined by comparing the comprehensive stability index with the driver's personal dynamic threshold, thereby triggering a warning strategy corresponding to the driving risk level and performing graded warnings. The personal dynamic threshold is extracted from a pre-constructed personal baseline model matched with the driver.

[0028] The individual baseline model is periodically and incrementally updated to optimize the individual dynamic threshold.

[0029] Based on the driver mental health monitoring method, system, and computer storage medium in this application embodiment, multimodal data is collected synchronously, emotional features are extracted, and the driver's emotional stability index and driving stability index are calculated based on the obtained multiple emotional state probability features and driving aggression index features, respectively. Through weighted fusion, a comprehensive stability index corresponding to the driver is obtained. The driving risk level is determined by comparing the comprehensive stability index with the driver's personal dynamic threshold, so as to trigger the graded early warning strategy corresponding to the driving risk level, and the personal baseline model is updated incrementally on a regular basis to optimize the personal dynamic threshold.

[0030] In this way, personalized monitoring is achieved through individual baseline models, significantly improving the accuracy of early warnings. By utilizing stability indices, the abstract psychological concept of "psychological resilience" is transformed into calculable and monitorable engineering parameters, enabling the measurement of the essence of safety risks. The intelligent and humanized hierarchical intervention system achieves true predictive safety, enabling early prediction of potential dangers and timely analysis and early warning. It achieves accurate and predictive assessment and intervention of mental health risks and driving stability, and has the ability to continuously learn and optimize, dynamically updating individual baseline models. This ensures the long-term adaptability and accuracy of the system, fundamentally improving the level of road traffic safety. Attached Figure Description

[0031] The accompanying drawings, which are included to provide a further understanding of the embodiments of this specification and form part of the embodiments of this specification, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0032] Figure 1 A flowchart illustrating a method for monitoring the mental health status of drivers provided in this application embodiment;

[0033] Figure 2 This is a schematic diagram of the system architecture corresponding to a method for monitoring the mental health status of drivers provided in an embodiment of this application;

[0034] Figure 3 A flowchart illustrating the specific application process of a driver's mental health monitoring method provided in this application embodiment;

[0035] Figure 4 This application provides a schematic diagram of the structure of a driver's mental health monitoring system.

[0036] Figure 5 This is a schematic diagram of the structure of a computer storage medium corresponding to a driver's mental health monitoring method provided in an embodiment of this application. Detailed Implementation

[0037] In existing technologies, on the one hand, fixed "road rage thresholds" and "duration thresholds" are used to evaluate all drivers, completely ignoring the significant differences in individual drivers' personalities, driving styles, emotional expression habits, and skill levels. This "one-size-fits-all" approach inevitably leads to frequent false alarms for drivers with habitually aggressive driving styles, while underreporting for drivers with habitually mild but occasional emotional fluctuations.

[0038] On the other hand, while some existing technologies have established personal operating baselines, these baselines rely solely on vehicle operating data and cannot directly perceive the driver's psychological state. They can only indirectly "infer" emotional problems through operational anomalies, and cannot distinguish whether operational anomalies stem from emotional outbursts or normal emergency avoidance. Furthermore, they cannot adapt to changes in driving habits or increased psychological maturity that drivers accumulate with experience. This can cause the system to gradually deviate from the driver's true state over time, resulting in decreased accuracy.

[0039] Therefore, this application provides a method, system, and computer storage medium for monitoring the mental health status of drivers. By synchronously collecting multimodal data and extracting emotional features, the driver's emotional stability index and driving stability index are calculated based on the obtained multiple emotional state probability features and driving aggression index features. Through weighted fusion, a comprehensive stability index corresponding to the driver is obtained. The driving risk level is determined by comparing the comprehensive stability index with the driver's personal dynamic threshold, thereby triggering a graded early warning strategy corresponding to the driving risk level. The personal baseline model is also updated incrementally on a regular basis to optimize the personal dynamic threshold.

[0040] In this way, personalized monitoring is achieved through individual baseline models, significantly improving the accuracy of early warnings. By utilizing stability indices, the abstract psychological concept of "psychological resilience" is transformed into calculable and monitorable engineering parameters, enabling the measurement of the essence of safety risks. The intelligent and humanized hierarchical intervention system achieves true predictive safety, enabling early prediction of potential dangers and timely analysis and early warning. It achieves accurate and predictive assessment and intervention of mental health risks and driving stability, and has the ability to continuously learn and optimize, dynamically updating individual baseline models. This ensures the long-term adaptability and accuracy of the system, fundamentally improving the level of road traffic safety.

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0043] Please see Figure 1 This is a flowchart illustrating a method for monitoring the mental health status of drivers, as provided in this application embodiment.

[0044] In the embodiments of this specification, the method for monitoring the driver's mental health status may specifically include the following steps:

[0045] S101: Based on the synchronously collected multimodal data, emotional features are extracted to obtain multiple emotional state probability features and driving aggression index features;

[0046] S103: Calculate the driver's emotional stability index and driving stability index based on multiple emotional state probability features and the driving aggression index features, respectively;

[0047] S105: The emotional stability index and the driving stability index are weighted and fused to obtain a comprehensive stability index corresponding to the driver;

[0048] S107: Using the comparison result between the comprehensive stability index and the personal dynamic threshold matched with the driver, the driving risk level is determined to trigger the early warning strategy corresponding to the driving risk level and perform graded early warning. The personal dynamic threshold is extracted from a pre-constructed personal baseline model matched with the driver.

[0049] S109: Periodically and incrementally update the personal baseline model to optimize the personal dynamic threshold.

[0050] In the embodiments described in this specification, a series of implementation processes, including personalized baseline modeling, stability quantification through multimodal fusion, system self-learning evolution, refined hierarchical early warning, and edge-cloud collaborative management, systematically address the shortcomings of existing technologies, achieve accurate, predictive, and personalized management and intervention of drivers' mental health risks, and fundamentally improve the level of road traffic safety.

[0051] As an application embodiment of this specification, for step S101, before extracting emotion features, the method may further include:

[0052] The multimodal data is collected in real time and synchronously, and the multimodal data includes at least the driver's voice signal, facial image, physiological signal and vehicle data;

[0053] The multimodal data is preprocessed to extract emotional features.

[0054] In the embodiments of this specification, multimodal data may specifically refer to data related to the driver's mental health status, such as the driver's voice signals, facial images, physiological signals, and vehicle data, etc., without being specifically limited here.

[0055] In addition, multimodal data can be collected using the data acquisition module installed in the vehicle terminal. For example, the driver's voice signal can be collected directionally using the microphone device in the vehicle, the driver's facial image can be captured using the camera, vehicle dynamic parameters and other vehicle data can be collected and read using the vehicle's standard CAN bus interface or OBD-II interface, and physiological signals such as the driver's heart rate can be collected using integrated non-contact or contact sensors. No specific limitations are made here.

[0056] To accurately determine changes in drivers' real-time mental health status, multimodal data used for assessment needs to be collected synchronously in real time. This ensures that the analysis and processing of multimodal data are synchronized, preventing asynchronous multimodal data from affecting subsequent mental health assessments. Specifically, a unified high-precision timestamp can be assigned to each synchronously collected multimodal data point, and for each moment, only the multimodal data corresponding to that high-precision timestamp is analyzed.

[0057] Furthermore, in step S101, based on the synchronously acquired multimodal data, emotion feature extraction is performed, which may specifically include:

[0058] Based on the preprocessed speech signal, determine the probability features of the first emotional state;

[0059] Based on the preprocessed facial images, determine the probability features of the second emotional state;

[0060] Based on the preprocessed physiological signals, the probability characteristics of the third emotional state are determined;

[0061] Based on the preprocessed vehicle data, driving behavior is analyzed to determine the characteristics of the driving aggression index.

[0062] In the embodiments of this specification, the multiple emotional state probability features may specifically include a first emotional state probability feature, a second emotional state probability feature, and a third emotional state probability feature, which are obtained by analyzing the preprocessed voice signal, facial image, and physiological signal, respectively. In this way, the driver's emotional state can be comprehensively analyzed from different perspectives.

[0063] Secondly, the characteristics of the aggressive driving index can specifically include aggressive acceleration index, aggressive deceleration index, aggressive steering index, abnormal lane keeping index, and horn frequency index, etc., without being specifically limited here. In this way, it is also possible to accurately and comprehensively analyze the changes in the driver's emotional state from different perspectives of the driver's driving behavior.

[0064] As another application embodiment of this specification, before real-time synchronous acquisition of the multimodal data, the method further includes:

[0065] Collect drivers' historical driving data;

[0066] Filter the historical driving data to select normal safe driving data;

[0067] Based on the normal safe driving data, the sample mean and sample standard deviation of the target emotional characteristics are calculated to obtain the emotional baseline corresponding to the driver.

[0068] Based on the vehicle data in the normal safe driving data, the sample mean vector and sample covariance matrix corresponding to the driving aggression index feature are calculated to obtain the driving operation baseline corresponding to the driver.

[0069] Based on the emotional baseline and the driving operation baseline, a personal baseline model corresponding to the driver is constructed.

[0070] In the embodiments of this specification, the personal baseline model is pre-built and collected from the driver's historical driving data under normal safe driving conditions without stress. This allows for the analysis of the driver's emotional baseline in a calm state. Under normal safe driving conditions, the analysis yields the sample mean vector and sample covariance matrix corresponding to each dimension of the driving aggression index feature, thus obtaining the driver's corresponding driving operation baseline. This provides an accurate basis for subsequent assessment of the driver's real-time mental health status.

[0071] In other words, each driver has a corresponding personal baseline model, and subsequent monitoring of each driver's emotional state is also based on the driver's own emotional state in a calm state. This comparison can better reflect the changes in the driver's emotional state.

[0072] In specific application scenarios, each driver's personal baseline model can be encrypted and stored in the vehicle terminal and cloud repository, which facilitates subsequent access and also ensures the security of the data model and the privacy of the driver.

[0073] As an application embodiment of this specification, for step S103, calculating the driver's emotional stability index and driving stability index based on multiple emotional state probability features and the driving aggression index features respectively may specifically include:

[0074] The multiple probabilistic features of the emotional states are fused to obtain a comprehensive emotional feature.

[0075] Extract the target emotion feature value from the comprehensive emotion features;

[0076] The driver's emotional stability index is obtained by calculating the standard score of the target emotional feature value relative to the emotional baseline.

[0077] The driver's driving stability index is obtained by calculating the Mahalanobis distance of the driving aggression index feature relative to the driving operation baseline.

[0078] In the embodiments of this specification, by fusing the first emotional state probability feature, the second emotional state probability feature, and the third emotional state probability feature, the emotions corresponding to the driver's voice signal, facial image, and physiological signal can be fused to obtain a comprehensive emotional feature. In this way, the comprehensive emotional feature can be used to more accurately indicate the driver's current emotional state.

[0079] The target emotional feature value can specifically be a key emotional feature that can indicate the driver's emotions, such as the driver's emotional calmness.

[0080] The driver's emotional stability index is obtained by calculating the standard score of the target emotional characteristic value relative to the emotional baseline. The higher the driver's emotional stability index, the more abnormal the emotional fluctuation.

[0081] The driving stability index of a driver is obtained by calculating the Mahalanobis distance between the driving aggression index features and the driving operation baseline. The Mahalanobis distance takes into account the correlation between features and can more accurately measure the degree of anomaly in multivariate data.

[0082] As an application embodiment of this specification, for step S105, the emotional stability index and the driving stability index are weighted and fused to obtain a comprehensive stability index corresponding to the driver, which may specifically include:

[0083] The weights of the emotional stability index and the driving stability index can be adjusted according to the strategy. For example, when driving safety requirements are high, the driving stability index corresponding to abnormal driving behavior can be given a higher weight.

[0084] As an application embodiment of this specification, step S107, which uses the comparison result between the comprehensive stability index and the driver's matched personal dynamic threshold to determine the driving risk level, may specifically include:

[0085] The individual dynamic threshold includes a first dynamic threshold, a second dynamic threshold, and a third dynamic threshold. If the comprehensive stability index is less than the first dynamic threshold, the driving risk level is determined to be low risk.

[0086] If the comprehensive stability index is greater than or equal to the first dynamic threshold and less than the second dynamic threshold, the driving risk level is determined to be medium risk.

[0087] If the comprehensive stability index is greater than or equal to the second dynamic threshold and less than the third dynamic threshold, the driving risk level is determined to be high risk.

[0088] If the comprehensive stability index is greater than or equal to the third dynamic threshold, the driving risk level is determined to be an emergency risk.

[0089] In the embodiments of this specification, the driver-matched personal dynamic thresholds include a first dynamic threshold, a second dynamic threshold, and a third dynamic threshold. These are thresholds set by the driver's corresponding personal baseline model based on the driver's historical driving data, representing different psychological states of the driver and indicating different indicators of whether the driver's psychological state is normal, abnormal, or dangerous during driving. The graded early warning strategy adopted under different driving risk levels can be specifically described as follows:

[0090] If the overall stability index is less than the first dynamic threshold, it indicates that the driver's mental health is within the normal fluctuation range and the driving risk level is low. In this case, there is no need to disturb the driver; simply record the data silently and upload it to the cloud for analysis.

[0091] If the comprehensive stability index is greater than or equal to the first dynamic threshold and less than the second dynamic threshold, it indicates that the driver's psychological state has begun to deviate from the baseline, suggesting a decrease in stability and a medium risk of driving. In this case, a level one warning is triggered, and soothing music or gentle voice prompts (such as "Long-distance driving is tiring, please relax") can be played in the vehicle.

[0092] If the comprehensive stability index is greater than or equal to the second dynamic threshold and less than the third dynamic threshold, it indicates that the driver's psychological state has significantly deviated from the baseline and the driving stability has obviously decreased. The driving risk level is determined to be high risk. In this case, a level 2 warning is triggered: a clear voice warning (such as "Your state fluctuation has been detected, please drive smoothly!") is issued, and the alarm information is pushed to the fleet safety management backend in real time.

[0093] If the comprehensive stability index is greater than or equal to the third dynamic threshold, it indicates that the driver's psychological state is extremely abnormal and the risk of an accident is extremely high. The driving risk level is determined to be an emergency risk. In this case, a three-level response is triggered: a strong audible and visual alarm in the vehicle; the system can automatically enter the vehicle restriction mode (such as speed limit); the background safety officer immediately intervenes in the voice call; and the emergency contact procedure is activated if necessary.

[0094] In specific application scenarios, if the driving risk level is determined to be an emergency risk, a vehicle-to-everything (V2X) collaborative early warning strategy can also be adopted. This involves using technologies such as V2X to broadcast the high-risk status of the target vehicle to surrounding vehicles of the target vehicle where the driver is located, so as to coordinate surrounding vehicles to avoid the vehicle and form regional collaborative safety.

[0095] It should be noted that the warning strategies corresponding to the graded warnings can be adjusted according to actual needs. The warning strategies mentioned above are only for illustrative purposes and do not limit the embodiments in this specification.

[0096] As an application embodiment of this specification, for step S109, periodically and incrementally updating the personal baseline model to optimize the personal dynamic threshold includes:

[0097] Regularly update the mean and standard deviation of the target sentiment characteristics incrementally;

[0098] The mean vector and covariance matrix of the driving stability index are updated periodically and incrementally.

[0099] The individual baseline model is updated based on the updated feature data to optimize the individual dynamic threshold.

[0100] In the embodiments described in this specification, a model update mechanism is designed to enable the personal baseline model to grow alongside the driver. The system periodically (e.g., monthly) or uses high-quality safe driving data determined to be low-risk during each trip to incrementally update the personal baseline model, thereby optimizing the personal dynamic threshold. In this way, the driving risk level can be accurately determined in real time based on changes in the driver's psychological state.

[0101] For example, if a driver's mental state is relatively stable in the early stages and they are not prone to road rage, their corresponding personal dynamic threshold is relatively high. However, after a period of time, the driver's driving data shows that their mental state is more prone to change. In this case, the driver's personal dynamic threshold needs to be lowered. This makes it easier to detect the driver's emotional fluctuations and anticipate potential dangers in a timely manner.

[0102] This specification provides a method for monitoring the mental health status of drivers. By synchronously collecting multimodal data and extracting emotional features, the method calculates the driver's emotional stability index and driving stability index based on the obtained multiple emotional state probability features and driving aggression index features. Through weighted fusion, a comprehensive stability index corresponding to the driver is obtained. The driving risk level is determined by comparing the comprehensive stability index with the driver's personal dynamic threshold, thereby triggering a graded early warning strategy corresponding to the driving risk level. The personal baseline model is also updated incrementally on a regular basis to optimize the personal dynamic threshold.

[0103] In this way, personalized monitoring is achieved through individual baseline models, significantly improving the accuracy of early warnings. By utilizing stability indices, the abstract psychological concept of "psychological resilience" is transformed into calculable and monitorable engineering parameters, enabling the measurement of the essence of safety risks. The intelligent and humanized hierarchical intervention system achieves true predictive safety, enabling early prediction of potential dangers and timely analysis and early warning. It achieves accurate and predictive assessment and intervention of mental health risks and driving stability, and has the ability to continuously learn and optimize, dynamically updating individual baseline models. This ensures the long-term adaptability and accuracy of the system, fundamentally improving the level of road traffic safety.

[0104] It should be noted that the above-described specific methods for monitoring the mental health status of drivers are merely specific application examples and do not limit the scope of the embodiments in this specification. Other specific embodiments may also be included, which will not be elaborated here.

[0105] Based on the same inventive concept, this specification also provides specific application examples of the above-described driver mental health monitoring method.

[0106] like Figure 2 The diagram shown is a system architecture diagram corresponding to a driver's mental health status monitoring method provided in an embodiment of this specification.

[0107] In the embodiments of this specification, the aim is to construct an edge-cloud collaborative active safety system that uses a personal baseline model as a benchmark, multimodal data fusion as a means, stability index as a quantitative indicator, and has self-learning capabilities. The system achieves accurate and predictive assessment and intervention of mental health risks and driving stability by monitoring the degree of deviation of the driver's state from his personal normal baseline in real time.

[0108] The system corresponding to the driver's mental health status monitoring method adopts a distributed architecture of "end-cloud" collaboration, consisting of two main parts: the vehicle terminal and the cloud service platform. Specifically, data interaction can be carried out through 4G / 5G wireless communication networks. This architecture not only ensures the low latency requirement for real-time processing at the vehicle end, but also utilizes the powerful storage and computing capabilities of the cloud for macro-analysis and model optimization.

[0109] The vehicle-mounted terminal acts as the nerve ending of the system, responsible for real-time data acquisition, processing, decision-making, and local early warning. Its core hardware is an edge computing unit integrating an AI acceleration chip, ensuring that complex model inference can be completed quickly on the vehicle side. Specifically, the vehicle-mounted terminal may include the following modules:

[0110] 1. Data Acquisition Module:

[0111] Specifically, the data acquisition module may include the following data acquisition units:

[0112] The speech recognition unit employs a microphone array with beamforming and active noise reduction capabilities to directionally acquire speech signals from the driver's position. The speech recognition unit not only records the speech content (optional function, subject to strict privacy protection), but more importantly, it extracts paralinguistic features in real time, including but not limited to: fundamental frequency (reflecting pitch), short-time energy (reflecting intensity), speech rate, formant frequencies, jitter rate, and shimmer (reflecting sound stability). These features are key inputs for emotion recognition.

[0113] Image recognition unit: It adopts a high dynamic range camera that supports near-infrared illumination to ensure that the driver's facial image can be clearly captured under complex lighting conditions such as strong light, backlight, and night. The camera captures video streams at a frame rate of 15-30fps, and runs face detection (MTCNN model) and key point localization (Dlib or deep learning model) in real time through an embedded AI chip, thereby extracting facial expression features (emotion probability distribution output by CNN model trained on FER2013 dataset) and head posture features (pitch, yaw, roll angle).

[0114] Vehicle data acquisition unit: Reads vehicle dynamic parameters in real time at a high frequency (100Hz) via the vehicle's standard CAN bus interface or OBD-II interface. Key data includes: accelerator pedal opening, brake pedal status and pressure, steering wheel angle and angular velocity, vehicle speed, longitudinal acceleration, lateral acceleration, turn signal status, horn trigger signal, GPS coordinates and timestamp. This data forms the basis for calculating driving behavior indicators.

[0115] Physiological signal unit (optional expansion): can integrate non-contact sensors, such as heart rate variability (HRV) monitoring based on millimeter-wave radar, or monitoring skin conductance (GSR) via a capacitive sensor on the steering wheel. Indicators such as the low-frequency / high-frequency power ratio of HRV are effective physiological indicators for assessing stress and mental load.

[0116] 2. Edge computing module:

[0117] The edge computing module is the brain of the in-vehicle terminal, undertaking core computing tasks, ensuring that all sensitive data processing is completed locally, protecting user privacy, and guaranteeing the real-time nature of data processing. Specifically, the edge computing module may include the following sub-modules:

[0118] Real-time feature extraction submodule:

[0119] The real-time feature extraction submodule can be used for speech emotion analysis, image emotion analysis, multimodal emotion fusion, and driving behavior analysis.

[0120] Specifically, speech emotion analysis involves preprocessing the collected speech signal (pre-emphasis, framing, windowing, and noise reduction), extracting feature vectors such as MFCC (Mel-frequency cepstral coefficients), fundamental frequency, and energy, inputting them into a pre-trained temporal classification model (e.g., LSTM, Transformer, etc.), and outputting a first emotion state probability feature.

[0121] Specifically, the probability feature of the first emotional state is P_voice = [p_calm, p_happy, p_angry, p_sad, p_surprised].

[0122] Image emotion analysis specifically involves extracting geometric features such as AU (action unit) intensity, eye opening and closing, and mouth opening from each frame of a face image, combining them with CNN deep features, inputting them into a facial expression recognition model, and outputting a second emotion state probability feature P_face.

[0123] Multimodal emotion fusion employs DS evidence theory or a dynamic weight fusion algorithm to fuse P_voice and P_face into a real-time comprehensive emotion feature P_fused(t). For example, P_face is given higher weight in good lighting conditions, while P_voice is given higher weight when the speech is clear.

[0124] Furthermore, multimodal emotion fusion can specifically employ dynamic weight fusion, where the fusion weights can be dynamically adjusted based on signal quality. For example, if image quality metrics Q_face (such as brightness and sharpness) and speech quality metrics Q_voice (such as signal-to-noise ratio) are defined, then the weight W_face of P_face = Q_face / (Q_face + Q_voice).

[0125] In another application embodiment of this specification, multimodal emotion fusion can also perform feature-level fusion, concatenating the original features of speech and images with vehicle data features at a lower level, and inputting them into an end-to-end deep learning network (such as...).

[0126] The Transformer is used for joint training to directly output the stability index or risk level.

[0127] Driving behavior analysis specifically involves calculating a set of driving aggression index features D(t) in real time based on vehicle data:

[0128] Aggressive acceleration index = f(accelerator pedal change rate, longitudinal acceleration)

[0129] Aggressive deceleration index = f(rate of change of brake pedal pressure, deceleration)

[0130] Steering Aggression Index = f(Steering wheel angular velocity, rate of change of lateral acceleration)

[0131] Lane keeping anomaly index = f(lane deviation distance, lane departure speed)

[0132] The horn frequency index = f(number of horn blasts per unit time, time interval between two horn blasts, duration of a single horn blast).

[0133] The Personal Baseline Model Storage Submodule is used to save the current driver's PBM (Personal Baseline Model) in the local secure storage area. This model is built during the learning phase and includes:

[0134] Emotional baseline B_emotion: {μ_e,σ_e}, which is the mean μ_e and standard deviation σ_e of the target emotional characteristic (such as "emotional calmness") when the driver is in a calm state.

[0135] The driving operation baseline B_driving: {μ_d, Σ_d}, represents the mean vector μ_d and covariance matrix Σ_d of each dimension of the driving aggression index characteristic under normal safe driving conditions. Σ_d is crucial, as it describes the inherent correlation patterns between different driving operations (such as acceleration and steering).

[0136] Furthermore, the personal baseline model can also adopt scenario-based segmentation modeling. For example, for drivers with significantly different driving modes (such as daytime urban delivery and nighttime long-distance highway driving), multiple scenario-based sub-baseline models (such as "urban commuting PBM", "highway cruising PBM", etc.) can be established and modeled using Gaussian mixture models (GMM). During evaluation, the model that best matches the current scenario can be selected.

[0137] Stability index calculation submodule:

[0138] Emotional Stability Index (ESI): Calculates the Z-score (standard score) of the current emotional characteristic value e(t) relative to the individual's emotional baseline. The higher the Emotional Stability Index value, the more abnormal the driver's emotional fluctuations.

[0139] Specifically, the emotional stability index can be calculated using the following formula (1):

[0140] ESI(t) = |e(t) - μ_e| / σ_e; Formula (1)

[0141] Where t is time, e(t) is the current emotional feature value, μ_e is the mean of the target emotional feature, and σ_e is the standard deviation of the target emotional feature.

[0142] Driving Stability Index (DSI): Calculates the Mahalanobis distance between the current driving vector D(t) and the individual driving baseline. The Mahalanobis distance takes into account the correlation between features and can more accurately measure the degree of anomalies in multivariate data.

[0143] Specifically, the driving stability index can be calculated using the following formula (2):

[0144] DSI(t) = sqrt((D(t) - μ_d)^T * Σ_d^(-1) * (D(t) - μ_d)). Formula (2)

[0145] Compared to Euclidean distance, Mahalanobis distance takes into account the covariance structure of the data and can more accurately reflect the degree of anomaly of points in a multivariate distribution.

[0146] Comprehensive Stability Index (SSI): A weighted fusion of ESI and DSI.

[0147] Specifically, the driving stability index can be calculated using the following formula (3):

[0148] SSI(t) = α * ESI(t) + β * DSI(t), formula (3)

[0149] Where α+β=1.

[0150] In practical applications, the weights can be adjusted according to the strategy. For example, when safety requirements are high, abnormal driving behavior (DSI) can be given a higher weight (β>α).

[0151] In another embodiment of the embodiments of this specification, the stability index calculation can also be based on anomaly detection of One-Class SVM. By using the safe driving data of an individual's baseline period as positive samples, a one-class support vector machine model is trained. Multimodal data collected in real time is input into the model, and its output "anomaly score" can be directly used as the stability index. This method can be specifically applied to situations where the distribution of baseline data does not satisfy the Gaussian assumption.

[0152] Risk Decision-Making and Early Warning Submodule:

[0153] The calculated SSI(t) is compared with the individual dynamic threshold (the initial value is set based on the statistical percentile and can be optimized through self-learning in the later stage) to determine the risk level and trigger the corresponding local early warning strategy.

[0154] 3. Local early warning module

[0155] The local warning module may include an in-vehicle speaker (for voice prompts), an instrument panel or HUD icon display, and optional steering wheel or seat vibrators to execute different levels of warnings.

[0156] 4. Communication module:

[0157] The communication module is responsible for securely uploading the anonymized data (SSI values, event logs, and safe driving data fragments required for model updates) to the cloud platform, and receiving model update packages or management instructions from the cloud.

[0158] Furthermore, regarding cloud service platforms:

[0159] The cloud platform serves as the system's "intelligent hub," responsible for macro-level management, in-depth analysis, and continuous optimization. It primarily includes the following modules:

[0160] 1) Driver Profile Database: Establish an independent digital profile for each driver in the fleet, and store their Personal Baseline Model (PBM), historical trip data, SSI trend chart, warning event records, etc. for a long time.

[0161] 2) Big Data Analysis Module: Analyzes data from the entire fleet to identify patterns of group risks (such as specific road sections or time periods that are prone to triggering emotional fluctuations), constructs driver mental health profiles, and provides data support for enterprise management.

[0162] 3) Model Training and Update Center: Leveraging the powerful computing capabilities and massive amounts of data in the cloud, this center continuously optimizes emotion recognition models and stability assessment algorithms. Optimized, lightweight models are periodically deployed to in-vehicle terminals to enable iterative system evolution.

[0163] 4) Safety Management Backend: Provides fleet managers with a web-based visual interface to enable real-time vehicle monitoring, high-risk alarm push notifications, driver health status report generation, safety training management, and other functions, forming a closed-loop management system.

[0164] The embodiments in this specification, through the collaborative architecture of "real-time processing by vehicle-mounted terminals + macro-analysis by cloud platforms", can not only adapt to various types of operating vehicles (trucks, buses, taxis, etc.), but also provide transportation companies with value-added services such as driver mental health profiling, risk trend analysis, and centralized model optimization through cloud platforms, forming a closed-loop ecosystem from real-time safety of individual vehicles to long-term health management of enterprises.

[0165] Based on the same inventive concept, this specification also provides a specific application embodiment of a method for monitoring the mental health status of drivers.

[0166] like Figure 3The diagram shown is a flowchart illustrating the specific application process of a driver's mental health monitoring method provided in an embodiment of this specification.

[0167] In the embodiments of this specification, the workflow of the entire system can be specifically divided into three stages: initialization learning stage, real-time monitoring application stage, and continuous self-learning stage.

[0168] The first phase is system initialization and personal baseline modeling (learning phase), which can be carried out when the system is first installed or when it is enabled for a new driver. The goal is to collect enough data to establish an accurate personal baseline model (PBM) under stress-free, normal safe driving conditions.

[0169] The first stage may specifically include the following steps:

[0170] S301: Data collection and filtering;

[0171] During normal driving hours, the system automatically filters driving time data that meets the following conditions for modeling:

[0172] The vehicle operates smoothly (e.g., during high-speed cruising or on clear city roads).

[0173] No aggressive driving (all driving indices are below the lenient general safety thresholds).

[0174] The recommended cumulative data collection time is 15-20 hours to ensure data representativeness.

[0175] S303: Perform model calculations, including calculations of emotional baselines and driving operation baselines;

[0176] Among them, the emotion baseline (B_e) ​​is calculated: for the selected data, the sample mean μ_e and sample standard deviation σ_e of the target emotion feature (such as “emotional calmness” extracted from P_fused(t)) are calculated.

[0177] Driving baseline (B_driving) calculation: Calculate the sample mean vector μ_d and sample covariance matrix Σ_d of the driving aggression index feature D(t). The introduction of the covariance matrix characterizes the intrinsic patterns of driving behavior.

[0178] S305: Store personal baseline models;

[0179] The calculated B_emotion and B_driving are packaged into the driver's PBM, encrypted, and stored in the vehicle terminal and cloud archive.

[0180] Furthermore, the second phase is the application phase, used for real-time monitoring and stability assessment. This second phase may specifically include the following steps:

[0181] S307: Synchronously collect multimodal data in real time;

[0182] The voice, image, and vehicle data collection units are started synchronously, and a unified high-precision timestamp is added to all multimodal data.

[0183] S309: Real-time feature extraction on the edge side;

[0184] The edge computing module runs in parallel and outputs P_fused(t) and D(t) in real time.

[0185] S311: Calculate the stability index;

[0186] S313: Evaluate the risk level based on the personal dynamic threshold;

[0187] Specifically, the risk level evaluation includes:

[0188] Low risk (Level 1): SSI(t) < Th1_personal (the first dynamic threshold). At this time, the driver's mental state is within the normal personal fluctuation range, and the system silently records the data and uploads it to the cloud for analysis.

[0189] Medium risk (Level 2): Th1_personal <= SSI(t) < Th2_personal (the second dynamic threshold). At this time, the driver's mental state begins to deviate from the baseline, indicating a decrease in stability, and a first-level warning is triggered.

[0190] High risk (Level 3): Th2_personal <= SSI(t) < Th3_personal (the third dynamic threshold). At this time, the driver's mental state deviates significantly, and the driving stability decreases significantly, triggering a second-level warning.

[0191] Emergency risk (Level 4): SSI(t) >= Th3_personal. At this time, the driver's mental state is extremely abnormal, and the accident risk is extremely high, triggering a third-level warning.

[0192] S315: Execute the hierarchical warning strategy corresponding to the risk level;

[0193] S317: Update the personal baseline model.

[0194] The step S317 is the third stage and is used for the continuous self-learning and optimization of the system.

[0195] To enable the system to grow alongside the driver, the embodiments in this specification incorporate a model update mechanism. The system periodically (e.g., monthly) or uses high-quality safe driving data classified as Level 1 (low risk) from each trip to incrementally update the Personal Baseline Model (PBM).

[0196] 1) Update of mean μ: The exponentially weighted moving average (EWMA) method is used. μ_new=λ*μ_old+(1-λ)*μ_safe_period, where λ is the forgetting factor (0<λ<1), which is used to control the weight of old data.

[0197] Among them, the forgetting factor λ is a key parameter that controls the update rate and stability of the personal baseline model. The value of λ is not fixed, but is dynamically and finely adjusted according to the system's operating status, the driver's characteristics, and the data quality.

[0198] On the one hand, the initial value of the forgetting factor λ can be preset according to the driver type.

[0199] For experienced and stable drivers, λ = 0.93-0.98. A higher λ value gives historical data a higher weight, while new safety cycle data only accounts for 2%-7% of the weight. This ensures the high stability of the personal baseline model and effectively prevents unnecessary drift of the model due to short-term, occasional good driving data, thus avoiding the "dilution" of the stable habits formed by the driver over a long period of time.

[0200] For newly hired drivers, young drivers, drivers with unstable driving styles, or drivers whose driving styles are still developing as detected by the system, λ = 0.81-0.92. The lower λ value gives new data a higher weight (8%-19%), which allows the model to absorb the driver's growth and changes more quickly, rapidly capture the improvement of their driving skills and the optimization of their habits, and enable the individual baseline model to quickly converge to its current true level.

[0201] It should be noted that the initial value of the preset forgetting factor λ can be adjusted according to the actual application scenario. The specific data mentioned above are only for illustrative purposes and do not limit the embodiments of this specification.

[0202] On the other hand, the forgetting factor λ can be dynamically adjusted based on the collected multimodal data. Specifically, adjustments can be made based on "learning confidence" or "model deviation warning".

[0203] Specifically, the adjustment based on "learning confidence" involves calculating a learning confidence score (C) for each segment of safe driving data used for model updates. This confidence score incorporates the following factors:

[0204] Data quality: signal-to-noise ratio of each sensor signal, image clarity, voice integrity, etc.;

[0205] Scenario representativeness: Is the driving scenario (such as urban roads, highways, etc.) in which this data segment is located common?

[0206] Driving stability: The variance of the Driving Stability Index (DSI) within this data segment.

[0207] In practical applications, C can be normalized to 0.9–1.0, without specific limitations here. The updated forgetting factor λ can be the product of the historical forgetting factor λ and C. When the data confidence is high (C→1.0), the new data is more reliable, and a higher effective λ can be used for a more aggressive update; when the data confidence is low, the new data is less reliable, and a lower effective λ can be used for a more cautious update.

[0208] Based on the "model deviation warning" adjustment, the system can continuously monitor the mean of the Mahalanobis distance between the new safety data and the old baseline model. If it finds that the mean is consistently at a low but non-zero level, it indicates that the driver's normal behavior may be undergoing a slow and continuous shift (e.g., the overall driving style becomes softer due to increased experience).

[0209] In this case, the system can temporarily and slightly reduce the λ value (e.g., by 0.02) for several update cycles to allow the model to catch up with the driver's actual changes. Once the deviation warning is lifted, the λ value is restored to the base value.

[0210] Furthermore, to ensure the safety of the model during the self-learning process, model update trigger conditions and model parameter verification can be set.

[0211] Specifically, for the model update triggering conditions, not all data is used for updates. Only when a complete driving trip is comprehensively judged as "Level 1 (low risk)" and no warning is triggered, the data of that trip is eligible to enter the EWMA update pool. This fundamentally eliminates the pollution of personal baseline by abnormal state data.

[0212] After each EWMA update, the system performs a rationality check on the new baseline parameters. For example, it checks whether the matrix remains positive definite and whether the variance is within a physiologically reasonable range. If the check fails, the update is discarded, the system rolls back to the model before the update, the anomaly is recorded, and the driver's λ value is slightly increased (e.g., by 0.01) to make subsequent learning more cautious.

[0213] By employing a comprehensive λ-value selection strategy that combines static presets, dynamic adjustments, and boundary protection, the system can provide the most suitable model evolution speed for drivers with different characteristics. It can also adjust the learning intensity based on the quality of the data itself and the system status, making the learning process more intelligent. Furthermore, it can ensure the safety and reliability of the model during the self-learning process and maintain the reliability of the system's long-term stable operation.

[0214] 2) Update of standard deviation σ and covariance matrix Σ: A similar recursive update algorithm is adopted to ensure that the model can slowly adapt to the long-term benign evolution of the driver's driving style, while avoiding contamination by short-term noisy data.

[0215] The driver mental health monitoring system and implementation process used in the embodiments of this specification can achieve the following technical effects:

[0216] (1) Personalized monitoring: Through the personal baseline model, the system truly achieves "personalized assessment for each individual", completely solving the problem of high misjudgment rate of general models and significantly improving the accuracy of early warning.

[0217] (2) Psychological resilience and driving stability can be quantified: The "Stability Index (SSI)" is introduced, which transforms the abstract psychological concept of "psychological resilience" into a calculable and monitorable engineering parameter, thus realizing the measurement of the essence of safety risks.

[0218] (3) Achieving true predictive safety: The system can issue an early signal when the driver’s stability just begins to decline (Level 2), much earlier than the occurrence of dangerous operations (Level 3 / 4), thus gaining valuable time for intervention.

[0219] (4) Intelligent and humanized hierarchical intervention system: The early warning strategy based on the degree of stability deviation not only effectively ensures safety, but also minimizes the ineffective interference to the driver and improves the system's acceptability.

[0220] (5) Possesses lifelong learning and evolution capabilities: The self-updating mechanism makes the system a "personal safety advisor" for drivers, dynamically adapting to changes and ensuring long-term effectiveness.

[0221] (6) Forming a closed loop for enterprise safety management: The cloud platform provides transportation companies with a comprehensive solution from real-time monitoring to long-term mental health, helping them to achieve a modern safety management transformation from "passive accident handling" to "proactive risk prevention".

[0222] The specific implementation process of the embodiments in this specification can be referred to the various implementation steps corresponding to the above embodiments, and will not be repeated here.

[0223] Based on the same inventive concept, embodiments of this specification also provide a driver's mental health monitoring system. For example... Figure 4 The diagram shown is a structural schematic of a driver mental health monitoring system provided in an embodiment of this specification.

[0224] Specifically, the driver mental health monitoring system may include:

[0225] The extraction module 401 extracts emotional features based on the synchronously collected multimodal data, and obtains multiple emotional state probability features and driving aggression index features;

[0226] The calculation module 402 calculates the driver's emotional stability index and driving stability index based on multiple emotional state probability features and the driving aggression index features, respectively.

[0227] The fusion module 403 performs weighted fusion of the emotional stability index and the driving stability index to obtain a comprehensive stability index corresponding to the driver.

[0228] The judgment module 404 uses the comparison result between the comprehensive stability index and the personal dynamic threshold matched with the driver to determine the driving risk level, so as to trigger the warning strategy corresponding to the driving risk level and perform graded warning. The personal dynamic threshold is extracted from a pre-constructed personal baseline model matched with the driver.

[0229] The update module 405 periodically and incrementally updates the personal baseline model to optimize the personal dynamic threshold.

[0230] based on Figure 4 The system described in this specification also provides some specific implementation schemes of the system, which will be described below.

[0231] Furthermore, before extracting emotional features, the system also includes:

[0232] The multimodal data is collected in real time and synchronously, and the multimodal data includes at least the driver's voice signal, facial image, physiological signal and vehicle data;

[0233] The multimodal data is preprocessed to extract emotional features.

[0234] Furthermore, based on the synchronously collected multimodal data, emotional features are extracted, which may specifically include:

[0235] Based on the preprocessed speech signal, determine the probability features of the first emotional state;

[0236] Based on the preprocessed facial images, determine the probability features of the second emotional state;

[0237] Based on the preprocessed physiological signals, the probability characteristics of the third emotional state are determined;

[0238] Based on the preprocessed vehicle data, driving behavior is analyzed to determine the characteristics of the driving aggression index.

[0239] Furthermore, the driving aggression index features include aggressive acceleration index, aggressive deceleration index, aggressive steering index, lane keeping abnormality index, and horn frequency index.

[0240] Furthermore, before acquiring the multimodal data in real time, the system also includes:

[0241] Collect drivers' historical driving data;

[0242] Filter the historical driving data to select normal safe driving data;

[0243] Based on the normal safe driving data, the sample mean and sample standard deviation of the target emotional characteristics are calculated to obtain the emotional baseline corresponding to the driver.

[0244] Based on the vehicle data in the normal safe driving data, the sample mean vector and sample covariance matrix corresponding to the driving aggression index feature are calculated to obtain the driving operation baseline corresponding to the driver.

[0245] Based on the emotional baseline and the driving operation baseline, a personal baseline model corresponding to the driver is constructed.

[0246] Furthermore, the driver's emotional stability index and driving stability index are calculated based on multiple emotional state probability features and the driving aggression index features, respectively, including:

[0247] The multiple probabilistic features of the emotional states are fused to obtain a comprehensive emotional feature.

[0248] Extract the target emotion feature value from the comprehensive emotion features;

[0249] The driver's emotional stability index is obtained by calculating the standard score of the target emotional feature value relative to the emotional baseline.

[0250] The driver's driving stability index is obtained by calculating the Mahalanobis distance of the driving aggression index feature relative to the driving operation baseline.

[0251] Furthermore, by comparing the comprehensive stability index with the driver's matched personal dynamic threshold, the driving risk level is determined, including:

[0252] The individual dynamic threshold includes a first dynamic threshold, a second dynamic threshold, and a third dynamic threshold. If the comprehensive stability index is less than the first dynamic threshold, the driving risk level is determined to be low risk.

[0253] If the comprehensive stability index is greater than or equal to the first dynamic threshold and less than the second dynamic threshold, the driving risk level is determined to be medium risk.

[0254] If the comprehensive stability index is greater than or equal to the second dynamic threshold and less than the third dynamic threshold, the driving risk level is determined to be high risk.

[0255] If the comprehensive stability index is greater than or equal to the third dynamic threshold, the driving risk level is determined to be an emergency risk.

[0256] Furthermore, if the driving risk level is determined to be an emergency risk, the high-risk status of the target vehicle is broadcast to surrounding vehicles of the target vehicle where the driver is located, in order to coordinate surrounding vehicles to avoid the vehicle.

[0257] This specification provides a driver mental health monitoring system that synchronously collects multimodal data, extracts emotional features, calculates the driver's emotional stability index and driving stability index based on the obtained multiple emotional state probability features and driving aggression index features, and obtains a comprehensive stability index corresponding to the driver through weighted fusion. By comparing the comprehensive stability index with the driver's personal dynamic threshold, the system determines the driving risk level, triggers a graded early warning strategy corresponding to the driving risk level, and periodically updates the personal baseline model incrementally to optimize the personal dynamic threshold.

[0258] In this way, personalized monitoring is achieved through individual baseline models, significantly improving the accuracy of early warnings. By utilizing stability indices, the abstract psychological concept of "psychological resilience" is transformed into calculable and monitorable engineering parameters, enabling the measurement of the essence of safety risks. The intelligent and humanized hierarchical intervention system achieves true predictive safety, enabling early prediction of potential dangers and timely analysis and early warning. It achieves accurate and predictive assessment and intervention of mental health risks and driving stability, and has the ability to continuously learn and optimize, dynamically updating individual baseline models. This ensures the long-term adaptability and accuracy of the system, fundamentally improving the level of road traffic safety.

[0259] Based on the same inventive concept, embodiments of this specification also provide an electronic device, including at least one processor and a memory, wherein the memory stores a program and is configured to be executed by the at least one processor in the following steps:

[0260] Based on the synchronously collected multimodal data, emotional features were extracted to obtain multiple emotional state probability features and driving aggression index features;

[0261] The driver's emotional stability index and driving stability index are calculated based on multiple emotional state probability features and driving aggression index features, respectively.

[0262] The emotional stability index and the driving stability index are weighted and fused to obtain a comprehensive stability index corresponding to the driver.

[0263] The driving risk level is determined by comparing the comprehensive stability index with the driver's personal dynamic threshold, thereby triggering a warning strategy corresponding to the driving risk level and performing graded warnings. The personal dynamic threshold is extracted from a pre-constructed personal baseline model matched with the driver.

[0264] The individual baseline model is periodically and incrementally updated to optimize the individual dynamic threshold.

[0265] Other functions of the processor can be found in the above embodiments, and will not be repeated here.

[0266] Based on the same inventive concept, embodiments of this specification also provide a computer-readable storage medium, including a program for use in conjunction with an electronic device, the program being executable by a processor to perform the following steps:

[0267] Based on the synchronously collected multimodal data, emotional features were extracted to obtain multiple emotional state probability features and driving aggression index features;

[0268] The driver's emotional stability index and driving stability index are calculated based on multiple emotional state probability features and driving aggression index features, respectively.

[0269] The emotional stability index and the driving stability index are weighted and fused to obtain a comprehensive stability index corresponding to the driver.

[0270] The driving risk level is determined by comparing the comprehensive stability index with the driver's personal dynamic threshold, thereby triggering a warning strategy corresponding to the driving risk level and performing graded warnings. The personal dynamic threshold is extracted from a pre-constructed personal baseline model matched with the driver.

[0271] The individual baseline model is periodically and incrementally updated to optimize the individual dynamic threshold.

[0272] Other functions of the processor can be found in the above embodiments, and will not be repeated here.

[0273] like Figure 5 As shown in the figure, this specification also provides a schematic diagram of the structure of a computer storage medium.

[0274] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0275] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0276] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0277] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0278] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0279] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0280] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0281] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0282] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0283] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0284] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0285] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0286] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of the claims of this application.

Claims

1. A method for monitoring the mental health status of drivers, characterized in that, The method for monitoring the driver's mental health status includes: Based on the synchronously collected multimodal data, emotional features were extracted to obtain multiple emotional state probability features and driving aggression index features; The driver's emotional stability index and driving stability index are calculated based on multiple emotional state probability features and driving aggression index features, respectively. The emotional stability index and the driving stability index are weighted and fused to obtain a comprehensive stability index corresponding to the driver. The driving risk level is determined by comparing the comprehensive stability index with the driver's personal dynamic threshold, thereby triggering a warning strategy corresponding to the driving risk level and performing graded warnings. The personal dynamic threshold is extracted from a pre-constructed personal baseline model matched with the driver. The individual baseline model is periodically and incrementally updated to optimize the individual dynamic threshold.

2. The method as described in claim 1, characterized in that, Before extracting emotional features, the method further includes: The multimodal data is collected in real time and synchronously, and the multimodal data includes at least the driver's voice signal, facial image, physiological signal and vehicle data; The multimodal data is preprocessed to extract emotional features.

3. The method as described in claim 2, characterized in that, Based on synchronously collected multimodal data, emotion features are extracted, including: Based on the preprocessed speech signal, determine the probability features of the first emotional state; Based on the preprocessed facial images, determine the probability features of the second emotional state; Based on the preprocessed physiological signals, the probability characteristics of the third emotional state are determined; Based on the preprocessed vehicle data, driving behavior is analyzed to determine the characteristics of the driving aggression index.

4. The method as described in claim 3, characterized in that, The driving aggression index features include aggressive acceleration index, aggressive deceleration index, aggressive steering index, abnormal lane keeping index, and horn frequency index.

5. The method as described in claim 3, characterized in that, Before real-time synchronous acquisition of the multimodal data, the method further includes: Collect drivers' historical driving data; Filter the historical driving data to select normal safe driving data; Based on the normal safe driving data, the sample mean and sample standard deviation of the target emotional characteristics are calculated to obtain the emotional baseline corresponding to the driver. Based on the vehicle data in the normal safe driving data, the sample mean vector and sample covariance matrix corresponding to the driving aggression index feature are calculated to obtain the driving operation baseline corresponding to the driver. Based on the emotional baseline and the driving operation baseline, a personal baseline model corresponding to the driver is constructed.

6. The method as described in claim 5, characterized in that, The driver's emotional stability index and driving stability index are calculated based on multiple emotional state probability features and the driving aggression index features, respectively, including: The multiple probabilistic features of the emotional states are fused to obtain a comprehensive emotional feature. Extract the target emotion feature value from the comprehensive emotion features; The driver's emotional stability index is obtained by calculating the standard score of the target emotional feature value relative to the emotional baseline. The driver's driving stability index is obtained by calculating the Mahalanobis distance of the driving aggression index feature relative to the driving operation baseline.

7. The method as described in claim 1, characterized in that, The driving risk level is determined by comparing the comprehensive stability index with the driver's matched personal dynamic threshold, including: The individual dynamic threshold includes a first dynamic threshold, a second dynamic threshold, and a third dynamic threshold. If the comprehensive stability index is less than the first dynamic threshold, the driving risk level is determined to be low risk. If the comprehensive stability index is greater than or equal to the first dynamic threshold and less than the second dynamic threshold, the driving risk level is determined to be medium risk. If the comprehensive stability index is greater than or equal to the second dynamic threshold and less than the third dynamic threshold, the driving risk level is determined to be high risk. If the comprehensive stability index is greater than or equal to the third dynamic threshold, the driving risk level is determined to be an emergency risk.

8. The method as described in claim 7, characterized in that, If the driving risk level is determined to be an emergency risk, the high-risk status of the target vehicle is broadcast to surrounding vehicles of the target vehicle where the driver is located, so as to coordinate surrounding vehicles to avoid it.

9. A driver's mental health status monitoring system, characterized in that, The driver mental health monitoring system includes: The extraction module extracts emotional features based on synchronously collected multimodal data, obtaining multiple emotional state probability features and driving aggression index features; The calculation module calculates the driver's emotional stability index and driving stability index based on multiple emotional state probability features and the driving aggression index features, respectively. The fusion module performs a weighted fusion of the emotional stability index and the driving stability index to obtain a comprehensive stability index corresponding to the driver. The judgment module uses the comparison result between the comprehensive stability index and the personal dynamic threshold matched with the driver to determine the driving risk level, so as to trigger the early warning strategy corresponding to the driving risk level and perform graded early warning. The personal dynamic threshold is extracted from a pre-constructed personal baseline model matched with the driver. The update module periodically and incrementally updates the individual baseline model to optimize the individual dynamic threshold.

10. A computer storage medium comprising a program for use in conjunction with an electronic device, the program being executable by a processor to perform the following steps: Based on the synchronously collected multimodal data, emotional features were extracted to obtain multiple emotional state probability features and driving aggression index features; The driver's emotional stability index and driving stability index are calculated based on multiple emotional state probability features and driving aggression index features, respectively. The emotional stability index and the driving stability index are weighted and fused to obtain a comprehensive stability index corresponding to the driver. The driving risk level is determined by comparing the comprehensive stability index with the driver's personal dynamic threshold, thereby triggering a warning strategy corresponding to the driving risk level and performing graded warnings. The personal dynamic threshold is extracted from a pre-constructed personal baseline model matched with the driver. The individual baseline model is periodically and incrementally updated to optimize the individual dynamic threshold.

Images