A driver fatigue detection method and system based on multi-modal data fusion

Abstract

The application belongs to the technical field of intelligent traffic and driving safety monitoring, and specifically discloses a driver fatigue detection method and system based on multi-modal data fusion. First, a heterogeneous fusion system of facial visual features, bracelet features and vehicle driving behavior features is constructed. Then, the extracted multi-modal features are subjected to individual standardization processing based on individual sober benchmarks. Next, three fatigue detection models are constructed, two of which receive bimodal input and one of which receives multi-modal input. Real-time multi-modal information is collected, and according to the failure or recovery state of the visual mode / bracelet mode, adaptive switching is performed between the three fatigue detection models, and finally the switched model is used for fatigue state detection. The application solves the problems of poor cross-driver generalization performance and insufficient detection robustness in complex environments from the root, and realizes accurate identification and monitoring of the fatigue state of the driver.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation and driving safety monitoring technology, and specifically relates to a driver fatigue detection method and system based on multimodal data fusion. Background Technology

[0002] Existing driver fatigue detection technologies are mainly divided into two categories: single-modal detection and multi-modal fusion detection. Among them, single-modal detection includes detection methods based on facial visual features and detection methods based on physiological signals.

[0003] Detection methods based on facial visual features typically use in-vehicle cameras to capture images of the driver's face and extract behavioral features such as eye closure ratio (PERCLOS), eye aspect ratio (EAR), blink frequency, mouth aspect ratio (MAR), and head posture. These features are then used to identify fatigue states through machine learning or deep learning models.

[0004] Such detection methods based on facial visual features have advantages such as being non-contact and easy to deploy, but their performance is easily affected by factors such as changes in lighting, occlusion (such as sunglasses and masks), and nighttime environments, leading to a decrease in detection stability.

[0005] Physiological signal-based detection methods can collect physiological indicators such as heart rate variability (HRV), skin conductance signals, or electroencephalogram (EEG) signals through wearable devices, and use these indicators to determine the driver's level of fatigue.

[0006] However, such detection methods based on physiological signals usually have high physiological relevance, but require wearing devices, and there are significant differences in physiological characteristics among different drivers, resulting in poor generalization ability of the model in cross-subject scenarios.

[0007] In recent years, multimodal fusion methods have gradually become a research hotspot, as they can improve detection accuracy by simultaneously utilizing visual features, physiological features, and vehicle dynamic information. However, in practical applications, multimodal systems still face two key challenges:

[0008] (1) Individual differences: Different drivers have significant differences in physiological and behavioral characteristics when they are awake, and traditional global standardization methods are difficult to effectively eliminate this individual difference problem.

[0009] (2) Modal reliability issues: In complex driving environments, some modalities may temporarily fail, such as low light at night causing failure of facial key point detection, or visual obstruction, which affects the stability of the fusion model.

[0010] To address the problems existing in the aforementioned background technology, the present invention aims to solve the following technical problems:

[0011] (1) Existing single-modal driving fatigue detection methods cannot simultaneously take into account non-invasiveness, detection accuracy, early warning capability and scenario robustness. They have inherent defects such as poor scenario adaptability, weak anti-interference capability, insufficient cross-subject generalization performance and strong detection lag, and cannot meet the all-weather detection requirements of real vehicle scenarios.

[0012] (2) Existing fatigue detection methods generally adopt a global standardization strategy, which ignores the significant individual baseline differences of drivers, making it easy for the model to misjudge the inherent characteristics of individuals as fatigue characteristics, resulting in poor cross-subject generalization ability and poor real-world application.

[0013] (3) Existing multimodal fusion schemes fail to address the modal bottleneck effect in dynamic environments. When visual modal data fails due to interference such as lighting or occlusion, the input of low-quality features directly reduces the detection accuracy of the entire fusion model. There is a lack of real-time evaluation and adaptive switching mechanism for modal quality, which makes it impossible to adapt to the complex and ever-changing environmental changes such as lighting, working conditions, and driver behavior in vehicle scenarios, resulting in insufficient continuous and reliable detection capability in all weather conditions.

[0014] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art. Summary of the Invention

[0015] The purpose of this invention is to propose a driver fatigue detection method based on multimodal data fusion. This method integrates driver facial visual features, wearable wristband signals, and vehicle dynamic information, and combines individual alertness benchmark standardization and modal validity strategies to facilitate accurate identification and monitoring of driver fatigue.

[0016] To achieve the above objectives, the present invention adopts the following technical solution:

[0017] A driver fatigue detection method based on multimodal data fusion includes the following steps:

[0018] Step 1. Simultaneously collect three main types of modal data from the driver: facial video data, wristband data, and driving behavior data, under both awake and fatigued states. Then, preprocess and extract features from the collected data.

[0019] Then, the extracted multimodal features are subjected to depersonalized feature standardization based on individual sobriety benchmarks;

[0020] Step 2. Build and pre-train two bimodal input fatigue detection models and one multimodal input fatigue detection model;

[0021] One bimodal input fatigue detection model takes facial visual features and driving behavior features as inputs; the other bimodal input fatigue detection model takes wristband features and driving behavior features as inputs.

[0022] The inputs to the multimodal input fatigue detection model are facial visual features, wristband features, and driving behavior features;

[0023] Based on the feature set standardized by de-personalization features in step 1, targeted pre-training was performed on the three fatigue detection models respectively;

[0024] Step 3. Collect facial video data, wristband data, and driving behavior data in real time, and perform preprocessing and feature extraction as in Step 1, as well as depersonalized feature standardization based on individual alertness benchmarks;

[0025] Using multimodal data that has undergone depersonalization feature standardization as input, the effectiveness of visual modality and wristband modality is quantitatively evaluated, and failure and recovery status are determined for visual modality or wristband modality respectively;

[0026] When the visual modality or wristband modality fails or recovers, the system will switch between the corresponding dual-modal input fatigue detection model and the multimodal input fatigue detection model, and use the switched model to detect fatigue status.

[0027] Furthermore, based on the aforementioned driver fatigue detection method based on multimodal data fusion, this invention also proposes a corresponding driver fatigue detection system based on multimodal data fusion, which adopts the following technical solution:

[0028] A driver fatigue detection system based on multimodal data fusion includes the following modules:

[0029] The preprocessing module is used to simultaneously collect three types of modal data from the driver in both awake and fatigued states: facial video data, wristband data, and driving behavior data. It also performs preprocessing and feature extraction on the collected data.

[0030] The extracted multimodal features are subjected to depersonalized feature standardization based on individual sobriety benchmarks;

[0031] The fatigue detection model building and pre-training module is used to build and pre-train two bimodal input fatigue detection models and one multimodal input fatigue detection model.

[0032] One bimodal input fatigue detection model takes facial visual features and driving behavior features as inputs; the other bimodal input fatigue detection model takes wristband features and driving behavior features as inputs.

[0033] The inputs to the multimodal input fatigue detection model are facial visual features, wristband features, and driving behavior features;

[0034] Based on the feature set standardized by de-personalization features, three fatigue detection models were pre-trained in a targeted manner.

[0035] And a fatigue state prediction module, used to collect facial video data, wristband data and driving behavior data in real time, and perform preprocessing, feature extraction and de-personalized feature standardization processing;

[0036] Using multimodal data that has undergone depersonalization feature standardization as input, the effectiveness of visual modality and wristband modality is quantitatively evaluated, and failure and recovery status are determined for visual modality or wristband modality respectively;

[0037] When the visual modality or wristband modality fails / recovers, the system switches between the corresponding dual-modal input fatigue detection model and the multimodal input fatigue detection model, and uses the switched model to detect fatigue status.

[0038] The present invention has the following advantages:

[0039] As described above, this invention proposes a driver fatigue detection method based on multimodal data fusion. This method first constructs a multi-dimensional fusion system of facial visual features, wristband features, and driving behavior features. Specifically, it captures facial fatigue features such as blink frequency, PERCLOS value, and yawning through an in-vehicle camera; simultaneously collects early physiological fatigue signals such as heart rate and heart rate variability, as well as motion features such as wrist movements caused by fatigue, through a portable physiological wristband; and acquires later driving behavior features such as vehicle driving behavior through the vehicle's CAN bus. These three types of features complement each other throughout the entire process. Simultaneously, dynamic fusion is achieved by combining real-time modal effectiveness, eliminating heterogeneous feature differences and invalid noise interference. Compared to traditional fixed-modal fusion schemes, this improves the comprehensiveness and accuracy of fatigue stage detection. Secondly, addressing the problem of poor model generalization caused by large differences in individual driver physiology and driving habits, this invention proposes a de-personalized feature standardization method based on individual alertness benchmarks. A benchmark library is established using the driver's alertness features, and feature standardization is completed with the driver's own baseline as a reference. This eliminates the interference of individual differences on detection results without collecting fatigue samples to retrain the model, significantly improving detection accuracy and deployment efficiency across different driver scenarios. Furthermore, this invention constructs an environment-perception-driven modal effectiveness quantitative evaluation system. For the failure mechanisms of the two core detection modalities—visual and wristband—in in-vehicle scenarios, multi-dimensional quantitative evaluation methods are designed. For the visual modality, it integrates two dimensions: facial key point detection reliability and facial region illumination suitability, comprehensively covering visual failure scenarios such as face occlusion, head-down, backlighting overexposure, and tunnel dim lighting. For the wristband modality, it integrates two dimensions: device connection status and physiological data quality, comprehensively covering data failure scenarios such as device disconnection, signal loss, and driving motion artifacts. Finally, a normalized modal comprehensive effectiveness score is output, achieving the quantification of modal usability. In addition, this invention constructs a modal effectiveness perception system to accurately identify modal failure and recovery states across all scenarios. When the visual and wristband modalities are in different states, it achieves adaptive model switching by linking with multiple pre-trained fatigue detection models, ensuring stable fatigue detection in different driving scenarios and avoiding detection anomalies caused by single modal failure. This invention addresses the core pain points of existing vehicle fatigue detection technologies, such as single multimodal fusion dimension, poor adaptability to individual differences, weak static fusion anti-interference capability, and lack of functional safety in interference scenarios. It constructs a closed-loop detection solution for the entire process, thereby achieving high-precision fatigue detection in complex vehicle environments and can be directly adapted to the engineering implementation of various vehicle scenarios. Attached Figure Description

[0040] Figure 1 This is a flowchart of a driver fatigue detection method based on multimodal data fusion in an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of facial information extraction based on Meidapipe in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of facial key points extracted in an embodiment of the present invention;

[0043] Figure 4 This is a flowchart illustrating the standardized model training process based on an individual consciousness benchmark in an embodiment of the present invention.

[0044] Figure 5 This is a flowchart of the fatigue detection model switching mechanism based on modal effectiveness in an embodiment of the present invention. Detailed Implementation

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0046] Example 1

[0047] like Figure 1 As shown, to address the core shortcomings of existing driver fatigue detection technologies, such as insufficient robustness of single-modal detection, strong equipment invasiveness, poor model generalization due to individual baseline differences, and a sharp drop in detection accuracy after modal failure in complex environments, this invention provides a driver fatigue detection method based on multimodal data fusion, which includes the following steps:

[0048] Step 1. Simultaneously collect three types of modal data from the driver: facial video data, wristband data, and driving behavior data, under both awake and fatigued states. Then, preprocess and extract features from the collected data.

[0049] This invention utilizes three hardware acquisition units: a low-invasive smart physiological wristband, an in-vehicle monocular camera, and a vehicle CAN bus interface. Combined with a unified system clock synchronization and linear interpolation frequency matching algorithm, it performs full-process processing on three core data types: driver physiological and wrist movement signals, facial video streams, and vehicle driving control.

[0050] First, configure the acquisition parameters for all hardware components uniformly:

[0051] The fixed physiological wristband has a sampling frequency of 30Hz, the vehicle camera has a 1080P resolution and a frame rate of 30fps, and the vehicle CAN data has a sampling frequency of 10Hz. All collected data are marked with a unified system timestamp distributed by the main control unit.

[0052] Secondly, the system clock of the main control unit is used as the sole reference to eliminate local clock errors of multiple devices.

[0053] Using linear interpolation, sampling frequency alignment is completed with 100ms as the smallest time unit, unifying all modal data to a sampling frequency of 10Hz, and achieving accurate spatiotemporal correspondence of multi-source data under the same timestamp.

[0054] Finally, data cleaning, outlier removal, and fatigue / awake state labeling were completed (0 represents awake state, 1 represents fatigue state). A standardized driving fatigue detection dataset was constructed, covering driving subjects of different ages, driving experience, and genders, and both awake and fatigued driving states. It includes three types of synchronous data: facial visual images, physiological wristband data, and driving behavior data.

[0055] This step helps provide a high-quality, spatiotemporally aligned data source for subsequent feature extraction, model training, and validation.

[0056] Step 1.1. To achieve data collection, data from multiple drivers under normal driving and fatigue driving conditions are collected simultaneously using an in-vehicle USB camera, a physiological wristband, and a driving simulator.

[0057] The two data collections for the same driver are independent, forming a sober driving dataset and a fatigue driving dataset, respectively.

[0058] Using facial images of the driver captured by a camera, the system achieves real-time detection of 468 facial key points based on the MediaPipe FaceMesh framework, focusing on extracting the three most fatigue-sensitive areas: eyes, mouth, and head posture.

[0059] A diagram illustrating the key points is attached. Figure 3 The specific implementation is as follows:

[0060] I. Eye aspect ratio EAR calculation is used to normalize and quantify the degree of eyelid opening and closing, which is the basis for blink detection and PERCLOS calculation, corresponding to the driver's eye opening and closing behavior.

[0061] Key point definition: For a single eye, select 6 key points.

[0062] in Key points for the left corner of the eye The key point is the right corner of the eye. , These are two key points on the upper eyelid. , Two key points on the lower eyelid; the EAR (early response ratio) is calculated for both eyes separately and then averaged as the final value. value.

[0063] Calculation formula (using the coordinates of key points in the calculation):

[0064] .

[0065] Set EAR=0.2 as the threshold for determining eyelid closure.

[0066] When EAR≤0.2, it is determined to be a closed eyelid state; when EAR>0.2, it is determined to be a closed eyelid state.

[0067] II. Percentage of eyelid closure (PERCLOS) is used to measure the percentage of time the eyelids are closed per unit of time. It is the core gold standard indicator for fatigue assessment and can reflect a continuous state of fatigue compared to a single EAR value.

[0068] The calculation window uses a 60-second sliding time window with a 10-second step size, and the calculation results are updated every 10 seconds.

[0069] Calculation formula: .in, This refers to the cumulative duration within a 60-second window where EAR ≤ 0.2. The total window duration is 60 seconds, and the result is calculated as a percentage.

[0070] III. Blink frequency (BF) calculation.

[0071] Changes in blinking frequency reflect fatigue levels; blinking frequency increases significantly when fatigued.

[0072] Blink event definition: A complete blink event is defined as an event in which the EAR is less than 0.2 in no more than 3 consecutive frames of the image, excluding interference from prolonged eye closure; Statistical method: A 60-second sliding time window is used to count the number of effective blinks within the window, in units of blinks / minute, and the results are updated every 60 seconds.

[0073] IV. Calculation of the mouth's aspect ratio (MAR).

[0074] Used to quantify the degree of lip opening and closing, corresponding to yawning behavior during driver fatigue.

[0075] Key point definition: Select 6 key points on the lips. The left corner of the mouth The right corner of the mouth , Two key points for the upper lip, , These are two key points for the lower lip.

[0076] Calculation formula (using the coordinates of key points in the calculation):

[0077] .

[0078] Yawning double threshold determination: When MAR>0.6 and the duration exceeds 0.5 seconds, it is determined as a yawning event, excluding short-term mouth opening interference during normal speech; the yawning frequency within a 60-second window is counted as the core feature.

[0079] V. Head pose Euler angle extraction.

[0080] Abnormal fluctuations in head posture can help determine fatigue levels. When a driver is fatigued, their neck muscles relax, and their head will frequently tilt downwards, lean to the side, or deviate from the normal driving posture.

[0081] In this invention, head pose is solved based on the matching relationship between 2D facial key points detected by MediaPipe FaceMesh and a standard 3D face model. The specific solution process is as follows:

[0082] First, the coordinates of two-dimensional key points on the face are obtained using a facial landmark detection algorithm. Then, feature points that can stably represent head posture are selected and a correspondence is established with the corresponding points in the pre-defined three-dimensional face standard model.

[0083] The rotation and translation matrices of the head relative to the camera coordinate system are then solved using the PnP method.

[0084] The PnP method is short for Perspective-n-Point method.

[0085] Finally, the rotation matrix is ​​decomposed into Euler angles to obtain the pitch, yaw, and roll angles of the head, thus achieving driver head attitude estimation. The specific calculation formulas and steps are as follows:

[0086] Let the coordinates of the key points in the 2D image be: , .

[0087] The corresponding 3D coordinates in the standard 3D face model are: , .

[0088] The standard 3D face model is established using a general face proportion model, with the tip of the nose as the origin of the 3D coordinate system, and the unit is mm.

[0089] The specific standard coordinates are as follows: nose tip left and right corners of the eyes , Left and right tragus , chin .

[0090] The mathematical model for the projection relationship from a 3D point to a 2D image point is as follows:

[0091] ;

[0092] in These are the coordinates of a three-dimensional point. These are the pixel coordinates of the image; It is a rotation matrix; It is a translation matrix; Scale factor; This is the camera intrinsic parameter matrix. Represented as:

[0093] ;

[0094] in , Focal length , These are the coordinates of the image center point. For a camera with a resolution of 1920×1080, =960, =540, the focal length is determined by the camera calibration. = =1100.

[0095] By establishing the projection relationship between 2D key points and 3D model points, the rotation matrix is ​​solved using the PnP algorithm. With translation matrix The objective of PnP is to minimize the reprojection error of 3D points projected onto the image plane. Its objective function is:

[0096] ;

[0097] In the formula The two-dimensional coordinates of a three-dimensional point obtained through a projection model; For the number of key points, =6.

[0098] After obtaining the rotation matrix Then, the rotation matrix is ​​decomposed into Euler angles to represent the head pose angles. The rotation matrix is ​​represented as:

[0099] .

[0100] Using the rotation order along the ZYX axes, the corresponding relationship is as follows:

[0101] ; ; ;

[0102] in, , , The yaw angle, pitch angle, and roll angle represent the angles at which the pilot's head turns left and right, tilts up and down, and leans back, respectively.

[0103] Data collected from driver facial video includes eye aspect ratio (EAR), mouth aspect ratio (MAR), eyelid closure percentage (PERCLOS), blink frequency (BF), head pitch angle, head yaw angle, and head roll angle.

[0104] The data collected by the physiological wristband includes heart rate signals and wrist movement signals.

[0105] Driving performance data such as longitudinal vehicle speed, lateral vehicle speed, accelerator pedal position, steering wheel angle, longitudinal acceleration, lateral acceleration, lane crossing time, and yaw rate are extracted via the CAN bus of the driving simulator.

[0106] Step 1.2. The collected raw data is cleaned. A method based on interquartile ranges is used to define the normal fluctuation range of the data, and extreme samples exceeding this range are removed from the dataset. To address high-frequency noise interference in the physiological signals, a Kalman filter algorithm is introduced for time-series smoothing to improve the quality and reliability of the signal data.

[0107] Step 1.3. Using a sliding time window, extract multiple features highly correlated with driving fatigue from the collected facial visual data, physiological wristband data, and driving behavior data, focusing on facial features, heart rate features, wrist movement features, and vehicle control features. The sliding time window length is 60 seconds, and the sliding step size is 10 seconds.

[0108] The facial features were extracted, including the mean and standard deviation of the eye aspect ratio, the mean and standard deviation of the mouth aspect ratio, the mean and standard deviation of the eyelid closure percentage, the mean and standard deviation of the blink frequency, and the mean and standard deviation of the head posture Euler angles (including pitch, yaw and roll), for a total of 14 features.

[0109] By analyzing the raw heart rate and removing outliers, a normal heartbeat interval (NN) sequence was obtained. Nine core HRV features were then extracted, including the mean of the heartbeat interval, the difference between adjacent NN intervals, the root mean square of the absolute value of the difference between adjacent NN intervals, the standard deviation of all NN intervals, low-frequency power, ultra-low-frequency power, very low-frequency power, high-frequency power, and the ratio of low-frequency power to high-frequency power. Specific descriptions are shown in Table 1 below.

[0110] Table 1 HRV characteristics

[0111]

[0112] The wristband's motion features include the average and standard deviation of X-axis acceleration, the average and standard deviation of X-axis angular velocity, the average and standard deviation of Y-axis acceleration, the average and standard deviation of Y-axis angular velocity, the average and standard deviation of Z-axis acceleration, and the average and standard deviation of Z-axis angular velocity, totaling 12 features. Therefore, after feature extraction, the wristband data has a total of 21 features.

[0113] Driving performance characteristics include the average and standard deviation of longitudinal vehicle speed, the average and standard deviation of lateral vehicle speed, the average and standard deviation of accelerator pedal speed, the average and standard deviation of steering wheel angle, the average and standard deviation of longitudinal acceleration, the average and standard deviation of lateral acceleration, the average and standard deviation of lane crossing time, and the average and standard deviation of yaw rate, totaling 16 dimensions.

[0114] Step 1.4. Based on the above description, construct a facial visual feature dataset, a wristband feature dataset, a driving behavior feature dataset, and a fusion dataset. The fusion dataset contains the above three modal datasets, totaling 51 dimensions.

[0115] By constructing a heterogeneous fusion system of facial visual features, wristband features, and vehicle driving behavior features, it covers the entire chain of fatigue representation from early physiological changes to later behavioral instability, and achieves fusion by combining modal real-time effectiveness.

[0116] Step 2. Based on the sober driving dataset and fatigue driving dataset constructed in Step 1, perform depersonalized feature standardization processing on the full driving features of drivers obtained after preprocessing and feature extraction, based on individual sober benchmarks.

[0117] To provide core support for improving the model's cross-subject generalization ability and to solve the problems of model misjudgment and poor generalization performance caused by differences in individual driver physiological / behavioral baselines, the multi-dimensional fatigue feature set output from step 1 is used as the input for this step.

[0118] Step 2 utilizes an individual sobriety baseline adaptive construction strategy and an improved Z-score standardization algorithm to perform de-personalization standardization processing on the original multi-dimensional fatigue feature set obtained in Step 1, categorized by the entire process scenario.

[0119] During the training phase, feature data from the driver's alertness dataset is extracted based on the constructed dataset. After removing outliers, the mean and standard deviation of each feature corresponding to the driver's alertness are calculated as the individual baseline parameters for that driver. In the testing phase, a leave-one-out-of-subject cross-validation model is adopted. The test set data is standardized using only the alertness baseline of the test subject itself, avoiding the training set baseline from masking individual differences. Based on the individual alertness baseline, the absolute values ​​of features are converted into changes relative to the individual alertness baseline using a standardization calculation formula. This results in a standardized feature set that eliminates interference from individual driver heterogeneity and has stronger cross-subject generalization ability, while also enabling seamless and rapid adaptation to completely new, unfamiliar drivers.

[0120] Step 2.1. First, perform data cleaning using 3D methods. The principle involves outlier detection for the raw data of each feature, removing outlier data segments that exceed the mean ± 3 standard deviations. The cleaned dataset is then grouped by driver ID, ensuring a one-to-one correspondence between each driver's alertness and fatigue data, providing data support for leave-one-subject cross-validation and individual benchmark construction.

[0121] Step 2.2. The cleaned dataset contains a total of One valid driver, verification process repeated. This round ensures that each driver completes an independent test as if they were a completely new and unfamiliar driver. The following is the [number]th round. The specific implementation process of round verification:

[0122] Select the first All data from the drivers, including their own conscious and fatigue state datasets, are used as independent test sets and are not included in this round of model training and benchmark construction, ensuring zero leakage of test set information.

[0123] The remaining The complete data of all drivers were merged and used as the training set for this round. .

[0124] Construct a standardized benchmark for the training set, using only data from within the training set. The complete conscious state datasets of all drivers were merged to form the training set conscious dataset, without distinction. The individual ID of the driver.

[0125] For each feature in the training set and the sober dataset, calculate the mean of the mixed data. with standard deviation subscript Representative training set, Representing the Round verification, Represents the feature dimension; .

[0126] Will and As a standardized benchmark for this round of training, it is used within the training set. The standardization of all data for a driver, including both alertness and fatigue data, is performed using the following formula:

[0127] ;

[0128] in The raw data for each feature, This represents the data after the features in the training set have been standardized.

[0129] Next, an independent test set benchmark was constructed, using only the test set's first test set. The dataset consists of the conscious states of individual drivers, without using any data from the training set. For the... For each characteristic of a driver in their conscious state, calculate the mean of the corresponding characteristic data for that individual driver in their conscious state. with standard deviation subscript This represents the test set.

[0130] Will and As the first The individual alertness baseline for each driver is used to standardize all test data of that driver, including alertness and fatigue data. The standardization formula is as follows:

[0131] .

[0132] in The raw data for each feature, This represents the standardized data of the features in the test set.

[0133] Through this transformation, the training set... The features of each driver were mapped to a uniform scale for model training.

[0134] The features of the t-th driver in the test set are mapped to a scale with their own conscious state as the zero point for model validation, ensuring that the validation results truly reflect the model's generalization ability when faced with a completely new and unfamiliar driver.

[0135] Repeat the above The verification process continues until each driver completes an independent test.

[0136] This invention establishes a dedicated individual alertness benchmark library based on driver alertness data, performs hierarchical standardization processing on three types of core features, decouples individual baseline differences from fatigue feature changes, and can complete cross-driver adaptation without collecting fatigue samples. This effectively solves the problems of poor individual adaptability, weak generalization ability, and high implementation cost of traditional solutions.

[0137] Step 3. Build and pre-train two bimodal input fatigue detection models and one multimodal input fatigue detection model.

[0138] One bimodal input fatigue detection model uses facial visual features and driving behavior features as input; the other uses wristband features and driving behavior features as input. A multimodal input fatigue detection model (i.e., with additional features) Figure 5 The multimodal fusion model shown in the article takes facial visual features, wristband features, and driving behavior features as inputs.

[0139] In this embodiment, the two bimodal input fatigue detection models are fallback backup models for when the visual modality or wristband modality fails. This ensures that if a modality fails, low-quality input from that modality is prevented from entering the model, thus avoiding decreased fatigue detection accuracy and misjudgments. The multimodal input fatigue detection model is used for high-precision fatigue identification when all modalities are available. By fully considering facial visual features, wristband features, and driving behavior features, it can better characterize the driver's fatigue state, ultimately improving fatigue detection in multimodal scenarios and meeting the requirements for high-precision driver fatigue detection.

[0140] Based on the feature set standardized by de-personalization features in step 1, targeted pre-training was performed on the three fatigue detection models respectively.

[0141] Based on the standardized feature set obtained in step 2, this invention splits the full set of features according to different modal combination methods, constructs corresponding feature subsets, and together with the original fatigue labels (0 represents the awake state and 1 represents the fatigue state) to form the training datasets of each target model (i.e., the dual-modal input fatigue detection model and the multimodal input fatigue detection model).

[0142] For each target model training dataset, machine learning methods are used for model training, and the model hyperparameters are optimized using a 5-fold cross-validation approach to obtain the optimal combination of model parameters, ensuring model accuracy while suppressing overfitting. Based on the optimal hyperparameter combination obtained through training, the final training of the corresponding XGBoost model is completed using the full sample of the target model-specific training dataset. The full dataset contains N drivers, and a leave-one-out cross-validation approach is used, selecting one driver as the test set in each round of validation, with the remaining drivers... A number of drivers were used as the training set; the training set adopted... All drivers' sobriety baselines are standardized, and the test set uses individual drivers' sobriety baselines. Through this method, the model can learn fatigue characteristic change patterns that are independent of individual differences, thereby improving the model's cross-subject generalization ability.

[0143] The training processes for both the dual-modal input fatigue detection model and the multimodal input fatigue detection model are as follows:

[0144] The facial visual features and driving behavior features in the feature set are combined to form the first bimodal training dataset; the facial visual features and driving behavior features in the feature set are combined to form the second bimodal training dataset.

[0145] The facial visual features, driving behavior features, and wristband features in the feature set are all combined to form a multimodal training dataset.

[0146] Based on the first bimodal training dataset consisting of 14-dimensional facial visual features and 16-dimensional driving behavior features, an XGBoost model is trained to obtain a pre-trained bimodal input fatigue detection model.

[0147] Based on the second bimodal training dataset consisting of a 21-dimensional wristband and 16-dimensional driving behavior features, an XGBoost model is trained to obtain another pre-trained bimodal input fatigue detection model.

[0148] Based on a multimodal training dataset consisting of 14-dimensional facial visual features, 16-dimensional driving behavior features, and 21-dimensional wristband features, an XGBoost model is trained to obtain a multimodal input fatigue detection model.

[0149] Of course, the base model in this embodiment is not limited to the XGBoost model mentioned above. For example, alternatives may include lightweight neural networks, random forests, SVMs, and other machine learning / deep learning models.

[0150] Using the XGBoost algorithm, Leave-one-out-of-subjects cross-validation (LOSO) strategy, and grid search hyperparameter optimization method, a complete fatigue detection model system was constructed according to modal combinations for the standardized multimodal feature set obtained in step 2. Model training, validation, and optimization were completed simultaneously. All models were classified with "0 = driver awake state, 1 = driver fatigue state" as binary labels. Model validation was performed using leave-one-out-of-subjects cross-validation, and hyperparameter optimization was performed within a preset range using grid search to ensure a balance between model fitting performance and anti-overfitting ability. Finally, a hierarchical model system covering all modal combinations and adapting to different vehicle operating conditions was constructed, providing complete model support for modal switching and real-time fatigue detection.

[0151] Step 4. Collect facial video data, wristband data, and driving behavior data in real time, and perform preprocessing and feature extraction as in Step 1, and perform depersonalized feature standardization based on individual alertness benchmarks as in Step 2.

[0152] Using the multimodal data processed by de-personalization and feature standardization in step 3 as input, the effectiveness of the visual modality and the wristband modality is quantitatively evaluated separately, and the failure and recovery states of the visual modality or wristband modality are determined. When the visual modality or wristband modality fails / recovers, an adaptive switch is performed between the corresponding bimodal input fatigue detection model and the multimodal input fatigue detection model, and the switched model is used to detect fatigue state in order to achieve matching for different driving conditions.

[0153] This invention first monitors two dimensions of parameters in parallel: facial video stream and wristband data. These parameters are: 1) the average confidence score of facial key points and average grayscale value of the image in the visual modality; and 2) the device connection status and the proportion of data anomalies in the physiological modality, enabling real-time quantitative assessment of modality effectiveness. Secondly, based on pre-calibrated quantization thresholds and duration rules, this invention employs a dual-threshold hysteresis protection mechanism to determine the failure and recovery of both the visual and wristband modalities. This filters out transient environmental interference and avoids frequent switching under critical conditions, thus achieving modality failure and recovery determination. Furthermore, based on the determination results, this invention achieves adaptive switching of the detection modality through pre-loaded model switching: under normal operating conditions, a multi-modal input fatigue detection model is activated for high-precision measurement; when the visual modality fails, it adaptively switches to a dual-modal input fatigue detection model (wristband and vehicle); when the wristband modality fails, it adaptively switches back to a dual-modal input fatigue detection model (visual and vehicle); and when both the visual and wristband modalities fail simultaneously, a system alarm is triggered. Ultimately, this achieves seamless adaptive switching of the detection modality in complex in-vehicle environments.

[0154] Step 4.1. Synchronous acquisition of multi-source data and time-aligned synchronous acquisition, including driver facial video data acquired by the vehicle camera, wristband data acquired by the wearable physiological wristband, and driving behavior data acquired by the vehicle CAN bus.

[0155] This step provides a solution for long-term adaptation to unfamiliar drivers when the vehicle is put into operation. The core is to build a personal baseline by automatically and seamlessly collecting the driver's valid alertness data. During the real-world use phase by unfamiliar drivers, the system automatically collects alertness data for the driver within 3 minutes of the first 5 minutes of each trip, based on a joint judgment rule that combines operational condition triggering and visual anomaly filtering. This data is used to seamlessly build and permanently store the driver's unique personal alertness baseline.

[0156] Specifically, the system monitors three types of parameters in real time: wristband features, driving behavior, and visual features. Data collection is only considered valid and begins when all three conditions are met simultaneously and for a sustained period of 3 minutes:

[0157] The vehicle ignition start signal was detected within the first 5 minutes of a single trip, and a baseline was established using the driver's alert state immediately upon starting the vehicle; the vehicle's current speed was >30km / h, and the longitudinal acceleration fluctuation range fed back by the CAN bus was ±0.2m / s². 2 The internal and lateral acceleration fluctuation range is ±0.15m / s². 2 Within the data acquisition window, no extreme fatigue indicators were detected in the visual modality, such as eye closure events with an eye aspect ratio (EAR) ≤ 0.2 and a duration > 1 second, or yawning events with a mouth aspect ratio (MAR) > 0.6 and a duration > 0.5 seconds. After meeting the above combined conditions, all 51-dimensional feature data within the 3-minute baseline acquisition window were automatically extracted. Following the calculation rules in step 1, the mean and standard deviation of each feature were calculated as the individual alertness baseline for the unfamiliar driver. The driver ID, linked to the vehicle, was permanently stored on the local in-vehicle terminal.

[0158] The collected multimodal data is time-stamped and interpolated for alignment. The data is preprocessed and features are extracted according to steps 1 and 2. The extracted multimodal features are then standardized based on individual sobriety benchmarks for depersonalization.

[0159] Step 4.2. Real-time perception of modal validity: Using the synchronous multimodal data output in Step 4.1 as input, perform multi-dimensional validity quantification evaluation on the visual modality and the wristband modality respectively, and output modal validity quantification index.

[0160] The real-time modal validity perception method described in step 4.2 addresses the feature failures and data anomalies that are prone to occur in visual and wristband modalities under complex vehicle scenarios. It constructs quantifiable modal validity perception models and outputs a unified modal comprehensive validity score, providing accurate numerical basis for subsequent modal state determination and adaptive switching.

[0161] Step 4.2.1. The process of obtaining the quantification index of visual modality effectiveness is as follows:

[0162] The method for quantifying the effectiveness of visual modalities is determined by two core dimensions: the reliability of facial keypoint detection and the availability of facial region illumination. A unified comprehensive visual effectiveness score is obtained through weighted fusion.

[0163] I. Reliability calculation of facial key points.

[0164] First, invalid points are filtered out based on the confidence level of facial key points in a single frame, eliminating bad points with extremely low confidence.

[0165] Define the set of valid key points for frame t. for:

[0166] .

[0167] In the formula For the t-th frame Confidence level of detection of key facial landmarks The single-point effective confidence threshold. =0.5.

[0168] Based on the set of valid key points, the average confidence score of valid key points in a single frame is calculated using the following formula:

[0169] .

[0170] In the formula Let be the average confidence level of valid key points in frame t, ranging from [0,1]. denoted as the number of valid keypoints in frame t.

[0171] A fixed-length sliding window is used to smooth the average confidence score of a single frame:

[0172] ;

[0173] The confidence score of key points after window smoothing, with a value range of [0,1]; To ensure the smooth sliding window length for confidence levels, in this embodiment, The value is set to 10 frames, which balances jitter suppression with system real-time performance.

[0174] II. Calculation of Facial Illumination Suitability.

[0175] To address the issue of visual feature failure caused by abnormal lighting conditions such as backlighting, tunnels, and direct strong light in vehicle environments, the acquired RGB face images are first converted to standard grayscale. A universal standard formula is used to ensure the stability and versatility of the conversion.

[0176] .

[0177] In the formula Image coordinates The grayscale value of the pixel at that location; This represents the values ​​of the three channels of the pixel in the RGB color space, with each value ranging from [0, 255].

[0178] Based on the Region of Interest (ROI) output by the face detection model, the average gray value within the region is calculated using the following formula:

[0179] ;

[0180] In the formula is the average gray value of the facial ROI region in frame t, with a value range of [0, 255]. This represents the total number of pixels within the facial ROI region; the average grayscale value within this region is used to characterize the overall illumination level of the facial area.

[0181] The absolute grayscale value is mapped to a lighting availability score, and a center normalization strategy is used. The closer the grayscale value is to the optimal lighting, the closer the score is to 1. When the grayscale value is too dark or too exposed, the score approaches 0. The formula is as follows:

[0182] .

[0183] In the formula Let be the illumination suitability score of the t-th frame image, and its value range is [0,1]. The optimal grayscale median value is fixed at 127; , This is a fixed value range for the grayscale image, with fixed values ​​of 0 and 255.

[0184] III. Visual modality comprehensive validity score.

[0185] The keypoint confidence score and the lighting suitability score are weighted and fused to obtain a unified overall visual effectiveness score:

[0186] ;

[0187] In the formula The visual modality integration effectiveness score for frame t is given, with a value range of [0,1]. .

[0188] Used to characterize the availability of a visual modality; the closer the value is to 1, the more stable and reliable the visual modality is.

[0189] Since the reliability of facial key point detection directly determines the extraction accuracy of core fatigue features such as PERCLOS and blink frequency, its impact on the detection results is greater than that of illumination suitability, and therefore it is given a higher weight.

[0190] The weights can be jointly optimized and calibrated using the vehicle-mounted dataset, with fatigue detection accuracy as the objective. , These are the key point confidence weights and illumination suitability weights, with a value range of [missing information]. [0.7, 0.8]、 [0.2, 0.3].

[0191] Step 4.2.2. The process of obtaining the quantitative indicators of wristband modal effectiveness is as follows:

[0192] The effectiveness of the wristband modality is determined by two dimensions: device connection status and physiological data quality. It comprehensively covers common failure scenarios of wearable devices in in-vehicle scenarios, such as disconnection, signal loss, and motion artifacts, and outputs a normalized wristband modality effectiveness score.

[0193] I. Device connection status determination.

[0194] definition The duration, in milliseconds, during which a physiological device fails to receive a valid data packet; the disconnection threshold. The unit is milliseconds (ms); when If the connection to the physiological device is lost, the validity of the wristband modality is set to 0.

[0195] II. Calculation of the proportion of data quality anomalies.

[0196] For the Inter-Beat Interval (IBI) data output by the physiological wristband, abnormal data judgment rules are defined based on the physiological limits of the human body to exclude invalid data caused by motion artifacts and signal interference. The process is as follows:

[0197] Define exception flags , used to mark the first Are any IBI data points outliers? If so... or ,but =1, indicating abnormal data; other cases... =0, is considered normal data.

[0198] In this embodiment, the 300ms-2000ms range corresponds to the human physiological limit heart rate range of 30 beats / min-200 beats / min. Data outside this range are considered invalid outliers.

[0199] Within a fixed-length time window The percentage of statistically abnormal data within the body characterizes the overall quality of physiological data, and the formula is as follows:

[0200] ;

[0201] In the formula For the first The percentage of abnormal IBI data within the window is counted continuously, with a value range of [0,1]. To count the total number of IBI data points within the window, the window length is set to 10 seconds.

[0202] III. Overall effectiveness score of wristband modality;

[0203] Combining device connectivity status and data quality, the final comprehensive effectiveness score of the wristband modality is output:

[0204] When the device is properly connected hour: ;

[0205] When the device disconnects hour: ;

[0206] In the formula For the first The overall modal effectiveness score of the wristband, with a value range of [0,1]. The closer the value is to 1, the more stable the connection of the physiological device and the more reliable the data quality.

[0207] Step 4.3. Modal state determination: Based on the modal validity quantification index output in Step 4.2, combined with the dual threshold mechanism and time persistence constraint, the failure and recovery states of each mode are determined.

[0208] Based on the visual comprehensive effectiveness score output in step 4.2 Combined with physiological effectiveness score This step employs a dual-threshold hysteresis mechanism and time persistence constraints to determine modal failure and recovery states. All threshold parameters in this step are determined through statistical distribution pre-calibration and 5-fold cross-validation using a multimodal driving dataset. Simultaneously, all recovery thresholds are strictly higher than their corresponding failure thresholds, forming hysteresis protection and fundamentally preventing repeated modal switching.

[0209] Step 4.3.1. The process of determining the failure and recovery status of the visual modality is as follows:

[0210] I. The process of determining the failure of visual modalities.

[0211] Preset visual modality failure threshold This threshold is calibrated using the 5th percentile of the visual effectiveness score sample distribution under normal driving scenarios, ensuring that over 95% of normal visual scenarios will not be falsely judged as failures. The calibration formula is as follows:

[0212] ;

[0213] In the formula For the sample distribution Quantiles This embodiment uses a sample set of visual effectiveness scores under normal driving scenarios. =0.05. Define the first... Frame-based visual modality single-frame failure condition The formula is as follows:

[0214] ;

[0215] To avoid misjudgments caused by momentary occlusion, a continuous frame accumulation judgment mechanism is introduced. The visual modality is only formally judged as failing when the abnormal state accumulates to a preset condition. The formula is as follows:

[0216] ;

[0217] In the formula The sliding window length is set to 10 frames to determine visual failure. This is the failure flag bit for the k-th frame. If it is established, then =1, otherwise =0; when satisfied When the visual modality fails, an adaptive model switch is executed. The minimum number of abnormal frames required to trigger the failure is set to 8 frames.

[0218] II. The process of determining the visual modality recovery.

[0219] To prevent frequent switching near the threshold, a hysteresis mechanism is designed, and a visual modality recovery threshold is defined. Strictly above the failure threshold, i.e., satisfying: .

[0220] in The visual effectiveness score was calibrated using the 10th percentile of the sample distribution under normal driving scenarios.

[0221] Define the qualification conditions for single-frame recovery of the visual modality in frame t. As shown below:

[0222] .

[0223] To ensure system stability, a continuous frame constraint is introduced, and the state transition is only completed when the recovery conditions are met for multiple consecutive frames:

[0224] ;

[0225] in This is the recovery qualification flag for the k-th frame.

[0226] like If it is established, then =1, otherwise =0; when satisfied At that time, the visual modality recovery is deemed effective, and the model adaptive switching is executed.

[0227] The number of consecutive qualified frames required for visual restoration is set to 20 frames in this embodiment.

[0228] Step 4.3.2. The process of determining the failure and recovery states of the wristband mode is as follows:

[0229] I. Determine the failure status of the wristband mode.

[0230] Preset wristband modal failure threshold The threshold is determined by the 5th percentile of the sample distribution of the wristband's effectiveness score under normal driving scenarios. The single-frame failure condition of the wristband modality in frame t is defined as follows:

[0231] .

[0232] Set duration constraint: when If the condition is met for a duration exceeding 2000ms, the physiological mode is deemed to have failed; when the above condition is met, the wristband mode is deemed to have failed, and adaptive model switching is performed.

[0233] II. Determining the recovery of the wristband mode.

[0234] Define the modal recovery threshold of the wristband Strictly above the failure threshold, satisfying:

[0235] .

[0236] Define the qualification conditions for single-frame recovery of physiological modality in frame t. As shown below:

[0237] .

[0238] To ensure system stability, a duration constraint is set: when When the condition is met for a duration exceeding 3000ms, the wristband is officially deemed to have recovered its modality.

[0239] When the wristband mode becomes valid again, an adaptive model switch is performed.

[0240] Step 4.4. Based on the state determination results of the visual modality or wristband modality, call the fatigue detection model pre-trained in step 2 that matches the current effective modality state, thereby completing the inference and output of the driver's fatigue state.

[0241] A one-to-one mapping relationship between modal state combinations and pre-trained models is established in advance. Based on the current modal state, the matching pre-trained model is adaptively invoked to ensure that optimal fatigue detection accuracy can be achieved under different modal combinations. The mapping relationship is as follows:

[0242] When the visual modality fails, the fatigue detection model is adaptively converted from a multimodal input model to a dual-modal input model with only wristband features and driving behavior features as input, and then used for fatigue state detection.

[0243] When the wristband mode fails, the fatigue detection model is adaptively converted from a multimodal input fatigue detection model to a dual-modal input fatigue detection model with only facial visual features and driving behavior features as input, and then used for fatigue state detection.

[0244] If both the visual modality and the wristband modality fail, the inference process of the fatigue detection model will be immediately suspended, unreliable fatigue detection results will be stopped, a system fault warning will be issued to the driver, and a system fault signal will be sent to the vehicle remote monitoring platform.

[0245] When the visual modality is recovered, the dual-modal input fatigue detection model, which only uses wristband features and driving behavior features as input, is adaptively converted into a multimodal input fatigue detection model, and then used for fatigue state detection.

[0246] When the wristband modality recovers, it adaptively transforms from a bimodal input fatigue detection model with only facial visual features and driving behavior features as input to a multimodal input fatigue detection model, and uses it for fatigue state detection.

[0247] This invention designs a dual-dimensional quantitative evaluation model for the two core modalities of vision and physiological movement: the reliability of key point detection and facial lighting suitability on the visual side, and the connection status and data quality of the device on the wristband side. It outputs a normalized modal comprehensive effectiveness score, accurately identifies the failure state of the modality in the whole scene, and blocks invalid features and noise interference.

[0248] Furthermore, this invention employs a dual-threshold mechanism and temporal constraints to complete modal state determination, and through linkage with the pre-trained model in step 3, achieves accurate matching between modal state and detection model, effectively avoiding the limitations of single-threshold determination and improving the accuracy and reliability of modal state determination.

[0249] Based on the standardized results of depersonalized features and the pre-trained hierarchical model system, this invention achieves adaptive switching of detection modes and reliable output of fatigue state through multimodal state perception and dynamic model scheduling.

[0250] This invention addresses the core pain points of poor cross-driver generalization performance and insufficient robustness in complex environments by integrating facial visual features, wristband signals, and vehicle driving behavior information. It employs de-personalized feature standardization processing, multi-modal complementary feature fusion, and an environment-aware modality switching mechanism. This achieves high-precision, high-generalization, low-intrusion, and all-weather real-time detection and early warning of driver fatigue, providing a reliable theoretical basis and a complete engineering implementation path for intelligent vehicle active safety systems. This invention can be directly applied to scenarios such as in-vehicle driver status monitoring systems, commercial vehicle driving safety early warning equipment, intelligent cockpit active safety protection systems, and commercial vehicle driving behavior monitoring platforms.

[0251] Example 2

[0252] This embodiment 2 describes a driver fatigue detection system based on multimodal data fusion, which has the same inventive concept as the driver fatigue detection method based on multimodal data fusion in embodiment 1 above.

[0253] A driver fatigue detection system based on multimodal data fusion includes the following modules:

[0254] The preprocessing module is used to simultaneously collect three types of modal data from the driver in both awake and fatigued states: facial video data, wristband data, and driving behavior data. It also performs preprocessing and feature extraction on the collected data.

[0255] The extracted multimodal features are subjected to depersonalized feature standardization based on individual sobriety benchmarks;

[0256] The fatigue detection model building and pre-training module is used to build and pre-train two bimodal input fatigue detection models and one multimodal input fatigue detection model.

[0257] One bimodal input fatigue detection model takes facial visual features and driving behavior features as inputs; the other bimodal input fatigue detection model takes wristband features and driving behavior features as inputs.

[0258] The inputs to the multimodal input fatigue detection model are facial visual features, wristband features, and driving behavior features;

[0259] Based on the feature set standardized by de-personalization features, three fatigue detection models were pre-trained in a targeted manner.

[0260] And a fatigue state prediction module, used to collect facial video data, wristband data and driving behavior data in real time, and perform preprocessing, feature extraction and de-personalized feature standardization processing;

[0261] Using multimodal data that has undergone depersonalization feature standardization as input, the effectiveness of visual modality and wristband modality is quantitatively evaluated, and failure and recovery status are determined for visual modality or wristband modality respectively;

[0262] When the visual modality or wristband modality fails / recovers, the system switches between the corresponding dual-modal input fatigue detection model and the multimodal input fatigue detection model, and uses the switched model to detect fatigue status.

[0263] It should be noted that any content not mentioned in the above-described functional modules of the system described in Embodiment 2 can be referred to the step description of the corresponding method in Embodiment 1 above, and will not be repeated in detail here.

[0264] Example 3

[0265] This embodiment 3 describes a computer device including a memory and one or more processors. Executable code is stored in the memory, and when the processor executes the executable code, it implements steps for a driver fatigue detection method based on multimodal data fusion.

[0266] In this embodiment, the computer device can be any device or apparatus with data processing capabilities, and will not be described in detail here.

[0267] Example 4

[0268] This embodiment 4 describes a computer-readable storage medium storing a program that, when executed by a processor, implements the steps of a driver fatigue detection method based on multimodal data fusion.

[0269] The computer-readable storage medium can be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc.

[0270] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A driver fatigue detection method based on multimodal data fusion, characterized in that, Includes the following steps: Step 1. Simultaneously collect three main types of modal data from the driver: facial video data, wristband data, and driving behavior data, under both awake and fatigued states. Then, preprocess and extract features from the collected data. Step 2. Perform depersonalized feature standardization on the extracted multimodal features based on individual sobriety benchmarks; Step 3. Build and pre-train two bimodal input fatigue detection models and one multimodal input fatigue detection model; One of the bimodal input fatigue detection models uses facial visual features and driving behavior features as inputs. Another dual-modal input fatigue detection model uses wristband features and driving behavior features as inputs. The inputs to the multimodal input fatigue detection model are facial visual features, wristband features, and driving behavior features; Based on the feature set standardized by de-personalization features in step 1, targeted pre-training was performed on the three fatigue detection models respectively; Step 4. Collect facial video data, wristband data, and driving behavior data in real time, and perform preprocessing and feature extraction as in Step 1, and perform depersonalized feature standardization based on individual alertness benchmarks as in Step 2. Using multimodal data that has undergone depersonalization feature standardization as input, the effectiveness of visual modality and wristband modality is quantitatively evaluated, and failure and recovery status are determined for visual modality or wristband modality respectively; When the visual modality or wristband modality fails / recovers, the system will switch between the corresponding dual-modal input fatigue detection model and the multimodal input fatigue detection model, and use the switched model to detect fatigue status. Step 4 specifically involves: Step 4.

1. Synchronous acquisition of multi-source data and time-aligned synchronous acquisition, including driver facial video data acquired by the vehicle camera, wristband data acquired by the wearable physiological wristband, and driving behavior data acquired by the vehicle CAN bus; The collected multimodal data is subjected to timestamp calibration and interpolation alignment. The data is preprocessed and features are extracted. The extracted multimodal features are then standardized based on individual sobriety benchmarks to remove individual features. Step 4.

2. Real-time perception of modal validity: Using the synchronous multimodal data output in Step 4.1 as input, perform multi-dimensional validity quantification evaluation on the visual modality and the wristband modality respectively, and output modal validity quantification index; Step 4.

3. Modal state determination: Based on the modal effectiveness quantification index output in Step 4.2, combined with the dual threshold mechanism and time persistence constraint, the failure and recovery states of each mode are accurately determined. Step 4.

4. Based on the state determination results of the visual modality or wristband modality, call the fatigue detection model pre-trained in step 2 that matches the current effective modality state, thereby completing the inference and output of the driver's fatigue state; When the visual modality fails, the fatigue detection model is adaptively converted from a multimodal input model to a dual-modal input model with only wristband features and driving behavior features as input, and then used for fatigue state detection. When the wristband mode fails, the fatigue detection model is adaptively converted from a multimodal input fatigue detection model to a dual-modal input fatigue detection model with only facial visual features and driving behavior features as input, and then used for fatigue state detection. When the visual modality is recovered, the dual-modal input fatigue detection model, which only uses wristband features and driving behavior features as input, is adaptively converted into a multimodal input fatigue detection model, and then used for fatigue state detection. When the wristband modality recovers, it adaptively transforms from a bimodal input fatigue detection model with only facial visual features and driving behavior features as input to a multimodal input fatigue detection model, and uses it for fatigue state detection.

2. The driver fatigue detection method based on multimodal data fusion according to claim 1, characterized in that, In step 1, the multimodal features obtained by feature extraction in step 1 have a total of 51 dimensions, including 14-dimensional facial visual features, 21-dimensional wristband features, and 16-dimensional driving behavior features. 14-dimensional facial visual features, specifically including: average and standard deviation of eye aspect ratio, average and standard deviation of mouth aspect ratio, average and standard deviation of eyelid closure percentage, average and standard deviation of blink frequency, and average and standard deviation of head posture Euler angles, namely pitch angle, yaw angle, and roll angle. The 21-dimensional wristband features include 9 HRV core features and 12 wrist movement features; The nine core features of HRV are: the mean of the heartbeat interval, the difference between adjacent NN intervals, the root mean square of the absolute value of the difference between adjacent NN intervals, the standard deviation of all NN intervals, low-frequency power, ultra-low-frequency power, very low-frequency power, high-frequency power, and the ratio of low-frequency power to high-frequency power. The 12 wrist motion characteristics of the bracelet are as follows: average and standard deviation of X-axis acceleration, average and standard deviation of X-axis angular velocity, average and standard deviation of Y-axis acceleration, average and standard deviation of Y-axis angular velocity, average and standard deviation of Z-axis acceleration, and average and standard deviation of Z-axis angular velocity. The 16-dimensional driving behavior characteristics are: average and standard deviation of longitudinal vehicle speed, average and standard deviation of lateral vehicle speed, average and standard deviation of accelerator pedal speed, average and standard deviation of steering wheel angle, average and standard deviation of longitudinal acceleration, average and standard deviation of lateral acceleration, average and standard deviation of lane crossing time, and average and standard deviation of yaw rate.

3. The driver fatigue detection method based on multimodal data fusion according to claim 1, characterized in that, In step 2, after preprocessing and feature extraction of the collected data, a sober driving dataset and a fatigued driving dataset are constructed, and depersonalized feature standardization processing based on individual soberness benchmarks is performed. The process is as follows: Step 2.

1. First, perform data cleaning using 3D methods. The principle is to perform outlier detection on the raw data of each feature and remove outlier data segments that exceed the mean ± 3 times the standard deviation. The cleaned dataset is grouped by driver ID, so that each driver's sobriety data and fatigue data are matched one-to-one. Step 2.

2. The cleaned dataset contains a total of One valid driver, verification process repeated. wheel; Regarding the first Round of verification, select the first All data from the famous drivers, including their own conscious state dataset and fatigue state dataset, are used as independent test sets and are not included in this round of model training and benchmark construction. The remaining The complete data of all drivers were merged and used as the training set for this round. ; Construct a standardized benchmark for the training set, using only data from within the training set. The complete conscious state datasets of all drivers were merged to form the training set conscious dataset, without distinction. The individual ID of the driver; For each feature in the training set and the sober dataset, calculate the mean of the mixed data. with standard deviation subscript Representative training set, Representing the Round verification, Represents the feature dimension; ; Will and As a standardized benchmark for this round of training, it is used within the training set. The standardization of all data for a driver, including both alertness and fatigue data, is performed using the following formula: ； in The raw data for each feature, This represents the data after the features in the training set have been standardized; Independently construct the test set benchmark, and only use the test set's first test set. Dataset of the conscious states of individual drivers; For the first For each characteristic of a driver in their conscious state, calculate the mean of the corresponding characteristic data for that individual driver in their conscious state. with standard deviation subscript Represents the test set; Will and As the first The individual alertness baseline for each driver is used to standardize all test data of that driver, including alertness and fatigue data. The standardization formula is as follows: ； in The raw data for each feature, This represents the standardized data of the features in the test set.

4. The driver fatigue detection method based on multimodal data fusion according to claim 1, characterized in that, In step 3, both the bimodal input fatigue detection model and the multimodal input fatigue detection model are based on the XGBoost model as the base model. The training process for each fatigue detection model is as follows: The facial visual features and driving behavior features in the feature set are combined to form the first bimodal training dataset; the wristband features and driving behavior features in the feature set are combined to form the second bimodal training dataset. The facial visual features, wristband features, and driving behavior features in the feature set are all combined to form a multimodal training dataset; Based on the first bimodal training dataset consisting of 14-dimensional facial visual features and 16-dimensional driving behavior features, an XGBoost model is trained to obtain a pre-trained bimodal input fatigue detection model. Based on the second bimodal training dataset consisting of a 21-dimensional wristband and 16-dimensional driving behavior features, one XGBoost model is trained to obtain another pre-trained bimodal input fatigue detection model. Based on a multimodal training dataset consisting of 14-dimensional facial visual features, 16-dimensional driving behavior features, and 21-dimensional wristband features, an XGBoost model is trained to obtain a pre-trained multimodal input fatigue detection model.

5. The driver fatigue detection method based on multimodal data fusion according to claim 1, characterized in that, In step 4.2, the process of obtaining the visual modality effectiveness quantification index is as follows: I. Reliability calculation of facial key points; First, invalid points are filtered out based on the confidence scores of facial key points in a single frame, and the set of valid key points in frame t is defined. for: ； In the formula For the t-th frame Confidence level of detection of key facial landmarks This is the single-point effective confidence threshold; Based on the set of valid key points, the average confidence score of valid key points in a single frame is calculated using the following formula: ； In the formula Let be the average confidence level of valid key points in frame t, ranging from [0,1]. Let be the number of valid keypoints in frame t; A fixed-length sliding window is used to smooth the average confidence score of a single frame: ； The confidence score of key points after window smoothing, ranging from [0,1]; The length of the smoothed sliding window is the confidence level. II. Calculation of Facial Illumination Suitability; To address the issue of visual feature failure caused by abnormal lighting, the acquired RGB face images are first converted to standard grayscale: ； In the formula Image coordinates The grayscale value of the pixel at that location; This represents the values ​​of the three channels of the pixel in the RGB color space, with each value ranging from [0, 255]. Based on the Region of Interest (ROI) output by the face detection model, the average gray value within the region is calculated using the following formula: ； In the formula is the average gray value of the facial ROI region in frame t, with a value range of [0, 255]. This represents the total number of pixels within the facial ROI region; the absolute grayscale values ​​are mapped to an illumination availability score using the following formula: ； In the formula Let be the illumination suitability score of the t-th frame image, and its value range is [0,1]. This is the optimal grayscale median value; , This refers to a fixed value range for grayscale images. III. Visual modality comprehensive validity score; The keypoint confidence score and the lighting suitability score are weighted and fused to obtain a unified overall visual effectiveness score: ； In the formula The visual modality integration effectiveness score for frame t is given, with a value range of [0,1]. , These are the key point confidence weights and illumination suitability weights, with a value range of [missing information]. [0.7, 0.8]、 [0.2, 0.3] .

6. The driver fatigue detection method based on multimodal data fusion according to claim 1, characterized in that, In step 4.2, the process of obtaining the quantitative index of wristband modality effectiveness is as follows: I. Determining Device Connection Status; definition The duration, in milliseconds, during which a physiological device fails to receive a valid data packet; the disconnection threshold. The unit is milliseconds (ms); when When the physiological device is disconnected, the validity of the wristband modality is set to 0. II. Calculation of the proportion of data quality anomalies; For the IBI (Inter-heart Rate Interval) data output by the physiological wristband, abnormal data judgment rules are defined based on the physiological limits of the human body to exclude invalid data caused by motion artifacts and signal interference. The process is as follows: Define exception flags , used to mark the first Are any IBI data points outliers? If so... or ,but =1, indicating abnormal data; other cases... =0, which is considered normal data; Within a fixed-length time window The percentage of internal statistical outliers is calculated using the following formula: In the formula For the first The percentage of abnormal IBI data within the window is counted continuously, with a value range of [0,1]. This represents the total number of IBI data points within the statistics window; III. Overall effectiveness score of wristband modality; Combining device connectivity status and data quality, the final comprehensive effectiveness score of the wristband modality is output: When the device is properly connected hour: ; When the device disconnects hour: ; In the formula For the first The overall validity score of the wristband modality is measured in the range [0,1].

7. The driver fatigue detection method based on multimodal data fusion according to claim 5, characterized in that, In step 4.3, the process of determining the failure and recovery status of the visual modality is as follows: I. Preset visual modality failure threshold The calibration formula is: ； In the formula For the sample distribution Quantiles This is a sample set of visual validity scores under normal driving scenarios; Definition of the first Frame-based visual modality single-frame failure conditions The formula is as follows: ； To avoid misjudgments caused by momentary occlusion, a continuous frame accumulation judgment mechanism is introduced. The visual modality is only formally judged as failing when the abnormal state accumulates to a preset condition. The formula is as follows: ； In the formula The length of the sliding window is used to determine visual failure. This is the failure flag bit for the k-th frame; like If it is established, then =1, otherwise =0; when satisfied When the visual modality fails, an adaptive model switch is executed. Minimum number of abnormal frames required to trigger a failure; II. Define the visual modality recovery threshold Strictly above the failure threshold, i.e., satisfying: ； Define the qualification conditions for single-frame recovery of the visual modality in frame t. As shown below: ； A continuous frame full-qualification constraint is introduced, and the state transition is only completed when the recovery conditions are met for multiple consecutive frames: ； in This is the recovery qualification flag for the k-th frame; like If it is established, then =1, otherwise =0; when satisfied When the visual modality recovery is deemed effective, an adaptive model switching is executed. The number of consecutive qualified frames required for visual restoration.

8. The driver fatigue detection method based on multimodal data fusion according to claim 6, characterized in that, In step 4.3, the process of determining the failure and recovery status of the wristband mode is as follows: I. Preset wristband mode failure threshold The single-frame failure condition for the wristband mode in frame t is defined as follows: ； Set duration constraint: when If the duration of continuous compliance exceeds 2000ms, the wristband mode is deemed to have failed; when the wristband mode fails, adaptive model switching is performed. II. Define the wristband modal recovery threshold Strictly above the failure threshold, satisfying: ； Define the qualification conditions for single-frame recovery of the wristband mode in frame t. As shown below: ； Set duration constraint: when If the condition is met for more than 3000ms, the wristband modal recovery is officially determined; when the wristband modal recovery is effective, the model adaptive switching is performed.

9. A driver fatigue detection system based on multimodal data fusion for implementing the driver fatigue detection method based on multimodal data fusion as described in claim 1, characterized in that, The driver fatigue detection system based on multimodal data fusion includes the following modules: The preprocessing module is used to simultaneously collect three types of modal data from the driver in both awake and fatigued states: facial video data, wristband data, and driving behavior data. It also performs preprocessing and feature extraction on the collected data. The extracted multimodal features are subjected to depersonalized feature standardization based on individual sobriety benchmarks; The fatigue detection model building and pre-training module is used to build and pre-train two bimodal input fatigue detection models and one multimodal input fatigue detection model. One of the bimodal input fatigue detection models uses facial visual features and driving behavior features as inputs. Another dual-modal input fatigue detection model uses wristband features and driving behavior features as inputs. The inputs to the multimodal input fatigue detection model are facial visual features, wristband features, and driving behavior features; Based on the feature set standardized by de-personalization features, three fatigue detection models were pre-trained in a targeted manner. And a fatigue state prediction module, used to collect facial video data, wristband data and driving behavior data in real time, and perform preprocessing, feature extraction and de-personalized feature standardization processing; Using multimodal data that has undergone depersonalization feature standardization as input, the effectiveness of visual modality and wristband modality is quantitatively evaluated, and failure and recovery status are determined for visual modality or wristband modality respectively; When the visual modality or wristband modality fails / recovers, the system switches between the corresponding dual-modal input fatigue detection model and the multimodal input fatigue detection model, and uses the switched model to detect fatigue status.

Images