Fatigue monitoring and automatic driving takeover method based on driving style adaptation and multi-modal fusion

Abstract

The application discloses a fatigue monitoring and automatic driving takeover method based on driving style self-adaptation and multi-modal fusion, and has the advantages that through multi-source data acquisition, multi-scale feature extraction and driving style online clustering, AHP-EWM combined weighting multi-modal fusion and fatigue state determination, based on the final fatigue index, scene complexity and driving style coefficient, the takeover urgency index is calculated, the hierarchical intervention strategy is formulated, and finally online feedback optimization is realized, realizing the qualitative change from 'rule type' weighting to 'experience rule-objective data' double-driven self-adaptation. The application effectively improves the anti-interference robustness of fatigue recognition and the monitoring accuracy under complex scenes and different driving styles.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving safety technology, specifically relating to a fatigue monitoring and autonomous driving takeover method based on driving style adaptation and multimodal fusion. Background Technology

[0002] According to statistics from traffic management departments, approximately 20% of traffic accidents in my country each year are related to fatigued driving. Fatigue significantly reduces a driver's perception and reaction abilities, posing a serious threat to road safety. With the development of autonomous driving technology, the driver's role has shifted from operator to monitor, placing higher demands on fatigue monitoring. Real-time and accurate fatigue monitoring and effective intervention have become crucial for improving the safety of autonomous driving.

[0003] However, existing multimodal fatigue monitoring technologies still have significant limitations. First, the use of fixed weights to fuse multi-source data makes them unsuitable for dynamic scenarios. For example, in heavy rain or strong light conditions, when single-modal data fails, fixed weights lead to a significant drop in overall monitoring accuracy. Second, using uniform feature thresholds to determine fatigue status ignores individual differences such as driver age and driving habits, easily resulting in false alarms and missed alarms. Third, there is a lack of a closed-loop mechanism for weight optimization; weights rely on offline presets and cannot be dynamically adjusted based on actual monitoring results, leading to significant performance degradation over long-term use. Furthermore, existing technologies generally ignore differences in driving styles. Research shows that aggressive, moderate, and conservative drivers exhibit drastically different behavioral characteristics when fatigued.

[0004] In summary, existing technologies are insufficient to meet the high requirements of fatigue monitoring in autonomous driving scenarios. Currently, there is a need for a fatigue monitoring and autonomous driving takeover method that can automatically identify driving styles, dynamically learn multimodal fusion weights, and has online evolution capabilities. Summary of the Invention

[0005] To address the shortcomings of existing multimodal fatigue monitoring technologies, such as fixed weights failing to adapt to dynamic scenarios, uniform thresholds lacking individual adaptability, and the absence of closed-loop optimization mechanisms, this invention provides a fatigue monitoring and autonomous driving takeover method based on driving style adaptation and multimodal fusion. Multimodal features are extracted through online clustering of driving styles, and then weighted using a combination of AHP (Analytic Hierarchy Process) and EWM (Entropy Weighting). Takeover risks are assessed, tiered intervention strategies are formulated, and finally, online feedback optimization is implemented, achieving a qualitative leap from "rule-based" weighting to a dual-driven adaptive approach based on "empirical rules and objective data."

[0006] The present invention achieves the above-mentioned technical objectives through the following technical means.

[0007] A fatigue monitoring and autonomous driving takeover method based on driving style adaptation and multimodal fusion:

[0008] (1) Multi-source data acquisition

[0009] Acquire the driver's visual information, physiological information, operational information, long-term driving behavior history data, as well as driving scenario information and vehicle information;

[0010] (2) Multi-scale feature extraction and online clustering of driving style

[0011] Preprocessing and feature extraction are performed on visual, physiological, and operational information to obtain standardized visual comprehensive feature values. Physiological comprehensive standardized characteristic values Combined standardized eigenvalues ​​of operations Meanwhile, driving style features are extracted based on long-term driving behavior historical data, and Gaussian mixture model is used for online clustering of driving styles to output driving style labels and corresponding style coefficients; multi-scale entropy and permutation entropy are introduced to extract the complexity features of each modality, which are used to construct the nonlinear complexity factor C.

[0012] (3) AHP-EWM combined weighted multimodal fusion and fatigue state determination

[0013] 1) Calculation of scene complexity index:

[0014] Calculate the scenario complexity index based on driving scenario information and vehicle information;

[0015] 2) Subjective weighting in the Analytic Hierarchy Process (AHP)

[0016] Constructing a hierarchical structure: The target layer consists of multimodal feature fusion weights. , , The criteria layer includes three dimensions: scenario adaptability, style sensitivity, and data reliability; the solution layer includes three modalities: visual, physiological, and operational.

[0017] Based on the inherent characteristics of driving safety scenarios, the weight priority of the criterion layer is set. The Saaty1-9 scaling method is used to construct a judgment matrix based on the relative importance of each dimension of the criterion layer, calculate the maximum eigenvalue of the matrix judgment, and calculate the consistency index to perform consistency verification and ensure the rationality of weight allocation.

[0018] Based on the aforementioned criteria layer weights, and combined with different scenario complexity ranges and driving style labels, subjective weights are set for the solution layer.

[0019] 3) Objective weighting using the entropy weight method

[0020] Regarding the , , Normalization is performed, and the probability of each mode is calculated based on the normalized data. Then, the information entropy and difference coefficient of each mode are calculated to obtain the objective weight.

[0021] 4) Combination weighting and fatigue index calculation:

[0022] A multiplicative combination method is used to fuse subjective and objective weights to obtain the final dynamic fused weight. This final dynamic fused weight is then weighted and fused with the corresponding standardized eigenvalues ​​to obtain the comprehensive fatigue index F. The nonlinear complexity factor C is used to dynamically correct the comprehensive fatigue index, resulting in the final fatigue index. Based on The fatigue level is determined by the numerical range.

[0023] (4) Risk assessment and differentiated intervention for takeover

[0024] Combined with the final fatigue index The scenario complexity S and driving style coefficient θ are used to calculate the takeover urgency index T. Based on the T value and driving style label, a style-differentiated hierarchical intervention strategy is generated and sent to the vehicle's domain controllers to achieve multi-domain collaborative control.

[0025] (5) Optimization of online feedback

[0026] Data on the driver's takeover response after each intervention is collected, an optimization objective function is constructed, and the scenario coefficients are iteratively optimized using the gradient descent method to ensure that the system adapts to changes in driving style and scenario differences over the long term.

[0027] Furthermore, the scene complexity index ,in, This refers to the vehicle's real-time speed. This is the highest speed limit on domestic highways. For real-time light intensity, The threshold for determining strong light environments. Rainfall level, α represents the unit of road curvature, and α, β, γ, and δ represent scene coefficients.

[0028] Furthermore, the priority of the criterion layer weights, from highest to lowest, is as follows: scene adaptability 0.4, style sensitivity 0.3, and data reliability 0.3. The judgment matrix uses scene adaptability, style sensitivity, and data reliability as evaluation dimensions, with scene adaptability having the highest priority, followed by style sensitivity, and data reliability having the lowest priority. The matrix form is as follows:

[0029]

[0030] Calculate the matrix to determine the largest eigenvalue This leads to the consistency index:

[0031]

[0032] Where n is the order of the matrix;

[0033] Combining the average random consistency index RI corresponding to the 3rd order matrix, through Perform a consistency check to ensure that CR < 0.1.

[0034] Furthermore, the subjective weights of the scheme layer include visual subjective weights. Physiological subjective weight Operational subjective weight ,and Based on the scenario complexity index, the method for allocating subjective weights is as follows:

[0035] When the scene complexity S < 2, in the conservative driving style: =0.5、 =0.3、 =0.2, in a stable driving style: =0.45、 =0.35、 =0.2, in an aggressive driving style: =0.4、 =0.3、 =0.3;

[0036] When the scene complexity 2 ≤ S < 4, in the conservative driving style: =0.4、 =0.4、 =0.2,

[0037] In a smooth driving style: =0.38、 =0.42、 =0.2, in an aggressive driving style: =0.35、 =0.35、 =0.3;

[0038] When the scene complexity S≥4, in the conservative driving style: =0.25、 =0.5、 =0.25, in a stable driving style: =0.2、 =0.5、 =0.3, in an aggressive driving style: =0.2、 =0.3、 =0.5.

[0039] Furthermore, the probability of each mode is calculated based on normalized data. Where x represents v, p, and o, namely visual, physiological, and operational modalities. The sampling point number, The total number of sampling points. For the first Normalized eigenvalues ​​of each sampling point; calculate the information entropy of each modality. Calculate the difference coefficients for each mode. The objective weight ,in for The objective weights of the modalities, and + + =1.

[0040] Furthermore, the final dynamic fusion weight ,in, for The final fusion weights of the modalities, For modality Subjective weights, among which represent These correspond to visual, physiological, and operational modalities, respectively; thus, the multimodal feature fusion weights of the target layer are obtained. , , ,and + + =1.

[0041] Furthermore, the comprehensive fatigue index Nonlinear complexity factor , To normalize the visual multi-scale entropy features, To normalize the physiological arrangement entropy, To normalize the multi-scale entropy, , , The corresponding weighting coefficients; final fatigue index: , This is a correction factor.

[0042] Furthermore, the statement based on the final fatigue index The numerical value is used to determine the fatigue level, specifically: <0.3 indicates level 0 is normal, 0.3≤ <0.6 indicates Level 1 mild fatigue, 0.6≤ <0.8 indicates Level 2 moderate fatigue. A value ≥0.8 indicates level 3 severe fatigue.

[0043] Furthermore, the aforementioned takeover urgency index ,in Let L2 norm be the style coefficient vector. , , The weighting coefficients are experimentally calibrated; the hierarchical intervention strategy generates style-differentiated hierarchical intervention strategies, which are sent to the vehicle's domain controllers to achieve multi-domain collaborative control, specifically:

[0044] When T < 0.4, an auxiliary reminder is executed, and the cockpit domain controller displays a green normal status icon;

[0045] When 0.4≤T<0.6, the recommended takeover is executed, triggering a voice prompt + yellow coffee cup warning icon. For aggressive drivers, an additional slight vibration feedback is added to the steering wheel.

[0046] When 0.6≤T<0.8, partial takeover is implemented. Based on Level 1 intervention, the seat experiences intermittent strong vibrations, and the seat belt is pre-tightened. Conservative drivers activate lane departure warning in advance.

[0047] When T≥0.8, forced takeover is executed, the vehicle output torque is limited, the lane keeping function is activated and the nearest safe stopping point is planned, and the hazard warning lights are turned on.

[0048] Furthermore, the optimization objective function ,in, To predict the fatigue index, This is the true fatigue index derived from the data on the response to the takeover.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] First, the AHP-EWM combined weighting method is adopted, which combines scene and style adaptation rules and real-time data entropy value to dynamically adjust the weights, thereby improving the monitoring accuracy in complex scenarios and under different driving styles.

[0051] Secondly, a multimodal fusion framework integrating vision, physiology, and operation is constructed. Through multi-scale feature extraction and nonlinear dynamic index mining, complementary enhancement of multi-source heterogeneous data is achieved, effectively improving the anti-interference robustness of fatigue recognition.

[0052] Finally, an online closed-loop optimization mechanism based on takeover feedback is introduced to achieve periodic iterative updates of weights and scenario coefficients, ensuring that the system adapts to driving habits and environmental changes in the long term. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the overall system architecture of the present invention;

[0054] Figure 2 This is a flowchart of the online clustering of driving styles and the combined weighting of AHP-EWM in this invention;

[0055] Figure 3 This is a schematic diagram of the takeover risk assessment and differentiated intervention strategy of the present invention. Detailed Implementation

[0056] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the scope of protection of the present invention is not limited thereto.

[0057] like Figure 1 As shown, this invention provides a fatigue monitoring and autonomous driving takeover method based on driving style adaptation and multimodal fusion, comprising the following steps:

[0058] S100: Through vehicle-mounted cameras, radar, and steering wheel sensors, it simultaneously collects the driver's visual information, physiological information, and operational behavior information to form a raw multimodal dataset.

[0059] S200: The collected raw data is denoised, filtered, outlier removed, and time synchronized. Visual features, physiological features, operational features, and nonlinear dynamic features (including visual multi-scale entropy features, physiological arrangement entropy features, and operational multi-scale entropy features) are extracted to form a standardized feature vector.

[0060] S300 uses algorithms such as Gaussian mixture model for online clustering to output driving style labels and corresponding style coefficients;

[0061] S400: Based on the scene information, subjective weighting of AHP and objective weighting of EWM are performed. The combined weighting method is used to calculate the comprehensive weight, and then the fatigue index is calculated and the fatigue level is determined.

[0062] S500: Calculate the takeover urgency index and implement tiered interventions according to the tiered strategy;

[0063] The S600 collects driver reaction data and vehicle status information after intervention, constructs a loss function, and iteratively optimizes model parameters to achieve online self-learning and self-optimization, thereby improving long-term monitoring accuracy and robustness.

[0064] The core of this invention lies in constructing a closed-loop system of "adaptive driving style and combined weighting of AHP-EWM". Data is collected by multiple sources of sensors, and the driver fatigue index is obtained through multi-scale feature extraction and combined weighting. Differentiated takeover intervention is then performed based on the index and driving style, and the model parameters are optimized online based on the intervention effect, forming a continuously optimized technical closed loop.

[0065] Through a five-layer architecture comprising a data acquisition layer, a feature extraction and style clustering layer, an AHP-EWM combined weighting and fusion layer, a takeover decision layer, and an online optimization layer, this system achieves accurate monitoring and differentiated takeover of driver fatigue in autonomous driving scenarios. All operations are performed using existing onboard hardware, requiring no additional dedicated equipment. Furthermore, key calculations strictly adhere to the formulas specified in the invention, ensuring accuracy and consistency in technology implementation while also considering the real-time performance and stability of the system, and adapting to the hardware interface standards of different vehicle brands.

[0066] I. After system startup, the system first enters the multi-source data acquisition phase. This phase requires synchronously acquiring four types of key data through the existing onboard sensor cluster to build a complete monitoring data foundation. The data sampling frequency is uniformly set to 10Hz to ensure data timeliness and continuity. Details are as follows:

[0067] The vehicle uses an infrared camera to capture real-time images of the driver's face. The infrared camera is installed below the rearview mirror inside the vehicle, and the lens angle is calibrated to be aimed at the upper middle part of the driver's face. It focuses on collecting visual information related to changes in eye contours, the degree of eyelid opening and closing, and blinking. The image data is temporarily stored in the vehicle's cache in JPEG format, and each frame of the image is accompanied by a timestamp accurate to milliseconds to avoid deviations in subsequent data synchronization.

[0068] Millimeter-wave radar is used to target the driver's chest area. The millimeter-wave radar probe is installed on the upper part of the seat back. By continuously receiving the micro-motion signals of the chest cavity (vibration frequency 0.8-3.0Hz), physiological indicators such as heart rate and heart rate variability are extracted. The radar data is transmitted to the vehicle computing unit in the form of raw intermediate frequency signals, reducing the distortion in the signal conversion process.

[0069] With the help of the steering wheel integrated angle and pressure sensors, the steering wheel angle data and the distribution of the driver's hand grip force are recorded as data related to the operation behavior. The sensor output signals are transmitted in 16-bit binary format via the Controller Area Network (CAN) bus to ensure data accuracy and meet computing requirements.

[0070] Simultaneously, the system connects to the vehicle's CAN bus and environmental sensors to acquire real-time vehicle speed (accuracy ±1km / h), ambient light intensity (measurement range 0-200000 lux, accuracy ±5%), rainfall level (mapped to levels 1-5 via voltage signals output from rain sensors, with level 1 representing light rain and level 5 representing heavy rain), road curvature (calculated by the onboard navigation system in conjunction with vehicle location information, unit 1 / km), and the current autonomous driving level of the vehicle (Level L2-L4, output by the autonomous driving domain controller). All scenario and vehicle data are updated every 100ms to ensure time synchronization with driver status data.

[0071] The system stores driver behavior data such as steering wheel angle change rate, following distance, lane change frequency, and acceleration change rate over a long period. The steering wheel angle change rate is obtained by differential calculation after collecting steering angle data from the angle sensor; the following distance is calculated by measuring the relative distance between the vehicle and the vehicle in front using millimeter-wave radar and combining it with real-time vehicle speed obtained from the CAN bus, using the formula "following distance = relative distance / real-time vehicle speed"; the lane change frequency is determined by the vehicle's infrared camera continuously identifying lane lines and the onboard computing unit detecting the vehicle's lateral displacement relative to the lane boundary. When a vehicle crosses the lane boundary from the original lane centerline into an adjacent lane and remains stable in the new lane for more than 1 second, or when the lateral displacement is continuously greater than 0.5 times the lane width, it is considered a valid lane change. The system counts the number of lane changes per unit time to obtain the lane change frequency; the acceleration change rate (Jerk) is obtained from the longitudinal acceleration signal obtained from the vehicle's CAN bus, after differential calculation and low-pass filtering, using the following formula:

[0072]

[0073] in, , These represent the longitudinal acceleration values ​​at adjacent sampling times. This represents the sampling time interval.

[0074] The aforementioned behavioral data is stored for 30 days. After compression, the data is stored in the vehicle's onboard storage (with a capacity of no less than 64GB) as the raw material for subsequent driving style clustering.

[0075] Specifically, when a driver uses this system for the first time, they need to click the "Baseline Acquisition" button on the in-vehicle central control screen to trigger the individual baseline acquisition mode. The system will continuously collect and store visual, physiological, and operational data for 30 minutes while the driver maintains a normal driving state (under the conditions of vehicle speed 30-100km / h, no sudden acceleration / deceleration, and no significant physical interference from the driver). After the data collection is completed, the system automatically calculates the mean, maximum, and minimum values ​​of each modality feature as the benchmark for subsequent normalization processing and weight calculation. This benchmark data is stored in encrypted format in the in-vehicle memory (capacity not less than 16GB). Each time the system is started subsequently, the baseline data of the currently logged-in driver will be automatically retrieved. If a driver change is detected (through seat pressure distribution or facial recognition assistance), the new driver will be prompted to complete the baseline acquisition to avoid monitoring deviations due to individual differences.

[0076] Second, after data acquisition is completed, the preprocessing and feature extraction stage begins. This stage runs in the on-board computing unit, and the processing latency is controlled within 100ms to ensure real-time performance.

[0077] (1) First, filter and denoise the various types of raw data. Specifically:

[0078] The visual image data is processed by Gaussian filtering to remove noise caused by changes in ambient light, and then histogram equalization is used to enhance the contrast of facial features, making the eye contours clearer. The processed image is used as the input for subsequent facial localization and feature extraction.

[0079] The millimeter-wave radar signal uses an adaptive filtering algorithm to eliminate interference from vehicle body vibration and retain the effective signal of chest cavity micro-movement;

[0080] The steering wheel angle and grip force data are filtered using a moving average to smooth high-frequency jitter, while outliers that exceed the reasonable range are removed to avoid abnormal data affecting subsequent calculations.

[0081] After all data processing is completed, timestamp synchronization technology (based on the CAN bus clock, with deviation controlled within ±10ms) is used to ensure the time consistency and validity of data from different sources, thus guaranteeing the accuracy of subsequent calculations.

[0082] S401. Input real-time feature values ​​(visual, physiological, and operational). In step S401, the real-time visual, physiological, and operational feature values ​​extracted in step S200 are used as input data to be normalized.

[0083] S402. Input individual baseline data (driver's normal state baseline). In step S402, the minimum and maximum values ​​of each modal feature stored during the individual baseline acquisition phase are used as the reference baseline for the driver's normal state;

[0084] S403. Based on individual baseline normalization. In step S403, based on the individual baseline data input in step S402, the real-time feature values ​​input in step S401 are normalized using the min-max method, and each modal feature is mapped to the interval [0, 1].

[0085] (2) During visual feature extraction, the facial region is first located based on a Haar-like feature classifier in the preprocessed visual image. Then, the Region of Interest (ROI) is determined using a pupil center localization algorithm. The ROI size is set to 120×60 pixels to ensure that the complete eyelid contour is included. The system continuously calculates the vertical eyelid opening within the ROI region, which is the vertical distance between the upper and lower eyelids, and normalizes it as a percentage relative to the maximum distance when the eyes are open, thus forming a time series of eyelid opening. The sampling frequency is 10Hz, which is consistent with the data acquisition frequency.

[0086] Use the ROI as a window, and define it within the window. The formula for calculating the percentage of eyelid closure time over the pulley over time (PERCLOS), a core indicator of fatigue, is as follows:

[0087]

[0088] in, The total duration of eyelid closure within a 3-second time window (the criterion for determining eyelid closure is: the ratio of the vertical distance of the eyelid to the maximum distance when the eyes are open is <20%, and this ratio is calculated in real time after extracting the upper and lower edge contours of the eyelid using the Canny edge detection algorithm). The total duration of the 3-second time window is recorded. Simultaneously, the system tracks the driver's blinking activity to provide auxiliary visual characteristics. The blink determination rule is: if eyelid closure is detected at two consecutive sampling points, it is counted as a complete blink. The system counts the number of complete blinks per minute, recorded as the blink frequency (BF).

[0089] To ensure consistency across the time dimension during feature fusion, the system integrates PERCLOS values ​​at the minute level. The real-time PERCLOS feature value used for subsequent fusion is defined as the moving average of PERCLOS calculated over all consecutive 3-second time windows within the current minute, denoted as . Therefore, the visual features involved in the fusion are two unified minute-level original feature values ​​at two different time scales: Blink frequency (BF).

[0090] To eliminate dimensional differences and achieve personalized assessment, the system performs normalization processing on the two real-time raw feature values ​​based on individual baselines.

[0091] The visual feature values ​​stored in the baseline data strictly adhere to the calculation rules for real-time features: in the individual baseline calibration mode, the system continuously collects data for 30 minutes and synchronously generates a new feature value every minute. The system records the minimum and maximum values ​​of all 30 sets of characteristic values ​​generated during the 30-minute sober driving period, denoted as: [value 1], [value 2], [value 3], [value 4], [value 5], [value 6], [value 7], [value 8], [value 9], [value 10], [value 11], [value 12], [value 13], [value 14], [value and , and This serves as the benchmark for subsequent normalization calculations.

[0092] Normalization uses the min-max method to map the original feature values ​​to the interval [0, 1]. The calculation formulas are as follows:

[0093]

[0094]

[0095] The two normalized feature values ​​are then weighted and fused to generate a visual comprehensive standardized feature value. :

[0096]

[0097] in, The optimal weight for visual feature fusion, determined through experiments, is 0.7. This value reflects the higher weight of PERCLOS as the core fatigue indicator compared to blink frequency in the discrimination process.

[0098] (3) In the physiological feature extraction stage, based on the millimeter-wave radar signal that has undergone preliminary filtering, a special preprocessing for heart rate variability analysis is performed. A Butterworth bandpass filter (order 4) with a passband frequency of 0.8-3.0Hz is used to further filter out respiratory interference (frequency 0.2-0.8Hz) and high-frequency noise (frequency > 3.0Hz). Then, the Pan-Tompkins algorithm is used to perform query per second (QRS) wave group detection on the filtered signal to locate the R wave peak and calculate the time interval between adjacent R waves to obtain the RR interval sequence. Based on this sequence, the standard deviation of normal-to-normal intervals (SDNN), a core indicator of heart rate variability, is calculated. The calculation formula is as follows:

[0099]

[0100] in, The time interval (in milliseconds) for the i-th heartbeat cycle. This represents the average interval over n heartbeat cycles. To ensure statistical reliability, the calculation window duration is set to 60 seconds by default. If the number of heartbeats in this window is less than 40, the calculation window will be automatically extended to 90 seconds to obtain sufficient RR interval data. In this case, n is the total number of heartbeats within the extended window.

[0101] Simultaneously, the root mean square of successive differences (RMSSD) is calculated for the differences between adjacent RR intervals. To ensure consistency in the time dimension during feature fusion, RMSSD is calculated based on the same RR interval sequence and calculation window as SDNN. The formula is as follows:

[0102]

[0103] SDNN is used as the core physiological feature, and RMSSD is used as the auxiliary physiological feature. The calculation results are retained to one decimal place to ensure data accuracy. Both SDNN and RMSSD are core indicators characterizing the state of autonomic nervous system function. Their values ​​are negatively correlated with the driver's fatigue level, that is, as fatigue deepens, the measured values ​​of SDNN and RMSSD tend to decrease.

[0104] Therefore, the physiological features involved in the fusion are the original feature values ​​of two time scales unified (based on the same analysis window): SDNN and RMSSD.

[0105] To eliminate dimensional differences and achieve personalized assessment, the system performs normalization processing on the aforementioned real-time raw feature values ​​based on individual baselines. The calculation rules for the physiological feature values ​​stored in the baseline data are strictly consistent with those for the real-time features: in the individual baseline calibration mode, the system continuously collects data for 30 minutes while the driver is awake, and simultaneously generates a set of SDNN and RMSSD values ​​based on the same 60-second (or extended to 90-second) analysis window. The system records the minimum and maximum values ​​among all 30 sets of feature values ​​generated within these 30 minutes, denoted as [reference needed]. and , and This serves as the benchmark for subsequent normalization calculations.

[0106] Normalization uses the min-max method to map the real-time raw feature values ​​to the [0, 1] interval. The calculation formulas are as follows:

[0107]

[0108]

[0109] The two normalized feature values ​​are weighted and fused to generate a comprehensive physiological standardized feature value. :

[0110]

[0111] in, The optimal weight for physiological feature fusion, determined through experiments, is 0.6. This value reflects the higher weight of SDNN as a core indicator of heart rate variability in physiological fatigue assessment, while also taking into account the sensitivity of RMSSD to instantaneous heart rate changes.

[0112] (4) During operation feature extraction, based on the aforementioned pre-filtered steering wheel angle data, a first-order Kalman filter is further employed to accurately calculate the operation stability index (the state equation is...). The observation equation is Where the state variable x is the true estimate of the steering wheel angle, and the observed variable is... The values ​​are sensor-acquired, and the process noise is... With observation noise The variances were calibrated and set to 0.01 and 0.05 respectively, and then finely smoothed to more effectively separate the actual steering intention from the high-frequency jitter of the sensor. The steering wheel angle data after the first-order Kalman filter were arranged in chronological order to form a steering wheel angle time series. The sampling frequency is 10Hz, consistent with the data acquisition frequency, and is used for subsequent multi-scale entropy feature extraction.

[0113] The standard deviation of the steering wheel angle within a 2-second time window is calculated as an indicator of handling smoothness. The calculation formula is as follows:

[0114]

[0115] in, Let j be the steering wheel angle value at the j-th sampling point (in degrees). This is the average steering wheel angle within a 2-second time window (20 sampling points), where m is the number of sampling points (i.e., 20).

[0116] Simultaneously, the difference in grip strength between the driver's two hands (grip strength at the 3 o'clock position minus grip strength at the 9 o'clock position) is calculated as an auxiliary operational feature. To unify the time scale for fusion, this grip strength difference feature is also calculated based on a 2-second sliding window, and its average value is denoted as... (Unit: N)

[0117] To eliminate dimensional differences and achieve personalized evaluation, the system first processes the aforementioned real-time raw feature values. and Normalization based on individual baselines is performed. The operational feature values ​​stored in the baseline data are calculated using rules strictly consistent with those of the real-time features. Normalization employs the min-max method, mapping the original real-time feature values ​​to the [0, 1] interval. The calculation formulas are as follows:

[0118]

[0119] in, , , , The baseline feature values ​​(minimum and maximum values) of the corresponding features recorded for the driver in the individual baseline pattern.

[0120] The two normalized eigenvalues ​​are then weighted and fused to generate the operational comprehensive standardized eigenvalues. :

[0121]

[0122] in, For the weighting of operational features, an optimal value of 0.6 was determined through experiments. This value reflects the core role of steering wheel operation stability in judging driving fatigue, while also reasonably incorporating grip symmetry as an effective supplement.

[0123] Thus, the comprehensive standardized feature value is obtained, i.e., step S404.

[0124] (5) When extracting visual multi-scale entropy features, the eyelid opening time series is analyzed. Perform coarse-graining treatment:

[0125]

[0126] Calculate the sample entropy for each coarse-grained sequence. The visual multi-scale entropy features are obtained as follows:

[0127]

[0128] in, This represents the scale factor, used to control the spatial resolution of the image signal for coarsening. This represents the total number of data points in the eyelid opening time series, where j is the index of the coarse-grained sequence; in this embodiment, we take... =20, embedding dimension m=2, similarity tolerance r=0.2 std(E) represents the standard deviation of the eyelid opening time series.

[0129] (6) Unlike vision and manipulation which use multi-scale entropy, physiological signals are discrete interval sequences, which are more suitable for being characterized by permutation entropy.

[0130] When extracting physiological permutation entropy features, the phase space of the RR interval sequence is reconstructed:

[0131]

[0132] Arrange each m-dimensional vector in ascending order to obtain a symbol sequence, and count the frequency of each arrangement. Then the permutation entropy is:

[0133]

[0134] After normalization, we get .

[0135] In this embodiment, m=3. =1.

[0136] (7) When performing multi-scale entropy feature extraction, the same multi-scale entropy calculation method as for vision is used for the steering wheel angle time series; for the steering wheel angle time series Perform coarse-graining treatment:

[0137]

[0138] in, Indicates the scale factor. This represents the total number of data points in the steering wheel angle time series, where j is the index of the coarse-grained sequence;

[0139] Calculate the sample entropy for each coarse-grained sequence. The operational multi-scale entropy features are obtained as follows:

[0140]

[0141] In this embodiment, the following is taken =20, embedding dimension m=2, similarity tolerance r'=0.2 std(E)', where std(E)' is the standard deviation of the steering wheel angle time series.

[0142] (8) Online clustering of driving style: Based on the driver's long-term driving behavior history data over 30 days, a driving style feature vector is constructed from the steering wheel angle change rate, following distance, lane change frequency, and acceleration change rate. .

[0143] A Gaussian mixture model is used to perform online clustering of driving style feature vectors, with the number of clusters set to k=3 (corresponding to three driving styles: aggressive, moderate, and conservative). The model parameters (mixture coefficients) are iteratively optimized using the EM algorithm. Mean vector Covariance matrix For a single driver sample, the posterior probability of belonging to each driving style is calculated using the following formula:

[0144]

[0145] Select the category k with the highest posterior probability and classify it as the driving style label corresponding to the driver.

[0146] This invention combines the physiological mechanisms of driving behavior with multimodal fatigue response characteristics to establish a preset mapping relationship between driving style categories and fatigue sensitivity coefficients. After determining the driving style label, fatigue sensitivity coefficients corresponding to visual, physiological, and operational modalities are obtained through table lookup matching, forming a driving style coefficient vector. The specific calibration value is: Aggressive type: Stable type: Conservative type: .

[0147] in, , , The fatigue sensitivity of visual, physiological, and operational modalities to the current driving style is quantified separately. This coefficient vector is used for the subsequent takeover urgency index calculation. Its L2 norm is used as the input parameter of the takeover urgency index T to reflect the differentiated impact of different driving styles on the degree of takeover urgency.

[0148] In S405, scene data and driving style labels are input. The real-time collected scene information and vehicle information, as well as the driving style labels output from online clustering of driving styles, are used as inputs for the dynamic weight calculation and fatigue state determination process.

[0149] III. After feature processing is completed, the system enters the dynamic weight calculation and fatigue state determination stage. The core decision fusion logic of this stage is as follows: Figure 2 As shown, the process is executed once every second to ensure real-time tracking of the driver's status. This process involves a large number of floating-point operations. To meet the real-time requirement of once per second, the system utilizes the high-performance parallel computing capabilities of the onboard computing unit to achieve this and improve computational efficiency.

[0150] S406. Scenario-based weight allocation: First, calculate the scenario complexity index S using the following formula:

[0151]

[0152] in, This represents the vehicle's real-time speed (unit: km / h). Set to 120km / h (the maximum speed limit on domestic highways); Real-time light intensity (unit: lux). Set to 100,000 lux (the threshold for strong light environments; exceeding this value will cause a decrease in the visual sensor's acquisition performance). α represents the rainfall level (levels 1-5, dimensionless, directly mapped from the rain sensor signal); β represents the road curvature (unit: 1 / km, curvature of straight road sections is 0, curvature of sharp curves is greater than 0.1); α=0.2, β=0.3, γ=0.2, δ=0.3 are scene coefficients, which are calibrated using a linear regression algorithm based on measured data from multiple sets of different scenes (covering sunny days, rainy days, nighttime, highways, urban areas, mountain roads, etc.). These coefficients correspond to the weights of scene factors such as vehicle speed, illumination, rainfall, and road curvature, respectively, ensuring that they accurately reflect the degree of influence of different scenes on the monitoring mode.

[0153] S407, AHP subjective weighting, follows a set three-level hierarchical structure, namely the target layer is the fusion weight of the three modalities of vision, physiology and operation ( , , The criteria layer consists of three core dimensions: scenario adaptability, style sensitivity, and data reliability. The solution layer consists of three modalities: visual, physiological, and operational, ensuring that the hierarchical logic and technical solutions are completely consistent.

[0154] Based on the inherent characteristics of driving safety scenarios, the priority of the criteria layer weights is set from high to low as follows: scenario adaptability 0.4, style sensitivity 0.3, and data reliability 0.3.

[0155] Based on the above priorities, the Saaty 1-9 scaling method is used to construct a judgment matrix for the relative importance of each dimension of the criteria layer. The matrix uses scenario adaptability, style sensitivity, and data reliability as evaluation dimensions, with scenario adaptability having the highest priority, followed by style sensitivity, and data reliability having the lowest priority. The matrix form is as follows:

[0156]

[0157] In the matrix, elements Indicates the first The factor relative to the first The importance of each factor is determined by the fact that the diagonal elements are always 1, indicating that each dimension is equally important compared to itself. The ratios of scene adaptability to style sensitivity and data reliability are 2 and 4 respectively, reflecting their relative importance. The ratio of style sensitivity to data reliability is 2, reflecting the difference in importance between the two. The matrix satisfies the property of a positively reciprocal matrix, i.e. = .

[0158] Calculate the largest eigenvalue of a matrix using eigenvalue decomposition. Then, the consistency index CI is calculated using the following formula:

[0159]

[0160] Where n=3 (matrix order); combined with the fixed average random consistency index RI=0.58 (industry-standard value) corresponding to a 3-order matrix, consistency verification is performed using the following formula:

[0161]

[0162] If CR < 0.1, the weight allocation is deemed reasonable and directly used for subsequent combination weighting. If CR ≥ 0.1, it indicates a logical inconsistency within the judgment matrix. In this case, the system will automatically call the preset backup weights for the corresponding scenario complexity range and driving style label. These backup weights are stored in the vehicle storage unit in the form of "scenario-driving style weight allocation rules," and their deviation from the main weights is strictly controlled within 0.05, and they have passed consistency verification (CR < 0.1) in advance. These backup weight rules have been calibrated through 100 sets of real-vehicle tests (covering different weather, roads, and driving groups) to ensure the logical rationality of the weight allocation at the criterion layer.

[0163] Based on the aforementioned criteria layer weights, and combined with different scenario complexity ranges and driving style labels, subjective weights are set for the solution layer.

[0164] in, , , These represent the subjective weights of the AHP in the visual, physiological, and operational modalities, respectively. The three weights satisfy the following conditions: To facilitate a unified expression of subsequent combination weighting formulas, let's denote... For modality The subjective weights of AHP, where represent (Corresponding to visual, physiological, and operational modalities, respectively). The allocation is based on the following criteria:

[0165] In simple scenarios (S<2, such as driving straight on a sunny highway with sufficient light and no rain), the recognition accuracy of visual features (such as blinking frequency and gaze direction) is the highest when there is sufficient light and the road is straight. Therefore, the visual weight is set to the highest value.

[0166] In moderate scenarios (2≤S<4, such as cloudy urban roads, light rain or moderate rain), the increased interference from changes in ambient light and slippery road surfaces leads to a decrease in visual recognition accuracy. It is necessary to appropriately increase physiological weights (such as heart rate and skin conductance) to enhance anti-interference capabilities.

[0167] In complex scenarios (S≥4, such as mountain roads at night during heavy rain, insufficient lighting, or many sharp bends), visual sensors are easily affected by severe weather or low light environments, resulting in a significant reduction in recognition reliability. In such cases, fatigue judgment is primarily based on physiological signals or operational behaviors (such as steering wheel vibration or accelerator pedal fluctuation).

[0168] Simultaneously, weights are adjusted based on individual differences in driving style: aggressive drivers exhibit more drastic changes in operational characteristics (such as steering wheel vibration) when fatigued, therefore their operational weight is appropriately increased; conservative drivers are more sensitive to changes in visual characteristics (such as blinking frequency) when fatigued, therefore their visual weight is higher; and the fatigue characteristics of steady drivers fall between the two, with their weight set to the middle value. Therefore:

[0169] When the scene complexity S < 2, in the conservative driving style: =0.5、 =0.3、 =0.2; In a smooth driving style: =0.45、 =0.35、 =0.2; In aggressive driving style: =0.4、 =0.3、 =0.3;

[0170] When the scene complexity 2 ≤ S < 4, in the conservative driving style: =0.4、 =0.4、 =0.2;

[0171] In a smooth driving style: =0.38、 =0.42、 =0.2; In aggressive driving style: =0.35、 =0.35、 =0.3;

[0172] When the scenario complexity S≥4, in the conservative driving style =0.25、 =0.5、 =0.25; In a smooth driving style: =0.2、 =0.5、 =0.3; In aggressive driving style: =0.2、 =0.3、 =0.5.

[0173] The aforementioned weight allocation rules are preset in the vehicle's storage unit in the form of "scenario-based and driving style-based weight allocation rules," and are automatically loaded when the system starts. These rules have been calibrated through 100 sets of real-vehicle tests (covering different weather conditions, roads, and driving groups) to ensure adaptability to real-world application scenarios.

[0174] After calculating the scenario complexity S-value in real time, the system determines its corresponding interval based on thresholds (the thresholds are calibrated and fixed at 2 and 4). Thresholds 2 and 4 are calibrated based on expert experience and real-vehicle test data, serving as the dividing points between simple, medium, and complex scenarios. Simultaneously, the system automatically matches the corresponding subjective weights from the preset rules, based on the style labels output by online clustering of driving styles. In case of abnormal situations (such as S-value exceeding the reasonable range of 0-10, or driving style labels failing to output), the system defaults to using the weights corresponding to scenario complexity 2 ≤ S < 4 and a stable driving style, ensuring the robustness of system operation.

[0175] S408 and EWM objective weighting, data normalization, and standardized feature values ​​for visual synthesis. Physiological comprehensive standardized characteristic values Operational comprehensive standardized characteristic values The min-max method is used for normalization, and the formula is:

[0176]

[0177] in represent (corresponding to visual, physiological, and operational modalities, respectively). , The minimum and maximum values ​​of the corresponding modal features recorded during the driver baseline acquisition phase (e.g., visual modality). Baseline period The minimum value, Baseline period The maximum value of the normalized data is strictly mapped to the interval [0, 1] to avoid dimensional differences between different modal features and individual baseline differences.

[0178] Based on the normalized feature data, the probability distribution of each modality is calculated according to the dimension of the sampling points, using the following formula:

[0179]

[0180] in The sampling point number is n=600 (corresponding to the cumulative data volume at a sampling frequency of 10Hz within 1 minute, taking into account both data representativeness and computational efficiency). For the first Normalized feature values ​​of each sampling point (e.g.) When i=10, (The visual normalized feature value of the 10th sampling point); probability. Characterizing the first The proportion of the feature value of each sampling point in all sampling points of that mode provides a basis for subsequent information entropy calculation.

[0181] Based on the probability distribution of each modality, the information entropy of each modality is calculated using the following formula to quantify the discriminative power of the feature data:

[0182]

[0183] in represent , (Corresponding to visual, physiological, and operational modalities), n=600 (total number of sampling points). The value range is [0, 1]. The smaller the entropy value, the more significant the difference between the modal features in the fatigue and non-fatigue states, and the higher the contribution to fatigue determination. If the entropy value of a certain modality is close to 1 (the feature discrimination is extremely low), the subsequent objective weight will be automatically reduced to reduce the interference of invalid features.

[0184] Difference coefficient calculation: The difference coefficient of each modality is derived by using information entropy, highlighting the weight proportion of high-discrimination features. The formula is as follows:

[0185]

[0186] Coefficient of difference Positively correlated with feature discriminative power The larger the value, the more accurately the modal feature can characterize changes in driver fatigue state, and the higher the subsequent objective weight will be assigned.

[0187] The difference coefficients of all modes are normalized to obtain the objective weights of each mode, ensuring that the sum of the weights is 1. The formula is as follows:

[0188]

[0189] in for Objective weights of modalities represent , , That is, the normalized objective weights of visual, physiological, and operational modalities, and satisfying + + =1, ensuring the rationality and operability of weight allocation.

[0190] S409. Combined weighting and fatigue state determination, dynamic fusion weight calculation: The multiplicative combination method is used to fuse AHP subjective weights and EWM objective weights, which not only reflects the inherent rules of scene and style adaptation, but also incorporates dynamic feedback from real-time data. The formula is as follows:

[0191]

[0192] in for The final fusion weights of the modalities, Representing v, p, and o, which are the normalized fusion weights for visual, physiological, and operational modalities, and satisfying the following conditions: + + =1.

[0193] S410. The final fusion weights are weighted and fused with the comprehensive standardized eigenvalues ​​of the corresponding modes to obtain the quantified comprehensive fatigue index F, as shown in the formula:

[0194]

[0195] The value of F ranges from [0, 1]. The larger the value, the higher the driver's fatigue level. If the modal weight is 0 (the feature is completely invalid in extreme cases), the modal feature value will not participate in the fatigue index calculation to ensure that the result is not affected by invalid data.

[0196] To introduce nonlinear dynamic evolution characteristics of signals and improve the sensitivity and precision of fatigue state identification, a nonlinear complexity factor C is defined.

[0197] Visual multiscale entropy features Physiological arrangement entropy characteristics Operational multi-scale entropy features The extreme value normalization method was used to achieve dimensional uniformity. Based on the maximum and minimum values ​​of each feature collected during the individual baseline stage, min–max normalization was performed. After normalization, the feature value range was unified to [0, 1].

[0198] Taking visual multi-scale entropy features as an example, the normalization calculation formula is:

[0199]

[0200] Similarly, the normalized physiological arrangement entropy characteristics are calculated sequentially. Multiscale entropy features with normalization operation .

[0201] We construct a nonlinear complexity factor C by weighting and fusing the three types of normalized nonlinear features:

[0202]

[0203] The weighting coefficients in the formula satisfy the normalization constraint. + + =1; This embodiment combines the calibration value with the driving fatigue response mechanism: =0.4、 =0.4、 =0.2, highlighting the dominant role of visual and physiological nonlinear characteristics in the fatigue evolution process.

[0204] Introducing correction coefficients By coupling the original comprehensive fatigue index with the complexity factor, the index is corrected to obtain the final fatigue index. :

[0205]

[0206] In the formula, the correction factor Belonging to [0, 1], it is used to adjust the proportion of the correction contribution of the nonlinear complexity factor; in this embodiment, it is taken as... =0.2, which means that the fatigue index of the original mode fusion is the main body and the nonlinear complexity feature is the auxiliary for collaborative correction.

[0207] Subsequent fatigue level determination and takeover urgency index calculation are both based on As the core quantitative basis.

[0208] S411. Determine the fatigue level based on the numerical range of the comprehensive fatigue index F: <0.3 indicates Level 0 is normal, meaning the driver is alert, focused, and shows no signs of fatigue; 0.3≤ A value <0.6 indicates Level 1 mild fatigue, where the driver shows slight signs of fatigue (such as a slightly increased blinking frequency) and a decline in attention; a value ≤0.6 indicates... <0.8 indicates Level 2 moderate fatigue, with the driver exhibiting significant fatigue, a marked decrease in reaction speed, and reduced operational stability; A fatigue level ≥0.8 indicates Level 3 severe fatigue, posing a significant safety risk to the driver and requiring immediate intervention. Once the fatigue level is determined, a corresponding tiered intervention strategy is generated and implemented.

[0209] IV. Risk Assessment and Differentiated Intervention for Takeover

[0210] S510, Calculate the urgency index of takeover.

[0211] Combined with comprehensive fatigue index The scenario complexity S and driving style coefficient θ quantify the urgency of the takeover request, and the formula is as follows:

[0212]

[0213] in The L2 norm of the style coefficient vector is calculated using the following formula:

[0214]

[0215] The range of values ​​is [0, [This is used to quantify the impact of driving style on takeover response (e.g., aggressive driver's...)] (usually higher) =0.5、 =0.3、 =0.2 is a weighting coefficient calibrated through testing, corresponding to the importance of fatigue state, scenario risk and driving style, respectively, to ensure that the index calculation fits the actual safety requirements.

[0216] S520. Implement differentiated interventions at levels 0-3 based on the range of values ​​for T.

[0217] Intervention command latency is strictly controlled within 200ms. It is distributed via the Controller Area Network (CAN) bus to the vehicle cockpit domain controller, body domain controller, powertrain domain controller, autonomous driving domain controller, steering system controller, and seatbelt controller, achieving multi-domain collaborative control. The specific strategy is consistent with the invention content, while supplementing execution details, such as... Figure 3 As shown:

[0218] Level 0 takeover (T<0.4): Assistance reminder is executed, and the cockpit domain controller displays a green normal status icon on the instrument panel (1 / 15 of the screen area, static display), without any additional audio or visual interference, to avoid affecting the driver's normal operation;

[0219] Level 1 takeover (0.4≤T<0.6): Implement recommended takeover. The cockpit domain controller triggers a voice prompt (volume 1.2 times the current volume, message: "Mild fatigue detected, please take a rest"), and simultaneously displays a yellow coffee cup warning icon (1 / 10 of the screen, flashing 2 times / second) for 30 seconds. For aggressive drivers, a slight vibration feedback is additionally output through the steering system controller (frequency 5Hz, amplitude ±2mm, lasting 1 second) to enhance the reminder effect without interfering with driving operations.

[0220] Level 2 Intervention (0.6≤T<0.8): Partial intervention is implemented. Based on Level 1 intervention, the vehicle domain controller drives the seat to intermittently vibrate (frequency 3Hz, amplitude ±5mm, 3 seconds of vibration, 2 seconds of stop, cycle 3 times), and the seat belt controller controls the seat belt to pre-tighten by 5cm (speed 5cm / s, held for 3 seconds). For conservative drivers, the lane departure warning system is activated in advance (warning threshold is reduced by 30% compared to the normal state). The driver can stop the vibrating and seat belt tightening through the custom button on the steering wheel, but the warning icon and voice prompt continue until F<0.3.

[0221] Level 3 Takeover (T≥0.8): Forced takeover is executed. The power domain controller limits the vehicle's output torque (maximum acceleration ≤0.05g) to avoid the risk of sudden acceleration. The autonomous driving domain controller activates the lane keeping function (lateral control accuracy ±0.2m) and plans the nearest safe parking point within 5km based on the onboard navigation (prioritizing service areas and emergency lanes). The body domain controller turns on the hazard warning lights (flashing frequency 1.5 times / second) and automatically closes the windows (speed 10cm / s) until the vehicle comes to a smooth stop and the engine is turned off. The driver is prohibited from disengaging the takeover control throughout the process to ensure driving safety.

[0222] V. Online Feedback Optimization

[0223] Step 1: Collect driver takeover response data after each intervention, including takeover response time (the time from the issuance of the intervention command to the driver's execution of the takeover operation, with an accuracy of ±100ms), operation correction range (steering wheel angle correction and vehicle speed adjustment within 3 seconds after takeover, with accuracies of ±0.5 degrees and ±0.1km / h, respectively), and correlate them with the corresponding fatigue index prediction values. Scene parameters (S value, driving style label).

[0224] Inferring the true fatigue index from control response data The reverse calculation rules have been verified through real-vehicle testing:

[0225] If the takeover response time is ≤1 second and the operation correction range is ≤5%, the prediction is judged to be conservative. = ×0.8;

[0226] If the takeover response time is greater than 3 seconds or the operation correction range is greater than 15%, the prediction is considered overly optimistic. = ×1.2;

[0227] In other cases: the prediction is generally accurate. = .

[0228] Step 2: Construct an optimization objective function to minimize the deviation between the predicted fatigue index and the actual fatigue index, ensuring that the system iteration direction aligns with the actual monitoring results. The formula is:

[0229]

[0230] Where n is the total number of intervention events within the optimization period (at least 10 to ensure the effectiveness of the optimization). For the first The predicted fatigue index value of the second intervention. For the first The true fatigue index of the intervention.

[0231] With the goal of minimizing the loss function Loss, the scene coefficients α, β, γ, and δ in the scene complexity calculation are iteratively optimized. The weights of the AHP criterion layer do not participate in the numerical iteration update, but still ensure their effectiveness through consistency checks. If the check fails, the corresponding backup weights are automatically called.

[0232] Step 3: Parameter Iterative Optimization. The gradient descent method is used to iteratively optimize the scenario coefficients α, β, γ, and δ. The optimization cycle is 24 hours, and it is executed in the background during periods when the vehicle is stationary and the system is idle, without affecting normal vehicle operation and safety control. The specific update formula is as follows:

[0233]

[0234]

[0235]

[0236]

[0237] in, For the updated scene coefficients, The initial scene coefficients are η = 0.001; the learning rate controls the optimization step size. This learning rate is chosen to balance the magnitude of parameter updates: an excessively large step size can lead to system instability during over-the-air (OTA) updates, while an excessively small step size cannot achieve effective convergence within the 24-hour optimization cycle. This value has been calibrated using multiple sets of scene data and verified through offline simulation, ensuring system stability and avoiding parameter oscillations while meeting the convergence requirements of this optimization mechanism.

[0238] The initial values ​​of the above scenario coefficients (α, β, γ, δ) are preset through offline data calibration. During system operation, they will be iteratively updated through this optimization mechanism: the system performs weight optimization once every 24 hours of accumulated driving data.

[0239] The optimized scenario coefficients must meet the constraint that the values ​​are between 0 and 0.5 and the sum is 1. Then, they are updated to the vehicle computing platform via OTA encryption to achieve self-performance improvement without manual intervention in a silent update manner. After the update, the module is restarted to make the new coefficients take effect.

[0240] The system is mounted on a conventional in-vehicle computing platform, and the program is compatible with the AUTOSAR architecture, making it easy to promote.

[0241] This implementation plan clarifies the sensor parameters, calculation formulas, and operation procedures, including multi-source data acquisition, feature extraction, dynamic weight calculation of AHP-EWM combined weighting, hierarchical intervention, and online feedback optimization. It is adapted to the engineering needs of automakers and provides an executable technical solution for driver fatigue monitoring and safe takeover in autonomous driving scenarios.

[0242] The present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program, it can implement all the steps of the above-mentioned fatigue monitoring method for autonomous driving based on multimodal perception, providing hardware support for the practical application of the system.

[0243] The embodiments described above are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A fatigue monitoring and autonomous driving takeover method based on driving style adaptation and multimodal fusion, characterized in that: (1) Multi-source data acquisition Acquire the driver's visual information, physiological information, operational information, long-term driving behavior history data, as well as driving scenario information and vehicle information; (2) Multi-scale feature extraction and online clustering of driving style Preprocessing and feature extraction are performed on visual, physiological, and operational information to obtain standardized visual comprehensive feature values. Physiological comprehensive standardized characteristic values and operation integrated standardized eigenvalues Meanwhile, driving style features are extracted based on long-term driving behavior historical data, and Gaussian mixture model is used to perform online clustering of driving styles, outputting driving style labels and corresponding style coefficients. Furthermore, multi-scale entropy and permutation entropy are introduced to extract the complexity features of each mode, which are used to construct the nonlinear complexity factor C; (3) AHP-EWM combined weighted multimodal fusion and fatigue state determination 1) Calculation of scene complexity index: Calculate the scenario complexity index based on driving scenario information and vehicle information; 2) Subjective weighting in the Analytic Hierarchy Process (AHP) Constructing a hierarchical structure: The target layer consists of multimodal feature fusion weights. , , The criteria layer includes three dimensions: scenario adaptability, style sensitivity, and data reliability; the solution layer includes three modalities: visual, physiological, and operational. Based on the inherent characteristics of driving safety scenarios, the weight priority of the criterion layer is set. The Saaty1-9 scaling method is used to construct a judgment matrix based on the relative importance of each dimension of the criterion layer, calculate the maximum eigenvalue of the matrix judgment, and calculate the consistency index to perform consistency verification and ensure the rationality of weight allocation. Based on the aforementioned criteria layer weights, and combined with different scenario complexity ranges and driving style labels, subjective weights are set for the solution layer. 3) Objective weighting using the entropy weight method Regarding the , , Normalization is performed, and the probability of each mode is calculated based on the normalized data. Then, the information entropy and difference coefficient of each mode are calculated to obtain the objective weight. 4) Combination weighting and fatigue index calculation: A multiplicative combination method is used to fuse subjective and objective weights to obtain the final dynamic fused weight. This final dynamic fused weight is then weighted and fused with the corresponding standardized eigenvalues ​​to obtain the comprehensive fatigue index F. The nonlinear complexity factor C is used to dynamically correct the comprehensive fatigue index, resulting in the final fatigue index. Then based on The fatigue level is determined by the numerical range. (4) Risk assessment and differentiated intervention for takeover Combined with the final fatigue index The scenario complexity S and driving style coefficient θ are used to calculate the takeover urgency index T. Based on the T value and driving style label, a style-differentiated hierarchical intervention strategy is generated and sent to the vehicle's domain controllers to achieve multi-domain collaborative control. (5) Optimization of online feedback Data on the driver's takeover response after each intervention is collected, an optimization objective function is constructed, and the scenario coefficients are iteratively optimized using the gradient descent method to ensure that the system adapts to changes in driving style and scenario differences over the long term.

2. The fatigue monitoring and automatic driving takeover method according to claim 1, characterized in that, The scenario complexity index ,in, This refers to the vehicle's real-time speed. This is the highest speed limit on domestic highways. For real-time light intensity, The threshold for determining strong light environments. Rainfall level, α represents the unit of road curvature, and α, β, γ, and δ represent scene coefficients.

3. The fatigue monitoring and automatic driving takeover method according to claim 1, characterized in that, The weights of the criteria layer, from highest to lowest priority, are: scene adaptability (0.4), style sensitivity (0.3), and data reliability (0.3). The judgment matrix uses scene adaptability, style sensitivity, and data reliability as evaluation dimensions, with scene adaptability having the highest priority, followed by style sensitivity, and data reliability having the lowest priority. The matrix form is as follows: Calculate the matrix to determine the largest eigenvalue This leads to the consistency index: Where n is the order of the matrix; Combining the average random consistency index RI corresponding to the 3rd order matrix, through Perform a consistency check to ensure that CR < 0.

1.

4. The fatigue monitoring and automatic driving takeover method according to claim 1, characterized in that, The subjective weights at the scheme layer include visual subjective weights. Physiological subjective weight Operational subjective weight ,and ; Based on the scenario complexity index, the method for allocating subjective weights is as follows: When the scene complexity S < 2, in the conservative driving style: =0.5、 =0.3、 =0.2, in a stable driving style: =0.45、 =0.35、 =0.2, in an aggressive driving style: =0.4、 =0.3、 =0.3; When the scene complexity 2 ≤ S < 4, in the conservative driving style: =0.4、 =0.4、 =0.2, In a smooth driving style: =0.38、 =0.42、 =0.2, in an aggressive driving style: =0.35、 =0.35、 =0.3; When the scene complexity S≥4, in the conservative driving style: =0.25、 =0.5、 =0.25, in a stable driving style: =0.2、 =0.5、 =0.3, in an aggressive driving style: =0.2、 =0.3、 =0.

5.

5. The fatigue monitoring and automatic driving takeover method according to claim 4, characterized in that, The probability of each mode is calculated based on normalized data. Where x represents v, p, and o, namely visual, physiological, and operational modalities. The sampling point number, The total number of sampling points. For the first Normalized eigenvalues ​​of each sampling point; calculate the information entropy of each modality. Calculate the difference coefficients for each mode. The objective weight ,in for The objective weights of the modalities, and + + =1.

6. The fatigue monitoring and automatic driving takeover method according to claim 5, characterized in that, The final dynamic fusion weight ,in, for The final fusion weights of the modalities, For modality Subjective weights, among which represent These correspond to visual, physiological, and operational modalities, respectively; thus, the multimodal feature fusion weights of the target layer are obtained. , , ,and + + =1.

7. The fatigue monitoring and automatic driving takeover method according to claim 6, characterized in that, The comprehensive fatigue index Nonlinear complexity factor , To normalize the visual multi-scale entropy features, To normalize the physiological arrangement entropy, To normalize the multi-scale entropy, , , The corresponding weighting coefficients; final fatigue index: , This is a correction factor.

8. The fatigue monitoring and automatic driving takeover method according to claim 7, characterized in that, According to the final fatigue index The numerical value is used to determine the fatigue level, specifically: <0.3 indicates level 0 is normal, 0.3≤ <0.6 indicates Level 1 mild fatigue, 0.6≤ <0.8 indicates level 2 moderate fatigue. A value ≥0.8 indicates level 3 severe fatigue.

9. The fatigue monitoring and automatic driving takeover method according to claim 8, characterized in that, The urgency index of the takeover ,in Let L2 norm be the style coefficient vector. , , The weighting coefficients are experimentally calibrated; the hierarchical intervention strategy generates style-differentiated hierarchical intervention strategies, which are sent to the vehicle's domain controllers to achieve multi-domain collaborative control, specifically: When T < 0.4, an auxiliary reminder is executed, and the cockpit domain controller displays a green normal status icon; When 0.4≤T<0.6, the recommended takeover is executed, triggering a voice prompt + yellow coffee cup warning icon. For aggressive drivers, an additional slight vibration feedback is added to the steering wheel. When 0.6≤T<0.8, partial takeover is implemented. Based on Level 1 intervention, the seat experiences intermittent strong vibrations, and the seat belt is pre-tightened. Conservative drivers activate lane departure warning in advance. When T≥0.8, forced takeover is executed, the vehicle output torque is limited, the lane keeping function is activated and the nearest safe stopping point is planned, and the hazard warning lights are turned on.

10. The fatigue monitoring and automatic driving takeover method according to claim 9, characterized in that, The optimization objective function ,in, To predict the fatigue index, This is the true fatigue index derived from the data on the response to the takeover.

Images