A Confidence-Driven Time-Window Adaptive Fatigue Detection and Response Method

Abstract

This invention discloses a confidence-driven time-window adaptive fatigue detection and response method, belonging to the field of intelligent driving safety technology. The method includes: dynamically initializing a first time window based on multiple factors such as driving scenario risk and driver state; acquiring driver image information within this time window, identifying fatigue characteristics and determining a first fatigue index, calculating a confidence level, and dynamically adjusting the time window based on the first fatigue index and the confidence level to obtain an optimized second time window; updating the assessment of driver fatigue state based on this second time window to determine a second fatigue index; and using the second fatigue index and the confidence level together as a decision-making basis to execute a graded response matching the fatigue level and risk scenario. This invention effectively solves the contradiction between real-time performance and computational efficiency in existing technologies, as well as the problems of a single and unreliable response mechanism, significantly improving the accuracy, real-time performance, and overall system safety of fatigue detection.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving safety technology, and more specifically, to a confidence-driven time window adaptive fatigue detection and response method. Background Technology

[0002] Fatigue driving detection technology is a key technology in the fields of intelligent transportation and active vehicle safety. It aims to determine a driver's fatigue state by analyzing their physiological and behavioral characteristics, thereby effectively preventing traffic accidents. With the development of computer vision and artificial intelligence technologies, vision-based non-contact fatigue detection methods have become mainstream. This method typically uses a camera to capture images of the driver's face and analyzes biometric indicators such as blink frequency (BPM), percentage of eyelid closure time (PERCLOS), and yawn frequency to assess the degree of fatigue.

[0003] However, existing technical solutions generally suffer from the following two main drawbacks:

[0004] (1) Most systems use a fixed time window for image acquisition and feature analysis. This static strategy is not adaptable to dynamic and changing driving scenarios. In low-risk scenarios (such as driving straight on a highway during the day), fixed high-frequency sampling will cause unnecessary waste of computing resources, which is particularly disadvantageous for embedded vehicle platforms with limited computing power; while in high-risk scenarios (such as complex road conditions at night), a fixed sampling frequency may not be sufficient to capture rapid changes in fatigue state, resulting in response delay and missing the best intervention opportunity.

[0005] (2) Most existing systems employ a simple "threshold-action" mapping mechanism. Once the detected fatigue index exceeds a preset threshold, a fixed alarm is triggered, such as a voice, light, or seat vibration. This response method is relatively simple and lacks graded and progressive intervention strategies for different levels of fatigue and different scenario risks. More importantly, it fails to fully consider the reliability of the detection results themselves and is prone to misjudgment due to factors such as changes in lighting, facial occlusion, or individual differences, thereby triggering unnecessary alarms, interfering with normal driving, and reducing user acceptance and trust.

[0006] Therefore, how to design a fatigue driving detection method that can dynamically adjust the monitoring strategy according to the driving scenario and classify and intelligently respond based on the reliability of the detection results is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0007] The present invention provides a confidence-driven time window adaptive fatigue detection and response method to solve the problems of difficulty in balancing real-time monitoring and computational efficiency, and the single and passive response mechanism in existing fatigue driving detection technologies.

[0008] To solve the above problems, the technical solution adopted by the present invention is as follows:

[0009] A confidence-driven time-window adaptive fatigue detection and response method includes:

[0010] The first time window is dynamically initialized based on multiple factors;

[0011] Based on the first time window, the driver's image information is collected and the driver's fatigue characteristics are identified, thereby determining the first fatigue index;

[0012] The confidence level of the first fatigue index is calculated, and based on the first fatigue index and the confidence level, the first time window is dynamically adjusted through a nonlinear double feedback adjustment function to obtain a second time window; wherein, the nonlinear double feedback adjustment function introduces a confidence adjustment term that increases as the confidence level decreases, and a fatigue index adjustment term that increases as the first fatigue index increases, to form a comprehensive adjustment factor; the method for determining the second time window includes: calculating the length of the first time window with the comprehensive adjustment factor, and using preset minimum and maximum time window values ​​to impose boundary constraints on the calculation result;

[0013] Based on the second time window update, the driver's fatigue status is assessed, the driver's fatigue characteristics are obtained, and the second fatigue index is determined.

[0014] Based on the second fatigue index and the confidence level, a graded response matching the fatigue state is executed.

[0015] Furthermore, methods for dynamically initializing the first time window based on multiple factors include:

[0016] The following methods are used to calculate scenario risk factors, driver state prior factors, and historical performance factors. The scenario risk factor is calculated by weighting and fusing risk coefficients representing different time periods, road types, and weather conditions. The driver state prior factor is calculated using a driving duration function that simulates the cumulative effect of fatigue, combined with a driver-specific coefficient learned from historical data. The historical performance factor is calculated by constructing a query vector, retrieving historical time window configurations and their response performance scores for similar scenarios from a historical database, and then weighting the retrieval results.

[0017] The first time window is obtained by weighted geometric mean fusion of the scenario risk factors, driver state prior factors, and historical performance factors.

[0018] Furthermore, in the process of identifying fatigue characteristics and determining the first fatigue index, personalized thresholds were used to determine fatigue characteristics, and personalized weighting coefficients were used for weighted fusion.

[0019] The method for generating the personalized threshold includes: recording the driver's baseline eye aspect ratio EAR_base and baseline mouth aspect ratio MAR_base within the initial wakefulness monitoring window after vehicle startup, and multiplying them by a preset scaling factor to generate a personalized EAR threshold and a personalized MAR threshold.

[0020] The method for determining the first fatigue index includes: extracting and normalizing blinking frequency, yawning frequency, and eyelid closure time percentage as fatigue features; dynamically assigning personalized weight coefficients to each normalized fatigue feature, wherein the method for assigning personalized weight coefficients includes referencing the historical statistical correlation between each fatigue feature and the actual fatigue state label; and summing the weighted fatigue features.

[0021] Furthermore, the method for calculating the confidence level of the first fatigue index includes:

[0022] Calculate the physiological characteristic consistency factor, vehicle behavior factor, and environmental disturbance assessment factor;

[0023] The calculation method for the physiological characteristic consistency factor includes assessing the degree of dispersion among multiple normalized physiological fatigue characteristic values, including blinking frequency, yawning frequency, and eyelid closure time percentage; the assessment method for the vehicle behavior factor includes determining whether the standard deviation of the vehicle's lateral displacement within the current time window exceeds a preset threshold; and the evaluation method for the environmental interference assessment factor includes comprehensively utilizing ambient light sensor data and camera image quality assessment results.

[0024] The confidence level is obtained by weighted fusion of the physiological characteristic consistency factor, vehicle behavior factor, and environmental interference assessment factor.

[0025] Furthermore, a graded response is implemented based on a response level arbitration mechanism. This arbitration mechanism includes: establishing a multi-level response strategy library encompassing levels ranging from neural regulation, tactile alert, vehicle restraint, active avoidance, to emergency rescue. For low response levels (neural regulation and tactile alert) involving early warning and mild intervention, triggering is based on whether the second fatigue index reaches the threshold of the corresponding range. For high response levels (vehicle restraint, active avoidance, and emergency rescue) involving active vehicle control or emergency rescue, the second fatigue index must reach a higher threshold, and the confidence level must also exceed the incremental confidence safety threshold set for that higher level before execution can be triggered.

[0026] Furthermore, when the neuromodulation level response is triggered, alpha brainwaves of a specific frequency are played through the vehicle audio system to neuromodulate the driver and promote his concentration. The generation of the operation command when the neuromodulation level response is triggered depends on the extracted eye and mouth fatigue characteristics and the calculated mild fatigue index.

[0027] When a tactile alert response is triggered, the vibration motor inside the steering wheel is controlled to generate pulse vibrations that are synchronized with the driver's physiological rhythm, which is calculated based on the frequency of blinking and yawning.

[0028] When the vehicle constraint level response is triggered, the vehicle powertrain is controlled to limit the vehicle speed to below a preset safe value. Combined with map data, when the vehicle is about to enter a curve, the safe passing speed is actively calculated based on the road curvature and adaptive speed reduction is performed in advance.

[0029] When the active avoidance level response is triggered, the vehicle automatically finds and plans a safe parking route to the nearest emergency parking area or service area based on the map and real-time centimeter-level positioning. At the same time, it coordinates with roadside units and surrounding vehicles through vehicle-to-the-world (V2X) communication to inform the vehicle of its status in advance and request the clearing of the target area.

[0030] Furthermore, the emergency rescue-level response is triggered when both the second fatigue index and confidence level reach the highest danger threshold. First, it controls the vehicle's air conditioning to release preset stimulating gas, and then automatically triggers an emergency rescue call through the vehicle's T-Box terminal, sending the vehicle's location and driver status information, including fatigue characteristic data, to the rescue center.

[0031] Furthermore, after executing the hierarchical response, it also includes link data recording and closed-loop learning: the complete event chain data from the initialization scenario vector to the final response result is structured and recorded and stored in the historical database. The recorded complete event chain data is used to continuously optimize and calibrate the historical performance factor, confidence calculation model, and trigger threshold of each response level in the initialization model of the first time window offline.

[0032] Furthermore, the image acquisition strategy based on resource adaptation acquires the driver's image information, including: forming decision conditions based on the scene risk factors determined after assessing the driving environment and the real-time monitoring of the system's central processing unit (CPU) load; if the scene risk factors or CPU load are higher than the preset high-risk threshold, a high-priority single-frame capture command is sent to the camera; if the high-risk decision conditions are not met, a standard video stream sampling mode is adopted to acquire short video segments according to the duration defined by the first time window.

[0033] Compared with the prior art, the beneficial effects of the present invention are:

[0034] (1) The present invention enables the system to automatically extend the monitoring interval in low-risk scenarios by using a time window mechanism based on multi-factor dynamic initialization and confidence feedback adjustment, thereby saving ineffective computational load and improving the operating efficiency on the embedded platform; while in high-risk or low-confidence scenarios, the monitoring window can be shortened quickly to achieve high-frequency sampling, ensuring timely capture and response to rapid changes in fatigue state, and significantly improving the real-time performance and scenario adaptability of the system.

[0035] (2) By introducing personalized feature threshold learning and a multimodal weighted fusion model, this invention can adapt to the physiological differences of different drivers (such as eye size, facial contour, etc.) and dynamically adjust the weight of each fatigue feature in the comprehensive evaluation according to the sensitivity of each individual. Compared with the traditional method using fixed thresholds and weights, the false alarm rate of this invention is greatly reduced, especially when dealing with complex situations such as wearing glasses and different lighting conditions, it shows stronger robustness.

[0036] (3) This invention innovatively introduces a confidence assessment mechanism, using both the fatigue index and confidence level as dual bases for response decisions, effectively avoiding driving interference caused by misjudgment of a single threshold. Based on this, a five-level progressive response strategy from neural modulation to emergency rescue is established, achieving a smooth transition from non-invasive early warning to active safety intervention. In real road tests, the false trigger rate of high-level intervention measures is greatly reduced, significantly improving the reliability of the system and user acceptance, and constructing comprehensive safety protection.

[0037] (4) By recording data and providing performance feedback on the complete event chain, this invention constructs a closed-loop system with self-learning and self-evolution capabilities. This system can continuously optimize its initialization strategy, confidence model, and response logic as usage time increases, adapting to changes in driving modes in different regions and seasons, and possessing long-term evolution capabilities that traditional static systems lack.

[0038] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, embodiments of the present invention are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the overall system workflow of an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the driving duration function used to simulate the cumulative effect of fatigue in an embodiment of the present invention.

[0042] Figure 3 This is a schematic diagram of eye feature points in an embodiment of the present invention.

[0043] Figure 4 This is a schematic diagram of the mouth feature points in an embodiment of the present invention.

[0044] Figure 5 This is a schematic diagram of the dynamic time window initialization model in an embodiment of the present invention.

[0045] Figure 6 This is a flowchart of the multimodal weighted fatigue index calculation in an embodiment of the present invention.

[0046] Figure 7 This is a flowchart of confidence calculation and window adjustment in an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0048] Reference Figure 1 This invention provides a confidence-driven time window-based adaptive fatigue detection and response method, comprising the following steps:

[0049] Step S1: Dynamically initialize the first time window based on multiple factors.

[0050] In this embodiment, the first time window T_initial (i.e., sampling interval and duration) for the first image sampling is determined by a weighted fusion model. Then, based on the first time window, the driver's image information (photos, videos) is acquired through the camera.

[0051] Combination Figure 3 and Figure 5 This step can be further divided into the following sub-steps:

[0052] S11, Initialization calculation of the first time window

[0053] 1. Set scenario risk factor E

[0054] Assumption This represents the scenario risk score, which is a weighted score combining the risks of time period, road, and weather. R_time represents the time period risk coefficient (late night = 0.9, dusk = 0.7, peak hour = 0.5, daytime = 0.3), R_road represents the road risk coefficient (highway = 0.8, urban road = 0.6, rural road = 0.4), and R_weather represents the weather risk coefficient (heavy rain = 0.9, fog = 0.8, light rain = 0.6, sunny = 0.1).

[0055] The calculation method for scenario risk factors includes introducing risk coefficients representing different time periods, road types, and weather conditions for weighted fusion, that is, first constructing a scenario risk score:

[0056]

[0057] Where t, r, and w are the weights of each risk, satisfying t + r + w = ​​1.

[0058] Then, the risk score is mapped to a window adjustment factor (i.e., the scenario risk factor). The higher the risk, the shorter the window should be. We use a negative exponential mapping relationship to calculate the scenario risk factor E:

[0059]

[0060] Where T_max and T_min are the maximum and minimum time windows allowed by the system (in seconds), respectively, and k is a scaling factor that controls the decay rate.

[0061] 2. Set the driver's state prior factor D

[0062] The method for calculating the driver state prior factor D involves using a driving duration function that simulates the cumulative effect of fatigue, combined with a driver personalization coefficient learned from historical data.

[0063] Reference Figure 2 The figure shows a schematic diagram of the driving duration function f(t_drive) simulating the cumulative effect of fatigue, which uses a logical sigmoid curve:

[0064]

[0065] Where t_drive represents the cumulative driving time after this ignition (in minutes), a is the coefficient of the steepness of the control curve, b is the inflection point of the fatigue acceleration accumulation time, and T_max and T_min are the maximum and minimum time windows allowed by the system (in seconds).

[0066] Then, based on the duration of this drive and the driver's historical behavior patterns, the driver's state prior factor D can be calculated using the following formula:

[0067]

[0068] Where C_driver represents the driver personalization coefficient (the average fatigue rate of the driver is learned from historical data).

[0069] 3. Set the historical performance factor H

[0070] Setting the historical performance factor H involves searching a vector database and using weighted voting to find the initialization configuration with the highest confidence level in similar scenarios from historical big data. In this embodiment, the calculation method for the historical performance factor H includes constructing a query vector, retrieving historical time window configurations and their response performance scores for similar scenarios from the historical database, and performing weighted calculations on the search results, as follows:

[0071] First, construct the query vector Q:

[0072] (After normalization)

[0073] Then, an approximate nearest neighbor search is performed:

[0074] Query the vector database for the K historical scene vectors S_i that are most similar to Q.

[0075] The similarity between the query vector Q and any historical scene vector S_i is measured by the similarity calculation function sim(Q, S_i). The calculation formula is as follows:

[0076]

[0077] In the formula, n represents the vector dimension (4 here, corresponding to the four components R_time, R_road, R_weather, and t_drive). and Let |Q| and |S_i| represent the j-th components of vectors Q and S_i, respectively, where |Q| and |S_i| represent the magnitudes of vectors Q and S_i, respectively. The function returns values ​​in the range [-1, 1], but after normalization, it can be used to represent the degree of similarity; a larger value indicates a higher degree of similarity.

[0078] Then, the performance weights are calculated. Each historical scene vector S_i corresponds to a time window T_i and a performance score P_i, which is obtained by the following formula:

[0079]

[0080] Where T_response represents the historical response time. It refers to the time elapsed from when the system detects fatigue to when it completes effective intervention (such as the driver returning to normal or the vehicle safely stopping) in a certain historical scenario S_i. The shorter the T_response, the higher the efficiency of the time window configuration T_i used in that historical scenario, and the higher the score P_i.

[0081] Finally, the historical performance factor H is calculated:

[0082]

[0083] 4. Calculate the first time window T_initial

[0084] Finally, the three factors mentioned above are fused using a weighted geometric mean to obtain the first window T_initial:

[0085]

[0086] Where u, v, and z are the weight coefficients of scenario risk factor E, driver state prior factor D, and historical effectiveness factor H, respectively, satisfying u+v+z=1.

[0087] S12, Resource-Adaptive Image Acquisition Execution

[0088] First, decision conditions are formed based on the calculated scenario risk score R_scenario and the real-time monitored system CPU load L_cpu:

[0089] If (R_scenario > R_high) or (L_cpu > L_high), then "high-priority single-frame capture" is used. A command is sent to the camera to interrupt the current task and immediately capture a high-resolution still image. The image resolution can be appropriately reduced to further reduce transmission and processing latency.

[0090] Otherwise, "standard video stream sampling" is used, and short video segments are collected at intervals defined by T_initial.

[0091] Where R_high represents the high-risk threshold of the scenario, and L_high represents the critical point at which the system's computing resources are saturated.

[0092] S13, Output

[0093] Based on the first time window T_initial, a segment of driver image data (video or image) is collected, and this data will be sent to step S2 for feature recognition.

[0094] In this embodiment, step S1 constructs a dynamic optimization model to adapt to the initialization time window by integrating multi-source prior information such as environment, driver status and historical performance. This achieves a precise match between computing resources and safety requirements, and significantly improves the real-time performance, accuracy and resource efficiency of the monitoring system.

[0095] Step S2: Identify driver fatigue characteristics based on image information.

[0096] Based on the first time window and driver image information obtained in step S1, the state of the eyes and mouth is analyzed in real time to extract key physiological features for fatigue assessment. This process utilizes personalized thresholds for judgment. The system first learns and generates personalized EAR and MAR thresholds specific to the driver at the initial stage of vehicle startup. Subsequently, through facial landmark detection technology, the eye aspect ratio (EAR) and mouth aspect ratio (MAR) of each frame are calculated. Based on this, the time series is analyzed to calculate key fatigue features such as blink frequency (BPM), yawn frequency (Yawn_Freq), and eyelid closure time percentage (PERCLOS). Details are as follows:

[0097] S21, Resource-Adaptive Feature Recognition Strategy

[0098] First, the instructions from S1 are parsed, and the program enters different processing branches:

[0099] If the instruction received from S1 is "high priority single frame capture", then only the fastest feature analysis is performed on the current single frame image to ensure low latency. Then, jump to step S22 to calculate the EAR and MAR values ​​of the current frame, without performing any time-consuming time series analysis (such as blink detection, yawn detection, PERCLOS calculation).

[0100] If the instruction received from S1 is "standard video stream sampling", then perform complete feature extraction and analysis on the short video segments captured at intervals defined by T_initial output by S1.

[0101] S22, Personalized Adaptive Threshold Learning Mechanism

[0102] First, based on the initial wakefulness monitoring window after vehicle startup (first 3-5 minutes), the driver's raw EAR and MAR values ​​EAR_base and MAR_base are continuously calculated and recorded. EAR_base is the stable average value of the driver's EAR value when the eyes are open during the baseline learning phase, and MAR_base is the stable average value of the driver's MAR value when the mouth is closed during the baseline learning phase.

[0103] Then, perform personalized threshold calculation:

[0104] The personalized EAR threshold (EAR_thresh_personal) is calculated as follows:

[0105]

[0106] Where k_ear is the eye closure ratio coefficient, which is an empirical value determined based on a large number of experimental statistics (preferred range 0.5-0.75).

[0107] Personalized MAR threshold (MAR_thresh_personal):

[0108]

[0109] Where k_mar is the mouth opening ratio coefficient, which is also an empirical value (preferred range 1.4-2.0).

[0110] S23. Extraction of basic features of the eyes and mouth

[0111] First, such as Figure 3 As shown, the coordinates of 6 feature points for each of the left and right eyes are extracted. The eye aspect ratio (EAR) can be calculated using the following formula:

[0112]

[0113] Wherein, P1 is the left corner of the eye (point 37 for the left eye and point 43 for the right eye), P2 is the midpoint of the upper boundary of the eye (point 38 for the left eye and point 44 for the right eye), P3 is the right corner of the eye (point 39 for the left eye and point 45 for the right eye), P4 is the midpoint of the lower boundary of the eye (point 40 for the left eye and point 46 for the right eye), P5 is the right half of the eye, located between P3 and P4 (point 41 for the left eye and point 47 for the right eye), and P6 is the left half of the eye, located between P1 and P2 (point 42 for the left eye and point 48 for the right eye).

[0114] Subsequently, as Figure 4 As shown, the coordinates of 20 feature points around the mouth are extracted, and the mouth aspect ratio (MAR) value can be calculated using the following formula:

[0115]

[0116] Where P48 is the left corner of the mouth (point 48), P51 is the midpoint of the upper boundary of the upper lip (point 51), P54 is the right corner of the mouth (point 54), P53 is a point on the right half of the upper lip (point 53), P57 is a point on the left half of the upper lip (point 57), and P59 is the midpoint of the lower boundary of the lower lip (point 59).

[0117] S24. Time Series Feature Analysis and Calculation

[0118] First, determine the blink rate:

[0119] Within the T_initial time window, monitor the EAR sequence. When the EAR value continuously falls below the personalized EAR threshold EAR_thresh_personal, it is considered a blink.

[0120] The blink frequency is then obtained using the following formula. :

[0121]

[0122] Count_blink represents the number of blinks within that window.

[0123] Then, yawn detection is performed: within the T_initial time window, the MAR sequence is monitored. When the MAR value continuously exceeds the personalized threshold MAR_thresh_personal for a predetermined number of frames (preferably 15 frames), it is determined as a yawn.

[0124] Finally, the yawning frequency is obtained using the following formula. :

[0125]

[0126] Count_yawn represents the number of yawns within that window.

[0127] S24, PERCLOS calculation

[0128] First, within the T_initial time window, the EAR_value calculated for each frame (i.e., the instantaneous value of the eye aspect ratio calculated in real time based on eye feature points, used to characterize the degree of eye opening in each frame) is compared with the personalized eye closure determination threshold EAR_thresh_personal calculated based on the driver's personal baseline features to determine the eyelid state of that frame:

[0129] If EAR_value < EAR_thresh_personal, the frame is determined to be in the "eyelid closed" state. Otherwise, the frame is determined to be in the "eyelid open" state.

[0130] Then, the total number of frames judged as "eyelid closed" within the T_initial time window is counted and denoted as N_closed. Meanwhile, the total number of video frames N_total within this time window can be calculated using the camera frame rate (FPS, frames per second) and the time window length T_initial (seconds).

[0131]

[0132] Finally, the PERCLOS value is obtained by the following formula:

[0133]

[0134] PERCLOS represents the percentage of time the driver's eyelids are closed (or the pupils are covered) within a specified time window.

[0135] In this embodiment, step S2 introduces baseline learning and scaling factor to transform the general algorithm into a personalized model for a specific driver, fundamentally solving the problem of misjudgment caused by individual physiological differences. The feature recognition depth of S2 is controlled by S1, and the feature judgment threshold is generated based on the driver's own state, forming a full-link adaptive mechanism of "scene-driven acquisition -> acquisition-driven analysis -> individual feature-driven judgment".

[0136] Step S3: Determine the first fatigue index.

[0137] In this embodiment, refer to Figure 4 Based on the normalized fatigue features (blinking frequency, yawning frequency, PERCLOS) extracted in step S2, a comprehensive and quantitative first fatigue index is calculated by fusing personalized weighting coefficients with a multimodal linear weighted model. Details are as follows:

[0138] S31. Multimodal fatigue feature extraction and normalization processing

[0139] First, obtain the following fatigue characteristic data calculated in real time from step S2:

[0140] Blink frequency (BPM) is measured in blinks per minute.

[0141] Yawning frequency (Yawn_Freq), measured in times per minute;

[0142] The percentage of eyelid closure time (PERCLOS) is defined as the percentage of time the eyelids are closed within a time window.

[0143] Subsequently, the blink frequency, yawn frequency, and PERCLOS were normalized and mapped to the [0,1] interval. The normalization formula is as follows:

[0144] Normalized BPM:

[0145]

[0146] Normalized Yawn_Freq:

[0147]

[0148] Normalized PERCLOS:

[0149]

[0150] Where BPM_min and BPM_max are the preset minimum and maximum blink frequencies, respectively; Yawn_Freq_min and Yawn_Freq_max are the minimum and maximum yawn frequencies, respectively; and PERCLOS_min and PERCLOS_max are the minimum and maximum PERCLOS frequencies, respectively. After normalization, we obtain BPM_norm, Yawn_Freq_norm, and PERCLOS_norm.

[0151] S32. Correlation calculation based on characteristics and fatigue state

[0152] First, calculate the statistical correlation between each fatigue feature and the actual fatigue state. Features with stronger correlations are assigned higher importance scores.

[0153]

[0154] in, It is an importance score for a specific feature. It is the Pearson correlation coefficient between this feature and the fatigue state label.

[0155] Subsequently, data from N fatigue events were extracted from the historical database, including feature values ​​and fatigue state labels: (For example: alertness = 0, mild fatigue = 1, severe fatigue = 2), the correlation coefficient between each feature and the fatigue state label is obtained by the following formula:

[0156]

[0157]

[0158]

[0159] in These are three fatigue characteristics and their corresponding labels for actual fatigue state: blink frequency (BPM), yawn frequency (Yawn_Freq), and eyelid closure time percentage (PERCLOS). The Pearson correlation coefficient between the features and fatigue state is used to quantify the degree of linear correlation between each feature and fatigue state. Its value ranges from -1 to 1, with positive values ​​indicating a positive correlation (the larger the feature value, the more severe the fatigue) and negative values ​​indicating a negative correlation. The closer the absolute value is to 1, the stronger the correlation. N represents the number of fatigue events. It's a fatigue status label. This indicates a fatigue status label. The arithmetic mean, This represents the arithmetic mean of all blink frequency samples from N fatigue events extracted from the historical database. This represents the arithmetic mean of the yawning frequency samples from N fatigue events extracted from the historical database. This represents the arithmetic mean of the percentage sample values ​​of all eyelid closure times from N fatigue events extracted from the historical database. This represents the value of the i-th blink frequency sample. This represents the frequency sample value of the i-th yawn. This represents the percentage value of the i-th eyelid closure time sample.

[0160] For the three correlation coefficients Importance rating Their calculation method is to take the absolute value of the corresponding correlation coefficient, that is:

[0161] , .

[0162] S33, Personalized Weight Adaptive Learning Mechanism

[0163] First, weights are dynamically assigned based on feature sensitivity. The weight calculation formula is as follows:

[0164]

[0165]

[0166]

[0167] Where α, β, and γ are The weighting coefficients satisfy α+β+γ= 1.

[0168] S34. Calculation of the first fatigue index of multimodal weighted index

[0169] Combination Figure 6 As shown, the first fatigue index is calculated using the following linear weighted model:

[0170]

[0171] In the formula, α, β, and γ are the weight coefficients obtained in S33, and BPM_norm, Yawn_Freq_norm, and PERCLOS_norm are the normalized BPM, Yawn_Freq, and PERCLOS in S32.

[0172] In this embodiment, step S3 integrates multiple fatigue features into a comprehensive index through multimodal feature normalization and personalized weight adaptive learning. This index considers the contribution of different features and adapts to individual differences among drivers, significantly improving the accuracy and robustness of fatigue detection.

[0173] Step S4: Calculate the confidence level and dynamically adjust the time window.

[0174] In this embodiment, based on the first fatigue index (Fatigue_Index1) obtained in step S3, and combining the auxiliary features extracted in step S2 with the real-time vehicle status information, the confidence level of the index is calculated using a multi-source cross-validation model. Then, based on the dual feedback mechanism of the confidence level and the fatigue index, the monitoring time window is dynamically adjusted. Figure 7 As shown, the details are as follows:

[0175] S41. Confidence calculation based on multi-source cross-validation

[0176] The confidence score is calculated using the following three factors obtained in real time, with a value range of [0, 1].

[0177] First, calculate the three factors:

[0178] 1. Physiological consistency factor (C_physio)

[0179] This factor assesses the consistency among multiple physiological fatigue characteristics used to calculate the fatigue index. In this invention, the "physiological fatigue characteristics" specifically refer to the normalized blink frequency (BPM_norm), yawn frequency (Yawn_Freq_norm), and eyelid closure time percentage (PERCLOS_norm) used in step S3. Their dispersion is calculated as follows:

[0180]

[0181] Where BPM_norm, Yawn_Freq_norm, and PERCLOS_norm are three normalized eigenvalues, and μ is the average of the three.

[0182] 2. Vehicle behavior factor (C_vehicle)

[0183] This factor, which assesses whether the vehicle's driving condition provides supporting evidence for fatigued driving, is derived from the following mathematical formula:

[0184] 1

[0185] in, This represents the standard deviation of the vehicle's lateral displacement within the current time window. This is a set threshold. The more unstable the vehicle's operation, the lower this factor becomes, thus reducing the confidence level of the fatigue index calculated in S3.

[0186] 3. Environmental disturbance assessment factor (C_environment)

[0187] This factor assesses whether the current environment is conducive to the accurate extraction of S2 features, and is derived from the following mathematical formula:

[0188] Light_score (1 Noise_score)

[0189] Light_score is a score based on ambient light sensor data (sufficient light = 1, insufficient light = 0.3), and Noise_score is a noise score based on camera image quality assessment (sharp image = 0, blurry image / high noise = 0.7).

[0190] Subsequently, the confidence score is used to quantify the reliability of the current system's determination of the first fatigue index (Fatigue_Index1), and is derived by a weighted fusion of the above three factors:

[0191] + +

[0192] in, , , The weighting coefficients for the physiological consistency factor C_physio, the vehicle behavior factor C_vehicle, and the environmental disturbance assessment factor C_environment in calculating the total confidence score satisfy the following conditions: =1.

[0193] S42. Time window adjustment based on dual feedback mechanism

[0194] First, based on the first fatigue index Fatigue_Index1 output in step S3 and the confidence_Score calculated in step S41, the first time window T_initial determined in step S1 is adjusted, and the second time window T_second is calculated using the following nonlinear double feedback adjustment function formula:

[0195]

[0196] Where λ is the base adjustment coefficient (0 < λ < 1), controlling the overall adjustment amplitude. k is the confidence sensitivity coefficient, controlling the influence of confidence on adjustment. m is the fatigue index sensitivity coefficient, controlling the influence of fatigue index on adjustment. The following is the adjustment logic for the time window:

[0197] 1. Low confidence level & low fatigue index:

[0198] Confidence adjustment term and fatigue index adjustment term The values ​​are all relatively large, with the largest adjustment range, significantly shortening the window to obtain more data and improve the accuracy of judgment.

[0199] 2. Low confidence level & high fatigue index:

[0200] Both are at a moderate level. The window should be shortened appropriately to ensure both response speed and accuracy verification.

[0201] 3. High confidence level & low fatigue index:

[0202] Both are relatively small, with minimal adjustment ranges, allowing the window to be maintained or extended appropriately to save computing resources.

[0203] 4. High confidence level & high fatigue index:

[0204] The confidence term is small, but the fatigue index term is large. The minimum amplitude shortens the window, and after confirming high risk, priority is given to ensuring real-time response.

[0205] S43, Time Window Boundary Constraints

[0206]

[0207] in, , These are the minimum and maximum time window values ​​allowed by the system, respectively, and T_second is the second time window obtained from S42.

[0208] In this embodiment, step S4 constructs a comprehensive confidence assessment system by reusing the scenario parameters of S1 and introducing historical and real-time performance data from S2 and S3. Based on a nonlinear dual-feedback adjustment mechanism of confidence and fatigue index, precise and adaptive adjustment of the time window is achieved. This not only allows the system to inherit the predictive advantage of S1, but also forms an adaptive optimization closed loop that runs through the entire monitoring process.

[0209] Step S5: Update Fatigue Index

[0210] In this embodiment, the system uses the optimized second time window T_second as the new sampling period to calculate the updated and more accurate second fatigue index Fatigue_Index2, as follows:

[0211] S51, Image Acquisition Based on Second Time Window

[0212] Using the second time window T_second obtained after adjustment in step S4 as the new sampling period, the driver's image information is acquired through the camera:

[0213] 1. Acquisition Strategy: Inheriting the resource adaptation logic from step S1, the system will re-evaluate the current R_scenario (scene risk score calculated by S11) and L_cpu (real-time monitored system CPU load) to decide whether to use "high-priority single-frame capture" or "standard video stream sampling".

[0214] 2. Optimized sampling frequency:

[0215] T_second has been optimized compared to T_initial:

[0216] If T_second ≤ T_initial, the system is collecting data at a higher frequency to capture dynamic changes in fatigue state more quickly or to obtain more samples at low confidence levels.

[0217] If T_second > T_initial, the system is collecting data at a lower frequency, saving computing resources and suitable for stable scenarios with high confidence and low fatigue index.

[0218] S52. Fatigue Feature Extraction Based on the Second Time Window

[0219] First, following the same method as step S2, the updated fatigue characteristics are calculated within the new T_second window: blink frequency (BPM_2), yawn frequency (Yawn_Freq_2), and PERCLOS value (PERCLOS_2).

[0220] S53, Calculation of the Second Fatigue Index

[0221] Based on the updated fatigue features extracted in step S52, the second fatigue index (Fatigue_Index2) is calculated.

[0222] First, feature normalization is performed:

[0223] Normalize BPM_2, Yawn_Freq_2, and PERCLOS_2 using the same normalization parameters as in S31 to obtain BPM_norm2, Yawn_Freq_norm2, and PERCLOS_norm2.

[0224] Then, the weights are applied:

[0225] The personalized weighting coefficients α, β, γ calculated for the driver in step S3 are used.

[0226] Finally, the second fatigue index is calculated using the same multimodal weighted model:

[0227]

[0228] BPM_norm2, Yawn_Freq_norm2, and PERCLOS_norm2 are the three normalized eigenvalues ​​obtained above.

[0229] Step S6: Execute a tiered response

[0230] The core of this step is a response level arbitration mechanism. The system will jointly decide based on the latest second fatigue index (Fatigue_Index2) and confidence score (Confidence_Score), triggering a progressive response strategy library containing five levels (L1~L5). For higher-level responses, both the fatigue index and confidence score thresholds must be met simultaneously. Specifically:

[0231] S61, Confidence-based Response Level Arbitration

[0232] The final determination of the response level is the result of a joint decision made by the second fatigue index (Fatigue_Index2) and the confidence score (Confidence_Score) to ensure the safety and reliability of the high-level response.

[0233] 1. Low-level response for warnings and mild interventions: Specifically refers to L1 (neuromodulation level) and L2 (tactile alert level) responses. These responses are mainly triggered directly based on the threshold range to which Fatigue_Index2 belongs, and their purpose is to provide non-invasive reminders and mild interventions to the driver.

[0234] 2. High-response levels involving active vehicle control or emergency rescue: Specifically referring to L3 (vehicle restraint level), L4 (active avoidance level), and L5 (emergency rescue level) responses. These responses carry higher risk because they directly involve vehicle control or initiating external rescue. Therefore, in addition to Fatigue_Index2 reaching the corresponding threshold, confidence conditions must also be met. For example, triggering an L3 response requires Confidence_Score > 0.7, triggering an L4 response requires Confidence_Score > 0.8, and triggering an L5 response requires Confidence_Score > 0.9, thus ensuring that critical commands such as vehicle control and life support are executed only when the system has a high degree of confidence.

[0235] S62, Five-level progressive response instruction generation and execution

[0236] Based on the arbitration result of S61, generate and execute the corresponding response instructions:

[0237] L1 response (neural regulation level): when 0.3 ≤ Fatigue_Index2 < 0.5

[0238] Generate biofeedback stimulus commands. Alpha brainwaves of a specific frequency are played through the vehicle's audio system to neuromodulate the driver and improve their concentration. The generation of these biofeedback stimulus commands relies on the accurate extraction of eye and mouth fatigue characteristics (PERCLOS, blinking, yawning) by S2 and the calculation of a mild fatigue index by S3, representing an early, non-invasive intervention.

[0239] L2 response (haptic alert level): when 0.5 ≤ Fatigue_Index2 < 0.7

[0240] Generate active tactile feedback commands. Control the vibration motor in the steering wheel to generate pulse vibrations synchronized with the driver's physiological rhythm (which can be preliminarily estimated based on the blinking and yawning frequencies extracted from S2), providing a strong and personalized tactile warning. Utilizing the timing characteristic frequency output from S2, the tactile warning is synchronized with the driver's physiological state, enhancing the warning effect.

[0241] L3 Response (Vehicle Constraint Level): When 0.7 ≤ Fatigue_Index2 < 0.8 and Confidence_Score > 0.7

[0242] The system generates vehicle dynamic constraint commands. It controls the powertrain to limit the vehicle speed to a preset safe value (e.g., 80 km / h). Combining high-precision map data, if there is a curve ahead, it proactively calculates the safe speed based on the curvature and intervenes in advance to adaptively reduce speed. This level of execution relies heavily on the high confidence level of the S4 calculations, ensuring the reliability of vehicle control commands and avoiding dangerous interventions due to misjudgments.

[0243] L4 Response (Active Risk Avoidance): A safety island parking instruction is generated when 0.8 ≤ Fatigue_Index2 < 0.9 and Confidence_Score > 0.8. Based on high-precision maps and real-time positioning, the system automatically finds and plans a route to the nearest emergency parking area or service area. Simultaneously, through V2X communication (vehicle-to-everything wireless communication technology) and in collaboration with roadside units, it informs following vehicles of its vehicle status in advance and requests the clearing of the target area, achieving collaborative obstacle clearance. This level is a higher-level application of S3 and S4 outputs, deeply integrating biometric recognition results with vehicle-road-cloud collaborative technology, achieving a leap in safety assurance from inside the vehicle to outside.

[0244] L5 Response (Emergency Rescue Level): When Fatigue_Index2 ≥ 0.9 and Confidence_Score > 0.9

[0245] An emergency life support command is generated. The system first controls the vehicle's air conditioning to release a preset waking gas in an attempt to wake the driver. Simultaneously, an emergency rescue call is automatically triggered, sending the vehicle's precise location and driver fatigue status (based on S2 and S3 data) to the rescue center via the T-Box (vehicle telematics terminal). This serves as the final safety guarantee, initiating the highest level of medical-grade intervention based on the accurate and high-confidence fatigue assessment provided by all preceding steps (S1-S5).

[0246] S63, Link Data Recording and Closed-Loop Learning

[0247] First, the entire event chain, from the scene vector in S1 to the final response and result in S6, is formed into a data record rich in context.

[0248] The record is then stored in the historical database for:

[0249] Optimize the H factor of S1: A successful response (such as timely driver recovery) can prove that the configuration is effective and improve its P_i (performance score).

[0250] Calibrate the confidence model of S4: The driver's state after the response is the gold standard for verifying the accuracy of the confidence model.

[0251] Improve S6's response logic: Analyze the actual effects of responses at each level to fine-tune thresholds and optimize response strategies.

[0252] In this embodiment, step S6 creatively integrates biometric recognition with vehicle control, vehicle-road cooperation, and even medical rescue, constructing an unprecedented, comprehensive fatigue driving response system. It is by no means a simple "threshold-action" mapping, but an intelligent decision-making center that fully inherits and utilizes the wisdom of previous steps (Fatigue_Index2, Confidence_Score).

[0253] This invention presents a confidence-driven time-window adaptive fatigue detection and response method. Through a confidence-driven dynamic adjustment mechanism of the time window, it achieves an adaptive balance between fatigue detection accuracy and computational efficiency. By employing personalized feature threshold learning and multimodal weighted fusion, it constructs a full-link individual adaptive fatigue assessment system. It innovatively adopts a multi-source cross-validation confidence calculation model, providing reliability assurance for system decision-making. Furthermore, it establishes a five-level progressive response mechanism from neural modulation to emergency rescue, forming a comprehensive safety assurance system. Ultimately, it constructs a closed-loop system with self-optimization capabilities, significantly improving the real-time performance, accuracy, and safety of fatigue driving detection.

[0254] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A confidence-driven time-window adaptive fatigue detection and response method, characterized in that, include: The first time window is dynamically initialized based on multiple factors, including: calculating scenario risk factors, driver state prior factors, and historical performance factors respectively; wherein, the calculation method of the scenario risk factors includes introducing risk coefficients representing different time periods, road types, and weather conditions for weighted fusion; the calculation method of the driver state prior factors includes using a driving duration function that simulates the cumulative effect of fatigue and combining it with a driver personalization coefficient learned from historical data; the calculation method of the historical performance factors includes constructing a query vector, retrieving historical time window configurations and their response performance scores for similar scenarios in a historical database, and performing weighted calculation on the retrieval results; and performing a weighted geometric mean fusion of the scenario risk factors, driver state prior factors, and historical performance factors to obtain the first time window; Based on the first time window, the driver's image information is collected and the driver's fatigue characteristics are identified, thereby determining the first fatigue index; The confidence level of the first fatigue index is calculated, and based on the first fatigue index and the confidence level, the first time window is dynamically adjusted through a nonlinear double feedback adjustment function to obtain a second time window; wherein, the nonlinear double feedback adjustment function introduces a confidence adjustment term that increases as the confidence level decreases, and a fatigue index adjustment term that increases as the first fatigue index increases, to form a comprehensive adjustment factor; the method for determining the second time window includes: calculating the length of the first time window with the comprehensive adjustment factor, and using preset minimum and maximum time window values ​​to impose boundary constraints on the calculation result; Based on the second time window update, the driver's fatigue status is assessed, the driver's fatigue characteristics are obtained, and the second fatigue index is determined. Based on the second fatigue index and the confidence level, a graded response matching the fatigue state is executed.

2. The method according to claim 1, characterized in that, In the process of identifying fatigue characteristics and determining the first fatigue index, personalized thresholds were used to determine fatigue characteristics, and personalized weight coefficients were used for weighted fusion. The method for generating the personalized threshold includes: recording the driver's baseline eye aspect ratio EAR_base and baseline mouth aspect ratio MAR_base within the initial wakefulness monitoring window after vehicle startup, and multiplying them by a preset scaling factor to generate a personalized EAR threshold and a personalized MAR threshold. The method for determining the first fatigue index includes: extracting and normalizing blinking frequency, yawning frequency, and eyelid closure time percentage as fatigue features; dynamically assigning personalized weight coefficients to each normalized fatigue feature, wherein the method for assigning personalized weight coefficients includes referencing the historical statistical correlation between each fatigue feature and the actual fatigue state label; and summing the weighted fatigue features.

3. The method according to claim 1, characterized in that, The method for calculating the confidence level of the first fatigue index includes: Calculate the physiological characteristic consistency factor, vehicle behavior factor, and environmental disturbance assessment factor; The calculation method for the physiological characteristic consistency factor includes assessing the degree of dispersion among multiple normalized physiological fatigue characteristic values, including blinking frequency, yawning frequency, and eyelid closure time percentage; the assessment method for the vehicle behavior factor includes determining whether the standard deviation of the vehicle's lateral displacement within the current time window exceeds a preset threshold; and the evaluation method for the environmental interference assessment factor includes comprehensively utilizing ambient light sensor data and camera image quality assessment results. The confidence level is obtained by weighted fusion of the physiological characteristic consistency factor, vehicle behavior factor, and environmental interference assessment factor.

4. The method according to claim 1, characterized in that, A tiered response mechanism based on response levels is implemented. This mechanism includes: establishing a multi-level response strategy library encompassing levels ranging from neural regulation, tactile alert, vehicle restraint, active avoidance, to emergency assistance. For the neural regulation and tactile alert levels, triggering is based on whether the second fatigue index reaches a threshold within the corresponding range. For the vehicle restraint, active avoidance, and emergency assistance levels, the second fatigue index must reach a higher threshold, and the confidence level must also exceed the incremental confidence safety threshold set for each level before execution can be triggered.

5. The method according to claim 4, characterized in that, When the neuromodulation level response is triggered, alpha brainwaves of a specific frequency are played through the vehicle audio system to neuromodulate the driver and promote his concentration. When a tactile alert response is triggered, the vibration motor inside the steering wheel is controlled to generate pulse vibrations that are synchronized with the driver's physiological rhythm, which is calculated based on the frequency of blinking and yawning. When the vehicle constraint level response is triggered, the vehicle powertrain is controlled to limit the vehicle speed to below a preset safe value. Combined with map data, when the vehicle is about to enter a curve, the safe passing speed is actively calculated based on the road curvature and adaptive speed reduction is performed in advance. When the active avoidance level response is triggered, the vehicle automatically finds and plans a safe parking route to the nearest emergency parking area or service area based on the map and real-time centimeter-level positioning. At the same time, it coordinates with roadside units and surrounding vehicles through vehicle-to-the-world (V2X) communication to inform the vehicle of its status in advance and request the clearing of the target area.

6. The method according to claim 4, characterized in that, The emergency rescue level response is triggered when both the second fatigue index and confidence level reach the highest danger threshold. First, it controls the vehicle's air conditioning to release preset stimulating gas, and then automatically triggers an emergency rescue call through the vehicle's T-Box terminal, sending the vehicle's location and driver status information, including fatigue characteristic data, to the rescue center.

7. The method according to claim 4, characterized in that, After executing the hierarchical response, the process also includes link data recording and closed-loop learning: the complete event chain data from the initialization scenario vector to the final response result is structured and recorded and stored in the historical database. The recorded complete event chain data is used to continuously optimize and calibrate the historical performance factor, confidence calculation model, and trigger threshold of each response level in the initialization model of the first time window offline.

8. The method according to claim 1, characterized in that, The image acquisition strategy based on resource adaptation acquires the driver's image information, including: forming decision conditions based on scene risk factors determined after assessing the driving environment and the real-time monitoring of the system's central processing unit (CPU) load; If the scene risk factor or CPU load is higher than the preset high-risk threshold, a high-priority single-frame capture command is sent to the camera; if the high-risk decision conditions are not met, the standard video stream sampling mode is adopted, and short video segments are captured according to the duration defined by the first time window.

Images