Fatigue screening method based on critical flash fusion frequency and multi-modal eye features

Abstract

The invention provides a fatigue screening method based on critical flash fusion frequency and multi-modal eye features, and the method comprises the steps: processing an eye response image set, extracting the diameter of a pupil, and forming reference eye position data; and finally generating a user fatigue value representing the fatigue sensitivity of the user in combination with a CFF critical value model. In the normal driving process, the system continues to collect a current monitoring image through flickering irradiation of the light source, and monitoring eye position data are extracted from the current monitoring image. And comparing the monitored eye position data with a previously generated user fatigue value, if the monitored eye position data exceeds the user fatigue value, considering that the driver is possibly in a fatigue state, generating a fatigue signal by the system, and triggering a corresponding early warning mechanism. In combination with flicker frequency induced visual perception change and eye feature analysis, the limitation of only depending on eye movement or CFF is broken through, and the detection precision of central fatigue is improved. The method is completed through light source and image acquisition and is suitable for continuous fatigue monitoring in the actual driving process.

Description

Technical Field

[0001] This invention relates to the field of fatigue monitoring technology, and in particular to a fatigue screening method based on critical flash fusion frequency and multimodal eye characteristics. Background Technology

[0002] During long drives, drivers are highly susceptible to central fatigue due to sustained visual concentration and high cognitive load. This fatigue not only reduces reaction speed and operational accuracy but also significantly increases traffic safety risks. Currently, fatigue detection methods are mostly based on eye movement characteristics, such as the duration of eye closure, blinking frequency, or facial expression recognition. However, these methods typically suffer from problems such as strong detection lag, significant individual differences, and sensitivity to external lighting and occlusion, making it difficult to achieve highly reliable real-time fatigue monitoring.

[0003] Critical flicker fusion frequency (CFF) is an important physiological indicator for assessing central nervous system excitability and is widely used in controlled experimental settings to measure fatigue levels. CFF refers to the change in the human eye's perception of flickering light as the flicker frequency increases. When the flicker frequency exceeds a certain critical point, the individual perceives it as continuous light; this critical frequency is the CFF value. Studies have shown that the CFF value decreases significantly under central nervous system fatigue, demonstrating good sensitivity and objectivity.

[0004] However, existing CFF measurement methods typically rely on statically focusing on a flickering light source and manually adjusting the flickering frequency with a knob until the subject subjectively perceives the light source as merging into a continuous beam. This measurement method, based on active cooperation and a static environment, is clearly unsuitable for real-time fatigue assessment during dynamic driving. Although fatigue detection schemes incorporating other ocular physiological characteristics such as eye movement trajectory and pupil diameter changes have emerged in recent years, these methods still face challenges such as poor stability and high false positive rates. In contrast, CFF, as a fundamental neurophysiological indicator reflecting the brain's cognitive state, is still widely considered one of the most objective and sensitive fatigue monitoring methods.

[0005] Therefore, there is an urgent need for a method or device that can dynamically and intermittently trigger visual stimuli during natural driving behavior without affecting the execution of driving tasks, and combine multimodal ocular physiological parameters to achieve rapid, accurate and reliable central fatigue screening, so as to improve driving safety and human intelligence. Summary of the Invention

[0006] Therefore, it is necessary to propose a fatigue screening method based on critical flash fusion frequency and multimodal eye features to address the above problems.

[0007] A fatigue screening method based on critical flash fusion frequency and multimodal eye features, the fatigue screening method based on critical flash fusion frequency and multimodal eye features includes:

[0008] Acquire continuous images of the eye under different frequencies of flickering light sources inside the vehicle to obtain continuous images;

[0009] A set of eye response images at the same frequency is selected from the continuous images based on the flicker frequency;

[0010] Reference eye position data is parsed from the set of eye reaction images, and a user fatigue value is generated based on the reference eye position data;

[0011] During monitoring, an eye monitoring image of the driver based on the flickering light source is acquired, and the monitoring eye position data is parsed from the monitoring image;

[0012] The monitored eye position data is compared with the user's fatigue value. If the monitored eye position data exceeds the user's fatigue value, a fatigue signal is generated, and an early warning is issued based on the fatigue signal.

[0013] In at least one embodiment of this application, the specific steps of parsing reference eye position data from the set of eye reaction images and generating a user fatigue value based on the reference eye position data include:

[0014] Based on the diameter of the pupil in each frame of the image, the eye reaction time of each set of the eye reaction images is calculated to obtain reaction time data;

[0015] The maximum reaction time value is obtained by filtering out the maximum reaction time from the reaction time data.

[0016] The reaction time values ​​adjacent to the maximum reaction time value are selected from the reaction time data to obtain the secondary time value;

[0017] Reference eye position data is generated based on the maximum reaction time value and the secondary time value;

[0018] A preset CFF critical threshold is obtained, and the user fatigue value is calculated based on the CFF critical threshold and the reference eye position data.

[0019] In at least one embodiment of this application, during the monitoring, acquiring an eye monitoring image of the driver based on light source flicker, and parsing the monitoring eye position data from the monitoring image, specifically includes the following steps:

[0020] Pupil data of the eye is obtained by parsing the pupil data of the eye from the monitored image;

[0021] Frame images corresponding to the maximum and minimum pupil values ​​are selected from the monitored pupil data to obtain monitoring image pairs;

[0022] The time difference is calculated based on the monitored images, and monitoring eye position data is generated based on the time difference.

[0023] In at least one embodiment of this application, the fatigue screening method based on critical flash fusion frequency and multimodal ocular features further includes:

[0024] The monitored eye position data is compared with the user's fatigue value. If the monitored eye position data does not exceed the user's fatigue value, then fatigue monitoring for the next cycle will begin.

[0025] In at least one embodiment of this application, the fatigue screening method based on critical flash fusion frequency and multimodal ocular features further includes:

[0026] Analyze the continuous images to obtain the ambient light intensity in the continuous images and generate a reference light intensity value;

[0027] The user fatigue value is calibrated based on the reference light intensity value to obtain a calibrated fatigue reference value.

[0028] In at least one embodiment of this application, the step of calibrating the user fatigue value based on the reference illumination intensity value to obtain a calibrated fatigue reference value specifically includes:

[0029] Obtain the critical ambient light value corresponding to the preset CFF critical threshold, and calculate the initial calibration difference based on the critical ambient light value and the reference light intensity value.

[0030] The user fatigue value is calibrated based on the initial calibration difference to obtain a calibrated fatigue reference value.

[0031] In at least one embodiment of this application, the fatigue screening method based on critical flash fusion frequency and multimodal ocular features further includes:

[0032] The average ambient light value of the monitored images was analyzed during the monitoring process.

[0033] The monitoring eye position data is calibrated based on the average ambient light value to obtain calibrated monitoring eye position data.

[0034] The calibrated fatigue reference value is compared with the calibrated monitoring eye position data. If the calibrated monitoring eye position data is greater than the calibrated fatigue reference value, a suspected fatigue command is generated.

[0035] In at least one embodiment of this application, the fatigue screening method based on critical flash fusion frequency and multimodal ocular features further includes:

[0036] Acquire the calibrated eye position data in the next monitoring process;

[0037] The average monitoring eye position data is calculated based on the calibrated monitoring eye position data described in this monitoring process and the calibrated monitoring eye position data described in the next monitoring process to obtain the average monitoring eye position data.

[0038] The average monitored eye position data is compared with the calibrated fatigue reference value. If the average monitored eye position data is greater than the calibrated fatigue reference value, a fatigue signal is generated, and an early warning is issued based on the fatigue signal.

[0039] In at least one embodiment of this application, the fatigue screening method based on critical flash fusion frequency and multimodal ocular features further includes:

[0040] The maximum height of the palpebral fissure is extracted from the continuous images to obtain a first palpebral fissure value;

[0041] The maximum height of the palpebral fissure is extracted from the monitored image to obtain a second palpebral fissure value;

[0042] Obtain the palpebral fissure threshold, and calculate the palpebral fissure reference value based on the first palpebral fissure value and the palpebral fissure threshold;

[0043] The second palpebral fissure value is compared with the palpebral fissure reference value. If the second palpebral fissure value is less than the palpebral fissure reference value, a monitoring command is generated and the light source is controlled to flash to obtain a monitoring image of the driver based on the flashing light source for monitoring.

[0044] In at least one embodiment of this application, the fatigue screening method based on critical flash fusion frequency and multimodal ocular features further includes:

[0045] Acquire eye-tracking images before the light source flashes;

[0046] Analyze the pupil coordinates and macula coordinates in the eye-tracking image;

[0047] Adjust the shooting angle of the camera according to the pupil coordinate position;

[0048] The focal length of the camera is adjusted according to the coordinate position of the macula.

[0049] The fatigue screening method based on critical flash fusion frequency and multimodal ocular features implemented in this embodiment will have at least the following beneficial effects:

[0050] The fatigue screening method based on critical flash fusion frequency and multimodal eye features provided above firstly involves a light source set up inside the vehicle that flashes at different frequencies. The system simultaneously acquires continuous eye images of the driver at each frequency to simulate the visual stimulation process in the traditional CFF test.

[0051] The acquired images are classified according to their flicker frequency, and multiple eye image sequences at the same frequency are extracted to obtain an eye response image set, which is used to analyze the eye images corresponding to each frequency.

[0052] These sets of eye response images are processed to extract the pupil diameter, forming reference eye position data. Combined with the CFF critical value model, a user fatigue value representing the user's fatigue sensitivity is finally generated, serving as a benchmark for subsequent judgments.

[0053] During normal driving, the system continues to illuminate the eyes with flashing light while simultaneously acquiring current monitoring images and extracting eye position data from them. This eye position data represents the eye's ability to respond to flashing stimuli in the current state.

[0054] The system compares the monitored eye position data with the previously generated user fatigue value. If the monitored eye position data exceeds the user fatigue value (i.e., the condition has deteriorated), it is considered that the driver may be in a state of fatigue. The system generates a fatigue signal and triggers the corresponding warning mechanism (such as reminder, vibration, etc.).

[0055] By combining visual perception changes induced by flicker frequency with eye feature analysis, the limitations of relying solely on eye movements or CFF alone are overcome, thus improving the detection accuracy of central fatigue.

[0056] By calculating the user fatigue value for each driver and avoiding the use of a uniform threshold, it is possible to better reflect the physiological characteristics of different individuals and reduce false alarms and missed alarms.

[0057] The method is completed through light source and image acquisition, which does not affect driving behavior and does not require the driver's active cooperation, making it suitable for continuous fatigue monitoring in actual driving processes. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] in:

[0060] Figure 1 This is a flowchart of a fatigue screening method based on critical flash fusion frequency and multimodal eye features in one embodiment;

[0061] Figure 2 This is a flowchart of a fatigue screening method based on critical flash fusion frequency and multimodal eye features in another embodiment;

[0062] Figure 3 This is a flowchart of a fatigue screening method based on critical flash fusion frequency and multimodal eye features based on illumination environment in one embodiment;

[0063] Figure 4 A flowchart of a fatigue screening method based on critical flash fusion frequency and multimodal eye features based on illumination environment in another embodiment;

[0064] Figure 5 The flowchart below shows a fatigue screening method based on critical flash fusion frequency and multimodal eye features in one embodiment, based on a calibrated flowchart.

[0065] Figure 6 This is a flowchart of a fatigue screening method based on critical flash fusion frequency and multimodal ocular features based on palpebral fissure monitoring in one embodiment;

[0066] Figure 7 This is a flowchart of a fatigue screening method based on critical flash fusion frequency and multimodal eye features, using eye tracking and focus adjustment, as described in one embodiment. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] This invention provides a fatigue screening method based on critical flash fusion frequency and multimodal eye features, the fatigue screening method based on critical flash fusion frequency and multimodal eye features includes:

[0069] S101. Acquire continuous images of the eye under different frequencies of flickering light sources inside the vehicle to obtain continuous images;

[0070] S102. Select a set of eye response images at the same frequency from the continuous images based on the flicker frequency;

[0071] S103. Extract reference eye position data from the set of eye reaction images, and generate user fatigue value based on the reference eye position data;

[0072] S104. During monitoring, acquire an eye monitoring image of the driver based on the flickering light source, and parse the monitoring eye position data from the monitoring image;

[0073] S105. Compare the monitored eye position data with the user fatigue value;

[0074] S106. If the monitored eye position data exceeds the user's fatigue value, a fatigue signal is generated, and an early warning is issued based on the fatigue signal.

[0075] Please refer to Figures 1-7 In this embodiment, firstly, the light source installed inside the vehicle will flash at different frequencies, and the system will simultaneously collect continuous eye images of the driver at each frequency to simulate the visual stimulation process in the traditional CFF test.

[0076] The acquired images are classified according to their flicker frequency, and multiple eye image sequences at the same frequency are extracted to obtain an eye response image set, which is used to analyze the eye images corresponding to each frequency.

[0077] These sets of eye response images are processed to extract the pupil diameter, forming reference eye position data. Combined with the CFF critical value model, a user fatigue value representing the user's fatigue sensitivity is finally generated, serving as a benchmark for subsequent judgments.

[0078] During normal driving, the system continues to illuminate the eyes with flashing light while simultaneously acquiring current monitoring images and extracting eye position data from them. This eye position data represents the eye's ability to respond to flashing stimuli in the current state.

[0079] The system compares the monitored eye position data with the previously generated user fatigue value. If the monitored eye position data exceeds the user fatigue value (i.e., the condition has deteriorated), it is considered that the driver may be in a state of fatigue. The system generates a fatigue signal and triggers the corresponding warning mechanism (such as reminder, vibration, etc.).

[0080] By combining visual perception changes induced by flicker frequency with eye feature analysis, the limitations of relying solely on eye movements or CFF alone are overcome, thus improving the detection accuracy of central fatigue.

[0081] By calculating the user fatigue value for each driver and avoiding the use of a uniform threshold, it is possible to better reflect the physiological characteristics of different individuals and reduce false alarms and missed alarms.

[0082] The method is completed through light source and image acquisition, which does not affect driving behavior and does not require the driver's active cooperation, making it suitable for continuous fatigue monitoring in actual driving processes.

[0083] It should be noted that flashing light sources and cameras are installed in any one or more of the following locations: above the steering wheel (steering column area), the center or bottom of the instrument panel, the top of the center console screen, the rearview mirror housing (above), the side of the A-pillar, or the roof. The flashing frequency is preset to the user's CFF frequency, or the preset value can be adjusted according to the environment.

[0084] The camera is used to capture the user's eye information. The flashing light source and the camera can be in the same position for easy direct image capture. Alternatively, the camera can be placed only on the steering wheel or dashboard, with multiple flashing light sources in other areas to obtain more detection samples. Multiple camera positions can also be used, not only to cooperate with the flashing light source for shooting, but also to help predict in advance whether the user will look at the location with the flashing light source. Because the user's gaze time is very short, and the CFF (Current Eye Filtration) judgment time is also very short, the camera's acquisition is real-time, but the flashing light source does not flash continuously; it only flashes when the user looks over and turns off after operation. Therefore, the corresponding flashing light source can be controlled to flash in advance based on the pupil drift time and position information to ensure timely acquisition of the eye image during flashing. For example, there can be a first camera and a second camera, with the first camera positioned above the steering wheel and the second camera positioned on the rearview mirror housing, with a first flashing light source corresponding to the second camera, and the second flashing light source corresponding to the first camera.

[0085] In at least one embodiment of this application, the specific steps of parsing reference eye position data from the set of eye reaction images and generating a user fatigue value based on the reference eye position data include:

[0086] S1031. Based on the diameter of the pupil in each frame image, calculate the eye reaction time for each set of the eye reaction images to obtain reaction time data.

[0087] S1032. Select the maximum reaction time value from the reaction time data to obtain the maximum reaction time value;

[0088] S1033. Select the reaction time values ​​adjacent to the maximum reaction time value from the reaction time data to obtain the secondary time value;

[0089] S1034. Generate reference eye position data based on the maximum reaction time value and the secondary time value;

[0090] S1035. Obtain a preset CFF critical threshold, and calculate the user fatigue value based on the CFF critical threshold and the reference eye position data.

[0091] Please refer to Figures 1-7In this embodiment, the system first performs frame-level analysis on the acquired eye response image set. Each image set corresponds to a specific light source flicker frequency and contains image frames of the driver's eyes at different time points under stimulation at that frequency.

[0092] The system extracts pupil diameter changes from these frames and, based on the dynamic response of the pupil to light flickering stimulation (such as the onset and peak times of contraction or dilation), calculates the visual reaction time of individuals in each image set for that frequency, thus obtaining reaction time data, which is the time delay between the light stimulus and the appearance of the physiological response.

[0093] After calculating the reaction time data for all image frames, the system analyzes this reaction time data and selects the value with the longest response time, which is recorded as the maximum response time value.

[0094] The maximum reaction time value represents the driver's slowest ocular response under the current stimulus conditions, and is usually a typical indicator of high central nervous system load or fatigue.

[0095] To enhance the stability and reliability of the maximum reaction time value, the system further extracts one or more reaction time values ​​that are adjacent to the maximum value in time sequence from the reaction time data, as secondary time values.

[0096] This secondary time value can smooth out fluctuations, avoid misjudging extreme values ​​due to occasional factors, and thus enhance the robustness of subsequent reference value calculations.

[0097] After obtaining the maximum reaction time value and the secondary reaction time value, the system synthesizes the two and uses methods such as weighted average, difference evaluation or distribution analysis to generate an intermediate parameter that comprehensively reflects the driver's current reaction state and obtain reference eye position data.

[0098] Subsequently, the system calls the pre-set CFF critical threshold. The pre-set CFF critical threshold can be determined by clinical research or experimental data, representing the lower limit of the normal fusion frequency of the human eye to flicker light in a non-fatigue state.

[0099] By comparing, converting, or interpolating the preset CFF critical threshold with reference eye position data, the system ultimately generates a specific value to obtain the user's fatigue value, which will serve as the judgment standard in the subsequent fatigue screening process.

[0100] By calculating the pupil's reaction time to visual stimuli, a fatigue measurement method based on neurophysiology is constructed, which significantly improves the scientific rigor and objectivity of the identification. The reference eye position parameters calculated by combining the user's own maximum reaction time and secondary time data can be adapted to the physiological differences of different users, making the subsequently generated user fatigue values ​​more targeted, thereby avoiding misjudgments or omissions caused by uniform standards.

[0101] By using secondary time values, fluctuations in the final reference value caused by individual extreme response times are effectively avoided, thereby improving the system's anti-interference and stability while maintaining sensitivity.

[0102] In at least one embodiment of this application, during the monitoring, acquiring an eye monitoring image of the driver based on light source flicker, and parsing the monitoring eye position data from the monitoring image, specifically includes the following steps:

[0103] S1041. Parse the pupil data of the eye from the monitoring image to obtain the monitoring pupil data;

[0104] S1042. Select the frame images corresponding to the maximum and minimum pupil values ​​from the monitored pupil data to obtain monitoring image pairs;

[0105] S1043. Calculate the time difference based on the monitored image pair, and generate monitoring eye position data based on the time difference.

[0106] Please refer to Figures 1-7 In this embodiment, the system continuously captures images of the driver's eye area using an onboard camera during monitoring. The monitoring images are acquired under conditions of flickering light to stimulate a light response (contraction or dilation) in the pupil.

[0107] The system uses image processing algorithms (such as edge detection, deep learning recognition, or grayscale fitting) to process each frame of the image and extract the diameter data of the pupil in each frame of the image to form the pupil monitoring data.

[0108] After obtaining the complete pupil monitoring data sequence, the system will scan and compare the sequence, and select the image frame corresponding to the maximum pupil diameter, which usually corresponds to the stage when the light stimulus fades or the pupil recovers and dilates. The image frame corresponding to the minimum pupil diameter usually corresponds to the rapid contraction when the light stimulus just applies. These two images constitute the monitoring image pair.

[0109] The system calculates the time interval between frames in the image sequence based on the timestamp (or frame number) information of the monitored images, obtaining the time difference. This time difference is the reaction time required for the pupil to change from minimum to maximum or from maximum constriction to minimum dilation.

[0110] The system defines this time difference as monitoring eye position data, which is a quantitative indicator characterizing the driver's neural response efficiency within the current cycle.

[0111] By directly extracting the physiological response time of the pupil as an indicator, it reflects the speed at which the brain processes light stimulation, providing stronger physiological basis and quantitative comparability.

[0112] This method does not rely on the driver's subjective feedback, nor on facial muscle recognition under lighting conditions (such as expression analysis). It quantifies objective, continuous and easily identifiable physiological changes in pupil diameter, making the judgment results more accurate.

[0113] The core data for fatigue judgment is extracted by extracting two extreme frames and calculating the time difference. The overall algorithm structure is lightweight and can run on low-power computing chips, making it suitable for deployment in vehicle systems or mobile terminals for real-time operation.

[0114] In at least one embodiment of this application, the fatigue screening method based on critical flash fusion frequency and multimodal ocular features further includes:

[0115] Compare the monitored eye position data with the user fatigue value;

[0116] S107. If the monitored eye position data does not exceed the user's fatigue value, then the fatigue monitoring for the next cycle will be performed.

[0117] Please refer to Figures 1-7 In this embodiment, the system calls the user's fatigue value. If the monitored eye position data is less than or equal to the user's fatigue value, it means that the driver's current reaction is still within the normal range and there are no obvious signs of central fatigue.

[0118] If the monitored eye position data exceeds this value, indicating obvious signs of central fatigue, a fatigue alarm procedure will be initiated.

[0119] If the monitored eye position data does not exceed the user's fatigue threshold within the current monitoring cycle, the system will automatically enter the next monitoring cycle and restart the process of light source stimulation, image acquisition, and response time calculation. This behavior can be set to be timed (e.g., every 3 minutes) or event-triggered (e.g., when driving duration reaches a certain threshold).

[0120] This forms a cyclical, non-invasive, and continuous fatigue detection closed-loop system, ensuring continuous assessment of the driver's condition throughout the entire driving process.

[0121] A continuous monitoring model is constructed through a monitoring-comparison-loop mechanism. The system can continuously update the driver's fatigue status identification results during driving tasks, respond promptly to sudden changes in conditions, and ensure safe driving.

[0122] By comparing each monitoring result with the user's fatigue value and triggering a response only when a threshold is exceeded, the system effectively avoids false alarms caused by slight fluctuations. The system is tolerant of normal physiological fluctuations, improving the user experience in actual use.

[0123] The system only performs anomaly handling and early warning response when the monitoring results are abnormal; in most other cases, the system performs only one simple comparison before entering the next round of monitoring, which significantly reduces system power consumption and computing resource consumption, and is conducive to the low power consumption of the vehicle hardware platform.

[0124] In at least one embodiment of this application, the fatigue screening method based on critical flash fusion frequency and multimodal ocular features further includes:

[0125] S201. Analyze the continuous image, obtain the ambient light intensity in the continuous image, and generate a reference light intensity value;

[0126] The user fatigue value is calibrated based on the reference light intensity value to obtain a calibrated fatigue reference value.

[0127] Please refer to Figures 1-7 In this embodiment, during the initial stage of fatigue monitoring, the system uses a flashing light source to flash and acquire multiple sets of continuous eye image sequences. These images are used to analyze pupil response behavior and also contain objective information about background illumination.

[0128] Calculate the average brightness value of non-eye areas in continuous images, extract background grayscale distribution features, or combine the ambient brightness data read by the photosensitive element to calculate the average ambient light intensity at that time and obtain the reference light intensity.

[0129] The calculation is performed using user fatigue data and reference light intensity values. In this embodiment, a fixed graded calibration table is set (e.g., brightness change ±30 lux, correction factor ±Δt).

[0130] The system takes the current reference light intensity value as an input parameter, calculates the amount of correction that should be made to the user's fatigue value (such as weighting factor, deviation value, etc.), and adds it to the original user fatigue value to obtain the calibrated fatigue reference value.

[0131] For example:

[0132] If the current illumination is much higher than the reference illumination, the system will predict that the pupil constriction time will be faster, and therefore the fatigue judgment threshold will be reduced accordingly.

[0133] If the light intensity is much lower than the reference value, the pupil response is expected to be slower, so the fatigue judgment threshold is increased to avoid misjudgment.

[0134] Before each fatigue monitoring session, the system calls up the calibrated fatigue reference value and compares it with the monitoring eye position data for the current period to ensure that the judgment criteria are always based on the real and reasonable threshold under the current lighting conditions.

[0135] By extracting ambient light intensity and dynamically correcting the user fatigue assessment, false alarms and missed alarms caused by changes in light intensity are effectively avoided.

[0136] This calibration mechanism, in conjunction with real-time monitoring, enables the judgment logic to have both forward adaptation and backward feedback capabilities, thereby improving the robustness and intelligence of the overall closed-loop fatigue identification system.

[0137] In at least one embodiment of this application, the step of calibrating the user fatigue value based on the reference illumination intensity value to obtain a calibrated fatigue reference value specifically includes:

[0138] S202. Obtain the critical ambient light value corresponding to the preset CFF critical threshold, and calculate the initial calibration difference based on the critical ambient light value and the reference light intensity value.

[0139] S203. The user fatigue value is calibrated based on the initial calibration difference to obtain a calibrated fatigue reference value.

[0140] In another embodiment, the system sets a CFF critical threshold, which is derived from group trials or clinical studies and represents the critical flicker fusion frequency when a person is at the fatigue-wake boundary under standard lighting conditions. Corresponding to the CFF critical threshold, there is also a preset critical ambient light value, that is, the standard ambient brightness value when the human eye is in the CFF response state, for example, set to 300 lux.

[0141] The system obtains the reference illumination intensity value under the current image acquisition conditions (e.g., the current actual value is 120 lux) through image analysis or sensors. Then, it compares the current reference illumination intensity value with the above-mentioned critical ambient illumination value to obtain the difference between the two, thus obtaining the initial calibration difference value, which reflects the degree of deviation between the current ambient light conditions and the model training standard.

[0142] After obtaining the initial calibration difference, the system corrects the user's fatigue value based on the initial calibration difference. If the current light intensity is much lower than the critical ambient light value, it is inferred that the current pupil dilation is slow and the reaction time is long. The system appropriately increases the fatigue threshold to prevent the light reaction from being misjudged as fatigue. If the current light intensity is much higher than the critical ambient light value, the reaction is faster. The system appropriately lowers the threshold to avoid missing fatigue.

[0143] We have obtained a more accurate dynamic assessment standard that reflects a driver's fatigue response under current lighting conditions—the calibrated fatigue reference value.

[0144] Under different lighting conditions, the human eye's ability to recognize flickering light sources varies significantly, thus affecting pupillary reaction time and CFF (Critical Fluctuation Factor) values. This embodiment establishes a correlation between critical CFF and light intensity, constructing a judgment standard with physical basis and physiological interpretability.

[0145] Since user fatigue values ​​are static values ​​generated from initial data collection, they are prone to inaccuracy in environments with frequently changing light and dark conditions if not corrected. By introducing an initial calibration difference, the threshold can be automatically corrected based on the current lighting conditions, enabling dynamic and scenario-based fatigue assessment.

[0146] This calibration mechanism introduces an environmental correction dimension on the basis of individualization, expanding fatigue judgment from a single-dimensional static threshold to a multi-factor judgment model of individual and environment, which is more in line with real driving scenarios and human characteristics.

[0147] In at least one embodiment of this application, the fatigue screening method based on critical flash fusion frequency and multimodal ocular features further includes:

[0148] S301, Average ambient light value of the monitored image during analysis;

[0149] S302. The monitoring eye position data is calibrated based on the average ambient light value to obtain calibrated monitoring eye position data.

[0150] S303. Compare the calibrated fatigue reference value with the calibrated monitoring eye position data. If the calibrated monitoring eye position data is greater than the calibrated fatigue reference value, generate a suspected fatigue command.

[0151] Please refer to Figures 1-7 In this embodiment, when performing real-time fatigue monitoring, the system collects a sequence of eye images of the driver to extract monitoring eye position data (such as pupil dynamic reaction time, maximum-minimum change, etc.).

[0152] At the same time, the overall brightness characteristics of the image are analyzed to infer the ambient light level at that time.

[0153] The methods for performing brightness analysis include, but are not limited to, the following:

[0154] The average brightness (average grayscale value) of the entire image set or a specific frame is calculated, the eye ROI region is removed, and the average brightness of the background area is extracted or the brightness is obtained using the ambient light sensor (ALS) integrated in the camera.

[0155] After obtaining the average ambient light value, the system needs to determine whether the current light deviates significantly from the standard (e.g., it is different from the critical light value during basic modeling).

[0156] If they are different, the system will monitor the average illumination value corresponding to the image, input the preset illumination-pupil response mapping model, calculate a calibration factor, and apply it to the current monitored eye position data.

[0157] Calibration can be achieved in the following ways:

[0158] Construct a fitting curve between illumination value and pupil reaction time, and set brightness segments (e.g., <100 lux, 100–300 lux, >300 lux) corresponding to different adjustment coefficients, or establish a function model (e.g., A=B–α·ΔL), where A is the calibrated monitoring eye position data, B is the original monitoring eye position data, ΔL is the illumination difference, α is the illumination compensation coefficient, and lux is the illumination intensity.

[0159] By calibrating the monitoring data in real time, we can ensure that the current data reflects changes in response caused by fatigue, rather than pupillary response deviations caused by changes in light intensity, thereby obtaining a more accurate judgment.

[0160] The method can automatically adapt to the current brightness conditions based on the image, making it more widely applicable.

[0161] In at least one embodiment of this application, the fatigue screening method based on critical flash fusion frequency and multimodal ocular features further includes:

[0162] S401. Obtain the calibrated monitoring eye position data in the next monitoring process;

[0163] S402. Calculate the average monitoring eye position data based on the calibrated monitoring eye position data in this monitoring process and the calibrated monitoring eye position data in the next monitoring process to obtain the average monitoring eye position data;

[0164] S403. The average monitored eye position data is compared with the calibrated fatigue reference value. If the average monitored eye position data is greater than the calibrated fatigue reference value, a fatigue signal is generated, and an early warning is issued based on the fatigue signal.

[0165] Please refer to Figures 1-7 In this embodiment, fatigue monitoring is performed periodically during continuous system operation. For example, eye images are collected and fatigue indicators are extracted every 3 minutes or every 10 kilometers.

[0166] After completing monitoring and obtaining calibrated eye position data for a certain period, the system does not immediately make a final judgment, but waits for the data collection of the next period to be completed. After light calibration, the calibrated eye position data for the next monitoring period is obtained.

[0167] The system processes the calibrated eye position data acquired in the two cycles, performs an arithmetic mean calculation, and generates average eye position data.

[0168] The system compares the average monitored eye position data with the calibrated fatigue reference values ​​after environmental correction.

[0169] If the monitored eye position data is greater than the calibrated fatigue reference value, it means that the driver's reaction speed in two consecutive cycles is significantly lower than the calibrated fatigue reference value. Based on this, the system judges that the driver has entered a state of central fatigue and triggers a fatigue signal. The fatigue signal can be used for subsequent processing by the system, such as audible and visual warnings, lane keeping assist intervention, etc.

[0170] Based on data from two consecutive time periods, the system effectively filters out the impact of short-term fluctuations through an averaging mechanism, thereby significantly improving the system's ability to identify trend-based fatigue states and ensuring the accuracy of the judgment results.

[0171] To prevent false alarms caused by brief visual disturbances or abnormalities, such as when the driver occasionally looks down, looks away, or changes in external lighting may cause abnormal pupil response within a cycle, this method will effectively tolerate one-time deviations and only issue fatigue signals when the trend continues to deviate, significantly reducing the system's false alarm rate and improving its practicality.

[0172] In at least one embodiment of this application, the fatigue screening method based on critical flash fusion frequency and multimodal ocular features further includes:

[0173] S501. The maximum height of the palpebral fissure is extracted from the continuous images to obtain a first palpebral fissure value;

[0174] S502. The maximum height of the palpebral fissure is extracted from the monitored image to obtain a second palpebral fissure value;

[0175] S503. Obtain the palpebral fissure threshold, and calculate the palpebral fissure reference value based on the first palpebral fissure value and the palpebral fissure threshold.

[0176] S504. The second palpebral fissure value is compared with the palpebral fissure reference value. If the second palpebral fissure value is less than the palpebral fissure reference value, a monitoring command is generated and the light source is controlled to flash to obtain a monitoring image of the driver based on the flashing light source for monitoring.

[0177] Please refer to Figures 1-7In this embodiment, before system monitoring, the system controls the light source frequency and acquires a continuous sequence of eye images of the driver. The system analyzes the palpebral fissure height in each frame of the image, that is, the vertical distance between the upper and lower eyelids, to obtain the first palpebral fissure value.

[0178] The system extracts the maximum palpebral fissure height value, which is the typical eye opening amplitude when the driver's eyes are fully open in a normal, conscious state.

[0179] During actual driving, the system will periodically or through event-triggered methods to collect eye images at the current moment.

[0180] The system also extracts the maximum palpebral fissure height from it to obtain the second palpebral fissure value, which represents the driver's actual eye opening degree during the current time period.

[0181] The system obtains a manually preset palpebral fissure threshold and calculates a palpebral fissure reference value by accumulating the palpebral fissure threshold and the first palpebral fissure value.

[0182] The system compares the second palpebral fissure value with the palpebral fissure reference value. If the second palpebral fissure value is greater than or equal to the palpebral fissure reference value, it indicates that the driver's eye-opening state is normal and the detection is not triggered. If the second palpebral fissure value is less than the palpebral fissure reference value, it indicates that the driver's eyes are currently obviously unable to open, and there are initial signs of fatigue.

[0183] At this point, the system determines that the current state may be pre-fatigue, but does not immediately issue an alarm; instead, it proceeds to the next step of the processing.

[0184] The system generates and executes monitoring commands, controlling the light sources in the cockpit to flash at a specific frequency, inducing visual nerve responses in the driver (such as pupil changes). Simultaneously, the eye image acquisition device is activated to obtain images of the driver's dynamic pupil changes under this light stimulation condition, thereby extracting monitoring eye position data (such as pupil reaction time) for subsequent comparison and analysis by the fatigue recognition module.

[0185] Narrowing of the palpebral fissure is a significant, non-subjective early sign of fatigue. It can help identify potential fatigue trends in drivers in advance, allowing for intervention before fatigue fully develops and facilitating prediction.

[0186] This method avoids the system frequently performing resource-intensive operations such as image flickering and pupil calculation. The detection process is only initiated when a suspicious state is identified, thereby significantly saving system resources and improving the overall energy efficiency ratio. It is suitable for deployment in vehicle-mounted embedded systems or mobile wearable platforms.

[0187] In at least one embodiment of this application, the fatigue screening method based on critical flash fusion frequency and multimodal ocular features further includes:

[0188] S108. Before the light source flashes, acquire an eye-tracking image;

[0189] S109. Analyze the pupil coordinates and macula coordinates in the eye-tracking image;

[0190] S110. Adjust the shooting angle of the camera according to the pupil coordinate position;

[0191] S111. Adjust the shooting focal length of the camera according to the position of the macula coordinates.

[0192] Please refer to Figures 1-7 In this embodiment, before performing fatigue monitoring, the system acquires one or more frames of eye images under normal lighting conditions as eye-tracking images. These eye-tracking images are used to determine the driver's current head posture and eye direction, to determine the approximate position of the pupil in the image, and to analyze whether the gaze direction has shifted.

[0193] The system performs image recognition and spatial analysis on eye-tracking images, resolving the pupil coordinates and macula coordinates in the eye-tracking images. Based on the offset of the pupil coordinates in the image, the system determines whether the current field of view of the camera is deviated from the center of the eyeball. If the pupil is deviated from the center of the image, it means that the viewing angle is not aligned. The system then controls the movable camera (such as a module with a micro-gimbal or motorized bracket) to make fine-tuning of the angle, so that the camera's visual axis is aligned with the direction of the center of the eyeball as much as possible, ensuring that the eye is always in the center of the image, avoiding edge blurring, distortion, or missing information, which helps to improve the stability of pupil feature extraction.

[0194] The macula's coordinates represent changes in the user's gaze direction or eye depth (such as looking up or down). By combining the macula's coordinates with the camera's position, the actual depth of the eye from the camera can be estimated, determining whether the lens's focus needs adjustment to accommodate eye movement. The system dynamically adjusts the camera's focus using methods such as motorized zoom or liquid lens control to ensure a clear image of the pupil area, improving image sharpness and edge sharpness, resulting in more accurate pupil boundary recognition, which is especially crucial when pupil diameter changes are small.

[0195] In CFF detection, the system needs to measure minute changes in the pupil. If the image is blurry or the shooting angle is off, fatigue data recognition will fail. This embodiment ensures that the image is centered and clear by automatically adjusting the angle and focal length.

[0196] During driving, drivers may make slight head / eye movements due to road conditions or operations. Static shooting systems are prone to deviating from the target, resulting in blind spots. Dynamic tracking and real-time correction enable the system to have stronger adaptive shooting capabilities, thereby improving robustness.

[0197] The above description is merely an embodiment of this application. It should be noted that those skilled in the art can make improvements without departing from the inventive concept of this application, but these improvements all fall within the protection scope of this application.

Claims

1. A fatigue screening method based on critical flicker fusion frequency and multi-modal ocular features, characterized in that, The fatigue screening method based on critical flicker fusion frequency and multi-modal eye features comprises: Obtaining continuous eye images under different flicker frequencies of an in-vehicle light source, and obtaining continuous images; Screening eye response image sets under the same frequency from the continuous images according to the flicker frequency; Analyzing reference eye position data from the eye response image sets, and generating a user fatigue value according to the reference eye position data; During monitoring, obtaining eye monitoring images of the driver based on the flicker of the light source, and analyzing monitoring eye position data from the monitoring images; Comparing the monitoring eye position data with the user fatigue value, and if the monitoring eye position data exceeds the user fatigue value, generating a fatigue signal and performing a warning according to the fatigue signal.

2. The fatigue screening method based on critical flicker fusion frequency and multi-modal eye features as claimed in claim 1, wherein, The specific steps of analyzing reference eye position data from the eye response image sets and generating a user fatigue value according to the reference eye position data comprise: According to the diameter of the pupil in each frame of image, calculating the eye response time of each group of eye response image sets to obtain reaction time data; Screening the maximum reaction time value from the reaction time data to obtain the maximum reaction time value; Screening the reaction time value adjacent to the maximum reaction time value from the reaction time data to obtain a secondary time value; Generating reference eye position data according to the maximum reaction time value and the secondary time value; Obtaining a preset CFF critical threshold, and calculating a user fatigue value according to the CFF critical threshold and the reference eye position data.

3. The fatigue screening method based on critical flicker fusion frequency and multi-modal eye features as claimed in claim 1 wherein, The specific steps of obtaining eye monitoring images of the driver based on the flicker of the light source during monitoring and analyzing monitoring eye position data from the monitoring images comprise: Analyzing pupil data of the eye from the monitoring images to obtain monitoring pupil data; Screening frame images corresponding to the maximum pupil value and the minimum pupil value from the monitoring pupil data to obtain a monitoring image pair; Calculating a time difference value according to the monitoring image pair, and generating monitoring eye position data according to the time difference value.

4. The critical flicker fusion frequency and multi-modal eye feature based fatigue screening method as claimed in claim 1, wherein, The fatigue screening method based on critical flicker fusion frequency and multi-modal eye features further comprises: If the monitoring eye position data does not exceed the user fatigue value, performing fatigue monitoring in the next cycle.

5. The critical flicker fusion frequency and multi-modal eye feature based fatigue screening method as claimed in claim 1, wherein, The fatigue screening method based on critical flicker fusion frequency and multi-modal eye features further comprises: Analyzing the continuous images to obtain the light intensity of the environment in the continuous images, and generating a reference light intensity value; Calibrating the user fatigue value according to the reference light intensity value to obtain a calibrated fatigue reference value.

6. The fatigue screening method based on critical flicker fusion frequency and multi-modal eye features as claimed in claim 5, wherein, The step of calibrating the user fatigue value according to the reference light intensity value to obtain a calibrated fatigue reference value comprises: Obtaining a critical environmental light value corresponding to a preset CFF critical threshold, and calculating a first calibration difference value according to the critical environmental light value and the reference light intensity value; Calibrating the user fatigue value according to the first calibration difference value to obtain a calibrated fatigue reference value.

7. The fatigue screening method based on critical flicker fusion frequency and multi-modal eye features as claimed in claim 5, wherein, The fatigue screening method based on critical flicker fusion frequency and multi-modal eye features further comprises: analyzing the average ambient light value of the monitoring image during the monitoring; calibrating the monitoring eye position data according to the average ambient light value to obtain calibrated monitoring eye position data; comparing the calibrated fatigue reference value with the calibrated monitoring eye position data, and if the calibrated monitoring eye position data is greater than the calibrated fatigue reference value, generating a suspected fatigue command.

8. The fatigue screening method based on critical flicker fusion frequency and multi-modal eye features as claimed in claim 7, wherein, The fatigue screening method based on the critical flicker fusion frequency and the multi-modal eye features further comprises: obtaining the calibrated monitoring eye position data in the next monitoring process; calculating average monitoring eye position data according to the calibrated monitoring eye position data in this monitoring process and the calibrated monitoring eye position data in the next monitoring process to obtain average monitoring eye position data; comparing the average monitoring eye position data with the calibrated fatigue reference value, and if the average monitoring eye position data is greater than the calibrated fatigue reference value, generating a fatigue signal and performing a pre-warning according to the fatigue signal.

9. The critical flicker fusion frequency and multi-modal eye feature based fatigue screening method as claimed in claim 1, wherein, The fatigue screening method based on the critical flicker fusion frequency and the multi-modal eye features further comprises: analyzing the maximum height of the palpebral fissure from the continuous image to obtain a first palpebral fissure value; analyzing the maximum height of the palpebral fissure from the monitoring image to obtain a second palpebral fissure value; obtaining a palpebral fissure threshold value, and calculating a palpebral fissure reference value according to the first palpebral fissure value and the palpebral fissure threshold value; comparing the second palpebral fissure value with the palpebral fissure reference value, and if the second palpebral fissure value is less than the palpebral fissure reference value, generating a monitoring instruction and controlling the light source to flash to obtain a monitoring image of the driver based on the light source flashing for monitoring.

10. The critical flicker fusion frequency and multi-modal eye feature based fatigue screening method as claimed in claim 1, wherein, The fatigue screening method based on the critical flicker fusion frequency and the multi-modal eye features further comprises: obtaining an eye tracking image before the light source flashes; analyzing the pupil coordinate position and the macular coordinate position in the eye tracking image; adjusting the shooting angle of the beat member according to the pupil coordinate position; adjusting the shooting focal length of the beat member according to the macular coordinate position.

Images