Driver fatigue status recognition method

Abstract

The present invention discloses a method for identifying a driver's fatigue state, which relates to the technical field of driver fatigue state identification, and includes the following steps: setting an initial low-frequency frame rate, continuously capturing the driver's facial image through a camera, thereby obtaining continuous image data of the driver's face, and providing a basis for subsequent eyelid motion feature extraction and analysis; using image processing technology to extract the driver's eyelid motion features from the captured facial image, and conducting an in-depth analysis of the extracted eyelid motion features. The present invention generates a fatigue coefficient through eyelid motion feature analysis and machine learning, effectively monitors the driver's fatigue state at a low frame rate, and after identifying potential fatigue, the system automatically increases the frame rate, enhances data accuracy and detection sensitivity, and then comprehensively analyzes detailed data, classifies fatigue states, and issues warnings for persistent fatigue, notifying the driver to rest, thereby improving warning accuracy and overall traffic safety.

Description

Technical Field

[0001] The invention relates to the technical field of driver fatigue state recognition, and in particular to a driver fatigue state recognition method. Background Art

[0002] Driver fatigue state recognition based on multimodal recognition technology is a method that uses multiple sensors and data sources to detect and evaluate the driver's fatigue state. This technology combines data from multiple modalities such as vision, sound, and physiological signals to determine whether the driver is in a fatigued state through comprehensive analysis. For example, the visual modality can capture the driver's facial expressions, eyelid movements, head posture and other information through the camera; the sound modality can detect the driver's voice characteristics through the microphone, such as slow speech speed and lower volume; the physiological signal modality can obtain the driver's heart rate, brain waves, skin galvanic response and other physiological indicators through wearable devices. By integrating data from these different modalities, the driver's fatigue state can be identified more accurately, providing timely warnings and intervention measures.

[0003] The advantage of multimodal recognition technology lies in the diversity and comprehensiveness of its data sources, which can make up for the shortcomings of a single modality in fatigue detection. For example, although the visual modality can provide intuitive fatigue performance, it may fail in conditions such as poor lighting or when the driver wears sunglasses; the sound modality can supplement the shortcomings of the visual modality, especially when the driver is driving alone; and the physiological signal modality can provide a more direct physiological response to fatigue, and further improve the accuracy and reliability of detection by monitoring the driver's physiological changes for a long time. In summary, the recognition of driver fatigue status based on multimodal recognition technology can achieve comprehensive monitoring and accurate recognition of the driver's fatigue status through multi-angle and multi-level data analysis, providing important technical support for road traffic safety.

[0004] The prior art has the following deficiencies:

[0005] When capturing the driver's eyelid movement through a camera to identify fatigue, it is common practice to set a lower frame rate to reduce the system's computing burden and data management storage pressure. However, when the driver is driving fatigued, maintaining a low frame rate to collect eyelid movement may not be able to efficiently detect eyelid movement in a fatigued state, such as prolonged eye closure and frequent blinking. These features often occur instantaneously. If the camera's frame rate is too low, the system may not be able to capture these key moments in time, resulting in failure to detect the driver's fatigue state; for example, if the camera only captures a dozen frames per second, it may miss the key moments when the driver blinks quickly or closes his eyes for a long time, thereby missing the fatigue state. When the driver shows signs of fatigue, the system needs more time to collect enough eyelid movement data for analysis. This delay may cause the early warning system to fail to issue an alarm in time, and the driver may have fallen into a dangerous fatigue state without being reminded, increasing the risk of traffic accidents.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0007] The purpose of the present invention is to provide a method for identifying the driver's fatigue state. By deeply analyzing the extracted eyelid movement features and generating a fatigue coefficient using a machine learning model, the method can effectively monitor and preliminarily identify the driver's fatigue state at a low frame rate to prevent fatigue driving. After identifying potential fatigue, the system automatically increases the camera frame rate, increases data capture accuracy, optimizes resource utilization, and enhances detection sensitivity and accuracy. Subsequently, a comprehensive analysis of the detailed data is performed to divide the fatigue state into short-term and continuous types, and an early warning is issued for the continuous fatigue state to notify the driver to rest, thereby improving the accuracy of the early warning and overall traffic safety, so as to solve the problems in the above-mentioned background technology.

[0008] In order to achieve the above object, the present invention provides the following technical solution: a method for identifying a driver's fatigue state, comprising the following steps:

[0009] An initial low-frequency frame rate is set to continuously capture the driver's facial image through the camera, thereby obtaining continuous image data of the driver's face, which provides a basis for subsequent eyelid movement feature extraction and analysis;

[0010] Using image processing technology, the driver's eyelid movement features are extracted from the captured facial images, and the extracted eyelid movement features are deeply analyzed. Based on the analysis results, the driver's potential fatigue state is preliminarily identified;

[0011] After initially identifying the potential fatigue state, the camera's target frame rate is automatically adjusted. By increasing the frame rate, the frequency of capturing the driver's eyelid movement characteristics is increased, and more detailed and accurate movement data is obtained;

[0012] The detailed data obtained after increasing the frame rate is comprehensively analyzed to comprehensively evaluate the driver's fatigue state. Based on the comprehensive analysis, the driver's fatigue state is further divided into short-term and continuous types. For the continuous fatigue state detected, the system issues an early warning to notify the driver to stop and rest.

[0013] Preferably, eye positioning image processing technology is used to extract the driver's eyelid movement features from the captured facial image, and the specific steps are as follows:

[0014] First, a face detection algorithm is used to identify the face area from the captured face image. The face detection algorithm generates a bounding box containing the face area by scanning the entire image and identifying the area with facial features.

[0015] In the detected face area, the feature point detection model is used to locate the position of the eyes;

[0016] According to the located eye positions, an eye region image is cropped from the facial image.

[0017] Preferably, the extracted eyelid movement features are deeply analyzed, wherein the extracted eyelid movement features include the speed of the eyelids from fully open to fully closed and then to fully open, and the variation range between the maximum angle and the minimum angle of eye opening. Under the detection window, after deeply analyzing the speed of the driver's eyelids from fully open to fully closed and then to fully open, and the variation range between the maximum angle and the minimum angle of eye opening, an eyelid opening and closing speed index and an eye opening angle variation index are generated respectively, the eyelid opening and closing speed index and the eye opening angle variation index are input into a pre-trained machine learning model, a fatigue coefficient is generated based on the model, and the driver's potential fatigue state is preliminarily identified through the fatigue coefficient.

[0018] Preferably, in the detection window, the fatigue coefficient generated by the machine learning model is compared and analyzed with a preset fatigue coefficient reference threshold to preliminarily identify the driver's potential fatigue state. The specific steps are as follows:

[0019] If the fatigue coefficient is less than the fatigue coefficient reference threshold, the driving state of the driver in the detection window is classified as fatigue driving;

[0020] If the fatigue coefficient is greater than or equal to the fatigue coefficient reference threshold, the driving state of the driver in the detection window is classified as normal driving.

[0021] Preferably, after the potential fatigue state is initially identified, that is, when the driver's driving state is classified as fatigue driving in the detection window, the target frame rate of the camera is automatically adjusted. The specific steps are as follows:

[0022] Use Fatigue γ and fatigue factor reference threshold F thr Calculate the frame rate adjustment factor. The calculation expression is: Among them, R x represents the frame rate adjustment factor, k1 and k2 are control parameters used to control the sensitivity and variation range of the frame rate adjustment factor, F max Indicates the maximum fatigue coefficient, which is used to describe the maximum fatigue state value that the system can identify during the detection process;

[0023] Adjust the factor R according to the frame rate x Calculate the target frame rate. The expression is: R target =R initial +(R max -R initial )·R x , where R target represents the target frame rate, R initial represents the initial low-frequency frame rate, R max Indicates the maximum frame rate;

[0024] The driver's eyelid motion features continue to be captured based on the adjusted target frame rate to obtain more detailed and accurate motion data.

[0025] Preferably, after increasing the camera frame rate, the driver's eyelid movement feature data is continuously collected, and the fatigue coefficient of each time window is calculated;

[0026] Collect all calculated fatigue coefficients and establish an analysis set for subsequent comprehensive analysis;

[0027] The clustering algorithm is used to comprehensively analyze the fatigue coefficients in the analysis set to identify different fatigue state modes. The k-means clustering algorithm is used to divide the fatigue coefficient set into k clusters. The expression for the division is: In the formula, D represents the objective function, S is the set of clusters, and S i is the i-th cluster, μ i is the center of the ith cluster, representing the average fatigue state of the cluster, k is the number of clusters, v is the data point, ||v-μ i || represents the data point v and the cluster center μ i The squared Euclidean distance between

[0028] Calculate the standard deviation of the fatigue coefficient using the expression: Here, σ represents the standard deviation of the fatigue coefficient.

[0029] Preferably, the standard deviation is compared with a preset standard deviation, and if the standard deviation is greater than the preset standard deviation, S iis a transient fatigue cluster. If the standard deviation is less than or equal to the preset standard deviation, then S i It is a persistent fatigue cluster;

[0030] When there is a persistent fatigue cluster in the analysis set, it indicates that the driver is driving with persistent fatigue, and the system will issue an early warning to notify the driver to stop and rest.

[0031] Preferably, in the detection window, after in-depth analysis of the speed of the driver's eyelids from fully open to fully closed and then fully open, the specific steps of generating the eyelid opening and closing speed index are as follows:

[0032] In the detected eye area, the eyelid key points are extracted using the feature point detection algorithm, and continuously tracked by the tracking algorithm. The coordinates of the eyelid key points detected at time t are represented by P(t);

[0033] Calculate the vertical distance between the upper eyelid and the lower eyelid. The calculation expression is: Among them, P u and P l are the key point coordinates of the upper eyelid and lower eyelid, D t is the eyelid opening and closing distance at time t, P u，x and P u，y are the horizontal and vertical coordinates of the upper eyelid key point coordinates, P l，x and P l，y They are the horizontal and vertical coordinates of the lower eyelid key point coordinates;

[0034] Calculate the speed from fully open to fully closed and then to fully open. The calculation expression is: Among them, V t is the eyelid opening and closing speed at time t, Δt is the time interval between adjacent frames;

[0035] The eyelid opening and closing speed in the detection window is analyzed and the eyelid opening and closing speed index is calculated. The calculation expression is: EOOSI α is the eyelid opening and closing speed index, N is the number of frames in the detection window, is the absolute value integral of the eyelid opening and closing speed in the time interval [t, t+Δt], max t≤τ≤Δt V τ and min t≤τ≤Δt V τ They represent the maximum and minimum values ​​of the eyelid opening and closing speed in the time interval [t, t+Δt] respectively.

[0036] Preferably, in the detection window, after in-depth analysis of the change range between the maximum angle and the minimum angle of the driver's eyes, the specific steps of generating the eye opening angle change index are as follows:

[0037] By calculating the vertical distance between the upper edge of the eyelid and the lower edge of the eyelid in each frame, the instantaneous opening and closing angle of the eye is obtained. The calculation expression is: In the formula, A p represents the instantaneous opening and closing angle in the p-th frame image;

[0038] In the detection window, the instantaneous opening and closing angles of the eyes are obtained, and a data set is established for the instantaneous opening and closing angles. The instantaneous opening and closing angles in the data set are sorted in order, and the maximum and minimum values ​​of the instantaneous opening and closing angles are obtained. Then, the change range of the opening and closing angles is calculated. The calculation expression is: ΔA = |A max -A min |, where ΔA represents the change in the instantaneous opening and closing angle of the eyes within the detection window, A max Indicates the maximum instantaneous opening and closing angle within the detection window, A min Indicates the minimum instantaneous opening and closing angle within the detection window;

[0039] Calculate the eye opening angle change index, the calculation expression is: Where, EOAVI β Indicates the eye opening angle change index.

[0040] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0041] The present invention conducts in-depth analysis of the extracted eyelid movement features and generates a fatigue coefficient using a pre-trained machine learning model, thereby being able to preliminarily identify the driver's potential fatigue state. This process ensures that fatigue signs can still be effectively monitored at low frame rates, and the driver's fatigue state can be discovered in time, which helps prevent fatigue driving at an early stage and thus reduce the risk of traffic accidents.

[0042] After the present invention preliminarily identifies the potential fatigue state, the system automatically adjusts the target frame rate of the camera. By increasing the frame rate, the frequency of capturing the driver's eyelid movement characteristics is improved. This dynamic adjustment mechanism not only reduces the computing burden and storage pressure of the system under normal conditions, but also ensures that more detailed and accurate data can be obtained when potential fatigue is detected. It not only optimizes resource utilization, but also enhances the sensitivity and accuracy of fatigue state detection, thereby improving the overall efficiency and reliability of the system.

[0043] After improving the frame rate, the present invention uses a system to conduct a comprehensive analysis of the acquired detailed data, comprehensively evaluate the driver's fatigue state, and on this basis further divide the fatigue state into short-term and continuous types. For the detected continuous fatigue state, the system issues an early warning prompt to notify the driver to stop and rest. This process refines the fatigue state classification to ensure that early warnings are issued only when necessary, avoiding false alarms and missed alarms, thereby improving the accuracy and effectiveness of early warnings, which not only protects the driver's safety, but also improves the overall traffic safety level. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0045] Figure 1 The present invention is a flowchart of the method for identifying driver fatigue status. DETAILED DESCRIPTION

[0046] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that the description of the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art.

[0047] The present invention provides Figure 1 The method for identifying the driver's fatigue state shown includes the following steps:

[0048] An initial low-frequency frame rate is set to continuously capture the driver's facial image through the camera, thereby obtaining continuous image data of the driver's face, which provides a basis for subsequent eyelid movement feature extraction and analysis;

[0049] First, a lower frame rate is set (i.e., a smaller number of image frames are captured per second) to reduce computing and storage pressure, but the driver's facial images can still be continuously captured. Through these continuous facial images, the system can provide the necessary data basis for subsequent eyelid movement feature extraction and analysis. The initial low-frequency frame rate setting standard is usually around 10 to 15 frames per second. Such a frame rate can not only reduce the system burden, but also identify the driver's potential fatigue driving state to a certain extent, such as by detecting longer periods of eye closure or more frequent blinking and other features to preliminarily judge signs of fatigue.

[0050] Using image processing technology, the driver's eyelid movement features are extracted from the captured facial images, and the extracted eyelid movement features are deeply analyzed. Based on the analysis results, the driver's potential fatigue state is preliminarily identified;

[0051] Using eye location image processing technology, the driver's eyelid movement features are extracted from the captured facial image. The specific steps are as follows:

[0052] First, a face detection algorithm (such as Haar cascade classifier or MTCNN) is used to identify the face area from the captured face image. The face detection algorithm generates a bounding box containing the face area by scanning the entire image and identifying the area with facial features.

[0053] This step ensures that the system can accurately locate the driver's facial area, providing a basis for subsequent eye positioning.

[0054] In the detected face area, use a feature point detection model (such as Dlib's 68-point feature detection model) to locate the position of the eyes;

[0055] These models can accurately mark the key points on the face, including the eyes, nose, mouth, etc. By identifying the key point locations of the eyes, the system can accurately determine the area of ​​the eyes.

[0056] According to the located eye positions, an eye region image is cropped from the facial image;

[0057] This step focuses on processing data related to eyelid movement, reducing the amount of calculation and improving processing efficiency. The cropped eye area image will be used for subsequent detailed feature extraction to ensure that the processed data is concentrated on the part of the eye that needs to be analyzed.

[0058] The extracted eyelid movement features are analyzed in depth, including the speed of the eyelids from fully open to fully closed and then fully open, and the change range between the maximum and minimum angles of eye opening. In a fatigued state, the speed of eyelid opening and closing tends to slow down, and the eyelid movement speed of the driver when fatigued will be significantly reduced. At the same time, in a fatigued state, the driver's eye opening angle will gradually decrease, and the angle change range will decrease. Under the detection window, after an in-depth analysis of the speed of the driver's eyelids from fully open to fully closed and then fully open, and the change range between the maximum and minimum angles of eye opening, the eyelid opening and closing speed index and the eye opening angle change index are generated respectively, and the eyelid opening and closing speed index and the eye opening angle change index are input into a pre-trained machine learning model, and a fatigue coefficient is generated based on the model, and the driver's potential fatigue state is preliminarily identified through the fatigue coefficient.

[0059] A slower eyelid movement from fully open to fully closed and then fully open indicates that the driver may be potentially fatigued. When driving fatigued, a person's reaction speed and muscle control ability will decrease significantly, and the slower eyelid movement speed is one of the obvious manifestations. When the driver is fatigued, the brain's activity becomes sluggish, causing the eyelids to open and close more slowly. This process is usually accompanied by prolonged eye closure and more frequent blinking, reflecting the driver's decreased attention and slow reaction. The slower eyelid movement speed not only affects the clarity of vision and driving concentration, but also increases the risk of traffic accidents. Therefore, by monitoring the changes in the eyelid opening and closing speed, the driver's fatigue state can be identified and warned in a timely manner.

[0060] Under the detection window, after in-depth analysis of the speed at which the driver's eyelids change from fully open to fully closed and then fully open, the specific steps for generating the eyelid opening and closing speed index are as follows:

[0061] In the detected eye area, the eyelid key points are extracted using a feature point detection algorithm (such as Dlib's 68-point feature detection model), and continuously tracked using a tracking algorithm (such as the KLT tracking algorithm). The coordinates of the eyelid key points detected at time t are represented by P(t);

[0062] Calculate the vertical distance between the upper eyelid and the lower eyelid. The calculation expression is: Among them, P u and P l are the key point coordinates of the upper eyelid and lower eyelid, D t is the eyelid opening and closing distance at time t, P u，x and P u，y are the horizontal and vertical coordinates of the upper eyelid key point coordinates, P l，x and P l，y They are the horizontal and vertical coordinates of the lower eyelid key point coordinates;

[0063] Calculate the speed from fully open to fully closed and then to fully open. The calculation expression is: Among them, V t is the eyelid opening and closing speed at time t, Δt is the time interval between adjacent frames;

[0064] The eyelid opening and closing speed in the detection window is analyzed and the eyelid opening and closing speed index is calculated. The calculation expression is: EOOSI α is the eyelid opening and closing speed index, N is the number of frames in the detection window, is the absolute value integral of the eyelid opening and closing speed in the time interval [t, t+Δt], max t≤τ≤Δt V τand min t≤τ≤Δt V τ They represent the maximum and minimum values ​​of the eyelid opening and closing speed in the time interval [t, t+Δt] respectively.

[0065] It can be seen from the eyelid opening and closing speed index that, under the detection window, after an in-depth analysis of the speed at which the driver's eyelids change from fully open to fully closed and then fully open, the smaller the performance value of the eyelid opening and closing speed index generated, the greater the potential risk of fatigue driving for the driver, and vice versa. This is because when the driver is fatigued, the driver's reaction speed and muscle control ability will significantly decrease, resulting in a slower eyelid opening and closing speed. On the contrary, when the driver is awake, the eyelid opening and closing speed is faster.

[0066] The decrease in the amplitude of the change between the maximum and minimum angles of the driver's eyes is indicative of a potential fatigue driving state. Normally, drivers in an awake state open and close their eyes to a greater extent, with their eyelids fully open when their eyes are open and fully closed when their eyes are closed. However, when the driver begins to feel fatigued, the control of the eyelid muscles decreases, the maximum angle of eye opening gradually decreases, and the minimum angle of eye closing will not be completely closed, resulting in a decrease in the overall amplitude of eyelid opening and closing. This change indicates that the driver's eyelid opening and closing movements become sluggish and incomplete when they are fatigued, reflecting a decrease in their attention and alertness. As fatigue increases, the driver's eyelid opening and closing amplitude will further decrease, and phenomena such as frequent partial eye closure and prolonged staring will become more obvious. These changes not only affect the driver's field of vision and reaction ability, but also increase the risk of traffic accidents. Therefore, the decrease in the amplitude of the change between the maximum and minimum angles of eye opening is an important indicator for identifying fatigue driving.

[0067] After an in-depth analysis of the change range between the maximum and minimum angles of the driver's eyes under the detection window, the specific steps for generating the eye opening angle change index are as follows:

[0068] By calculating the vertical distance between the upper edge of the eyelid (upper eyelid) and the lower edge of the eyelid (lower eyelid) in each frame of the image, the instantaneous opening and closing angle of the eye is obtained. The calculation expression is: In the formula, A p represents the instantaneous opening and closing angle in the p-th frame image;

[0069] In the detection window, the instantaneous opening and closing angles of the eyes are obtained, and a data set is established for the instantaneous opening and closing angles. The instantaneous opening and closing angles in the data set are sorted in order, and the maximum and minimum values ​​of the instantaneous opening and closing angles are obtained. Then, the change range of the opening and closing angles is calculated. The calculation expression is: ΔA = |A max -A min|, where ΔA represents the change in the instantaneous opening and closing angle of the eyes within the detection window, A max Indicates the maximum instantaneous opening and closing angle within the detection window, A min Indicates the minimum instantaneous opening and closing angle within the detection window;

[0070] Calculate the eye opening angle change index, the calculation expression is: Where, EOAVI β Indicates the eye opening angle change index.

[0071] Under the detection window, after an in-depth analysis of the change range between the maximum and minimum angles of the driver's eye opening, the smaller the performance value of the generated eye opening angle change index, the smaller the driver's eyelid movement, which means that the eyes are not fully opened or the eye opening angle is small, which is usually a sign of fatigue. Therefore, the smaller the eye opening angle change index, the greater the potential risk of fatigue driving for the driver; conversely, if the eye opening angle change index is large, it means that the eye opening and closing range is normal, the driver's attention and alertness are high, and the potential risk of fatigue driving is smaller.

[0072] The machine learning model is not specifically limited here, and can achieve the eyelid opening and closing speed index EOOSI α and eye opening angle variation index EOAVI β Perform comprehensive analysis to generate fatigue coefficient γ In order to realize the technical solution of the present invention, the present invention provides a specific implementation method;

[0073] Fatigue coefficient γ The resulting calculation formula is:

[0074]

[0075] , where r1 and r2 are the eyelid opening and closing speed index EOOSI respectively. α and eye opening angle variation index EOAVI β The preset proportional coefficient, and r1 and r2 are both greater than 0.

[0076] It can be seen from the fatigue coefficient that, under the detection window, after an in-depth analysis of the speed at which the driver's eyelids change from fully open to fully closed and then to fully open, the smaller the performance value of the eyelid opening and closing speed index generated, after an in-depth analysis of the change range between the maximum and minimum angles of the driver's eyes, the smaller the performance value of the eye opening angle change index generated, indicating that the driver has a greater potential risk of fatigue driving, and vice versa.

[0077] In the detection window, the fatigue coefficient generated by the machine learning model is compared and analyzed with the preset fatigue coefficient reference threshold to preliminarily identify the driver's potential fatigue state. The specific steps are as follows:

[0078] If the fatigue coefficient is less than the fatigue coefficient reference threshold, the driving state of the driver in the detection window is classified as fatigue driving;

[0079] If the fatigue coefficient is greater than or equal to the fatigue coefficient reference threshold, the driving state of the driver in the detection window is classified as normal driving.

[0080] After initially identifying the potential fatigue state, the camera's target frame rate is automatically adjusted. By increasing the frame rate, the frequency of capturing the driver's eyelid movement characteristics is increased, and more detailed and accurate movement data is obtained;

[0081] After the potential fatigue state is initially identified, that is, when the driver's driving state is classified as fatigue driving under the detection window, the target frame rate of the camera is automatically adjusted. The specific steps are as follows:

[0082] Use Fatigue γ and fatigue factor reference threshold F thr Calculate the frame rate adjustment factor. The calculation expression is: Among them, R x represents the frame rate adjustment factor, k1 and k2 are control parameters used to control the sensitivity and range of the frame rate adjustment factor. No specific restrictions are given here. max Indicates the maximum fatigue coefficient, which is used to describe the maximum fatigue state value that the system can identify during the detection process;

[0083] Adjust the factor R according to the frame rate x Calculate the target frame rate. The expression is: R target =R initial +(R max -R initial )·R x , where R target represents the target frame rate, R initial represents the initial low-frequency frame rate, R max Indicates the maximum frame rate, which is the highest frame rate the system can adjust to when fatigue is detected.

[0084] The driver's eyelid motion features continue to be captured based on the adjusted target frame rate to obtain more detailed and accurate motion data.

[0085] Comprehensively analyze the detailed data obtained after increasing the frame rate to comprehensively evaluate the driver's fatigue status. Based on the comprehensive analysis, the driver's fatigue status is further divided into short-term and continuous types. For the continuous fatigue status detected, the system issues an early warning to notify the driver to stop and rest.

[0086] After increasing the camera frame rate, the driver's eyelid movement feature data is continuously collected and the fatigue coefficient of each time window is calculated;

[0087] Collect all calculated fatigue coefficients and establish an analysis set for subsequent comprehensive analysis;

[0088] The clustering algorithm is used to comprehensively analyze the fatigue coefficients in the analysis set to identify different fatigue state modes. The k-means clustering algorithm is used to divide the fatigue coefficient set into k clusters. The expression for the division is: In the formula, D represents the objective function, S is the set of clusters, and S i is the i-th cluster, μ i is the center of the ith cluster, representing the average fatigue state of the cluster, k is the number of clusters, v is the data point (here is the fatigue coefficient), ||v-μ i || represents the data point v and the cluster center μ i The squared Euclidean distance between

[0089] Calculate the standard deviation of the fatigue coefficient using the expression: Where σ represents the standard deviation of the fatigue coefficient;

[0090] Transient fatigue: The fatigue coefficient is clustered in a certain period of time, but does not form a continuous high fatigue state. The standard deviation of the transient fatigue cluster is large, which means that the fatigue coefficient fluctuates greatly.

[0091] Persistent fatigue: The fatigue coefficient remains high over multiple time windows. The standard deviation of the persistent fatigue cluster is small, which means that the fatigue coefficient is stable and high.

[0092] Compare the standard deviation with the pre-set standard deviation. If the standard deviation is greater than the pre-set standard deviation, then S i is a transient fatigue cluster. If the standard deviation is less than or equal to the preset standard deviation, then S i It is a persistent fatigue cluster;

[0093] When there is a persistent fatigue cluster in the analysis set, it indicates that the driver is driving with persistent fatigue, and the system will issue an early warning to notify the driver to stop and rest.

[0094] The present invention conducts in-depth analysis of the extracted eyelid movement features and generates a fatigue coefficient using a pre-trained machine learning model, thereby being able to preliminarily identify the driver's potential fatigue state. This process ensures that fatigue signs can still be effectively monitored at low frame rates, and the driver's fatigue state can be discovered in time, which helps prevent fatigue driving at an early stage and thus reduce the risk of traffic accidents.

[0095] After the present invention preliminarily identifies the potential fatigue state, the system automatically adjusts the target frame rate of the camera. By increasing the frame rate, the frequency of capturing the driver's eyelid movement characteristics is improved. This dynamic adjustment mechanism not only reduces the computing burden and storage pressure of the system under normal conditions, but also ensures that more detailed and accurate data can be obtained when potential fatigue is detected. It not only optimizes resource utilization, but also enhances the sensitivity and accuracy of fatigue state detection, thereby improving the overall efficiency and reliability of the system.

[0096] After improving the frame rate, the present invention uses a system to conduct a comprehensive analysis of the acquired detailed data, comprehensively evaluate the driver's fatigue state, and on this basis further divide the fatigue state into short-term and continuous types. For the detected continuous fatigue state, the system issues an early warning prompt to notify the driver to stop and rest. This process refines the fatigue state classification to ensure that early warnings are issued only when necessary, avoiding false alarms and missed alarms, thereby improving the accuracy and effectiveness of early warnings, which not only protects the driver's safety, but also improves the overall traffic safety level.

[0097] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0098] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center containing one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0099] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0100] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0101] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for identifying a driver's fatigue state, characterized in that: The following steps are involved: An initial low-frequency frame rate is set to continuously capture the driver's facial image through the camera, thereby obtaining continuous image data of the driver's face, which provides a basis for subsequent eyelid movement feature extraction and analysis; Using image processing technology, the driver's eyelid movement features are extracted from the captured facial images, and the extracted eyelid movement features are deeply analyzed. Based on the analysis results, the driver's potential fatigue state is preliminarily identified; After initially identifying the potential fatigue state, the camera's target frame rate is automatically adjusted. By increasing the frame rate, the frequency of capturing the driver's eyelid movement characteristics is increased, and more detailed and accurate movement data is obtained; Comprehensively analyze the detailed data obtained after increasing the frame rate to comprehensively evaluate the driver's fatigue status. Based on the comprehensive analysis, the driver's fatigue status is further divided into short-term and continuous types. For the continuous fatigue status detected, the system issues an early warning to notify the driver to stop and rest. The extracted eyelid movement features are deeply analyzed, wherein the extracted eyelid movement features include the speed of the eyelid from fully open to fully closed and then fully open and the change range between the maximum angle and the minimum angle of the eye opening. Under the detection window, after deeply analyzing the speed of the driver's eyelid from fully open to fully closed and then fully open and the change range between the maximum angle and the minimum angle of the eye opening, an eyelid opening and closing speed index and an eye opening angle change index are generated respectively, and the eyelid opening and closing speed index and the eye opening angle change index are input into a pre-trained machine learning model, a fatigue coefficient is generated based on the model, and the driver's potential fatigue state is preliminarily identified through the fatigue coefficient; In the detection window, the fatigue coefficient generated by the machine learning model is compared and analyzed with the preset fatigue coefficient reference threshold to preliminarily identify the driver's potential fatigue state. The specific steps are as follows: If the fatigue coefficient is less than the fatigue coefficient reference threshold, the driving state of the driver in the detection window is classified as fatigue driving; If the fatigue coefficient is greater than or equal to the fatigue coefficient reference threshold, the driving state of the driver in the detection window is classified as normal driving.

2. The method for identifying driver fatigue status according to claim 1, characterized in that: Using eye location image processing technology, the driver's eyelid movement features are extracted from the captured facial image. The specific steps are as follows: First, a face detection algorithm is used to identify the face area from the captured face image. The face detection algorithm generates a bounding box containing the face area by scanning the entire image and identifying the area with facial features. In the detected face area, the feature point detection model is used to locate the position of the eyes; According to the located eye positions, an eye region image is cropped from the facial image.

3. The method for identifying driver fatigue status according to claim 1, characterized in that: After the potential fatigue state is initially identified, that is, when the driver's driving state is classified as fatigue driving under the detection window, the target frame rate of the camera is automatically adjusted. The specific steps are as follows: Use Fatigue γ and fatigue factor reference threshold F thr Calculate the frame rate adjustment factor. The calculation expression is: Among them, R x represents the frame rate adjustment factor, k1 and k2 are control parameters used to control the sensitivity and variation range of the frame rate adjustment factor, F max Indicates the maximum fatigue coefficient, which is used to describe the maximum fatigue state value that the system can identify during the detection process; Adjust the factor R according to the frame rate x Calculate the target frame rate. The expression is: R target =R initial +(R max -R initial )·R x , where R target represents the target frame rate, R initial represents the initial low-frequency frame rate, R max Indicates the maximum frame rate; The driver's eyelid motion features continue to be captured based on the adjusted target frame rate to obtain more detailed and accurate motion data.

4. The method for identifying driver fatigue status according to claim 1, characterized in that: After increasing the camera frame rate, the driver's eyelid movement feature data is continuously collected and the fatigue coefficient of each time window is calculated; Collect all calculated fatigue coefficients and establish an analysis set for subsequent comprehensive analysis; The clustering algorithm is used to comprehensively analyze the fatigue coefficients in the analysis set to identify different fatigue state modes. The k-means clustering algorithm is used to divide the fatigue coefficient set into k clusters. The expression for the division is: In the formula, D represents the objective function, S is the set of clusters, and S i is the ith cluster, μ i is the center of the ith cluster, representing the average fatigue state of the cluster, k is the number of clusters, v is the data point, ||v-μ i || represents the data point v and the cluster center μ i The squared Euclidean distance between Calculate the standard deviation of the fatigue coefficient using the expression: Here, σ represents the standard deviation of the fatigue coefficient.

5. The method for identifying driver fatigue status according to claim 4, characterized in that: Compare the standard deviation with the pre-set standard deviation. If the standard deviation is greater than the pre-set standard deviation, then S i is a short-term fatigue cluster. If the standard deviation is less than or equal to the preset standard deviation, then S i It is a persistent fatigue cluster; When there is a persistent fatigue cluster in the analysis set, it indicates that the driver is driving with persistent fatigue, and the system will issue an early warning to notify the driver to stop and rest.

6. The method for identifying driver fatigue status according to claim 1, characterized in that: Under the detection window, after in-depth analysis of the speed at which the driver's eyelids change from fully open to fully closed and then fully open, the specific steps for generating the eyelid opening and closing speed index are as follows: In the detected eye area, the eyelid key points are extracted using the feature point detection algorithm, and continuously tracked by the tracking algorithm. The coordinates of the eyelid key points detected at time t are represented by P(t); Calculate the vertical distance between the upper eyelid and the lower eyelid. The calculation expression is: Among them, P u and P l are the key point coordinates of the upper eyelid and lower eyelid, D t is the eyelid opening and closing distance at time t, P u，x and P u，y are the horizontal and vertical coordinates of the upper eyelid key point coordinates, P l，x and P l，y They are the horizontal and vertical coordinates of the lower eyelid key point coordinates; Calculate the speed from fully open to fully closed and then to fully open. The calculation expression is: Among them, V t is the eyelid opening and closing speed at time t, Δt is the time interval between adjacent frames; The eyelid opening and closing speed in the detection window is analyzed and the eyelid opening and closing speed index is calculated. The calculation expression is: EOOSI α is the eyelid opening and closing speed index, N is the number of frames in the detection window, is the absolute value integral of the eyelid opening and closing speed in the time interval [t, t+Δt], max t≤τ≤Δt V τ and min t≤τ≤Δt V τ They represent the maximum and minimum values ​​of the eyelid opening and closing speed in the time interval [t, t+Δt] respectively.

7. The method for identifying driver fatigue status according to claim 6, characterized in that: After an in-depth analysis of the change range between the maximum and minimum angles of the driver's eyes under the detection window, the specific steps for generating the eye opening angle change index are as follows: By calculating the vertical distance between the upper edge of the eyelid and the lower edge of the eyelid in each frame, the instantaneous opening and closing angle of the eye is obtained. The calculation expression is: In the formula, A p represents the instantaneous opening and closing angle in the p-th frame image; In the detection window, the instantaneous opening and closing angles of the eyes are obtained, and a data set is established for the instantaneous opening and closing angles. The instantaneous opening and closing angles in the data set are sorted in order, and the maximum and minimum values ​​of the instantaneous opening and closing angles are obtained. Then, the change range of the opening and closing angles is calculated. The calculation expression is: ΔA = |A max -A min |, where ΔA represents the change in the instantaneous opening and closing angle of the eyes within the detection window, A max Indicates the maximum instantaneous opening and closing angle within the detection window, A min Indicates the minimum instantaneous opening and closing angle within the detection window; Calculate the eye opening angle change index, the calculation expression is: Where, EOAVI β Indicates the eye opening angle change index.

Images