A real-time drowsiness detection system

Abstract

A system equipped with dual cameras configured to capture video data simultaneously, with one facing a user and other facing towards a road, utilizes state-of-the-art machine learning and deep learning technology for face and facial landmark detection models to calculate eye aspect ratio (EAR), which is further used to derive the real-time mean open eye aspect ratio (MOE) that determines user's eye openness and attentiveness and subsequently MOE threshold values are carefully selected based on EAR calibration threshold for varying eye sizes, and alerts are given to user when real-time MOE falls below MOE threshold level enabling user to take preventive measures to enhance driver safety and optimize driving experiences.

Description

[0001] A REAL-TIME DROWSINESS DETECTION SYSTEM

[0002] Field of invention

[0003] The present invention relates to the system for drowsiness detection with machine learning and deep learning based face and facial landmark detection models to calculate Eye Aspect Ratio (EAR) to derive Mean Open Eye Aspect Ratio (MOE) which is an essential indicator for identifying drowsy events in a user.

[0004] Background of invention

[0005] Driving is an essential part of daily routine of humans. Transport industry is highly dependent on road ways for on land freight movement. Apart from industrial use there is plethora of usages in daily life form public transport to private commute. Livelihood of many depends on driving vehicles.

[0006] Road commute also involves risk of road accidents by variety of reasons; drowsy driving is one of them. Even with the increasing presence of automation, human operator continues to play important role in critical task execution. According to global studies 21% of all the fatal accidents happen due to drowsy driving. National Safety Council (NSC) reports that approx 100.000 accidents are reported every year with cause of drowsy driving, among those approx 71,000 injuries and approx 6400 fatal causalities are included. Financial loss due to this accidents accounts for close to $100 Billion according to National highway traffic safety administration (NHTSA) only considering USA. Driving the vehicle after 20 hours without sleep increases your blood-alcohol level to 0.08% which is above legal drinking limit while driving in USA. As per the current studies data is only available for certain part of the world but such cases of fatigue driving is observed in every region of the world.

[0007] Detecting the drowsiness in driver can be done through different methods. Primarily state of drowsiness is detected by monitoring three type of measurements: physiological-based (using EEG, ECG, EOG and EMG signals), Vehicle operation based (focusing of deviation from lane position, movement of the steering wheel and pressure on the acceleration pedal) and behavioral based measurements (including yawning, eye closure, head pose etc.).

[0008] The patent application number KR10206808481 discloses a method of detecting fatigue by capturing a user's face for a predetermined period of time and observing the blink time of user's eyes it makes a prediction of fatigue state based on the frequent blinking. The invention lacks in making an accurate prediction as only blinking duration cannot be the sufficient parameter for detecting drowsiness in user, other factors like behavior while eyes are open is essential to detect drowsiness beforehand.

[0009] The patent application number 202111024738 discloses a system to prevent drowsy driving utilizes an eye closure ratio measured using the blinking sensors to arrive at the conclusion. It lacks the consideration of other parameters associated with drowsiness for example face orientation, physiological parameters, user's behavior while he is awake, early signs of drowsiness by heavy eye lids leading to unconscious staring to loss of focus.

[0010] Current available systems lack the ability to predict the driver's tendency to fall asleep as they are not personalized for different users. Already available system uses the data involving the closed-eye and related behaviors, which is not sufficient to make prediction. Available solutions use face detection model to calculate EAR of closed eyes or blinking duration of eyes to arrive on the conclusion of drowsy driving. While starting the drive, drivers will not be feeling drowsiness hence will be alert but will gradually start feeling sleepy and drowsy that may lead falling asleep on steering wheel while driving and attention span of such period plays a crucial role so closed eye behavior of measuring the EAR while person is about to sleep might not be sufficient to arrive on reliable conclusion.

[0011] Hence, the present invention uses Eye Aspect Ratio (EAR) while user's eyes are open to provide a reliable system for drowsiness detection in user.

[0012] Object of Invention

[0013] The main object of "A REAL-TIME DROWSINESS DETECTION SYSTEM" is to identify signs of drowsiness in user with machine learning and deep learning modules to detect users face and facial landmarks in real time. Furthermore, the device can also employ machine learning to recognize specific patterns in the driver's behavior and adopt its warnings based on individual driving style and preferences, making the system more personalized and affective.

[0014] Yet another object of the present invention is to monitor the driver's state by tracking the eye movements and head orientation to gauge the driver's attention to the road and driving conditions.

[0015] Another, object of the present invention is to identify landmark using the road side camera such as vehicle, road signs, and pedestrian crossing, enabling system to provide timely navigation assistance and safety alerts.

[0016] Another, object of the present invention is to continuously adapt and improve its drowsiness detection algorithms by learning from the continuously evolving data base of drowsy driving scenarios and driver responses, the system can refine its accuracy and sensitivity over time by utilizing its machine learning and deep learning capabilities.

[0017] These and other objects will be apparent based on the disclosure herein.

[0018] Summary of invention

[0019] The present invention pertains to a system and method for drowsiness detection in user that employs machine learning and deep learning models to detect a user's face with facial landmarks and calculate the mean open eye aspect ratio (MOE) from values of Eye Aspect Ratio (EAR). In view of the foregoing, an embodiment herein provides a system equipped with dual cameras configured to capture video data simultaneously, with one facing a user and other facing towards a road, with a face and facial landmarks detection model combined with MOE calculation module; and an alarm device, wherein the road facing camera captures the input and system process the visuals for object detection and keeps user informed of the upcoming obstacles, and the user facing camera captures the input and system process the visuals of user while driving the vehicle using a face and facial landmark detection model comprising of machine learning and deep learning algorithms used for calculating eye aspect ratio (EAR); essentially specifying upon deriving a Mean Open Eye Aspect Ratio (MOE) using MOE module that derives the personalized MOE depending on the user behavior and eye size; and an alarm device to provide alerts in case of obstacle on road or detection of drowsiness in user.

[0020] Brief description of drawings

[0021] Other objects, advantages and novel features of the invention will become apparent from the following detailed description of the present embodiment when taken in conjunction with the accompanying drawings.

[0022] Fig. 1 illustrates a Block Diagram of various components of the systems

[0023] Fig. 2 illustrates general system flow that is initiated by input data from capturing devices and provides output Fig. 3 illustrates MOE Calculation Module and its flow denoting the conditions and requirements of the MOE calculation process

[0024] Fig. 4 illustrates Face Detection

[0025] Fig. 5 illustrates Face detection with Landmark

[0026] Fig. 6 illustrates Eye Landmark points being considered for calculating EAR

[0027] Fig. 7 illustrates MOE values vs. Time (in minutes) graph showing drowsiness event

[0028] Fig. 8 illustrates MOE Values vs. Time (For smaller eyes) graph showing drowsiness event

[0029] Fig. 9 illustrates MOE Values vs. Time (In Minutes) graph showing no drowsiness event

[0030] Fig. 10 illustrates MOE Values vs. Time (In Minutes) graph showing no drowsiness event

[0031] Detailed Description of Invention

[0032] Before explaining the present invention in detail, it is to be understood that the invention is not limited in its application to the details of the construction and arrangement of parts illustrated in the accompany drawings. The invention is capable of other embodiment, as depicted in different figures as described above and of being practiced or carried out in a variety of ways. It is to be understood that the phraseology and terminology employed herein is for the purpose of description and not of limitation.

[0033] It is to be also understood that the term "comprises" and grammatical equivalents thereof are used herein to mean that other components, ingredients, steps, etc. are optionally present. For example, an article "comprising" (or "which comprises") components A, B, and C can consist of (i.e., contain only) components A, B, and C, or can contain not only components A, B, and C but also contain one or more other components.

[0034] A hardware device equipped with dual cameras, one facing the road and the other directed towards the driver's side, utilizes state-of-the-art machine learning and deep learning technology to enhance driver safety and optimize driving experiences. The primary focus of the driver side camera is to monitor and analyze the driver's behavior and facial expressions to ensure his attentiveness and emotional state during the journey.

[0035] The driver side camera, powered by machine learning and deep learning modules, accurately detects and tracks the driver's face in real-time [as shown in fig. 1], By continually analyzing facial landmarks and expressions [as shown in figure 2], the system can identify signs of drowsiness, fatigue, distraction, or even potential signs of impairment. If the driver displays signs of drowsiness or distraction, the device can issue timely alerts, such as audio warnings, prompting the driver to regain focus or take a break to prevent accidents. Furthermore, the device can also employ machine learning to recognize specific patterns in the driver's behavior and adapt its individual driving styles and preferences, making the system more personalized and effective.

[0036] Apart from monitoring the driver's state, the driver side camera can assist in enhancing driving performance. It can track eye movements and head orientation to gauge the driver's attention to the road and driving conditions. Additionally, the camera can analyze driving habits and provide feedback on factors like following distance, lane-keeping, and reaction times, promoting safer and more responsible driving behavior.

[0037] For instance, drooping eyelids, frequent blinking, or a lack of eye movement may signal that the driver is becoming drowsy and potentially at risk of falling asleep behind the wheel. When the system detects these drowsiness-related cues, it promptly issues warnings and these warnings can be in the form of audio alerts, or even haptic feedback to grab the driver's attention. The device may suggest taking a break, stopping at a nearby rest area, or switching drivers if applicable.

[0038] Moreover, machine learning capabilities of the system enable it to continuously adapt and improve its drowsiness detection algorithm. By learning from a vast dataset of drowsy driving scenarios and driver responses, the system can refine its accuracy and sensitivity over time

[0039] Facial Landmark detection model and implementation:

[0040] Deep-learning-based facial landmark detection systems have made impressive strides in recent years. Drowsiness is characterized by yawning, heavy eyelids, daydreaming, eye rubbing, an inability to concentrate, and lack of attention. The detection of drowsiness in drivers is facilitated by using facial landmarks and computer vision-based techniques. The system comprising of machine learning and deep learning algorithms is equipped with face and landmark detection models that have been specifically trained on a comprehensive driver dataset, ensuring their accuracy and reliability in identifying critical facial features and movements associated with drowsiness. After predicting landmarks, out of all facial landmarks it is eye region landmark that are extracted to calculate eye aspect ratio (EAR) values.

[0041] Method for calculating Eye Aspect Ratio (EAR):

[0042] Eye Aspect Ratio (EAR) is a metric commonly used in computer vision and facial recognition applications to quantify and analyze eye-related movements and behaviors. It is particularly useful in tasks like eye tracking, gaze estimation, and drowsiness detection. EAR is calculated based on the positions of specific facial landmarks, especially from eyes obtained through facial landmark detection algorithms.

[0043] To understand EAR, it needs to define the relevant facial landmarks. These landmarks consist of points representing the corners of the eyes, both on the horizontal and vertical edges. Typically, six landmarks are used for each eye: two for horizontal points (denoted as Pl and P4) [as shown in fig. 3] and four vertical points in pair of 2 points for the (denoted as P2-P6 and P3- P5) [as shown in fig. 3]. The position of these landmarks can be accurately tracked in video or and an image.

[0044] EAR is calculated using the following formula:

[0045] EAR= ( I I P2 - P6 I I + I IP3 - P5 I I ) / 2 * I IP1 - P4 I I

[0046] Here, I I Pn - Pm I I represents the Euclidean distance between two points Pn and Pm on the face.

[0047] Euclidean distance is separately and simultaneously calculated for both left eye and right eye.

[0048] The EAR value is a representation of the ratio between the vertical distance (the distance between the top and bottom eyelid landmarks) and the horizontal distance (the distance between the inner and outer corner landmarks) of the eye. In other words, it reflects how open or closed the eye is. When the eye is fully open, the EAR value will be higher, and as the eye starts closing or blinking, the EAR value decreases. For example, when a person blinks or has heavy eyelids due to drowsiness, the EAR value will decrease momentarily as the vertical distance between the eyelids becomes smaller. Conversely, when the eye is open wide, such as during moments of alertness, the EAR value will be higher.

[0049] In drowsiness detection systems, a certain threshold for EAR is set based on the behavior of the eye during wakeful and drowsy states. When the EAR falls below EAR calibrated threshold level, it indicates the closed or partially closed eyes; it indicates that the person may be experiencing drowsiness. When the eyes are open or close, the Eye Aspect Ratio (EAR) - a numerical measure - exhibits distinct changes. As the eyes blink, the EAR value either increases rapidly or decreases significantly.

[0050] Method of determining EAR Calibrated Threshold:

[0051] The system will calculate the open eye aspect ratio of the driver for the duration of two minutes comprising 1200 frames. This data will then be used to determine the median value of the open eye aspect ratio (EAR) over this specified duration. Based on this median value, which corresponds to the size of the driver's eyes, the system will dynamically set the threshold to either 0.16 or 0.18.

[0052] EAR threshold of 0.16 serves as the minimum value for smaller eyes, indicating when they may be considered closed or partially closed. After thorough analysis of multiple data sets, it has been determined that for relatively larger eyes, a threshold of 0.18 is the minimum value required to reliably discern whether the individual with normally-sized eyes has closed them or not. This approach allows for tailored thresholds based on eye size variations, ensuring accurate detection across a range of eye sizes and shapes.

[0053] Mean Open Eye Aspect Ratio (MOE):

[0054] After calculating EAR values and determining an EAR calibrated threshold to differentiate between open and closed eyes, the mean open eye aspect ratio (MOE) is calculated. MOE represents the average EAR values observed for open eyes over a specific time, providing valuable insights into a person's eye behavior. By continuously monitoring and computing the EAR values while the eyes are open, the MOE reflects the baseline level of eye openness for an individual.

[0055] Method for MOE Calculation:

[0056] The process described involves collecting EAR data for duration of two minutes, comprising 1200 frames of video. E ring this period, the system ensures the quality of the data by considering certain conditions. The first condition is that the face must be straight for the EAR data to be considered. This helps eliminate potential inaccuracies caused by varying head orientations during data collection.

[0057] The second condition involves using an EAR calibrated threshold, which is set at 0.16 or 0.18 (depending on eye size) to differentiate between open and closed eyes. Only EAR values greater than this threshold are taken into consideration. This step ensures that outliers or inaccurate data points that fall below the calibrated threshold (which might indicate closed eyes or other anomalies) are not included in the analysis.

[0058] When a frame meets both criteria (face is straight, and EAR value is greater than the threshold), the EAR value is added to the data collection. However, if either of the conditions is not met (the face is not straight or the EAR value is below the threshold), the system inserts a placeholder value of (-1) instead of the EAR value. This ensures that the ongoing data collection is not disrupted and maintains a consistent duration of 1200 frames. After the two-minute data collection period, the system performs data analysis by calculating the mean of the EAR values collected. However, before calculating the mean, the (-1) values are removed from the data to exclude the placeholder values. This step ensures that the calculated mean is representative of the actual EAR values during the two-minute period and is not influenced by the placeholder values.

[0059] By following this data preprocessing and analysis approach, the system can obtain a reliable and consistent mean EAR value for the two- minute duration, which serves as the Mean Open Eye Aspect Ratio (MOE). The MOE provides valuable insights into the driver's attentiveness and helps predict the likelihood of drowsiness during extended periods of driving. This information is crucial for timely interventions and ensuring road safety by addressing drowsiness before it escalates into a potential hazard on the road.

[0060] Method for implementing Mean Open Eye Aspect Ratio (MOE):

[0061] Drowsiness is indeed a gradual process that progresses through various stages before reaching complete eye closure. Recognizing this journey and predicting drowsiness beforehand is crucial for preventing accidents caused by inattentiveness on the road. The development of the Mean Open Eye Aspect Ratio (MOE) considers the critical stages in this drowsy cycle, with a focus on the mid-stage during heavy eye lids and frequent eye closure.

[0062] At the initial stage of drowsiness, drivers may experience yawning, which is often an early sign of fatigue. As drowsiness sets in further, heavy eyelids become noticeable, and drivers may find it increasingly challenging to keep their eyes open consistently. Frequent eye closures start to occur, where the eyes involuntarily shut momentarily, compromising the driver's alertness and reaction times. It is during these mid-stages of drowsiness that MOE plays a vital role. MOE is based on data from these critical mid-stages; it becomes a robust predictor of a driver's attentiveness. A higher MOE suggests that the person is generally more attentive, while a lower MOE indicates a higher propensity for drowsiness.

[0063] MOE serves as a critical indicator for predicting a person's behavior state, especially in scenarios like drowsiness detection or attentiveness monitoring. A higher MOE suggests that the person tends to keep their eyes open more often, indicating alertness and attentiveness. On the other hand, a lower MOE signifies that the person's eyes are frequently closing or blinking, which might indicate drowsiness or reduced attentiveness.

[0064] By comparing real-time MOE values with the established MOE threshold, the system can make predictions about the person's current behavior state. If the MOE value falls below the threshold established by system, it could signal a closed eye or partially closed eye, potentially indicating drowsiness or fatigue. Overall, MOE plays a crucial role in improving safety and enhancing performance by providing valuable information about a person's eye behavior state and enabling proactive interventions when necessary. MOE Result analysis using graphical representation:

[0065] A 1 hour of video data was taken and inferred that data using the face detection models and extracted EAR values and calculated MOE from above mentioned methodology. After calculating MOE, a graph was plotted describing MOE data with different devices and different users.

[0066] 1. The graph of real-time MOE Values vs. Time given in figure 4 shows MOE levels of Person A over the period of time, when EAR calibrated threshold of 0.18 was selected for calculating MOE, it was observed that the values of MOE was falling below threshold line of 0.25 when driver felt drowsy.

[0067] 2. The graph of real-time MOE Values vs. Time given in figure 5 shows MOE levels of Person B over the period of time having smaller eyes, when EAR calibrated threshold of 0.16 was selected for calculating MOE, it was observed that the values of MOE was falling below threshold line of 0.22 when driver felt drowsy.

[0068] 3. The graph of real-time MOE Values vs. Time given in figure 6 shows MOE levels of Person C over the period of time, when EAR calibrated threshold of 0.18 was selected for calculating MOE, it was observed that the values of MOE does not fall blow threshold line of 0.25, it stays above the threshold line for whole duration, that indicates that there was no drowsiness event detected throughout the period.

[0069] 4. The graph of real-time MOE Values vs. Time given in figure 6 shows MOE levels of Person C over the period of time, when EAR calibrated threshold of 0.18 was selected for calculating MOE, the values of MOE does not fall blow threshold line of 0.25, it stays above the threshold line for whole duration, that indicates that there was no drowsiness event detected throughout the period.

[0070] MOE Threshold determination based on observations and analysis:

[0071] After calculating the Mean Open Eye Aspect Ratio (MOE) from continuous video data for drivers exhibiting drowsy behavior, the MOE values are critical to identify patterns that could serve as indicators of an impending drowsy event. Remarkably, it is consistently observed that there was a distinct drop in MOE values before a valid drowsy event in more than 20 datasets collected from different devices and users.

[0072] To set a reliable threshold for identifying drowsy behavior based on MOE values, it was further analyzed that individuals with an EAR threshold of 0.16 must have an arithmetic MOE threshold greater than 0.20 to be categorised as not sleepy. If their arithmetic MOE falls below 0.20, any value will be considered indicative of a sleepy stage. Similarly, for those with an EAR threshold of 0.18, their arithmetic MOE threshold must remain above 0.23 to be classified as not sleepy. If their arithmetic MOE is less than 0.23, it will indicate an sleepy stage.

[0073] By plotting and examining the MOE values for numerous occurrences of drowsy behavior, it is established that these thresholds strike a balance between sensitivity and specificity in drowsiness detection. A higher MOE threshold ensures effective capturing of instances where the EAR values decrease significantly, indicating a higher likelihood of drowsiness. However, the threshold is carefully chosen to avoid false positives and unnecessary alerts. Using these MOE thresholds, this drowsiness detection system can accurately predict the onset of drowsy behavior in real-time.

[0074] The invention has been explained in relation to specific embodiment. It is inferred that the foregoing description is only illustrative of the present invention and it is not intended that the invention be limited or restrictive thereto. Many other specific embodiments of the present invention will be apparent to one skilled in the art from the foregoing disclosure.

[0075] All substitution, alterations and modification of the present invention which come within the scope of the following claims are to; which the present invention is readily susceptible without departing from the invention. The scope of the invention should therefore be determined not with reference to the above description but should be determined with reference to appended claims along with full scope of equivalents to which such claims are entitled.

[0076] List of Reference Numerals

[0077] 1 Hardware Device

[0078] 2 Road Facing Camera

[0079] 3 User Facing Camera 4 Machine hearing and Deep Learning Based Obstacle Detection Model

[0080] 5 Machine learnig and Deep hearing Based Face and Facial Landmark Detection Model

[0081] 6 Mean Open Eye Aspect Ratio (MOE) Calculation Module

[0082] 7 Alarm Device

Claims

We Claim:

1. A real-time drowsiness detection system comprising: a hardware device (1) equipped with dual cameras configured to capture video data simultaneously, with one road facing camera (2) and other user facing camera (3); a machine leaning and deep learning based obstacle detection model (4) configured to detect objects on road; a machine learning and deep learning based face and facial landmarks detection model (5) for identifying and real-time monitoring of facial landmarks of user to extract data for calculating the eye aspect ratio (EAR) using eye region landmarks; a Mean Open Eye Aspect Ratio (MOE) calculation module (6) configured to calculate real-time MOE of user by extracted values of EAR; and an alarm device (7) configured to provide appropriate notifications and alerts based on triggers caused by either of road side obstacles detection or user side drowsiness detection or both simultaneous.

2. The system as claimed in claim 1, wherein the MOE calculation module (6) calculates real-time MOE of user while user's eyes are open and EAR value to determine the openness state of an eye is referred herein as EAR calibration threshold.

3. A Method of calculating personalized MOE of user, comprising; a input received from the user facing camera (3);a machine learning and deep learning based face and facial landmarks detection model (5); a Mean Open Eye Aspect Ratio (MOE) calculation module (6); an alarm device (7); wherein the MOE calculation module (6) takes data of 120 seconds comprising 1200 frames with precondition that the user has straight face and concurrently captured EAR value by a machine learning and deep learning based face and facial landmarks detection model (5) is above EAR calibrated threshold, falling to satisfy either or both of these two conditions flags a placeholder value (-1) to be considered in place to ensure consistent data collection.

4. The method as claimed in claim 3, wherein EAR values are continually being collected and fetched to the MOE calculation module (6) for calculating real-time mean open eye aspect ratio (MOE).

5. The method as claimed in claim 3, wherein the EAR calibrated threshold value is determined based on the median value of the open eye aspect ratio (EAR) and based on calculated median value of EAR, the system will dynamically set the threshold to either 0.16 or 0.18.

6. The method as claimed in claim 3, wherein the realtime MOE is calculed based on the values of EAR that are above the EAR calibrated threshold of 0.16 or 0.18.

7. The method as claimed in claim 3, wherein the MOE threshold is selected based on EAR calibrated threshold, MOE threshold of 0.20 is selected when EAR calibrated threshold is 0.16 and MOE threshold of 0.23 when EAR calibrated threshold is 0.18 and so on.

8. The method as claimed in claim 3, wherein the alarm device provides timely alerts in form of audio-visual signal for instances when realtime MOE falls below MOE threshold, indicates the drowsiness event that is detected in user.

Images