Eyeglass-removable fatigue detection method and system based on eye aspect ratio change rate

Abstract

The application discloses a fatigue detection method and system based on eye aspect ratio change rate, and relates to the field of video image processing, and solves the problem that the existing fatigue detection technology excessively relies on blink detection, leading to inaccurate detection; the application comprises the following steps: S1, collecting driver video information and processing the information into pictures frame by frame; S2, removing glasses and glasses shadows in the pictures frame by frame by using a glasses removal algorithm; S3, synthesizing the pictures processed in S2 into a video frame by frame, detecting a face region in the video by using dlib, identifying key points of the face in the face region, and calculating an average value of a single frame eye; and S4, judging an eye fatigue state by using a unit time average value change rate of the eye in combination with a fatigue degree early warning index; the application continuously judges the fatigue degree by using the average value change rate of the eye changing with time, and predicts a blink frame parameter by using a regression function, so that the change rate is not affected by different blink habits of different people.

Description

Technical Field

[0001] The present invention relates to the field of video image processing, and in particular to a glasses-free fatigue detection method and system based on eye aspect ratio change rate. Background Art

[0002] Fatigue driving has become one of the main causes of traffic accidents. Researching fatigue driving detection systems and algorithms with practical application value is of great significance to preventing traffic accidents and protecting personal and property safety.

[0003] In recent years, researchers at home and abroad have conducted extensive research on fatigue detection systems and proposed the following detection methods:

[0004] 1. Detection methods based on physiological indicators. Using contact devices for real-time measurement, such as electroencephalogram (EEG) detection, has high fatigue judgment accuracy, but requires multiple instruments and affects the driver's driving environment.

[0005] 2. Fatigue detection methods based on vehicle or driver behavior. These methods primarily analyze the driver's pedal and steering wheel operation, as well as the vehicle's trajectory, to determine whether the driver is in a fatigued driving state. However, due to the influence of driving habits, the detection is not very robust.

[0006] 3. A non-contact fatigue detection method based on machine vision and facial key points. This method captures driver images and analyzes key points of the human eye. This method offers good accuracy, is easy to deploy, and does not affect the driver's operation. Summary of the Invention

[0007] In order to solve the problems existing in the above-mentioned prior art, the present invention provides a glasses-free fatigue detection method and system based on the eye aspect ratio change rate, which aims to solve the problem that the existing fatigue detection technology relies too much on blink detection, resulting in inaccurate detection.

[0008] The method for detecting fatigue without glasses based on the change rate of eye aspect ratio comprises the following steps:

[0009] S1: Collect driver video information and process it into frame-by-frame images;

[0010] S2: Use the glasses removal algorithm to remove glasses and glasses shadows from the image frame by frame;

[0011] S3: Synthesize the images processed by S2 into a video frame by frame, then use dlib to detect the face area in the video, identify the key points of the face in the face area, and calculate the eye average of a single frame;

[0012] S4: Use the rate of change of the average eye value per unit time combined with the fatigue warning index to determine the state of human eye fatigue.

[0013] Preferably, the glasses removal algorithm in S2 is implemented using a take-glass-net network, which includes a mask prediction stage and a mask removal stage. The mask prediction stage estimates the glasses mask Mg and the shadow mask Ms, and the mask removal stage obtains an image of the driver with the glasses and the glasses shadow removed.

[0014] Preferably, said S2 comprises the following steps:

[0015] S21: In the mask prediction stage, for the input portrait I with glasses, the glasses mask Mg and shadow mask Ms are estimated using the cross-domain segmentation module. The cross-domain segmentation module consists of a DA network, a Glass MaskNetwork, and a ShadowMaskNetwork. Specifically:

[0016] First, the original image I is processed into a uniform feature map F. The prediction process adopts a multi-step strategy. First, the glasses mask Mg is predicted using GlassMaskNetwork. Then, based on the predicted position of the glasses mask Mg, ShadowMaskNetwork is used to predict the position of the shadow mask Ms in the feature map F.

[0017] S22: In the mask removal stage, the two preset masks are used as position clues and a multi-step strategy is adopted. First, the original image I, the glasses mask Mg, and the shadow mask Ms are input into the De-Shadow Network to remove the glasses shadow in the original image I. The glasses mask Mg provides a position guide for the shadow mask Ms. Then, the original image I and the glasses mask Mg are input into the De-Glass Network to remove the glasses in the original image I. Finally, the driver image with the glasses and the glasses shadow removed is obtained.

[0018] Preferably, said S3 comprises the following steps:

[0019] S31: synthesize the images processed by S2 into a video frame by frame;

[0020] S32: Use the face detection function in dlib to predict the face area of ​​the video obtained in S31;

[0021] S33: Use the face key point detector shapepredictor in dlib to identify multiple key points of the face on the face area detected by S32;

[0022] S34: Extract key point parameters of the left eye and the right eye, and calculate the average value of the single frame eye.

[0023] Preferably, the calculation method of the single-frame eye average value in S3 is as follows:

[0024] Let the vertical length of the eye be H and the horizontal length be W, then the aspect ratio of the eye is:

[0025] Aspect=H / W

[0026] The calculation formula for the average value of a single frame eye:

[0027] Aspect eye =(Aspect L +Aspect R ) / 2

[0028] Among them, Aspect L Indicates the aspect ratio of the left eye; Aspect R Indicates the aspect ratio of the right eye.

[0029] Preferably, the S4 comprises the following steps:

[0030] S41: accumulating the average eye value of each frame within the unit time and dividing it by the total number of frames within the unit time to obtain the average eye value within the unit time;

[0031] S42: Perform regression analysis on the average eye value per unit time within a period of time to obtain the rate of change of the average eye value per unit time;

[0032] S43: The first fatigue stage, the second fatigue stage, and the third fatigue stage are set according to the fatigue level, and the threshold of the rate of change of the average eye value per unit time in different stages is set, that is, the fatigue level prompt indicator. Then, the rate of change of the average eye value per unit time is compared with the threshold in real time. If the threshold is reached, it is determined that the person being tested has entered the next fatigue stage.

[0033] Preferably, in step S43, a warning prompt is given when it is determined that the person being detected has entered the second fatigue stage; and an alarm prompt is given when it is determined that the person being detected has entered the third fatigue stage.

[0034] A glasses-removable fatigue detection system based on the eye aspect ratio change rate includes a video acquisition module, a glasses removal module, a single-frame eye average calculation module, and a fatigue stage judgment module; the video acquisition module is used to acquire video data in real time and convert it into frame-by-frame images, the glasses removal module is used to remove glasses and glasses shadows in the images frame by frame, the single-frame eye average calculation module synthesizes the images into videos frame by frame, and then uses dlib to detect the face area in the video, identify the key points of the face in the face area, and calculate the single-frame eye average; the fatigue stage judgment module first uses the single-frame eye average to calculate the eye average per unit time, and then uses the rate of change of the eye average per unit time combined with a threshold to judge the current fatigue stage of the detected person and provide corresponding prompts.

[0035] A computer-readable storage medium stores a fatigue detection program, which, when executed by a processor, implements the steps of a glasses-free fatigue detection method based on the eye aspect ratio change rate.

[0036] The beneficial effects of the present invention include:

[0037] The present invention continuously determines fatigue by measuring the rate of change of the average eye aspect ratio over time. It then predicts blink frame parameters using a regression function, ensuring that the rate of change is not affected by individual blinking habits. Finally, the rate of change of the average eye aspect ratio over time is measured to provide an early warning to the driver. Compared to traditional eye fatigue assessment criteria such as PERCLOS, which rely on blink statistics, the present invention's assessment is more objective, has a wider range of applications, and is more accurate.

[0038] In addition, this application combines a glasses removal algorithm, which can eliminate interference from glasses for people being detected and still maintain a high judgment accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The flowchart of the eye fatigue detection method based on the eye aspect ratio change rate can be removed without glasses; wherein EAR stands for eye aspect ratio.

[0040] Figure 2 This is the Take-glass-net network diagram involved in Example 1.

[0041] Figure 3 This is a detection diagram for removing eyeglass fatigue involved in Example 1.

[0042] Figure 4 This is a regression curve diagram of the rate of change of the average eye value per unit time under different fatigue levels of the subjects involved in Example 1.

[0043] Figure 5 This is a regression curve diagram of the change rates corresponding to the actual detection parameters of the person being detected in Example 1, the detection parameters when wearing glasses, and the detection parameters after removing glasses. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0045] Example 1

[0046] The following is combined with Figure 1-5 The specific embodiments of the present invention are described in detail;

[0047] The method for detecting fatigue without glasses based on the change rate of eye aspect ratio comprises the following steps:

[0048] S1: Collect driver video information and process it into frame-by-frame images;

[0049] S2: Use the glasses removal algorithm to remove glasses and glasses shadows from the image frame by frame;

[0050] The glasses removal algorithm is implemented using a take-glass-net network, which consists of two stages: a mask prediction stage to estimate the glasses mask Mg and the shadow mask Ms, and a mask removal stage to obtain a driver image with the glasses and the glasses shadow removed.

[0051] Specifically, the S2 includes the following steps:

[0052] S21: In the mask prediction stage, for the input portrait I with glasses, the glasses mask Mg and shadow mask Ms are estimated using the cross-domain segmentation module. The cross-domain segmentation module consists of a DA network, a Glass MaskNetwork, and a ShadowMaskNetwork. Specifically:

[0053] First, the original image I is processed into a uniform feature map F. The prediction process adopts a multi-step strategy. First, the GlassMaskNetwork is used to predict the glasses mask. Then, based on the predicted glasses mask position, the ShadowMaskNetwork is used to predict the shadow mask position in the feature map.

[0054] S22: In the mask removal stage, the two preset masks are used as position clues and a multi-step strategy is adopted. First, the original image I, the glasses mask Mg, and the shadow mask Ms are input into the De-Shadow Network to remove the glasses shadow in the original image I. The glasses mask Mg provides a position guide for the shadow mask Ms. Then, the original image I and the glasses mask Mg are input into the De-Glass Network to remove the glasses in the original image I. Finally, the driver image with the glasses and the glasses shadow removed is obtained.

[0055] S3: Synthesize the images processed by S2 into a video frame by frame, then use dlib to detect the face area in the video, identify the key points of the face in the face area, and calculate the eye average of a single frame;

[0056] S31: synthesize the images processed by S2 into a video frame by frame;

[0057] S32: Use the face detection function in dlib to predict the face area of ​​the video obtained in S31;

[0058] S33: Use the face key point detector shapepredictor in dlib to identify multiple key points of the face on the face area detected by S32;

[0059] S34: Extract key point parameters of the left eye and the right eye, and calculate the average value of the single frame eye.

[0060] Specifically, the calculation method of the single-frame eye average value in S3 is as follows:

[0061] Let the vertical length of the eye be H and the horizontal length be w, then the aspect ratio of the eye is:

[0062] Aspect=H / W Single frame eye average calculation formula:

[0063] Aspect eye =(Aspect L +Aspect R ) / 2

[0064] Among them, Aspect L Indicates the aspect ratio of the left eye; Aspect R Indicates the aspect ratio of the right eye.

[0065] S4: Use the rate of change of the average eye value per unit time combined with the fatigue warning index to determine the state of human eye fatigue.

[0066] S41: Accumulate the average eye value of each frame within the unit time and divide it by the total number of frames within the unit time to obtain the average eye value per unit time. Since the average number of blinks per minute for a person is 12-20, we take the minimum value and assume that a person blinks 12 times per minute to reduce the computational cost. Set 5 seconds as the unit time;

[0067] S42: Perform regression analysis on the average eye value per unit time within a period of time to obtain the rate of change of the average eye value per unit time, where a period of time here refers to 10 minutes;

[0068] Specifically, the average eye count per unit time within 10 minutes is shown in Table 1-3 below:

[0069] Table 1. Eye aspect ratios in the first fatigue stage

[0070] Eye aspect ratio Frame rate ratio_5 0.3488258309221826 2030 ratio_5 0.3539365399908740 2175 ratio_5 0.3410974605697282 2320 ratio_5 0.3516518893915801 2465 ratio_5 0.3403471846619106 2610 ratio_5 0.3538802607995771 2755 ratio_5 0.3360529604569680 2900 ratio_5 0.3491896554055731 3045

[0071] Table 2. Eye aspect ratios in the second fatigue stage

[0072]

[0073]

[0074] Table 3. Eye aspect ratios in the third fatigue stage

[0075] Eye aspect ratio Frame rate ratio_5 0.2906835348826286 2030 ratio_5 0.3040500985987624 2175 ratio_5 0.2985520126925376 2320 ratio_5 0.3133626374784978 2465 ratio_5 0.3127012091133299 2610 ratio_5 0.3138253305975456 2755 ratio_5 0.2846445291891444 2900 ratio_5 0.3049489859066432 3045

[0076] Specifically, the results of regression analysis of the data in the table are as follows: Figure 4 As shown in the figure, regression line 1 and observation 1 refer to the data related to the first fatigue stage; regression line 2 and observation 2 refer to the data related to the second fatigue stage; regression line 3 and observation 3 refer to the data related to the third fatigue stage.

[0077] It can be seen that the rate of change of the eye aspect ratio is different under different fatigue conditions. When using the eye aspect ratio as the long-term detection standard, we use regression equations of different degrees as reference indicators, and calculate the average eye aspect ratio in a unit of 5 seconds. The average eye aspect ratio of 10 minutes is subjected to regression analysis to detect the change in the fatigue level of the person being tested over a period of time. Compared with the perclos detection method, this method of detecting the rate of change of the eye aspect ratio does not rely on blink detection, and can reflect the changing trend of the fatigue level of the person being tested, thereby achieving the purpose of early prediction.

[0078] S43: The first fatigue stage, the second fatigue stage, and the third fatigue stage are set according to the fatigue level, and the threshold of the rate of change of the average eye value per unit time in different stages is set, that is, the fatigue level prompt indicator. Then, the rate of change of the average eye value per unit time is compared with the threshold in real time. If the threshold is reached, it is determined that the person being tested has entered the next fatigue stage.

[0079] Among them, when it is determined that the person being tested has entered the second fatigue stage, a warning prompt is given; when it is determined that the person being tested has entered the third fatigue stage, an alarm prompt is given.

[0080] In addition, during the actual test, the change rate of the eye aspect ratio after removing the glasses is closer to the actual test results. The specific test results are shown in Table 4 below;

[0081] Table 4. Comparison of eye aspect ratio detection before and after removing glasses

[0082]

[0083] Note: Left-Eye refers to the left eye aspect ratio, Right-Eye refers to the right eye aspect ratio, Eye-average value refers to the actual single-frame eye average, Take-Glass-Eye refers to the single-frame eye average after removing glasses detection, and Eye refers to the single-frame eye average of the original video without removing glasses.

[0084] After the above data are subjected to regression analysis, the parameter change rate is as follows Figure 5 As shown, regression line 1 and observation 1 refer to the actual parameters; regression line 2 and observation 2 refer to the original video detection parameters; regression line 3 and observation 3 refer to the detection parameters after removing the glasses. Analysis shows that:

[0085] Actual parameters Regression Function: y = -0.01x + 0.31

[0086] Original video detection parameters Regression Function: y = -0.02x + 0.37

[0087] Remove glasses detection parameters Regression Function: y = -0.01x + 0.18

[0088] It can be seen that the detection parameters after removing the glasses are closer to the actual parameters than the detection parameters before removing the glasses in the original video because the problem of key points moving upward caused by the glasses is solved.

[0089] Example 2

[0090] The fatigue detection system based on the eye aspect ratio change rate can be removed from the glasses. It should be understood that the system is the same as the above-mentioned Figure 1 The method embodiment corresponds to the following Figure 1 The various steps involved in the method embodiment and the specific functions of the system can be found in the description above. To avoid repetition, detailed descriptions are appropriately omitted here. Specifically, it includes a video acquisition module, a glasses removal module, a single-frame eye average value calculation module and a fatigue stage judgment module. The video acquisition module is used to acquire video data in real time and convert it into frame-by-frame pictures. The glasses removal module is used to remove glasses and glasses shadows in the picture frame by frame. The single-frame eye average value calculation module synthesizes the pictures into a video frame by frame and then uses dlib to detect the face area in the video and identify the key points of the face in the face area and calculate the single-frame eye average value. The fatigue stage judgment module first uses the single-frame eye average value to calculate the eye average value per unit time, and then uses the rate of change of the eye average value per unit time combined with the threshold to judge the current fatigue stage of the detected person and provide corresponding prompts.

[0091] Example 3

[0092] A computer-readable storage medium stores a fatigue detection program, which, when executed by a processor, implements the steps of a glasses-free fatigue detection method based on the eye aspect ratio change rate.

[0093] The above-described embodiments merely represent specific implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of protection of the present application. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the technical concept of the present application, and all such variations and improvements fall within the scope of protection of the present application.

Claims

1. A glasses-free fatigue detection method based on the eye aspect ratio change rate, characterized in that: The following steps are involved: S1: Collect driver video information and process it into frame-by-frame images; S2: Use the glasses removal algorithm to remove glasses and glasses shadows from the image frame by frame; S3: Synthesize the images processed by S2 into a video frame by frame, then use dlib to detect the face area in the video, identify the key points of the face in the face area, and calculate the eye average of a single frame; S4: Use the average change rate of the eye per unit time combined with the fatigue warning index to determine the state of human eye fatigue; The glasses removal algorithm in S2 is implemented using a take-glass-net network, which includes a mask prediction stage and a mask removal stage. The mask prediction stage estimates the glasses mask Mg and the shadow mask Ms, and the mask removal stage obtains the driver image with the glasses and the glasses shadow removed. The S2 comprises the following steps: S21: In the mask prediction stage, for the input portrait I with glasses, the glasses mask Mg and shadow mask Ms are estimated using the cross-domain segmentation module. The cross-domain segmentation module consists of a DA network, a Glass MaskNetwork, and a ShadowMaskNetwork. Specifically: First, the original image I is processed into a uniform feature map F. The prediction process adopts a multi-step strategy. First, the glasses mask Mg is predicted using GlassMaskNetwork. Then, based on the predicted position of the glasses mask Mg, ShadowMaskNetwork is used to predict the position of the shadow mask Ms in the feature map F. S22: In the mask removal stage, the two preset masks are used as position clues and a multi-step strategy is adopted. First, the original image I, the glasses mask Mg, and the shadow mask Ms are input into the De-Shadow Network to remove the glasses shadow in the original image I. The glasses mask Mg provides a position guide for the shadow mask Ms. Then, the original image I and the glasses mask Mg are input into the De-Glass Network to remove the glasses in the original image I. Finally, the driver image with the glasses and the glasses shadow removed is obtained.

2. The method for detecting fatigue based on eye aspect ratio change rate without glasses according to claim 1, characterized in that: The S3 includes the following steps: S31: synthesize the images processed by S2 into a video frame by frame; S32: Use the face detection function in dlib to predict the face area of ​​the video obtained in S31; S33: Use the face key point detector shape predictor in dlib to identify multiple key points of the face on the face area detected in S32; S34: Extract key point parameters of the left eye and the right eye, and calculate the average value of the single frame eye.

3. The method for fatigue detection based on eye aspect ratio change rate according to claim 1, wherein: The calculation method of the single-frame eye average value described in S3 is as follows: Let the vertical length of the eye be H and the horizontal length be W, then the aspect ratio of the eye is: ； The calculation formula for the average value of a single frame eye: ； Among them, Aspect L Indicates the aspect ratio of the left eye; Aspect R Indicates the aspect ratio of the right eye.

4. The method for fatigue detection based on eye aspect ratio change rate without glasses according to claim 1, characterized in that: The S4 comprises the following steps: S41: accumulating the average eye value of each frame within the unit time and dividing it by the total number of frames within the unit time to obtain the average eye value within the unit time; S42: Perform regression analysis on the average eye value per unit time within a period of time to obtain the rate of change of the average eye value per unit time; S43: The first fatigue stage, the second fatigue stage, and the third fatigue stage are set according to the fatigue level, and the threshold of the rate of change of the average eye value per unit time in different stages is set, that is, the fatigue level prompt indicator. Then, the rate of change of the average eye value per unit time is compared with the threshold in real time. If the threshold is reached, it is determined that the person being tested has entered the next fatigue stage.

5. The method for fatigue detection based on eye aspect ratio change rate according to claim 4, wherein: In step S43, when it is determined that the person being detected has entered the second fatigue stage, a warning prompt is given; when it is determined that the person being detected has entered the third fatigue stage, an alarm prompt is given.

6. A glasses-free fatigue detection system based on the eye aspect ratio change rate, characterized in that: It includes a video acquisition module, a glasses removal module, a single-frame eye average calculation module and a fatigue stage judgment module; the video acquisition module is used to collect video data in real time and convert it into frame-by-frame pictures, the glasses removal module is used to remove glasses and glasses shadows in the pictures frame by frame, the single-frame eye average calculation module synthesizes the pictures into videos frame by frame, and then uses dlib to detect the face area in the video, identify the key points of the face in the face area and calculate the single-frame eye average; the fatigue stage judgment module first uses the single-frame eye average to calculate the eye average per unit time, and then uses the change rate of the eye average per unit time combined with the threshold to judge the current fatigue stage of the detected person and provide corresponding prompts.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a fatigue detection program, which, when executed by a processor, implements the steps of the eye-aspect ratio-based fatigue detection method without glasses as described in any one of claims 1 to 5.

Images