Bus driver fatigue parameter compensation and double-track parallel detection method

Abstract

The bus driver fatigue parameter compensation and double-track parallel detection method comprises the following steps: S1, labeling the eye blinking and yawning image data of the driver; S2, calculating the frame number of the bus driver closing eyes and opening mouth in the front face state by using a face key point algorithm, and comparing with the current data preprocessing method, and compensating according to the ratio; S3, double-track division of the fatigue state time sequence; after the image frame is detected according to the time sequence, the detection results of each area are output, and the complete time segment results are divided into double-track time sequence according to the vehicle speed and vehicle condition; S4, setting the double-track early warning mechanism of the fatigue state time sequence; the analysis data includes the eye blinking frame number, eye blinking frequency, mouth opening frame number and yawning frequency; the perclos eye The ratio of the number of closed eye frames to the total number of frames in a calculation unit can reflect the ratio of the eye closure duration to the detection time and the fatigue state of the driver.

Description

Technical Field

[0001] The present invention relates to the field of bus driver fatigue state detection methods, and in particular to bus driver fatigue parameter compensation and dual-track parallel detection methods. Background Art

[0002] Bus accidents caused by driver fatigue can have serious consequences and adverse impacts. Bus safety is directly related to driver behavior, and driver fatigue directly affects their driving behavior. Therefore, real-time detection and identification of driver fatigue can reduce these types of traffic accidents and improve overall bus operation efficiency and safety.

[0003] Currently, facial keypoint algorithms are widely used in practical applications to calculate blinking and yawning frequency. While this method is effective for real-time fatigue detection during normal driving, it still has certain issues. 1) Current detection methods only target normal driving conditions. However, bus driving often involves frequent stops, waiting for traffic lights, and yielding to pedestrians, resulting in frequent idling. These conditions are still considered working time for bus drivers. Subsequent bus simulation driving experiments found that idling tasks such as stops and waiting for traffic lights account for 15% to 20% of total driving time, while blinking accounts for 30% to 35% of the total. During idling, bus drivers' attention shifts, leading to a tendency to relax, resulting in increased blinking and yawning frequencies, which more strongly indicate fatigue and reveal their true state of fatigue. 2) When at a stop, bus drivers need to observe whether passengers scan the QR code and engage in a certain degree of communication. During this period, bus drivers frequently turn their heads and nod frequently, adjusting their masks several times. Facial keypoint algorithms cannot accurately locate the eye and mouth areas for these images, and significant offsets occur. This leads to inaccurate fatigue parameter calculations and an inability to accurately determine the driver's fatigue status. 3) The current threshold setting only applies to normal driving conditions. During idling conditions, such as at stops or waiting at traffic lights, bus drivers' attention shifts, leading to increased blinking and yawning rates.

[0004] To address the problem that facial key point algorithms are unable to detect, many researchers use target detection algorithms for identification. However, since quantitative standards such as P80 (when the eye pupil is covered by the eyelid for more than 80% of the entire face, the eyes are considered closed) cannot be used in the image labeling process, the labeling standards are prone to differences.

[0005] In view of the above problems, it is urgent to compensate the fatigue parameters of the bus driver and to detect the fatigue parameters in parallel in two tracks, that is, the yawning and blinking fatigue parameters obtained by the target detection algorithm are compensated, and the fatigue state of the bus driver in the normal driving state and the idling state (stopping at the station, waiting for the traffic signal, etc.) is detected, the threshold values are respectively set, the fatigue state of the bus driver during the whole driving is real-timely warned, and the driving safety of the bus driver is improved. SUMMARY

[0006] The present application overcomes the above-mentioned shortcomings of the prior art and provides a method for compensating fatigue parameters of a bus driver and detecting fatigue parameters in parallel in two tracks.

[0007] The fatigue state detection method for the bus driver combines the bus speed and the vehicle condition to distinguish the normal driving state and the idling state (stopping at the station, waiting for the traffic signal, etc.) of the bus driver, uses the image recognition algorithm to real-timely recognize the open and closed eyes and the open and closed mouth of the bus driver, and attributes the recognition results to different driving state time dimensions according to the bus speed and the vehicle condition, that is, the recognition results of the normal driving state and the idling state are respectively extracted and spliced, and the fatigue state parameters are compensated and analyzed in parallel in two tracks in the respective time dimensions.

[0008] The technical scheme of the present application is as follows:

[0009] The method for compensating fatigue parameters of a bus driver and detecting fatigue parameters in parallel in two tracks comprises the following steps:

[0010] S1, image data label is set up;

[0011] The current target detection algorithm has excellent detection effect for image classification and positioning, for example, SSD, YOLO, Faster-RCNN, etc., which can be used in this method. Since the face fatigue parameters are calculated by using image data, the blinking and yawning phenomena directly reflect the fatigue degree of the driver. The image data label setting method of the present application is as follows:

[0012] a) Blinking: the image data is labeled from the perspective of time sequence, that is, in the complete process from open eyes to closed eyes and then from closed eyes to open eyes, the image data of completely closed eyes is labeled as closed eye state by artificial judgment, and the rest of the data is labeled as open eye state.

[0013] b) Yawning: Since yawning is from closed mouth to open mouth, and then from open mouth to closed mouth, it is difficult to accurately define the open mouth critical state. According to the observation of multiple experiments, the duration of yawning is usually 2-3 seconds, and some even longer, lasting for 4-5 seconds. According to the yawning image data with and without wearing a mask, different labeling methods are set. The yawning data without wearing a mask is labeled from the perspective of continuous time sequence, and the mouth region of the image is labeled, avoiding confusion with normal communication and other states. After multiple experiments, it is determined that the middle X% of the image data of the complete time sequence image of yawning without wearing a mask is labeled as open mouth, and the rest is labeled as closed mouth.

[0014] During the experiment, it was found that when the driver wears a mask, yawning will appear to squeeze the eyes, raise the eyebrows, wrinkle the eyebrows, and other expressions, and the eye area will change significantly, accompanied by a large increase in the mask. Single feature may be confused with other actions and expressions, for example, when a person is angry, the eyes will be squeezed, the eyebrows will be wrinkled, and other situations will occur; when laughing, the mask will be greatly enlarged. Therefore, the features of the two regions (mask region, eye and nearby region) are combined, and the yawning image data with a mask is labeled from the perspective of the complete time sequence of the image. Since the mask almost disappears after wearing the mask, the start and end of yawning no longer greatly open the mask and the facial expression begins to recover, so as to improve the accuracy of yawning counting when wearing a mask, the middle Y% of the image data of the complete time sequence image of yawning with a mask is labeled as yawning.

[0015] S2, establish a parameter compensation method;

[0016] As can be seen from the principle of image data preprocessing method, the current closed eye, open mouth, and yawning image frame prediction number is less than the face key point algorithm, so a fatigue parameter compensation method is proposed. The number of frames of closed eyes and open mouths of bus drivers in the front face state is calculated by using the face key point algorithm (Dlib library), and compared with the current data preprocessing method, and the ratio is compensated.

[0017] S21) Calculate the eye aspect ratio;

[0018] E AR The ratio of the Euclidean distance between the eye longitudinal landmark and the transverse landmark can directly reflect the degree of eye closure. According to the positions of the six key points of the eyes, E AR The calculation formula is

[0019]

[0020] When the eyes of the bus driver are open, E ARvalue keeps dynamic balance, i.e. in small fluctuations. But when the bus driver's eyes are closed, E AR value will decrease rapidly, and recover to dynamic balance rapidly when the eyes are opened again. Therefore E AR value can directly reflect the state of eye opening and closing after combining with P80 standard. Eyeball images of multiple bus simulator drivers in the state of frontal face are intercepted, and the E AR values of the left and right eyes of the driver are calculated by using the face key point detector in the Dlib library, and the average value is taken.

[0021] S22) Calculate the mouth aspect ratio;

[0022] Referring to the definition method of eye aspect ratio, the mouth aspect ratio M AR is calculated according to the 8 feature points of the mouth. The feature point calculation formula is

[0023]

[0024] The mouth aspect ratio is calculated from the 8 feature points of the inner contour of the mouth, which can directly reflect the opening degree of the mouth. According to the figure, the bus driver will frequently communicate with the passengers getting on and off at the stop, and the M AR threshold value is set to distinguish between yawning and normal talking mouth opening amplitude.

[0025] S23) Establish E AR and M AR threshold value;

[0026] In order to determine the fluctuation range of E AR value and M AR value of the bus driver in the state of frontal face, the data in the state of frontal face in the video are intercepted, and the maximum value and the minimum value of E AR and M AR in the processed video stream data are calculated respectively. Finally, the E AR and M AR threshold values are determined by referring to the P80 index in perclos, and the threshold value calculation formula is

[0027] E AR = (E AR,max -E AR,min )(1-X1)+E AR,min (3)

[0028] M AR = (M AR,max -M AR,min )(1-X2)+M AR,min (4)

[0029] S24) Obtain compensation parameters;

[0030] E AR and MAR Threshold, complete intercept bus driver face state under the blink of an eye, yawn (not wearing a mask) image sequence frame, according to the threshold to determine the closed eye and the number of yawn frames, and compare this frame number with the target detection algorithm label frame number, to calculate the ratio, the formula for calculating the ratio S is

[0031]

[0032] The number of key point frames and target detection label frames are counted respectively when the number of blinks is 100, 500 and 1000 times, and the ratio S is calculated eye The interval. By increasing the number of blink samples, the accuracy of the ratio S eye can be improved, and it is more convincing.

[0033] The number of yawn frames is counted in the same way as the number of blinks, and the ratio S of the number of key point frames to the number of target detection label frames is calculated yawn interval.

[0034] Since the ratio S yawn is not wearing a mask, according to the target detection label preprocessing method, the value of S yawn-mask under the condition of wearing a mask is calculated as

[0035]

[0036] S3, time sequence double-track division of fatigue state;

[0037] After the image frames are detected in time sequence, the detection results of each area are output, and the complete time segment results are combined with the vehicle speed and vehicle condition to be divided into double-track time sequence, i.e. normal driving sequence and idling driving sequence. The normal driving sequence is the normal driving state of the bus, and the idling driving sequence refers to the driving state of the bus at the bus stop, waiting for traffic signal lights, and giving way to pedestrians, etc. The two sequences will be spliced respectively, and the fatigue state will be analyzed from the perspective of continuous time in each sequence. The specific steps are as follows:

[0038] S31: Combine the real-time bus speed obtained by the vehicle speed and vehicle condition sensor, and recheck the consistency of the time dimension of the vehicle speed and vehicle condition collection device and the mobile image collection device.

[0039] S32: The bus speed is 0 km / h and the speed is restored from 0 km / h as the time slice division node. Avoid breaking the complete sequence of eye detection (blinking) and mouth detection (yawning) when dividing the node, so compare the frame number of the counting sequence before and after the node, and divide the counting sequence completely to the side with more frame numbers.

[0040] S33: If the bus speed is 0 km / h within 10 seconds, the image frame detection result corresponding to the 10-second time slice is directly divided into the normal driving sequence; if the node appears, the image frame detection result corresponding to the time slice before the node is divided into the normal driving sequence, and the idling time of the bus is calculated from the node; if the idling time is less than 5 seconds, that is, the bus speed is restored within 5 seconds (commonly seen at intersections or zebra crossings for a short time), it is known through multiple times that the bus driver's attention does not shift significantly during this period and is still focused on the road conditions, and is in a normal driving state. Therefore, the node is divided from the speed recovery node, and the time slice and related results are divided into the normal driving sequence; if the idling time is greater than 5 seconds and less than 10 seconds (commonly seen at stops without passengers getting on or off or pedestrians), the time slice and results are divided into the idling driving sequence; if the idling time is greater than 10 seconds (commonly seen at traffic lights, traffic jams, and stops), each 10 seconds is divided as a time slice, and the last time slice less than or equal to 10 seconds is sequentially divided into the idling driving sequence.

[0041] S34: The two driving sequences are respectively spliced into the respective time slices and the corresponding image frame detection results, and the image frame results are re-corresponded to the two sequence time dimensions, that is, each starts from the beginning and takes the frame number as the horizontal coordinate.

[0042] S4, Establishing a fatigue state time sequence double-track early warning mechanism;

[0043] Since the time sequence double-track division is performed, it is necessary to set up a warning mechanism for the two driving sequences respectively. The data that can be analyzed includes the number of blinking frames, blinking frequency, number of mouth opening frames, and number of yawns. Define perclos eye The ratio of the number of closed-eye frames to the total number of frames in the unit can reflect the ratio of the duration of eye closure to the detection time and the fatigue state of the driver, as shown in formula 7. The number of blinks N eye Then add the counts of each sequence.

[0044]

[0045] In the formula, t eye represents the number of eye closure frames; T eye represents the total number of frames per unit time.

[0046] According to the relationship between the time and the number of frames per minute, define perclos mouth and perclos mouth-mask , so that the fatigue evaluation can describe the increase of the mouth opening feature frequency in the unit time as a local continuity, as shown in formulas 8 and 9.

[0047]

[0048]

[0049] wherein t mouth represents the number of frames of opening mouth; T mouth represents the total number of frames per unit time; t mouth-mask represents the number of frames of yawning; T mouth-mask represents the total number of frames per unit time.

[0050] S41) Setting a normal driving sequence warning mechanism:

[0051] It is found through the simulation driving platform experiment that the blink frequency of the bus driver is significantly reduced when focusing on driving, and the phenomenon of multiple blinking in a short time occurs due to eye dryness after focusing for a long time. When the driver is in a fatigue state, the eyes will be sluggish, and the blinking frequency will be lower than the normal value; in addition, some drivers may resist fatigue by frequent blinking, resulting in a blinking frequency higher than the normal value, so both too low or too high blinking frequency indicates deepening of fatigue. According to the above phenomenon, the normal driving sequence driving warning mechanism is as follows (the unit time is 1 minute):

[0052]

[0053] wherein perclos eye represents the ratio of the number of frames of closing eyes to the unit time, perclos mouth represents the ratio of the number of frames of opening mouth to the unit time, perclos mouth-mask represents the ratio of the number of frames of yawning to the unit time when wearing a mask, N eye represents the number of times of blinking per unit time, states eye represents the eye detection result. If any of the above conditions is met, it is determined that the bus driver is in a fatigue state under the normal driving sequence, and a warning is given.

[0054] S42) Setting an idling driving sequence warning mechanism:

[0055] It is found through the simulation driving platform experiment that the bus driver's overall concentration decreases after the vehicle is in an idling state due to higher concentration in the normal driving condition, and the yawning and blinking frequency is significantly improved, but the vehicle can still be driven normally after a short correction in the idling stage. According to the above phenomenon, the idling driving sequence driving warning mechanism is as follows (the unit time is 1 minute):

[0056]

[0057] wherein the characters represent the same meaning as formula 10.

[0058] Preferably, in step S23), X1 is 0.8, that is, when the degree of eye closure is greater than 80% of the eyelid covering the pupil, it is considered as closing eyes; similarly, X2 is 0.2, that is, when the degree of mouth opening is greater than 80% of the maximum opening, it is considered as yawning action.

[0059] The working principle of the present application is as follows: the bus speed and the vehicle condition are combined to distinguish the normal driving state and the idling state (stopping at stations, waiting for traffic signals, etc.) of the bus driver, the target detection algorithm is used to identify the open and closed eyes and the open and closed mouth of the bus driver in real time, the detection results are compensated by parameters, the identification results are attributed to different driving state time dimensions according to the speed and the vehicle condition, and are extracted and spliced, the fatigue state parameters are calculated and analyzed in parallel in the respective time dimensions, and different warning thresholds are set.

[0060] The advantage of the present application is that the fatigue state of the bus driver is detected by the target algorithm in the normal driving state and the idling state, and the parameters are compensated to be more in line with the fatigue parameter quantization standard, and different thresholds are set to provide real-time warning for the fatigue state of the bus driver during the whole journey, and to improve the safety of bus driving. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 (a)~(c) are the image annotation schematic diagrams of the present application, Figure 1 (c) is the image annotation schematic diagram of the present application, Figure 1 (a) is the open and closed eye annotation schematic diagram, Figure 1 (b) is the open and closed mouth annotation schematic diagram, Figure 1 (c) is the yawning annotation schematic diagram when wearing a mask;

[0062] Figure 2 (a)~(c) are the image annotation schematic diagrams of the present application, Figure 2 (b) is the eye feature point schematic diagram of the present application, wherein Figure 2 (a) is the open eye state, Figure 2 (b) is the closed eye state;

[0063] Figure 3 (a)~(c) are the image annotation schematic diagrams of the present application, Figure 3 (b) is the mouth feature point schematic diagram of the present application, wherein Figure 3 (a) is the open mouth state, Figure 3 (b) is the closed mouth state;

[0064] Figure 4 is the fatigue state time sequence double-track division schematic diagram of the present application. DETAILED DESCRIPTION

[0065] The technical solutions of the present application will be further described below with reference to the drawings.

[0066] The fatigue parameter compensation and double-track parallel detection method of the bus driver comprises the following steps:

[0067] S1, image data label making;

[0068] Current target detection algorithm has excellent detection effect for image classification and positioning, for example, SSD, YOLO, Faster-RCNN, etc., which can be used in the method. Since the face fatigue parameters are calculated by using image data, the blinking and yawning phenomenon directly reflects the fatigue degree of the driver. The image data label making method of the application is as shown in the figure, and the specific steps are as follows: Figure 1

[0069] a) Blinking: the image data is labeled from the perspective of time sequence, that is, in the complete process from opening eyes to closing eyes, and from closing eyes to opening eyes, the image data of completely closing eyes is labeled as closing eyes by artificial judgment, and the rest is labeled as opening eyes.

[0070] b) Yawning: since yawning is from closing mouth to opening mouth, and from opening mouth to closing mouth, it is difficult to accurately define the critical state of opening mouth. According to many experimental observations, the duration of yawning is usually 2-3 seconds, and some are even longer, lasting for 4-5 seconds. According to the yawning image data with and without wearing masks, different labeling methods are set. From the perspective of continuous time sequence, the mouth region of the image is labeled, and the state of normal communication is avoided. After many experiments, it is determined that the middle X% (70%) of the complete time sequence image of the unmasked yawning is taken and the mouth region is labeled as opening mouth, and the rest is labeled as closing mouth.

[0071] During the experiment, it is found that when the driver wears a mask, yawning will appear to squeeze the eyes, raise the eyebrows, wrinkle the eyebrows, and the eye region will change significantly, and the mask will be opened significantly. Single feature may be confused with other actions and expressions, for example, when a person is angry, he will also appear to squeeze his eyes and wrinkle his eyebrows; when laughing, the mask will be significantly enlarged. Therefore, the characteristics of the two regions (mask region, eye region and nearby region) are combined, and the yawning image data with mask is labeled from the perspective of complete time sequence. Since the mask feature almost disappears after wearing the mask, the start and end of yawning no longer significantly opens the mask and the facial expression starts to recover. Therefore, in order to improve the accuracy of yawning counting when wearing a mask, the middle Y% (50%) of the complete time sequence image of the yawning with mask is labeled as yawning.

[0072] S2, parameter compensation method is established;

[0073] ​From the principle of image data preprocessing method, the current closed eyes, mouth, yawning image frame prediction quantity is less than the human face key point algorithm, so the fatigue parameter compensation method is proposed. The number of frames of the bus driver in the front face state is calculated by using the face key point algorithm (Dlib library), and compared with the current data preprocessing method, and the compensation is made according to the ratio.

[0074] 1) Calculate the eye aspect ratio;

[0075] E AR The ratio of the Euclidean distance between the longitudinal and transverse landmarks of the eye can directly reflect the degree of eye closure. According to the position of the six key points of the eye, as shown in Figure 2 E AR The calculation formula is

[0076]

[0077] When the eyes of the bus driver are open, E AR value remains dynamic balance, that is, in a small fluctuation. But when the eyes of the bus driver are closed, E AR value will rapidly decrease, and will rapidly recover to dynamic balance when the eyes are opened again. Therefore, E AR value can directly reflect the open and closed state of the eyes after combining P80 standard. The blinking image of the bus driver in the front face state is intercepted, and the E AR value of the left and right eyes of the driver is calculated by using the face key point detector in the Dlib library, and the average value is taken.

[0078] 2) Calculate the mouth aspect ratio;

[0079] Referring to the definition method of eye aspect ratio, according to the eight feature points of the mouth, as shown in Figure 3 M AR The feature point calculation formula is

[0080]

[0081] The mouth aspect ratio calculated from the eight feature points of the mouth contour can directly reflect the opening degree of the mouth. According to the figure, the bus driver will frequently communicate with the passengers getting on and off at the stop, and the M AR threshold value is set to distinguish the yawning and normal speaking of the mouth opening amplitude.

[0082] 3) Establish E AR and M AR threshold;

[0083] In order to determine the E AR value and M ARThe fluctuation range of the value, the video of the face state data is intercepted, and the maximum value and the minimum value of E AR and M AR in the processed video stream data are calculated respectively. Finally, the E AR and M AR threshold values are determined by referring to the P80 index in the perclos. The threshold value calculation formula is

[0084] E AR =(E AR ,max-E AR,min )(1-X1)+E AR,min (3)

[0085] M AR =(M AR ,max-M AR,min )(1-X2)+M AR,min (4)

[0086] X1 is taken as 0.8, that is, when the degree of eye closure is greater than 80% of the eyelid covering the pupil, it is considered as closing the eyes. Similarly, X2 is taken as 0.2, that is, when the degree of mouth opening is greater than 80% of the maximum opening, it is considered as yawning. The determined threshold values are shown in Table 1, that is, E AR is less than 0.19, and M AR is greater than 0.60, it is considered as yawning.

[0087] Table 1 E AR and M AR threshold values

[0088]

[0089] 4) Obtain compensation parameters;

[0090] The E AR and M AR threshold values are determined by the above method, the image sequence frames of the bus driver in the face state are completely intercepted, the number of eye closure and yawning frames is determined according to the threshold values, and the frame number is compared with the frame number set by the target detection algorithm label, so as to calculate the ratio. The calculation formula of the ratio S is

[0091]

[0092] The number of eye closure frames is shown in Table 2. According to the table, the key point frame number and the target detection label frame number are respectively counted when the number of eye blinking is 100 times, 500 times and 1000 times, and the ratio S eye is in the interval of 1.31-1.34. By increasing the number of eye blinking samples, the accuracy of the ratio S eye can be improved, which is more convincing. In this case, S eye is taken as 1.32.

[0093] Table 2 closed eye frame number statistics

[0094]

[0095] The yawning frame number statistics are shown in Table 3. Due to the small number of yawning samples, only 5, 10 and 20 key point frames and target detection label frames are counted respectively, and the ratio S yawn The interval is 1.21-1.22, and the current S yawn is 1.21.

[0096] Table 3 yawning frame number statistics

[0097]

[0098] Since the ratio S yawn is not wearing a mask, according to the target detection label preprocessing method, the S yawn-mask value calculation formula for wearing a mask is

[0099]

[0100] According to the current calculation result, the current S yawn is 1.21.

[0101] S3, fatigue state time series double-track division;

[0102] After the image frame is detected according to the time sequence, the detection results of each region are output, and the vehicle speed and vehicle condition are combined to divide the complete time segment results into double-track time sequences, i.e. normal driving sequence and idling driving sequence. The normal driving sequence is that the bus is in a normal driving state, and the idling driving sequence refers to the driving state of the bus at the bus stop, waiting for traffic signal lights, and giving way to pedestrians, etc. The two sequences will splice the time pieces respectively, and analyze the fatigue state from the continuous time angle on their own sequences. The fatigue state time series double-track division is shown in Figure 4 , and the specific steps are as follows:

[0103] Step one: combine the real-time bus speed obtained by the speed and condition sensor, use Simulink in the bus simulation driving platform to collect the data of the driving simulator (Logitech G29), and the sampling frequency is 30 times / second, which is consistent with the image frame rate. Again, check the consistency of the time dimension of the speed and condition acquisition device and the mobile image acquisition device.

[0104] Step two: Set the bus speed of 0 km / h and the speed from 0 km / h as the time slice division node. Avoid breaking the complete sequence of eye detection (blinking) and mouth detection (yawning) count when dividing the node, so compare the frame count of the count sequence before and after the node, and divide the complete count sequence to the side of the time sequence with more frame count.

[0105] Step three: If the bus speed is 0 km / h within 10 seconds, the image frame detection result corresponding to the 10-second time slice is directly divided into the normal driving sequence; if the node appears, the image frame detection result corresponding to the time slice before the node is divided into the normal driving sequence, and the idle speed duration is calculated from the node; if the idle speed duration is less than 5 seconds, that is, the bus speed is restored within 5 seconds (commonly seen at intersections or zebra crossings for a short pause), it is known through multiple times that the bus driver's attention does not shift significantly during this period and remains focused on road conditions, remaining in a normal driving state. Therefore, the node is divided from the speed recovery, and the time slice and related results are divided into the normal driving sequence; if the idle speed duration is greater than 5 seconds and less than 10 seconds (commonly seen at stops without passengers getting on or off or giving way to pedestrians), the time slice and results are divided into the idle speed driving sequence; if the idle speed duration is greater than 10 seconds (commonly seen at traffic signals, traffic jams, and stops), the time slice length is divided every 10 seconds, and the last time slice less than or equal to 10 seconds is sequentially divided into the idle speed driving sequence.

[0106] Step four: The two driving sequences are respectively divided into their own time slices and corresponding image frame detection results, and the image frame results are re-corresponded to the two sequence time dimensions, that is, each starts from the beginning with frame number as the horizontal coordinate.

[0107] After completing the time sequence double-track division according to the above method, the idle speed state accounts for about 20% of the total time in the current bus simulation driving experiment. However, the number of blinks accounts for 30% to 35% of the total. During the idle speed state, the bus driver's attention shifts, and he is prone to a relaxed state, with increased blinking frequency and yawning frequency, expressing a stronger fatigue state and revealing his true fatigue state more clearly.

[0108] S4, set the time sequence double-track warning mechanism for the fatigue state;

[0109] Since the time sequence double-track division is performed, a warning mechanism needs to be set for the two driving sequences respectively. The data that can be analyzed includes the number of blinking frames, blinking frequency, mouth opening frames, and yawning times. Define perclos eye The ratio of the number of closed eye frames to the total number of frames in the calculation unit can reflect the ratio of the duration of eye closure to the detection time and the fatigue state of the driver, as shown in formula 7. The number of blinks N eye Then add the counts of each sequence.

[0110]

[0111] In the formula, t eye represents the number of eye closing frames; T eye represents the total number of frames per unit time.

[0112] According to the relationship between time and frame number per minute, define perclos mouth and perclos mouth-mask , so the fatigue evaluation can be described as the increase of mouth opening feature frequency per unit time in local continuity, as shown in formulas 8 and 9.

[0113]

[0114]

[0115] In the formula, t mouth represents the number of mouth opening frames; T mouth represents the total number of frames per unit time; t mouth-mask represents the number of yawning frames; T mouth-mask represents the total number of frames per unit time.

[0116] 1) Establish a normal driving sequence warning mechanism:

[0117] Through the simulation driving platform experiment, it is found that the blink frequency of bus drivers is significantly reduced when they are focused on driving, and after a long time of focus, the phenomenon of multiple blinks in a short time will appear due to eye dryness. When the driver is in a fatigue state, the eye will be sluggish, and the blink frequency will be lower than the normal value; in addition, some drivers may resist fatigue by frequent blinking, resulting in a higher blink frequency than the normal value, so both too low and too high blink frequencies indicate deepening of fatigue. According to the above phenomenon, the normal driving sequence driving warning mechanism is as follows (the unit time is 1 minute):

[0118]

[0119] In the formula, perclos eye represents the ratio of eye closing frames per unit time, perclos mouth represents the ratio of mouth opening frames per unit time, perclos mouth-mask represents the ratio of yawning frames per unit time when wearing a mask, N eye represents the number of blinks per unit time, states eye represents the eye detection result. If any of the above conditions is met, it is determined that the bus driver is in a fatigue state under the normal driving sequence, and a warning is given.

[0120] 2) Establish an idling driving sequence warning mechanism:

[0121] Through the simulation driving platform experiment, it is found that the bus driver's concentration is higher in normal driving, and the overall concentration is reduced after the vehicle is in idle state, yawning and blinking frequency is greatly improved, but it can still drive the vehicle normally after a short correction in the idle stage. According to the above phenomenon, the idle driving sequence driving warning mechanism is as follows (unit time is 1 minute):

[0122]

[0123] In the formula, the character represents the same meaning as formula 10.

Claims

1. A method for compensating fatigue parameters of a bus driver and detecting two parallel tracks, comprising the following steps: S1, establishing image data labels; The image data of the driver's blinking and yawning are labeled, specifically including: a) Blinking: the image data is labeled from the perspective of time sequence, that is, in the complete process from opening eyes to closing eyes, and then from closing eyes to opening eyes, the image data of complete closing eyes is labeled as closing eyes by artificial judgment, and the rest is labeled as opening eyes; b) Yawning: different labeling methods are set according to the image data of yawning with and without wearing a mask; the image data of yawning without wearing a mask is labeled from the perspective of continuous time sequence, and the mouth area of the image is labeled, avoiding confusion with the normal communication state; after multiple experiments, the middle X% of the image data of the complete time sequence image of yawning without wearing a mask is taken and the mouth area is labeled as opening, and the rest is labeled as closing; The characteristics of the mask area, eye area and nearby area are combined, and the image data of yawning with a mask is labeled from the perspective of complete time sequence, that is, the mask area, eye area and nearby area of the image are labeled; since the feature of the mouth area almost disappears after wearing a mask, the beginning and end of yawning no longer greatly open the mask and the facial expression begins to recover calm, so in order to improve the accuracy of yawning counting when wearing a mask, the middle Y% of the image data of the complete time sequence image of yawning with a mask is labeled as yawning; S2, establishing a parameter compensation method; As known from the principle of image data preprocessing method, the predicted number of image frames of current closing eyes, opening mouth and yawning is less than that of face key point algorithm, therefore a fatigue parameter compensation method is established; Based on the Dlib library, the number of frames of closing eyes and opening mouth of the bus driver in the front face state is calculated by using the face key point algorithm, and compared with the current data preprocessing method, and the compensation is made according to the ratio; S21) Calculate the aspect ratio of the eyes; E AR The ratio of the Euclidean distance between the longitudinal landmarks and the transverse landmarks of the eye can directly reflect the degree of eye closure. According to the positions of the 6 key points of the eye 6; E AR The calculation formula is When the eyes of the bus driver are open, E AR value remains in dynamic balance, that is, in small fluctuations; but when the eyes of the bus driver are closed, E AR value will rapidly decrease and rapidly recover to dynamic balance when the eyes are open again; therefore E AR value can directly reflect the open and closed state of the eyes after combining with the P80 standard; intercept the blinking image of the bus driver under the condition of the front face, and calculate the E AR value of the left and right eyes of the driver by using the face key point detector in the Dlib library and taking the average value; S22) Calculate the aspect ratio of the mouth; With reference to the eye aspect ratio definition method, the mouth aspect ratio M is defined according to the 8 feature points of the mouth AR The feature point calculation formula is The mouth length-width ratio is calculated from 8 feature points of the inner contour of the mouth, which can directly reflect the opening degree of the mouth; the bus driver will frequently communicate with passengers getting on and off the bus at the stop, and the mouth opening degree needs to be set M AR Threshold value distinguishes between yawning and normal talking S23) Establish E AR and M AR threshold value; To determine the fluctuation range of E AR and M AR values in the frontal face state of bus drivers, the video data in the frontal face state are intercepted, and the maximum and minimum values of E AR and M AR in the processed video stream data are calculated respectively; finally, the E AR and M AR threshold values are determined with reference to the P80 index in perclos, and the threshold value calculation formula is E AR = (E AR,max -E AR,min )(1 - X1) + E AR,min (3) M AR = (M AR,max -M AR,min )(1 - X2) + M AR,min (4) X1 is 0.8, which means that when the degree of eye closure is greater than 80% of the eyelid covering the pupil, it is considered as closing eyes; similarly, X2 is 0.2, which means that when the degree of mouth opening is greater than 80% of the maximum opening, it is considered as yawning action; S24) Obtain compensation parameters; E is determined by the above method AR and M AR Threshold, complete intercept bus driver face state under the frame of blink, yawn, not wearing a mask image sequence, according to the threshold to determine the number of closed eyes and yawn frames, and compare this frame number with the target detection algorithm label setting frame number, to calculate the ratio, the calculation formula of the ratio S is The key point frame number and the target detection label frame number are counted respectively when the blink number is 100, 500 and 1000, and the ratio S is calculated eye The interval; the accuracy of the ratio S is improved by increasing the number of blink samples eye ​ The yawp frame number statistics method is same with the blink, and the key point frame number and the target detection label frame number ratio S is calculated yawn Interval; Since the ratio S yawn is not wearing a mask, according to the target detection label preprocessing method, wearing a mask S yawn-mask The value calculation formula is S3, double-track division of time sequence of fatigue state; After the image frames are detected according to time sequence, the detection results of each area are output, and the complete time segment results are combined with the vehicle speed and vehicle condition to divide into double-track time sequence, that is, normal driving sequence and idling driving sequence; the normal driving sequence refers to the driving state of the bus in normal driving state, and the idling driving sequence refers to the driving state of the bus in the state of stopping at the station, waiting for traffic signal, and drivers having a long waiting time for pedestrians; the two sequences are spliced respectively, and the fatigue state is analyzed from the perspective of continuous time in each sequence, and the specific steps are as follows: S31: Combine the real-time bus speed obtained by the vehicle speed and vehicle condition sensor to recheck the consistency of the time dimension of the vehicle speed and vehicle condition collection device and the mobile terminal image collection device; S32: Taking the bus speed of 0 km / h and the speed recovery from 0 km / h as the time slice division node; avoiding breaking the complete sequence of eye blinking and yawning count when dividing the node, thus comparing the frame count of the count sequence before and after the node, and dividing the complete count sequence to the time sequence with more frame count; S33: If the bus speed of 0 km / h does not appear within 10 seconds, the image frame detection result corresponding to the 10-second time slice is directly divided into the normal driving sequence; if the node appears, the image frame detection result corresponding to the time slice before the node is divided into the normal driving sequence, and the idling time of the bus is calculated from the node; If the idling time is less than 5 seconds, that is, the bus speed is recovered within 5 seconds, it is known through multiple times that the bus driver's attention does not shift significantly during this period and is still focused on the road conditions, in the normal driving state; therefore, the node of speed recovery is divided, and the time slice segment from the node of 0 km / h speed to the node of speed recovery and the related results are divided into the normal driving sequence; if the idling time is greater than 5 seconds and less than 10 seconds, the time slice segment from the node of 0 km / h speed to the node of speed recovery and the related results are divided into the idling driving sequence; if the idling time is greater than 10 seconds, the time slice length is divided every 10 seconds, and the divided time slice segment and the last time slice segment less than or equal to 10 seconds are sequentially divided into the idling driving sequence; S34: The two driving sequences are respectively spliced with the divided time slices and the corresponding image frame detection results, and the image frame results are re-corresponded to the two sequence time dimensions, that is, each starts from the beginning with the frame number as the horizontal coordinate; S4, set up a double-track early warning mechanism for the fatigue state time sequence; Because of the time series double-track division, it is necessary to set the early warning mechanism for two driving sequences respectively; the analysis data includes the number of blinking frames, blinking frequency, the number of mouth opening frames, and the number of yawning times; define perclos eye The ratio of the number of closed-eye frames to the total number of frames in the unit can reflect the ratio of the duration of eye closure to the detection time and the fatigue state of the driver, as shown in formula 7; the number of blinks N eye Then add the count of each sequence. In the formula, t eye represents the number of eye-closed frames; T eye represents the total number of frames per unit time; The perclos is defined according to the relationship between the time within a minute and the number of frames mouth and the perclos mouth-mask The fatigue evaluation described in this way is the increase of mouth opening feature frequency within a unit time as a local continuity, as shown in formulas 8 and 9. In the formula, t mouth indicates the frame number of opening mouth; T mouth indicates the total frame number per unit time; t mouth-mask indicates the frame number of yawning; T mouth-mask represents the total number of frames per unit time; S41) Set up a normal driving sequence early warning mechanism: It is found through simulation driving platform experiments that the blink frequency of the bus driver is significantly reduced when focusing on driving, and after focusing for a long time, the phenomenon of multiple blinks in a short time due to eye dryness occurs; When the driver is in a fatigue state, the eyes will be sluggish, and the blink frequency will be lower than the normal value. In addition, some drivers will resist fatigue by frequent blinking, resulting in a blink frequency higher than the normal value, so both too low and too high blink frequencies indicate deepening of fatigue. According to the above phenomenon, the normal driving sequence driving early warning mechanism is as follows, with a unit time of 1 minute: In the formula, perclos eye represents the ratio of the number of closed-eye frames to the unit time, perclos mouth represents the ratio of the number of mouth-opening frames to the unit time, perclos mouth-mask represents the ratio of the number of yawning frames to the unit time when wearing a mask, N eye represents the number of blinks per unit time, states eye represents the eye detection result; if any of the above conditions is met, it is determined that the bus driver is in a fatigue state under normal driving sequence, and a warning reminder is given. S42) Set up an idling driving sequence early warning mechanism: It is found through simulation driving platform experiments that the bus driver has a high degree of concentration in normal driving conditions, and after the vehicle is in an idling state, the overall concentration decreases, and there is a significant increase in blinking frequency and yawning, but after a short correction during the idling phase, the vehicle can still be driven normally. According to the above phenomenon, the idling driving sequence driving early warning mechanism is as follows, with a unit time of 1 minute: In the formula, the character represents the same meaning as formula 10.

Images