A driver fatigue detection method for special vehicles

Abstract

The application discloses a driver fatigue detection method for special vehicles, which realizes early warning and identification of the driver fatigue state, and comprises the following steps: a behavior response time prediction model is constructed by using the response period of an abnormal event, the response period of the next abnormal event is estimated, and early warning of the fatigue state is realized; a face attribute identification model is used to identify the shielding condition of the face and mouth area of the driver, corresponding identification methods are executed according to the identification result, and identification of the fatigue state is realized. The application can advance the detection time window of the fatigue event, early warning of the fatigue state of the driver is realized, and fatigue detection can be performed on the driver wearing special equipment, so that the accident rate is reduced.

Description

Technical Field

[0001] This invention relates to the field of safe driving in automobiles, and more specifically to a method for detecting driver fatigue in special vehicles. Background Technology

[0002] Fatigue driving affects road safety and is one of the main causes of traffic accidents. Drivers become fatigued due to prolonged driving or insufficient rest. In this state, their reaction ability decreases and their control over the vehicle weakens. This manifests as a lack of concentration, an inability to accurately judge and handle sudden abnormal situations, which creates driving hazards and endangers traffic safety.

[0003] Driver fatigue detection technology is mainly researched in two categories: contact and non-contact. Contact-based technologies require drivers to wear additional sensors to acquire physiological data. The fatigue level is determined by analyzing parameters such as heart rate, blood oxygen saturation, pulse, respiratory rate, electromyography (EMG), and electroencephalography (EEG). Non-contact technologies primarily use cameras to capture driver images and analyze behavioral characteristics or vehicle driving information to assess fatigue. Behavioral characteristics mainly utilize facial images to identify behaviors such as closing eyes and yawning. Vehicle driving information includes speed, brake pedal pressure, steering wheel grip strength, and total driving time. Generally, contact-based technologies have higher detection accuracy than non-contact technologies. However, the physiological data acquisition equipment required for these technologies is typically expensive and can influence driver habits, making them less suitable for practical applications.

[0004] Currently, due to the difficulty in obtaining vehicle driving information, the most widely used fatigue detection method for ordinary vehicles is non-contact fatigue detection based on driver facial image features. This method incorporates driver eye-closing and yawning as key indicators of fatigue. However, this method has significant shortcomings, especially for special vehicles. Firstly, due to the special nature of special vehicles, drivers must wear helmets and masks, significantly obscuring their faces and making it impossible to obtain eye and mouth features. Secondly, yawning already indicates a relatively deep state of fatigue, which is unsuitable for the application scenarios of special vehicles. Therefore, it is necessary to pre-determine the fatigue detection time window for drivers of special vehicles and provide early warnings of fatigue status. Currently, fatigue warnings mainly rely on accumulated driving time, but road conditions, individual driver differences, and mental state vary in their impact on the duration of fatigue, easily leading to false alarms. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to propose a method and device for driver fatigue detection of special vehicles, so as to realize fatigue warning and detection of special vehicle drivers.

[0006] In a first aspect, embodiments of the present invention provide a driver fatigue warning method, comprising:

[0007] Detect abnormal events during vehicle operation and record the entire process until the abnormal event disappears as a time period;

[0008] A behavior response time prediction model is constructed based on the recorded time period;

[0009] Based on the behavioral response model, predict the response time of the next abnormal event, determine the current driver's fatigue state, and provide voice reminders;

[0010] The behavioral response time prediction model is updated based on the difference between the predicted response time of the next abnormal event and the actual response time of the next abnormal event.

[0011] Optionally, in this embodiment, the detection of abnormal events during vehicle operation and the recording of the entire process until the abnormal event disappears as a time period includes the following steps:

[0012] Anomaly detection is performed in real time based on a pre-trained deep learning-based anomaly detection model, and the working time of the anomaly detection model is recorded.

[0013] The timing starts after an abnormal event is detected and ends after the abnormal event disappears. The time difference between the two times constitutes a response time period.

[0014] Repeatedly record the response time cycle of abnormal events until a preset number is reached or the working time of the abnormal event detection model meets the preset duration.

[0015] Optionally, in this embodiment, the step of constructing a behavior response time prediction model based on the recorded time period includes the following steps:

[0016] Generate a driver response cycle table from the recorded response cycles of multiple abnormal events;

[0017] Density clustering algorithm is used to cluster the data in the driver response cycle table, and the maximum boundary of the most cluster is extracted as the fatigue warning threshold.

[0018] A behavioral response time prediction model is constructed by fitting a driver response periodic table using a neural network.

[0019] Optionally, in this embodiment, the step of predicting the response time of the next abnormal event based on the behavior response time prediction model and determining the current driver's fatigue state includes the following steps:

[0020] The response time period of the next abnormal event is predicted based on the obtained behavior response time prediction model.

[0021] The driver's fatigue state is determined based on the relationship between the predicted response time of the next abnormal event and the fatigue warning threshold.

[0022] A voice prompt is triggered based on the driver's fatigue level.

[0023] Optionally, in this embodiment, updating the behavioral response time prediction model based on the difference between the predicted response time of the next abnormal event and the actual response time of the next abnormal event includes the following steps:

[0024] Obtain the actual response time for the next exception event;

[0025] Determine the difference between the predicted response time to the abnormal event and the actual response time;

[0026] If the difference is greater than the preset threshold, the actual response time is added to the driver response cycle table, and the behavior response time prediction model is updated.

[0027] Secondly, embodiments of the present invention provide a method for driver fatigue recognition, comprising:

[0028] Acquire the driver's facial images, posture images, and voice;

[0029] Based on the driver's facial image, facial attributes are detected using a facial attribute model;

[0030] Based on the detected facial attribute results, execute the corresponding driver fatigue recognition method;

[0031] Fatigue alerts will be issued based on the recognition results.

[0032] Optionally, in this embodiment, acquiring the driver's facial image, posture image, and voice includes the following steps:

[0033] The driver's facial image is captured in each frame of video using the front-facing camera;

[0034] Use a side-mounted camera to acquire time-series images of the driver's attitude;

[0035] The driver's voice information is obtained using a voice sensing module.

[0036] Optionally, in this embodiment, the step of detecting facial attributes using a facial attribute model based on the driver's facial image includes the following steps:

[0037] The system identifies whether the mouth and eyes are obscured based on a pre-trained deep learning-based facial attribute model.

[0038] Optionally, in this embodiment, the step of using different fatigue recognition models to identify individuals based on the facial attribute results includes the following steps:

[0039] Based on the facial attribute recognition results, the corresponding driver fatigue recognition method is executed, as follows:

[0040] If the mouth and eyes are not obscured, the facial landmark detection model is used to detect the key points of the eyes and mouth. Then, the classification model is used to identify the opening and closing of the eyes and mouth in the key point area. By counting the number of times the eyes are closed and the number of times the mouth is opened within a preset time period, it is determined whether the driver is fatigued.

[0041] If the eyes are covered but the mouth is not, the facial landmark detection model is used to detect the key points of the mouth, and then the classification model is used to identify whether the mouth is open or closed. By statistically analyzing the duration of mouth opening and the speech recognition model, it is determined whether the driver is fatigued.

[0042] If the mouth is covered but the eyes are not, the facial landmark detection model is used to detect the key points of the eyes, and then the classification model is used to identify whether the eyes are open or closed. The driver's fatigue is determined by statistically analyzing the duration of closed eyes.

[0043] If the mouth and eyes are covered, a driver behavior recognition model is used to identify driver behavior and determine whether the driver is fatigued by head and hand movements.

[0044] Optionally, in this embodiment, the fatigue reminder based on the recognition result is specifically as follows:

[0045] Fatigue assessment is made based on the identification results;

[0046] If fatigue is detected, an audio-visual reminder will be given.

[0047] Thirdly, embodiments of the present invention provide a driver fatigue detection device, comprising:

[0048] Sensing module; processing module; alert module.

[0049] Optionally, in this embodiment, the sensing module includes:

[0050] The driver's cab voice acquisition unit is used to acquire the driver's voice.

[0051] The driver's cab image acquisition unit is used to acquire facial and posture images of the driver;

[0052] The driver's side external image acquisition unit is used to acquire road images and images of the environment around the vehicle.

[0053] Optionally, in this embodiment, the processing module includes:

[0054] Memory is used to store computer programs that can be executed in a processor;

[0055] The processor, which is communicatively connected to the memory and equipped with a computing accelerator card, is a computer program for executing the driver fatigue warning method and driver fatigue recognition method described in the first and second aspects above.

[0056] Optionally, in this embodiment, the reminder module includes:

[0057] A sound alert is used during the fatigue warning phase;

[0058] The fatigue detection stage uses audible and visual alarms.

[0059] As can be seen from the above description, the embodiments of the present invention have the following beneficial effects:

[0060] This invention provides a method and device for detecting driver fatigue in special vehicles. It detects driver fatigue from both early warning and identification perspectives. By recording the driver's response time to abnormal events, it provides early warning of accumulated driver fatigue. Simultaneously, it utilizes the driver's facial, head, voice, and posture features to construct a fatigue recognition model to determine whether the driver is fatigued and to issue corresponding audio-visual alerts. This allows for earlier detection of fatigue events and provides alerts when the driver is fatigued, thereby improving safety during the driving of special vehicles. Attached Figure Description

[0061] Figure 1 This is a composition diagram of a driver fatigue detection method according to an embodiment of the present invention;

[0062] Figure 2 This is a flowchart of a driver fatigue warning method according to an embodiment of the present invention;

[0063] Figure 3 This is a flowchart of a driver fatigue recognition method according to an embodiment of the present invention;

[0064] Figure 4 This is a flowchart of a first type of driver fatigue recognition method according to an embodiment of the present invention;

[0065] Figure 5 This is a flowchart of the second type of recognition method in a driver fatigue recognition method according to an embodiment of the present invention;

[0066] Figure 6 This is a flowchart of a third type of driver fatigue recognition method according to an embodiment of the present invention;

[0067] Figure 7This is a flowchart of the fourth type of driver fatigue recognition method according to an embodiment of the present invention;

[0068] Figure 8 This is a schematic diagram of a driver fatigue detection device according to an embodiment of the present invention; Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described below with reference to the accompanying drawings. The various details of the embodiments of the present invention are only for understanding purposes and do not limit the scope of protection of the present invention.

[0070] It should be noted that in the description of this invention, "a plurality of" means two or more. The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are only used to distinguish objects and do not emphasize order or sequential relationship. Furthermore, the embodiments mentioned in this invention are exemplary, and the features, structures, and characteristics described in conjunction with the embodiments can be included in at least one embodiment of this invention.

[0071] like Figure 1 The diagram shown is a compositional diagram of a driver fatigue detection method according to an embodiment of the present invention, comprising:

[0072] Driver fatigue warning method 101 provides an early warning of driver fatigue by estimating the driver's response time in abnormal situations at the next moment.

[0073] Driver fatigue recognition method 102 identifies the driver's fatigue state by analyzing the driver's facial, posture and voice information at the current moment.

[0074] like Figure 2 The diagram shown is a flowchart of a driver fatigue warning method according to an embodiment of the present invention, the method comprising the following steps:

[0075] Step S201: Acquire images of the road ahead and the surrounding environment of the vehicle.

[0076] This includes using a camera mounted on the upper center of the windshield of the driver's cab to capture images of the road ahead, and using cameras mounted on the left, right, and rear of the vehicle to capture images of the surrounding environment. Optionally, the camera capturing the road ahead image can be a wide-angle camera with a resolution of 1080P and infrared illumination, while the camera capturing the images of the surrounding environment can be a fisheye camera with a field of view greater than 180°.

[0077] Step S202: Detect abnormal events based on a pre-trained abnormal event detection model based on deep learning;

[0078] This includes a deep learning-based anomaly detection model that extracts semantic information from acquired images using convolutional neural networks to detect anomalies such as obstacles and lane departures. Specifically, the image of the vehicle's surrounding environment will be obtained by pre-calibrating images acquired from different locations on the vehicle to create a panoramic view. The anomaly detection model will then perform detection on this image, starting synchronously with the processor and timing its operation, denoted as t. AEW ;

[0079] Step S203: Continuously record the response time cycle for handling abnormal events;

[0080] Includes: timing t starting from the time an abnormal event is detected. AEC1 The timer t ends after the abnormal event ends. AEC2 Record this time period as an exception event handling response time cycle: T AEC =t AEC2 -t AEC1 ;Record the response time cycle of abnormal event handling in a loop, up to a count T. n Reaching the preset value N AEC Or the abnormal event detection model continues to work until the threshold T is reached. AEW In one embodiment of the present invention, N is... AEC Set to 40, T AEW Set to 1 hour;

[0081] Step S204: Generate a driver response cycle table, construct a behavior response time prediction model, and obtain a fatigue warning threshold;

[0082] This includes: creating a table of driver response time cycles by recording multiple abnormal event handling response cycles. AE The table is fitted with driver response curves using a backpropagation (BP) neural network to construct a behavior response time prediction model. Simultaneously, the table is clustered using the density clustering algorithm DBSCAN, with the largest boundary of the most numerous cluster serving as the fatigue warning threshold Th. The BP neural network is defined as follows:

[0083]

[0084] The BP neural network uses a three-layer model, x i For network input, O k For network output, g(x) is the activation function, and w is the network output. ij For the input layer to hidden layer weights, w jk For the weights from the hidden layer to the output layer, a j For the input layer to hidden layer bias, b kThe hidden layer is used for the bias from the output layer, where l is the number of hidden layer nodes, n is the number of input layer nodes, i is the input layer node label, and j is the hidden layer node label. In this embodiment of the invention, l is set to 10 and n is set to 1.

[0085] Step S205: Predict the next abnormal event response cycle and issue a fatigue warning;

[0086] This includes: estimating the response period T for the next abnormal event based on a behavioral response time prediction model. AEP Fatigue warnings are issued by comparing the result with a fatigue warning threshold Th, according to the following formula:

[0087] |T AEP -Th|>ξ

[0088] Wherein, ξ is a critical fluctuation value, which is set to 2 seconds in this embodiment of the invention; if T AEP When the absolute value of the difference between the driver and the fatigue warning threshold Th mentioned in step S204 exceeds the critical fluctuation value, it is considered that the driver is about to become fatigued, and a voice reminder is given.

[0089] Step S206: Update the behavior response time prediction model.

[0090] This includes: the response time period T for detecting and handling the next abnormal event. AER The abnormal event response period T obtained from judgment and prediction AEP The difference between the two values, if greater than a preset difference threshold Td, satisfies the following formula:

[0091] |T AER -T AEP |<Td

[0092] Where, if T AER With T AEP If the difference is greater than Td, the behavioral response time prediction model is considered to have made an error, and T is adjusted accordingly. AER Add to driver response cycle table S AE The behavior response time prediction model is then updated. In one specific embodiment of the invention, Td is set to 1 second.

[0093] Figure 3 The diagram shown is a flowchart of a driver fatigue recognition method according to an embodiment of the present invention, the method comprising the following steps:

[0094] Step S301: Obtain the driver's facial image, posture image, and voice information;

[0095] This includes: real-time acquisition of the driver's facial image via a camera installed directly in front of the driver's cab; acquisition of the driver's posture image via a camera installed on the side of the driver's cab; and acquisition of the driver's voice information via a voice acquisition device installed directly above the driver. Optionally, the facial image acquisition camera is a wide dynamic range camera with a 30° field of view, the posture image acquisition camera is a wide dynamic range camera with a 45° field of view, and the voice acquisition device is a voice device with a sensitivity of -37.5dB (±2dB).

[0096] Step S302: Recognize facial attributes based on a pre-trained deep learning-based facial attribute recognition model;

[0097] This includes a deep learning-based facial attribute model that uses convolutional neural networks to extract information features from acquired images, thereby enabling attribute recognition of whether a driver is obscuring their mouth and eyes. The model structure of this convolutional neural network is built using a multi-label learning approach, and the model evaluation metric is as follows:

[0098]

[0099] Where L is the number of attributes, N is the number of samples, i is the attribute label, and TP i and TN i P represents the number of correctly classified positive and negative samples, respectively. i and N i L represents the total number of positive and negative samples, and in this embodiment of the invention, L is set to 2. In the application scenario of this invention, the background is relatively fixed and simple, and the number of facial attribute categories is not large. In this embodiment of the invention, the model is optimized based on the MobileNetV2 model.

[0100] Step S303: Execute the corresponding recognition method based on the facial attribute recognition result;

[0101] This includes classifying facial attribute recognition results into the following four categories:

[0102] (1) The eyes and mouth are not covered;

[0103] (2) Eyes covered, mouth not covered;

[0104] (3) The eyes are not covered, but the mouth is covered;

[0105] (4) Eyes covered, mouth covered;

[0106] Step S304: Perform fatigue judgment and reminder based on the results of the recognition method.

[0107] This includes: determining the driver's fatigue state based on the results of the above recognition method, outputting fatigue information corresponding to the fatigue state via voice through the reminder module, and simultaneously triggering flashing lights to help the driver improve attention, alleviate fatigue, and thus ensure driving safety.

[0108] Figure 4 The method shown is for identifying the class (1) identification result in step S303 according to an embodiment of the present invention, including:

[0109] Step S401: Obtain facial key points using a key point detection model;

[0110] This includes: a deep learning-based facial landmark detection model that extracts deep semantic information from the acquired images through a convolutional neural network to complete the detection of key points of the face, eyes, nose, and mouth. In this embodiment of the invention, five key points are used, namely the center point of the left eye, the center point of the right eye, the tip of the nose, the left corner of the mouth, and the right corner of the mouth.

[0111] Step S402: Calculate the head tilt angle based on the key points of the left and right eyes;

[0112] Including: The formula for calculating the head tilt angle s is as follows:

[0113]

[0114] Where θ is the head tilt angle, (x1, y1) are the coordinates of the left eye, and (x2, y2) are the coordinates of the right eye.

[0115] Step S403: Perform face correction based on the tilt angle;

[0116] This includes: calculating an affine transformation matrix using the obtained head tilt angle, with the rotation center at the location of the nose key point; using the calculated affine transformation matrix to perform face correction, positioning the face in a horizontal, frontal position; and determining the coordinates of the corrected key point for the left eye. right eye nose Left corner of the mouth Right corner of the mouth

[0117] Step S404: Extract the images of the eye and mouth regions based on the standard face model;

[0118] This includes: a standard face model that divides the face horizontally into 5 equal parts and vertically into 3 equal parts; the center of the eyes divides the face into 2 equal parts; and the extraction of the eye and mouth regions satisfies the following formula:

[0119] (1) Left eye region:

[0120] ①

[0121] ②

[0122] ③

[0123] ④

[0124] (2) Right eye area:

[0125] ①

[0126] ②

[0127] ③

[0128] ④

[0129] (3) Mouth area:

[0130] ①

[0131] ②

[0132] ③

[0133] ④

[0134] Where face_w is the width of the detected face, face_h is the length of the detected face, (x l1 y l1 (x) represents the coordinates of the upper left corner of the left eye region. l2 y l2 (x) represents the coordinates of the lower right corner of the left eye region. r1 y r1 (x) represents the coordinates of the upper left corner of the right eye region. r2 y r2 (x) represents the coordinates of the lower right corner of the right eye region. m1 y m1 (x) represents the coordinates of the upper left corner of the mouth region. m2 y m2 () represents the coordinates of the lower right corner of the mouth area. The coordinates of the center point of the left eye are: The coordinates of the center point of the right eye are... The coordinates of the left corner of the mouth are... Let be the coordinates of the right corner of the mouth. a, b, c, and d are scaling factors used to scale the eye and mouth areas when the eyes are open and closed, and when the mouth is open and closed. In this embodiment of the invention, a is set to 0.15, b to 0.25, c to 0.15, and d to 0.35.

[0135] Step S405: Classify the eye and mouth areas;

[0136] This includes: using a deep learning-based classification model to classify the eye and mouth regions as open / closed eyes and open / closed mouths. In this embodiment of the invention, this classification model is an improvement on the MobileNetV3 model, where the loss function L is the cross-entropy loss function, as shown in the following formula:

[0137]

[0138] Where N is the total number of all samples, y i For the i-th sample, This is the value output by the model for the i-th sample.

[0139] Step S406: Perform fatigue assessment based on the classification results.

[0140] This includes: setting a fatigue continuous frame threshold Tk, a head tilt threshold Ts, and recording fatigue for k consecutive frames during which eye-closing and mouth-opening behaviors occur. In this embodiment of the invention, Tk is set to 10 and Ts is set to 45. The fatigue judgment condition satisfies the following formula:

[0141]

[0142] Where s is the head tilt angle calculated in step S402, and the driver is considered to be in a state of fatigue when the formula is satisfied.

[0143] Furthermore, Figure 5 The method shown is for identifying the type (2) identification result in step S303 according to an embodiment of the present invention, including:

[0144] Step S501: Obtain facial key points using a key point detection model;

[0145] This includes: a deep learning-based facial landmark detection model that extracts deep semantic information from the acquired images through a convolutional neural network to complete the detection of key points on the face and mouth; in one embodiment of the present invention, only the key points on the left and right corners of the mouth are needed.

[0146] Step S502: Extract and classify the mouth region image;

[0147] This includes: extracting the mouth region according to the extraction rules for the mouth region in step S304, and classifying the open and closed mouth states according to the classification model in step S405.

[0148] Step S503: Recognize speech information based on the speech recognition model;

[0149] This includes: when the classification model determines that the current frame is a mouth-open state, it triggers the speech recognition model to recognize the sound frequency captured by the speech collector.

[0150] Step S504: Perform fatigue assessment.

[0151] The process includes: if the duration of the mouth-opening state exceeds a preset time threshold upper limit Tc, it is determined to be a fatigued state; if the duration of the mouth-opening state is less than a preset time threshold lower limit Tb, it is determined to be a non-fatigue state; when the duration of the mouth-opening state is between Tb and Tc, it is determined whether there is a similar frequency of continuous duration Td within the duration based on speech recognition. If it exists, it is determined to be a fatigued state; otherwise, it is determined to be a non-fatigue state. In one embodiment of the present invention, Tc is set to 10 frames, Tb is set to 2 frames, and Td is set to 4 frames.

[0152] Figure 6 The method shown is for identifying the type of identification result in step S303 (3) according to an embodiment of the present invention, including:

[0153] Step S601: Obtain facial key points using a key point detection model;

[0154] This includes: a deep learning-based facial landmark detection model that extracts deep semantic information from the acquired images through a convolutional neural network to detect facial and eye landmarks. In this embodiment of the invention, only the left and right eye landmarks are needed.

[0155] Step S602: Extract and classify the eye region image;

[0156] This includes: extracting the eye region according to the extraction rules for the eye region in step S404, and classifying the open and closed eye states using the classification model in step S405.

[0157] Step S603: Perform fatigue assessment.

[0158] The method includes: when both eyes are continuously detected to be closed, if the duration of continuous eye closure exceeds a preset threshold Tec for both eyes, it is determined to be a fatigue state; if the duration of continuous eye closure exceeds a preset threshold Ted for a single eye, it is determined to be a fatigue state; if the duration of eye closure exceeds a preset threshold Tee, it is determined to be a fatigue state; the determination priority is as follows: Tec > Ted > Tee. In one embodiment of the present invention, Tec is set to 6 frames, Ted is set to 8 frames, and Tee is set to 10 frames.

[0159] Figure 7 The method shown is for identifying the type of identification result in step S303 (4) according to an embodiment of the present invention, including:

[0160] Step S701: Identify driver behavior using a driver behavior recognition model;

[0161] This includes: inputting 10 frames of images into a deep learning-based driver behavior recognition model at a time, recognizing the driver's hand and head movements within the video time frames, and detecting whether the driver exhibits behaviors such as tilting their head back, covering their mouth, or stretching their back. The driver behavior recognition model is composed of 3D convolutional kernels.

[0162] Step S702: Perform fatigue assessment.

[0163] This includes: determining the driver's actions in the current time sequence using the driver behavior recognition model; if the duration of the action exceeds a preset action duration threshold Tac, then determining that the driver is in a state of fatigue. In one embodiment of the present invention, Tac is set to 10 frames.

[0164] Figure 8 The image shown is a driver fatigue detection device according to an embodiment of the present invention, comprising:

[0165] The sensing module 801 includes an external camera, an internal camera, and a voice acquisition device. The external camera is used to acquire road condition images and images of the environment around the vehicle. The internal camera is used to acquire facial and posture images of the target driver. The voice acquisition device is used to acquire the voice information of the target driver.

[0166] The processing module 802 includes a memory and a processor, wherein the memory is used to store computer programs that can be executed in the processor; the processor is communicatively connected to the memory and is equipped with a computing acceleration card for executing computer programs that implement the above-described driver fatigue warning method and driver fatigue recognition method.

[0167] The reminder module 803 is used to provide reminders based on the results of the driver fatigue warning method and the driver fatigue recognition method. The reminder methods include voice reminders and light reminders. Specifically, the driver fatigue warning method uses voice broadcast reminders, while the driver fatigue recognition method uses sound, light, and voice reminders.

[0168] It is understood that driver fatigue detection devices in practical applications also include other necessary components, and there are no restrictions on the connection methods between components. All driver fatigue detection devices that can implement the embodiments of the present invention are within the protection scope of the present invention. Furthermore, the technical parts within the protection scope of the present invention are not limited to the specific embodiments given in this application; all technologies that do not contradict the solutions of the present invention are included within the protection scope of the present invention.

Claims

1. A method for detecting driver fatigue for special purpose vehicles, characterized in that Comprise: A) performing driver fatigue warning; B) performing driver fatigue identification, Wherein: Step A comprises: A1) acquiring front road image and vehicle body surrounding environment image, including using the camera installed in the middle position of the windshield of the driver's cab to collect the front road image, and using the cameras installed on the left side, right side and tail of the vehicle body to collect the surrounding environment image; A2) detecting abnormal events according to the pre-trained abnormal event detection model based on deep learning, comprising: Building an abnormal event detection model to detect obstacles and lane deviation abnormal events; Record the working time of the abnormal event detection model; A3) the cycle record abnormal event processing response time period, comprising: from detecting the abnormal event to the end of the abnormal event as an abnormal event processing response time period T AEC ; the cycle record abnormal event processing response time period meets that the number reaches a preset value N AEC or the abnormal event detection model continues to work to reach a threshold T AEW ; A4) generating a driver response cycle table and building a behavior response time prediction model and obtaining a fatigue warning threshold, comprising: generating a driver response cycle table S AE ; According to the driver response cycle table, a behavior response time prediction model is built using BP back propagation neural network, which is expressed as: where BP neural network adopts three-layer model, x i is network input, O k is network output, g(x) is excitation function, w ij is input layer to hidden layer weight, w jk is hidden layer to output layer weight, a j is input layer to hidden layer bias, b k is hidden layer to output layer bias, l is hidden layer node number, n is input layer node number, i is input layer node label, and j is hidden layer node label. Using the density clustering algorithm DBSCAN to cluster the table, and taking the maximum boundary of the cluster with the most number as the fatigue warning threshold Th: A5) predicting the next abnormal event response cycle and performing fatigue warning, comprising: According to the behavioral response time prediction model, the next abnormal event response period T is estimated AEP ; Comparing with the fatigue warning threshold Th to perform fatigue warning, expressed as: |T AEP -Th|>ξ Wherein, ξ is a critical fluctuation value, which is considered to be fatigue when it exceeds the value, and voice prompt is performed; A6) updating the behavior response time prediction model, comprising: detecting the next abnormal event handling response time period T AER ; The abnormal event response period T obtained from judgment and prediction AEP The relationship between the difference and the preset difference threshold Td is expressed as: |T AER -T AEP |<Td Wherein, if the difference between T AER and T AEP is greater than Td, it is considered that the behavior response time prediction model is incorrect, and T AER is added to the driver response cycle table S AE , and the behavior response time prediction model is updated. Step B comprises: B1) acquiring the face image, posture image and voice information of the driver, comprising: Using the camera installed in the front of the driver's cab to collect the face image of the driver in real time; Using the camera installed on the side of the driver's cab to obtain the time sequence posture image of the driver; Using the voice collector installed directly above the driver to obtain the voice information of the driver, B2) identifying the face attribute according to the pre-trained face attribute recognition model based on deep learning, comprising: Building a face attribute recognition model based on deep learning, and the evaluation index is expressed as: where L is the number of attributes, N is the number of samples, i is the attribute label, TP i and TN i are the number of correctly classified positive and negative samples, respectively, P i and N i are the total number of positive and negative samples, respectively. Completing the attribute recognition of whether the driver's mouth and eyes are blocked, B3) according to the face attribute recognition result, executing the corresponding recognition method, comprising: B31) in the case that the driver's eyes are not blocked and the mouth is not blocked, the eye and mouth key points are detected by using the face key point detection model, and then the classification model is used to identify the open and close of the eyes and mouth, and whether the driver is tired is determined by counting the number of closed eyes and the number of mouth opening in the preset time period; B32) in the case that the driver's eyes are blocked and the mouth is not blocked, the mouth key points are detected by using the face key point detection model, and then the classification model is used to identify the open and close of the mouth, and whether the driver is tired is determined by counting the mouth opening duration and the voice recognition model; B33) in the case that the driver's eyes are not blocked and the mouth is blocked, the eye key points are detected by using the face key point detection model, and then the classification model is used to identify the open and close of the eyes, and whether the driver is tired is determined by counting the closed eye duration; B34) In the case of driver eye occlusion and mouth occlusion, driver behavior recognition is performed using a driver behavior recognition model, and whether the driver is tired is determined through head and hand movements, B4) fatigue judgment and prompting are performed according to the results of the recognition method.

2. The driver fatigue detection method according to claim 1, characterized in that Step B31 includes: Obtaining facial key points using a key point detection model; Calculating the head tilt angle according to the left and right eye key points, which is represented as: Wherein θ is the head tilt angle, (x1, y1) is the left eye coordinate, and (x2, y2) is the right eye coordinate; Correcting the face according to the tilt angle; According to the standard face model, the eye and mouth region images are extracted, and the extraction of the eye and mouth regions is represented as: Left eye region: Right eye region: Mouth region: wherein face_w is the width of the detected face, face_h is the length of the detected face, (x l1 , y l1 ) is the top-left corner coordinate of the left eye region, (x l2 , y l2 ) is the bottom-right corner coordinate of the left eye region, (x r1 , y r1 ) is the top-left corner coordinate of the right eye region, (x r2 , y r2 ) is the bottom-right corner coordinate of the right eye region, (x m1 , y m1 ) is the top-left corner coordinate of the mouth region, (x m2 , y m2 ) is the bottom-right corner coordinate of the mouth region, is the coordinate of the center of the left eye, is the coordinate of the center of the right eye, is the coordinate at the left corner of the mouth, is the coordinate at the right corner of the mouth, and a, b, c, d are scaling factors for opening and closing eyes and opening and closing mouth, respectively, for scaling the eye and mouth regions when eyes are opened and closed and mouth is opened and closed. Classifying the eye and mouth regions, and the loss function L is selected as the cross-entropy loss function, which is represented as: where N is the total number of samples, y i is the true actual value for the i-th sample, is the value output by the model for the i-th sample; Fatigue determination is performed according to the classification results, and the judgment formula is represented as: Wherein s is the head tilt angle, k is the continuous time frame, and Tk and Ts are the set fatigue continuous frame threshold and head tilt threshold, respectively.

3. The driver fatigue detection method according to claim 2, characterized in that Step B32 includes: Obtaining facial key points using a key point detection model; Extracting the mouth region image and classifying it; Recognizing the voice information according to the voice recognition model; Fatigue judgment.

4. The driver fatigue detection method according to claim 2, characterized by Step B33 includes: Obtaining facial key points using a key point detection model; Extracting the eye region image and classifying it; Fatigue judgment.

5. The driver fatigue detection method according to claim 2, characterized by Step B34 includes: Recognizing the driver's behavior using a driver behavior recognition model; Fatigue judgment.

6. The driver fatigue detection method according to claim 2, characterized by Step B4 includes: Fatigue judgment according to the recognition results; Outputting the fatigue information corresponding to the fatigue state through the prompting module; Triggering the light to flash to improve the driver's attention.

7. A computer-readable storage medium storing a computer program, which can enable a processor to execute the method according to any one of claims 1-6.

Images