Fatigue driving detection method and device

By extracting facial features on the front video stream of the driver, calculating the opening and closing states of the mouth and eyes, and generating facial expression state frequency data, the problem of low accuracy of fatigue driving detection in the prior art is solved, and efficient fatigue is achieved Driving detection.

CN118314558BActive Publication Date: 2024-11-05THE QUARTERMASTER RES INST OF THE GENERAL LOGISTICS DEPT OF THE CPLA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410486488.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-22
Publication Date
2024-11-05
Estimated Expiration
2044-04-22

AI Technical Summary

Technical Problem

The existing fatigue driving detection methods are affected by factors such as light intensity, light angle and color, making it difficult to accurately extract facial feature points of the driver, with low detection accuracy and high false alarm rate, and lack of accurate and lightweight facial feature extraction methods.

Method used

By collecting 1 frame of the driver's front video stream, positioning the face and outputting the mouth and eye area images, calculating its variance, generating an open and closed state data stream, extracting facial expression state frequency data, inferring fatigue degree and providing alarm prompts .

Benefits of technology

It realizes that facial feature extraction calculations are small, high accuracy, and quickly obtains facial dynamic expression characteristics, providing an efficient reference for fatigue driving detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118314558B_ABST
    Figure CN118314558B_ABST
Patent Text Reader

Abstract

The present invention discloses a fatigue driving detection method and device, the method comprising: obtaining a driver front image data set, the driver front image data set comprising N driver front images and time information of the N driver front images; processing the driver front image data set to obtain a mouth opening and closing state data set and an eye opening and closing state data set; performing dynamic feature extraction on the mouth opening and closing state data set to obtain first feature information; performing dynamic feature extraction on the eye opening and closing state data set to obtain dynamic feature information, the dynamic feature information comprising second feature information, third feature information and fourth feature information; processing the first feature information and the dynamic feature information to obtain driver fatigue degree information. The facial feature extraction method of the present invention has a small amount of calculation and a high accuracy, and can quickly obtain facial dynamic expression features, providing a reference for fatigue driving detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a fatigue driving detection method and device. Background Art

[0002] Fatigue driving is one of the main causes of dangerous events such as traffic accidents. The main methods currently used are based on vehicle status, human status, and a combination of the two. Human status detection mainly detects the driver's fatigue level by identifying and inferring the driver's facial features. However, due to the influence of many factors such as the intensity of light in the car, the angle of light, and similar colors, it is difficult to accurately extract the driver's facial feature points, resulting in low accuracy and high false alarm rate for fatigue driving detection. Therefore, it is very important to propose an accurate and lightweight facial feature extraction method for fatigue driving detection. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a fatigue driving detection method and device, which mainly includes three parts: facial static feature extraction, facial expression dynamic feature extraction, and fatigue state analysis and alarm. Facial static feature extraction is mainly achieved by collecting one frame of image in the driver's front video stream, locating the driver's face, and finally locating the mouth and eyes and outputting the mouth and eye area images. Then, according to the driver's mouth and eye area images, their variances are calculated respectively, and the opening and closing states are judged to generate the mouth and eye opening and closing state data streams, thereby forming various facial expression state frequency data. Finally, the driver's fatigue level is inferred, and a corresponding level of alarm prompt is issued.

[0004] In order to solve the above technical problems, the first aspect of the embodiment of the present invention discloses a fatigue driving detection method, the method comprising:

[0005] S1, obtaining a driver front image dataset; the driver front image dataset includes N driver front images and time information of the N driver front images being captured, where N is a positive integer;

[0006] S2, processing the driver's frontal image dataset to obtain a mouth opening and closing state dataset and an eye opening and closing state dataset;

[0007] S3, extracting dynamic features from the mouth opening and closing state data set to obtain first feature information;

[0008] S4, extracting dynamic features from the eye opening and closing state data set to obtain dynamic feature information; the dynamic feature information includes second feature information, third feature information and fourth feature information;

[0009] S5: Process the first characteristic information and the dynamic characteristic information to obtain driver fatigue level information.

[0010] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the driver's front image dataset to obtain a mouth opening and closing state dataset and an eye opening and closing state dataset includes:

[0011] S21, performing face positioning on any one of the driver's frontal images in the driver's frontal image data set to obtain a driver's facial image;

[0012] S22, performing mouth location on the driver's facial image to obtain an mouth area image;

[0013] S23, performing eye positioning on the driver's facial image to obtain an eye region image;

[0014] S24, judging the mouth opening and closing state of the mouth area image to obtain mouth opening and closing state information;

[0015] S25, determining the eye opening and closing state of the eye region image to obtain eye opening and closing state information;

[0016] S26, integrating the mouth opening and closing state information and the corresponding time information of the driver's front image acquisition to obtain a mouth opening and closing state data set;

[0017] S27, integrating the eye opening and closing state information and the corresponding time information of the driver's front image acquisition to obtain an eye opening and closing state data set.

[0018] As an optional implementation, in the first aspect of the embodiment of the present invention, the step of determining the mouth opening and closing state of the mouth region image to obtain the mouth opening and closing state information includes:

[0019] S241, performing variance calculation on the mouth area image to obtain mouth variance information;

[0020] S242, comparing the mouth variance information with a preset variance threshold, when the mouth variance information is greater than the preset variance threshold, the mouth opening and closing state information is an open state, otherwise the mouth opening and closing state information is a closed state.

[0021] As an optional implementation, in the first aspect of the embodiment of the present invention, the step of extracting dynamic features from the mouth opening and closing state data set to obtain first feature information includes:

[0022] S31, processing the mouth opening and closing state data set to obtain the opening time;

[0023] S32, when the opening time is greater than 3 seconds, confirming that the current state is a yawning state;

[0024] S33, recording the start times tma1 and tma2 of two adjacent yawning states;

[0025] S34, processing the start times tma1 and tma2 of the two adjacent yawning states to obtain first feature information;

[0026] The expression of the first characteristic information is:

[0027] Fa=1 / (tma1-tma2)*60 times / minute

[0028] Where Fa is the first feature information, * represents multiplication, and / represents division.

[0029] As an optional implementation, in the first aspect of the embodiment of the present invention, the step of extracting dynamic features from the eye opening and closing state data set to obtain dynamic feature information includes:

[0030] S41, processing the eye opening and closing state data set to obtain the eye closing time;

[0031] S42, processing the eye closing time to obtain second characteristic information and third characteristic information;

[0032] S43, processing the eye opening and closing state data set to obtain the eye opening state duration;

[0033] S44, processing the duration of the eye-opening state to obtain fourth feature information.

[0034] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the eye closure time to obtain the second feature information and the third feature information includes:

[0035] S421, determining the eye closing time, and when the eye closing time is greater than 2.5 seconds, confirming that the current state is a slowly closing eye state;

[0036] S422, recording the start times tmb1 and tmb2 of the slow closing states of two adjacent eyes;

[0037] S423, processing the start times tmb1 and tmb2 of the two adjacent eyes in the slow closing state to obtain second feature information;

[0038] The expression of the second characteristic information is:

[0039] Fb=1 / (tmb1-tmb2)*60 times / minute

[0040] Where, Fb is the second feature information, * represents multiplication, and / represents division;

[0041] S424, determining the eye closure time, and when the eye closure time is less than 1 second, confirming that the current state is a blinking state;

[0042] S425, recording the start times tmc1 and tmc2 of two adjacent blinking states;

[0043] S426, processing the start times tmc1 and tmc2 of the two adjacent blinking states to obtain third feature information;

[0044] The expression of the third characteristic information is:

[0045] Fc=1 / (tmc1-tmc2)*60 times / minute

[0046] Wherein, Fc is the third feature information, * represents multiplication, and / represents division.

[0047] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the duration of the eye-opening state to obtain the fourth feature information includes:

[0048] S441, determining the duration of the eye-opening state, and when the eye-opening state duration is greater than 10 seconds, determining that the current state is a staring state;

[0049] S442, recording the start times tmd1 and tmd2 of two adjacent gaze states;

[0050] S443, processing the start times tmd1 and tmd2 of the two adjacent gaze states to obtain fourth feature information;

[0051] The expression of the fourth characteristic information is:

[0052] Fd=1 / (tmd1-tmd2)*60 times / minute

[0053] Wherein, Fd is the fourth feature information, * represents multiplication, and / represents division.

[0054] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the first feature information and the dynamic feature information to obtain the driver fatigue level information includes:

[0055] Using a driver fatigue degree judgment model, the first feature information and the dynamic feature information are processed to obtain driver fatigue degree information;

[0056] The driver fatigue degree judgment model is:

[0057]

[0058] In the formula, F a is the first feature information, F b is the second feature information, F c is the third feature information, F d It is the fourth characteristic information, FD is the driver's fatigue level information, FD ≥ 0.9 is severe fatigue, and a first-level warning is issued; 0.7 ≤ FD < 0.9 is moderate fatigue, and a second-level warning is issued; 0.4 ≤ FD < 0.7 is mild fatigue, and a third-level warning is issued; FD < 0.4 is weak fatigue, and no warning is issued for the time being.

[0059] A second aspect of an embodiment of the present invention discloses a fatigue driving detection device, the device comprising:

[0060] A data acquisition module is used to acquire a driver front image data set; the driver front image data set includes N driver front images and time information of the N driver front images being acquired, where N is a positive integer;

[0061] A data set processing module, used for processing the driver's front image data set to obtain a mouth opening and closing state data set and an eye opening and closing state data set;

[0062] A first feature extraction module, used for performing dynamic feature extraction on the mouth opening and closing state data set to obtain first feature information;

[0063] A second feature extraction module is used to extract dynamic features from the eye opening and closing state data set to obtain dynamic feature information; the dynamic feature information includes second feature information, third feature information and fourth feature information;

[0064] The driver fatigue level judgment module is used to process the first feature information and the dynamic feature information to obtain the driver fatigue level information.

[0065] As an optional implementation, in the second aspect of the embodiment of the present invention, the processing of the driver's front image dataset to obtain a mouth opening and closing state dataset and an eye opening and closing state dataset includes:

[0066] S21, performing face positioning on any one of the driver's frontal images in the driver's frontal image data set to obtain a driver's facial image;

[0067] S22, performing mouth location on the driver's facial image to obtain an mouth area image;

[0068] S23, performing eye positioning on the driver's facial image to obtain an eye region image;

[0069] S24, judging the mouth opening and closing state of the mouth area image to obtain mouth opening and closing state information;

[0070] S25, determining the eye opening and closing state of the eye region image to obtain eye opening and closing state information;

[0071] S26, integrating the mouth opening and closing state information and the corresponding time information of the driver's front image acquisition to obtain a mouth opening and closing state data set;

[0072] S27, integrating the eye opening and closing state information and the corresponding time information of the driver's front image acquisition to obtain an eye opening and closing state data set.

[0073] As an optional implementation, in the second aspect of the embodiment of the present invention, the determining of the mouth opening and closing state of the mouth region image to obtain the mouth opening and closing state information includes:

[0074] S241, performing variance calculation on the mouth area image to obtain mouth variance information;

[0075] S242, comparing the mouth variance information with a preset variance threshold, when the mouth variance information is greater than the preset variance threshold, the mouth opening and closing state information is an open state, otherwise the mouth opening and closing state information is a closed state.

[0076] As an optional implementation, in the second aspect of the embodiment of the present invention, the step of extracting dynamic features from the mouth opening and closing state data set to obtain first feature information includes:

[0077] S31, processing the mouth opening and closing state data set to obtain the opening time;

[0078] S32, when the opening time is greater than 3 seconds, confirming that the current state is a yawning state;

[0079] S33, recording the start times tma1 and tma2 of two adjacent yawning states;

[0080] S34, processing the start times tma1 and tma2 of the two adjacent yawning states to obtain first feature information;

[0081] The expression of the first characteristic information is:

[0082] Fa=1 / (tma1-tma2)*60 times / minute

[0083] Where Fa is the first feature information, * represents multiplication, and / represents division.

[0084] As an optional implementation, in the second aspect of the embodiment of the present invention, the step of extracting dynamic features from the eye opening and closing state data set to obtain dynamic feature information includes:

[0085] S41, processing the eye opening and closing state data set to obtain the eye closing time;

[0086] S42, processing the eye closing time to obtain second characteristic information and third characteristic information;

[0087] S43, processing the eye opening and closing state data set to obtain the eye opening state duration;

[0088] S44, processing the duration of the eye-opening state to obtain fourth feature information.

[0089] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the processing of the eye closure time to obtain the second feature information and the third feature information includes:

[0090] S421, determining the eye closing time, and when the eye closing time is greater than 2.5 seconds, confirming that the current state is a slowly closing eye state;

[0091] S422, recording the start times tmb1 and tmb2 of the slow closing states of two adjacent eyes;

[0092] S423, processing the start times tmb1 and tmb2 of the two adjacent eyes in the slow closing state to obtain second feature information;

[0093] The expression of the second characteristic information is:

[0094] Fb=1 / (tmb1-tmb2)*60 times / minute

[0095] Where, Fb is the second feature information, * represents multiplication, and / represents division;

[0096] S424, determining the eye closure time, and when the eye closure time is less than 1 second, confirming that the current state is a blinking state;

[0097] S425, recording the start times tmc1 and tmc2 of two adjacent blinking states;

[0098] S426, processing the start times tmc1 and tmc2 of the two adjacent blinking states to obtain third feature information;

[0099] The expression of the third characteristic information is:

[0100] Fc=1 / (tmc1-tmc2)*60 times / minute

[0101] Wherein, Fc is the third feature information, * represents multiplication, and / represents division.

[0102] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the processing of the duration of the eye-opening state to obtain fourth feature information includes:

[0103] S441, determining the duration of the eye-opening state, and when the eye-opening state duration is greater than 10 seconds, determining that the current state is a staring state;

[0104] S442, recording the start times tmd1 and tmd2 of two adjacent gaze states;

[0105] S443, processing the start times tmd1 and tmd2 of the two adjacent gaze states to obtain fourth feature information;

[0106] The expression of the fourth characteristic information is:

[0107] Fd=1 / (tmd1-tmd2)*60 times / minute

[0108] Wherein, Fd is the fourth feature information, * represents multiplication, and / represents division.

[0109] As an optional implementation manner, in the second aspect of the embodiment of the present invention, the processing of the first feature information and the dynamic feature information to obtain the driver fatigue level information includes:

[0110] Using a driver fatigue degree judgment model, the first feature information and the dynamic feature information are processed to obtain driver fatigue degree information;

[0111] The driver fatigue degree judgment model is:

[0112]

[0113] In the formula, F a is the first feature information, F b is the second feature information, F c is the third feature information, F dIt is the fourth characteristic information, FD is the driver's fatigue level information, FD ≥ 0.9 is severe fatigue, and a first-level warning is issued; 0.7 ≤ FD < 0.9 is moderate fatigue, and a second-level warning is issued; 0.4 ≤ FD < 0.7 is mild fatigue, and a third-level warning is issued; FD < 0.4 is weak fatigue, and no warning is issued for the time being.

[0114] The third aspect of the present invention discloses another fatigue driving detection device, the device comprising:

[0115] A memory storing executable program code;

[0116] a processor coupled to the memory;

[0117] The processor calls the executable program code stored in the memory to execute part or all of the steps in the fatigue driving detection method disclosed in the first aspect of the embodiment of the present invention.

[0118] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0119] The present invention proposes a fatigue driving detection method and device, which mainly include three parts: facial static feature extraction, facial expression dynamic feature extraction, and fatigue state analysis and alarm. Facial static feature extraction is mainly to locate the driver's face by collecting one frame of image in the driver's front video stream, and finally locate the mouth and eyes and output the mouth and eye area images. Then, according to the driver's mouth and eye area images, their variances are calculated respectively, and the opening and closing states are judged to generate mouth and eye opening and closing state data streams, thereby forming various facial expression state frequency data. Finally, the driver's fatigue degree is inferred, and a corresponding level of alarm prompt is issued. The facial feature extraction method of the present invention has a small amount of calculation and a high accuracy rate. It can quickly obtain facial dynamic expression features, providing a reference for the detection of fatigue driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0120] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0121] Figure 1 It is a flow chart of a fatigue driving detection method disclosed in an embodiment of the present invention;

[0122] Figure 2 It is a flow chart of another fatigue driving detection method disclosed in an embodiment of the present invention;

[0123] Figure 3is a flow chart of facial static feature extraction disclosed in an embodiment of the present invention;

[0124] Figure 4 It is a flow chart of facial expression dynamic feature flow extraction disclosed in an embodiment of the present invention;

[0125] Figure 5 It is a structural schematic diagram of a fatigue driving detection device disclosed in an embodiment of the present invention;

[0126] Figure 6 It is a structural schematic diagram of another fatigue driving detection device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0127] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0128] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0129] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0130] Through research, it is found that the driver's facial information, especially the eyes and mouth, can best reflect the driver's fatigue state when driving a motor vehicle. When the driver is tired and dozing off, he is often accompanied by facial expressions such as yawning, slowly closing his eyes, blinking, and no change in his line of sight for a long time (staring). Therefore, according to the above characteristics of fatigue driving. The present invention discloses a fatigue driving detection method and device, which includes: obtaining a driver's front image data set, the driver's front image data set includes N driver front images and time information of the acquisition of N driver front images; processing the driver's front image data set to obtain a mouth opening and closing state data set and an eye opening and closing state data set; performing dynamic feature extraction on the mouth opening and closing state data set to obtain first feature information; performing dynamic feature extraction on the eye opening and closing state data set to obtain dynamic feature information, the dynamic feature information includes second feature information, third feature information and fourth feature information; processing the first feature information and the dynamic feature information to obtain driver fatigue information. The facial feature extraction method of the present invention has a small amount of calculation and high accuracy, and can quickly obtain facial dynamic expression features, providing a reference for the detection of fatigue driving. The following are detailed descriptions.

[0131] Embodiment 1

[0132] See also Figure 1 , Figure 1 : is a flow chart of a fatigue driving detection method disclosed in an embodiment of the present invention. Figure 1 The described fatigue driving detection method is used to infer the driver's fatigue level and issue a corresponding level of warning prompt, which is not limited in the embodiments of the present invention. Figure 1 As shown, the fatigue driving detection method may include the following operations:

[0133] S1, obtaining a driver front image dataset; the driver front image dataset includes N driver front images and time information of the N driver front images being captured, where N is a positive integer;

[0134] S2, processing the driver's frontal image dataset to obtain a mouth opening and closing state dataset and an eye opening and closing state dataset;

[0135] S3, extracting dynamic features from the mouth opening and closing state data set to obtain first feature information;

[0136] S4, extracting dynamic features from the eye opening and closing state data set to obtain dynamic feature information; the dynamic feature information includes second feature information, third feature information and fourth feature information;

[0137] S5: Process the first characteristic information and the dynamic characteristic information to obtain driver fatigue level information.

[0138] Optionally, the processing of the driver's frontal image dataset to obtain a mouth opening and closing state dataset and an eye opening and closing state dataset includes:

[0139] S21, performing face positioning on any one of the driver's frontal images in the driver's frontal image data set to obtain a driver's facial image;

[0140] S22, performing mouth location on the driver's facial image to obtain an mouth area image;

[0141] S23, performing eye positioning on the driver's facial image to obtain an eye region image;

[0142] S24, judging the mouth opening and closing state of the mouth area image to obtain mouth opening and closing state information;

[0143] S25, determining the eye opening and closing state of the eye region image to obtain eye opening and closing state information;

[0144] S26, integrating the mouth opening and closing state information and the corresponding time information of the driver's front image acquisition to obtain a mouth opening and closing state data set;

[0145] S27, integrating the eye opening and closing state information and the corresponding time information of the driver's front image acquisition to obtain an eye opening and closing state data set.

[0146] Optionally, the judging the mouth opening and closing state of the mouth area image to obtain the mouth opening and closing state information includes:

[0147] S241, performing variance calculation on the mouth area image to obtain mouth variance information;

[0148] S242, comparing the mouth variance information with a preset variance threshold, when the mouth variance information is greater than the preset variance threshold, the mouth opening and closing state information is an open state, otherwise the mouth opening and closing state information is a closed state.

[0149] Optionally, the extracting dynamic features of the mouth opening and closing state data set to obtain first feature information includes:

[0150] S31, processing the mouth opening and closing state data set to obtain the opening time;

[0151] S32, when the opening time is greater than 3 seconds, confirming that the current state is a yawning state;

[0152] S33, recording the start times tma1 and tma2 of two adjacent yawning states;

[0153] S34, processing the start times tma1 and tma2 of the two adjacent yawning states to obtain first feature information;

[0154] The expression of the first characteristic information is:

[0155] Fa=1 / (tma1-tma2)*60 times / minute

[0156] Where Fa is the first feature information, * represents multiplication, and / represents division.

[0157] Optionally, the extracting dynamic features of the eye opening and closing state data set to obtain dynamic feature information includes:

[0158] S41, processing the eye opening and closing state data set to obtain the eye closing time;

[0159] S42, processing the eye closing time to obtain second characteristic information and third characteristic information;

[0160] S43, processing the eye opening and closing state data set to obtain the eye opening state duration;

[0161] S44, processing the duration of the eye-opening state to obtain fourth feature information.

[0162] Optionally, the processing of the eye closure time to obtain the second feature information and the third feature information includes:

[0163] S421, determining the eye closing time, and when the eye closing time is greater than 2.5 seconds, confirming that the current state is a slowly closing eye state;

[0164] S422, recording the start times tmb1 and tmb2 of the slow closing states of two adjacent eyes;

[0165] S423, processing the start times tmb1 and tmb2 of the two adjacent eyes in the slow closing state to obtain second feature information;

[0166] The expression of the second characteristic information is:

[0167] Fb=1 / (tmb1-tmb2)*60 times / minute

[0168] Where, Fb is the second feature information, * represents multiplication, and / represents division;

[0169] S424, determining the eye closure time, and when the eye closure time is less than 1 second, confirming that the current state is a blinking state;

[0170] S425, recording the start times tmc1 and tmc2 of two adjacent blinking states;

[0171] S426, processing the start times tmc1 and tmc2 of the two adjacent blinking states to obtain third feature information;

[0172] The expression of the third characteristic information is:

[0173] Fc=1 / (tmc1-tmc2)*60 times / minute

[0174] Wherein, Fc is the third feature information, * represents multiplication, and / represents division.

[0175] Optionally, the processing of the duration of the eye-opening state to obtain fourth feature information includes:

[0176] S441, determining the duration of the eye-opening state, and when the eye-opening state duration is greater than 10 seconds, determining that the current state is a staring state;

[0177] S442, recording the start times tmd1 and tmd2 of two adjacent gaze states;

[0178] S443, processing the start times tmd1 and tmd2 of the two adjacent gaze states to obtain fourth feature information;

[0179] The expression of the fourth characteristic information is:

[0180] Fd=1 / (tmd1-tmd2)*60 times / minute

[0181] Wherein, Fd is the fourth feature information, * represents multiplication, and / represents division.

[0182] Optionally, the processing of the first feature information and the dynamic feature information to obtain driver fatigue level information includes:

[0183] Using a driver fatigue degree judgment model, the first feature information and the dynamic feature information are processed to obtain driver fatigue degree information;

[0184] The driver fatigue degree judgment model is:

[0185]

[0186] In the formula, F a is the first feature information, F b is the second feature information, F c is the third feature information, F dIt is the fourth characteristic information, FD is the driver's fatigue level information, FD ≥ 0.9 is severe fatigue, and a first-level warning is issued; 0.7 ≤ FD < 0.9 is moderate fatigue, and a second-level warning is issued; 0.4 ≤ FD < 0.7 is mild fatigue, and a third-level warning is issued; FD < 0.4 is weak fatigue, and no warning is issued for the time being.

[0187] Optionally, a frontal image of the driver is obtained, and a feature-based cascade classifier can be used to detect faces in the image or video. The purpose of this step is to locate and identify the face area.

[0188] Facial feature calibration: In the detected face area, a facial feature calibration algorithm (such as a method based on facial key point detection) is used to calibrate facial key points, such as eyes, nose, mouth, etc.

[0189] Mask generation: Based on the calibrated facial key points, a mask is generated to cover the unwanted facial features. The mask can be a fixed-shaped area or dynamically adjusted according to the position and shape of the facial key points.

[0190] Optionally, the facial feature calibration method is:

[0191] The detected faces are sampled to obtain N sub-images, and the sub-images are scaled normalized without changing the structure of each sub-image. The sub-images are filled around them to obtain sub-images of the same size. The features of the sub-images are extracted using the VGG19 network, and then the features output by each layer of the VGG19 network are clustered using the agglomerative hierarchical clustering method to obtain the clustering feature parameters of each sub-image. The clustering feature parameters are used as the key points of the face.

[0192] Optionally, the method of processing the first feature information and the dynamic feature information to obtain the driver fatigue level information may be as follows:

[0193] A segment of video data of the driver is extracted, and features are extracted for each frame of the video data to obtain first feature information, second feature information, third feature information, and fourth feature information. For a segment of video data, a first feature information matrix, a second feature information matrix, a third feature information matrix, and a fourth feature information matrix can be obtained. The first feature information matrix, the second feature information matrix, the third feature information matrix, and the fourth feature information matrix are fused in pairs to obtain fused feature information, specifically: the first feature information matrix and the second feature information matrix are fused to obtain a fifth feature information matrix, the third feature information matrix and the fourth feature information matrix are fused to obtain a sixth feature information matrix, and the fifth feature information matrix and the sixth feature information matrix are fused to obtain fused feature information.

[0194] The fusion method is:

[0195] The feature information matrices that need to be fused are X and Y, X is an n′×m′ dimensional matrix, Y is a p×m′ dimensional matrix, m′ represents the number of samples, n′ and p represent the dimensions of the two features, and the two matrices are projected to 1 dimension for linear representation, corresponding to the projection vectors a1 and b1 respectively, then the feature matrix after projection becomes:

[0196]

[0197] The purpose is to maximize the correlation coefficient between X' and Y', so as to obtain the projection vectors a1 and b1 when the correlation coefficient is maximum, that is:

[0198]

[0199] Before projection, the data is standardized. The purpose of standardization is to make the mean of the data 0 and the variance 1. We can get:

[0200] cov(X',Y')=cov(a1 T X,b1 T Y)=E(<a1 T X,b1 T Y>)=E((a1 T X)(b1 T Y) T )=a1 T E(XY T )b1

[0201]

[0202] D(Y′)=D(b1 T Y)=b1 T E(Y T Y)b1

[0203] D(X)=cov(X,X)=E(X T X)

[0204] D(Y)=cov(Y,Y)=E(Y T Y)

[0205] cov(X,Y)=E(XY T ),cov(Y,X)=E(YX T )

[0206] S XX =cov(X,X), then the solution objective is transformed into:

[0207]

[0208] Step 1: Calculate the variance S of X and YXX and S YY , the covariance S of XY and YX XY =S YX T ;

[0209] Step 2: Calculate the matrix

[0210] Step 3: Solve the singular values ​​of M′ to obtain the largest singular value and its preceding and following singular vectors u, v;

[0211] Step 4: The projection vectors a1 and b1 of X and Y are:

[0212]

[0213] Step 5: The fused feature information is

[0214] Using a preset fatigue detection and evaluation model, the fused feature information is processed to obtain driver fatigue level information;

[0215] The fatigue detection evaluation model is as follows: first, a classifier is trained using n1 fused feature information as training samples. The output categories of the classifier include level 1 warning, level 2 warning, level 3 warning, and no warning. Then, a fused feature information sample to be detected is used as an input vector with a length of d and input into the classifier. The classifier determines its category through calculation.

[0216] The m1 categories are represented by 1 to m1, and the encoding vectors corresponding to these categories are: When sample x belongs to the i′th category, then y (i′) =1, otherwise y (i′) =0; the training sample set consists of n1 training samples The probability that sample x belongs to category i′ is:

[0217]

[0218] where i′∈{1,…,m1}, ω (i′) is the weighted feature vector corresponding to category i′, with the superscript Τ is the transpose of the matrix;

[0219] Perform maximum likelihood estimation on ω:

[0220]

[0221]

[0222] Among them, p(ω) is a function of ω, expressed as:

[0223] p(ω)∝exp(-λ||ω||1)

[0224] where ||ω||1=∑ l |ω|,|ω| represents the absolute value, ∑ l |ω| means calculating the l1 norm, It means taking the maximum likelihood value of l(ω).

[0225] It can be seen that the present invention proposes a fatigue driving detection method and device, which mainly include three parts: facial static feature extraction, facial expression dynamic feature extraction, and fatigue state analysis and alarm. Facial static feature extraction is mainly to locate the driver's face by collecting 1 frame of image in the driver's front video stream, and finally locate the mouth and eyes and output the mouth and eye area images. Then, according to the driver's mouth and eye area images, their variances are calculated respectively, and the opening and closing states are judged to generate mouth and eye opening and closing state data streams, thereby forming various facial expression state frequency data. Finally, the driver's fatigue level is inferred, and a corresponding level of alarm prompt is issued. The facial feature extraction method of the present invention has a small amount of calculation and high accuracy, and can quickly obtain facial dynamic expression features, providing a reference for the detection of fatigue driving.

[0226] Embodiment 2

[0227] See also Figure 2 , Figure 2 is a flow chart of another fatigue driving detection method disclosed in an embodiment of the present invention. Figure 2 The described fatigue driving detection method is used to infer the driver's fatigue level and issue a corresponding level of warning prompt, which is not limited in the embodiments of the present invention. Figure 2 As shown, the fatigue driving detection method includes three parts: facial static feature extraction, facial expression dynamic feature extraction, and fatigue state analysis and warning.

[0228] Furthermore, the facial static feature extraction steps are as follows: collecting 1 frame of image from the driver's front video stream, locating the face based on the SSD deep neural network, locating the mouth based on the attention mechanism and outputting the mouth area image I m , eye location based on attention mechanism and output eye area image I e ,like Figure 3 The acquisition of one frame of image in the front video stream of the driver is to read one frame of image I0 from the video stream and record the acquisition time t0 of the current frame image; the face positioning based on the SSD deep neural network is to use the existing trained SSD deep neural network to recognize and locate the human head in image I0, and output the driver's head area image I h; The mouth location based on the attention mechanism and output of the mouth area image I m , is the driver's head area image I through the neural network of the attention mechanism h Identify and locate image I h The mouth area in the image is output as the driver's mouth area image I m The eye location based on the attention mechanism and output of the eye area image I e , is the driver's head area image I through the neural network of the attention mechanism h Identify and locate image I h The eye area in the image is outputted as the driver's eye area image I e .

[0229] Furthermore, the facial expression dynamic feature extraction, such as Figure 4 As shown, it is based on the driver's mouth area image Im and eye area image I e The variance is calculated respectively, and the opening and closing states are determined to generate the mouth and eye opening and closing state data streams, thereby forming various facial expression state frequency data. The specific steps are as follows: First, the mouth area image I is calculated m Variance DX m , determine the mouth opening and closing state, if the variance DX m >T m , then it is in the open state; if the variance DX m <T m , then it is in the closed state, where T m The threshold for determining the mouth opening and closing state is generally recommended to be 10 (the specific value can be determined based on the mouth area image I m The mouth closing state of the current frame is associated with the image acquisition time t0 to form a mouth opening and closing state data stream S with a time stamp. m Similarly, according to the eye area image I e , calculate and form the eye opening and closing state data stream S with time stamp e Secondly, according to the mouth opening and closing state data stream S m and eye opening and closing status data stream S e , calculate the frequency F of yawning, slow eye closing, blinking, and long-term unchanged line of sight (staring) respectively a 、F b 、F c 、F d The yawning expression is based on the mouth opening and closing state data stream S m The time of opening the mouth is calculated to determine whether it is greater than 3 seconds. If it is greater than 3 seconds, it means that the person is yawning. The time when the mouth starts to yawn is recorded as t ma1Similarly, determine and record the next yawning time as t ma2 , calculate and record the yawning state frequency F a =1 / (t ma1 -t ma2 )*60 times / minute; the slowly closing expression of the eyes, according to the eye opening and closing state data stream S e A judgment is made to calculate whether the eye closure time is greater than 2.5 seconds. If it is greater than 2.5 seconds, it means that the eyes are in a state of slow eye closure. The moment when the eyes start to close slowly is recorded as t mb1 Similarly, determine and record the time when the eyes slowly close next time as t mb2 , calculate and record the frequency F of the slowly closed eye expression state b =1 / (t mb1 -t mb2 )*60 times / minute; the blinking expression, eye opening and closing state data stream S e The eye closing time is calculated to determine whether it is less than 1 second. If it is less than 1 second, it means that the eye is blinking at this time, and the time when the eye starts to close after blinking is recorded as t mc1 Similarly, determine and record the time of the next blink as t mc2 , calculate and record the blinking state frequency F c =1 / (t mc1 -t mc2 )*60 times / minute; the expression of the state of sight without change (staring) for a long time is recorded through the data stream S of the eye opening and closing state e If the eye-open state lasts for more than 10 seconds, it is considered as a long-term unchanged line of sight (gaze) state, and the time of the most recent eye closure is recorded as t md1 Similarly, the time when the line of sight remains unchanged for a long time (gaze) is determined and recorded as t md2 , calculate and record the frequency of the state of long-term unchanged gaze (staring) F d =1 / (t md1 -t md2 )*60 times / minute.

[0230] Furthermore, the fatigue state analysis and warning is to infer the driver's fatigue level based on the facial expression state frequency data and issue a corresponding level of warning prompt. The driver's fatigue level FD is calculated according to the empirical FD calculation method as shown in the following formula:

[0231]

[0232] The corresponding level of alarm prompt is determined according to FD. FD>=0.9 is severe fatigue, and a level one alarm is issued; 0.9>FD>=0.7 is moderate fatigue, and a level two alarm is issued; 0.7>FD>=0.4 is mild fatigue, and a level three alarm is issued; 0.4>FD is slight fatigue, and no alarm is issued for the time being.

[0233] Embodiment 3

[0234] See also Figure 5 , Figure 5 : is a schematic diagram of the structure of a fatigue driving detection device disclosed in an embodiment of the present invention. Figure 5 The described fatigue driving detection device is used to infer the driver's fatigue level and issue a corresponding level of warning prompt, which is not limited in the embodiments of the present invention. Figure 5 As shown, the fatigue driving detection device may include the following operations:

[0235] S301, a data acquisition module, used to acquire a driver front image data set; the driver front image data set includes N driver front images and time information of the N driver front images being acquired, where N is a positive integer;

[0236] S302, a data set processing module, used to process the driver's front image data set to obtain a mouth opening and closing state data set and an eye opening and closing state data set;

[0237] S303, a first feature extraction module, configured to extract dynamic features from the mouth opening and closing state data set to obtain first feature information;

[0238] S304, a second feature extraction module, configured to extract dynamic features from the eye opening and closing state data set to obtain dynamic feature information; the dynamic feature information includes second feature information, third feature information and fourth feature information;

[0239] S305, a driver fatigue level determination module is used to process the first feature information and the dynamic feature information to obtain driver fatigue level information.

[0240] Embodiment 4

[0241] See also Figure 6 , Figure 6 : is a schematic diagram of another structure of a fatigue driving detection device disclosed in an embodiment of the present invention. Figure 6 The described fatigue driving detection device is used to infer the driver's fatigue level and issue a corresponding level of warning prompt, which is not limited in the embodiments of the present invention. Figure 6 As shown, the fatigue driving detection device may include the following operations:

[0242] A memory 401 storing executable program codes;

[0243] a processor 402 coupled to the memory 401;

[0244] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the fatigue driving detection method described in the first embodiment and the second embodiment.

[0245] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0246] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0247] Finally, it should be noted that the fatigue driving detection method and device disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting fatigue driving, characterized in that: The method comprises: S1, obtaining a driver front image dataset; the driver front image dataset includes N driver front images and time information of the N driver front images being captured, where N is a positive integer; S2, processing the driver's frontal image dataset to obtain a mouth opening and closing state dataset and an eye opening and closing state dataset; S3, extracting dynamic features from the mouth opening and closing state data set to obtain first feature information, including: S31, processing the mouth opening and closing state data set to obtain the opening time; S32, when the opening time is greater than 3 seconds, confirming that the current state is a yawning state; S33, recording the start times tma1 and tma2 of two adjacent yawning states; S34, processing the start times tma1 and tma2 of the two adjacent yawning states to obtain first feature information; The expression of the first characteristic information is: Fa=1 / (tma1-tma2)*60 times / minute Where Fa is the first feature information, * represents multiplication, and / represents division; S4, extracting dynamic features from the eye opening and closing state data set to obtain dynamic feature information; the dynamic feature information includes second feature information, third feature information and fourth feature information, including: S41, processing the eye opening and closing state data set to obtain the eye closing time; S42, processing the eye closing time to obtain second feature information and third feature information, including: S421, determining the eye closing time, and when the eye closing time is greater than 2.5 seconds, confirming that the current state is a slowly closing eye state; S422, recording the start times tmb1 and tmb2 of the slow closing states of two adjacent eyes; S423, processing the start times tmb1 and tmb2 of the two adjacent eyes in the slow closing state to obtain second feature information; The expression of the second characteristic information is: Fb=1 / (tmb1-tmb2)*60 times / minute Where, Fb is the second feature information, * represents multiplication, and / represents division; S424, determining the eye closing time, and when the eye closing time is less than 1 second, confirming that the current state is a blinking state; S425, recording the start times tmc1 and tmc2 of two adjacent blinking states; S426, processing the start times tmc1 and tmc2 of the two adjacent blinking states to obtain third feature information; The expression of the third characteristic information is: Fc=1 / (tmc1-tmc2)*60 times / minute Where, Fc is the third feature information, * represents multiplication, and / represents division; S43, processing the eye opening and closing state data set to obtain the eye opening state duration; S44, processing the duration of the eye-opening state to obtain fourth feature information, including: S441, determining the duration of the eye-opening state, and when the eye-opening state duration is greater than 10 seconds, determining that the current state is a staring state; S442, recording the start times tmd1 and tmd2 of two adjacent gaze states; S443, processing the start times tmd1 and tmd2 of the two adjacent gaze states to obtain fourth feature information; The expression of the fourth characteristic information is: Fd=1 / (tmd1-tmd2)*60 times / minute Wherein, Fd is the fourth feature information, * represents multiplication, and / represents division; S5, processing the first feature information and the dynamic feature information to obtain driver fatigue level information, including: Using a driver fatigue degree judgment model, the first feature information and the dynamic feature information are processed to obtain driver fatigue degree information; The driver fatigue degree judgment model is: In the formula, F a is the first feature information, F b is the second feature information, F c is the third feature information, F d It is the fourth characteristic information, FD is the driver's fatigue level information, FD ≥ 0.9 is severe fatigue, and a first-level warning is issued; 0.7 ≤ FD < 0.9 is moderate fatigue, and a second-level warning is issued; 0.4 ≤ FD < 0.7 is mild fatigue, and a third-level warning is issued; FD < 0.4 is weak fatigue, and no warning is issued for the time being.

2. The fatigue driving detection method according to claim 1, characterized in that: The processing of the driver's frontal image dataset to obtain a mouth opening and closing state dataset and an eye opening and closing state dataset includes: S21, performing face positioning on any one of the driver's frontal images in the driver's frontal image data set to obtain a driver's facial image; S22, performing mouth location on the driver's facial image to obtain an mouth area image; S23, performing eye positioning on the driver's facial image to obtain an eye region image; S24, judging the mouth opening and closing state of the mouth area image to obtain mouth opening and closing state information; S25, determining the eye opening and closing state of the eye region image to obtain eye opening and closing state information; S26, integrating the mouth opening and closing state information and the corresponding time information of the driver's front image acquisition to obtain a mouth opening and closing state data set; S27, integrating the eye opening and closing state information and the corresponding time information of the driver's front image acquisition to obtain an eye opening and closing state data set.

3. The fatigue driving detection method according to claim 2, characterized in that: The step of judging the mouth opening and closing state of the mouth region image to obtain the mouth opening and closing state information includes: S241, performing variance calculation on the mouth area image to obtain mouth variance information; S242, comparing the mouth variance information with a preset variance threshold, when the mouth variance information is greater than the preset variance threshold, the mouth opening and closing state information is an open state, otherwise the mouth opening and closing state information is a closed state.

4. A fatigue driving detection device, characterized in that: The device comprises: A data acquisition module is used to acquire a driver front image data set; the driver front image data set includes N driver front images and time information of the N driver front images being acquired, where N is a positive integer; A data set processing module, used for processing the driver's front image data set to obtain a mouth opening and closing state data set and an eye opening and closing state data set; The first feature extraction module is used to extract dynamic features from the mouth opening and closing state data set to obtain first feature information, including: Processing the mouth opening and closing state data set to obtain the opening time; When the opening time is greater than 3 seconds, it is confirmed that the current state is a yawning state; Record the start times tma1 and tma2 of two adjacent yawning states; Processing the start times tma1 and tma2 of the two adjacent yawning states to obtain first feature information; The expression of the first characteristic information is: Fa=1 / (tma1-tma2)*60 times / minute Where Fa is the first feature information, * represents multiplication, and / represents division; The second feature extraction module is used to extract dynamic features from the eye opening and closing state data set to obtain dynamic feature information; the dynamic feature information includes second feature information, third feature information and fourth feature information, including: Processing the eye opening and closing state data set to obtain eye closing time; The eye closing time is processed to obtain second feature information and third feature information, including: The eye closing time is judged, and when the eye closing time is greater than 2.5 seconds, it is confirmed that the current state is a slowly closing eye state; Record the start times tmb1 and tmb2 of the slow closing state of two adjacent eyes; Processing the start times tmb1 and tmb2 of the two adjacent eyes in the slow closing state to obtain second feature information; The expression of the second characteristic information is: Fb=1 / (tmb1-tmb2)*60 times / minute Where, Fb is the second feature information, * represents multiplication, and / represents division; Determining the eye closing time, and when the eye closing time is less than 1 second, confirming that the current state is a blinking state; Record the start times tmc1 and tmc2 of two adjacent blink states; Processing the start times tmc1 and tmc2 of the two adjacent blinking states to obtain third feature information; The expression of the third characteristic information is: Fc=1 / (tmc1-tmc2)*60 times / minute Where, Fc is the third feature information, * represents multiplication, and / represents division; Processing the eye opening and closing state data set to obtain the eye opening state duration; The duration of the eye-opening state is processed to obtain fourth feature information, including: Determining the duration of the eye-opening state, and when the eye-opening state duration is greater than 10 seconds, confirming that the current state is a staring state; Record the start time tmd1 and tmd2 of two adjacent gaze states; Processing the start times tmd1 and tmd2 of the two adjacent gaze states to obtain fourth feature information; The expression of the fourth characteristic information is: Fd=1 / (tmd1-tmd2)*60 times / minute Wherein, Fd is the fourth feature information, * represents multiplication, and / represents division; A driver fatigue level judgment module, used for processing the first feature information and the dynamic feature information to obtain driver fatigue level information, including: Using a driver fatigue degree judgment model, the first feature information and the dynamic feature information are processed to obtain driver fatigue degree information; The driver fatigue degree judgment model is: In the formula, F a is the first feature information, F b is the second feature information, F c is the third feature information, F d It is the fourth characteristic information, FD is the driver's fatigue level information, FD ≥ 0.9 is severe fatigue, and a first-level warning is issued; 0.7 ≤ FD < 0.9 is moderate fatigue, and a second-level warning is issued; 0.4 ≤ FD < 0.7 is mild fatigue, and a third-level warning is issued; FD < 0.4 is weak fatigue, and no warning is issued for the time being.

5. A fatigue driving detection device, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the fatigue driving detection method described in any one of claims 1-3.