Fatigue driving recognition method based on facial features and related device

Abstract

The embodiment of the invention relates to the field of fatigue detection, and provides a fatigue driving recognition method based on facial features and a related device, and the method comprises the steps: collecting a driver image set of a driver during driving through a vehicle-mounted camera device; processing the driver images in the driver image set by using a multi-task neural network to obtain a head deflection angle and a face key point coordinate corresponding to each driver image; performing image cutting on corresponding driver images by adopting the face key point coordinates, performing state recognition on the driver according to a left eye image, a right eye image and a mouth image corresponding to each driver image, and obtaining a mouth state and an eye state corresponding to each driver image; and performing fatigue detection on the driver by adopting the head deflection angle, the mouth state and the eye state corresponding to each driver image to obtain the fatigue state of the driver, thereby realizing high accuracy and high robustness of real-time identification of the driving state of the driver.

Description

Technical Field

[0001] The present application relates to the technical field of fatigue detection, and in particular to a method and related device for identifying fatigue driving based on facial features. Background Art

[0002] With the rapid growth of my country's economy and the gradual improvement of the highway system, cars have now become the most commonly used means of transportation for Chinese citizens. Real-time and accurate recognition of driver fatigue characteristics is of great significance for reducing the occurrence of traffic accidents.

[0003] The current fatigue driving recognition methods are divided into three categories according to the characteristics of the judgment: recognition based on the driver's physiological signals, recognition based on vehicle features, and recognition based on the driver's facial features. The recognition method based on the driver's physiological signals is to obtain physiological signals such as telecardiogram, electroencephalogram and pulse in real time by wearing medical equipment, and analyze and identify the driver's fatigue state; although this type of method has high accuracy, it requires the driver to wear corresponding professional equipment and is highly invasive. The recognition method based on vehicle status features is to identify the driver's fatigue state by analyzing vehicle braking, lane deviation and speed changes, but these features are affected by the external environment, road conditions and the driver's operating level and habits, and the reliability is not high. The fatigue driving recognition method based on facial features uses computer vision methods to identify whether the driver's mouth, eyes and head have fatigue features to identify fatigue driving. This type of method has the advantages of good real-time performance, high accuracy, low invasiveness and low cost, but the existing methods have problems such as single features and poor robustness. Summary of the invention

[0004] The embodiment of the present application provides a method and related device for identifying fatigue driving based on facial features, which comprehensively analyzes the three major facial features of the driver's head, mouth and eyes, and achieves high accuracy and high robustness in real-time identification of the driver's driving status.

[0005] A first aspect of an embodiment of the present application provides a method for identifying fatigue driving based on facial features, the method comprising:

[0006] A driver image set of the driver while driving is collected by a vehicle-mounted camera device, wherein the driver image set is a set of driver images collected within a preset time interval;

[0007] The driver images in the driver image collection are processed by using a multi-task neural network of head posture estimation and facial key point detection to obtain the head deflection angle and facial key point coordinates corresponding to each driver image;

[0008] Using the facial key point coordinates to crop the corresponding driver images, obtaining a left eye image, a right eye image, and a mouth image corresponding to each driver image;

[0009] Inputting the left eye image, the right eye image and the mouth image corresponding to each driver image into a lightweight multi-classification neural network to perform driver status recognition, and obtaining the mouth status and eye status corresponding to each driver image;

[0010] The driver's fatigue state is detected by using the head deflection angle, mouth state and eye state corresponding to each driver image to obtain the driver's fatigue state.

[0011] Compared with the prior art, the present invention has the following advantages and beneficial effects: 1. The present invention uses a camera to collect the driver's image from the front angle for fatigue driving identification. The driver does not need to wear equipment and is not highly invasive. 2. The present invention utilizes the high correlation between head posture estimation and face key point detection tasks, introduces a multi-task learning mechanism, sets head posture estimation as the main task, and uses face key point detection to improve the accuracy of head posture estimation. 3. The present invention uses multiple indicators to comprehensively analyze the driver's fatigue state, and different indicators do not rely on a single network for identification. The present invention still has high generalization and robustness under complex environmental conditions. 4. The present invention uses a shared backbone network in the multi-task detection network and only adds a branch network form, and uses a lightweight network framework in the multi-classification neural network according to the task. Under the premise of ensuring recognition performance, it does not significantly increase the consumption of hardware resources and recognition time.

[0012] A second aspect of an embodiment of the present application provides a device for identifying fatigue driving based on facial features, the device comprising:

[0013] A collection unit, used to collect a set of driver images of the driver while driving through a vehicle-mounted camera device, wherein the set of driver images is a set of driver images collected within a preset time interval;

[0014] A processing unit, used to process the driver images in the driver image set by using a multi-task neural network of head posture estimation and facial key point detection to obtain a head deflection angle and facial key point coordinates corresponding to each driver image;

[0015] A cropping unit, used to crop the corresponding driver image using the facial key point coordinates to obtain a left eye image, a right eye image and a mouth image corresponding to each driver image;

[0016] A recognition unit, used for inputting the left eye image, the right eye image and the mouth image corresponding to each driver image into a lightweight multi-classification neural network to perform state recognition on the driver, and obtain the mouth state and eye state corresponding to each driver image;

[0017] The detection unit is used to perform fatigue detection on the driver using the head deflection angle, mouth state and eye state corresponding to each driver image to obtain the driver's fatigue state.

[0018] A third aspect of an embodiment of the present application provides a terminal, comprising a processor, an input device, an output device and a memory, wherein the processor, input device, output device and memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the step instructions in the first aspect of the embodiment of the present application.

[0019] The fourth aspect of the embodiments of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps described in the first aspect of the embodiments of the present application.

[0020] A fifth aspect of the embodiments of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiments of the present application. The computer program product may be a software installation package. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A flowchart of a method for identifying fatigue driving based on facial features is provided for an embodiment of the present application;

[0023] Figure 2 A schematic diagram of the positions of key points of a face is provided for an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a head deflection angle is provided for an embodiment of the present application;

[0025] Figure 4A schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0026] Figure 5 A structural schematic diagram of a facial feature-based fatigue driving identification device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0029] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0030] See also Figure 1 , Figure 1 A flowchart of a method for identifying fatigue driving based on facial features is provided for an embodiment of the present application. Figure 1 As shown, the method includes:

[0031] 101. Collect a set of driver images of a driver while driving using a vehicle-mounted camera device, where the driver image set is a set of driver images collected within a preset time interval.

[0032] The vehicle-mounted camera device may be installed on the dashboard. The front view image of the driver may be collected in real time to obtain a driver image set. The preset time interval may be a continuous time interval.

[0033] Specifically, a camera with a 1080P resolution and a frame rate of 30FPS is generally used and installed above the dashboard so that it can fully capture the driver's frontal perspective image.

[0034] 102. The driver images in the driver image set are processed using a multi-task neural network of head posture estimation and facial key point detection to obtain the head deflection angle and facial key point coordinates corresponding to each driver image.

[0035] The head pose estimation and face key point detection multi-task neural network (HFMNet) is as follows:

[0036] HFMNet selects 8,000 images from the open source dataset 300-LP and 2,000 images from the BIWI dataset as the training and test sets of the network, respectively. The annotation information of each image in the dataset includes 68 facial key points and the angles of the head deflection in three directions;

[0037] HFMNet consists of three parts: the backbone network, the face key point detection branch, and the head posture estimation branch. HFMN selects the first four segments of GhostNET as the backbone network to extract shared features. GhostNet can generate redundant feature maps through linear operations, thereby simplifying the feature extraction process. The extracted feature maps are then sent to the face key point detection branch and the head posture estimation branch respectively. The face key point detection branch and the head posture estimation branch both have the last segment of GhostNET for extracting specific features of the branch tasks. Finally, each will regress the shared features and specific features through the pooling layer and the fully connected layer to obtain the results of the tasks of each branch.

[0038] When estimating the entire head posture, HFMNet divides the angle interval [-90°, 90°] into 12 non-overlapping subintervals, each of which represents a range of 15 degrees; each interval is further divided into 3 equally divided intervals. Finally, by calculating the final expected value, the classification result is converted into a regression problem to perform head posture estimation. The head posture is calculated by the method shown in the following formula:

[0039]

[0040] where p k is the probability that the angle is in the kth large interval, is the probability that the angle is in the i-th small interval of the k-th large interval, α(i k ) is the representative value of the ith small interval of the kth large interval, and θ is the head deflection angle;

[0041] The sum of the losses of the two tasks of face key point detection and head pose estimation is the total loss of HFMN, and the loss function is:

[0042] Loss total =Loss HPE +φLoss FKD ,

[0043] Among them, Loss total is the total loss function of HFMN;

[0044] Loss HPE and Loss FKD They are the loss functions for the head pose estimation task and the face key point task, respectively, as follows:

[0045]

[0046] in are the predicted values ​​of the three head deflection angles, are the actual values ​​of the three head deflection angles, are the predicted value and actual value of the i-th key point of the face.

[0047] Specifically, Figure 2 As shown, Figure 2 A schematic diagram of the positions of key points of a human face is shown. The figure shows multiple key point labels, and the key point labels of the human face used subsequently are consistent with those in the figure. Figure 3 A schematic diagram of a head deflection angle is shown.

[0048] 103. Use the facial key point coordinates to crop the corresponding driver images to obtain a left eye image, a right eye image, and a mouth image corresponding to each driver image.

[0049] Among them, 6 key points of the left eye and the right eye and four key points of the mouth numbered 49, 52, 55 and 58 are selected;

[0050] For the left eye area, select the maximum value x among the x coordinates of the six key points max and the minimum value x min and the maximum value y in the y coordinate max and the minimum value y min , the center coordinates (x center ,y center ):

[0051]

[0052] Around the center coordinate, and The left eye image is cropped based on the length and width of the candidate area, and the right eye and mouth images are obtained using the same principle.

[0053] 104. Input the left eye image, right eye image and mouth image corresponding to each driver image into a lightweight multi-classification neural network to perform driver status recognition, and obtain the mouth status and eye status corresponding to each driver image.

[0054] The lightweight multi-classification neural network is an improved ShuffleNetV2 network:

[0055] The dataset of the improved ShuffleNetV2 network is 2500 mouth and eye pictures obtained from the Kaggle platform. The pictures are divided into four categories: eyes closed, eyes open, yawning, and not yawning according to the eye aspect ratio EAR and mouth aspect ratio MAR. Among them, there are 1316 pictures of eyes closed, 1184 pictures of eyes open, 1236 pictures of yawning, and 1264 pictures of not yawning. They are divided into training set and test set in a ratio of 7:1.

[0056] The eye aspect ratio EAR and mouth aspect ratio MAR calculation formula can be calculated by the method shown in the following formula:

[0057]

[0058] Among them, M i is the coordinate information of the i-th key point of the face, M 62 is the coordinate information of the 62nd key point of the face, M 38 is the coordinate information of the 38th key point of the face, M 42 is the coordinate information of the 42nd key point of the face, M 39 is the coordinate information of the 39th key point of the face, M 41 is the coordinate information of the 41st key point of the face, M 68 is the coordinate information of the 68th key point of the face, M 64 is the coordinate information of the 64th key point of the face, M 65 is the coordinate information of the 65th key point of the face, M 61 The coordinate information of the 61st key point of the face;

[0059] The ShuffleNetV2 network is mainly composed of 3 module layers composed of ShuffleNetV2 Unit2 and ShuffleNetV2Unit1;

[0060] The αH-swish activation function is used instead of the ReLU activation function used in the original ShuffleNetV2 network. α is a very small number. The αH-swish activation function is mathematically defined as follows:

[0061]

[0062] 105. Use the head deflection angle, mouth state and eye state corresponding to each driver image to perform fatigue detection on the driver to obtain the driver's fatigue state.

[0063] The fatigue status includes blinking frequency, longest eye closure time, yawning, nodding frequency and longest head lowering time. The specific judgment basis is as follows:

[0064] It usually takes 0.2-0.3 seconds for a driver to blink each time, so if the eyes change from open to closed to open in 6 consecutive frames, it is considered a blink.

[0065] If the eyes are identified as closed for more than 60 frames, it is identified as a long period of eye closure;

[0066] The driver usually keeps his mouth open in a yawning state for no more than 1 second, so 30 consecutive frames of the mouth being in a yawning state is considered one yawn;

[0067] The normal pitch angle range of the driver's head is 0-15°. Within 30 consecutive frames, if the head pitch angle changes from the normal range to greater than 15° and then returns to the normal range, it is considered a nod;

[0068] If the head pitch angle is greater than 15° for more than 90 frames, it is recognized as a long period of head lowering.

[0069] Furthermore, based on the fatigue level measurement scale in Table 1, when it is detected that the driver meets the standards of a certain fatigue level, it is determined to be fatigue driving to the corresponding degree.

[0070] Table 1 Fatigue level scale

[0071]

[0072] Therefore, after extracting the blinking frequency, the longest eye closing time, whether yawning, the nodding frequency and the longest head lowering time, the driver's fatigue state can be determined by looking up Table 1.

[0073] In a specific implementation, the present application embodiment also provides another method for identifying fatigue driving based on facial features, as follows:

[0074] Step 1: Install the vehicle-mounted camera above the dashboard to collect the driver's image from a positive angle in real time;

[0075] Specifically, a camera with a 1080P resolution and a frame rate of 30FPS is generally used and installed above the dashboard so that it can fully capture the driver's frontal perspective image.

[0076] Step 2: Use the head posture estimation and facial key point detection multi-task neural network to detect the data collected in step 1 to obtain the driver's head deflection angle and facial key point coordinates;

[0077] Specifically, the head pose estimation and face key point detection multi-task neural network (HFMNet) selects 8000 images from the open source dataset 300-LP and 2000 images from the BIWI dataset as the training set and test set of the network respectively. The annotation information of each image in the dataset includes 68 face key points and the angles of the head deflection in three directions. The schematic diagram of the position of face key points and the schematic diagram of the head deflection angle are shown in Figure 2. Figure 2 and Figure 3 shown.

[0078] Furthermore, HFMNet consists of three parts: the backbone network, the facial key point detection branch, and the head posture estimation branch. HFMN selects the first four segments of GhostNET as the backbone network to extract shared features. GhostNet can generate redundant feature maps through linear operations, thereby simplifying the feature extraction process. The extracted feature maps are then sent to the facial key point detection branch and the head posture estimation branch respectively. The facial key point detection branch and the head posture estimation branch both have the last segment of GhostNET to extract specific features for branch tasks. Finally, each will regress the shared features and specific features through the pooling layer and the fully connected layer to obtain the results of the tasks of each branch.

[0079] Furthermore, in the problem of head posture estimation, the head posture angle has the characteristics of continuous distribution. If direct regression is used, there is often a large difference between the predicted value and the true value. HFMNet transforms the entire head posture estimation from a regression problem to a classification problem, and divides the entire interval [90°, 90°] into 12 non-overlapping subintervals, each of which represents a range of 15 degrees. In order to further improve the accuracy, each interval is further divided into 3 equally divided intervals. Finally, the classification result is transformed into a regression problem by calculating the final expected value to perform head posture estimation. The calculation method is as shown in formula (1):

[0080]

[0081] where p k is the probability that the angle is in the kth large interval, is the probability that the angle is in the i-th small interval of the k-th large interval, α(i k ) is the representative value of the i-th small interval of the k-th large interval, and θ is the head deflection angle.

[0082] Furthermore, the sum of the losses of the two tasks of face key point detection and head pose estimation is the total loss of HFMN. In order to ensure that the head pose is the main task, a weight parameter φ is introduced into the loss function. The loss function is as follows:

[0083] Loss total =Loss HPE +φLoss FKD (2)

[0084] Among them, Loss total is the total loss function of HFMN;

[0085] LOss HPE and Loss FKD They are the loss functions for the head pose estimation task and the face key point task, respectively. The calculation formulas are as follows:

[0086]

[0087] in are the predicted values ​​of the three head deflection angles, are the actual values ​​of the three head deflection angles, are the predicted value and actual value of the i-th key point of the face.

[0088] Furthermore, the initial learning rate is set to 0.001, the batch size is set to 4, and a total of 100 iterations are trained. The learning rate is reduced by 1 / 10 every 10 iterations. The trained network is tested on the test set. The absolute errors of Yaw, Pitch and Roll are 4.39, 3.97 and 4.91 respectively, and the average absolute error of facial key points is 3.74, which proves the effectiveness of the network. HFMNet recognition results are shown in Figure 4 and Figure 5 .

[0089] Step 3: Based on the facial key point coordinate data obtained by the head posture estimation and facial key point detection multi-task neural network, locate and crop the driver's left eye, right eye and mouth images;

[0090] Specifically, select 6 key points for the left eye and the right eye, and four key points for the mouth labeled 49, 52, 55, and 58. For the left eye area, select the maximum value x among the x coordinates of the 6 key points. max and the minimum value x min and the maximum value y in the y coordinate max and the minimum value y min , the center coordinates (x) of the candidate region are calculated by equations (5) and (6) center ,y center ):

[0091]

[0092] Furthermore, around the center coordinate, and The left eye image is cropped based on the length and width of the candidate area, and the right eye and mouth images are obtained using the same principle.

[0093] Step 4: Input the eye and mouth images obtained in step 3 into a lightweight multi-classification neural network, identify the input into four categories: yawning, not yawning, eyes closed, and eyes open, and identify the state of the driver's mouth and eyes;

[0094] Specifically, 2500 mouth and eye images were obtained from the Kaggle platform. According to the eye aspect ratio EAR and mouth aspect ratio MAR, the images were divided into four categories: eyes closed, eyes open, yawning, and not yawning. Among them, 1316 eyes were closed, 1184 eyes were open, 1236 were yawning, and 1264 were not yawning. They were divided into training set and test set according to the ratio of 7:1. The calculation formula of eye aspect ratio EAR and mouth aspect ratio MAR can be expressed by formula (7) and formula (8):

[0095]

[0096]

[0097] Among them, M i is the coordinate information of the i-th key point of the face;

[0098] Furthermore, the lightweight multi-classification neural network is based on the ShuffleNetV2 network, which is mainly composed of three module layers composed of ShuffleNetV2 Unit2 and ShuffleNetV2 Unit1. The present invention connects an efficient channel attention module (ECA) to each ShuffleNetV2 Unit1 and Uint2 so that the network performs global average pooling without dimensionality reduction, thereby avoiding the adverse effects of dimensionality reduction on channel attention learning. The module is simple and has a small computational burden.

[0099] Furthermore, the original ShuffleNetV2 network uses the ReLU activation function, which is simple and efficient, but some function values ​​less than 0 are 0, which makes some neurons unable to be activated. The present invention uses the αH-swish activation function instead of the ReLU activation function, which not only keeps the simple and efficient characteristics of the activation function, but also solves the problem that the neurons cannot be activated due to the disappearance of the gradient when the input is less than -3. α is a very small number, and in the present invention, it is taken as 0.05. The mathematical definition of the αH-swish activation function is formula (9):

[0100]

[0101] Furthermore, the initial learning rate was set to 0.01, the batch size was set to 8, and a total of 50 iterations were trained, with the learning rate decreasing by 1 / 10 every 10 iterations. The trained network was tested on the test set with an accuracy of 0.973, proving the effectiveness of the network in eye and mouth posture classification.

[0102] Step 5: Based on the driver's facial features obtained in steps 2 and 4, statistically analyze the driver's blinking frequency, longest eye closure time, yawning, nodding frequency, and longest head lowering time over a period of time, and finally identify the driver's fatigue state based on the initial threshold set by multiple indicators.

[0103] Specifically, the results obtained in step 2 and step 4 are single-frame states. When a driver is driving normally, he or she may also have closed eyes or open mouth, so the results cannot be used as a basis for judgment. Therefore, statistical analysis of single-frame states is required to form features that can be used to identify fatigue driving, such as blinking frequency, yawning frequency, nodding frequency, and prolonged nodding or eye closure. The basis for judging each feature is as follows (the video frame rate is 30FPS):

[0104] (1) It usually takes 0.2-0.3 seconds for a driver to blink each time. Therefore, if the eyes change from open to closed to open in 6 consecutive frames, it is considered a blink.

[0105] (2) A driver usually does not yawn with his mouth open for more than 1 second, so 30 consecutive frames of yawning are considered one yawn;

[0106] (3) The normal pitch angle range of the driver's head is 0-15°. Within 30 consecutive frames, if the head pitch angle changes from the normal range to greater than 15° and then returns to the normal range, it is considered a nod;

[0107] (4) If the eyes are identified as closed for more than 60 frames, it is identified as a long period of eye closure;

[0108] (5) If the head pitch angle is greater than 15° for more than 90 frames, it is recognized as a long period of head bowing.

[0109] Furthermore, based on the fatigue level measurement scale in Table 1, when it is detected that the driver meets the standards of a certain fatigue level, it is determined to be fatigue driving to the corresponding degree.

[0110] For the above embodiments, please refer to Figure 4 , Figure 4A schematic diagram of the structure of a terminal provided in an embodiment of the present application, as shown in the figure, includes a processor, an input device, an output device and a memory, the processor, the input device, the output device and the memory are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, the processor is configured to call the program instructions, and the program includes instructions for executing the following steps;

[0111] A driver image set of the driver while driving is collected by a vehicle-mounted camera device, wherein the driver image set is a set of driver images collected within a preset time interval;

[0112] The driver images in the driver image collection are processed by using a multi-task neural network of head posture estimation and facial key point detection to obtain the head deflection angle and facial key point coordinates corresponding to each driver image;

[0113] Using the facial key point coordinates to crop the corresponding driver images, obtaining a left eye image, a right eye image, and a mouth image corresponding to each driver image;

[0114] Inputting the left eye image, the right eye image and the mouth image corresponding to each driver image into a lightweight multi-classification neural network to perform driver status recognition, and obtaining the mouth status and eye status corresponding to each driver image;

[0115] The driver's fatigue state is detected by using the head deflection angle, mouth state and eye state corresponding to each driver image to obtain the driver's fatigue state.

[0116] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process on the method side. It is understandable that in order to realize the above functions, the terminal includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0117] The embodiment of the present application can divide the terminal into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0118] In line with the above, see Figure 5 , Figure 5 The present invention provides a schematic diagram of a structure of a device for identifying fatigue driving based on facial features. Figure 5 As shown, the device comprises:

[0119] The acquisition unit 501 is used to acquire a driver image set of the driver while driving through a vehicle-mounted camera device, wherein the driver image set is a set of driver images acquired within a preset time interval;

[0120] A processing unit 502 is used to process the driver images in the driver image set using a multi-task neural network of head posture estimation and facial key point detection to obtain the head deflection angle and facial key point coordinates corresponding to each driver image;

[0121] A cropping unit 503 is used to crop the corresponding driver image using the facial key point coordinates to obtain a left eye image, a right eye image and a mouth image corresponding to each driver image;

[0122] The recognition unit 504 is used to input the left eye image, the right eye image and the mouth image corresponding to each driver image into a lightweight multi-classification neural network to perform state recognition on the driver, and obtain the mouth state and eye state corresponding to each driver image;

[0123] The detection unit 505 is used to perform fatigue detection on the driver using the head deflection angle, mouth state and eye state corresponding to each driver image to obtain the driver's fatigue state.

[0124] In a possible implementation, the head pose estimation and face key point detection multi-task neural network (HFMNet) is specifically as follows:

[0125] HFMNet selects 8,000 images from the open source dataset 300-LP and 2,000 images from the BIWI dataset as the training and test sets of the network, respectively. The annotation information of each image in the dataset includes 68 facial key points and the angles of the head deflection in three directions;

[0126] HFMNet consists of three parts: the backbone network, the face key point detection branch, and the head posture estimation branch. HFMN selects the first four segments of GhostNET as the backbone network to extract shared features. GhostNet can generate redundant feature maps through linear operations, thereby simplifying the feature extraction process. The extracted feature maps are then sent to the face key point detection branch and the head posture estimation branch respectively. The face key point detection branch and the head posture estimation branch both have the last segment of GhostNET for extracting specific features of the branch tasks. Finally, each will regress the shared features and specific features through the pooling layer and the fully connected layer to obtain the results of the tasks of each branch.

[0127] When estimating the entire head posture, HFMNet divides the angle interval [-90°, 90°] into 12 non-overlapping subintervals, each of which represents a range of 15 degrees; each interval is further divided into 3 equally divided intervals. Finally, by calculating the final expected value, the classification result is converted into a regression problem to perform head posture estimation. The head posture is calculated by the method shown in the following formula:

[0128]

[0129] where p k is the probability that the angle is in the kth large interval, is the probability that the angle is in the i-th small interval of the k-th large interval, α(i k ) is the representative value of the i-th small interval of the k-th large interval, and θ is the head deflection angle;

[0130] The sum of the losses of the two tasks of face key point detection and head pose estimation is the total loss of HFMN, and the loss function is:

[0131] Loss total =Loss HPE +φLoss FKD

[0132] Among them, Loss total is the total loss function of HFMN;

[0133] Loss HPE and Loss FKD They are the loss functions for the head pose estimation task and the face key point task, respectively, as follows:

[0134]

[0135] in are the predicted values ​​of the three head deflection angles, are the actual values ​​of the three head deflection angles, are the predicted value and actual value of the i-th key point of the face.

[0136] In a possible implementation, the clipping unit is specifically used for:

[0137] Select 6 key points of the left eye and the right eye, and four key points of the mouth numbered 49, 52, 55 and 58;

[0138] For the left eye area, select the maximum value x among the x coordinates of the six key points max and the minimum value x min and the maximum value y in the y coordinate max and the minimum value y min , the center coordinates (x center ,y center ):

[0139]

[0140] Around the center coordinate, and The left eye image is cropped based on the length and width of the candidate area, and the right eye and mouth images are obtained using the same principle.

[0141] In a possible implementation, the lightweight multi-classification neural network is an improved ShuffleNetV2 network:

[0142] The dataset of the improved ShuffleNetV2 network is 2500 mouth and eye pictures obtained from the Kaggle platform. The pictures are divided into four categories: eyes closed, eyes open, yawning, and not yawning according to the eye aspect ratio EAR and mouth aspect ratio MAR. Among them, there are 1316 pictures of eyes closed, 1184 pictures of eyes open, 1236 pictures of yawning, and 1264 pictures of not yawning. They are divided into training set and test set in a ratio of 7:1.

[0143] The eye aspect ratio EAR and mouth aspect ratio MAR calculation formula can be calculated by the method shown in the following formula:

[0144]

[0145] Among them, M i is the coordinate information of the i-th key point of the face;

[0146] The ShuffleNetV2 network is mainly composed of 3 module layers composed of ShuffleNetV2 Unit2 and ShuffleNetV2Unit1;

[0147] The αH-swish activation function is used instead of the ReLU activation function used in the original ShuffleNetV2 network. α is a very small number. The αH-swish activation function is mathematically defined as follows:

[0148]

[0149] In a possible implementation, the fatigue state includes blinking frequency, longest eye closure time, yawning, nodding frequency and longest head lowering time, and the specific judgment basis is as follows:

[0150] It usually takes 0.2-0.3 seconds for a driver to blink each time, so if the eyes change from open to closed to open in 6 consecutive frames, it is considered a blink.

[0151] If the eyes are identified as closed for more than 60 frames, it is identified as a long period of eye closure;

[0152] The driver usually keeps his mouth open in a yawning state for no more than 1 second, so 30 consecutive frames of the mouth being in a yawning state is considered one yawn;

[0153] The normal pitch angle range of the driver's head is 0-15°. Within 30 consecutive frames, if the head pitch angle changes from the normal range to greater than 15° and then returns to the normal range, it is considered a nod;

[0154] If the head pitch angle is greater than 15° for more than 90 frames, it is identified as a long period of head bowing. The present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables the computer to execute part or all of the steps of any facial feature-based fatigue driving identification method described in the above method embodiment.

[0155] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program enables a computer to execute part or all of the steps of any facial feature-based fatigue driving identification method recorded in the above method embodiments.

[0156] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0157] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0158] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical or other forms.

[0159] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] In addition, the functional units in the various embodiments of the application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software program modules.

[0161] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program codes.

[0162] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which can include: a flash drive, a read-only memory, a random access memory, a magnetic disk or an optical disk, etc.

[0163] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for identifying fatigue driving based on facial features, characterized in that: The fatigue driving identification method comprises: A driver image set of the driver while driving is collected by a vehicle-mounted camera device, wherein the driver image set is a set of driver images collected within a preset time interval; The driver images in the driver image collection are processed by using a multi-task neural network of head posture estimation and facial key point detection to obtain the head deflection angle and facial key point coordinates corresponding to each driver image; Using the facial key point coordinates to crop the corresponding driver images, obtaining a left eye image, a right eye image, and a mouth image corresponding to each driver image; Inputting the left eye image, the right eye image and the mouth image corresponding to each driver image into a lightweight multi-classification neural network to perform driver status recognition, and obtaining the mouth status and eye status corresponding to each driver image; The driver's fatigue state is detected by using the head deflection angle, mouth state and eye state corresponding to each driver image to obtain the driver's fatigue state.

2. The method for identifying fatigue driving based on facial features according to claim 1, characterized in that: The head posture estimation and face key point detection multi-task neural network HFMNet is as follows: HFMNet selects 8,000 images from the open source dataset 300-LP and 2,000 images from the BIWI dataset as the training and test sets of the network, respectively. The annotation information of each image in the dataset includes 68 facial key points and the angles of the head deflection in three directions; HFMNet consists of three parts: the backbone network, the face key point detection branch, and the head posture estimation branch. HFMN selects the first four segments of GhostNET as the backbone network to extract shared features. GhostNet can generate redundant feature maps through linear operations, thereby simplifying the feature extraction process. The extracted feature maps are then sent to the face key point detection branch and the head posture estimation branch respectively. The face key point detection branch and the head posture estimation branch both have the last segment of GhostNET for extracting specific features of the branch tasks. Finally, each will regress the shared features and specific features through the pooling layer and the fully connected layer to obtain the results of the tasks of each branch. HFMNet divides the angle interval [-90°, 90°] into 12 non-overlapping subintervals when estimating the entire head posture, each of which represents a range of 15 degrees; each interval is further divided into 3 equally divided intervals, and finally the classification result is converted into a regression problem by calculating the final expected value to estimate the head posture. The head posture is calculated by the method shown in the following formula: where p k is the probability that the angle is in the kth large interval, is the probability that the angle is in the i-th small interval of the k-th large interval, α(i k ) is the representative value of the ith small interval of the kth large interval, and θ is the head deflection angle; The sum of the losses of the two tasks of face key point detection and head pose estimation is the total loss of HFMN, and the loss function is: Loss total =Loss HPE +φLoss FKD , Among them, Loss total is the total loss function of HFMN; Loss HPE and Loss FKD They are the loss functions for the head pose estimation task and the face key point task, respectively, as follows: in are the predicted values ​​of the three head deflection angles, are the actual values ​​of the three head deflection angles, are the predicted value and actual value of the i-th key point of the face.

3. The method for identifying fatigue driving based on facial features according to claim 2, characterized in that: The corresponding driver images are cropped using the facial key point coordinates to obtain a left eye image, a right eye image and a mouth image corresponding to each driver image, including: Select 6 key points of the left eye and the right eye, and four key points of the mouth numbered 49, 52, 55 and 58; For the left eye area, select the maximum value x among the x coordinates of the six key points max and the minimum value x min and the maximum value y in the y coordinate max and the minimum value y min , the center coordinates (x center ,y center ): Around the center coordinate, and The left eye image is cropped based on the length and width of the candidate area, and the right eye and mouth images are obtained using the same principle.

4. The method for identifying fatigue driving based on facial features according to claim 3, characterized in that: In step 4, the lightweight multi-classification neural network is an improved ShuffleNetV2 network: The dataset of the improved ShuffleNetV2 network is 2500 mouth and eye pictures obtained from the Kaggle platform. The pictures are divided into four categories: eyes closed, eyes open, yawning, and not yawning according to the eye aspect ratio EAR and mouth aspect ratio MAR. Among them, there are 1316 pictures of eyes closed, 1184 pictures of eyes open, 1236 pictures of yawning, and 1264 pictures of not yawning. They are divided into training set and test set in a ratio of 7:

1. The eye aspect ratio EAR and mouth aspect ratio MAR calculation formula can be calculated by the method shown in the following formula: Among them, M i is the coordinate information of the i-th key point of the face; The ShuffleNetV2 network is mainly composed of 3 module layers composed of ShuffleNetV2 Unit2 and ShuffleNetV2 Unit1; The αH-swish activation function is used instead of the ReLU activation function used in the original ShuffleNetV2 network. α is a very small number. The αH-swish activation function is mathematically defined as follows:

5. The method for identifying fatigue driving based on facial features according to claim 4, characterized in that: The fatigue status includes blinking frequency, longest eye closure time, yawning, nodding frequency and longest head lowering time. The specific judgment basis is as follows: It usually takes 0.2-0.3 seconds for a driver to blink each time, so if the eyes change from open to closed to open in 6 consecutive frames, it is considered a blink. If the eyes are identified as closed for more than 60 frames, it is identified as a long period of eye closure; The driver usually keeps his mouth open in a yawning state for no more than 1 second, so 30 consecutive frames of the mouth being in a yawning state is considered one yawn; The normal pitch angle range of the driver's head is 0-15°. Within 30 consecutive frames, if the head pitch angle changes from the normal range to greater than 15° and then returns to the normal range, it is considered a nod; If the head pitch angle is greater than 15° for more than 90 frames, it is recognized as a long period of head lowering.

6. A fatigue driving identification device based on facial features, characterized in that: The device comprises: A collection unit, used to collect a set of driver images of the driver while driving through a vehicle-mounted camera device, wherein the set of driver images is a set of driver images collected within a preset time interval; A processing unit, used to process the driver images in the driver image set by using a multi-task neural network of head posture estimation and facial key point detection to obtain a head deflection angle and facial key point coordinates corresponding to each driver image; A cropping unit, used to crop the corresponding driver image using the facial key point coordinates to obtain a left eye image, a right eye image and a mouth image corresponding to each driver image; A recognition unit, used for inputting the left eye image, the right eye image and the mouth image corresponding to each driver image into a lightweight multi-classification neural network to perform state recognition on the driver, and obtain the mouth state and eye state corresponding to each driver image; The detection unit is used to perform fatigue detection on the driver using the head deflection angle, mouth state and eye state corresponding to each driver image to obtain the driver's fatigue state.

7. A terminal, characterized in that: The method comprises a processor, an input device, an output device and a memory, wherein the processor, the input device, the output device and the memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 5.

Images