Fatigue detection system, method, medium and detection equipment based on facial skeleton model

Abstract

The present invention discloses a driving fatigue detection system, method, medium and detection equipment based on a facial skeleton model, and relates to the field of image recognition technology. The obtained image is recognized by using a human posture detection model; the obtained recognition image is 3D skeletonized by a pinhole camera model, and the head posture features and facial features are extracted by using a face detection model, and fatigue features are calculated by using the extracted head and facial posture features; the obtained fatigue features are detected by using a classifier, and a fatigue detection model is obtained, and fatigue is graded and determined by using the fatigue detection model, and an APP is used to interact with the user. The present invention analyzes fatigue behavior, extracts fatigue recognition features, and obtains a fatigue detection model integrating multiple fatigue features by using a random forest algorithm. The entire system is implemented on a Raspberry Pi, and the development of the backend and the APP end is completed. The entire fatigue detection system has a high integrity.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition technology, and in particular relates to a fatigue detection system, method, medium and detection equipment based on a facial skeleton model. Background Art

[0002] At present, with the continuous development of the automobile industry, driving has become one of the main ways of travel. The car ownership rate of Chinese families has continued to rise, and more and more people have driver's licenses, which has brought about problems such as increased traffic congestion and frequent traffic accidents. The importance of traffic safety issues has continued to rise and has become one of the major threats to national life safety. According to relevant information, from 2010 to the present, the traffic accident rate, especially the probability of major traffic accidents, has increased every year, and the rate of increase has increased every year, which has brought serious challenges to people's traffic safety. According to statistics from my country's Transportation Bureau, nearly 30,000 people will die in traffic accidents every year, of which nearly 20% of traffic accidents are caused by or indirectly caused by fatigue driving. And according to surveys, fatigue driving accounts for a higher proportion of the causes of major traffic accidents.

[0003] Among the factors related to the causes of fatigue driving, the most common one is the long-term continuous driving time. Because when mild fatigue comes, the driver will not notice that his concentration has begun to decline to a large extent, and he will often only feel backache. This mild fatigue is often considered by the driver to be caused by incorrect sitting posture or other reasons, and the driver ignores his own fatigue. This is also one of the reasons why fatigue driving is difficult to detect and identify. When the driver enters a mild fatigue state, if he continues to drive without resting, it is easy to enter a deeper state of fatigue. Over time, the driver will experience drowsiness, trance, slow reaction and other problems, which are very likely to cause traffic accidents. Moreover, fatigue driving often occurs on highways and at night. Under the condition of high-speed driving, coupled with the low visibility at night, fatigue driving adds a layer of risk to high-speed driving at night, so fatigue driving is very likely to cause major traffic accidents.

[0004] Through the above analysis, the problems and defects of the prior art are as follows:

[0005] (1) Among the causes of major traffic accidents, traffic accidents caused by fatigue driving or indirectly caused by fatigue driving account for the largest proportion. To solve this problem, a system is needed that can detect driver fatigue in real time and remind the driver to stop and rest before an accident occurs. However, the current fatigue detection system still has many problems such as poor convenience, real-time performance, and accuracy.

[0006] (2) The wearable fatigue detection system in the prior art is not convenient to wear.

[0007] (3) The existing fatigue driving detection has many problems, such as the driver needs to wear special detection equipment, the process is cumbersome, the fatigue detection system equipment is too large and difficult to install, and the recognition rate of fatigue detection is not high.

[0008] The significance of solving the above problems and defects is:

[0009] The present invention aims to realize a fatigue detection system based on facial skeleton, and uses Raspberry Pi as a development and deployment platform. The depth image and RGB image are obtained through the Intel D435 depth camera, and the skeleton recognition of the human body is performed using the OpenPose human posture recognition model. The 3D coordinates of the key points of the human body are obtained by using the obtained depth information and combining the camera internal parameters. The posture features are extracted using the 3D coordinates, and the facial feature points are recognized using the OpenFace model, and the features of the mouth and eyes of the face are extracted. After the feature extraction is completed, the fatigue behavior is analyzed, and the fatigue recognition features are extracted. After comparing several machine learning algorithms, a random forest fatigue detection model based on the fusion of multiple fatigue features is trained using random forests. Finally, the back-end design and front-end APP development for the system are carried out to complete the implementation of the entire fatigue detection system. Summary of the invention

[0010] In order to overcome the problems existing in the related art, the embodiments disclosed in the present invention provide a fatigue detection system, method, medium and detection device based on a facial skeleton model.

[0011] The purpose of the present invention is to realize a fatigue driving detection system based on a facial skeleton model to detect the real-time situation of the driver. When the driver is driving, if the system finds that the driver has fatigue driving behavior, the system will issue an early warning after judgment and give the driver a reminder. The purpose of doing so is to prevent the occurrence of traffic accidents, effectively reduce the incidence of traffic accidents, and protect the traffic safety of drivers and passers-by. If an accident occurs, the traffic police can use this system to determine the cause of the accident, effectively solve the problem that fatigue driving is difficult to identify, and help reduce fatigue driving through legal means.

[0012] The present invention also aims to: (1) establish a human skeleton model. The depth information and key point information of the human body are extracted and combined, and the human skeleton model is obtained through the camera's internal parameters and coordinate conversion formula, and the human skeleton model is adjusted to complete the establishment of the human skeleton model.

[0013] (2) Extracting human body and facial features. The behavior of the head and the eyes and mouth are important features for detecting fatigue driving. Extracting their features can well reflect a person's fatigue level. By calculating the aspect ratio of the eyes and mouth, as well as the head forward angle, shoulder angle and other features, the driver's driving status can be well displayed, which is of great significance for fatigue detection.

[0014] (3) Training of the random forest fatigue detection model. Using the facial and head features obtained, fatigue features such as nodding frequency and longest head-down time are further extracted for the training of the fatigue detection model. The model and the selected features are then continuously adjusted to obtain a random forest fatigue detection model with good accuracy.

[0015] (4) Implementation of the fatigue detection system. The application scenario of this fatigue driving detection system is in the car. The entire system is completely built and deployed on the Raspberry Pi. It can detect fatigue in real time and upload the detection data to the cloud platform. The interactive interface between the back-end and front-end APP of the system is designed. When the system detects the driver's fatigue, it will pass the detection result to the APP through the interface. The mobile phone APP will give different reminders to the driver according to the level of fatigue.

[0016] The technical solution is as follows:

[0017] According to a first aspect of an embodiment disclosed in the present invention, a driving fatigue detection method based on a facial skeleton model is provided, comprising:

[0018] Step 1: Use the camera to obtain the driver's real-time RGB image and depth image, and use the OpenPose human posture detection model to recognize the obtained images;

[0019] Step 2: 3D skeletonize the obtained recognition image through the pinhole camera model, extract the head posture features and facial features using the OpenFace face detection model, and calculate the fatigue features using the extracted head and facial posture features;

[0020] Step three, use a classifier to detect the obtained fatigue features, obtain a fatigue detection model, use the fatigue detection model to grade fatigue, and use the APP to interact and remind the user.

[0021] According to a second aspect of the disclosed embodiment of the present invention, there is provided a fatigue detection system based on a facial skeleton model, comprising:

[0022] The video stream processing module is used to read real-time RGB and depth images through the Intel realsense D435 depth camera, detect the human body posture and facial feature point model, obtain the coordinates of the key points of the human body and the key points of the face, accelerate the model using the Intel neural acceleration computing stick, convert the caffe framework's CNN model into the IR model intermediate form (.xml and .bin files), and use the inference engine to complete its deployment. Through the internal parameters of the D435 camera, the pinhole camera model is used to convert the 2D coordinate points into 3D coordinate points, and the 18 body key points and 68 facial key points are used to extract the body and facial posture features;

[0023] The fatigue detection module is used to classify the fatigue state into two states: mild fatigue and severe fatigue. It counts the state of the eyes, mouth and head in a certain period of time, confirms the fatigue behavior characteristics, generates an 8-dimensional feature vector representing the fatigue state, and uses the random forest model to train a model for classifying the driver's state to complete the fatigue detection of the system.

[0024] The interactive module is used to complete the writing of the backend interface through the Flask backend framework and transmit the detection results to the mobile phone APP; the mobile phone APP includes two modules: fatigue display and fatigue reminder. The fatigue display module intuitively displays the driver's real-time fatigue through a dial, and the fatigue reminder module reminds or alarms the driver according to the results of detecting mild fatigue or severe fatigue, and warns of fatigue driving.

[0025] According to a third aspect of the disclosed embodiment of the present invention, there is provided a storage medium for receiving a program input by a user, wherein the stored computer program enables an electronic device to execute the driving fatigue detection method based on a facial skeleton model.

[0026] According to a fourth aspect of the disclosed embodiments of the present invention, there is provided a vehicle-mounted fatigue detection device based on a facial skeleton model, wherein the vehicle-mounted fatigue detection device based on a facial skeleton model executes the driving fatigue detection method based on a facial skeleton model.

[0027] The technical solution provided by the embodiments disclosed in the present invention may have the following beneficial effects:

[0028] The present invention provides a fatigue detection system and method based on facial skeleton. The system is developed and deployed with Raspberry Pi, which solves the convenience problem of traditional wearable devices. Real-time RGB images and depth images are obtained through Intel realsense cameras, the driver's posture is detected using the OpenPose posture recognition model, the 3D skeletonization of the human body is completed through the pinhole camera model, the facial features are extracted using the OpenFace model, the extracted head and facial posture features are used to calculate the features of fatigue detection, the fatigue detection model is obtained using a classifier, fatigue is graded and determined, and the interaction with the user is completed using an APP to remind the user.

[0029] The present invention solves the problem that the wearable fatigue detection system is inconvenient to wear. The system is developed and deployed using Raspberry Pi, which is very suitable for the small space inside the car. The facial skeleton model is used to identify the driver and perform graded fatigue detection on the driver.

[0030] Compared with the prior art, the advantages of the present invention further include:

[0031] The 2D key point coordinates obtained by the OpenPose posture recognition model are combined with depth information to complete the 3D skeletonization of the human body and the extraction of posture features.

[0032] The extraction of mouth and eye features is completed through the facial feature point model.

[0033] The fatigue behavior was analyzed, the features of fatigue identification were extracted, and the random forest algorithm was used to obtain a fatigue detection model that integrated multiple fatigue features.

[0034] The entire system was implemented on the Raspberry Pi, and the back-end and APP development were completed. The entire fatigue detection system has a high degree of integrity.

[0035] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0037] Figure 1 It is a flow chart of a driving fatigue detection method based on a facial skeleton model provided in an embodiment of the present invention.

[0038] Figure 2 It is a schematic diagram of a fatigue detection system based on a facial skeleton model provided in an embodiment of the present invention.

[0039] Figure 2 In: 1. Video stream processing module; 2. Fatigue detection module; 3. Interaction module.

[0040] Figure 3 It is a 3D human skeleton diagram provided by an embodiment of the present invention.

[0041] Figure 4 It is a facial feature point recognition map provided by an embodiment of the present invention.

[0042] Figure 5 These are the characteristic coordinates of the eye positions detected by the facial feature point model provided by the embodiment of the present invention. The facial feature point model can obtain 6 coordinate point diagrams for the detection of the eyes.

[0043] Figure 6 It is an interface diagram of height1, height2, and length obtained according to an embodiment of the present invention.

[0044] Figure 7 This is a blink action diagram recorded using the EAR value provided by an embodiment of the present invention.

[0045] Figure 8 It is a schematic diagram of the accuracy of the random forest model provided by an embodiment of the present invention.

[0046] Fig. 9 It is a schematic diagram of the architecture of a fatigue detection system based on a facial skeleton model provided in an embodiment of the present invention.

[0047] Fig.10 is a graph of the interface return results provided by an embodiment of the present invention. Fig.10 a) Normal driving; Fig.10 b) Mild fatigue; Fig.10 c) Severe fatigue. DETAILED DESCRIPTION

[0048] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0049] 1. The technical solution of the present invention is further described below in conjunction with specific embodiments.

[0050] Example

[0051] like Figure 1As shown, according to an embodiment disclosed in the present invention, a driving fatigue detection method based on a facial skeleton model is provided, comprising:

[0052] S101, obtaining a real-time RGB image and depth image of the driver through a camera device, and recognizing the obtained image using an OpenPose human posture detection model;

[0053] S102, 3D skeletonizing the obtained recognition image by using a pinhole camera model, extracting head posture features and facial features by using an OpenFace face detection model, and calculating fatigue features by using the extracted head and facial posture features;

[0054] S103, using a classifier to detect the obtained fatigue features, obtaining a fatigue detection model, using the fatigue detection model to grade fatigue, and using an APP to interact and remind the user.

[0055] like Figure 2 As shown, according to an embodiment disclosed in the present invention, a fatigue detection system based on a facial skeleton model is provided, comprising:

[0056] Video stream processing module 1 is used to read real-time RGB and depth images through the Intel realsense D435 depth camera, detect using the human posture and facial feature point model, obtain the coordinates of the key points of the human body and the key points of the face, accelerate the model using the Intel neural acceleration computing stick, convert the caffe framework's CNN model into an IR model intermediate form (.xml and .bin files), and use the inference engine to complete its deployment. Through the internal parameters of the D435 camera, the pinhole camera model is used to convert the 2D coordinate points into 3D coordinate points, and the 18 body key points and 68 facial key points are used to extract the body and facial posture features;

[0057] Fatigue detection module 2 is used to classify fatigue status into two states: mild fatigue and severe fatigue. It counts the state of eyes, mouth and head in a certain period of time, confirms the fatigue behavior characteristics, generates an 8-dimensional feature vector representing the fatigue state, and uses the random forest model to train a model for classifying the driver's state to complete the fatigue detection of the system.

[0058] Interaction module 3 is used to complete the writing of the backend interface through the Flask backend framework and transmit the detection results to the mobile phone APP; the mobile phone APP includes two modules: fatigue display and fatigue reminder. The fatigue display module intuitively displays the driver's real-time fatigue through a dial, and the fatigue reminder module reminds or alarms the driver according to the results of detecting mild fatigue or severe fatigue, and warns of fatigue driving.

[0059] 2. The technical solution of the present invention is further described below in conjunction with the acquisition and analysis of the skeleton model.

[0060] The depth information of the image is obtained through the Intel realsense D435 depth camera, and combined with the coordinates of the key points of the human body obtained by the OpenPose model. The camera intrinsic parameters of D435 are used to obtain the 3D key point coordinates of the human body through the pinhole camera model, complete the 3D skeletonization of the human body, and calculate the posture characteristics of the human body.

[0061] 2.1 Acquisition of Depth Image

[0062] Depth images are different from RGB images. They contain the distance information between the camera and the object. They are single-channel images, while the pixel value of RGB represents the brightness of a certain pixel. They are three-channel images and are depth images obtained through Intel D435. Depth images can be used to reconstruct scenes, restore the surface shape and three-dimensional structure information of objects, and perform object recognition. The more well-known kinect and Intel realsense series for obtaining depth images are based on structured light for depth image recognition. The basis of 3D structured light is infrared light. It is equipped with a dot matrix transmitter and emits infrared rays to determine the distance of objects. Since the principle used is infrared rays, the depth image is not greatly affected by light, and it still has good results in dark conditions.

[0063] The present invention uses Intel's D435 camera to complete the acquisition of RGB images and depth images, aligns the depth image and RGB image through timestamps, and obtains real-time data. Intel's D415 and D435 are both professional depth cameras, which can achieve the acquisition of depth images and 3D reconstruction while collecting RGB images. According to previous tests, compared with D415, D435 obtains more complete depth points of pictures and fewer missing depth points, so the present invention uses D435 depth camera to complete the acquisition of depth data. The use of realsense series cameras requires the installation of their corresponding SDK and the corresponding configuration work. The config is configured and the collected depth image and color image are obtained by streaming through pipe. The pixel ratio of the image obtained is set to 480*640, and the fps is 30. The depth image and color image are aligned through timestamps, and the frames with lost depth information are supplemented and discarded to complete the real-time data acquisition work.

[0064] 2.2 Obtaining the coordinates of key points of the human body

[0065] OpenPose is the basis of this facial skeleton-based fatigue detection system and is also a very meaningful project. It is a mature model for posture recognition developed by CMU. After deploying OpenPose, the present invention can easily recognize posture behaviors such as human body movements and gestures. It is based on the CNN convolutional neural network built on the Caffe framework, which can input pictures and videos, and can also perform real-time detection of the camera's video stream to obtain a two-dimensional skeleton result of the human body, with extremely high accuracy and robustness.

[0066] The specific implementation process of OpenPose posture recognition:

[0067] (1) OpenPose first identifies all possible key points in the image and outputs them through feature maps. It then outputs the possible connections between the key points in the form of PAFS, which is called joint affinity field. These operations are performed simultaneously, which means that OpenPose completes key point detection and connection at the same time.

[0068] (2) The first 10 layers of VGGNet are used to extract features, and the heatmap and pafs prediction network are used. The network has two levels in total, each of which is continuously fine-tuned in 6 stages, and featuremaps are generated and fused in each stage.

[0069] (3) After the fusion is completed, the confidence map will be in a 2D form. If it is an ideal state, there should be only one peak in the confidence map if there is only one person in the image. When there are multiple people in the image, there will be a peak in the corresponding confidence map for each visible key point of each person. Especially when performing multi-person pose estimation, there may be many possible relationships between the key points in the above figure. There are PAFS between all detected human bones to estimate the possible connection between two joints.

[0070] (4) In this way, there will be many connection situations for key points. OpenPose uses the affinity field component of PAFS and combines the posture form of the human body to maintain the relative relationship between each torso and ensure that the connected two-dimensional skeletonization result conforms to the posture characteristics of the human body.

[0071] 2.3 3D human skeletonization

[0072] The OpenPose human posture detection model can be used to identify 18 joints of the human body. Each ID corresponds to a joint of the body, including: nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right foot, left hip, left knee, left foot, right eye, left eye, right ear, and left ear. Since the application scenario of the fatigue detection system is in the car, the cab space is too small, and the driver is too close to the camera, the image collected by the camera loses the detection of the feature points of the lower body, and only retains the feature points of the left ear, right ear, nose, neck, left eye, right eye, left shoulder, and right shoulder. The present invention can use the feature points of the driver's upper body to identify its posture.

[0073] When using the D435 depth camera for detection, you can get a frame of aligned RGB and depth images at the same time. Using the depth information detected by the depth camera, the 2D coordinates of the key points of the human body identified by OpenPose and the depth information of the corresponding position are combined to get the triplet of the key points of the human body. The first two elements of each triplet represent the x and y coordinates of the key point of the human body, and the third element represents the depth information corresponding to this position. If each value of this triplet is -1.00, it means that OpenPose lost the detection of this key point when detecting this frame of image, so all of them are set to -1.00 for representation.

[0074] The 2D key point coordinates obtained by the posture recognition model and the depth information collected by the depth camera are saved. The data can be converted into three-dimensional coordinate points through the camera internal parameters of D435, so that more realistic human posture information can be obtained. The camera's changing internal parameters are: camera_factor, camera_cx, camera_cy, camera_fx and camera_fy. The transformation mainly uses the pinhole camera model to restore the captured objects into 3D through depth and 2D coordinates.

[0075] The converted 3D key point coordinates of the human body are saved. Compared with 2D coordinates, 3D coordinates are more accurate for posture recognition and can identify more features. The coordinate information of 18 3D key points of the human body after conversion can be obtained.

[0076] In order to make the 3D skeleton diagram of the human skeleton diagram more convenient to view, the skeleton diagram needs to be rotated to a certain extent, and the 3D skeletonized figure is rotated to the front. Such rotation does not change the relative position of the key points of the human body, so it has no effect on the solution of the posture feature of the present invention, and does not affect the angle between the bones and the posture of the human body. First, starting from the center point between the buttocks and the neck of the human body, the center point is translated to the origin, and then rotated around the three axes of x, y, and z on the basis of this origin. The rotation matrix formula for rotating around the axis is as follows. Formula (2-1), formula (2-2), and formula (2-3) are the formulas for counterclockwise rotation around the x-axis, y-axis, and z-axis, respectively.

[0077]

[0078]

[0079]

[0080] By performing relative translation and rotation operations on the skeleton image, a 3D human skeleton image is obtained. Figure 3 shown.

[0081] 2.4 Extraction of head posture features

[0082] After obtaining the 3D skeleton coordinates of the human body, according to the needs of posture recognition of the present invention, the posture features of the human body are extracted as a preparation for the subsequent training of the random forest fatigue detection model. The main features to be obtained and their definitions are as follows:

[0083] Definition 1: The flexion angle α1 is defined as the angle between the vector formed by the neck and nose and the normal line of the y-axis. This feature is used to detect frequent nodding and long-term head-down movements that often occur when fatigue occurs. The calculation formula is shown in formula (2-4):

[0084]

[0085] Definition 2: The shoulder angle α2 is defined as the angle between the neck and the two vectors of the left and right shoulders. Severe fatigue is often accompanied by a large left and right head tilt. This feature is used to detect the head tilt action of a person when fatigued. The calculation formula is shown in formula (2-5):

[0086]

[0087] Definition 3: The shoulder-neck ratio RATE is defined as the ratio of the width of the shoulder to the length of the neck. The detected shoulder width will not change significantly, while the length of the neck will fluctuate significantly due to the head-down action. This ratio is used to represent the driver's head-down action when he is tired. The calculation formula is shown in formula (2-6):

[0088]

[0089] Faming mainly analyzed the depth image and the OpenPose human posture recognition model. The depth information was extracted through the D435 camera, combined with the coordinates of the key points of the human body obtained by the OpenPose model, and the camera's internal parameters and the pinhole camera model were used to complete the transformation from the 2D key point coordinates of the human body to the 3D key point coordinates. The three-dimensional skeleton diagram of the human body was generated and the three-dimensional skeleton diagram was rotated to make the skeleton diagram more intuitive and clear. The obtained 3D coordinates were used to extract the posture features of the human body, including the forward lean angle, shoulder angle, etc., according to the needs of the posture recognition of the present invention, as preparation for the subsequent training of the random forest fatigue detection model.

[0090] 3. The technical solution of the present invention is further described below in conjunction with the extraction and analysis of facial features.

[0091] After comparing multiple face detection models, the present invention selects the OpenFace model for face recognition, converts and accelerates the model using OpenVINO, and deploys the model on the Raspberry Pi. The relative coordinates of 68 feature points of the face are obtained through detection, and the features of the eyes and mouth are extracted using the coordinates of the facial feature points.

[0092] 3.1 Extraction of facial feature points

[0093] The recognition rate of the algorithm is relatively high now, but some situations still need to be solved: in any given image, the location of the face is uncertain and can be anywhere, which makes detection very difficult. There are two mainstream processing methods, one is to change the sliding window, and the other is to classify the area by the similarity of pixels. The first method is a bit like a brute force solution, and the second method has certain optimizations. So will there be a better solution to this problem? We are still looking forward to it, and we will continue to pay attention to this topic. In normal face detection, it can be assumed that the face is facing the device, but in actual applications, it may be tilted, or some images are blocked by hands, hats, etc., which is also difficult to detect. Now there is a face alignment to solve the image perspective problem, but once it exceeds a certain perspective, it is difficult to solve. Infrared can only be said to solve the problem of occlusion caused by wearing sunglasses.

[0094] The facial feature point extraction attempt selected two models: the OpenFace model based on convolutional neural network and the facial feature point model based on SVM+Hog. The two models have their own advantages. The facial feature point model based on SVM can reach 20 frames per second, with good real-time performance. However, in complex driving scenes, factors such as light will affect the recognition of faces, and the accuracy is poor. The OpenFace model based on convolutional neural network performs better, but its real-time performance is worse than the former. Figure 4 shown.

[0095] The facial feature point model marks 68 feature points at key positions of the face, including key parts such as eyes, nose and mouth. In the fatigue detection system, the present invention mainly uses 12 key points of the left and right eyes and 20 key points of the mouth to extract features and detect the opening and closing of the eyes and mouth.

[0096] 3.2 Model transformation and reasoning

[0097] Since the application environment of the fatigue detection system is in the car, the system cannot be deployed and developed by computer, but Raspberry Pi must be used instead. Raspberry Pi is similar to a single-chip microcomputer. Although it is small in size, it can complete many functions. The Raspberry Pi 4B used in the present invention is a 32-bit ARM architecture with a storage space of 32GB. It does not have an image processing accelerator such as a GPU, so it is impossible to use the image processing completed by the Raspberry Pi itself, and the model needs to be accelerated and converted.

[0098] Intel NCS2 Neural Compute Accelerator Stick, referred to as VPU, is an acceleration device specially launched by Intel for deploying deep learning models and implementing AI algorithms on edge devices. It is compact in shape, similar to a USB flash drive, and is powered by a USB interface. It can be easily plugged into the Raspberry Pi for use. Currently, it only supports the Linux system, which is very compatible with the Raspberry Pi. The use of VPU in the present invention can greatly improve the processing performance of the neural network, reduce the time used to deploy the model, and make the AI ​​development of the present invention more convenient.

[0099] Before using the Intel Neural Compute Stick to accelerate the model, you need to use the OpenVINO component exclusive to the Neural Compute Stick to complete the model conversion and reasoning. OpenVINO is a Pipeline tool set developed by Intel for deep learning configuration. It can convert and accelerate common CNN networks. It also includes common toolkits such as OpenCV that are compatible with models trained by various open source frameworks. OpenVINO has various capabilities for online deployment of algorithm models. As long as you master this tool, you can easily and quickly deploy pre-trained models on Intel's CPU.

[0100] The present invention needs to deploy the caffe skeleton model of the human body and face on the Raspberry Pi, and first use the model optimizer of OpenVINO to convert it into an intermediate form of IR (xml and bin files), and then use the C++ interface provided by the reasoning engine of OpenVINO to read the network structure and weights, configure the input and output parameters, and complete the acceleration of the entire model. The model can be accelerated on the neural computing stick. Under the acceleration of the neural computing stick, the deployment of the model on the Raspberry Pi was successfully completed. Although the fps of real-time image processing on the Raspberry Pi using the model is still much lower than that of the computer, it can meet the needs of feature extraction.

[0101] 3.3 Extraction of facial features

[0102] According to the OpenFace face detection model, 68 feature points of the face are obtained to extract the eye and mouth features.

[0103] EAR (Eye Aspect Ratio) is a conceptual model for blink detection proposed in 2016. Based on the actual data set, using the facial key point detector as input, an effective blink detection algorithm is proposed. By locating the outline of the eye, the aspect ratio of the eye is obtained to estimate the state of the eye. It is a simple and efficient eye recognition method. It detects the closure of the human eye through the face 68 key point model, and uses the relative coordinates of each position of the eye to complete the calculation of the eye aspect ratio. Through a large number of experiments and tests, it is found that the eye aspect ratio can be used to detect the opening and closing of the eyes. When the eyes are open, the EAR value is stable in the range of 0.2 to 0.3. When the eyes are closed, the EAR value of this frame will suddenly drop below 0.1. When the eyes are opened again, the EAR value returns to about 0.2 again. Therefore, the present invention can detect blinks through a change process of the EAR value.

[0104] like Figure 5 As shown in the figure, the feature coordinates of the eye position detected by the facial feature point model. The facial feature point model can detect the eyes and obtain 6 coordinate points, namely P1 to P6. The coordinate points of the upper eyelid are P2 and P3, and the symmetrical coordinate points of the lower eyelid are P6 and P5. From the above figure, the present invention can know that the two heights of the eyes are height1 = P6-P2, height2 = P5-P3, and the width of the eyes is length = P4-P1. Figure 6 As shown, height1, height2, and length are obtained.

[0105] EAR represents the aspect ratio of the eye. The formula of EAR is shown in (3-1). p1 to p6 in the formula correspond to the six coordinates of the eye. In the formula, the present invention needs to use all six coordinates. This data processing method can avoid the impact on the final detection result due to a large length error of a group, and improve the accuracy of EAR calculation as much as possible.

[0106]

[0107] MAR represents the aspect ratio of the mouth. Similar to the calculation of the eye aspect ratio, the height and length are calculated using the detected mouth coordinate points to obtain the ratio of the two, which is used to estimate the degree of mouth opening and closing. The formula for MAR is shown in formula (3-2).

[0108]

[0109] Unlike EAR, the facial feature point model can detect 20 feature points for the feature point detection of the mouth. The detection of the upper lip is divided into the upper and lower parts of the lip. At the beginning, the calculation of the lip height is the coordinates of the feature points of the lower part of the upper lip and the upper part of the lower lip, which makes the height difference of the two feature point coordinates too close, so that the obtained MAR value often appears close to 0 or even negative, which is not conducive to data processing. After consulting the relevant MAR data for mouth detection, it is chosen to replace the feature points of the upper part of the upper lip and the lower part of the lower lip. After testing, when the mouth is closed, the height value is stable at about 15, and the MAR value is close to 0.1 to 0.2. When the mouth is open, the MAR value is stable between 0.6 and 0.7. There is a big difference in the MAR values ​​in the two states.

[0110] The present invention completes the extraction of facial features, uses OpenVINO to transform and deploy the facial feature point model, deploys the model on the Raspberry Pi, and uses the Intel Neural Compute Stick to accelerate the model, so that the facial skeleton model of the present invention can run on the Raspberry Pi. The coordinates of the facial feature points are used to complete the extraction of mouth and eye features, and the two main features of EAR and MAR are extracted to identify the behavior of the eyes and mouth, paving the way for the extraction of fatigue features and the training of fatigue detection models in the later stage.

[0111] 4. The technical solution of the present invention is further described below in conjunction with the identification of fatigue behavior.

[0112] After extracting the head and facial features, we analyzed the state of people when they are tired by consulting a large amount of information, obtained the behavioral characteristics of people when they are tired, extracted the features for fatigue identification from the set fixed time window, trained the collected labeled fatigue data, and tried two mainstream classification algorithms, SVM and random forest. After comparison and adjustment, we finally obtained a random forest fatigue detection model with higher accuracy.

[0113] 4.1 Extraction of fatigue features

[0114] Fatigue is a continuous process that gradually deepens. According to the survey of the current mainstream fatigue detection methods, fatigue is mainly classified into two states: mild fatigue and severe fatigue. For common fatigue detection algorithms, the detection of fatigue behavior is often regarded as a classification problem. By consulting relevant papers and materials and summarizing fatigue behavior, when the driver is awake, due to the need to wear a seat belt, the body posture and position are relatively fixed and will not change significantly. The driver's head keeps looking straight ahead for a long time, occasionally tilting his head to observe the surrounding situation, opening his mouth to speak, etc. The driver's overall posture is relatively fixed. When the driver is slightly fatigued, he will squint, the opening and closing of his eyes will decrease, and he will begin to yawn. When the driver enters severe fatigue, the driver will yawn frequently and persistently, squint or even close his eyes continuously, and his head will be greatly lowered and tilted.

[0115] Based on the three states of normal, mild fatigue and severe fatigue of the driver, and the large number of papers and materials consulted by the present invention, it is found that the conditions of the eyes, mouth and head all play a key role in fatigue recognition. Finally, the fatigue behavior characteristics are confirmed through analysis, and an 8-dimensional feature vector representing the fatigue state is generated. The physical meaning of each value is shown in Table 1.

[0116] Table 1 Fatigue behavior characteristics

[0117]

[0118]

[0119] PERCLOS in Table 1 was proposed by Carnegie Mellon University in the United States. It is the most widely recognized and accepted fatigue detection indicator in the industry. Through investigations by the US Highway Administration and tests on simulated fatigue driving, a comparison of mainstream fatigue characteristics was completed. Although these fatigue characteristics can predict fatigue to a certain extent, PERCLOS's performance is the best among these fatigue characteristics. The physical meaning of PERCLOS is the ratio of the time the eyes are closed to the time window within a certain time window. The degree of eye closure can be set artificially. There are usually several commonly used standards, such as 60% eye closure, 50% eye closure, etc. Figure 7 The figure of a blink action recorded with the EAR value shows the variation of the EAR value during a blink action. From the figure, the present invention can obtain that when the eyes are closed, the EAR value of this frame will suddenly drop from about 0.2 to below 0.1, and then quickly rise to above 0.2, completing a blink action. Therefore, the present invention selects when the EAR value is less than 0.1 to determine that this frame represents eye closure.

[0120]

[0121] The calculation formula of PERCLOS is shown in formula (4-1). The present invention counts the number of frames with EAR less than 0.1 in this time window, divides it by the size of this time window, and obtains the PERCLOS value in this time interval.

[0122] The calculations of the longest eye-closing time EAR_Time, the longest head-down time Nod_Time, the longest coughing duration Yawn_Time, and the longest head-tilting duration Left_Time are similar, all of which are calculated by setting a threshold value and determining the longest continuous number of frames exceeding the threshold value. For example, the longest coughing duration Yawn_Time is calculated by MAR (mouth aspect ratio). Through experiments and tests, the present invention can obtain that when the mouth is normally closed, the value of MAR is stable between 0.1 and 0.2, when speaking, the value of MAR is in the range of 0.25 to 0.3, and when yawning, the value of MAR is stable between 0.6 and 0.7. It can be seen from the MAR and EAR when the eyes are open and the mouth is open and when the eyes are closed and the mouth is closed that when the eyes are closed and the mouth is closed, the EAR of the left and right eyes is around 0.1, and the MAR of the mouth is around 0.1. When the eyes are open and the mouth is open, the EAR of the left and right eyes is between 0.3 and 0.4, and the MAR is between 0.5 and 0.6. The above data show that MAR can well represent mouth movements and make judgments about mouth behaviors.

[0123] For Nod_Count (nod count), Yawn_Count (yawn count), and Left_Count (head tilt count), the present invention analyzes this continuous action to understand the amplitude range and change process of this continuous action, thereby setting thresholds to calculate these fatigue characteristics. Take the yawn count as an example. The completion of a yawn action includes the process from MAR stabilization to MAR rise and then to MAR decline. According to the test and analysis, the present invention sets that when the MAR rises significantly in the two frames before and after and exceeds the threshold of 0.4, it is recorded as a yawn action. The calculation method of Nod_Count is similar to Left_Count. The cosine value of the forward tilt angle of the previous frame is recorded. When the absolute value of the cosine value of the forward tilt angle of the current frame differs by more than 0.3, it is considered that a nodding action has occurred, and Nod_Count is increased by 1.

[0124] Since fatigue behavior needs to be detected once within a time interval, by estimating the model detection speed and the effect to be achieved in the actual application of the present invention, it is guaranteed that the result of fatigue detection can be obtained in about 5 seconds. Therefore, the present invention sets the time window size of fatigue detection to 10 frames. After obtaining the coordinates and depth information of the key points of the human body for 10 frames, they are converted into 3D coordinates of the human body posture, and the key point coordinates of the face are obtained at the same time. The features of the body and face of each frame are calculated and uploaded to the cloud platform of the laboratory for storage. At the same time, data processing is performed to extract the fatigue driving behavior characteristics of the present invention. In order to prevent a certain fatigue action from running through the middle of the two time windows and causing errors in fatigue detection, the accuracy of the system of the present invention is improved as much as possible. A sliding window method is adopted. After repeated tests, three frames are selected as the interval of one sliding. Whenever the data of ten frames is detected, the first three frames of data in the queue are dequeued, and the next three frames of data are queued. The fatigue detection model of the present invention is continuously input for detection, and the fatigue detection results of the present invention are continuously obtained.

[0125] 4.2 Random Forest

[0126] Random forest is an ensemble learning concept that combines many decision trees together as a training model. For a new input data, the prediction result of the random forest model refers to putting the new data into all decision trees for judgment, outputting the predicted categories in turn, and then counting all categories, using a voting method, and taking the category with the most votes as the prediction result of the random forest.

[0127] If the random forest ignores its randomness, it is the same as the general decision tree model. Randomness refers to the fact that the training set composition of each decision tree in the random forest is arbitrary, and random sampling is used for the training set of each decision tree, and the samples will be put back after the training is completed. Secondly, randomness refers to the random selection of features for each decision tree. When training each decision tree, you can selectively select a part of the features for training without training all the features. Randomly select k feature values ​​from the feature list (there are m feature values ​​in total, m is greater than k), and start to repeatedly establish decision trees to obtain multiple different decision trees. When testing, put the test set into each decision tree to output its predicted results. Each tree obtains a predicted result, and the one with the highest proportion in the result is used as the final result of the entire random forest. This makes the random forest have a higher ability to resist overfitting, and it can also increase the importance of the features that the random forest training hopes to pay attention to, so as to facilitate the adjustment of the model.

[0128] Random forest has many advantages such as stability and resistance to overfitting. Its stability lies in that only when more than half of the decision trees are wrong, the final prediction result of random forest will be wrong. The emergence of new data points will only affect one decision tree, and have little impact on the entire random forest model. Strong resistance to overfitting means that even if the decision trees in the random forest are overfitted, the final result will not be affected much, because all decision trees vote to decide, thus reducing the risk of overfitting.

[0129] The random forest algorithm has high accuracy and robustness, and can adapt to training environments with smaller samples and has higher efficiency.

[0130] 4.2.1 Model Training

[0131] For the training of the random forest fatigue detection model, the sklearn (Scikit-Learn) library should be pre-installed. It is a Python-based algorithm learning library with a simple algorithm interface design. The present invention can easily pre-process data, train models, score models, and save them. Before installing it, you need to pre-install libraries such as numpy and scipy.

[0132] After installing the environment, start writing the code for the training model. First, load the training fatigue data. The collected data correspond to normal driving status, slightly fatigued driving status and severely fatigued driving status. The first step is to preprocess the data. Because the training of the fatigue model is supervised learning, the fatigue status should be manually marked on the collected fatigue data, where 0 represents normal, 1 represents mild fatigue, and 2 represents severe fatigue. After marking the data, because the saved data is preceded by a label with the data name, use regular expressions to separate the data into separate values ​​and store them in an array, and convert them into numpy format. Extract fatigue features once every ten data and store them as a new training data set. At this point, each data in the training set is a 9-dimensional feature vector, of which the first eight are the features of fatigue detection extracted from ten frames of images by the present invention, and the last data represents the state of fatigue.

[0133] After the data is prepared, the algorithm in the sklearn library can be used to train the fatigue detection model of the present invention.

[0134] First, the present invention uses the train_test_split function to select 73% of the data from the data set for training the random forest fatigue detection model of the present invention, and retains the remaining 27% of the data for testing after the model of the present invention is generated to verify the accuracy of the model trained by the present invention.

[0135] Then, the random forest classifier in the sklearn library is called, and the training data divided before the present invention is loaded to train the model. When the random forest classifier is generated, the present invention can set the parameters of the random forest for pre-pruning. The number of decision trees generated is set to 2000, the minimum number of samples contained in the decision tree leaf node is 20, the maximum depth of the decision tree is set to 7, and the maximum number of features considered in the generated decision tree is 5. By continuously adjusting these parameters, the random forest model of the present invention has higher accuracy.

[0136] 4.2.2 Model Testing

[0137] The present invention uses the previously divided 27% test set to test the random forest fatigue detection model of the present invention, and uses the accuracy_score() method to obtain the accuracy of the random forest fatigue detection model, such as Figure 8 The accuracy of the random forest model shows that the accuracy of the normal state, mild fatigue, and severe fatigue three-classification model is 82%. Call the feature_importances_ method to output the top five most important features for model classification, and observe which features have the greatest impact on the training of this model, so that the present invention can make adjustments when selecting features.

[0138] The obtained random forest model is exported and saved, and the real-time fatigue characteristics are input into the model. While obtaining the prediction results, the prediction probabilities for the three categories of normal driving, mild fatigue, and severe fatigue are output to pave the way for subsequent use.

[0139] 4.3 SVM Classifier

[0140] SVM is a traditional and efficient small sample machine learning method. The SVM algorithm can be used to easily implement reasoning, greatly simplifying classification and regression problems. Its prediction results rely on a small number of support vectors. The difficulty of the operation is affected by the number of support vectors and is almost unaffected by the dimension. This allows SVM to obtain good results even when the amount of training set data is small, and has good robustness.

[0141] 4.3.1 Model Training

[0142] The training of the SVM support vector machine fatigue detection model is similar to that of the random forest. The Python sklearn library also includes the implementation of SVM. The present invention is the same as the training of the random forest. Similarly, 73% of the data is used as the random forest fatigue detection model of the present invention for training, and the remaining 27% of the data is used to verify the accuracy of the model. Before training, the data needs to be loaded into the pandas DataFrame, the label index is reset, the training data is loaded, the SVM svc model is called, and the random forest fatigue detection model is trained. After the training, the model is saved.

[0143] 4.3.2 Model Testing

[0144] Using the 27% data previously segmented as the test set, the model is directly tested and the accuracy is output. It can be seen that the accuracy of the model trained by SVM is only 67%.

[0145] Re-split the test set, and control the division operation of the training set and the test set by setting the random number seed. If the random number seed is set to 0, each time a training and test set containing different data is obtained, and the model can be trained repeatedly using different training sets to obtain an average model accuracy. If it is set to 1, each time a training set and a test set containing the same data are obtained, which makes the present invention use a set of data for model training each time, and facilitates comparison of the accuracy differences of different algorithms for the random forest fatigue detection model. By setting the random number seed during data segmentation, under the same conditions, the two models mentioned above are used for training respectively, and the accuracy of the random forest is between 78% and 85, while the accuracy of the SVM is between 67% and 73%. It was found that the accuracy of the SVM is still lower than that of the random forest. After analysis, the random forest algorithm is more suitable for multivariate classification than the support vector machine. The random forest algorithm can handle high-dimensional data. Since the random forest can generate multiple decision trees and use voting to produce the final result, the numerous decision trees determine that each high-dimensional data will be considered, and the random forest can output important features for model classification and observe the importance of each feature for the training of this model. The random forest still maintains good accuracy when features are missing, and the processing is better.

[0146] Based on the head and facial posture features extracted earlier, the three states of fatigue, normal, mild fatigue, and severe fatigue, were used to count the states of the eyes, mouth, and head within a certain period of time, confirm the fatigue behavior characteristics, and generate an 8-dimensional feature vector representing the fatigue state. For the training of the random forest fatigue detection model, two classifiers, random forest and SVM support vector machine, were selected, and two machine learning methods were introduced. For the same training set and test set, training and testing were performed multiple times, and two different classification algorithms were compared. Finally, it was concluded that in the current data set, the performance of random forest is better, with an accuracy rate close to 80%.

[0147] 5. The technical solution of the present invention is further described below in conjunction with the implementation of a fatigue detection system.

[0148] The entire fatigue detection system is fully deployed on the Raspberry Pi, which uses the Raspberry Pi for image processing and algorithm implementation. The detected data is uploaded to the built cloud platform for storage, completing the construction of the system backend, and using the mobile phone APP to complete the interaction with the user and provide real-time feedback on the fatigue detection results.

[0149] 5.1 Overall structure of the system, such as Fig. 9 shown.

[0150] The facial skeleton-based fatigue detection system was developed and deployed using the Raspberry Pi 4B. The Raspberry Pi is a 32-bit Linux microcomputer. The size of the Raspberry Pi is similar to that of an ID card. It is lightweight and uses an SD card for storage. However, it has a complete range of functions. Its small size is very suitable for the application scenario of fatigue driving detection. However, due to its own computing and operating capabilities, as well as the design of the ARM architecture, it has great limitations on the development and application of programs.

[0151] The fatigue detection system based on facial skeleton can be roughly divided into three parts, namely the video stream processing layer, the fatigue detection layer, and the UI layer for interaction. The system first reads the real-time RGB and depth images through the Intel realsense D435 depth camera, uses the human posture and facial feature point model for detection, obtains the coordinates of the key points of the human body and the key points of the face, accelerates the model using the Intel neural acceleration computing stick, converts the caffe framework's CNN model into the IR model intermediate form (.xml and .bin files), and uses the inference engine to complete its deployment. Through the internal parameters of the D435 camera, the pinhole camera model is used to convert the 2D coordinate points into 3D coordinate points, and the 18 body key points and 68 facial key points are used to extract the body and facial posture features. By consulting a large amount of information and papers, fatigue states are classified into mild fatigue and severe fatigue. The state of eyes, mouth and head position within a certain period of time are counted, the fatigue behavior characteristics are confirmed, and an 8-dimensional feature vector representing the fatigue state is generated. A model for classifying driver states is trained using the random forest model. After a large number of tests, the accuracy of the model is about 80%. The fatigue detection module of the system is completed, and the back-end interface is written through the Flask back-end framework. The detection results are transmitted to the mobile phone APP. The mobile phone APP includes two modules: fatigue display and fatigue reminder. The fatigue display module intuitively displays the driver's real-time fatigue through a dial. The fatigue reminder module reminds or alarms the driver according to the results of detecting mild fatigue or severe fatigue, and warns of fatigue driving.

[0152] 5.2 System related information

[0153] Machine: Raspberry Pi 4B

[0154] System: Ubuntu 20.04

[0155] Camera: Intel realsense D435 depth camera

[0156] Acceleration stick: Intel compute stick2 neural computing stick

[0157] Programming language: C++, Python 3.6.1.

[0158] 5.3 System backend construction

[0159] 5.3.1 Implementation of backend interface

[0160] Because Python was used for fatigue feature extraction and fatigue model training, the backend framework used Python's Flask framework for the simplicity and ease of writing of the backend, making the entire system easier to integrate and shortening the development time.

[0161] Flask is an open source web framework with a good community atmosphere. You can find answers to almost all common configuration and usage questions related to Flask, which provides great convenience for backend development. Compared with other mainstream frameworks, it is flexible, lightweight, easy to use, and highly customizable. You can add new features as needed. Unlike the heavyweight framework Django, it does not provide modules for function expansion, such as orm models, user authentication mechanisms, admin background management tools and components, but retains the core functions of the web framework, which can achieve lightweight, free, and flexible customization, expand functions through a rich plug-in library, and develop multiple websites.

[0162] The backend interface design of this fatigue detection system does not require much data to be interacted with, so the design process is relatively simple. Only two interfaces need to be implemented for data transmission. Since the deployment of the posture recognition model and the extraction of features are implemented using C++ code, and the extraction and data processing of fatigue features and the training of the random forest fatigue detection model are all programmed in Python, it is necessary to design an interface to pass the feature information detected by the C++ module to Python for processing, and another interface to realize the interaction between the Raspberry Pi and the mobile phone. The mobile phone APP requests the backend interface of the fatigue detection system, and the backend returns the real-time detected fatigue status results to the APP for interaction with the user.

[0163] (1)Interface name: getdata

[0164] Function: APP request interface, return the fatigue result of detection

[0165] Request URL: / getdata

[0166] Request method: POST or GET

[0167] The return parameter table of the interface is shown in Table 2.

[0168] Table 2 Parameters returned by the getdata interface

[0169]

[0170] (2)Interface name: redirect

[0171] Function: Used to pass the detection results of the posture recognition module to the Python module

[0172] Request URL: / redirect

[0173] Request method: POST or GET

[0174] The request parameter table and return parameter table of the redirect interface are shown in Table 3 and Table 4 respectively.

[0175] Table 3 Parameters of redirect interface request

[0176]

[0177] Table 4 Parameters returned by the redirect interface

[0178]

[0179] 5.3.2 Backend interface testing

[0180] (1) Get the test results

[0181] The test case for getting results uses a GET request to return the test results. The prerequisite for this interface is that the fatigue detection system has been started, the system has identified the driver, and the test results have been obtained through the fatigue detection model. This use case is used to test possible return errors, as shown in Table 5.

[0182] Table 5 getdata interface test case

[0183]

[0184] The description of the test case for obtaining results is a detailed supplement to the test case for obtaining results, as shown in Table 6. In actual applications:

[0185] In the backend of the fatigue detection system, a global queue with a length of 5 is declared to store the results because of concerns that the speed of fatigue detection and the speed of APP requests may not match, resulting in errors when the results are returned on the APP side. This test case mainly tests when the fatigue results of the detection are different and the length of the queue is different.

[0186] Table 6 getdata interface test case description

[0187]

[0188]

[0189] (2) Data transmission test

[0190] The data transfer test case uses a GET request to pass the processed data to the fatigue detection module of Python. The prerequisite of this interface is that the fatigue detection system is turned on, the driver is detected, and the data of the corresponding time window size is obtained. This test case is used to test possible data transfer errors, as shown in Table 7.

[0191] Table 7 redirect interface test case

[0192]

[0193] The data transfer test case description is a detailed supplement to the data transfer use case, as shown in Table 8. In actual applications:

[0194] This test case is mainly aimed at, since the time window size of fatigue detection setting is 10, every time 10 data are obtained, the data needs to be passed to the fatigue detection module, so the data type and length of the passed data need to be checked.

[0195] Table 8 redirect interface test case description

[0196]

[0197]

[0198] 5.4 APP implementation

[0199] The application scenario of the fatigue detection system based on facial skeleton is in the car, using the camera to obtain real-time images, extracting features through the skeleton model, and putting them into the random forest fatigue detection model for detection. According to the different levels of fatigue detected, different feedback is given to the driver to achieve interaction with the user, which is used to warn and remind fatigue driving, eliminate the safety hazards of fatigue driving as much as possible, and ensure the safety of drivers. Based on the above purpose, in order to better interact with the driver and give fatigue reminders to users more conveniently, considering the application scenario in the car, it is not suitable to use the web side for feedback, so the mobile phone APP is used as the front end of this fatigue detection system to interact with the user and feedback the real-time fatigue status.

[0200] In the APP interface of the fatigue detection system, the APP is generally composed of three parts. Including the fatigue display module in the center of the interface, it uses a dashboard to provide real-time feedback on the driver's fatigue value based on the fatigue returned by the back end. The driver can intuitively see his fatigue and status during driving. Below the APP are two buttons, Start and Close. When the user clicks Start, the APP starts to continuously send requests to the back end to obtain the fatigue data detected by the back end. After testing, the speed of sending requests is basically consistent with the speed of generating back-end detection results. When the user clicks Close, the APP stops feedback on the driver's fatigue and no longer sends requests and fatigue warning functions.

[0201] There is a WIFI signal on the upper right corner of the APP. After the user clicks it, a pop-up window will appear, asking the user to enter the request address and port number used to return the detected fatigue data. The reason for designing this function is mainly due to the convenience of debugging the system. Since the test scenarios and situations often change, the Raspberry Pi is sometimes connected via a network cable and sometimes via WIFI. The local area network where the Raspberry Pi and the mobile phone are located often changes. In order to facilitate configuration when testing the system, the backend address and port of each request are set to manual input. This solves the cumbersome problem of changes in the IP every time the local area network is changed, making the present invention more convenient for system development and testing.

[0202] At the same time, in order to make the system more convenient to test, the back end of the system is deployed on the Raspberry Pi and the Alibaba Cloud server respectively, so that the Raspberry Pi detects fatigue characteristics and transmits them to the server, and the server detects fatigue behavior. The mobile phone APP can directly request the public network IP and interface of the server, and there is no need to put the Raspberry Pi and the mobile phone in a local area network. All mobile phones can directly request the server, and no WIFI connection is required, and the APP of the present invention can be used. At the same time, the Raspberry Pi can also be used as a hotspot, as a wireless transmission signal, and the back end is deployed on the Raspberry Pi, and the mobile phone is connected to the hotspot of the Raspberry Pi, which can also achieve the effect of being in a local area network, which is suitable for transferring the device to the car later. When the car passes a section with poor network or when the network signal is not good, the use of the server will definitely be greatly affected, and the Raspberry Pi can be used as a hotspot to solve this problem well and cope with the situation of poor network.

[0203] 5.5 System Demonstration and Testing

[0204] 5.5.1 System Display

[0205] Because the command of Raspberry Pi for facial skeleton recognition is too cumbersome, it is necessary to enter the running program, the model location used, the camera ID, and the name of the accelerator to start it. At the same time, in order to facilitate future deployment on the car, it can deal with people who cannot operate the Raspberry Pi in the car or are not familiar with the system. Therefore, a shell script is written to automatically start the fatigue detection of the Raspberry Pi when the Raspberry Pi is turned on, replacing the original cumbersome command input startup method. When the driver is ready to drive the car, he only needs to turn on the power of the Raspberry Pi and connect the mobile phone to the WiFi of the Raspberry Pi to start the fatigue detection system. No other redundant operations are required, nor is there any need to understand the system. It is very user-friendly and the practical applicability of the entire fatigue driving detection system has been greatly improved.

[0206] After the user turns on the Raspberry Pi, open the mobile app and connect to the Raspberry Pi's WiFi, then click the start button on the screen to begin fatigue recognition detection.

[0207] When the driver is in a normal state, the fatigue detection system detects the driver's driving behavior. The mobile APP continuously obtains the detection results by requesting the back-end interface. When it detects that the driver is driving normally, slightly fatigued, or severely fatigued, Fig.10 The interface returns the result, where: Fig.10 a) Normal driving; Fig.10 b) Mild fatigue; Fig.10 c) Severe fatigue.

[0208] The result is obtained by calling the getdata interface of the backend with postman. The result is expressed in json format, including fatigue rate, fatigue result, and status.

[0209] 5.5.2 Overall test of the system

[0210] Due to time constraints, the overall test of the system was completed in the laboratory. By simulating normal, mildly fatigued, and severely fatigued states during driving, the system's detection accuracy, real-time detection, and the reminder function of the APP were tested. This provided a comprehensive understanding of the overall functional completeness of the system and the possible problems with the fatigue detection system, which will facilitate further improvement of the system in the future and improve user satisfaction with the fatigue detection system.

[0211] When the mobile phone APP receives the normal driving result returned by the back-end interface, the fatigue dashboard displays the normal fatigue level of the driver (29%) at this time. The APP has no other response, and the driver is not driving fatigued and can continue driving.

[0212] When the mobile phone APP receives the mild fatigue result returned by the backend interface, the fatigue dashboard displays the mild fatigue level of the driver (61%), and the APP makes a "ding dong" sound and displays the orange words "mild fatigue status" to remind the driver. When the driver is mildly fatigued, there is a high probability that the fatigue will continue to increase. At this time, giving a moderate reminder can make the driver pay more attention to driving, be alert, and take a timely rest, which can have a good effect.

[0213] When the mobile APP receives the severe fatigue result returned by the backend interface, the fatigue dashboard displays the driver's severe fatigue level (88%) at this time, and the APP issues an alarm "Please note that you have entered fatigue driving, please take a rest in time", and displays the red words "Severe fatigue state" to warn the driver. When the driver is in severe fatigue, his line of sight is completely away from the road ahead. If he continues to drive the vehicle at this time, it is easy to cause a traffic accident, resulting in serious consequences. At this time, the driver should be given a serious warning, informing him that he is in a state of severe fatigue and must stop driving and take a rest.

[0214] The fatigue detection system has achieved overall functional implementation, but due to the computing power limitations of edge devices such as the Raspberry Pi, when the driver's body is greatly offset, the facial skeleton model will lose the detection of facial feature points. Although the facial features are lost at this time, due to the posture characteristics of the head, the severe fatigue state in this posture can still be identified and a fatigue driving alarm can be issued.

[0215] At the same time, since the input of the facial skeleton model is a three-channel RGB image, the camera cannot collect effective image information in the dark, so it is difficult to detect the driver's posture. An infrared camera can be used to obtain a single-channel driver image to detect fatigue in the dark.

[0216] The present invention completes the introduction of the entire fatigue detection system. The entire fatigue detection system is deployed on the Raspberry Pi, realizes the function of automatic startup, uses the Flask framework to complete the writing of the back-end interface, and uses the mobile phone APP to realize the interaction with the user. It can remind the user of voice and text according to the different driving fatigue states detected, and completes the reasonable interaction with the user. The problem of IP change is solved by using the server and the Raspberry Pi as a hotspot, making the use of the entire system more convenient. After the system is completed, the back-end interface and system are tested to ensure the smooth operation of the system.

[0217] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

[0218] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure should be limited by the appended claims.

Claims

1. A driving fatigue detection method based on a facial skeleton model, characterized in that: The driving fatigue detection method based on the facial skeleton model comprises: Step 1: Use the camera to obtain the driver's real-time RGB image and depth image, and use the OpenPose human posture detection model to recognize the obtained images; Step 2: 3D skeletonize the obtained recognition image through the pinhole camera model, extract the head posture features and facial features using the OpenFace face detection model, and calculate the fatigue features using the extracted head and facial posture features; Step 3: Use a classifier to detect the obtained fatigue features, obtain a fatigue detection model, use the fatigue detection model to grade fatigue, and use the APP to interact and remind the user; The step 2 of extracting head posture features using the OpenFace face detection model includes: Definition 1: The flexion angle α1 is defined as the angle between the vector formed by the neck and nose and the normal of the y-axis. This feature is used to detect frequent nodding and long-term head-down movements that often occur when fatigue occurs. The calculation formula is as follows: Definition 2: The shoulder angle α2 is defined as the angle between the neck and the left and right shoulder vectors. Severe fatigue is often accompanied by a large left and right head tilt. This feature is used to detect the head tilt action of a person when fatigued. The calculation formula is as follows: Definition 3: The shoulder-neck ratio RATE is defined as the ratio of the width of the shoulder to the length of the neck. The detected shoulder width will not change significantly, while the length of the neck will fluctuate significantly due to the head-down movement. This ratio is used to represent the driver's head-down action when he is tired. The calculation formula is as follows: The extraction of facial feature points comprises: According to the OpenFace face detection model, 68 facial feature points are obtained to extract eye and mouth features; Blink detection is performed through a change in EAR value: EAR represents the aspect ratio of the eye. The formula for EAR is As shown, p1 to p6 in the formula correspond to the six coordinates of the eye; MAR represents the aspect ratio of the mouth. Similar to the calculation of the aspect ratio of the eyes, the height and length are calculated using the detected mouth coordinate points, and the ratio of the two is obtained to estimate the degree of mouth opening and closing. The formula for MAR is as follows: The MAR value approaches 0.1 to 0.2, and the MAR value stabilizes at 0.6 to 0.7 when the mouth is open; The step 2 uses the extracted head and face posture features to calculate fatigue features including: When the EAR value is less than 0.1, it is determined that the frame represents eye closure; Count the number of frames with EAR less than 0.1 in this time window, and divide it by the size of this time window to get the PERCLOS value in this time interval; The longest eye closing time EAR_Time, the longest head lowering time Nod_Time, the longest yawn duration Yawn_Time, and the longest head tilt duration Left_Time are calculated by setting a threshold and measuring the longest continuous number of frames exceeding this threshold; the longest yawn duration Yawn_Time is calculated by MAR, when the mouth is closed normally, the MAR value is 0.1 to 0.2, when speaking, the MAR value is 0.25 to 0.3, when yawning, the MAR value is 0.6 to 0.7; when closing the eyes and mouth, the EAR of the left and right eyes is around 0.1, and the MAR of the mouth is around 0.1; when opening the eyes and mouth, the EAR of the left and right eyes is 0.3 to 0.4, and the MAR is 0.5 to 0.6; judge the behavior of the mouth based on the above data; For Nod_Count, Yawn_Count, and Left_Count, the continuous actions are analyzed to obtain the amplitude range and change process of the continuous actions, so as to set the threshold to calculate these fatigue characteristics; among them, Nod_Count is the nod count, Yawn_Count is the yawn count, and Left_Count is the head tilt count.

2. The driving fatigue detection method based on the facial skeleton model according to claim 1 is characterized in that: The step 1 of acquiring the depth image includes: Use the D435 depth camera to complete the acquisition of depth data. Configure the config and stream it through the pipe to obtain the acquired depth image and color image. Set the obtained image pixel ratio to 480*640 and fps to 30. Align the depth image and color image through the timestamp, complete and discard the frames with lost depth information, and complete the real-time data acquisition. Using the OpenPose human posture detection model to identify the obtained images includes: The first step is to obtain the coordinates of the key points of the human body: The CNN convolutional neural network built based on the Caffe framework performs real-time detection on the camera's video stream to obtain a two-dimensional skeleton image of the human body; The second step is to identify the two-dimensional skeleton image of the human body obtained in the first step: (1) First, identify all possible key points in the image and output them through feature maps. Then, output the possible connections between the key points in the form of joint affinity field (PAFS). (2) The first 10 layers of VGGNet are used to extract features, and then the heatmap and pafs prediction network are used. The network has two layers in total, each of which is fine-tuned in 6 stages, and feature maps are generated and fused in each stage. (3) After the fusion is completed, the confidence map is in a 2D form. If there is only one person in the image, the confidence map has only one peak. When there are multiple people in the image, each visible key point of each person has a peak in the corresponding confidence map. When performing multi-person pose estimation, there may be many possible relationships between the key points in the image. There are estimated PAFS between two joints between all detected human bones. (4) For key points, the affinity field components of PAFS are used in combination with the posture form of the human body to maintain the relative relationship between the various torsos.

3. The driving fatigue detection method based on the facial skeleton model according to claim 1 is characterized in that: The 3D skeletonization in step 2 includes: The OpenPose human posture detection model is used to identify the feature points of the driver's upper body for posture recognition; When using the D435 depth camera for detection, a frame of aligned RGB and depth images is obtained at the same time. Using the depth information detected by the depth camera, the 2D coordinates of the key points of the human body identified by the OpenPose human posture detection model are combined with the depth information of the corresponding position to obtain a triplet of the key points of the human body. The first two elements of each triplet represent the x and y coordinates of the key point of the human body, and the third element represents the depth information corresponding to this position. If each value of this triplet is -1.00, it means that when OpenPose detects this frame of image, it loses the detection of this key point, and all are set to -1.00 for representation; The 2D key point coordinates obtained by the posture recognition model and the depth information collected by the depth camera are saved. The data is converted into three-dimensional coordinate points through the camera internal parameters of D435 to obtain more realistic human posture information. The camera's internal parameters are: camera_factor, camera_cx, camera_cy, camera_fx and camera_fy. The pinhole camera model is used for transformation to restore the captured object into 3D through depth and 2D coordinates. The transformed 3D key point coordinates of the human body are saved to obtain the coordinate information of multiple 3D key points of the human body after the transformation; The obtained skeleton images of multiple 3D key point coordinate information of the human body are rotated to rotate the 3D skeletonized image to the front, specifically including: first starting from the center point between the hips and the neck of the human body, the center point is translated to the origin, and then the rotation is performed around the x, y, and z axes based on the origin. The rotation matrix formulas for rotating around the axes are as follows, which are the formulas for counterclockwise rotation around the x-axis, y-axis, and z-axis respectively; 4. The driving fatigue detection method based on the facial skeleton model according to claim 1 is characterized in that: The step three specifically includes: First, the training fatigue data is loaded. The collected data corresponds to normal driving state, slightly fatigued driving state and severely fatigued driving state, where 0 represents normal, 1 represents slightly fatigued, and 2 represents severely fatigued. After marking the data, the saved data is preceded by a label with the data name. The data is separated into individual values ​​using regular expressions and stored in an array, which is then converted into numpy format. Fatigue features are extracted every ten data items and stored as a new training data set. After preparing the data, the algorithm in the Python-based algorithm learning library is used to train the random forest fatigue detection model: the train_test_split function is used to select 73% of the data from the dataset for training the random forest fatigue detection model, and the remaining 27% of the data is retained; Then call the random forest classifier in the Python-based algorithm learning library, load the previously divided training data to train the model, set the parameters of the random forest for pre-pruning when generating the random forest classifier, and continuously adjust the parameters of the random forest.

5. A fatigue detection system based on a facial skeleton model for implementing the driving fatigue detection method based on a facial skeleton model according to any one of claims 1 to 4, characterized in that: The fatigue detection system based on the facial skeleton model is deployed on the Raspberry Pi, which uses the Raspberry Pi for image processing and algorithm implementation. The detected data is uploaded to the constructed cloud platform for storage, the system backend is built, and the mobile phone APP is used to complete the interaction with the user and provide real-time feedback on the fatigue detection results.

6. The fatigue detection system based on facial skeleton model according to claim 5, characterized in that: The fatigue detection system based on the facial skeleton model specifically includes: The video stream processing module is used to read real-time RGB and depth images through the Intel realsense D435 depth camera, detect the human body posture and facial feature point model, obtain the coordinates of the key points of the human body and the key points of the face, accelerate the model using the Intel neural acceleration computing stick, convert the caffe framework's CNN model into the IR model intermediate form .xml and .bin files, and use the inference engine to complete its deployment; through the internal parameters of the D435 camera, use the pinhole camera model to convert the 2D coordinate points into 3D coordinate points, and use the obtained 18 body key points and 68 facial key points to extract body and facial posture features; The fatigue detection module is used to classify the fatigue state into two states: mild fatigue and severe fatigue. It counts the state of the eyes, mouth and head in a certain period of time, confirms the fatigue behavior characteristics, generates an 8-dimensional feature vector representing the fatigue state, and uses the random forest model to train a model for classifying the driver's state to complete the fatigue detection of the system. The interactive module is used to complete the writing of the backend interface through the Flask backend framework and transmit the detection results to the mobile phone APP; the mobile phone APP includes two modules: fatigue display and fatigue reminder. The fatigue display module intuitively displays the driver's real-time fatigue through a dial, and the fatigue reminder module reminds or alarms the driver according to the results of detecting mild fatigue or severe fatigue, and warns of fatigue driving.

7. A storage medium for receiving a program input by a user, wherein the stored computer program enables an electronic device to execute the driving fatigue detection method based on a facial skeleton model as claimed in any one of claims 1 to 4.

8. A vehicle-mounted fatigue detection device based on a facial skeleton model, characterized in that: The vehicle-mounted fatigue detection device based on a facial skeleton model executes the driving fatigue detection method based on a facial skeleton model according to any one of claims 1 to 4.

Images