Method for detecting and warning driver fatigue based on deep learning

Abstract

The application discloses a kind of detection and early warning method of driver fatigue driving based on deep learning, comprising the following steps: the head with the face of driver is carried out image acquisition by camera, and the image data collected is transferred to neural network model;The image collected is predicted by neural network model, and the head with the face of driver in image is segmented out;The face key point of driver in segmented image is detected and whether the face of driver is with sunglasses is judged by neural network model;When it is judged that driver is in severe fatigue state, mild fatigue state or non-fatigue state.The application can accurately judge whether driver is in fatigue state in real time, avoid driver into fatigue state, improve driving safety, and ensure the safety of highway vehicle driving.

Description

TECHNICAL FIELD

[0001] The present application relates to a driver fatigue driving detection and early warning method based on deep learning. BACKGROUND

[0002] The existing fatigue driving prevention means has its limitations, for example, the contact physiological parameter detection required by most fatigue driving detection equipment often needs invasive collection devices, which undoubtedly interferes with the normal driving behavior of the driver, brings new risks to safe driving, and the fatigue driving detection index tends to be single and not universal. For example, the driving habits of different drivers will affect the device that warns according to driving behavior. Therefore, we use the current computer vision, machine learning, and deep learning theory supported fatigue driving detection and recognition system to greatly improve the shortcomings of the existing technology.

[0003] The traditional fatigue driving detection system needs to rely on contact sensors, which to some extent brings new risks to driving, and causes uncomfortable driving feelings to the driver, and the detection index of fatigue driving is single and not universal, which reduces the detection accuracy to some extent. The technical solution adopts the theory of deep learning, relies on the vehicle-mounted camera for detection, solves the problem of relying on contact sensors, and has breakthroughs in the selection and determination of detection indexes, so that the detection accuracy is improved to some extent. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a driver fatigue driving detection and early warning method based on deep learning, which can quickly and accurately judge whether the driver is in a fatigue state in real time, avoid the driver entering a fatigue state, improve driving safety, and ensure the safety of highway vehicle driving.

[0005] The technical solution adopted by the present application to solve the above technical problems is:

[0006] A driver fatigue driving detection and early warning method based on deep learning, characterized in that it comprises the following steps:

[0007] Step 1: image acquisition of the head with the driver's face by the camera, and transmission of the collected image data to the trained neural network model;

[0008] Step 2: analyzing the collected image by the neural network model, and using image segmentation technology to segment the head region with the driver's face in the image;

[0009] Step 3, detecting the driver's face key points in the segmented image through the neural network model, and judging whether the driver wears sunglasses based on the features and shapes of the driver's eye region in the image;

[0010] Step 4, when it is judged that the driver does not wear sunglasses, if the driver's eye is detected to show fatigue features or both the eye and the mouth show fatigue features, it is considered that the driver is in a severe fatigue state, if only the driver's mouth is detected to show fatigue features, it is considered that the driver is in a mild fatigue state, and if neither the driver's eye fatigue features nor the mouth fatigue features are detected, it is considered that the driver is in a non-fatigue state (i.e. normal state);

[0011] When it is judged that the driver wears sunglasses, if the driver's head is detected to show fatigue features or both the head and the mouth show fatigue features, it is considered that the driver is in a severe fatigue state, if only the driver's mouth is detected to show fatigue features, it is considered that the driver is in a mild fatigue state, and if neither the driver's head fatigue features nor the mouth fatigue features are detected, it is considered that the driver is in a non-fatigue state.

[0012] According to the above technical solution, the neural network model comprises a first convolutional neural network model and a second convolutional neural network model;

[0013] In step 2, the first convolutional neural network model is used to predict the collected image, and the head part with the driver's face is segmented out and input to the second convolutional neural network model;

[0014] In step 3, the second convolutional neural network model is used to detect the driver's face key points in the segmented image and judge whether the driver wears sunglasses, and the state of the driver is judged through the image (the state includes a severe fatigue state, a mild fatigue state and a non-fatigue state).

[0015] According to the above technical solution, in step 4, when it is judged that the driver does not wear sunglasses, the specific detection process of whether the driver's eye shows fatigue features is as follows: K frames of images in the normal state of the driver are collected at the beginning of the driving process, and the average eye height of the driver is calculated to initialize the non-fatigue state eye height a; for the subsequent images, the eye height average b is calculated every m times of the time interval of the camera video collection, and clo=b / a; if the clo value measured for multiple times is less than the eye fatigue setting threshold W, it is considered that the eye shows fatigue features, otherwise, the case that the single clo is less than W is regarded as blinking or other temporary phenomenon, and is not regarded as a sign of fatigue.

[0016] According to the technical scheme, m is 0.2-0.6; K is 4-6, and the optimal selection is 5; if the clo value measured for 3-5 times continuously is less than the fatigue setting threshold W, the eye is considered to exhibit the fatigue feature; and the eye fatigue setting threshold W is 0.68.

[0017] According to the technical scheme, in the step 4, the highly specific calculation process for the eye is as follows: the eye height is calculated as the average difference between the upper eyelid key point position value and the lower eyelid key point position value in the driver's face image.

[0018] According to the technical scheme, in the step 4, the specific detection process for whether the driver's mouth exhibits the fatigue feature is as follows: the mouth opening degree average value in the collected video frame is calculated, and the mouth opening degree greater than the mouth fatigue threshold is considered to be yawning, exhibiting the fatigue feature.

[0019] According to the technical scheme, the mouth fatigue threshold is 40%, that is, the mouth opening degree greater than 40% is considered to be yawning; in addition, if the mouth opening degree greater than the mouth fatigue threshold is detected, and the driver's speaking sound is detected at the same time, the driver is considered to be speaking with the mouth open, rather than yawning, and the mouth opening degree information of other adjacent frames can also be considered by the system to reduce the misjudgment when the driver speaks with the mouth open.

[0020] According to the technical scheme, the calculation of the mouth opening degree is as follows: the mouth opening degree is equal to the ratio of the distance between the upper lip key point position and the lower lip key point position in the mouth key points to the lip width.

[0021] According to the technical scheme, in the step 4, the specific detection process for whether the driver's head exhibits the fatigue feature is as follows: n frames of images in the normal state of the driver at the beginning of driving are collected, the nose bridge key point or the forehead key point position is taken as the initialization head position index c in the non-fatigue state, and the head position average value d is calculated every time interval t, and the head movement index is defined as d / c, and when the head movement index is lower than the head fatigue threshold, the head is considered to exhibit the fatigue feature.

[0022] According to the technical scheme, N is 30-70; n is 30-70; the time t is twice the time of collecting the video by the camera; and the head fatigue threshold is 0.4.

[0023] According to the technical scheme, in the steps 2 and 3, the training process of the first convolutional neural network model and the second convolutional neural network model is as follows: the images in the data set are classified; the images in the data set are classified into severe fatigue, mild fatigue and normal state according to different combinations of the eye opening degree, the mouth opening degree and the head posture of the images in the data set;

[0024] Then image augmentation is carried out, which plays a role in expanding the data set while relieving the occurrence of overfitting problem;

[0025] Then the Mask R-CNN algorithm is used to simultaneously realize the two tasks of target object detection and object instance segmentation;

[0026] In this way, the portrait and the background in the picture are separated, and the picture is sent into the first convolutional neural network model for learning;

[0027] Then the fatigue detection system model is built and tested, the convolutional neural network is used in cooperation with the SVM classifier to build a training model, the data set with labels is sent into the model in batches for training, after the training is completed, a second convolutional neural network model is formed, the trained model is used to test the prediction accuracy on a test set different from the training data set, and finally the prediction accuracy of the model is obtained, and through continuous optimization, a model with the required accuracy of the project is obtained and deployed to the detection equipment.

[0028] The present application has the following beneficial effects:

[0029] The present application predicts the state of the driver by the image of the driver captured by the camera, forms a non-contact detection, can quickly and accurately judge whether the driver is in a fatigue state in real time, avoids the driver entering a fatigue state, improves driving safety, and ensures the safety of vehicle driving on the highway. BRIEF DESCRIPTION OF DRAWINGS

[0030] Fig. 1 It is a processing process schematic diagram of the driver fatigue driving detection and early warning method based on deep learning in the embodiment of the present application;

[0031] Fig. 2 It is a value assignment schematic diagram of each part position key point of the driver face image in the embodiment of the present application; DETAILED DESCRIPTION

[0032] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary, only for explaining the present application, and cannot be understood as limiting the present application.

[0033] In the description of the present application, it needs to be understood that, if there are terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise" and the like indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second" can be explicitly or implicitly included one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified and limited.

[0034] In the description of the present application, it needs to be noted that, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integral connection. It can be mechanical connection, or electrical connection. It can be directly connected, or indirectly connected through intermediate medium, or the internal communication of two elements or the interaction relationship of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0035] The present application will be described in detail below with reference to the drawings and examples.

[0036] Referring to Figs. 1-2 An embodiment 1 of the present application provides a method for detecting and warning driver fatigue driving based on deep learning, which is characterized by comprising the following steps:

[0037] Step 1: image acquisition of the head with the driver's face by the camera, and transmission of the collected image data to the neural network model;

[0038] Step 2: analysis of the collected image by the neural network model, and segmentation of the head region with the driver's face in the image using image segmentation technology;

[0039] Step 3: detection of the driver's face key points in the segmented image by the neural network model, and judgment of whether the driver wears sunglasses based on the features and shapes of the driver's eye region in the image;

[0040] Step 4, when it is judged that the driver does not wear sunglasses, if the driver's eye shows fatigue characteristics or both the eye and the mouth show fatigue characteristics, it is considered that the driver is in a severe fatigue state, if only the driver's mouth shows fatigue characteristics, it is considered that the driver is in a mild fatigue state, and if no eye fatigue characteristics and mouth fatigue characteristics of the driver are detected, it is considered that the driver is in a non-fatigue state;

[0041] When it is judged that the driver wears sunglasses, if the driver's head shows fatigue characteristics or both the head and the mouth show fatigue characteristics, it is considered that the driver is in a severe fatigue state, if only the driver's mouth shows fatigue characteristics, it is considered that the driver is in a mild fatigue state, and if no head fatigue characteristics and mouth fatigue characteristics of the driver are detected, it is considered that the driver is in a non-fatigue state.

[0042] The neural network model comprises a first convolutional neural network model and a second convolutional neural network model;

[0043] In step 2, the first convolutional neural network model is used to predict the collected image, and the head part with the driver's face in the image is segmented and input to the second convolutional neural network model;

[0044] In step 3, the second convolutional neural network model is used to detect the driver's face key points in the segmented image and judge whether the driver wears sunglasses, and judge the state of the driver through the image (the state includes a severe fatigue state, a mild fatigue state and a non-fatigue state).

[0045] When it is judged that the driver is in a severe fatigue state, a signal can be sent to an alarm device connected to the device to remind the driver, and a signal can also be sent to a connected network to the driver's friends or relatives or a supervisory unit for further reminding. The present application mainly utilizes the related technology of the convolutional neural network of deep learning, and the trained convolutional neural network model can correctly predict new pictures according to the features learned on the training data.

[0046] Further, the first convolutional neural network model is a Mask R-CNN model, and the second convolutional neural network model is a Yolo v5 model.

[0047] Embodiment 2

[0048] On the basis of embodiment 1, the driver's eye fatigue characteristics are further judged, and the performance of the limited embodiment 2 is more excellent.

[0049] In the step 4, when it is judged that the driver does not wear sunglasses, the specific detection process of whether the driver's eyes show fatigue features is as follows: K frames of images of the driver in a normal state are collected at the beginning of driving, and the eye height average value is calculated to initialize the non-fatigue state eye height a; the eye height average value b is calculated every m times of the time interval of the camera video collection for the subsequent images, and clo = b / a; if the clo value measured for 3-5 times continuously is less than the eye fatigue setting threshold W, it is considered that the eyes show fatigue features, otherwise, the case that the single clo is less than the fatigue setting threshold W is regarded as blinking or other temporary phenomenon, and is not regarded as a sign of fatigue.

[0050] Further, the eye height refers to the distance between the upper eyelid and the lower eyelid.

[0051] m is 0.2-0.6, and the optimal selection is 0.4. The "0.4" of m is a unitless coefficient, which is multiplied by the frame rate (unit: "frame / second") of the camera to obtain a new frame rate for the time interval of each detection;

[0052] K is 4-6, and the optimal selection is 5;

[0053] The eye fatigue setting threshold W is 0.68.

[0054] Further, in the step 4, the specific calculation process of the eye height is as follows: the eye position key points in the driver's face image are assigned values, and the average difference between the upper eyelid key point position value and the lower eyelid key point position value is taken as the eye height. The eye position key points are obtained by deep learning model analysis, and specifically, the eye position key points include the eye corner position, the upper eyelid position and the lower eyelid position, which are used to calculate the average height difference between the upper and lower eyelids to determine the opening and closing degree of the eyes.

[0055] Further, in the step 4, the calculation of the eye height refers to Fig. 2 Taking the right eye (43-48) as an example, the eye height is the average of the distance between the eye key points 44 and 48 and the distance between the eye key points 45 and 47.

[0056] Further, when it is detected that the eyes show fatigue features, the average value of the mouth opening degree in the batch of video frames in which the eyes show fatigue features is calculated at the same time, and the mouth opening degree greater than 40% is regarded as yawning, which shows fatigue features.

[0057] Embodiment 3

[0058] On the basis of embodiments 1 and 2, the driver's mouth fatigue features are further judged, and the performance of the limited embodiment 3 is more excellent.

[0059] In the step 4, the specific detection process of whether the driver's mouth shows fatigue features is: calculating the average of the mouth opening degree in the collected video frames, and regarding the mouth opening degree greater than the mouth fatigue threshold as yawning, which shows fatigue features.

[0060] Further, the mouth fatigue threshold is 40%, that is, the mouth opening degree greater than 40% is regarded as yawning; in addition, if the mouth opening degree greater than the mouth fatigue threshold is detected, and the driver's speaking sound is also detected, it is considered that the driver is speaking with his mouth open, not yawning, and other adjacent frame mouth opening degree information can also be considered by the system to reduce the misjudgment when the driver speaks with his mouth open.

[0061] Further, the calculation of the mouth opening degree is: the mouth opening degree is equal to the ratio of the distance between the upper lip key point position and the lower lip key point position in the mouth key points to the mouth width, that is, the distance between the upper lip key point position assigned point 53 and the lower lip key point position assigned point 57 is divided by the sum of the distance between the assigned point 51 and the assigned point 59, and the distance between the assigned point 49 and the assigned point 55 is multiplied by 100%. The number and position of the mouth key points are automatically calibrated by the deep learning model when analyzing the driver's face image, and can be adaptively adjusted and calibrated according to the face structure of different people.

[0062] Further, the calculation of the mouth opening degree is: referring to Fig. 2 , the mouth opening degree is equal to the sum of the distance between the mouth key points 53 and 57 and the distance between 51 and 59 divided by twice the distance between 49 and 55*100%.

[0063] Embodiment 4

[0064] On the basis of embodiments 1-3, the driver's head fatigue features are further judged, and the performance of the limited embodiment 4 is more excellent.

[0065] In the step 4, the specific detection process of whether the driver's head shows fatigue features is: collecting n consecutive images of the driver in a normal state at the beginning of driving, taking the position of the nose bridge key point or the forehead key point as the initialization head position index c in the non-fatigue state, and calculating the head position average d every time interval t thereafter, and defining the head movement index as d / c, when the head movement index is lower than the head fatigue threshold, it is considered that the head shows fatigue features.

[0066] Further, in this embodiment, the vertical coordinate average of the assigned points 29 and 30 of the nose bridge key point is taken as the head position index, wherein the assigned points 29 and 30 (as well as the assigned points of the eye) are the head feature points calibrated by the deep learning model when analyzing the driver's face image, and the two points can be used to calculate the position change and movement amplitude of the head; to evaluate the fatigue state of the driver.

[0067] Further, N is 30-70; n is 30-70; time t is 2 times the time of the camera collecting video; the head fatigue threshold is 0.4.

[0068] Further, the optimal choice of N is 50, and the optimal choice of n is 50.

[0069] Further, in steps 2 and 3, the training process of the first convolutional neural network model and the second convolutional neural network model is as follows: first, classify the images in the data set; according to the different combinations of eye opening degree, mouth opening degree and head posture of the images in the data set, the images in the data set are divided into severe fatigue, mild fatigue and normal state; here, the identification method of severe fatigue, mild fatigue and normal state is consistent with the identification method in step 4;

[0070] Then, image augmentation is performed, which plays a role in expanding the data set while relieving the occurrence of overfitting problem;

[0071] Then, the Mask R-CNN algorithm is used to simultaneously realize the two tasks of object detection and object instance segmentation;

[0072] In this way, the portrait and the background in the picture are separated, and the picture is sent to the first convolutional neural network model for learning;

[0073] Then, the fatigue detection system model is built and tested, and the convolutional neural network is used to build a training model with a SVM (SVM is a support vector machine) classifier. The data set with labels is sent to the model in batches for training. After training, a second convolutional neural network model is formed. The trained model is tested on a test set different from the training data set to test the prediction accuracy. Finally, the accuracy of the model prediction is obtained, and through continuous optimization, a model with the required accuracy of the project is obtained and deployed to the detection equipment.

[0074] In summary, 1. Improve image preprocessing, optimize the training process of the model: before the image data is input into the training system, the image is further processed, and the related algorithm of mask R-CNN is used in the instance segmentation of the image to train the portrait instance segmentation network of mask R-CNN. Compared with traditional instance segmentation, the image processing speed and instance segmentation accuracy are greatly improved after adding the mask R-CNN algorithm. After performing instance segmentation on the portrait and background, the model training will not learn irrelevant features, reducing the training amount of the model, improving the training effect of the model to a certain extent, and improving the reliability of the early warning system. At the same time, in actual detection, the image data collected by the camera is first separated by the portrait instance segmentation network of mask R-CNN, and then transmitted into the trained fatigue driving detection model to improve the model recognition speed and detection accuracy.

[0075] 2. Multi-system fusion to improve accuracy: use a deep learning model as a benchmark model, from image preprocessing to convolutional neural network, use multi-system joint training, and through additional auxiliary classifiers in the training process, further optimize the effect of the classifier, improve the detection accuracy, and realize high-precision recognition.

[0076] 3. Use lightweight network MobileNet combined with vehicle-mounted camera: compared with traditional technology, MobileNet, which uses depthwise separable convolution, can not only reduce model calculation complexity, but also greatly reduce model size. MobileNet combined with vehicle-mounted camera makes the system only need to connect the vehicle-mounted camera input to run in actual application, solves the problem of relying on contact sensors, and can expand to connect other warning devices to achieve the warning effect when the driver is fatigue driving.

[0077] It should be noted that in this article, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment.

[0078] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary of the principles and application of the present application. Numerous modifications and adaptions can be effected without departing from the spirit and scope of the present application, which is not limited to the exact construction and arrangement described. It is intended, therefore, to cover all modifications and adaptions that fall within the scope of the claims and their equivalents.

Claims

1. A method for detecting and warning driver fatigue driving based on deep learning, characterized in that, The method comprises the following steps: Step 1, image acquisition of the head with the driver's face by the camera, and transmission of the acquired image data to the neural network model; Step 2, analysis of the acquired image by the neural network model, and segmentation of the head region with the driver's face in the image by using an image segmentation technology; Step 3, detection of the driver's face key points in the segmented image by the neural network model, and judgment of whether the driver wears sunglasses based on the features and shapes of the eye region of the driver in the image; Step 4, when it is judged that the driver does not wear sunglasses, if the driver's eye region is detected to exhibit fatigue features or both the eye region and the mouth region exhibit fatigue features, it is considered that the driver is in a severe fatigue state, if only the driver's mouth region is detected to exhibit fatigue features, it is considered that the driver is in a mild fatigue state, and if neither the eye region nor the mouth region of the driver is detected to exhibit fatigue features, it is considered that the driver is in a non-fatigue state; When it is judged that the driver wears sunglasses, if the driver's head region is detected to exhibit fatigue features or both the head region and the mouth region exhibit fatigue features, it is considered that the driver is in a severe fatigue state, if only the driver's mouth region is detected to exhibit fatigue features, it is considered that the driver is in a mild fatigue state, and if neither the head region nor the mouth region of the driver is detected to exhibit fatigue features, it is considered that the driver is in a non-fatigue state; In step 4, when it is judged that the driver does not wear sunglasses, the specific detection process of whether the driver's eye region exhibits fatigue features is as follows: K frames of images in the normal state of the driver are acquired at the beginning of the driving process, and the eye height mean value of the driver is calculated for initialization of the non-fatigue state eye height a; the eye height mean value b is calculated once every m times of the time interval of the video acquisition of the camera for the subsequent images, and clo=b / a; if the clo value measured for multiple times is less than the eye fatigue setting threshold W, it is considered that the eye region exhibits fatigue features, otherwise, the case that the single clo is less than W is regarded as blinking or other temporary phenomena, and is not regarded as a sign of fatigue; In step 4, the specific detection process of whether the driver's head region exhibits fatigue features is as follows: n frames of images in the normal state of the driver are acquired at the beginning of the driving, the position of the nose bridge key point or the forehead key point is taken as the initialization of the head position index c in the non-fatigue state, and the head position mean value d is calculated once every time interval t thereafter, and the head movement index is defined as d / c, and when the head movement index is lower than the head fatigue threshold, it is considered that the head region exhibits fatigue features. 2.The method of claim 1, wherein, The neural network model comprises a first convolutional neural network model and a second convolutional neural network model; In step 2, the first convolutional neural network model is used to predict the acquired image, and the head with the driver's face in the image is segmented and input to the second convolutional neural network model; In step 3, the second convolutional neural network model is used to detect the driver's face key points in the segmented image, judge whether the driver wears sunglasses, and determine the state of the driver through the image. 3.The method of claim 1, wherein, m is 0.2-0.6; K is 4-6, and the optimal selection is 5; if the clo value is less than the fatigue setting threshold W for 3-5 consecutive times, the eye is considered to exhibit fatigue characteristics; The eye fatigue setting threshold W is 0.

68. 4.The method of claim 1, wherein, In the step 4, the specific calculation process of the eye height is as follows: the position key points of the eye in the driver's face image are assigned values, and the average difference between the upper eyelid key point position value and the lower eyelid key point position value is taken as the eye height. 5.The method of claim 1, wherein, In the step 4, the specific detection process of whether the driver's mouth exhibits fatigue characteristics is as follows: the average opening degree of the mouth in the collected video frames is calculated, and the mouth opening degree greater than the mouth fatigue threshold is considered as yawning, which exhibits fatigue characteristics. 6.The method of claim 5, wherein, The calculation of the mouth opening degree is as follows: the mouth opening degree is equal to the ratio of the distance between the upper lip key point position and the lower lip key point position in the mouth key points to the mouth width. 7.The method of claim 1, wherein, N is 30-70; n is 30-70; the time t is 2 times the video acquisition time of the camera; The head fatigue threshold is 0.

4. 8.The method of claim 2, wherein, In the steps 2 and 3, the training process of the first convolutional neural network model and the second convolutional neural network model is as follows: first, the images in the data set are classified; the images in the data set are divided into severe fatigue, mild fatigue and normal state according to different combinations of the eye opening degree, the mouth opening degree and the head posture of the images in the data set; Then, image augmentation is performed, which plays a role in expanding the data set and relieving the occurrence of overfitting; Then, the Mask R-CNN algorithm is used to realize the two tasks of target object detection and object instance segmentation at the same time; In this way, the portrait and the background in the picture are separated, and the picture is sent to the first convolutional neural network model for learning; Then, the fatigue detection system model is built and tested, the convolutional neural network is used to build the training model, the data set with labels is sent to the model in batches for training, after the training is completed, the second convolutional neural network model is formed, the trained model is used to test the prediction accuracy on the test set different from the training data set, and finally the prediction accuracy of the model is obtained, and through continuous optimization, the model with the required accuracy of the project is obtained and deployed to the detection equipment.

Images