Fatigue driving detection method, system, computer equipment and storage medium

Abstract

The invention provides a fatigue driving detection method, system, computer equipment and storage medium, which belong to the field of fatigue driving detection. The method comprises the following steps: constructing eye, mouth and head data sets; adding SimAM in YOLOv8n, and taking Detect and Pose as outputs respectively to obtain YOLOv8-ROI and YOLOv8-Keypoints, and using YOLOv8-ROI to extract images of regions of interest of eyes and mouth; using YOLOv8-Keypoints to extract key point information of the head; inputting the eye and mouth images into a MobileNetv3-Small neural network for classification, and distinguishing the opening and closing states of the eyes and mouth; and according to the key point information of the head, using coordinate system transformation to calculate the three-dimensional posture change of the head and distinguishing the head state, and obtaining the fatigue state of the driver. The invention can improve the fatigue driving detection accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of fatigue driving detection, and in particular relates to a fatigue driving detection method, system, computer equipment and storage medium. Background Art

[0002] With the rapid development of my country's economy, the improvement of living standards and road construction, cars have become the most commonly used means of transportation for Chinese citizens. In recent years, the actual traffic volume on the road has continued to grow, and the number of motor vehicle drivers has also continued to rise. As of the end of June this year, there were 513 million motor vehicle drivers in my country, of which 475 million were car drivers. But while cars provide convenience to our lives, they also bring a series of problems, such as the frequent occurrence of traffic accidents. According to previous statistics, among the many causes of traffic accidents, fatigue driving is one of the factors that cause the most deaths, and the number of deaths caused by accidents each year accounts for more than 10.9% of the number of deaths in motor vehicle accidents. Traffic accidents caused by fatigue driving account for about 14%-20%, of which large trucks and traffic accidents on highways account for about 37%, and heavy traffic accidents account for about 43%. If the driver can be reminded in time when there are signs of fatigue driving, the driver can be warned and reminded to take a rest in time, it will effectively reduce the occurrence of traffic accidents caused by driver fatigue driving. Therefore, an effective fatigue driving detection method is of great significance to reality.

[0003] Since the fatigue driving detection method based on computer vision has the advantages of being non-invasive, not affecting driving behavior, and being easier to apply in actual scenarios, more and more scholars are now studying the fatigue driving detection method based on computer vision. However, current research mainly focuses on the processing and identification of the driver's frontal image, and there is little research on the identification of the side image. However, in reality, the position of the camera in the driver's car is random, and the collected image may be angularly deflected, resulting in low accuracy of the existing fatigue driving detection method. Summary of the invention

[0004] In order to overcome the disadvantage of low accuracy of existing fatigue driving detection methods, the present invention provides a fatigue driving detection method, comprising the following steps:

[0005] Acquire driving video data of the driver during driving, obtain video frame images from the video data, and construct eye datasets, mouth datasets, and head datasets based on the video frame images;

[0006] Based on the YOLOv8n model, a parameter-free attention mechanism SimAM is added between the bottleneck layer C2f and the spatial pyramid pooling layer SPPF of the Backbone of the YOLOv8n model to obtain YOLOv8-SimAM; the detection layer Detect and the posture layer Pose are used as the output of YOLOv8-SimAM, respectively, and the network structure YOLOv8-ROI and the key point extraction network YOLOv8-Keypoints are obtained accordingly;

[0007] The data in the eye dataset and mouth dataset are respectively input into YOLOv8-ROI for feature extraction, and the eye region of interest ROI images and mouth region of interest ROI images in the driver's multi-angle side state are obtained; the head dataset is input into YOLOv8-Keypoints for facial key point positioning, and the driver's head key point information at multiple angles is obtained;

[0008] Input the eye ROI image and the mouth ROI image into the MobileNetv3-Small neural network, classify the eye ROI image and the mouth ROI image, and determine the open and closed states of the eyes and the mouth in multiple angles;

[0009] According to the key point information of the head, the three-dimensional posture change of the head is calculated using the coordinate system transformation method, and the head state is determined according to the three-dimensional posture change value;

[0010] The driver's fatigue status is obtained by comprehensively analyzing the opening and closing status of the eyes and mouth and the identified head status at multiple angles.

[0011] Preferably, the three-dimensional posture change of the head is calculated using a coordinate system transformation method, comprising the following steps:

[0012] Get the 3D world coordinate system of the key points of the head;

[0013] Converting the three-dimensional world coordinate system into a two-dimensional camera coordinate system;

[0014] Use the conversion formula to transform the two-dimensional camera coordinate system into the image coordinate system to obtain the rotation matrix;

[0015] Solve the rotation matrix to get the Euler angles;

[0016] Determine head pose changes based on Euler angles.

[0017] Preferably, the conversion formula is:

[0018]

[0019] Among them, f x 、f y 、cx 、c y is the camera intrinsic parameter matrix coefficient, r ij With t i are the rotation and translation matrices, s is the scale factor, and U, V, and W are the coordinates in the world coordinate system.

[0020] Preferably, the calculation formula of the Euler angle is as follows:

[0021]

[0022]

[0023]

[0024] Where α, β, and γ correspond to the x, y, and z rotation angles in the world coordinate system, respectively.

[0025] Preferably, before inputting the eye dataset and the mouth dataset into YOLOv8-ROI for feature extraction, YOLOv8-ROI needs to perform the following operations:

[0026] Collect multi-angle side data sets of the driver at 0°, 30°, 60°, and 90°;

[0027] Annotating the eye positions and mouth positions of the images in the multi-angle profile data set to obtain an annotated sample data set;

[0028] The YOLOv8-ROI is trained using a labeled sample dataset.

[0029] Preferably, before inputting the eye ROI image and the mouth ROI image into the MobileNetv3-Small neural network, the following steps are also included:

[0030] Get the open source dataset CEW of closed and open eyes samples;

[0031] Get the sample data set Yawn of the mouth's closed and open states;

[0032] MobileNetv3-Small is trained using the CEW, Yawn and labeled sample data sets.

[0033] Preferably, before inputting the head data set into YOLOv8-Keypoints for facial key point positioning, the wild human face facial key point data set WiderFace is used to train YOLOv8-Keypoints.

[0034] The present invention also provides a fatigue driving detection system, comprising:

[0035] The data set acquisition module is used to acquire driving video data of the driver during driving, and obtain video frame images from the video data, and construct eye data sets, mouth data sets and head data sets according to the video frame images;

[0036] The model building module is used to add a parameter-free attention mechanism SimAM between the bottleneck layer C2f and the spatial pyramid pooling layer SPPF in the Backbone of the YOLOv8n model to obtain YOLOv8-SimAM; the detection layer Detect and the posture layer Pose are respectively used as the output of YOLOv8-SimAM, and the network structure YOLOv8-ROI and the key point extraction network YOLOv8-Keypoints are obtained accordingly;

[0037] The feature extraction module is used to input the data in the eye dataset and the mouth dataset into YOLOv8-ROI for feature extraction, and obtain the eye region of interest ROI image and the mouth region of interest ROI image in the driver's multi-angle side state; input the head dataset into YOLOv8-Keypoints for facial key point positioning, and obtain the driver's head key point information at multiple angles;

[0038] A first state discrimination module is used to input the eye ROI image and the mouth ROI image into the MobileNetv3-Small neural network, classify the eye ROI image and the mouth ROI image, and discriminate the open and closed states of the eyes and the mouth in multiple angles;

[0039] The second state discrimination module is used to calculate the three-dimensional posture change of the head according to the key point information of the head using the coordinate system conversion method, and to discriminate the head state according to the three-dimensional posture change value;

[0040] The fatigue status analysis module is used to comprehensively analyze the opening and closing status of the eyes and mouth and the identified head status under multiple angles to obtain the driver's fatigue status.

[0041] The present invention also provides a computer device, characterized in that it includes a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute a fatigue driving detection method, system, computer device and storage medium.

[0042] The present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and the computer program is suitable for loading by a processor to execute a fatigue driving detection method, system, computer device and storage medium.

[0043] The fatigue driving detection method, system, computer device and storage medium provided by the present invention have the following features:

[0044] Beneficial effects:

[0045] The present invention adds a parameter-free attention mechanism SimAM to YOLOv8 to obtain YOLOv8-SimAM, and uses Detect and Pose as the output of YOLOv8-SimAM respectively, so as to obtain a network structure YOLOv8-ROI and a key point extraction network YOLOv8-Keypoints; by using YOLOv8-ROI, eye ROI images and mouth ROI images can be extracted, and the average extraction accuracy of the eye and mouth ROI in all angles of the side state reaches 98.7%; and after the successful extraction, the eye and mouth states are distinguished by MobileNetv3, and the average distinguishing accuracy of the eye and mouth states in all angles of the side state reaches 90.18%. In addition, the key points of the head at all angles can be located by YOLOv8-Keypoints, and the average accuracy reaches 85.9%, and the three-dimensional head posture is calculated according to the coordinate system conversion method to distinguish the head state, and the average distinguishing accuracy of the head state in all angles of the side state reaches 87.41%. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiment of the present invention and its design scheme, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 A flowchart of a fatigue driving detection method, system, computer device, and storage medium according to an embodiment of the present invention;

[0048] Figure 2 This is the network structure diagram of YOLOv8;

[0049] Figure 3 The network structure diagram of YOLOv8-ROI and YOLOv8-Keypoints;

[0050] Figure 4 is the image of the region of interest of the eye;

[0051] Figure 5 The image of the region of interest of the mouth;

[0052] Figure 6 is the relationship between the camera and the three-dimensional coordinate system;

[0053] Figure 7Illustration of the three-dimensional change direction of head posture. DETAILED DESCRIPTION

[0054] In order to enable those skilled in the art to better understand the technical solution of the present invention and implement it, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention.

[0055] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the technical solutions of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0056] In addition, the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified or limited, the terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. In the description of the present invention, unless otherwise specified, "plurality" means two or more, which will not be described in detail here.

[0057] Example

[0058] The present invention provides a method for detecting fatigue driving, specifically Figure 1 As shown, the following steps are included:

[0059] Step 1: Obtain driving video data of the driver during driving, and obtain video frame images from the video data, and construct eye datasets, mouth datasets, and head datasets based on the video frame images.

[0060] Since the public dataset lacks the dataset of the driver's side view and the dataset containing the eye and mouth position annotations in the side view, the dataset must be collected first. The present invention collects the driver's multi-angle side view dataset at 0°, 30°, 60°, and 90°, and annotates the eye and mouth positions of some datasets to form a training set for model training.

[0061] The head data set of the present invention uses a field human face facial key point data set (WiderFace). The WiderFace data set is currently recognized as one of the largest face detection data sets in the field of target detection. It contains a large number of face images of different types, sizes, postures and occlusion levels, and is therefore widely used in the training and performance evaluation of face detection algorithms. The data set contains 32,203 face images in various indoor and outdoor scenes and under different backgrounds, covering a large number of faces in real scenes. In addition, the data set contains data of people with multiple postures at various angles, which can realize the function of locating facial key points in different angles of side faces.

[0062] Step 2: Based on the YOLOv8n model, add a parameter-free attention mechanism SimAM between the bottleneck layer C2f and the spatial pyramid pooling layer SPPF of the Backbone of the YOLOv8n model to obtain YOLOv8-SimAM; use the detection layer Detect and the posture layer Pose as the output of YOLOv8-SimAM, respectively, and obtain the network structure YOLOv8-ROI and the key point extraction network YOLOv8-Keypoints.

[0063] like Figure 2 As shown in the figure, the network structure of YOLOv8 consists of Input, Backbone, Neck and Head. The Input uses Mosaic data enhancement and adaptive anchor box calculation. The Backbone part, as the main body of the backbone network, mainly includes convolution module (CBS), bottleneck layer (C2f) and spatial pyramid pooling (SPPF). Its C2f is further lightweight than the previous C3. Neck performs multi-scale fusion between Backbone and Head, and the Head part completes the output of target detection results through convolution operation (Conv). YOLOv8 uses DFL Loss+CIOU Loss as the classification loss function to calculate the bounding box loss.

[0064] In order to further improve the feature extraction capability of the YOLOv8 backbone, some local feature enhancement methods are used in the network structure, such as adding an attention mechanism to improve the model accuracy. Among the various attention mechanisms, adding a parameter-free attention mechanism is a good method, which will not increase the number of model parameters, and therefore can better ensure the lightweight and detection speed of the model. The present invention improves YOLOv8n by adding a parameter-free attention mechanism. The position where the attention mechanism is added is between C2f and SPPF. The final attention mechanism selected is SimAM, which has no parameters and has the best comprehensive performance after experimental comparison of multiple attention mechanisms. Then, Detect and Pose are used as the outputs of YOLOv8-SimAM, respectively, and the network structure YOLOv8-ROI (Region Of Interest, ROI) and the key point extraction network YOLOv8-Keypoints are obtained accordingly. The improved network structure is as follows Figure 3 shown.

[0065] Step 3: Input the data in the eye dataset and the mouth dataset into YOLOv8-ROI for feature extraction, and obtain the eye region of interest ROI image and mouth region of interest ROI image in the driver's multi-angle side state; input the head dataset into YOLOv8-Keypoints for facial key point positioning, and obtain the driver's head key point information from multiple angles.

[0066] (1) Train YOLOv8-ROI and extract the ROI images of the eye region of interest and the mouth region of interest.

[0067] Since there is a lack of data sets in the public data set for the driver's side profile and data sets with eye and mouth position annotations in the side profile, we first need to collect data sets. We collected multi-angle side profile data sets of drivers at 0°, 30°, 60°, and 90°, and annotated the eye and mouth positions of some data sets to form a training set for model training. In the experiment, the training data set mainly uses self-built data samples of the eye and mouth areas annotated at multiple angles of the side face. The training set has a total of 7760 sample images, and the validation set has a total of 2229 sample images.

[0068] When extracting the eye and mouth ROI images as the discrimination criteria, it is usually necessary to find the positions of the eyes and mouth in the pre-detected face area and divide the corresponding ROI images. The driver's eye ROI image extracted by YOLOv8-ROI in the present invention is as follows: Figure 4 As shown, the mouth ROI image is as follows Figure 5 As shown in the figure, by training the self-built data set, the function of extracting the eye and mouth ROIs of the driver in the multi-angle side states of 0°, 30°, 60°, and 90° is realized.

[0069] (2) Extraction of head key point information.

[0070] The present invention uses the WiderFace dataset to train YOLOv8-Keypoints to realize the head feature extraction function based on facial key points at different angles.

[0071] The method for extracting the three-dimensional posture change features of the driver's head is to obtain the facial feature points and then obtain the changes in the driver's head posture through algorithms or calculations. YOLOv8-Keypoints is also an improvement on YOLOv8n. Its improvement method is the same as YOLOv8-ROI. Both are based on YOLOv8n and add a parameterless attention mechanism between C2f and SPPF in its Backbone. After multiple experimental verifications, it was finally found that the performance was optimal after adding the parameterless attention mechanism SimAM. Therefore, YOLOv8-Keypoints is based on YOLOv8-SimAM, uses the Pose in the model as the final output, and finally forms the YOLOv8-Keypoints network structure. YOLOv8-Keypoints adds the key point positioning function on the basis of YOLOv8, adds the parameterless attention mechanism SimAM to improve performance, and finally realizes the positioning of the driver's facial key points at multiple angles of 0°, 30°, 60°, and 90°. Subsequently, the three-dimensional head posture is calculated based on the extracted key points to determine the head state.

[0072] Step 4: Input the eye ROI image and the mouth ROI image into the MobileNetv3-Small neural network, classify the eye ROI image and the mouth ROI image, and determine the opening and closing states of the eyes and mouth in multiple angles.

[0073] MobileNetv3 is an efficient and lightweight deep neural network architecture that can be used on mobile devices and embedded devices. MobileNetv3 achieves efficient model compression and acceleration by using technologies such as deep separable convolution, inverted residual connection, and SE module, which reduces the complexity and number of parameters of the model while maintaining high model performance. Among them, deep separable convolution reduces the complexity and number of parameters of the model by reducing the number of parameters of the convolution layer; inverted residual connection improves the expressiveness of the model by introducing nonlinear activation functions; SE (Squeeze-and-Excitation) module further improves model performance through global pooling operations and channel attention mechanism. In addition, MobileNetv3 also has two variants, MobileNetv3-Large and MobileNetv3-Small, which can meet application scenarios with different computing resources and latency requirements. Therefore, the image classification function of MobileNetv3 is used to classify multi-angle eye and mouth images, and to distinguish the state of eyes and mouth under multiple angles. MobileNetv3-Small was selected for classification in the experiment, and its network structure is shown in Table 1.

[0074] Table 1 MobileNetv3-Small network structure

[0075]

[0076]

[0077] MobileNetv3-Small was used in the experiment, and the dataset was used to train MobileNetv3. The dataset mainly used the open source dataset CEW to obtain closed and open eye samples, Yawn and other mouth-related datasets to obtain data samples of the closed and open states of the mouth, and self-built side face multi-angle annotated data samples of the eyes and mouth opening and closing states. The training set has a total of 7760 sample images, including 1429 closed eyes, 2349 closed mouths, 1623 open eyes, and 2359 open mouths. The validation set has a total of 2229 sample images, including 592 closed eyes, 768 closed mouths, 251 open eyes, and 618 open mouths. The test set has a total of 1984 sample images, including 420 closed eyes, 603 closed mouths, 360 open eyes, and 601 open mouths. Through the three data sets of CEW, Yawn and self-built data sets, the data set contains pictures of various eye and mouth states under various lighting conditions and multi-angle side face states, which is conducive to better distinguishing the eye and mouth states at various frontal and side angles.

[0078] Then, the pre-extracted eye ROI and mouth ROI images at different multi-angle states of 0°, 30°, 60°, and 90° are input into MobileNetv3-Small, and the input information is classified to determine the opening and closing states of the eyes and mouth at multiple angles of 0°, 30°, 60°, and 90°.

[0079] Step 5: Based on the key point information of the head, use the coordinate system transformation method to calculate the three-dimensional posture change of the head, and judge the head state based on the three-dimensional posture change value; conduct a comprehensive analysis of the opening and closing status of the eyes and mouth in multiple angles and the judged head state to obtain the driver's fatigue state.

[0080] The head posture calculation dataset uses the Wider Facial Landmarks in-the-wild (WFLW) dataset and a self-built dataset. After processing, the two datasets have a total of 10,000 facial images of various postures for training, testing and verification. In the experiment, 8,000 images are used for training, 1,000 for testing, and 1,000 for verification. The dataset contains various states and data in various environments, including facial occlusion, multiple angles, multiple lighting environments, and multiple clarity. Such a rich variety of datasets is closer to reality and more reference for practical applications. Through the training of the dataset, the calculation of various head postures of the driver's multi-angle side face states of 0°, 30°, 60°, and 90° in the self-built dataset is finally realized.

[0081] Before collecting head features, the following four steps need to be performed: extract 2D face key point information; establish 3D coordinates and match 2D facial features; use the conversion formula to solve the dimension conversion relationship; use the Rodriguez rotation matrix to perform algebraic operations to solve the Euler angle. The present invention calculates the three-dimensional posture change of the driver's head based on the 2D key point information processing, and then determines the driver's head state. The specific process is as follows:

[0082] (1) Coordinate system transformation to solve three-dimensional posture changes

[0083] The key to solving the Euler angle is to transform the rotation matrix. Assume that we know that the position of a 3D point P in the world coordinate system is (U, V, W), the corresponding (x, y) is its 2D position in the image coordinate system, and the position of point P in the camera coordinate system is (X, Y, Z). The transformation relationship between various coordinate systems is as follows: Figure 6 As shown, the positional relationship has two transformation processes. The first is the transformation process between the 3D world coordinate system and the 2D camera coordinate system, and the second is the two-dimensional mapping between the camera coordinate system and the image coordinate system to obtain the rotation matrix R.

[0084] During the coordinate system conversion process, the driver's head posture is first collected through the camera, and the matrix camera model is converted according to formula (1).

[0085]

[0086] Among them, f x 、f y 、c x 、c y is the camera intrinsic parameter matrix coefficient, r ij With t i is the rotation matrix. Since the scale factor s and the translation matrix are both known parameters, the rotation matrix R can be directly solved. Then the rotation matrix R can be used to represent the Euler angle, and its calculation formula is shown in formula (2-4).

[0087]

[0088]

[0089]

[0090] Substituting the above formula into R=Rx(α)·Ry(β)·Rz(γ), it can be expressed as the sine and cosine matrix of the Euler angles, and the calculation formula is (5).

[0091]

[0092] By solving the equation, the Euler angle can be calculated as shown in formula (6).

[0093]

[0094] The calculated α, β, and γ correspond to the x, y, and z rotation angles in the world coordinate system, respectively. Therefore, the three-dimensional head posture can be calculated through the above coordinate system conversion method.

[0095] (2) Determine the head state by changing the three-dimensional posture

[0096] When a driver is in a fatigued state, his head posture often changes accordingly, usually manifested as frequent nodding and abnormal head posture. The angle change of head posture is divided into three angles: Pitch, Yaw, and Roll. Its three-dimensional change direction is as follows Figure 7 shown.

[0097] like Figure 7As shown in the figure, frequent nodding can be regarded as the continuous swing of the head around the Pitch axis, so the angle on the Pitch axis will have obvious fluctuations; abnormal head posture includes tilt and shaking of the head, which can be regarded as changes around the Roll axis; and changes in head posture around the Yaw axis often occur in head swing. The rotation angles of the driver's head on the three-dimensional x, y, and z axes at multiple angles were calculated based on the coordinate system conversion, and the head state was judged based on this.

[0098] Ferrariod et al. found that the range of head posture changes under normal circumstances has a fixed value. After research, they gave the maximum range of changes in the three angles of the general human head, among which the maximum angle range of rotation around the Pitch axis is [-60.4°, 69.6°], the maximum range of rotation around the Yaw axis is [-90°, 75°], and the maximum range of rotation around the Roll axis is [-40.9°, 36.3°]. Therefore, when the Pitch axis posture changes beyond this standard, it can be judged that nodding behavior has occurred, and when the Yaw and Roll axis posture changes exceed the standard, it can be judged that abnormal head behavior has occurred. The driver's fatigue state can be judged based on the judgment of nodding behavior and abnormal head behavior.

[0099] The present invention also provides a fatigue driving detection system based on the side state of the driver, including a data set acquisition module, a model construction module, a feature extraction module, a first state discrimination module, a second state discrimination module and a fatigue state analysis module. The data set acquisition module is used to obtain driving video data of the driver during driving, and obtain video frame images from the video data, and construct eye data sets, mouth data sets and head data sets according to the video frame images; the model construction module is used to add a parameter-free attention mechanism SimAM between the bottleneck layer C2f and the spatial pyramid pooling layer SPPF in the Backbone of the YOLOv8n model based on the YOLOv8n model to obtain YOLOv8-SimAM; the detection layer Detect and the posture layer Pose are respectively used as the output of YOLOv8-SimAM, and the network structure YOLOv8-ROI and the key point extraction network YOLOv8-Keypoints are correspondingly obtained; the feature extraction module is used to input the data in the eye data set and the mouth data set into YOLOv8-ROI for feature extraction, and obtain the driver's multi-data set. The eye region of interest ROI image and mouth region of interest ROI image in the side angle state; the head data set is input into YOLOv8-Keypoints to locate the facial key points, and the driver's head key point information at multiple angles is obtained; the first state discrimination module is used to input the eye ROI image and the mouth ROI image into the MobileNetv3-Small neural network, classify the eye ROI image and the mouth ROI image, and distinguish the opening and closing states of the eyes and mouth in multiple angle states; the second state discrimination module is used to calculate the three-dimensional posture change of the head according to the head key point information using the coordinate system transformation method, and distinguish the head state according to the three-dimensional posture change value; the fatigue state analysis module is used to comprehensively analyze the opening and closing states of the eyes and mouth in multiple angle states and the judged head state to obtain the driver's fatigue state.

[0100] The present invention also provides a computer device, including a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute a fatigue driving detection method, system, computer device and storage medium.

[0101] The present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and the computer program is suitable for loading by a processor to execute a fatigue driving detection method, system, computer device and storage medium.

[0102] The present invention obtains an improved YOLOv8-SimAM by adding a parameter-free attention mechanism SimAM to YOLOv8. By using YOLOv8-SimAM, eye ROI images and mouth ROI images can be extracted, and the average extraction accuracy of the eye and mouth ROI in all angles of the side state reaches 98.7%, and the reasoning speed is 5.1ms, which can be used for real-time detection; and after the successful extraction, the eye and mouth states are distinguished by MobileNetv3, and the average discrimination accuracy of the eye and mouth states in all angles of the side state reaches 90.18%. Secondly, the positioning of the head key points at all angles is realized by YOLOv8-Keypoints, and the average accuracy reaches 85.9%, and the reasoning speed is 3.1ms, which can be used for real-time detection; and the three-dimensional head posture is calculated according to the coordinate system conversion method to distinguish the head state, and the average discrimination accuracy of the head state in all angles of the side state reaches 87.41%.

[0103] Example 2

[0104] In order to evaluate the performance of the proposed method, corresponding experiments were designed for the four methods for verification. This paper first introduces the data set and data set processing method used in each experiment, then introduces the verification experiment design of each method, and finally summarizes and analyzes the verification results of each method.

[0105] The feature extraction and discrimination experiments of driver fatigue status are tested under the driving behavior of multiple drivers in real environments. The features considered when formulating the test conditions are as follows: eye features, mouth features, head posture. All people will be captured in a one-minute driving video in a real driving environment with a video frame rate of 30 frames / s. Each frame of the video is captured for training, verification and testing of feature extraction and feature discrimination experiments.

[0106] A 10-fold cross validation was performed, repeating the process 10 times, with each of the 10 subjects used only once as a test dataset. In each fold, the data of nine subjects were used as a training dataset, while the remaining subject was used as a test dataset. The detection performance was measured in terms of accuracy, sensitivity, and specificity. A good performance for any classification is defined by high accuracy, high sensitivity, and high specificity. To select a suitable model, the average values ​​of accuracy, sensitivity, and specificity of the experiment were considered.

[0107] In order to maintain the randomness of the data, the ratio of the training set, validation set, and test set is basically maintained at 8:1:1 when dividing the data set. In addition, in order to maintain the randomness of the data set after data enhancement, the present invention divides the training set, validation set, and test set after data enhancement, rather than enhancing the data on the basis of the original training set, validation set, and test set. When training the model, the hyperparameters are all default, and the experimental environment is shown in Table 2.

[0108] Table 2 Experimental environment

[0109]

[0110] 1. Eye and mouth feature extraction experiment

[0111] (1) Experimental design

[0112] In order to analyze the state of the eyes and mouth, we first need to locate and extract the images of the eye and mouth regions of interest on the side face at various angles. In data preprocessing, all videos with a video frame rate of 30 frames / s in the self-collected dataset are converted into image datasets frame by frame. The eye and mouth feature extraction experiment completes the extraction of each person's eye and mouth features through the training set in the model training image dataset.

[0113] In this experiment, the four common attention mechanisms SimAM, GCT, SE, and GAM are used to optimize the backbone network of YOLOv8n and compared with YOLOv8n, YOLOv8s, and YOLOv5n algorithms. The algorithm's inference time for each image is used to measure the real-time performance of the algorithm, and Time is taken as the average value of 10 times; "+SimAM" means adding a SimAM attention mechanism module to the backbone network, "+SE" means adding an SE attention mechanism module to the backbone network, "+GCT" means adding a GCT attention mechanism module to the backbone network, and "+GAM" means adding a GAM attention mechanism module to the backbone network.

[0114] (2) Experimental results and analysis

[0115] The experimental results of eye and mouth feature extraction are shown in Table 3.

[0116] Table 3 Experimental results of eye and mouth feature extraction

[0117]

[0118] According to the experimental results, the results of eye and mouth feature extraction are different at different angles. According to the experimental results, it can also be found that the network structure based on the YOLOv8n network with the non-parameter SimAM attention mechanism in the current method has the best comprehensive effect in terms of detection accuracy, speed, etc. As can be seen from the table, on the basis of keeping the model lightweight, SimAM can effectively improve the original accuracy of the model compared with other algorithms. While reducing the number of parameters and the amount of calculation in the above algorithm, the detection accuracy of the model is also reduced, but the detection accuracy can be improved by other means, such as the data enhancement method of the present invention, the box regression loss function, etc. All increase the reasoning time of the model, which also shows that these methods are not conducive to the real-time detection of the model. The reason is that: SE and GAM both use grouped convolution. Although grouped convolution can reduce the complexity of calculation, it also brings more memory access consumption, thereby reducing the reasoning speed; on the other hand, SimAM and GCT do not increase the model size because they are non-parameter attention mechanism modules. Therefore, the experiment verifies that the improved YOLOv8-ROI algorithm can more effectively increase the accuracy of the model, and the improved algorithm has also been improved in detection speed. YOLOv8-ROI corresponds to the YOLOv8n+SimAM algorithm in the experiment of the present invention. In order to improve the effect of eye and mouth feature extraction, it is reasonable to use the YOLOv8-ROI algorithm.

[0119] 2. Head feature extraction experiment

[0120] (1) Experimental design

[0121] Since the calculation of head posture requires five key points of the face, we first need to accurately locate the five key points of the face on the side face at various angles. The WiderFace dataset contains a large number of face images of different types, sizes, postures, and occlusion levels. It can be used for the driver key point extraction experiment in the side state at multiple angles, so it is used to perform the head feature extraction experiment based on facial key points.

[0122] In this experiment, the four common attention mechanisms SimAM, GCT, SE, and GAM are used to optimize the backbone network of YOLOv8n-pose and compared with YOLOv8n-pose and YOLOv8s-pose algorithms. The algorithm's inference time for each image is used to measure the real-time performance of the algorithm, and Time is taken as the average value of 10 times; "+SimAM" means adding the SimAM attention mechanism module to the backbone network, "+SE" means adding the SE attention mechanism module to the backbone network, "+GCT" means adding the GCT attention mechanism module to the backbone network, and "+GAM" means adding the GAM attention mechanism module to the backbone network.

[0123] (2) Experimental results and analysis

[0124] The experimental results of head feature extraction are shown in Table 4.

[0125] Table 4 Experimental results of head feature extraction

[0126]

[0127] According to the experimental results, the network structure based on the YOLOv8n-pose network with the non-parameter SimAM attention mechanism has the best comprehensive effect in the current method. As with the extraction of eye and mouth features, it can be seen from the table that, on the basis of keeping the model lightweight, SimAM can effectively improve the original accuracy of the model compared with other algorithms. In summary, the YOLOv8-Keypoints formed by the improvement of the YOLOv8n-pose algorithm by SimAM can more effectively improve the accuracy of the model while improving the detection speed. In order to improve the extraction effect of multi-angle facial key points and facilitate the subsequent head posture calculation, it is reasonable to use YOLOv8-Keypoints.

[0128] 3. Eye and mouth state discrimination experiment

[0129] (1) Experimental design

[0130] The MobileNetv3 eye and mouth state discrimination network uses the multi-threaded input data framework provided by Tensorflow, and uses batch processing to disrupt the data sequence to improve the model training speed and accuracy. Before the training starts, the training cycle is set to 20, the batch size is set to 100, the initial learning rate is set to 0.01, and the adaptive momentum (Adam) optimizer is used. After the training is completed, the callback function is used to check the loss rate of the validation set, and the network model with the minimum loss rate is saved as the final training model. In order to verify the performance of MobileNetv3 in eye and mouth state discrimination, the performance of the algorithm was tested using the data set, and the average value of 10 tests was taken.

[0131] The experimental results of eye and mouth state discrimination are shown in Table 5.

[0132] Table 5 Experimental results of eye and mouth state discrimination

[0133]

[0134] According to the experimental results, the method performs well in various scenarios considered. The results are verified by comprehensive detection at different angles of 0°, 30°, 60°, and 90°, among which the results of eye and mouth feature state discrimination reach 89.77% and 90.58% respectively. The multi-angle eye and mouth state discrimination is realized, which can be applied to the driver fatigue state discrimination at various angles.

[0135] 4. Head posture discrimination experiment

[0136] (1) Experimental design

[0137] In order to verify the accuracy of the head posture identification method, the experimenters made 10 videos to simulate the driving status of the driver, including normal, nodding and various abnormal head posture behaviors. In order to verify the performance of the coordinate system transformation method in head state identification, the algorithm was tested using the data set, and the average value of 10 times was taken.

[0138] (2) Experimental results and analysis

[0139] The experimental results of head state discrimination are shown in Table 6.

[0140] Table 6 Experimental results of head state discrimination

[0141]

[0142] As shown in the table, according to the experimental results, the head state discrimination results obtained by comprehensive detection and verification at different angles of 0°, 30°, 60°, and 90° have achieved good results in nodding behavior and abnormal head state behavior, with F1 scores reaching 88.29% and 87.50% respectively. The head state discrimination at multiple angles is realized, which can be applied to the discrimination of driver fatigue state at various angles.

[0143] From the above description, it can be seen that the present invention proposes a fatigue driving detection method, system, computer device and storage medium, which provides significant benefits to the driver and can escort the driver while driving. The current method based on the driver's side state takes the actual application environment into consideration more comprehensively. The method used is a non-invasive, non-irritating and non-interference detection method. The detection method that adapts to multiple angles is less likely to block the driver's line of sight and affect driving behavior. The driver's eye, mouth and head features are well extracted and identified at multiple angles. First, in order to solve the problem of extracting eye and mouth ROI and distinguishing the state of eye and mouth under multiple angles, YOLOv8 was improved by adding the non-parameter attention mechanism SimAM, and YOLOv8-ROI and YOLOv8-Keypoints were obtained. YOLOv8-ROI realized the extraction of eye ROI and mouth ROI, and its average extraction accuracy of eye and mouth ROI in all angles of side state reached 98.7%, and the inference speed was 5.1ms, which can be used for real-time detection; and after successful extraction, MobileNetv3 was used to distinguish the state of eye and mouth, and its average discrimination accuracy of eye and mouth in all angles of side state reached 90.18%. Secondly, YOLOv8-Keypoints was used to locate facial key points at all angles, and its average accuracy reached 85.9%, and the inference speed was 3.1ms, which can be used for real-time detection; and the three-dimensional head posture was calculated according to the coordinate system conversion method to distinguish the head state, and its average discrimination accuracy of head state in all angles of side state reached 87.41%.

[0144] In summary, the method proposed in the present invention can detect blinking, yawning and head abnormalities at multiple angles. Compared with other general target detection models, it has the performance advantages of faster speed and higher detection accuracy. The present invention has conducted experiments at various angles of 0°, 30°, 60°, and 90°. The experimental results show that there is still room for improvement in the comprehensive average accuracy and inference speed. The experiments can be further subdivided in the future, and model improvement and optimization can be considered to ensure the best effect in more difficult scenarios in actual application.

[0145] The embodiments described above are only preferred specific implementation modes of the present invention, and the protection scope of the present invention is not limited thereto. Any simple changes or equivalent replacements of the technical solutions that can be obviously obtained by any technician familiar with the field within the technical scope disclosed in the present invention belong to the protection scope of the present invention.

Claims

1. A method for detecting fatigue driving, characterized in that: The steps include: Acquire driving video data of the driver during driving, obtain video frame images from the video data, and construct eye datasets, mouth datasets, and head datasets based on the video frame images; Based on the YOLOv8n model, a parameter-free attention mechanism SimAM is added between the bottleneck layer C2f and the spatial pyramid pooling layer SPPF of the Backbone of the YOLOv8n model to obtain YOLOv8-SimAM; the detection layer Detect and the posture layer Pose are used as the output of YOLOv8-SimAM, respectively, and the network structure YOLOv8-ROI and the key point extraction network YOLOv8-Keypoints are obtained accordingly; The data in the eye dataset and mouth dataset are respectively input into YOLOv8-ROI for feature extraction, and the eye region of interest ROI images and mouth region of interest ROI images in the driver's multi-angle side state are obtained; the head dataset is input into YOLOv8-Keypoints for facial key point positioning, and the driver's head key point information at multiple angles is obtained; Input the eye ROI image and mouth ROI image into the MobileNetv3-Small neural network, classify the eye ROI image and mouth ROI image, and identify the open and closed states of the eyes and mouth at multiple angles; According to the key point information of the head, the three-dimensional posture change of the head is calculated using the coordinate system transformation method, and the head state is determined according to the three-dimensional posture change value; The driver's fatigue status is determined by comprehensively analyzing the opening and closing status of the eyes and mouth and the head status under multiple angles.

2. The fatigue driving detection method according to claim 1, characterized in that: The coordinate system transformation method is used to calculate the three-dimensional posture change of the head, including the following steps: Get the 3D world coordinate system of the key points of the head; Converting the three-dimensional world coordinate system into a two-dimensional camera coordinate system; Use the conversion formula to transform the two-dimensional camera coordinate system into the image coordinate system to obtain the rotation matrix; Solve the rotation matrix to get the Euler angles; Determine head pose changes based on Euler angles.

3. The fatigue driving detection method according to claim 2, characterized in that: The conversion formula is: Among them, f x 、f y 、c x 、c y is the camera intrinsic parameter matrix coefficient, r ij With t i are the rotation matrix and translation matrix, s is the scale factor, U, V and W are the coordinates of the world coordinate system, and x and y are the coordinate values ​​of the spatial point on the x-axis and y-axis respectively.

4. The fatigue driving detection method according to claim 2, characterized in that: The calculation formula of the Euler angle is as follows: Where α, β, and γ correspond to the x, y, and z rotation angles in the world coordinate system, respectively, and R is the rotation matrix. x (α) is the rotation matrix around the x-axis, R y (β) is the rotation matrix around the y-axis, R z (γ) is the rotation matrix around the z-axis.

5. The fatigue driving detection method according to claim 1, characterized in that: Before inputting the eye dataset and the mouth dataset into YOLOv8-ROI for feature extraction, YOLOv8-ROI needs to be operated as follows: Collect multi-angle side data sets of the driver at 0°, 30°, 60°, and 90°; Annotating the eye positions and mouth positions of the images in the multi-angle profile data set to obtain an annotated sample data set; The YOLOv8-ROI is trained using a labeled sample dataset.

6. The fatigue driving detection method according to claim 5, characterized in that: Before inputting the eye ROI image and the mouth ROI image into the MobileNetv3-Small neural network, the following steps are also included: Get the open source dataset CEW of closed and open eyes samples; Get the sample data set Yawn of the mouth's closed and open states; MobileNetv3-Small is trained using the CEW, Yawn and labeled sample data sets.

7. The fatigue driving detection method according to claim 1, characterized in that: Before inputting the head data set into YOLOv8-Keypoints for facial key point positioning, the wild human face facial key point data set WiderFace is used to train YOLOv8-Keypoints.

8. A fatigue driving detection system, characterized in that: include: The data set acquisition module is used to acquire driving video data of the driver during driving, and obtain video frame images from the video data, and construct eye data sets, mouth data sets and head data sets according to the video frame images; The model building module is used to add a parameter-free attention mechanism SimAM between the bottleneck layer C2f and the spatial pyramid pooling layer SPPF in the Backbone of the YOLOv8n model to obtain YOLOv8-SimAM; the detection layer Detect and the posture layer Pose are respectively used as the output of YOLOv8-SimAM, and the network structure YOLOv8-ROI and the key point extraction network YOLOv8-Keypoints are obtained accordingly; The feature extraction module is used to input the data in the eye dataset and the mouth dataset into YOLOv8-ROI for feature extraction, and obtain the eye region of interest ROI image and the mouth region of interest ROI image in the driver's multi-angle side state; input the head dataset into YOLOv8-Keypoints for facial key point positioning, and obtain the driver's head key point information at multiple angles; The first state discrimination module is used to input the eye ROI image and the mouth ROI image into the MobileNetv3-Small neural network, classify the eye ROI image and the mouth ROI image, and discern the open and closed states of the eyes and mouth in multiple angles; The second state discrimination module is used to calculate the three-dimensional posture change of the head according to the key point information of the head using the coordinate system conversion method, and to discriminate the head state according to the three-dimensional posture change value; The fatigue status analysis module is used to comprehensively analyze the opening and closing status of the eyes and mouth and the identified head status under multiple angles to obtain the driver's fatigue status.

9. A computer device, characterized in that: The method comprises a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the method according to any one of claims 1 to 7.

Images