Fatigue driving detection method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of face recognition, and discloses a fatigue driving detection method and device, computer equipment and a storage medium, and the method comprises the steps: collecting video stream data in a driving process, and employing a preset target detection model to position an initial face region based on the video stream data; fitting face feature points based on the initial face region, constructing a face feature triangle based on the face feature points, constructing a face feature vector based on the face feature triangle, and constructing a driver state analysis data set according to a time sequence; based on the state analysis data sets and a preset sliding window, obtaining facial feature vectors in all the state analysis data sets, and projecting all the facial feature vectors to a preset face projection reference plane to obtain a facial motion feature point set; and calculating a facial motion information entropy based on the facial motion feature point set to detect the driving state of the driver. The problems that the driving state of a driver cannot be detected in real time in a complex scene and the detection accuracy is low are solved.

Description

Technical Field

[0001] The present invention relates to the field of face recognition technology, and in particular to a fatigue driving detection method, device, computer equipment and storage medium. Background Art

[0002] In recent years, with the rapid development of the transportation industry, road traffic accidents have become one of the greatest threats to public safety, and driver fatigue has become a major cause of these accidents. Fatigue is a phenomenon characterized by physiological and psychological dysfunction in drivers, often occurring after prolonged driving. Fatigue can severely impact a driver's driving ability, posing a serious safety threat to both the driver and other road users. Research has shown that improving a driver's reaction time by 0.5 seconds could prevent approximately 60% of road traffic accidents. Therefore, research on driver fatigue detection algorithms is of great practical significance.

[0003] In recent years, research on driver fatigue detection algorithms has become a hot topic in the field of road traffic safety. Fatigue detection algorithms based on the driver's facial features use a standard camera to capture video images of the driver while driving. By analyzing the driver's facial images, the driver's fatigue state is objectively evaluated. This method is non-invasive, does not affect the driver's normal operation, and is relatively low-cost, making it a current research hotspot. With the development of computer technology, fatigue detection algorithms and systems have seen significant improvements and enhancements in accuracy, stability, and real-time performance. However, the following issues still exist:

[0004] 1) Physiological and behavioral detection methods typically require drivers to wear or install additional physiological information monitoring equipment, which affects the driver's driving comfort. Moreover, the equipment used to collect physiological information is usually expensive and easily damaged, which is not conducive to the popularization of fatigue driving systems.

[0005] 2) Vision-based detection methods typically use Adaboost (Adaptive Boosting, an ensemble learning algorithm) for face location. However, Adaboost cannot accurately locate faces when the driver is wearing glasses or daylight goggles, when lighting conditions change, or when the face is partially obscured, making it impossible to provide timely warnings for driver fatigue.

[0006] 3) Currently, most commonly used algorithms are based on the PERCLOS (Percentage of Eyelid Closure Overtime) metric, which uses the driver's eye opening and closing status to determine fatigue. However, when the driver's eyes are too small, the algorithm's misjudgment rate is high. Furthermore, other fatigue assessment metrics are less commonly used and less reliable.

[0007] Therefore, it is necessary to design a new driver fatigue detection method to solve the problem of being unable to realize real-time detection of the driver's driving status and low detection accuracy in complex scenarios. Summary of the Invention

[0008] In view of this, the present invention provides a fatigue driving detection method, device, computer equipment and storage medium to solve the problem that the driver's driving status cannot be detected in real time and the detection accuracy is low in complex scenarios.

[0009] In a first aspect, the present invention provides a method for detecting fatigue driving, the method comprising:

[0010] Collect video stream data of the driver during driving, and use a preset target detection model to locate the initial face area based on the video stream data;

[0011] Fitting facial feature points based on the initial face area, constructing facial feature triangles of different areas based on the facial feature points, and constructing facial feature vectors within a preset time period based on the facial feature triangles of different areas;

[0012] Based on the facial feature vectors within a preset time period, a driver status analysis dataset is constructed in chronological order;

[0013] Based on the state analysis data set and the preset sliding window, all facial feature vectors in all state analysis data sets are obtained, and all facial feature vectors are projected onto the preset face projection reference plane to obtain a facial motion feature point set;

[0014] The facial motion information entropy corresponding to the facial motion feature point set is calculated, and the driving state of the driver is detected based on the relationship between the facial motion information entropy and a preset threshold.

[0015] The present invention provides a fatigue driving detection method that uses a preset target detection model to locate the initial facial region. This method rapidly processes video stream data, ensuring detection accuracy while meeting the stringent real-time requirements of the in-vehicle environment and promptly responding to changes in the driver's state. Facial feature triangles and facial feature vectors are constructed based on facial feature points. This method not only considers the geometric relationships between key facial parts but also captures subtle facial changes through triangles of varying areas. This method provides a more comprehensive and sensitive reflection of the driver's facial state, providing rich information for subsequent analysis. Facial feature vectors are projected onto a preset facial projection reference plane, effectively eliminating interference from factors such as differences in facial size and shooting distance between drivers. This achieves feature normalization, making data from different drivers and under different shooting conditions comparable and improving the stability and universality of the detection system. A driver state analysis dataset is constructed based on a preset time period, and data is dynamically acquired using a preset sliding window. This allows the system to continuously and dynamically analyze the driver's state, promptly capturing the onset and progression of fatigue, and improving detection reliability. By calculating the facial motion information entropy corresponding to facial motion feature points, the complexity and activity of the driver's facial movements are quantified from the perspective of information theory. Combined with the preset threshold, the driving status is judged. This is in line with the physiological characteristics of reduced facial movements and distracted attention during fatigue driving. It can detect fatigue driving status in a more real-time, scientific and accurate manner, solving the problem of being unable to achieve real-time detection of the driver's driving status and low detection accuracy in complex scenarios.

[0016] In an optional embodiment, the preset target detection model is an improved YOLOv3-tiny network model, and the improved YOLOv3-tiny network model is constructed in the following manner:

[0017] Determine the convolution layer, maximum pooling layer, and sampling layer of the preset YOLOv3-tiny convolutional neural network according to the preset number, and use the determined convolution layer, maximum pooling layer, and sampling layer as the backbone feature extraction network of the preset YOLOv3-tiny convolutional neural network;

[0018] The cross loss function is used as the loss function for training the driver face or non-face classification task, and the mean square error loss function is used as the loss function for training the driver face bounding box fitting task. The joint loss function of the preset YOLOv3-tiny convolutional neural network is determined based on the cross loss function and the mean square error loss function.

[0019] The WIDER FACE dataset is used as driving data, and data preprocessing is performed on the WIDER FACE dataset. The preprocessed WIDER FACE dataset is divided into training set, test set, and validation set according to the preset ratio;

[0020] The preset YOLOv3-tiny convolutional neural network is trained based on the training set and the joint loss function. The trained preset convolutional neural network is tested and verified using the test set and validation set to obtain the detection results and the initial improved YOLOv3-tiny network model.

[0021] The detection results are decoded to obtain target classification detection and position regression. Based on the situation where there are multiple detection boxes for the same target in the target classification detection and position regression, the non-maximum suppression algorithm is used to select the optimal bounding box;

[0022] An improved YOLOv3-tiny network model is constructed based on the initial improved YOLOv3-tiny network model, target classification detection and position regression, and the optimal bounding box determined by the non-maximum suppression algorithm.

[0023] The present invention provides a fatigue driving detection method. The improved YOLOv3-tiny convolutional neural network optimizes the feature extraction layer, adjusts network parameters, or introduces new modules based on the characteristics of driving scenarios to enhance the recognition ability of drivers' faces under different postures and lighting conditions. The significant reduction in computational complexity directly leads to an improvement in operating efficiency. The improved YOLOv3-tiny convolutional neural network performs driver face detection, simplifying the network structure while ensuring a high face recognition accuracy.

[0024] In an optional embodiment, locating an initial face region from video stream data using a preset target detection model includes:

[0025] Extract a frame of data from the video stream and input the extracted frame of data into the improved YOLOv3-tiny network model for face detection to determine whether the face area is detected;

[0026] If a face region is detected, the face region is positioned as the initial face region; if no face region is detected, the next frame of data in the video stream is judged until a face region is detected.

[0027] The present invention provides a fatigue driving detection method, in which the YOLOv3-tiny network model is used as a lightweight target detection model with low computational complexity and fast detection speed. When processing video stream data, it is possible to quickly perform face detection on each frame of the image. Even in the face of high-frame-rate videos, it is possible to quickly complete the single-frame detection task, meeting the stringent real-time requirements in the vehicle-mounted environment, ensuring that the face area is located in a short time, and buying time for subsequent fatigue detection and other operations. By adopting a method of frame-by-frame detection and continuous judgment until the face area is detected, when faced with the situation in which single-frame face detection fails due to occlusion, lighting changes, abnormal posture and other factors in the video stream, the detection process will not be interrupted, effectively avoiding the problem of missed detection caused by local interference factors, and ensuring the integrity and reliability of face detection.

[0028] In an optional embodiment, facial feature points are fitted based on the initial face area, facial feature triangles of different areas are constructed based on the facial feature points, and facial feature vectors within a preset time period are constructed based on the facial feature triangles of different areas, including:

[0029] Based on the initial face region, a preset toolkit is used to fit multiple facial feature points in the face image;

[0030] Determining three endpoints of a plurality of facial feature triangles based on a plurality of facial feature points, and constructing facial feature triangles of different areas based on the corresponding three endpoints;

[0031] The area and center of gravity coordinates of the driver's facial feature triangle are calculated based on the facial feature triangle, and the facial feature vector within a preset time period is constructed based on the area and center of gravity coordinates.

[0032] The present invention provides a fatigue driving detection method that uses a preset toolkit to fit multiple facial feature points in a facial image. It can accurately locate key feature points such as the corners of the eyes, corners of the mouth, and the tip of the nose under different postures, expressions, and lighting conditions. The three endpoints of the facial feature triangle are determined based on the multiple facial feature points, and facial feature triangles of different areas are constructed. This method innovatively geometrizes facial features. Triangles of different areas can capture subtle changes in the face from multiple angles. By calculating the area and centroid coordinates of the facial feature triangle and constructing a facial feature vector within a preset time period, the facial features are quantitatively expressed. The area reflects the size change of the facial area, and the centroid coordinates reflect the location information of the distribution of facial features. The feature vector composed of the two converts complex facial states into calculable and analyzable numerical data in a concise and scientific way. Constructing the facial feature vector within a preset time period introduces the time dimension, making the facial feature analysis continuous and dynamic.

[0033] In an optional embodiment, based on the state analysis data set and a preset sliding window, all facial feature vectors in the state analysis data set are obtained, including:

[0034] Obtain a preset sliding window size and sliding step size, and start overwriting facial feature vectors of the preset sliding window size each time at the starting data position of the state analysis data set to obtain the facial feature vectors within the current sliding window;

[0035] After completing a facial feature vector covering a preset sliding window size, the preset sliding window is moved in the state analysis data set according to the sliding step size until the state analysis data set is traversed, and all facial feature vectors in the state analysis data set are obtained.

[0036] The present invention provides a fatigue driving detection method that dynamically captures facial feature vector data within different time periods by presetting a sliding window and sliding step size. The sliding window covers a certain number of facial feature vectors each time to obtain the driver's facial status information within the current time period. The sliding step size determines the frequency of data updates, which can capture subtle changes in the driver's status in real time, promptly detect the emergence and development trends of fatigue status, and avoid missed detections due to data processing lags. Starting from the starting position of the status analysis data set, the sliding window is moved according to fixed rules until the entire data set is traversed, ensuring that all facial feature vectors are processed and analyzed. This avoids the problem of information omission caused by improper data selection, enabling the system to fully grasp the driver's facial status information throughout the entire detection period, not missing any characteristic changes that may reflect fatigue status, thereby providing a complete data foundation for accurately judging driving status and effectively improving the reliability of fatigue driving detection results.

[0037] In an optional embodiment, all facial feature vectors are projected onto a preset face projection reference plane to obtain a facial motion feature point set, including:

[0038] The projection area of ​​the preset face projection reference plane is obtained, and all facial feature vectors are projected based on the projection area and the preset projection formula to obtain a set of facial motion feature points.

[0039] The present invention provides a method for detecting fatigue driving. Different drivers have naturally different facial sizes and shapes, and changes in shooting distance and angle will also cause the face to appear different in the image. By presetting a facial projection reference plane and a specific projection formula, all facial feature vectors can be uniformly mapped to a reference plane of the same area, normalizing these complex and variable facial feature data. Regardless of the size of the driver's face or the distance from the camera, after projection processing, the facial motion feature points are all under the same standard, effectively eliminating the interference of individual differences and different shooting conditions on feature analysis, making the data of different drivers comparable and improving the universality of the detection system.

[0040] In an optional embodiment, calculating facial motion information entropy corresponding to a facial motion feature point set, and detecting the driver's driving state based on a relationship between the facial motion information entropy and a preset threshold, includes:

[0041] Calculate the center point of the facial motion feature point set, and calculate multiple Euclidean distances from other facial motion feature points to the center point except the center point;

[0042] Calculate the mean and standard deviation of multiple Euclidean distances, and calculate the interval based on the mean and standard deviation;

[0043] Count the number of Euclidean distances falling within the interval, and calculate the facial motion information entropy based on the number and the number of facial motion feature points;

[0044] Based on the preset classifier, the facial motion information entropy under different driving conditions is trained to obtain the driver's facial motion information entropy classifier, and the real-time video stream data of the driver's driving process is input into the driver's facial motion information entropy classifier to obtain the predicted facial motion information entropy;

[0045] The driver's fatigue state or non-fatigue state is detected based on the relationship between the predicted facial motion information entropy and a preset threshold.

[0046] The present invention provides a fatigue driving detection method that converts the complex changes in the driver's facial movements into quantifiable numerical indicators by calculating the center point, Euclidean distance, mean, standard deviation, etc. of facial movement feature points. Facial movement information entropy, as a comprehensive measurement indicator, can keenly reflect the activity and regularity of the driver's facial muscle activity, head rotation, and other movements. Based on the mean and standard deviation, a specific interval is calculated, and the number of Euclidean distances falling within the interval is counted. This statistical interval analysis method effectively filters out noise and abnormal fluctuations in the data. A preset classifier is used to train the facial movement information entropy under different driving conditions to construct a driver's facial movement information entropy classifier. By learning a large amount of sample data under fatigue and non-fatigue conditions, the facial movement information entropy classifier can accurately grasp the information entropy feature patterns corresponding to different states. The real-time collected video stream data is input into the trained classifier to achieve real-time dynamic detection of the driver's status.

[0047] In a second aspect, the present invention provides a fatigue driving detection device, the device comprising:

[0048] The driver's face positioning module is used to collect video stream data of the driver during driving and use a preset target detection model to locate the initial face area based on the video stream data;

[0049] A driver feature vector extraction module is used to fit facial feature points based on the initial facial area, construct facial feature triangles of different areas based on the facial feature points, and construct facial feature vectors within a preset time period based on the facial feature triangles of different areas;

[0050] The driver state analysis data set establishment module is used for the facial motion feature point determination module, which is used to construct the driver state analysis data set in chronological order based on the facial feature vectors within a preset time period, obtain all facial feature vectors in all state analysis data sets based on the state analysis data set and a preset sliding window, and project all facial feature vectors onto a preset face projection reference plane to obtain a facial motion feature point set;

[0051] The driver fatigue determination module is used to calculate the facial motion information entropy corresponding to the facial motion feature point set, and detect the driver's driving status based on the relationship between the facial motion information entropy and a preset threshold.

[0052] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the fatigue driving detection method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0053] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the fatigue driving detection method of the first aspect or any corresponding embodiment thereof.

[0054] In a fifth aspect, the present invention provides a computer program product comprising computer instructions for causing a computer to execute the fatigue driving detection method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 is a flow chart of a fatigue driving detection method according to an embodiment of the present invention;

[0057] Figure 2 is a flow chart of another fatigue driving detection method according to an embodiment of the present invention;

[0058] Figure 3 is a flowchart of another fatigue driving detection method according to an embodiment of the present invention;

[0059] FIG4( a ) is a flow chart of another fatigue driving detection method according to an embodiment of the present invention;

[0060] FIG4( b ) is a flow chart of another fatigue driving detection method combined with a fatigue driving detection device according to an embodiment of the present invention;

[0061] Figure 5 is a structural block diagram of a fatigue driving detection device according to an embodiment of the present invention;

[0062] Figure 6 1 is a network structure diagram of an improved YOLOv3-tiny according to an embodiment of the present invention;

[0063] FIG7( a ) is a schematic diagram of acquiring facial feature points of a driver based on the Dlib toolkit according to an embodiment of the present invention;

[0064] FIG7( b ) is another schematic diagram of the effect of locating the facial feature points of a driver based on the Dlib toolkit according to an embodiment of the present invention;

[0065] FIG8( a ) is a schematic diagram of a facial feature triangle according to an embodiment of the present invention;

[0066] FIG8( b ) is a schematic diagram of the facial feature triangle construction effect according to an embodiment of the present invention;

[0067] Figure 9 is a schematic diagram of different facial motion states and FFV differences according to an embodiment of the present invention;

[0068] Figure 10 is a diagram showing a projection principle of a facial feature vector according to an embodiment of the present invention;

[0069] FIG11( a ) is a schematic diagram of a facial motion feature point set in a fatigue driving state according to an embodiment of the present invention;

[0070] FIG11( b ) is a schematic diagram of a facial motion feature point set in a normal driving state according to an embodiment of the present invention;

[0071] Figure 12 is a structural block diagram of another fatigue driving detection device according to an embodiment of the present invention;

[0072] Figure 13 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0073] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0074] According to an embodiment of the present invention, an embodiment of a fatigue driving detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0075] In this embodiment, a fatigue driving detection method is provided, which can be used on a computer device or a vehicle. A fatigue driving detection device is installed on the computer device or the vehicle, such as Figure 5 As shown, the device includes a driver face positioning module, a driver feature vector extraction module, a driver state analysis data set establishment module and a driver fatigue determination module, wherein:

[0076] Driver face positioning module: During fatigue state recognition, a camera is used to capture video of the driver's driving process. The captured video sequence is used as the original data source. Based on deep learning theory, an improved YOLOv3-tiny network model is applied to locate suspected face areas in complex backgrounds.

[0077] Driver Feature Vector Extraction Module: Based on the coordinates of the suspected face area, it searches for Dlib facial feature points and constructs the driver's FFT (Facial Feature Triangle). It then calculates the area and centroid coordinates of the driver's facial feature triangle. Based on the area and centroid coordinates, it constructs the driver's FFV (Facial Feature Vector).

[0078] Driver state analysis data set establishment module: extract the driver's facial feature vector FFV within a specified time, establish the driver state analysis data set according to the extraction time sequence, and project it onto the driver's face projection reference plane.

[0079] Driver fatigue determination module: Design a sliding window for driver fatigue analysis and move the sliding window on the state analysis data set to obtain the facial feature point set formed by all FFVs in the current sliding window, calculate the facial motion information entropy corresponding to the current facial feature point set, and compare the facial motion information entropy with the pre-set threshold to evaluate the driver's fatigue status.

[0080] Figure 1 FIG. 1 is a flow chart of a fatigue driving detection method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0081] Step S101: collect video stream data of the driver during driving, and use a preset target detection model to locate the initial face area based on the video stream data.

[0082] Specifically, the driver face positioning module is used to collect the video stream of the driver during the fatigue state recognition process with a camera. The collected video stream is used as the original data source. Combined with the basic theory of deep learning, the improved YOLOv3-tiny network model is applied to locate the suspected face area from the complex background.

[0083] Among them, the improved YOLOv3-tiny network model is built based on the convolutional neural network model. According to the regression idea of ​​the YOLO model, the regression problem of multiple targets is improved to the regression problem of a single target. The improved YOLOv3-tiny model is designed to achieve the positioning of suspected face areas.

[0084] Step S102 , fitting facial feature points based on the initial face area, constructing facial feature triangles of different areas based on the facial feature points, and constructing facial feature vectors within a preset time period based on the facial feature triangles of different areas.

[0085] Specifically, the driver feature vector extraction module applies the Dlib toolkit's facial landmark detection model based on the coordinates of the suspected facial region to extract fine-grained features of the driver's face. The Dlib toolkit contains 68 facial landmarks, and its detection principle uses cascaded shape regression to query key points of facial components.

[0086] The Dlib toolkit is used to find Dlib facial feature points and establish the driver's facial feature triangle (FFT). The area and centroid coordinates of the driver's facial feature triangle are then calculated. Finally, the driver's facial feature vector (FFV) is established based on the area and centroid coordinates.

[0087] FFT shows significant differences when the driver's face is in different motion states (i.e., the driver's eye gaze and head orientation are different). Furthermore, FFV also shows significant differences when the driver's face is in different motion states, consistent with FFV. Therefore, FFV can be used as a parameter to characterize facial motion state in driver fatigue detection algorithms.

[0088] Step S103: constructing a driver status analysis dataset in chronological order based on the facial feature vectors within a preset time period.

[0089] Specifically, the driver status analysis dataset is constructed based on the driver's facial feature information. It is a key data set used to analyze driving status and plays a core role in applications such as fatigue driving detection.

[0090] In terms of data structure, it uses the driver's facial feature vector (FFV) extracted within a preset time period as its basic unit. Each FFV contains geometric feature information such as the area and centroid coordinates of the facial feature triangle, recording the driver's facial state at a specific moment. These facial feature vectors are arranged and combined in the order of extraction time to form the driver state analysis dataset. For example, during a one-minute monitoring period, the system extracts 10 facial feature vectors per second. Therefore, this state analysis dataset contains 600 ordered facial feature vectors, fully presenting the evolution of the driver's facial state during that one-minute period.

[0091] The driver state analysis data set establishment module is used to extract the driver's facial feature vector FFV within a specified time. According to the extraction time sequence, the driver state analysis data set is established and projected onto the driver's face projection reference plane.

[0092] Step S104: Based on the state analysis data set and the preset sliding window, all facial feature vectors in all state analysis data sets are obtained, and all facial feature vectors are projected onto a preset face projection reference plane to obtain a facial motion feature point set.

[0093] Specifically, a sliding window is set with a specific sliding window size and sliding step size. The driver fatigue determination module is used to move the sliding window on the state analysis data set to obtain a facial feature point set formed by all FFVs in the current sliding window.

[0094] Step S105 , calculating the facial motion information entropy corresponding to the facial motion feature point set, and detecting the driver's driving state based on the relationship between the facial motion information entropy and a preset threshold.

[0095] Specifically, to perceive the road traffic scene in real time and make control decisions, the driver needs to observe areas of vision, such as the front of the vehicle and the left and right rearview mirrors. This requires the driver to constantly adjust their eye gaze and head direction. When the driver is not fatigued, their attention is focused and they are highly alert, and their gaze and head direction can usually be switched quickly. However, when fatigued, their attention is distracted, their organs and body are in a state of decline, and the switching of gaze and head direction becomes slower.

[0096] To compare the differences in the frequency and amplitude of gaze points and head orientations under the two driving conditions, facial motion feature point sets were statistically analyzed over several consecutive frames based on facial motion feature points. Intuitively, the facial feature motion point sets in the non-fatigue driving state are more divergent and chaotic than those in the fatigue driving state. To characterize this characteristic, this embodiment applies facial motion information entropy. When the driver is in a non-fatigue state, they are focused and highly alert, and their gaze points and head orientations can usually switch quickly. Therefore, the facial motion information entropy during normal driving is also large.

[0097] Therefore, the driver fatigue judgment module is used to calculate the facial motion information entropy corresponding to the current facial feature point set, and the facial motion information entropy is compared with the pre-set threshold to evaluate the driver's fatigue state.

[0098] The fatigue driving detection method provided in this embodiment uses a preset target detection model to locate the initial facial region. It can rapidly process video stream data, ensuring detection accuracy while meeting the stringent real-time requirements of the in-vehicle environment and promptly responding to changes in the driver's state. It constructs facial feature triangles and facial feature vectors based on facial feature points. This not only considers the geometric relationships between key facial parts but also captures subtle facial changes through triangles of varying areas. This provides a more comprehensive and sensitive reflection of the driver's facial state, providing rich information for subsequent analysis. Projecting the facial feature vectors onto a preset facial projection reference plane effectively eliminates interference from factors such as differences in facial size and shooting distance between drivers, achieving feature normalization and making data comparable across different drivers and shooting conditions, thereby improving the stability and universality of the detection system. By constructing a driver state analysis dataset based on a preset time period and dynamically acquiring data using a preset sliding window, the system can continuously and dynamically analyze the driver's state, promptly capturing the onset and progression of fatigue, and improving detection reliability. By calculating the facial motion information entropy corresponding to facial motion feature points, the complexity and activity of the driver's facial movements are quantified from the perspective of information theory. Combined with the preset threshold, the driving status is judged. This is in line with the physiological characteristics of reduced facial movements and distracted attention during fatigue driving. It can detect fatigue driving status in a more real-time, scientific and accurate manner, solving the problem of being unable to achieve real-time detection of the driver's driving status and low detection accuracy in complex scenarios.

[0099] In this embodiment, a fatigue driving detection method is provided, which can be used on a computer device or a vehicle. A fatigue driving detection device is installed on the computer device or the vehicle. The device includes a driver face positioning module, a driver feature vector extraction module, a driver state analysis data set establishment module, and a driver fatigue determination module. Figure 2 FIG. 1 is a flow chart of a fatigue driving detection method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0100] Step S201: collect video stream data of the driver during driving, and use a preset target detection model to locate the initial face area based on the video stream data.

[0101] Specifically, this embodiment uses the YOLOv3-tiny convolutional neural network. Based on the regression concept of the YOLO model, the regression problem of multiple targets is improved to a single target regression problem. An improved YOLOv3-tiny model is designed to locate the suspected face area.

[0102] The preset target detection model is the improved YOLOv3-tiny network model, which is constructed in the following way:

[0103] Step a1: Determine the convolution layer, maximum pooling layer, and sampling layer of the preset YOLOv3-tiny convolutional neural network according to the preset number, and use the determined convolution layer, maximum pooling layer, and sampling layer as the backbone feature extraction network of the preset YOLOv3-tiny convolutional neural network.

[0104] Specifically, the YOLO (You Only Look Once) model is a fast object detection model based on deep learning. It is a single, end-to-end network that transforms object detection into a regression problem. Specifically, it applies regression methods and convolutional neural networks (CNNs) to replace the sliding window and hand-crafted features used in traditional object detection to extract features of the driver's face. This feature extraction method is less affected by the external environment and can quickly extract target features.

[0105] In order to simplify the computational network design, reduce the amount of computation and port code, based on the regression idea of ​​the YOLO model, the regression problem of multiple targets is improved to the regression problem of a single target, and an improved YOLOv3-tiny network is designed. This achieves the positioning of the suspected face area. YOLOv3-tiny is a simplified version of YOLOv3. It reduces the YOLOv3-tiny feature detection network darknet-53 to 7 layers of conventional convolutional layers (Conv+BatchNorm+LeakyReLu), 6 layers of Max Pooling (maximum pooling layer) and 1 layer of Up Sample (sampling layer). The improved YOLOv3-tiny network is as follows: Figure 6 As shown, the improved YOLOv3-tiny model uses seven layers of conventional convolutional layers (Conv+BatchNorm+LeakyReLu), six layers of Max Pooling, and one layer of Up Sample as the backbone extraction network. This network not only efficiently extracts features from the input image, but also reduces the model's computational complexity and facilitates code porting. Conv (Convolution) is a convolutional layer. Batch Norm (Batch normalization) is a method used for regularization, accelerating convergence, and preventing overfitting. Leaky ReLU is an activation function. Concat is tensor concatenation. NMS (Non Maximum Suppression) is used to eliminate redundant driver face markers and find the optimal suspected driver face region.

[0106] In step a2, a cross loss function is used as the loss function for training the driver's face or non-face classification task, and a mean square error loss function is used as the loss function for training the driver's face bounding box fitting task, and a joint loss function of the preset YOLOv3-tiny convolutional neural network is determined based on the cross loss function and the mean square error loss function.

[0107] Specifically, the loss function used to train the driver face or non-face classification task is the cross entropy loss function, and the calculation formula is as follows:

[0108] L1=-y i log(p i )+(1-y i )(1-log(p i ))(1);

[0109] Among them, p i The probability that the network predicts the sample to detect the face area, y i is the true label of the sample.

[0110] The loss function used for training the driver's face bounding box fitting task is the mean square error loss function, and the calculation formula is as follows:

[0111] L2=||p i -y i || 2 (2);

[0112] Among them, p i Predict sample face bounding box coordinates for the network, y i is the true coordinate of the face bounding box in the sample.

[0113] Therefore, the calculation formula of the joint loss function involved in this embodiment is as follows:

[0114] L = αL1 + (1-α)L2 (3);

[0115] Among them, α is the weight coefficient.

[0116] In step a3, the WIDER FACE dataset is used as driving data, and data preprocessing is performed on the WIDER FACE dataset. The preprocessed WIDER FACE dataset is divided into a training set, a test set, and a validation set according to a preset ratio.

[0117] Specifically, the preset ratio is set according to actual conditions, such as 6:2:2 or 5:4:1.

[0118] During the offline training phase of the improved YOLOv3-tiny model, the present invention uses the WIDER FACE (Face Detection Data Set and Benchmark) dataset as the driving data. The previously prepared WIDER FACE dataset is converted to VOC format and partitioned into training, validation, and test sets using a 6:2:2 or 5:4:1 ratio. The images in the WIDER FACE dataset are then rescaled and fed into the model for training.

[0119] Among them, the WIDER FACE dataset includes a total of 32,203 images and 393,703 labeled faces. It is one of the current face databases, covering different scales, postures, occlusions, expressions, makeup, lighting, etc.

[0120] The WIDER FACE dataset has the following characteristics:

[0121] (1) The dataset is divided into three types: training set, test set and validation set, which account for 60%, 20%, 20% or 50%, 40%, 10% of the dataset respectively.

[0122] (2) There are a large number of faces in each image, with an average of 12.2 faces per image.

[0123] (3) The images have high resolution and are all color images.

[0124] In step a4, the preset YOLOv3-tiny convolutional neural network is trained based on the training set and the joint loss function, and the trained preset convolutional neural network is tested and verified using the test set and the validation set to obtain the detection results and the initial improved YOLOv3-tiny network model.

[0125] Specifically, the video is first converted into frames of images, the images are scaled to the size required by the model, and preprocessing operations such as normalization are performed. The preprocessed images are then input into the YOLOv3-tiny convolutional neural network to obtain a trained YOLOv3-tiny convolutional neural network. The trained YOLOv3-tiny convolutional neural network is then tested and verified through a test set and verification to obtain the detection results and the initial improved YOLOv3-tiny network model.

[0126] In step a5, the detection results are decoded to obtain target classification detection and position regression. Based on the situation where there are multiple detection boxes for the same target in the target classification detection and position regression, the optimal bounding box is selected using the non-maximum suppression algorithm.

[0127] Specifically, the detection results are decoded to obtain target classification detection and position regression; finally, for the case where there may be multiple detection frames for the same target, the optimal detection frame is selected through the non-maximum suppression algorithm.

[0128] In the target classification detection and position regression part, the WIDER FACE dataset images are first resized to 10 different sizes. Grid cells are then divided into 13×13 and 26×26 grids on the resized images. The driver's face is then located and classified in non-overlapping grid cells. For each grid cell, the network outputs B bounding boxes and corresponding confidence scores, as well as the conditional probability of the driver's face category. Finally, redundant bounding boxes are suppressed using non-maximum filtering. The confidence score formula is:

[0129]

[0130] Where: P r (Object) indicates whether there is a driver’s face target. If there is no driver’s face target, then P r (Object)=0; otherwise, P r (Object) = 1; is the intersection-over-union (IoU) of the bounding box and the ground-truth box.

[0131] Step a6: construct an improved YOLOv3-tiny network model based on the initial improved YOLOv3-tiny network model, target classification detection and position regression, and the optimal bounding box determined by the non-maximum suppression algorithm.

[0132] Specifically, the YOLOv3-tiny network loss function consists of a bounding box center error term, a bounding box width and height error term, a prediction confidence error term, and a prediction category error term. The improved YOLOv3-tiny network model, trained offline, regresses the bounding box center error term and the bounding box width and height error term. Based on these regressed values, the prior box is repositioned to locate the suspected driver's face area, providing an accurate facial image for subsequent algorithms.

[0133] The fatigue driving detection method provided in this embodiment uses an improved YOLOv3-tiny convolutional neural network to optimize the feature extraction layer, adjust network parameters, or introduce new modules based on the characteristics of driving scenarios. This enhances the ability to recognize drivers' faces under different postures and lighting conditions. This significantly reduces the amount of computation and directly improves operating efficiency. The improved YOLOv3-tiny convolutional neural network performs driver face detection, simplifies the network structure while ensuring a high face recognition accuracy, and greatly reduces the amount of computation.

[0134] The above step S201 includes:

[0135] Step S2011: extract a frame of data from the video stream, and input the extracted frame of data into the improved YOLOv3-tiny network model to perform face detection to determine whether a face area is detected.

[0136] Specifically, a frame of data is first extracted from the video stream, and the improved YOLOv3-tiny network model is used for face detection to determine whether a face is detected.

[0137] Step S2012: If a face region is detected, the face region is positioned as an initial face region; if no face region is detected, the next frame of data in the video stream is judged until a face region is detected.

[0138] Specifically, if no face is detected, the next frame of the video stream is processed. If a face is detected, facial feature points are located. If the feature points are successfully located, the FFV is calculated. If the feature points are not successfully located, the next frame of the video stream is processed. The process then determines whether the data in the video stream has been processed. If not, the next frame of the video stream is processed. This process continues until all frames of the video stream have been processed.

[0139] This embodiment provides a fatigue driving detection method, in which the YOLOv3-tiny network model is used as a lightweight target detection model with low computational complexity and fast detection speed. When processing video stream data, it is possible to quickly perform face detection on each frame of the image. Even in the face of high-frame-rate videos, it is possible to quickly complete the single-frame detection task, meeting the stringent real-time requirements in the vehicle environment, ensuring that the face area is located in a short time, and buying time for subsequent fatigue detection and other operations. By adopting a method of frame-by-frame detection and continuous judgment until the face area is detected, when faced with the situation where single-frame face detection fails due to occlusion, lighting changes, abnormal posture and other factors in the video stream, the detection process will not be interrupted, effectively avoiding the problem of missed detection caused by local interference factors, and ensuring the integrity and reliability of face detection.

[0140] Step S202 , fitting facial feature points based on the initial face area, constructing facial feature triangles of different areas based on the facial feature points, and constructing facial feature vectors within a preset time period based on the facial feature triangles of different areas.

[0141] Specifically, the above step S202 includes:

[0142] Step S2021: Fitting multiple facial feature points in the facial image using a preset toolkit based on the initial facial region.

[0143] Specifically, the default toolkit is the Dlib toolkit. Using the face region image obtained by the improved YOLOv3-tiny network model as input, the Dlib toolkit is used to fit the 68 facial feature points in the face image, as follows:

[0144] As shown in Figure 7(a), the Dlib toolkit's facial landmark detection model is applied to the driver's face region located by the improved YOLOv3-tiny network model to extract fine-grained features of the driver's face. The Dlib toolkit contains 68 facial landmarks, and its detection principle uses cascaded shape regression to query key points of facial components.

[0145] The Dlib toolkit's facial landmark detection algorithm first extracts features from the input image. It then maps these features to facial landmarks, including those for the facial contour, eyebrows, eyes, nose, and mouth, using a learned regressor. For example, when given an image, the algorithm generates an initial shape of key facial components. It then uses gradient boosting to reduce this initial shape, compares it to the true shape, and calculates the sum of squared errors between the two. Finally, the least squares method is used to minimize the error, which is then used to calculate the cascaded regressors at each level.

[0146] The Dlib toolkit's facial landmark detection method uses a set of regression trees to estimate facial feature points. This method is fast, with detection of all 68 key points on a face taking approximately 1ms. Furthermore, it can handle missing feature points in the training set. It uses a cascade of regressors. The algorithm's iterative process is shown in the following formula:

[0147]

[0148] Where T is the number of regression rounds, is the current shape estimate; each regressor R t (,) According to the input image I and To predict an increment, The initial shape used is the average shape of the training data. The update strategy employs the Gradient Boosting Decision Tree (GBDT) algorithm, which trains one weak classifier at a time. Each weak classifier's prediction is then weighted and combined to approximate the true value of the portion it is supposed to predict. After the driver's face is detected, the algorithm is used to obtain facial feature points in real time, as shown in Figure 7(b).

[0149] Step S2022: determining three endpoints of a plurality of facial feature triangles based on a plurality of facial feature points, and constructing facial feature triangles of different areas based on the corresponding three endpoints.

[0150] Specifically, after the driver's face is detected, the Dlib toolkit is used to obtain the facial feature points in real time. In view of this, this embodiment provides a facial feature triangle to characterize the driver's facial motion state. Specifically, based on the facial feature point positioning based on the Dlib toolkit, a facial feature triangle (Face FeatureTriangle, abbreviated as FFT) is defined, that is, the driver's left eye midpoint A, the right eye midpoint B and the mouth midpoint C are the three endpoints of the FFT respectively. The three endpoints are connected to form the FFT. The schematic diagram of the facial feature triangle and the construction effect is shown in Figures 8(a) and 8(b).

[0151] Step S2023: Calculate the area and center-of-gravity coordinates of the driver's facial feature triangle based on the facial feature triangle, and construct a facial feature vector within a preset time period based on the area and center-of-gravity coordinates.

[0152] Specifically, based on the facial feature triangle, the facial feature vector (FFV) is defined as follows:

[0153]

[0154] Among them: F x ,F y , S are shown in the following formulas:

[0155]

[0156] Among them, point (A x ,A y ), point (B x ,B y ), point (C x ,C y ) define the following formulas respectively:

[0157]

[0158] When the driver's face is in different motion states (i.e. the driver's eye gaze point and head direction are different), FFT has significant differences. Figure 9 As shown in Figure 3, FFV also shows significant differences when the driver's face is in different motion states, which is consistent with FFV. Therefore, FFV can be used as a parameter to characterize facial motion state in driver fatigue detection algorithms.

[0159] Step S203: Based on the facial feature vectors within a preset time period, a driver status analysis dataset is constructed in chronological order. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0160] Step S204: Based on the state analysis data set and the preset sliding window, all facial feature vectors in all state analysis data sets are obtained, and all facial feature vectors are projected onto the preset face projection reference plane to obtain a facial motion feature point set. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0161] Step S205: Calculate the facial motion information entropy corresponding to the facial motion feature point set, and detect the driver's driving state based on the relationship between the facial motion information entropy and a preset threshold. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.

[0162] The fatigue driving detection method provided in this embodiment uses a preset toolkit to fit multiple facial feature points in a facial image. It can accurately locate key feature points such as the corners of the eyes, corners of the mouth, and tip of the nose under different postures, expressions, and lighting conditions. The three endpoints of the facial feature triangle are determined based on the multiple facial feature points, and facial feature triangles of different areas are constructed. This method innovatively geometrizes facial features. Triangles of different areas can capture subtle facial changes from multiple angles. By calculating the area and centroid coordinates of the facial feature triangle and constructing a facial feature vector within a preset time period, facial features are quantitatively expressed. The area reflects the size change of the facial area, and the centroid coordinates reflect the location information of the distribution of facial features. The feature vector composed of the two converts complex facial states into calculable and analyzable numerical data in a concise and scientific manner. Constructing the facial feature vector within a preset time period introduces the time dimension, making facial feature analysis continuous and dynamic.

[0163] In this embodiment, a fatigue driving detection method is provided, which can be used on a computer device or a vehicle. A fatigue driving detection device is installed on the computer device or the vehicle. The device includes a driver face positioning module, a driver feature vector extraction module, a driver state analysis data set establishment module, and a driver fatigue determination module. Figure 3 FIG. 1 is a flow chart of a fatigue driving detection method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0164] Step S301: Collect video stream data of the driver during driving and use a preset target detection model to locate the initial face area based on the video stream data. Figure 2Step S201 of the illustrated embodiment will not be described in detail here.

[0165] Step S302: Fitting facial feature points based on the initial face area, constructing facial feature triangles of different areas based on the facial feature points, and constructing facial feature vectors within a preset time period based on the facial feature triangles of different areas. Figure 2 Step S202 of the illustrated embodiment will not be described in detail here.

[0166] Step S303: Based on the facial feature vectors within a preset time period, a driver status analysis dataset is constructed in chronological order. Figure 2 Step S203 of the illustrated embodiment will not be described in detail here.

[0167] Step S304: Based on the state analysis data set and the preset sliding window, all facial feature vectors in all state analysis data sets are obtained, and all facial feature vectors are projected onto a preset face projection reference plane to obtain a facial motion feature point set.

[0168] Specifically, by moving the sliding window on the state analysis data set, the facial feature point set formed by all FFVs in the current sliding window can be obtained, and the facial motion information entropy corresponding to the current facial feature point set can be calculated. The facial motion information entropy is compared with a pre-set threshold to evaluate the driver's fatigue state. The main parts can be divided into: (1) extracting facial motion information entropy; (2) designing a driver's facial motion information entropy classifier based on SVM; (3) fatigue judgment based on facial motion information entropy. The above step S304 includes:

[0169] Step S3041, obtain the preset sliding window size and sliding step size, and start covering the facial feature vectors of the preset sliding window size each time from the starting data position of the state analysis data set to obtain the facial feature vectors in the current sliding window.

[0170] Step S3042: After completing the facial feature vector covering the preset sliding window size, move the preset sliding window in the state analysis data set according to the sliding step size until the state analysis data set is traversed to obtain all facial feature vectors in the state analysis data set.

[0171] Specifically, a sliding window for driver fatigue analysis is designed and moved across the state analysis dataset to obtain facial feature vectors formed by all FFVs in the current sliding window. The sliding window includes the sliding window size and the sliding step size.

[0172] Step S3043: Obtain the projection area of ​​a preset face projection reference plane, and project all facial feature vectors based on the projection area and a preset projection formula to obtain a facial motion feature point set.

[0173] Specifically, when applying the facial feature triangle FFT, the distance between the driver's head and the camera interferes with the change of the center point of the facial feature triangle. To eliminate this interference, the present invention introduces the driver's face projection reference plane. Figure 10 As shown, for the same driver, facial feature triangles of different areas are projected onto a preset area as a projection reference plane.

[0174] The area of ​​the projection reference plane is set to S0. The extracted facial feature vector is projected onto the projection reference plane according to its eigenvalue. The specific projection formula is as follows:

[0175]

[0176] Where row and col are the number of rows and columns of the input image, respectively. The point (x, y) projected onto the reference projection plane is defined as the driver's facial motion feature point.

[0177] This embodiment constructs a facial motion feature point set for the driver's facial movements by counting facial motion feature points across a number of frames. This model then builds a statistical model of the driver's facial motion state, reflecting the level of activity and regularity of facial movements. The specific steps for constructing this statistical model can be found in related art and will not be detailed here.

[0178] Step S305 , calculating the facial motion information entropy corresponding to the facial motion feature point set, and detecting the driver's driving state based on the relationship between the facial motion information entropy and a preset threshold.

[0179] Specifically, to perceive the road traffic scene in real time and make control decisions, the driver needs to observe areas of vision, such as the front of the vehicle and the left and right rearview mirrors. This requires the driver to constantly adjust their eye gaze and head direction. When the driver is not fatigued, their attention is focused and they are highly alert, and their gaze and head direction can usually be switched quickly. However, when fatigued, their attention is distracted, their organs and body are in a state of decline, and the switching of gaze and head direction becomes slower.

[0180] To compare the differences in gaze and head orientation frequency and amplitude between the two driving states, facial motion feature points were collected over several consecutive frames. Figures 11(a) and 11(b) show the facial motion feature points for the fatigued and non-fatigued states, respectively. Intuitively, the facial motion feature points for the non-fatigued state are more dispersed and chaotic than those for the fatigued state.

[0181] Specifically, the above step S305 includes:

[0182] Step S3051 , calculating the center point of the facial motion feature point set, and calculating multiple Euclidean distances from other facial motion feature points to the center point except the center point.

[0183] Specifically, to perceive the road traffic scene in real time and make control decisions, the driver needs to observe areas of vision, such as the front of the vehicle and the left and right rearview mirrors. This requires the driver to constantly adjust their eye gaze and head direction. When the driver is not fatigued, their attention is focused and they are highly alert, and their gaze and head direction can usually be switched quickly. However, when fatigued, their attention is distracted, their organs and body are in a state of decline, and the switching of gaze and head direction becomes slower.

[0184] To compare the differences in gaze and head orientation frequency and amplitude between the two driving states, facial motion feature points were used as the basis for statistical analysis of facial motion feature point sets over several consecutive frames. Intuitively, the facial feature motion point sets in the non-fatigue driving state are more divergent and chaotic than those in the fatigue driving state. To characterize this characteristic, this embodiment applies information entropy and provides a mathematical expression for calculating information entropy:

[0185]

[0186] Based on facial feature point location achieved using the Dlib toolkit, the driver's FFV (Facial Feature Vector) was extracted to establish a state analysis dataset. Based on the concept of information entropy, the driver's facial motion information entropy was defined. From this, a quantity was established to assess the degree of disorder of the facial motion feature point set. The calculation method is as follows:

[0187] Calculate the center point of the facial feature point set N is the number of feature points in the facial motion feature point set.

[0188]

[0189] Calculate the Euclidean distance from each feature point except the center point to the center point, denoted as l i .

[0190]

[0191] Where i = 1, 2...N.

[0192] Step S3052: Calculate the mean and standard deviation of multiple Euclidean distances, and calculate the falling interval based on the mean and standard deviation.

[0193] Specifically, the mean distance μ is calculated according to the following formula: l and calculate the standard deviation σ of the distance l ;

[0194]

[0195] Step S3053, count the number of Euclidean distances that fall within the interval, and calculate the facial motion information entropy based on the number and the number of facial motion feature points, and input the real-time video stream data of the driver's driving process into the driver's facial motion information entropy classifier to obtain the predicted facial motion information entropy.

[0196] Specifically, the definition falls into interval I i , where i=1,2...i max ,i max As shown below;

[0197] I i =[(i-1)*μ l / σ l ,i*μ l / σ l ] (15);

[0198]

[0199] According to the distance from each feature point to the center point, the statistics fall into interval I i The number of distances is recorded as n i ; Calculate facial motion information entropy H according to the following formula F (X):

[0200]

[0201] Among them, p(x i ) is the probability of falling into the interval.

[0202] When a driver is not fatigued, they are focused and alert, and their gaze and head direction typically switch rapidly. Therefore, facial motion entropy during normal driving is also high. This embodiment calculates facial motion entropy under different driving conditions in a training set and uses this to train an SVM (Support Vector Machine) classifier to determine the driver's driving state.

[0203] SVM is a machine learning model that uses the structural risk minimization criterion under the framework of statistical learning theory. It is a linear classifier model defined with the largest interval in the feature space. Given a training data set S={(x i ,y i ),i=1,2,...N}, where x i ∈R d is the i-th input sample, y i ∈{+1,-1} is x i The corresponding label. When y i= +1, called x i is a positive sample, when y i = -1, called x i is a negative sample.

[0204] Usually, there exists a linear discriminant function f(x)=w in d-dimensional space T x i +b can distinguish the two types of data, and the classification hyperplane can be described as:

[0205] w *T x+b * =0(18);

[0206] Normal vector w T The intercept b can determine the super classification surface function. According to the basic idea of ​​SVM, the constrained optimization problem of linearly separable support vector machine can be obtained, and the formula is as follows:

[0207]

[0208] In summary, during the training phase of the driver’s face marking frame, the improved YOLOv3-tiny network model is used as the training network, and the training set is used to detect the driver’s face. Based on the positioning information of the facial feature points in the Dlib toolkit, the driver’s facial motion information entropy is calculated. i = +1, x i is a positive sample, indicating that the driver is in a non-fatigue driving state; when y i = -1, x i is a negative sample, indicating that the driver is in a fatigue driving state. Combined with the constraints of the linearly separable support vector machine, the hyperplane parameter w can be solved T and b. Thus, the driver's facial motion information entropy classifier is designed.

[0209] Experiments have shown that different values ​​of the projection reference area S0 will affect the parameter w of the driver's facial motion information entropy classifier. T and b, S0=10000 was selected in the experiment.

[0210] Step S3054: Detecting the driver's fatigue state or non-fatigue state based on the relationship between the predicted facial motion information entropy and a preset threshold.

[0211] Specifically, the preset threshold is the driver fatigue state facial motion information entropy judgment threshold

[0212] The original image of driver monitoring is obtained by the on-board camera, and the improved YOLOv3-tiny network model is used to detect the driver's face. If the driver's face is detected, the facial area is cropped as the input sub-image, and the feature points of the face are marked using the Dlib toolkit. In order to reduce the false detection rate of the detection system, when the improved YOLOv3-tiny network model does not detect the face or the facial feature point positioning fails or the facial feature points are inaccurate due to excessive side face, this frame image is discarded as a failed frame and does not participate in the subsequent fatigue driving judgment. Based on the facial feature point marking, the facial feature vector FFV is calculated according to the coordinates of the eye feature points and the mouth feature point coordinates. Within a certain number of frames (the present invention sets the number of frames to more than 1000 frames), the facial feature vector FFV of each frame image of the driver is counted, and then a sliding window is set, the window size is set to 1000, and the sliding step size is set to 100. Each time the sliding window slides, the facial feature vector FFV of 1000 frames in the current sliding window is obtained first, then the facial motion feature point set of the current window is obtained, and finally the facial motion information entropy H of the current window is calculated. F (X), if H F (X) <Th HF(X) , Th HF(X) is the facial motion information entropy judgment threshold of the driver's fatigue state, then the driver is judged to be in a fatigue state. Otherwise, the current window is judged to be in a non-fatigue state, and the sliding window is moved to calculate the facial motion information entropy value of the next window until all facial motion feature points are judged.

[0213] The fatigue driving detection method provided in this embodiment converts the complex changes in the driver's facial movements into quantifiable numerical indicators by calculating the center point, Euclidean distance, mean, standard deviation, and other factors of facial motion feature points. Facial motion information entropy, as a comprehensive measurement indicator, can keenly reflect the level of activity and regularity of the driver's facial muscle activity, head rotation, and other movements. A specific interval is calculated based on the mean and standard deviation, and the number of Euclidean distances falling within this interval is counted. This statistical interval analysis method effectively filters out noise and abnormal fluctuations in the data. A preset classifier is used to train the facial motion information entropy under different driving conditions to construct a driver facial motion information entropy classifier. By learning from a large amount of sample data under fatigue and non-fatigue conditions, the facial motion information entropy classifier can accurately grasp the information entropy feature patterns corresponding to different states. Real-time video stream data collected in real time is input into the trained classifier to achieve real-time dynamic detection of the driver's state.

[0214] As one or more specific application embodiments of the present invention, Figures 4(a) to 11(b) The fatigue driving detection method provided by the present invention is further described in detail, specifically including:

[0215] The fatigue driving detection method provided in this embodiment is applied to a fatigue driving detection device, such as Figure 5 As shown, it includes 4 modules: driver face positioning module, driver feature vector extraction module, driver state analysis data set establishment module and driver fatigue judgment module. The flow chart of the method of the present invention is shown in Figure 4 (a) and Figure 4 (b): Input the video stream, first extract a frame of data in the video stream, use the YOLOv3-tiny network to perform face detection, and judge whether a face is detected. If no face is detected, the next frame of video stream data is judged; if a face is detected, the facial feature points are located. If the feature points are successfully located, FFV is calculated; if the feature points are not successfully located, the next frame of video stream data is judged. Judge whether the video ends, if not, the next frame of video stream data is judged; if it ends, the FFV data set is constructed through the sliding window, and the facial motion information entropy H is calculated. F (X), if is the threshold for judging the facial motion information entropy of the driver’s fatigue state, then the driver is judged to be in a fatigue state. Otherwise, the current window is judged to be in a non-fatigue state, and the facial motion information entropy value of the next window is calculated by sliding the window:

[0216] Step S1: During offline training, the WIDER FACE (Face Detection Data Set and Benchmark) dataset is used as the driving data. Combined with basic deep learning theory, this data is fed into an improved YOLOv3-tiny network for training to obtain face detection weights. During online recognition, a camera captures driving video, using the captured video sequence as the raw data source. The improved YOLOv3-tiny network is then applied to locate suspected faces within a complex background.

[0217] Step S1 includes the following two steps:

[0218] Step S1.1, offline training process:

[0219] Using the WIDER FACE dataset as driving data, the dataset is first converted into VOC format and divided into training, validation, and test sets in a ratio of 6:2:2 or 5:4:1. The images in the WIDER FACE dataset are then resized to 10 different sizes and the data is preprocessed. Finally, the processed dataset is input into a convolutional neural network for training to obtain face detection weights.

[0220] like Figure 6As shown in the figure, the convolutional neural network used in the present invention, based on the regression concept of the YOLO model, improves the regression problem of multiple targets into a single target regression problem, designs an improved YOLOv3-tiny model, and thereby realizes the location of suspected face areas. Among them, the loss function used for training the driver face / non-face classification task is the cross entropy loss function, which is calculated as follows:

[0221] L1=-y i log(p i )+(1-y i )(1-log(p i ))(1);

[0222] Among them, p i The probability that the network predicts the sample to detect the face area, y i is the true label of the sample.

[0223] The loss function used for training the driver's face bounding box fitting task is the mean square error loss function, and the calculation formula is as follows:

[0224] L2=||p i -y i || 2 (2);

[0225] Among them, p i Predict sample face bounding box coordinates for the network, y i is the true coordinate of the face bounding box in the sample.

[0226] Therefore, the calculation formula of the joint loss function involved in the present invention is as follows:

[0227] L = αL1 + (1-α)L2 (3);

[0228] Among them, α is the weight coefficient.

[0229] Step S1.2, online recognition process:

[0230] First, the video is converted into frame-by-frame images, which are then scaled to the model's required size and subjected to preprocessing operations such as normalization. The preprocessed images are then fed into a convolutional neural network to generate detection results. The detection results are then decoded to obtain object classification detection and position regression. Finally, when multiple detection boxes may exist for the same object, the optimal one is selected using a non-maximum suppression algorithm.

[0231] In the target classification detection and position regression phase, the input image data is divided into a grid cell. The driver's face is then located and classified within non-overlapping grid cells. For each grid cell, the network outputs B bounding boxes, a corresponding confidence score, and the conditional probability of the driver's face class. Finally, redundant bounding boxes are suppressed using non-maximum filtering. The confidence score formula is:

[0232]

[0233] Where: P r (Object) indicates whether there is a driver’s face target. If there is no driver’s face target, then P r (Object)=0; otherwise, P r (Object) = 1; is the intersection-over-union (IoU) of the bounding box and the ground-truth box.

[0234] In step S2, based on the coordinates of the suspected face area, the Dlib toolkit facial feature points are searched and the driver's FFT (facial feature triangle) is established. The area and centroid coordinates of the driver's facial feature triangle are then calculated. Finally, the driver's FFV (facial feature vector) is established based on the area and centroid coordinates.

[0235] The above step S2 includes:

[0236] In step S2.1, as shown in Figure 7(a) and Figure 7(b), a face detection algorithm is applied to obtain a face image in a complex driving scene. Then, the obtained face image is used as input and the Dlib toolkit is used to fit 68 feature points in the facial image.

[0237] In step S2.2, based on facial feature point location using the Dlib toolkit, a facial feature triangle (FFT) is defined. The driver's left eye midpoint A, right eye midpoint B, and mouth midpoint C are the three endpoints of the FFT. These three endpoints are connected to form the FFT, as shown in Figures 8(a) and 8(b).

[0238] Step S2.3: construct a facial feature vector (FFV) based on the facial feature triangle.

[0239]

[0240] Among them: F x ,F y , S are shown in the following formulas:

[0241]

[0242] Among them, point (A x ,A y ), point (B x ,B y ), point (C x ,C y ) is defined as follows:

[0243]

[0244] like Figure 9 As shown in FIG, when the driver's face is in different motion states (ie, the driver's eye gaze point and head direction are different), the FFT has significant differences.

[0245] Step S3: extract the driver's facial feature vector FFV within a specified time period, establish a driver state analysis data set in the order of extraction time, and project it onto the driver's face projection reference plane.

[0246] The above step S3 includes:

[0247] Step S3.1: Based on the FFVs extracted within a certain period of time, a driver status analysis dataset is established in the order of extraction time.

[0248] Step S3.2: Introduce the driver's face projection reference plane. For the same driver, project the facial feature triangles of different areas onto the pre-set projection reference plane. Figure 10 As shown, the area of ​​the projection reference plane is set to S0, and projection is performed on the projection reference plane according to the eigenvalues ​​of the extracted facial feature vectors. The specific projection formula is as follows:

[0249]

[0250] Where row and col are the number of rows and columns in the input image, respectively. The point (x, y) projected onto the reference projection plane is defined as the driver's facial motion feature point. This method constructs a feature point set for the driver's facial motion by counting feature points from several frames, and then constructs a statistical model of the driver's facial motion state to reflect the activity and regularity of facial movements.

[0251] In step S4, the sliding window is moved on the state analysis data set to obtain the facial feature point set formed by all FFVs in the current sliding window, and the facial motion information entropy corresponding to the current facial feature point set is calculated. The facial motion information entropy is compared with the pre-set threshold to evaluate the driver's fatigue state.

[0252] The above step S4 includes:

[0253] Step S4.1, extracting facial motion information entropy:

[0254] To perceive the traffic scene in real time and make control decisions, drivers need to observe areas of vision, such as the front of the vehicle and the left and right rearview mirrors. This requires drivers to constantly adjust their gaze and head direction. When drivers are not fatigued, they are focused and alert, and their gaze and head direction can usually be quickly adjusted. However, when fatigued, their attention is distracted, their organs and body are in a state of decline, and these adjustments become slower.

[0255] To compare the differences in gaze and head orientation frequency and amplitude between the two driving states, facial motion feature points were statistically analyzed over several consecutive frames. Intuitively, the facial motion feature point set is more dispersed and chaotic in the non-fatigue state compared to the fatigue state. To characterize this characteristic, information entropy was applied, and the mathematical expression for calculating information entropy was given:

[0256]

[0257] Based on Dlib facial feature point positioning, we extracted the driver's FFV (Facial Feature Vector) and established a state analysis dataset. Based on the concept of information entropy, we defined the driver's facial motion information entropy. From this, we established a metric to assess the degree of disorder in the facial motion feature point set. The calculation method is as follows.

[0258] Calculate the center point of the facial motion feature point set N is the number of feature points in the facial motion feature point set.

[0259]

[0260] Find the Euclidean distance from each facial motion feature point to the center point, denoted as l i , where i = 1, 2...N.

[0261]

[0262] Calculate the mean μ of the distance l and calculate the standard deviation σ of the distance l ;

[0263]

[0264]

[0265] Define interval I i , where i=1,2...i max ,i max As shown in the following formula:

[0266] I i=[(i-1)*μ l / σ l ,i*μ l / σ l ] (15);

[0267]

[0268] According to the distance from each feature point to the center point, the statistics fall in interval I i The number of distances is recorded as n i ;

[0269] Calculate facial motion information entropy H F (X):

[0270]

[0271] Step S4.2, design of driver's facial motion information entropy classifier based on SVM:

[0272] When a driver is not fatigued, they are focused and alert, and their gaze and head direction can usually switch quickly. Therefore, the facial motion information entropy during normal driving is also high. The present invention calculates the facial motion information entropy under different driving states in the training set and uses it to train an SVM classifier to determine the driver's driving state.

[0273] SVM is a machine learning model that uses the structural risk minimization criterion under the framework of statistical learning theory. It is a linear classifier model defined with the largest interval in the feature space. Given a training data set S={(x i ,y i ),i=1,2,...N}, where x i ∈R d is the i-th input sample, y i ∈{+1,-1} is x i The corresponding label. When y i = +1, called x i is a positive sample, when y i = -1, called x i is a negative sample.

[0274] Usually, there exists a linear discriminant function f(x)=w in d-dimensional space T x i +b can distinguish the two types of data, and the classification hyperplane can be described as:

[0275] w *T x+b * =0(18);

[0276] Normal vector w TAnd the intercept b can determine the super classification surface function. According to the basic idea of ​​SVM, the constrained optimization problem of linearly separable support vector machine can be obtained:

[0277]

[0278] In the driver's face marking frame training phase, the improved YOLOv3-tiny is used as the training network and the training set is used to detect the driver's face. Based on the positioning information of the facial feature points in the Dlib toolkit, the driver's facial motion information entropy is calculated. i = +1, x i is a positive sample, indicating that the driver is in a non-fatigue driving state; when y i = -1, x i is a negative sample, indicating that the driver is in a fatigue driving state. Combined with the constraints of the linearly separable support vector machine, the hyperplane parameter w can be solved T and b. Thus, the driver's facial motion information entropy classifier is designed.

[0279] Experiments have shown that different values ​​of the projection reference area S0 will affect the parameter w of the driver's facial motion information entropy classifier. T and b. In the experiment, S0=10000 was selected.

[0280] Step S4.3, fatigue determination based on facial motion information entropy is specifically as follows:

[0281] The original image of driver monitoring is obtained by the on-board camera, and the improved YOLOv3-tiny network is used to detect the driver's face. If the driver's face is detected, the facial area is cropped as the input sub-image, and the facial feature points are marked using Dlib. In order to reduce the false detection rate of the detection system, when the improved YOLOv3-tiny network does not detect the face or the facial feature point positioning fails or the facial feature points are inaccurate due to excessive side face, this frame image is discarded as a failed frame and does not participate in the subsequent fatigue driving judgment. Based on the facial feature point marking, the FFV is calculated according to the coordinates of the eye feature points and the mouth feature points. Within a certain number of frames (the present invention sets the number of frames to more than 1000 frames), the FFV of each frame image of the driver is counted, and then a sliding window is set with the window size set to 1000 and the sliding step size set to 100. Each time the sliding window slides, the FFV of 1000 frames in the current sliding window is first obtained, then the facial motion feature point set of the current window is obtained, and finally the facial motion information entropy H of the current window is calculated. F (X), if is the facial motion information entropy judgment threshold of the driver's fatigue state, then the driver is judged to be in a fatigue state. Otherwise, the current window is judged to be in a non-fatigue state, and the sliding window is calculated to calculate the facial motion information entropy value of the next window.

[0282] To perceive the traffic scene in real time and make control decisions, drivers need to observe areas of vision, such as the front of the vehicle and the left and right rearview mirrors. This requires drivers to constantly adjust their gaze and head direction. When drivers are not fatigued, they are focused and alert, and their gaze and head direction can usually be quickly adjusted. However, when fatigued, their attention is distracted, their organs and body are in a state of decline, and these adjustments become slower.

[0283] To compare the differences in gaze and head orientation frequency and amplitude between the two driving states, facial motion feature points were collected over several consecutive frames. Figures 11(a) and 11(b) show the facial motion feature points for the fatigued and non-fatigued states, respectively. Intuitively, the facial motion feature points for the non-fatigued state are more dispersed and chaotic than those for the fatigued state.

[0284] The method design of this embodiment improves the YOLOv3-tiny convolutional neural network for driver face detection, simplifying the network structure while ensuring a high face recognition accuracy, greatly reducing the amount of calculation. According to the Dlib toolkit, facial feature points are extracted, and the position characteristics of the driver's eyes and mouth are analyzed to establish the driver's FFT (facial feature triangle) and the driver's FFV (facial feature vector) based on the FFT area and center of gravity coordinates. In order to eliminate the interference of FFT area size changes on fatigue driving discrimination, the driver's face projection reference plane is introduced, and the projection principle is applied to extract the motion feature points of the driver's face. Based on the motion feature points, the present invention also provides a facial motion information entropy index to quantitatively characterize the degree of confusion of the driver's facial motion feature point set, and judge the driver's fatigue state based on this. The present invention can be deployed on a vehicle, requires less equipment, is flexible to deploy, and has low cost. It can significantly improve driving safety and enhance the level of intelligence in the transportation field.

[0285] This embodiment also provides a fatigue driving detection device for implementing the above-mentioned embodiments and preferred embodiments. Details already described will not be repeated. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0286] This embodiment provides a fatigue driving detection device, such as Figure 12 Shown, including:

[0287] The driver face positioning module 1201 is used to collect video stream data of the driver during driving and use a preset target detection model to locate the initial face area based on the video stream data.

[0288] The driver feature vector extraction module 1202 is used to fit facial feature points based on the initial facial area, construct facial feature triangles of different areas based on the facial feature points, and construct facial feature vectors within a preset time period based on the facial feature triangles of different areas.

[0289] The driver state analysis data set establishment module 1203 is used for the facial motion feature point determination module, which is used to construct the driver state analysis data set in chronological order based on the facial feature vectors within a preset time period, obtain all facial feature vectors in all state analysis data sets based on the state analysis data set and the preset sliding window, and project all facial feature vectors onto the preset face projection reference plane to obtain a facial motion feature point set.

[0290] The driver fatigue determination module 1204 is used to calculate the facial motion information entropy corresponding to the facial motion feature point set, and detect the driver's driving state based on the relationship between the facial motion information entropy and a preset threshold.

[0291] In some optional implementations, the driver face positioning module 1201 includes:

[0292] The data input unit is used to extract a frame of data from the video stream and input the extracted frame of data into the improved YOLOv3-tiny network model for face detection to determine whether a face area is detected.

[0293] The face region detection unit is used to locate the face region as the initial face region if a face region is detected; if no face region is detected, the next frame data in the video stream is judged until a face region is detected.

[0294] In some optional implementations, the driver feature vector extraction module 1202 includes:

[0295] The facial feature point fitting unit is used to fit multiple facial feature points in the facial image based on the initial facial region using a preset toolkit.

[0296] The facial feature triangle construction unit is used to determine three endpoints of multiple facial feature triangles based on multiple facial feature points, and to construct facial feature triangles of different areas based on the corresponding three endpoints.

[0297] The facial feature vector construction unit is used to calculate the area and center of gravity coordinates of the driver's facial feature triangle based on the facial feature triangle, and to construct a facial feature vector within a preset time period based on the area and center of gravity coordinates.

[0298] In some optional implementations, the driver status analysis data set establishment module 1203 includes:

[0299] The sliding window facial feature vector acquisition unit is used to obtain a preset sliding window size and a sliding step size, and to overwrite the facial feature vector of the preset sliding window size each time starting from the starting data position of the state analysis data set to obtain the facial feature vector within the current sliding window.

[0300] The traversal unit is used to move the preset sliding window in the state analysis data set according to the sliding step size after completing a facial feature vector covering the preset sliding window size, until the state analysis data set is traversed and all facial feature vectors in the state analysis data set are obtained.

[0301] The projection unit is used to obtain the projection area of ​​a preset face projection reference plane, and project all facial feature vectors based on the projection area and a preset projection formula to obtain a facial motion feature point set.

[0302] In some optional implementations, the driver fatigue determination module 1204 includes:

[0303] The Euclidean distance calculation unit is used to calculate the center point of the facial motion feature point set and calculate multiple Euclidean distances from other facial motion feature points to the center point except the center point.

[0304] The falling interval calculation unit is used to calculate the mean and standard deviation of multiple Euclidean distances, and calculate the falling interval based on the mean and standard deviation.

[0305] The facial motion information entropy calculation unit is used to count the number of Euclidean distances falling within the interval and calculate the facial motion information entropy based on the number and the number of facial motion feature points.

[0306] The real-time detection unit is used to train the facial motion information entropy under different driving conditions based on a preset classifier to obtain a driver's facial motion information entropy classifier, and input the real-time video stream data of the driver's driving process into the driver's facial motion information entropy classifier to obtain the predicted facial motion information entropy.

[0307] The driver state detection unit is used to detect the driver's fatigue state or non-fatigue state based on the relationship between the predicted facial movement information entropy and a preset threshold.

[0308] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0309] The fatigue driving detection device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0310] The embodiment of the present invention also provides a computer device having the above Figure 12 The fatigue driving detection device shown.

[0311] See also Figure 13 , Figure 13 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 13 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 13 A processor 10 is taken as an example.

[0312] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0313] The memory 20 stores instructions that can be executed by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0314] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0315] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0316] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 13 The bus connection is taken as an example.

[0317] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, an indicator stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display, and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0318] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0319] A portion of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0320] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for detecting fatigue driving, characterized in that: The method comprises: Collecting video stream data of the driver during driving, and using a preset target detection model to locate the initial face area based on the video stream data; Fitting facial feature points based on the initial facial area, constructing facial feature triangles of different areas based on the facial feature points, and constructing facial feature vectors within a preset time period based on the facial feature triangles of different areas; Based on the facial feature vectors within a preset time period, a driver status analysis dataset is constructed in chronological order; Based on the state analysis data set and the preset sliding window, all facial feature vectors in all state analysis data sets are obtained, and all facial feature vectors are projected onto a preset face projection reference plane to obtain a facial motion feature point set; The facial motion information entropy corresponding to the facial motion feature point set is calculated, and the driving state of the driver is detected based on the relationship between the facial motion information entropy and a preset threshold.

2. The method according to claim 1, characterized in that The preset target detection model is an improved YOLOv3-tiny network model, which is constructed in the following way: Determine the convolution layer, maximum pooling layer, and sampling layer of the preset YOLOv3-tiny convolutional neural network according to the preset number, and use the determined convolution layer, maximum pooling layer, and sampling layer as the backbone feature extraction network of the preset YOLOv3-tiny convolutional neural network; Using a cross loss function as a loss function for training a driver face or non-face classification task, using a mean square error loss function as a loss function for training a driver face bounding box fitting task, and determining a joint loss function of a preset YOLOv3-tiny convolutional neural network based on the cross loss function and the mean square error loss function; Using the WIDER FACE dataset as driving data, performing data preprocessing on the WIDER FACE dataset, and dividing the preprocessed WIDER FACE dataset into a training set, a test set, and a validation set according to a preset ratio; The preset YOLOv3-tiny convolutional neural network is trained based on the training set and the joint loss function, and the trained preset convolutional neural network is tested and verified using the test set and the validation set to obtain the detection results and the initial improved YOLOv3-tiny network model; Decoding the detection results to obtain target classification detection and position regression, and selecting an optimal bounding box using a non-maximum suppression algorithm based on the presence of multiple detection boxes for the same target in the target classification detection and position regression; An improved YOLOv3-tiny network model is constructed based on the initial improved YOLOv3-tiny network model, target classification detection and position regression, and the optimal bounding box determined by the non-maximum suppression algorithm.

3. The method according to claim 1, characterized in that The method of using a preset target detection model to locate an initial face region from the video stream data includes: Extract a frame of data from the video stream and input the extracted frame of data into the improved YOLOv3-tiny network model for face detection to determine whether the face area is detected; If a face region is detected, the face region is positioned as an initial face region; if no face region is detected, the next frame of data in the video stream is judged until a face region is detected.

4. The method according to claim 1, wherein The step of fitting facial feature points based on the initial facial area, constructing facial feature triangles of different areas based on the facial feature points, and constructing facial feature vectors within a preset time period based on the facial feature triangles of different areas includes: Fitting multiple facial feature points in the facial image using a preset toolkit based on the initial facial region; Determining three endpoints of a plurality of facial feature triangles based on the plurality of facial feature points, and constructing facial feature triangles of different areas based on the corresponding three endpoints; The area and center-of-gravity coordinates of the driver's facial feature triangle are calculated based on the facial feature triangle, and a facial feature vector within a preset time period is constructed based on the area and center-of-gravity coordinates.

5. The method according to claim 1, wherein Based on the state analysis data set and the preset sliding window, all facial feature vectors in the state analysis data set are obtained, including: Obtaining a preset sliding window size and a sliding step size, and starting from the starting data position of the state analysis data set, overwriting the facial feature vector of the preset sliding window size each time to obtain the facial feature vector within the current sliding window; After completing a facial feature vector covering a preset sliding window size, the preset sliding window is moved in the state analysis data set according to the sliding step size until the state analysis data set is traversed, thereby obtaining all facial feature vectors in the state analysis data set.

6. The method according to claim 1, characterized in that Project all facial feature vectors onto the preset face projection reference plane to obtain a set of facial motion feature points, including: The projection area of ​​a preset face projection reference plane is obtained, and all the facial feature vectors are projected based on the projection area and a preset projection formula to obtain a facial motion feature point set.

7. The method according to claim 1, characterized in that Calculating facial motion information entropy corresponding to the facial motion feature point set, and detecting the driver's driving state based on a relationship between the facial motion information entropy and a preset threshold, including: Calculating the center point of the facial motion feature point set, and calculating multiple Euclidean distances from other facial motion feature points to the center point except the center point; Calculating the mean and standard deviation of the plurality of Euclidean distances, and calculating the falling interval based on the mean and standard deviation; Counting the number of Euclidean distances falling within the interval, and calculating facial motion information entropy based on the number and the number of facial motion feature points; Based on a preset classifier, facial motion information entropy under different driving conditions is trained to obtain a driver facial motion information entropy classifier, and real-time video stream data collected during the driver's driving process is input into the driver facial motion information entropy classifier to obtain predicted facial motion information entropy; The driver's fatigue state or non-fatigue state is detected based on the relationship between the predicted facial motion information entropy and a preset threshold.

8. A fatigue driving detection device, characterized in that: The device comprises: A driver face positioning module is used to collect video stream data of the driver during driving and locate the initial face area based on the video stream data using a preset target detection model; a driver feature vector extraction module, configured to fit facial feature points based on the initial facial region, construct facial feature triangles of different areas based on the facial feature points, and construct facial feature vectors within a preset time period based on the facial feature triangles of different areas; The driver status analysis data set establishment module is used for the facial motion feature point determination module, which is used to construct the driver status analysis data set in chronological order based on the facial feature vectors within a preset time period; The driver fatigue determination module is used to obtain all facial feature vectors in all state analysis data sets based on the state analysis data set and the preset sliding window, and project all facial feature vectors onto a preset face projection reference plane to obtain a facial motion feature point set; calculate the facial motion information entropy corresponding to the facial motion feature point set, and detect the driver's driving state based on the relationship between the facial motion information entropy and a preset threshold.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the fatigue driving detection method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the fatigue driving detection method according to any one of claims 1 to 7.

Images