Fatigue detection method based on machine learning and mixed features

Abstract

The invention relates to a fatigue detection method based on machine learning and mixed features, and belongs to the technical field of vehicle safety. Face recognition is realized by training a target detection algorithm YOLOv8, face key point detection is performed based on a MediaPipe framework, and after standardization processing of data is completed based on a subject independent method, a fatigue detection classification model is obtained through multiple machine learning modeling. A self-correction feature extraction module and a preprocessing module are further designed, classification models are integrated to construct a fatigue state classification module, a driver real-time fatigue detection system is formed, and automatic detection and classification of the driver fatigue state are achieved. According to the method, twenty facial features such as the sight line concentration ratio, the eye movement amplitude and the head movement are introduced, and the good generalization ability is achieved when fatigue detection is conducted on an unknown driver; a driver fatigue detection system is constructed by using integrated learning, and a lightweight real-time driver fatigue detection system is realized.

Description

Technical Field

[0001] The invention belongs to the technical field of vehicle safety and relates to a fatigue detection method based on machine learning and hybrid features. Background Art

[0002] Fatigue driving is one of the main causes of traffic accidents. In recent years, the academic community has conducted a lot of research in the field of fatigue detection, mainly focusing on detection methods based on vehicle signals, physiological signals and facial signals. Among them, the fatigue detection method based on the driver's facial signal shows broad application prospects due to its non-contact, low-cost and easy promotion characteristics. With the advancement of electronic hardware technology, this method has gradually become a research hotspot.

[0003] At present, most fatigue detection studies based on facial signals mainly rely on traditional features such as blinking frequency, yawning frequency, and nodding frequency. The following methods are often used to classify fatigue status: discrimination is achieved by setting thresholds, the extracted features are input into machine learning models for classification, or the images are directly input into deep learning models to complete classification. However, due to individual differences among drivers, different people will show diverse characteristics when in a fatigue state. The use of threshold methods faces the difficulty of numerically adjusting fixed thresholds or dynamic threshold ranges; using only traditional features to train machine learning models to identify fatigue states may lead to insufficient model generalization and poor robustness; and directly inputting images into deep learning models for classification has the problem of slow processing speed and difficulty in meeting the real-time requirements for deployment.

[0004] In view of the above problems, the present invention proposes a fatigue detection method based on machine learning and hybrid features. Based on traditional features, this method further introduces twenty facial features such as gaze concentration, eye movement amplitude, blink wave velocity and head movement, and realizes classification through machine learning modeling. This method not only enriches the accuracy of detection, but also effectively solves the model generalization problem caused by individual differences of drivers, thereby significantly improving the accuracy and robustness of fatigue detection, and further improving the real-time performance of the system. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a fatigue detection method based on machine learning and hybrid features. In view of the problem that traditional methods only consider traditional features and ignore the problem that different drivers will present diverse performances, this method introduces a total of 20 facial features such as line of sight concentration, eye movement amplitude, blink wave velocity and head movement, and realizes classification through machine learning modeling. It can have good generalization ability when performing fatigue detection on unknown drivers, further improving the accuracy and comprehensiveness of the driver fatigue detection system.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A fatigue detection method based on machine learning and hybrid features, comprising the following steps:

[0008] S1: Data collection: Obtain the UTA-RLDD dataset and process it into non-overlapping video segments;

[0009] S2: Face detection: Train the object detection algorithm YOLOv8 to achieve face recognition;

[0010] S3: Dataset construction: Face key point detection based on MediaPipe framework, feature calculation using key points, screening and labeling to build the dataset;

[0011] S4: Data preprocessing: Based on the subject-independent method LOPO, the continuous data is normalized, the continuous data input into the multi-layer perceptron is normalized, and the data with outliers is filtered out;

[0012] S5: Model training: Input the above processed experimental data into the constructed model, including random forest RF, multi-layer perceptron MLP and extreme gradient boosting XGBoost, obtain the fatigue detection classification model through autonomous learning training, and evaluate the model performance through evaluation indicators to obtain the optimal model;

[0013] S6: System construction: Design self-correcting feature extraction module and preprocessing module, build fatigue status classification module through voting method in ensemble learning, form real-time driver fatigue detection system, and realize automatic detection and classification of driver fatigue status.

[0014] Furthermore, S1 is specifically as follows: each video in the real-life fatigue dataset is divided into non-overlapping segments of 30 seconds, and each segment is used as an independent data sample; in order to avoid overfitting due to the high correlation of features in adjacent time periods, all segments are ensured to be non-overlapping during the segmentation process.

[0015] Further, the S2 comprises the following steps:

[0016] S2.1: The model is lightweight by replacing the YOLOv8 backbone network with Faster-Net and trained using the Wider-Face dataset. During deployment, the model is quantized using TensorRT technology to achieve hardware acceleration and low-latency face detection.

[0017] S2.2: Use YOLOv8 for face recognition and obtain the region of interest (ROI) to ensure that the face is always located in the center of the image, so as to reduce the impact of the change in the position of the face in the two-dimensional image on the detection of key point coordinates by the face mesh model.

[0018] Further, the S3 comprises the following steps:

[0019] S3.1: The MediaPipe framework is used to extract the 2D coordinates of 478 key points from the face region to complete feature calculation; these features can be divided into four categories: eye features, head features, mouth features, and sight features, a total of 20 features, and these features are represented as feature vectors as x, x∈R m ,m=20;

[0020] Among the non-traditional features, eye movement amplitude i , Velocity i The proportion of eye closure time in the time window Perclos is calculated by combining the eye aspect ratio EAR with the time window:

[0021] Formula (1) defines eye movement amplitude i Calculation method:

[0022]

[0023] Among them, EAR Start is the EAR value at the beginning of the blink process, EAR End is the EAR value at the end of the blink process, and EAR Peak It is the EAR value at the lowest point in the entire blinking process;

[0024] Formula (2) defines the eye opening velocity Velocity i Calculation method:

[0025]

[0026] Among them, Time Open The time it takes for an eye to go from closing to opening;

[0027] Formula (3) defines the calculation method of Perclos, the proportion of eye closure time in the time window:

[0028]

[0029] Among them, Time Close is the time the eyes are closed within the time window, Time Length is the length of the time window. The time window length used is 30s and the data frame rate used is 30FPS. Length The value of is 900;

[0030] Among the non-traditional features, the average head pitch angle PA, head activity HA, and head down time per unit time HD are calculated by combining the pitch angle α, yaw angle β, and rotation angle γ of the head posture with the time window:

[0031] Formula (4) defines the calculation method of the average head pitch angle (PitchAverage, PA), which characteristically interprets the numerical value of the subject's head drooping degree within the time window:

[0032]

[0033] Among them, α i represents the head pitch angle, i represents the frame number;

[0034] The head activity HA indicates the frequency of head orientation changes within the time window. The head orientation is defined by formula (5):

[0035]

[0036] Roll T 、Pitch T , Yaw T They are the values ​​of rotation angle, pitch angle and yaw angle in normal orientation, which are obtained by the self-correction module; define θ y >(Pitch T +15°) is head up, θ y <(Pitch T -15°) is head down; θ z >(Yaw T +15°) is left turn, θ z <(Yaw T -15°) is a right turn of the head; thus completing the estimation of the head posture and accurately detecting whether nodding and turning occur; when the head orientation changes, the value of the head activity HA will increase;

[0037] The head down time HD (Head Down) per unit time is defined by formula (6):

[0038]

[0039] Among them, HeadDown indicates the time that the head changes from other orientations to lowered orientation within the time window;

[0040] Among the non-traditional features, the gaze activity index GA and the fixation center time are obtained by estimating the posture of the eyeball:

[0041] GazeDir estimates the eyeball posture and obtains the Euler angle group: rotation angle P α , pitch angle Pβ , yaw angle P γ Formula (7) defines the calculation method of gaze posture GazeDir:

[0042] GazeDir=Rovtvec(EOR·HOR)=(P α ,P β ,P γ ) (7)

[0043] Among them, Rotvec is a function provided by scipy, which is used to convert the rotation matrix into the form of Rodriguez rotation vector; EOR represents the rotation angle of the eyeball, which is the relative rotation angle between the left and right boundaries of the eyeball in the two-dimensional image calculated by the face key point algorithm; HOR represents the rotation matrix of the head;

[0044] The specific direction of the line of sight is defined by equation (8):

[0045]

[0046] Among them, Pu T represents the pupil yaw angle P obtained by the self-correction module when the subject is looking at the center of the screen normally β The average value within the correction time window; when P β >Pu T +0.5 when the line of sight is biased to the left; when P β <Pu T When -0.5, the sight is biased to the right; in other cases, the sight is defined as the gaze center. When the sight is looking toward the center within the time window, the feature is introduced: the gaze center time, which increases when the sight is in the gaze center state;

[0047] In order to refine the extraction of sight features, the feature is introduced: the sight activity index GA, when P β When a certain amplitude of change occurs, GA will increase even if the specific posture change conditions are not met;

[0048] S3.3: The annotation method was completed by four students from the majors of electronic information and communication engineering, combining the original annotations provided by the real-life sleepiness dataset UTA-RLDD and the knowledge of the Karolinska Sleepiness Scale (KSS). They watched the video and added annotations in an independent environment. They were not allowed to communicate with each other during the work to ensure the independence of the annotations. When there was a deviation between the annotations and the original annotations, the data of the segment would be excluded and would no longer participate in subsequent experiments.

[0049] S3.4: The constructed dataset contains 1126 cases of awake state, 1084 cases of mild fatigue, and 1166 cases of severe fatigue, with a balanced distribution.

[0050] Further, the S4 comprises the following steps:

[0051] S4.1: The subject-independent method LOPO is used for standardization. The object of standardization is the continuous features in the data set. The mean and standard deviation are calculated by extracting the feature values ​​of each subject in the awake state, and the data provided by the corresponding subject are standardized. The specific calculation method is shown in formula (9):

[0052]

[0053] Among them, F n,m represents the standardized eigenvalue, F n,m represents the original eigenvalue, μ n,m With σ n,m Respectively represent the mean and standard deviation of the continuous data collected from feature n and subject m in the awake state.

[0054] Further, the S5 comprises the following steps:

[0055] S5.1: The data set is divided into two parts using the leave-out method, 10-fold cross validation, stratified cross validation, and independent subject validation (LOPO) method. The random forest RF, multi-layer perceptron MLP, and extreme gradient boosting XGBoost models trained by the corresponding training sets are evaluated based on the test sets generated based on different proportions, data balance, and independence to obtain the optimal model.

[0056] Further, the S6 comprises the following steps:

[0057] S6.1: Design of self-correction feature extraction module: The period after the detection starts is the correction window. During this window, the extracted eye aspect ratio EAR, mouth aspect ratio MAR, eyeball pitch angle P β and the rotation angle Roll when the head is facing normally T , Pitch T , Yaw T As calibration data, it is used to set the blink threshold, yawn threshold, head posture correction and sight line correction. The data collected within the calibration window is not input into the model. The default window length is thirty seconds.

[0058] The correction method of blink threshold and yawn threshold is shown in formula (10):

[0059]

[0060] Among them, A EAR is the blink detection threshold, is the maximum value of EAR in each blink within the calibration window. is the minimum value of EAR in each blink within the calibration window, N is the number of blinks within the calibration window, C EAR is the EAR adjustment factor, the value is 0.2; A MAR is the yawning threshold, is the maximum value of MAR in the correction window in each blink. is the minimum value of MAR in each blink within the correction window, C MAR is the MAR adjustment factor, the value is 0.7;

[0061] The head posture correction method is defined as formula (5), and the sight line correction method is defined as formula (8);

[0062] S6.2: Preprocessing module: After completing the self-calibration, the mean and standard deviation of the continuous features of the first 1 / 3 in the awake state are calculated based on the data collected during the calibration period, which are used as parameters for subsequent data standardization; the subsequent collected data are standardized by using the parameters to ensure the stability and accuracy of the model operation. The preprocessing method is defined as formula (9);

[0063] S6.3: Integrated model classification module: Save the trained optimal model as a persistent file, load the model file when using it, and classify the standardized features; through the integrated learning method, the classification results of multiple models are weighted to form the final prediction result, where the weight ratio of random forest RF, multi-layer perceptron MLP and extreme gradient boosting XGBoost is 2:1:3;

[0064] S6.4: The self-correcting feature extraction module, the preprocessing module, the integrated model classification module are combined with the target detection model and the face key point detection model to construct a real-time driver fatigue detection system.

[0065] The beneficial effects of the present invention are: in view of the problem that traditional methods only consider traditional features and ignore the diverse performance of different drivers, this method introduces a total of 20 facial features such as eye concentration, eye movement amplitude, blink wave speed and head movement, and realizes classification through machine learning modeling, which can have good generalization ability when performing fatigue detection on unknown drivers, further improving the accuracy and comprehensiveness of the driver fatigue detection system. Based on this method and integrated learning, a driver fatigue detection system is constructed, realizing a lightweight real-time driver fatigue detection system that can meet deployment requirements and is faster than traditional methods.

[0066] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0068] Figure 1 This is an overall flow chart of the fatigue detection method based on machine learning and hybrid features proposed in the present invention.

[0069] Figure 2 This is a block diagram of the fatigue detection system based on integrated learning and hybrid features of the present invention. DETAILED DESCRIPTION

[0070] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0071] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0072] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0073] like Figure 1 As shown, based on the traditional features, this method further introduces twenty facial features such as gaze concentration, eye movement amplitude, blink wave velocity and head movement, and achieves classification through machine learning modeling. This method not only enriches the accuracy of detection, but also effectively solves the model generalization problem caused by individual differences of drivers, thereby significantly improving the accuracy and robustness of fatigue detection, and further improving the real-time performance of the system. The technical solution adopted by the present invention to achieve the above-mentioned invention purpose includes the following steps:

[0074] S1: Data collection: Obtain the UTA-RLDD dataset and process it into non-overlapping video segments;

[0075] S2: Face detection: Train the object detection algorithm YOLOv8 to achieve face recognition;

[0076] S3: Dataset construction: Face key point detection based on MediaPipe framework, feature calculation using key points, screening and labeling to build the dataset;

[0077] S4: Data preprocessing: Based on the subject-independent method LOPO, the continuous data is normalized, the continuous data input into the multi-layer perceptron is normalized, and the data with outliers is filtered out;

[0078] S5: Model training: Input the above processed experimental data into the constructed model, including random forest RF, multi-layer perceptron MLP and extreme gradient boosting XGBoost, obtain the fatigue detection classification model through autonomous learning training, and evaluate the model performance through evaluation indicators to obtain the optimal model;

[0079] S6: System construction: Design self-correcting feature extraction module and preprocessing module, build fatigue status classification module through voting method in ensemble learning, form real-time driver fatigue detection system, and realize automatic detection and classification of driver fatigue status.

[0080] Furthermore, in S1:

[0081] S1.1: The real-life fatigue dataset UTA-RLDD is provided by sixty healthy subjects of different races, ages, and facial accessories. Each volunteer recorded three different categories of videos of about ten minutes in length: awake, mild fatigue, and severe fatigue. A total of 180 RGB videos with a total duration of thirty minutes are composed of the dataset; in this invention, each video in the real-life fatigue dataset is cut into 30-second non-overlapping segments, and each non-overlapping segment is used as an independent data sample. In order to avoid overfitting due to the high correlation of features in adjacent time periods, all segments are ensured to be non-overlapping during the segmentation process.

[0082] In S2:

[0083] S2.1: In the present invention, the backbone network of the target detection model YOLOv8 developed based on Torch is replaced with FasterNet to achieve the lightweight model, and the Wider-Face dataset is used for training. During the deployment process, TensorRT technology is used to achieve model quantization to achieve hardware acceleration and low-latency face detection.

[0084] S2.2: Use YOLOv8 to detect faces and obtain the region of interest (ROI) to ensure that the face input into the MediaPipe framework is always located in the center of the image, so as to reduce the impact of the different positions of the face in the two-dimensional image on the key point coordinate detection of the MediaPipe face mesh model.

[0085] In the S3:

[0086] S3.1: Calculate features based on the coordinates of facial key points, including the following steps:

[0087] The MediaPipe framework is used to extract the 2D coordinates of 478 key points from the face area to complete feature calculation. These features can be divided into four categories: eye features, head features, mouth features, and sight features, a total of 20 features, and these features are represented as feature vectors as x, x∈R m ,m=20, the calculated feature list is shown in Table 1.

[0088] Table 1

[0089] Feature number Feature Name 1 Blinking times 2 Eye closure duration 3 Average blink time 4 EAR average 5 Average eye movement amplitude 6 Maximum eye movement amplitude 7 Minimum eye movement amplitude 8 Average eye opening velocity 9 Maximum eye opening speed 10 Minimum eye opening velocity 11 Perclos 12 Eye activity 13 Eyesight Concentration 14 Number of dozes 15 Average head pitch angle 16 Head down time 17 Head activity 18 Number of yawns 19 Whether yawning occurs at a nearby time 20 Is it in low action state?

[0090] Among the non-traditional features considered by the present invention, eye movement amplitude i , Velocity i The proportion of eye closure time in the time window Perclos is calculated by combining the eye aspect ratio EAR with the time window:

[0091] Formula (11) defines the calculation method of the eye aspect ratio (EAR) for detecting the blinking process:

[0092]

[0093] Among them, E n is the key point number of the eye area marked by the MediaPipe tool, Mean(A,B) represents the average value of A and B, and Dis(a,b) represents the Euclidean distance between two points. Its calculation formula (12) is as follows:

[0094]

[0095] Among them, Px and Py represent the two-dimensional coordinates of the corresponding facial key points on the image.

[0096] When the EAR value meets the setting of the self-correction module, it will be marked as the beginning of the blinking process. During the blinking process, relevant features can be recorded according to the changes in the EAR value.

[0097] Formula (13) defines the eye movement amplitude i Calculation method:

[0098]

[0099] Among them, EAR Start is the EAR value at the beginning of the blink process, EAR End is the EAR value at the end of the blink process, and EAR Peak It is the EAR value at the lowest point in the entire blinking process.

[0100] Formula (14) defines the calculation method of the eye opening velocity:

[0101]

[0102] Among them, Time Open The time it takes for an eye to go from closing to opening.

[0103] Perclos (Percentage of Eye Closure) is an important indicator for measuring fatigue status. Formula (15) defines the calculation method of Perclos, the proportion of eye closure time in the time window:

[0104]

[0105] Among them, Time Close is the time the eyes are closed within the time window, Time Length The time window length used in the present invention is 30s, and the data frame rate used is 30FPS, that is, Time Length The value is 900.

[0106] Among the non-traditional features considered in the present invention, the average head pitch angle PA, the head activity HA, and the head down time HD per unit time are calculated by combining the pitch angle α, yaw angle β, and rotation angle γ of the head posture with the time window:

[0107] The principle of extracting head signal features and sight signal features is based on the two-dimensional image output by the network camera and the pre-modeled three-dimensional coordinate data. This process completes the mapping from the three-dimensional coordinates of the head and eyeball in the real world to the three-dimensional coordinates of the camera in the real world, and then to the two-dimensional coordinates in the pixel world. This set of mapping relationships can be obtained by formula (16):

[0108]

[0109] Among them, s is a scale factor representing the depth information, x and y are the two-dimensional coordinates in the pixel world that need to be mapped. The key points from which these coordinates come are located at: left corner of the eye, right corner of the eye, forehead, nose tip, left side of the mouth, right side of the mouth; and u, v and w are the three-dimensional coordinates that need to be mapped. This set of data is obtained by measuring the 3D face model. C refers to the camera's intrinsic parameter matrix CameraMatrix, and its expression is expanded as shown in formula (17):

[0110]

[0111] where f x and f y are the focal lengths of the camera in the pixel x direction and pixel y direction, respectively. In this study, they are set to the width of the input image as an approximation; c x With c y It is the horizontal and vertical coordinates of the center point of the image. Another camera parameter that needs to be used is the distortion coefficient matrix DistMatrix, which can be set to an all-zero matrix before camera calibration.

[0112] Combining equations (17) and (18) with the two-dimensional coordinate groups x and y, and the three-dimensional coordinate groups u, v and w, the SolvePnP method in OpenCV can be used to invert the affine transformation matrix [RT] composed of the Rodriguez rotation vector R and the translation vector T of the head posture in equation (15), and its expanded form is shown in equation (18):

[0113]

[0114] The Rodriguez rotation vector R in equation (18) can be converted into the expression of Euler angles through equation (19), thereby completing the calculation of the head posture.

[0115]

[0116] Among them, α, β, and γ represent the pitch angle, yaw angle, and rotation angle of the estimated object respectively. Based on these data, the head posture at this time and whether the head is lowered can be judged.

[0117] Formula (20) defines the calculation method of the average head pitch angle PA (PitchAverage), which interprets the numerical value of the subject's head lowering degree within the time window.

[0118]

[0119] Among them, Pitch i represents the head pitch angle, and i represents the frame number.

[0120] The head activity HA indicates the frequency of head orientation changes within the time window. In the present invention, the head orientation is defined by formula (21):

[0121]

[0122] Roll T 、Pitch T , Yaw T are the values ​​of the rotation angle, pitch angle, and yaw angle in the normal orientation, which are obtained by the self-correction module. y >(Pitch T +15°) is head up, θ y <(Pitch T -15°) is for lowering the head; θ z >(Yaw T +15°) is left turn, θ z <(Yaw T -15°) is a right turn of the head; thus completing the estimation of the head posture and accurately detecting whether nodding and turning occur. When the head orientation changes, the value of the head activity HA will increase.

[0123] The head down time HD (Head Down) per unit time is defined by formula (22):

[0124]

[0125] Among them, HeadDown indicates the time that the head changes from other orientations to lowered orientation within the time window.

[0126] Among the non-traditional features considered in the present invention, the gaze activity index GA and the fixation center time are obtained by estimating the posture of the eyeball:

[0127] The Euler angle estimated by the eyeball from the gaze posture GazeDir: rotation angle P α , pitch angle P β , yaw angle P γ Formula (23) defines the calculation method of the gaze posture GazeDir:

[0128] GazeDir=Rovtvec(EOR·HOR)=(P α ,P β ,P γ ) (twenty three)

[0129] Wherein, Rotvec is a function provided by scipy, which is used to convert the rotation matrix into the form of the Rodriguez rotation vector; EOR represents the rotation angle of the eyeball, which is the relative rotation angle between the left and right boundaries of the eyeball in the two-dimensional image calculated by the face key point algorithm; HOR represents the rotation matrix of the head, which is the matrix form of the Rodriguez rotation vector R of the head posture defined in formula (16).

[0130] The specific direction of the line of sight in the present invention is defined by formula (24):

[0131]

[0132] Among them, Pu T represents the pupil yaw angle P obtained by the self-correction module when the subject is looking at the center of the screen normally β The average value within the correction time window. The present invention defines: when P β >Pu T +0.5 when the line of sight is biased to the left; when P β <Pu T When P is -0.5, the line of sight is biased to the right; in other cases, the line of sight is defined as the center of gaze. In order to refine the extraction of line of sight features, the present invention also introduces the line of sight activity index GA. β When a certain amplitude of change occurs, GA will increase even if the specific posture change conditions are not met.

[0133] According to formula (25), the aspect ratio of the mouth can be calculated:

[0134]

[0135] When the MAR value meets the setting of the self-correction module, it will be marked as the beginning of the yawning process.

[0136] S3.3: The annotation method is mainly based on the original annotations provided by the real-life sleepiness dataset UTA-RLDD and the knowledge of the Karolinska Sleepiness Scale (KSS). It was completed by four students from the majors of electronic information and communication engineering. They watched the video in an independent environment (rewinding or fast-forwarding was allowed) and added annotations. They were not allowed to communicate with each other during work to ensure the independence of the annotations. When there is a large deviation between the annotation and the original data, the data of this segment will be excluded and will no longer participate in subsequent experiments.

[0137] S3.4: The final constructed dataset contained 1126 cases of awake state, 1084 cases of mild fatigue, and 1166 cases of severe fatigue, with a balanced distribution.

[0138] In S4:

[0139] S4.1: The present invention adopts the subject-independent method LOPO for standardization, and the object of standardization is the continuous features in the data set. Specifically, the mean and standard deviation are calculated by the feature values ​​extracted from each subject in the awake state, and the data provided by the corresponding subject are standardized. The specific calculation method is shown in formula (26):

[0140]

[0141] in, represents the standardized eigenvalue, F n,m represents the original eigenvalue, μ n,m With σ n,m Respectively represent the mean and standard deviation of the continuous data collected from feature n and subject m in the awake state.

[0142] S4.2: The input data of the multilayer perceptron model needs to be standardized by the subject-independent method and then processed again by the deviation normalization method. For the random forest and extreme gradient boosting models, only the subject-independent method is used for standardization, and no normalization method is used.

[0143] S4.3: The present invention adopts a variety of machine learning algorithms to complete modeling, including random forest RF, multi-layer perceptron MLP and extreme gradient boosting XGBoost algorithm.

[0144] S4.3.1: Random forest is a machine learning algorithm that improves the accuracy and stability of classification or regression by building multiple decision trees and combining their results. Its principles are as follows:

[0145] First, multiple sub-datasets d are generated from the training dataset D containing N samples by sampling with replacement j , where j is the number of decision trees.

[0146] For each sub-dataset d j , random forest will train a decision tree. In the process of building each tree, a randomly selected subset is used on the feature selection of each node to increase the diversity of the model. The feature selection steps are shown in formula (27):

[0147]

[0148] Among them, Ft is the complete set of features, Ft jis the candidate feature subset of node j, in Ft j Select an optimal feature f * and its corresponding splitting threshold t * To maximize the purity after splitting, the commonly used purity index is the Gini index, which is defined as shown in formula (28):

[0149]

[0150] Among them, p k It is the proportion of samples of the kth category in the data set. When splitting nodes, the features and thresholds that maximize the reduction of the Gini index are selected to reduce impurity.

[0151] Repeat the feature subset selection and node splitting process at each child node until the maximum depth of the tree is reached or the number of samples at the leaf node is lower than the set threshold. Repeat this decision tree construction step j times, and the generated random forest is shown in formula (29):

[0152] Forest = {Tree 1 ,Tree 2 ,...,Tree j} (29)

[0153] For classification tasks, random forest votes on the prediction results of each decision tree and takes the category with the most occurrences as the final classification result.

[0154] S4.3.2: A multilayer perceptron usually consists of an input layer, a hidden layer, and an output layer. Suppose there are L hidden layers between the input layer and the output layer. The output of the lth hidden layer is denoted as h l , then the structure of the model is shown in formula (30):

[0155]

[0156] Among them, X represents the feature matrix, W l and b l They represent the weight matrix and bias of the lth layer respectively. f is an activation function, which is used to introduce nonlinearity. In the present invention, the activation function uses the ReLu function.

[0157] The last layer maps the activation value of the hidden layer to the label of the fatigue state and uses the Softmax function to complete the probability normalization, as shown in formula (31):

[0158]

[0159] in, Represents the model output fatigue state classification result of the i-th sample.

[0160] The training of the multilayer perceptron depends on the minimization of the loss function. The cross entropy loss Loss is used in the present invention, which is shown in formula (32):

[0161]

[0162] Among them, y i,c is the true fatigue state label of the i-th sample, and the weight matrix W is continuously adjusted through back propagation l , the model can learn the mapping relationship between features and labels.

[0163] S4.3.3: The extreme gradient boosting algorithm is an additive model that integrates decision trees through the boosting method. Its objective function is shown in formula (33):

[0164]

[0165] Among them, Ω is the regularization term, f k is the prediction function of the kth tree. In the extreme gradient boosting algorithm, the t-1th round of iteration is for the sample x i Classification results It is expressed as shown in formula (34):

[0166]

[0167] Among them, f t (x i ) is the t-th tree for sample x i predictions, is the predicted value for round t-1.

[0168] The core idea of ​​the extreme gradient boosting algorithm is to construct multiple decision trees. Each tree learns the prediction error of the previous round, and continuously reduces the model error through iteration to obtain the final model. The output result of the final model is the combination of all trees.

[0169] In S5:

[0170] S5.1: The present invention adopts methods such as hold-out method (20% / 30%), 10-fold cross validation, stratified cross validation and subject independent validation LOPO to divide the data set, and the test sets generated based on different proportions, data balance and independence are used to evaluate the random forest RF, multi-layer perceptron MLP and extreme gradient boosting XGBoost models trained by the corresponding training sets to obtain the optimal model.

[0171] The specific steps are as follows:

[0172] Holdout method: randomly divide the dataset into training dataset (70-80%) and test dataset (20-30%), with a ratio of 80:20 and 70:30 respectively;

[0173] K-fold cross validation method: divide the data set into K sub-folds, iteratively use K-1 folds for training, and 1 fold for validation;

[0174] Stratified K-fold cross-validation method: Based on the K-fold cross-validation method, ensure that the labels of each category (awake, mild fatigue, severe fatigue) are evenly distributed in the training data set and the test data set;

[0175] Subject independent validation method: The data were grouped according to the subject number, creating 59 groups for 60 subjects. This cross-validation was performed 60 times, with each iteration using the data of one subject for validation and the data of the remaining subjects for training.

[0176] It should be noted that in the hold-out method, K-fold cross-validation, and stratified K-fold cross-validation, the training data set and the test data set are first divided, and then the wakefulness state in the training data set is used to extract continuous features, and their mean and standard deviation are calculated. The data is standardized according to the subject number, and the data of the test set is completely independent of the calculation of the mean and standard deviation of the training set to avoid the risk of data leakage.

[0177] The label distribution of the holdout method is shown in Table 2.

[0178] Table 2

[0179]

[0180] The label distribution of K-fold cross validation method and stratified K-fold cross validation method is shown in Table 3, where 0 represents the awake state, 5 represents the mild fatigue state, and 10 represents the severe fatigue state. K=10.

[0181] Table 3

[0182]

[0183] S5.2: The present invention adopts a variety of indicators to test and compare the performance of the scheme. It is assumed that the number of normal samples classified as normal is defined as TP, the number of normal samples classified as abnormal is defined as FP, the number of abnormal samples classified as normal is defined as FN, and the number of abnormal samples classified as abnormal is defined as TN.

[0184] The calculation formulas for accuracy, recall, precision, and F1 score are as follows:

[0185]

[0186]

[0187]

[0188] Among them, since the method in the present invention is aimed at multi-classification problems, the recall rate Recall and precision Precision in the evaluation indicators are obtained by micro-averaging method after calculating each category separately.

[0189] S5.3: Use the holdout method provided in Table 2, the K-fold cross-validation method and the stratified K-fold cross-validation method provided in Table 3, and the subject independent LOPO division method to divide the test set and build a model for debugging to obtain the optimal model.

[0190] In S6:

[0191] S6.1: Design of self-correction feature extraction module: The present invention stipulates that a period of time after the start of detection is a correction window, and within this window, the extracted eye aspect ratio EAR, mouth aspect ratio MAR, eyeball pitch angle P β and the rotation angle Roll when the head is facing normally T , Pitch T , Yaw T The calibration data is used to set the blink threshold, yawn threshold, head posture correction and line of sight correction. The data collected within the calibration window is not input into the model, and the default window length is thirty seconds.

[0192] The correction method of blink threshold and yawn threshold is shown in formula (38):

[0193]

[0194] Among them, A EAR is the corresponding threshold value obtained by calculation, is the average of the maximum values ​​of EAR in each blink within the calibration window, is the mean of the minimum values ​​of EAR in each blink within the calibration window, N is the number of blinks within the calibration window, and C EAR A is the EAR adjustment factor, and its value is 0.2. MAR is the corresponding threshold value obtained by calculation, is the mean of the maximum values ​​of MAR in each blink within the correction window, is the mean of the minimum values ​​of MAR in each blink within the correction window, C MAR is the MAR adjustment factor, and its value is 0.7.

[0195] The head posture correction method is defined as formula (21), and the sight line correction method is defined as formula (24).

[0196] S6.2: Preprocessing module: After completing the self-calibration, the mean and standard deviation of the continuous features in the first 1 / 3 of the awake state are calculated based on the data collected during the calibration period, which are used as parameters for subsequent data standardization. The subsequent collected data are standardized using this parameter to ensure the stability and accuracy of the model operation. The preprocessing method is defined as formula (26).

[0197] S6.3: Integrated model classification module: Save the trained optimal model as a persistent file, load the model file when using it, and classify the standardized features. Through the integrated learning method, the classification results of multiple models are weighted to form the final prediction result, where the weight ratio of random forest RF, multi-layer perceptron MLP and extreme gradient boosting XGBoost is 2:1:3.

[0198] S6.4: The self-correction feature extraction module, preprocessing module, integrated model classification module are combined with the target detection model and the face key point detection model to construct a real-time driver fatigue detection system. The overall structure of the system is as follows: Figure 2 shown.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A fatigue detection method based on machine learning and hybrid features, characterized in that: The following steps are involved: S1: Data collection: Obtain the UTA-RLDD dataset and process it into non-overlapping video segments; S2: Face detection: Train the object detection algorithm YOLOv8 to achieve face recognition; S3: Dataset construction: Face key point detection based on MediaPipe framework, feature calculation using key points, screening and labeling to build the dataset; S4: Data preprocessing: Based on the subject-independent method LOPO, the continuous data is normalized, the continuous data input into the multi-layer perceptron is normalized, and the data with outliers is filtered out; S5: Model training: Input the above processed experimental data into the constructed model, including random forest RF, multi-layer perceptron MLP and extreme gradient boosting XGBoost, obtain the fatigue detection classification model through autonomous learning training, and evaluate the model performance through evaluation indicators to obtain the optimal model; S6: System construction: Design self-correcting feature extraction module and preprocessing module, build fatigue status classification module through voting method in ensemble learning, form real-time driver fatigue detection system, and realize automatic detection and classification of driver fatigue status.

2. The fatigue detection method based on machine learning and hybrid features according to claim 1, characterized in that: Specifically, S1 includes: dividing each video in the real-life fatigue dataset into non-overlapping segments of 30 seconds, with each segment being an independent data sample; and ensuring that all segments do not overlap during the segmentation process in order to avoid overfitting due to the high correlation of features in adjacent time periods.

3. The fatigue detection method based on machine learning and hybrid features according to claim 2, characterized in that: The S2 comprises the following steps: S2.1: The model is lightweight by replacing the YOLOv8 backbone network with Faster-Net and trained using the Wider-Face dataset. During deployment, the model is quantized using TensorRT technology to achieve hardware acceleration and low-latency face detection. S2.2: Use YOLOv8 for face recognition and obtain the region of interest (ROI) to ensure that the face is always located in the center of the image, so as to reduce the impact of the change in the position of the face in the two-dimensional image on the detection of key point coordinates by the face mesh model.

4. The fatigue detection method based on machine learning and hybrid features according to claim 3 is characterized in that: The S3 comprises the following steps: S3.1: The MediaPipe framework is used to extract the 2D coordinates of 478 key points from the face region to complete feature calculation; these features can be divided into four categories: eye features, head features, mouth features, and sight features, a total of 20 features, and these features are represented as feature vectors as x, x∈R m ,m=20; Among the non-traditional features, eye movement amplitude i , Velocity i The proportion of eye closure time in the time window Perclos is calculated by combining the eye aspect ratio EAR with the time window: Formula (1) defines eye movement amplitude i Calculation method: Among them, EAR Start is the EAR value at the beginning of the blink process, EAR End is the EAR value at the end of the blink process, and EAR Peak It is the EAR value at the lowest point in the entire blinking process; Formula (2) defines the eye opening velocity Velocity i Calculation method: Among them, Time Open The time it takes for an eye to go from closing to opening; Formula (3) defines the calculation method of Perclos, the proportion of eye closure time in the time window: Among them, Time Close is the time the eyes are closed within the time window, Time Length is the length of the time window. The time window length used is 30s and the data frame rate used is 30FPS. Length The value of is 900; Among the non-traditional features, the average head pitch angle PA, head activity HA, and head down time per unit time HD are calculated by combining the pitch angle α, yaw angle β, and rotation angle γ of the head posture with the time window: Formula (4) defines the calculation method of the average head pitch angle (PitchAverage, PA), which characteristically interprets the numerical value of the subject's head drooping degree within the time window: Among them, α i represents the head pitch angle, i represents the frame number; The head activity HA indicates the frequency of head orientation changes within the time window. The head orientation is defined by formula (5): Roll T 、Pitch T , Yaw T They are the values ​​of rotation angle, pitch angle and yaw angle in normal orientation, which are obtained by the self-correction module; define θ y >(Pitch T +15°) is head up, θ y <(Pitch T -15°) is head down; θ z >(Yaw T +15°) is left turn, θ z <(Yaw T -15°) is a right turn of the head; thus completing the estimation of the head posture and accurately detecting whether nodding and turning occur; when the head orientation changes, the value of the head activity HA will increase; The head down time HD(HeadDown) per unit time is defined by formula (6): Among them, HeadDown indicates the time that the head is kept in a downward direction after changing from other directions within the time window; Among the non-traditional features, the gaze activity index GA and the fixation center time are obtained by estimating the eye posture: GazeDir estimates the eyeball posture and obtains the Euler angle group: rotation angle P α , pitch angle P β , yaw angle P γ Formula (7) defines the calculation method of gaze posture GazeDir: GazeDir=Rovtvec(EOR·HOR)=(P α ,P β ,P γ ) (7) Among them, Rotvec is a function provided by scipy, which is used to convert the rotation matrix into the form of Rodriguez rotation vector; EOR represents the rotation angle of the eyeball, which is the relative rotation angle between the left and right boundaries of the eyeball in the two-dimensional image calculated by the face key point algorithm; HOR represents the rotation matrix of the head; The specific direction of the line of sight is defined by equation (8): Among them, Pu T represents the pupil yaw angle P obtained by the self-correction module when the subject is looking at the center of the screen normally β The average value within the correction time window; when P β >Pu T +0.5 when the line of sight is biased to the left; when P β <Pu T When -0.5, the sight is biased to the right; in other cases, the sight is defined as the gaze center. When the sight is looking toward the center within the time window, the feature is introduced: the gaze center time, which increases when the sight is in the gaze center state; In order to refine the extraction of sight features, the feature is introduced: the sight activity index GA, when P β When a certain amplitude of change occurs, GA will increase even if the specific posture change conditions are not met; S3.3: The annotation method was completed by four students from the majors of electronic information and communication engineering, combining the original annotations provided by the real-life sleepiness dataset UTA-RLDD and the knowledge of the Karolinska Sleepiness Scale (KSS). They watched the video and added annotations in an independent environment. They were not allowed to communicate with each other during the work to ensure the independence of the annotations. When there was a deviation between the annotations and the original annotations, the data of the segment would be excluded and would no longer participate in subsequent experiments. S3.4: The constructed dataset contains 1126 cases of awake state, 1084 cases of mild fatigue, and 1166 cases of severe fatigue, with a balanced distribution.

5. The fatigue detection method based on machine learning and hybrid features according to claim 4 is characterized in that: The S4 comprises the following steps: S4.1: The subject-independent method LOPO is used for standardization. The object of standardization is the continuous features in the data set. The mean and standard deviation are calculated by extracting the feature values ​​of each subject in the awake state, and the data provided by the corresponding subject are standardized. The specific calculation method is shown in formula (9): in, represents the standardized eigenvalue, F n,m represents the original eigenvalue, μ n,m With σ n,m Respectively represent the mean and standard deviation of the continuous data collected from feature n and subject m in the awake state.

6. The fatigue detection method based on machine learning and hybrid features according to claim 5, characterized in that: The S5 comprises the following steps: S5.1: The data set is divided into two parts using the leave-out method, 10-fold cross validation, stratified cross validation, and independent subject validation (LOPO) method. The random forest RF, multi-layer perceptron MLP, and extreme gradient boosting XGBoost models trained by the corresponding training sets are evaluated based on the test sets generated based on different proportions, data balance, and independence to obtain the optimal model.

7. The fatigue detection method based on machine learning and hybrid features according to claim 6, characterized in that: The S6 comprises the following steps: S6.1: Design of self-correction feature extraction module: The period after the detection starts is the correction window. During this window, the extracted eye aspect ratio EAR, mouth aspect ratio MAR, eyeball pitch angle P β and the rotation angle Roll when the head is facing normally T , Pitch T , Yaw T As calibration data, it is used to set the blink threshold, yawn threshold, head posture correction and sight line correction. The data collected within the calibration window is not input into the model. The default window length is thirty seconds. The correction method of blink threshold and yawn threshold is shown in formula (10): Among them, A EAR is the blink detection threshold, is the maximum value of EAR in each blink within the calibration window. is the minimum value of EAR in each blink within the calibration window, N is the number of blinks within the calibration window, C EAR is the EAR adjustment factor, the value is 0.2; A MAR is the yawning threshold, is the maximum value of MAR in the correction window in each blink. is the minimum value of MAR in each blink within the correction window, C MAR is the MAR adjustment factor, the value is 0.7; The head posture correction method is defined as formula (5), and the sight line correction method is defined as formula (8); S6.2: Preprocessing module: After completing the self-calibration, the mean and standard deviation of the continuous features of the first 1 / 3 in the awake state are calculated based on the data collected during the calibration period, which are used as parameters for subsequent data standardization; the subsequent collected data are standardized by using the parameters to ensure the stability and accuracy of the model operation. The preprocessing method is defined as formula (9); S6.3: Integrated model classification module: Save the trained optimal model as a persistent file, load the model file when using it, and classify the standardized features; through the integrated learning method, the classification results of multiple models are weighted to form the final prediction result, where the weight ratio of random forest RF, multi-layer perceptron MLP and extreme gradient boosting XGBoost is 2:1:3; S6.4: The self-correcting feature extraction module, the preprocessing module, the integrated model classification module are combined with the target detection model and the face key point detection model to construct a real-time driver fatigue detection system.

Images