A fatigue detection method based on machine learning and mixed features

Abstract

The present application relates to a kind of fatigue detection method based on machine learning and mixed features, belong to vehicle safety technical field.The present application realizes face recognition by training target detection algorithm YOLOv8, carries out face key point detection based on MediaPipe framework, after completing the standardization processing of data based on subject independent method, obtains fatigue detection classification model by a variety of machine learning modeling.The present application also designs self-correcting feature extraction module, pre-processing module, by integrating classification model to build fatigue state classification module, forms driver real-time fatigue detection system, realizes the automatic detection and classification of driver fatigue state.The present application introduces 20 kinds of facial features such as line of sight concentration, eye movement amplitude and head movement, realizes good generalization ability when fatigue detection is carried out to unknown driver;Driver fatigue detection system is constructed using ensemble learning, and a lightweight real-time driver fatigue detection system is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of vehicle safety, and relates to a fatigue detection method based on machine learning and mixed features. BACKGROUND

[0002] Fatigue driving is one of the main causes of traffic accidents. In recent years, the academic field has carried out a large amount of research in the field of fatigue detection, mainly focusing on detection methods based on vehicle signals, physiological signals and facial signals. Among them, the fatigue detection method based on the facial signals of the driver shows broad application prospects due to its non-contact, low cost and easy-to-promote characteristics. With the progress of electronic hardware technology, this method has gradually become a research hotspot.

[0003] At present, most fatigue detection research based on facial signals mainly relies on traditional features such as blink frequency, yawn frequency and nodding frequency, and the following methods are often used for fatigue state classification: setting thresholds to achieve discrimination, inputting extracted features into machine learning models for classification, or directly inputting images into deep learning models to complete classification. However, due to individual differences among drivers, different people will exhibit diverse characteristics in a fatigue state. Using the threshold method faces the difficulty of numerical adjustment of fixed thresholds or dynamic threshold ranges; using only traditional features to train machine learning models to identify fatigue states may result in insufficient model generalization ability and poor robustness; and directly inputting images into deep learning models for classification has the problems of slow processing speed and difficulty in meeting the real-time requirements of deployment.

[0004] In view of the above problems, the application provides a fatigue detection method based on machine learning and mixed features. The method further introduces twenty facial features such as gaze concentration, eye movement amplitude, blink wave speed and head movement on the basis of traditional features, and realizes classification through machine learning modeling. The method not only improves the accuracy of detection, but also effectively solves the model generalization problem caused by individual differences among drivers, thereby significantly improving the precision and robustness of fatigue detection and further improving the real-time performance of the system. SUMMARY

[0005] Therefore, the purpose of the application is to provide a fatigue detection method based on machine learning and mixed features. In view of the problem that traditional methods only consider traditional features and ignore the diversified performance of different drivers, the method introduces twenty facial features such as gaze concentration, eye movement amplitude, blink wave speed and head movement, and realizes classification through machine learning modeling, which can have good generalization ability for fatigue detection of unknown drivers, and further improves the accuracy and comprehensiveness of the driver fatigue detection system.

[0006] To achieve the above purpose, the application provides the following technical solutions:

[0007] A fatigue detection method based on machine learning and mixed features, comprising the following steps:

[0008] S1: Data collection: Obtain the UTA-RLDD dataset and process it into non-overlapping video segments;

[0009] S2: Face detection: Train the target detection algorithm YOLOv8 to realize face recognition;

[0010] S3: Dataset construction: Perform face key point detection based on the MediaPipe framework, use the key points to complete feature calculation, and filter and label to construct the dataset;

[0011] S4: Data preprocessing: Complete the standardization processing of continuous data based on the subject independent method LOPO, complete the normalization processing of continuous data in the input multilayer perceptron, and filter out the data with abnormal values;

[0012] S5: Model training: Input the above processed experimental data into the constructed model, including random forest RF, multilayer perceptron MLP and extreme gradient boosting XGBoost, obtain the fatigue detection classification model through autonomous learning and training, and evaluate the model performance through evaluation indexes to obtain the optimal model;

[0013] S6: System building: Design a self-correcting feature extraction module and a preprocessing module, build a fatigue state classification module through the voting method in ensemble learning, form a driver real-time fatigue detection system, and realize the automatic detection and classification of the fatigue state of the driver.

[0014] Further, the S1 is specifically: each video in the real-life fatigue dataset is cut into 30-second non-overlapping segments, and each segment is taken as an independent data sample; in order to avoid overfitting caused by the high correlation of features in adjacent time periods, it is ensured that all segments are non-overlapping during the cutting process.

[0015] Further, the S2 includes the following steps:

[0016] S2.1: Replace the backbone network of YOLOv8 with Faster-Net to realize the lightweight of the model, and use the Wider-Face dataset for training; in the deployment process, use the TensorRT technology to realize model quantization, to realize hardware acceleration and low-delay face detection;

[0017] S2.2: Use YOLOv8 to perform face recognition and obtain the region of interest (ROI), to ensure that the face is always located in the center of the picture, so as to reduce the influence of the change of the position of the face in the two-dimensional image on the detection of the key point coordinates by the face grid model.

[0018] Further, the S3 includes the following steps:

[0019] S3.1: Complete feature calculation using the two-dimensional coordinates of the 478 key points extracted from the face area by the MediaPipe framework; these features can be divided into four categories: eye features, head features, mouth features, and gaze features, a total of 20 features, and these features are expressed as a feature vector x, x ∈ R m ,m = 20;

[0020] Among the non-traditional features, the eye movement amplitude Amplitude i , the open eye wave speed Velocity i and the proportion of eye closure time Perclos in the time window are calculated by combining the eye aspect ratio EAR with the time window:

[0021] Formula (1) defines the calculation method of eye movement amplitude Amplitude i :

[0022]

[0023] Where EAR Start is the EAR value at the beginning of the blinking process, EAR End is the EAR value at the end of the blinking process, and EAR Peak is the EAR value at the lowest point in the entire blinking process;

[0024] Formula (2) defines the calculation method of open eye wave speed Velocity i :

[0025]

[0026] Where Time Open is the time required for the eye to close and open during blinking;

[0027] Formula (3) defines the calculation method of the proportion of eye closure time Perclos in the time window:

[0028]

[0029] Where Time Close is the time of eye closure in the time window, Time Length is the length of the time window, the length of the time window used is 30s, and the data frame rate used is 30FPS, i.e. the value of Time Length is 900;

[0030] Among the non-traditional features, the average head pitch angle PA, head activity HA, and head droop duration HD per unit time are calculated by combining the head posture pitch angle α, yaw angle β, and rotation angle γ with a time window.

[0031] Equation (4) defines the calculation method for the average head pitch angle (PA), which characteristically interprets the numerical value of the degree of head drooping of the subject within the time window:

[0032]

[0033] Where, α i This indicates the head pitch angle, and 'i' represents the frame number.

[0034] Head activity HA represents the frequency of head orientation changes within a time window, and head orientation is defined by equation (5):

[0035]

[0036] Roll T Pitch T Yaw T These are the rotation angle, pitch angle, and yaw angle values ​​under normal heading conditions, obtained by the self-correction module; θ is defined. y >(Pitch) T +15° is the head tilt, θ y <(Pitch) T -15° is considered head down; θ z >(Yaw T +15° is a left turn, θ z <(Yaw) T -15° is a right head turn; thus, the head posture is estimated and the head nodding and turning are accurately detected; when the head orientation changes, the value of head activity HA will increase;

[0037] The duration of head drooping (HD) per unit time is defined by equation (6):

[0038]

[0039] HeadDown represents the time during which the head remains tilted down after changing from another orientation within the time window;

[0040] Among the non-traditional features, the gaze activity index (GA) and fixation center time are obtained by estimating the eye's pose.

[0041] The Euler angle set obtained by GazeDir for eye pose estimation: rotation angle P α Pitch angle Pβ yaw angle P γ The gaze direction GazeDir is defined by the composition of the formula (7):

[0042] GazeDir = Rovtvec(EOR·HOR) = (P α ,P β ,P γ ) (7)

[0043] where Rotvec is a function provided by scipy, which is used to convert the rotation matrix into the form of the Rodrigues rotation vector; EOR represents the rotation angle of the eyeball, which is calculated by the face key point algorithm to obtain the relative rotation angle of the left boundary and the right boundary of the eyeball in the two-dimensional image; HOR represents the rotation matrix of the head;

[0044] The specific direction of the gaze is defined by formula (8):

[0045]

[0046] where Pu T represents the pupil yaw angle P β of the subject obtained by the self-correction module when the subject is looking at the center of the normal fixation picture; the average value in the correction time window; when P β > Pu T + 0.5, the gaze is deviated to the left; when P β < Pu T - 0.5, the gaze is deviated to the right; in other cases, the gaze is defined as the fixation center, and when the gaze is looking at the center in the time window, the feature of fixation center time is introduced, which will increase when the gaze is in the fixation center state;

[0047] To refine the extraction of the gaze feature, the feature of gaze activity index GA is introduced, which will increase when P β changes by a certain amplitude, even if the specific attitude change condition is not reached;

[0048] S3.3: Based on the original annotation provided by the real life drowsiness dataset UTA-RLDD and the understanding of the Karolinska Sleepiness Scale KSS, four students from the Electronic Information and Communication Engineering major completed the annotation by watching the video in an independent environment and adding the annotation, and they were not allowed to communicate with each other during the work to ensure the independence of the annotation; when the annotation deviates from the original annotation, the segment data will be excluded and will not participate in the subsequent experiment;

[0049] S3.4: The constructed dataset contains 1126 cases of wakefulness, 1084 cases of mild fatigue, and 1166 cases of severe fatigue, which are evenly distributed.

[0050] Further, the S4 comprises the following steps:

[0051] S4.1: Standardization is performed using the LOPO method, and the continuous features in the dataset are standardized. The mean and standard deviation are calculated by the feature values of each subject in the awake state, and the data provided by the corresponding subject is standardized. The specific calculation method is shown in equation (9):

[0052]

[0053] where F n,m represents the standardized feature value, F n,m represents the original feature value, μ n,m and σ n,m represent the mean and standard deviation of the continuous data collected from subject m in the awake state.

[0054] Further, the S5 comprises the following steps:

[0055] S5.1: The data set is divided using the LOPO method, 10-fold cross-validation, stratified cross-validation, and subject-independent verification. The test set generated based on different proportions, data balance, and independence is used to evaluate the random forest RF, multilayer perception MLP, and extreme gradient boosting XGBoost models trained on the corresponding training set, to obtain the optimal model.

[0056] Further, the S6 comprises the following steps:

[0057] S6.1: Self-correcting feature extraction module design: A period of time after detection is set as the correction window, and the extracted eye aspect ratio EAR, mouth aspect ratio MAR, eye pitch angle P β , and the rotation angle Roll T , pitch angle Pitch T , and yaw angle Yaw T of the head posture in the normal direction are used as calibration data to set the blink threshold, yawn threshold, head posture correction, and gaze correction. The data collected in the correction window is not input into the model, and the default window length is thirty seconds.

[0058] The correction method of the blink threshold and yawn threshold is shown in equation (10):

[0059]

[0060] where A EAR is the blink threshold, is the maximum value of EAR in each blink in the correction window, is the minimum value of EAR in each blink within the correction window, N is the number of blinks within the correction window, C EAR is the EAR adjustment factor, the value is 0.2; A MAR is the yawp judgment threshold, is the maximum value of MAR in each blink within the correction window, is the minimum value of MAR in each blink within the correction window, C MAR is the MAR adjustment factor, the value is 0.7;

[0061] The head posture correction method is defined as formula (5), and the line of sight correction method is defined as formula (8);

[0062] S6.2: Pretreatment module: after self-correction is completed, the mean and standard deviation of the continuous features in the first 1 / 3 of the wakeful state are calculated based on the data collected during the correction period, which are used as parameters for subsequent data standardization; the subsequent collected data is standardized by using the parameters to ensure the stability and accuracy of the model operation, and the pretreatment method is defined as formula (9);

[0063] S6.3: Integrated model classification module: save the trained optimal model as a persistent file, load the model file when using, and classify the standardized features; through the integrated learning method, the classification results of multiple models are combined to form the final prediction result by weighting, and the weight ratio of random forest RF, multilayer perception MLP and extreme gradient boosting XGBoost is 2:1:3;

[0064] S6.4: Combine the self-correction feature extraction module, the pretreatment module, the integrated model classification module, the target detection model and the face key point detection model to build a real-time driver fatigue detection system.

[0065] The beneficial effects of the present application are that: in view of the problem that the traditional method only considers traditional features and ignores the diversified performance of different drivers, the present method introduces twenty facial features such as line of sight concentration, eye movement amplitude, blink wave speed and head movement, and realizes classification through machine learning modeling, which can have good generalization ability for fatigue detection of unknown drivers, and further improves the accuracy and comprehensiveness of the driver fatigue detection system. And based on this method and integrated learning, a driver fatigue detection system is built, realizing a lightweight real-time driver fatigue detection system that can meet the deployment requirements and is faster than the traditional method.

[0066] Additional advantages, objects, and features of the application will be apparent to those skilled in the art upon examination of the following detailed description, it being understood that each of the foregoing general statements are true of the particular embodiments of the application and that such statements are not intended to serve as limitations upon the scope of the application. BRIEF DESCRIPTION OF DRAWINGS

[0067] In order to make the objects, technical solutions and advantages of the present application clearer, the preferred embodiments of the present application will be described in detail below with reference to the drawings, in which:

[0068] Figure 1 The overall flowchart of the fatigue detection method based on machine learning and mixed features proposed by the present application.

[0069] Figure 2 The block diagram of the fatigue detection system based on ensemble learning and mixed features of the present application. DETAILED DESCRIPTION

[0070] The embodiments of the present application are described below by way of specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the disclosure. The present application can also be implemented or applied by different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the diagrams provided in the following examples only illustrate the basic concept of the present application in a schematic manner, and the following examples and features in the examples can be combined with each other without conflict.

[0071] The drawings are only used for illustrative description, and the representation is only a schematic diagram, not a physical diagram, and cannot be understood as a limitation on the present application; in order to better illustrate the embodiments of the present application, some components in the drawings are omitted, enlarged or reduced, and do not represent the actual size of the product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings can be omitted.

[0072] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that the orientations or positional relationships indicated by terms such as "upper", "lower", "left", "right", "front", "back" and the like are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only used for illustrative description, and cannot be understood as a limitation on the present application, for those skilled in the art, the specific meanings of the above terms can be understood according to the specific circumstances.

[0073] As Figure 1 shown, the method further introduces twenty kinds of facial features such as line of sight concentration, eye movement amplitude, blink wave speed and head movement on the basis of traditional features, and realizes classification through machine learning modeling. The method not only enriches the accuracy of detection, but also effectively solves the model generalization problem caused by individual differences of drivers, thereby significantly improving the precision and robustness of fatigue detection, and further improving the real-time performance of the system. The technical solution adopted by the present application to achieve the above application purpose comprises the following steps:

[0074] S1: data collection: obtain the UTA-RLDD dataset and process it into non-overlapping video segments;

[0075] S2: face detection: train the target detection algorithm YOLOv8 to realize face recognition;

[0076] S3: dataset construction: perform face key point detection based on the MediaPipe framework, use the key points to complete feature calculation, and filter and label to construct the dataset;

[0077] S4: data preprocessing: complete the standardization processing of continuous data based on the subject independent method LOPO, complete the normalization processing of continuous data in the input multilayer perceptron, and filter out the data with abnormal values;

[0078] S5: model training: input the processed experimental data into the constructed model, including random forest RF, multilayer perceptron MLP and extreme gradient boosting XGBoost, obtain the fatigue detection classification model through autonomous learning and training, and evaluate the model performance through evaluation indexes to obtain the optimal model;

[0079] S6: system building: design a self-correcting feature extraction module and a preprocessing module, construct a fatigue state classification module through the voting method in ensemble learning, form a driver real-time fatigue detection system, and realize automatic detection and classification of the fatigue state of the driver.

[0080] Further, in S1:

[0081] S1.1: the real-life fatigue dataset UTA-RLDD is provided by sixty healthy subjects from different races, different ages and different facial accessories, each volunteer records three different categories of videos with a length of about ten minutes: awake, mild fatigue and severe fatigue, a total of 180, a total length of thirty minutes of RGB video dataset; In the present application, each video in the real-life fatigue dataset is divided into 30-second non-overlapping segments, and each non-overlapping segment is taken as an independent data sample. In order to avoid overfitting caused by the high correlation of features in adjacent time periods, it is ensured that all segments are non-overlapping during the division process.

[0082] In S2:

[0083] S2.1: In the present application, the lightweight of the model is realized by replacing the backbone network of the target detection model YOLOv8 developed based on Torch with FasterNet, and the model is trained using the Wider-Face dataset. In the deployment process, the model quantization is realized by using the TensorRT technology, so as to realize the hardware acceleration and the low-delay face detection.

[0084] S2.2: The YOLOv8 is used for face detection to obtain the region of interest (ROI), and the face input to the MediaPipe framework is always located in the center of the picture, so as to reduce the influence of the different positions of the face in the two-dimensional image on the key point coordinate detection of the MediaPipe face grid model.

[0085] In S3:

[0086] S3.1: Feature calculation according to face key point coordinates, including the following steps:

[0087] The two-dimensional coordinates of 478 key points extracted from the face area by the MediaPipe framework are used to complete the feature calculation. These features can be divided into four categories: eye features, head features, mouth features and gaze features, a total of 20 features, and these features are expressed as x, x∈R m ,m=20, and the calculated feature list is shown in Table 1.

[0088] Table 1

[0089] Feature Number Feature Name 1 Blink Count 2 Eye Closure Duration 3 Average Blink Duration 4 EAR Average 5 Eye Movement Amplitude Average 6 Eye Movement Amplitude Maximum 7 Eye Movement Amplitude Minimum 8 Eye Opening Velocity Average 9 Eye Opening Velocity Maximum 10 Eye Opening Velocity Minimum 11 Perclos 12 Gaze Activity 13 Gaze Concentration 14 Drowsiness Count 15 Head Pitch Average 16 Head Droop Duration 17 Head Activity 18 Yawn Count 19 Whether Yawn Occurred in Close Time 20 Whether in Low Motion State

[0090] Among the non-traditional features considered in the present application, the eye movement amplitude Amplitude i , the open eye wave velocity Velocity i and the proportion of eye closure time Perclos in the time window are calculated by combining the eye aspect ratio EAR with the time window:

[0091] Formula (11) defines the calculation method of the eye aspect ratio (EAR), which is used to detect the blinking process:

[0092]

[0093] wherein E n is the eye region key point number labeled by the MediaPipe tool, Mean(A,B) represents the average value of A and B, and Dis(a,b) represents the Euclidean distance between two points, and the calculation formula (12) is as follows:

[0094]

[0095] wherein Px and Py represent the two-dimensional coordinates of the corresponding facial landmark in the image.

[0096] When the value of EAR meets the setting of the self-correction module, it is marked as the start of the blinking process, and in the blinking process, according to the change of the value of EAR, the relevant features can be recorded.

[0097] Formula (13) defines the calculation method of the eye movement amplitude Amplitude i .

[0098]

[0099] wherein EAR Start is the value of EAR at the start of the blinking process, EAR End is the value of EAR at the end of the blinking process, and EAR Peak is the value of EAR at the lowest point in the entire blinking process.

[0100] Formula (14) defines the calculation method of the open-eye wave velocity Velocity

[0101]

[0102] wherein Time Open is the time required for the eye to open from closing.

[0103] Perclos (Percentage of Eye Closure) is an important indicator for measuring the fatigue state, and formula (15) defines the calculation method of the proportion of eye closure time in the time window Perclos

[0104]

[0105] wherein Time Close is the eye closure time in the time window, Time Length is the length of the time window, the length of the time window used in the present application is 30s, the data frame rate used is 30FPS, and the value of Time Length is 900.

[0106] Among the non-traditional features considered by the present application, the head pitch angle average value PA, the head activity HA and the head droop time length HD per unit time are calculated by combining the pitch angle a, the yaw angle b and the rotation angle g of the head posture with the time window:

[0107] For the extraction of head signal features and gaze signal features, the principle is based on the two-dimensional image output by the network camera and the pre-modeled three-dimensional coordinate data. The process completes the mapping of the three-dimensional coordinates of the head and eyeball in the real world to the three-dimensional coordinates of the camera in the real world, and then to the two-dimensional coordinates in the pixel world. This set of mapping relationships can be obtained by equation (16):

[0108]

[0109] where s is a scale factor representing depth information, x and y are two-dimensional coordinates in the pixel world that need to be mapped, which come from the key points located at the left corner of the eye, the right corner of the eye, the forehead, the nose tip, the left side of the mouth, and the right side of the mouth; and u, v, and w are three-dimensional coordinates that need to be mapped, which are obtained by calculating the 3D face model. C refers to the intrinsic matrix of the camera CameraMatrix, whose expression is expanded as shown in equation (17):

[0110]

[0111] where f x and f y are the focal lengths of the camera in the pixel x direction and the pixel y direction, respectively, which are set to the width of the input image as an approximation in this research scheme; c x and c y are the horizontal and vertical coordinates of the image center point. Another camera parameter that needs to be used is the distortion coefficient matrix DistMatrix, which can be set to a zero matrix before camera calibration.

[0112] Combining equation (17), equation (18), and the two-dimensional coordinate set x and y, and the three-dimensional coordinate set u, v, and w, the affine transformation matrix [R T] composed of the Rodrigues rotation vector R and the translation vector T of the head pose in equation (15) can be inversely deduced by the SolvePnP method in OpenCV, whose expanded form is shown in equation (18):

[0113]

[0114] The Rodrigues rotation vector R in equation (18) can be converted to the expression form of Euler angles by equation (19), thereby completing the calculation of the head pose.

[0115]

[0116] where α, β, and γ represent the estimated object's pitch angle, yaw angle, and roll angle, respectively. According to these data, the head pose and whether low head occurs can be determined.

[0117] Formula (20) defines the calculation method of the head pitch angle average value PA (PitchAverage), which interprets the value of the head droop degree of the subject in the time window.

[0118]

[0119] wherein, Pitch i represents the head pitch angle, and i represents the frame number.

[0120] The head activity HA represents the frequent degree of the head orientation change in the time window, and the head posture orientation in the application is defined by formula (21) :

[0121]

[0122] wherein, Roll T , Pitch T , Yaw T are the values of the roll angle, the pitch angle and the yaw angle in the normal orientation respectively, which are obtained by the self-correction module. In the application, θ y > (Pitch T + 15°) is the head lifting, θ y < (Pitch T - 15°) is the head lowering; θ z > (Yaw T + 15°) is the head turning left, and θ z < (Yaw T - 15°) is the head turning right; so as to complete the estimation of the head posture and accurately detect whether the nodding and turning of the head occur. When the head orientation changes, the value of the head activity HA will increase.

[0123] The head down time HD (Head Down) in unit time is defined by formula (22) :

[0124]

[0125] wherein, HeadDown represents the time that the head remains in the head down state after changing from other orientations to the head down state in the time window.

[0126] Among the non-traditional features considered in the application, the gaze activity indicator GA and the gaze center time are obtained by the posture estimation of the eyeball:

[0127] The gaze posture GazeDir is composed of the Euler angles obtained by the eyeball estimation: the roll angle P α , the pitch angle P β , and the yaw angle P γ , and formula (23) defines the calculation method of the gaze posture GazeDir:

[0128] GazeDir = Rovtvec(EOR·HOR) = (P α ,P β ,P γ ) (23)

[0129] where Rotvec is a function provided by scipy to convert rotation matrix to Rodrigues rotation vector form; EOR represents the rotation angle of the eyeball, which is the relative rotation angle between the left and right boundaries of the eyeball in the two-dimensional image calculated by the face key point algorithm; HOR represents the rotation matrix of the head, which is the matrix form of the Rodrigues rotation vector R of the head posture defined in formula (16).

[0130] The specific direction of the gaze in the application is defined by formula (24):

[0131]

[0132] where Pu T represents the pupil yaw angle P β u acquired by the self-correction module when the subject is staring at the center of the normal fixation picture, and the average value in the correction time window. The application defines: when P β > Pu T + 0.5, the gaze is deviated to the left; when P β < Pu T - 0.5, the gaze is deviated to the right; in other cases, the gaze is defined as the fixation center. In order to refine the extraction of the gaze feature, the application also introduces a gaze activity index GA, which will increase when P β changes within a certain amplitude, even if the specific posture change condition is not met.

[0133] The mouth aspect ratio (MAR) can be calculated according to formula (25):

[0134]

[0135] When the value of MAR meets the setting of the self-correction module, it will be marked as the start of the yawning process.

[0136] S3.3: The annotation method mainly combines the original annotation provided by the real life fatigue data set UTA-RLDD and the understanding of the Karolinska Sleepiness Scale (KSS), which is completed by four students majoring in electronic information and communication engineering, respectively. They watch the video (allowing rewinding or fast forwarding) in an independent environment and add annotations, and they are not allowed to communicate with each other during work to ensure the independence of the annotations. When the annotation deviates greatly from the original data, the segment data will be excluded and will not participate in the subsequent experiment.

[0137] S3.4: The final constructed dataset contains 1126 cases of sober state, 1084 cases of mild fatigue, and 1166 cases of severe fatigue, which are evenly distributed.

[0138] In S4:

[0139] S4.1: The present application adopts the subject-independent method LOPO for standardization, and the standardized object is the continuous feature in the dataset. Specifically, the mean and standard deviation are calculated by the feature value extracted from each subject in the sober state, and the data provided by the corresponding subject is standardized. The specific calculation method is shown in formula (26):

[0140]

[0141] Wherein, F represents the standardized feature value, F n,m μ represents the original feature value, μ n,m σ and σ n,m respectively represent the mean and standard deviation of the continuous data collected from subject n and subject m in the sober state.

[0142] S4.2: The data input into the multi-layer perception model needs to be standardized by the subject-independent method and then processed again by the deviation normalization method. For the random forest and the limit gradient boosting model, only the subject-independent method is used for standardization, and the normalization method is not used.

[0143] S4.3: In the present application, a variety of machine learning algorithms are used to complete modeling, including random forest RF, multi-layer perception MLP and limit gradient boosting XGBoost algorithm.

[0144] S4.3.1: Random forest is a machine learning algorithm that improves the accuracy and stability of classification or regression by constructing multiple decision trees and combining their results. Its principle is as follows:

[0145] First, a plurality of sub-datasets d j is generated by sampling with replacement from the training dataset D containing N samples.

[0146] For each sub-dataset d j , a random forest trains a decision tree, and in the process of building each tree, a randomly selected subset is used for feature selection at each node to increase the diversity of the model, and the feature selection step is shown in formula (27):

[0147]

[0148] Wherein, Ft is the full set of features, Ft jis the candidate feature subset of node j, in Ft j * * and its corresponding split threshold t k to maximize the purity after split, the commonly used purity index is Gini index, which is defined as formula (28):

[0149]

[0150] where p k is the proportion of the k-th class sample in the data set, when splitting the node, select the feature and threshold that can maximize the reduction of Gini index to reduce impurity.

[0151] Repeat the feature subset selection and node splitting process in each sub-node until the maximum depth of the tree or the number of samples in the leaf node is below the set threshold. Repeat this decision tree construction step j times, at this time the generated random forest is shown in formula (29):

[0152] Forest={Tree1,Tree2,...,Tree j} (29)

[0153] For classification tasks, random forest makes a final classification result by voting on the prediction results of each decision tree, and the class with the most occurrences is taken as the final classification result.

[0154] S4.3.2: Multi-layer perception is usually composed of input layer, hidden layer and output layer, between the input layer and the output layer, assuming there are L layers of hidden layers, the output of the l-th layer of hidden layer is h l , then the structure of its model is shown in formula (30):

[0155]

[0156] where X represents the feature matrix, W l and b l represent the weight matrix and bias of the l-th layer respectively. f is the activation function to introduce nonlinearity, and the activation function used in the present application is ReLu function.

[0157] The last layer maps the activation value of the hidden layer to the label of the fatigue state, and uses the Softmax function to complete the probability normalization, which is shown in formula (31):

[0158]

[0159] where y represents the model output fatigue state classification result of the i-th sample.

[0160] The training of the multi-layer perceptron relies on the minimization of the loss function, and the cross-entropy loss Loss is used in the present application, which is shown in equation (32):

[0161]

[0162] where y i,c is the true fatigue state label of the i-th sample, and the weight matrix W l is constantly adjusted through backpropagation, so that the model can learn the mapping relationship between the features and the labels.

[0163] S4.3.3: The extreme gradient boosting algorithm is an additive model that integrates decision trees through the boosting method. The objective function is shown in equation (33):

[0164]

[0165] where Ω is the regularization term, and f k is the prediction function of the k-th tree. In the extreme gradient boosting algorithm, the classification result of the sample x i after t-1 iterations is represented as:

[0166]

[0167] where f t (x i ) is the prediction of the t-th tree for the sample x i , and is the prediction value of the t-1 iteration.

[0168] The core idea of the extreme gradient boosting algorithm is to construct multiple decision trees, each of which learns the prediction error of the previous round, and through iterative methods, the model error is continuously reduced, so as to obtain the final model. The output result of the final model is the combination of all trees.

[0169] In the S5:

[0170] S5.1: In the present application, the data set is divided by the hold-out method (20% / 30%), 10-fold cross-validation, stratified cross-validation, and subject-independent validation LOPO. The test sets generated based on different proportions, data balance, and independence are used to evaluate the random forest RF, multi-layer perceptron MLP, and extreme gradient boosting XGBoost models trained by the corresponding training sets, so as to obtain the optimal model.

[0171] The specific steps are as follows:

[0172] ​Leave-one-out method: randomly divide the dataset into training dataset (70-80%) and test dataset (20-30%), respectively using 80:20 and 70:30 ratio split;

[0173] K-fold cross-validation method: divide the dataset into K sub-folds, iteratively use K-1 folds for training and 1 fold for validation;

[0174] Stratified K-fold cross-validation method: on the basis of K-fold cross-validation method, ensure that the labels of each class (awake, mild fatigue, severe fatigue) are evenly distributed in the training dataset and the test dataset;

[0175] Subject-independent validation method: group the data according to the subject number, create 59 groups for 60 subjects, this cross-validation is performed 60 times, each iteration uses the data of one subject for validation and the data of the remaining subjects for training.

[0176] It should be noted that in the leave-one-out method, K-fold cross-validation and stratified K-fold cross-validation, the training dataset and test dataset are divided first, then the awake state in the training dataset is used to extract continuous features, the mean and standard deviation are calculated, the data is standardized according to the subject number, and the test set data is completely independent of the calculation of the mean and standard deviation of the training set, so as to avoid the risk of data leakage.

[0177] The label distribution of the leave-one-out method is shown in Table 2.

[0178] Table 2

[0179]

[0180] The label distribution of K-fold cross-validation method and stratified K-fold cross-validation method is shown in Table 3, where 0 represents the awake state, 5 represents the mild fatigue state, and 10 represents the severe fatigue state. K=10.

[0181] Table 3

[0182]

[0183] S5.2: In the present invention, multiple indicators are used to test and compare the performance of the scheme, assuming that the number of normal samples classified as normal is defined as TP, the number of normal samples classified as abnormal is defined as FP, the number of abnormal samples classified as normal is defined as FN, and the number of abnormal samples classified as abnormal is defined as TN.

[0184] The calculation formulas of accuracy, recall, precision and F1 score are as follows:

[0185]

[0186]

[0187]

[0188] Among them, since the method in the application is aimed at multi-classification problem, the recall rate Recall and the precision rate Precision in the evaluation index are obtained by micro-averaging method after individual calculation of each class.

[0189] S5.3: Use the hold-out method provided in Table 2, the K-fold cross-validation method provided in Table 3, and the stratified K-fold cross-validation method and the division method of subject-independent LOPO to divide the test set and build a model for debugging, and obtain the optimal model.

[0190] In the S6:

[0191] S6.1: Self-correcting feature extraction module design: In the application, a period of time after detection is defined as a correction window, and the extracted eye aspect ratio EAR, mouth aspect ratio MAR, eye pitch angle P β and the rotation angle Roll of the normal orientation of the head posture T , the pitch angle Pitch T , the yaw angle Yaw T are used as calibration data to set the blink threshold, the yawning threshold, the head posture correction, and the line of sight correction. The data collected in the correction window is not input into the model, and the default window length is thirty seconds.

[0192] The correction method of the blink threshold and the yawning threshold is shown in formula (38):

[0193]

[0194] Among them, A EAR is the calculated corresponding threshold, is the average of the maximum value of EAR in each blink in the correction window, is the average of the minimum value of EAR in each blink in the correction window, N is the number of blinks in the correction window, C EAR is the EAR adjustment factor, and the value is 0.2. A MAR is the calculated corresponding threshold, is the average of the maximum value of MAR in each blink in the correction window, is the average of the minimum value of MAR in each blink in the correction window, C MAR is the MAR adjustment factor, and the value is 0.7.

[0195] The head posture correction method is defined as formula (21), and the line of sight correction method is defined as formula (24).

[0196] S6.2: Preprocessing module: After self-correction is completed, the mean and standard deviation of the first 1 / 3 continuous features in the wakeful state are calculated based on the data collected during the correction period, which are used as the parameters for subsequent data normalization. The subsequent collected data is standardized by using the parameters to ensure the stability and accuracy of the model operation, and the preprocessing method is defined as formula (26).

[0197] S6.3: Integrated model classification module: Save the trained optimal model as a persistent file, load the model file when using, and classify the standardized features. Through the integrated learning method, the classification results of multiple models are combined to form the final prediction result by weighting, and the weight ratio of random forest RF, multilayer perception MLP and extreme gradient boosting XGBoost is 2:1:3.

[0198] S6.4: Combine the self-correction feature extraction module, the preprocessing module, the integrated model classification module, the target detection model and the face key point detection model to build a driver real-time fatigue detection system, and the overall structure of the system is shown in Figure 2 .

[0199] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the purpose and scope of the present technical solutions, which should be covered in the scope of the claims of the present application.

Claims

1. A method for fatigue detection based on machine learning and mixed features, characterized in that: The method comprises the following steps: S1: data collection: obtain the UTA-RLDD dataset and process it into non-overlapping video clips; S2: face detection: train the target detection algorithm YOLOv8 to realize face recognition; S3: dataset construction: perform face key point detection based on the MediaPipe framework, complete feature calculation using the key points, and screen and label to construct a dataset; S4: data preprocessing: complete the standardization processing of continuous data based on the subject independent method LOPO, complete the normalization processing of continuous data in the input multilayer perceptron, and filter out the data with abnormal values; S5: model training: input the processed experimental data into the constructed model, including random forest RF, multilayer perceptron MLP and extreme gradient boosting XGBoost, obtain a fatigue detection classification model through autonomous learning and training, evaluate the model performance through evaluation indexes, and obtain an optimal model; S6: system construction: design a self-correcting feature extraction module and a preprocessing module, construct a fatigue state classification module through the voting method in ensemble learning, form a driver real-time fatigue detection system, and realize automatic detection and classification of the fatigue state of the driver; The S3 comprises the following steps: S3.1: The feature calculation is completed by using the two-dimensional coordinates of the 478 key points extracted from the face area by the MediaPipe framework; these features can be divided into four categories: eye features, head features, mouth features, and line-of-sight features, a total of 20 features, and these features are expressed as a feature vector x, x ∈ R m ,m = 20; Among the non-conventional features, the eye movement Amplitude i , the opening eye Velocity i and the eye closure Perclos are calculated by the eye aspect ratio EAR in combination with a time window: Amplitude of eye movement Amplitude is defined by the formula (1) i Method of calculation: where EAR Start is the EAR value at the beginning of the blink process, EAR End is the EAR value at the end of the blink process, and EAR Peak is the EAR value at the lowest point of the blink process. Velocity of the opening wave is defined by formula (2) i Method of calculation: wherein Time Open is the time required for a blink from closure to opening; Formula (3) defines the calculation method of the proportion Perclos of the eye closing time in the time window: wherein Time Close is the time of eye closure within the time window, Time Length is the length of the time window, the length of the time window used is 30s, the data frame rate used is 30FPS, i.e. Time Length has a value of 900; Among the non-traditional features, the head pitch angle average value PA, the head activity HA and the head down time HD per unit time are calculated by combining the pitch angle a, the yaw angle β and the rotation angle γ of the head posture with the time window: Formula (4) defines the calculation method of the head pitch angle average value PA, and the feature explains the numerical value of the head down degree of the subject in the time window: wherein a i denotes the head pitch angle, i denotes the frame number; The head activity HA represents the frequency of the change of the head orientation in the time window, and the head posture orientation is defined by formula (5): where Roll T , Pitch T , Yaw T are the values of the rotation angle, the pitch angle, the yaw angle respectively in the normal orientation, obtained from the self-correction module; define θ y > (Pitch T + 15°) as the head-up, θ y < (Pitch T - 15°) as the head-down; θ z > (Yaw T + 15°) as the head-left, θ z < (Yaw T - 15°) as the head-right; thus the estimation of the head posture is completed, and the nodding and turning of the head are accurately detected; when the head orientation changes, the value of the head activity HA will increase; The head down time HD per unit time is defined by formula (6): Among the non-traditional features, the gaze activity index GA and the gaze center time are obtained by estimating the posture of the eyeball: Wherein, Rotvec is a function provided by scipy, which is used to convert the rotation matrix into the form of Rodrigues rotation vector; EOR represents the rotation angle of the eyeball, which is the relative rotation angle between the left boundary and the right boundary of the eyeball in the two-dimensional image calculated by the face key point algorithm; HOR represents the rotation matrix of the head; The Euler angle set: roll angle P α , pitch angle P β , and yaw angle P γ comprise the gaze pose GazeDir, and formula (7) defines the calculation method of the gaze pose GazeDir: GazeDir = Rovtvec(EOR H0R) = (P α ,P β ,P γ ) (7) The specific orientation of the gaze is defined by formula (8): S3.3: Based on the original annotation provided by the real life drowsiness dataset UTA-RLDD and the understanding of the Karolinska Sleepiness Scale KSS, four students majoring in electronic information and communication engineering complete the annotation respectively, watch the video in an independent environment and add the annotation, and mutual communication is not allowed during the work to ensure the independence of the annotation; when the annotation deviates from the original annotation, the clip data will be excluded and will not participate in the subsequent experiment; where Pu T represents the pupil yaw angle P of the subject when looking at the center of the normal fixation picture, obtained by the self-correction module β is the average value in the correction time window; when P β > Pu T , the line of sight is directed to the left; when P β < Pu T -0.5, the line of sight is directed to the right; in other cases, the line of sight is defined as the fixation center, and when the line of sight looks at the center in the time window, a feature, fixation center time, is introduced, which increases when the line of sight is in the fixation center state; To refine the extraction of gaze features, a feature, gaze activity index GA, is introduced. When P β When a certain amplitude of change occurs, even if the specific posture change condition is not reached, GA will increase accordingly; ​ S3.4: The constructed dataset contains 1126 cases of sober state, 1084 cases of mild fatigue, and 1166 cases of severe fatigue, and the distribution is balanced; The S4 includes the following steps: S4.1: Standardization is performed using the subject-independent method LOPO, and the standardized object is the continuous features in the dataset; the mean and standard deviation are calculated from the feature values extracted from each subject in the sober state, and the data provided by the corresponding subject is standardized, and the specific calculation method is shown in formula (9): in, F represents the standardized eigenvalues. n,m μ represents the original eigenvalues. n,m With σ n,m Let n and m represent the mean and standard deviation of continuous data collected from feature n and subject m while they are awake, respectively. The S6 includes the following steps: S6.1: Self-correction feature extraction module design: A period of time after detection starts is the correction window, the extracted eye aspect ratio EAR, mouth aspect ratio MAR, eye pitch angle P β and the rotation angle of the normal head posture downward Roll T , pitch angle Pitch T , yaw angle Yaw T as calibration data for setting the blink threshold, the yawning threshold, the head posture correction and the line of sight correction, the data collected in the correction window is not input into the model, and the default window length is thirty seconds; The correction method of the blinking threshold and the yawning threshold is shown in formula (10): wherein A EAR is a blink judgment threshold, is the maximum value of EAR in each blink within the correction window, is the minimum value of EAR in each blink within the correction window, N is the number of blinks within the correction window, C EAR is an EAR adjustment factor, and has a value of 0.2; A MAR is a yawn judgment threshold, is the maximum value of MAR in each blink within the correction window, is the minimum value of MAR in each blink within the correction window, C MAR is an MAR adjustment factor, and has a value of 0.7; The head posture correction method is defined as formula (5), and the line of sight correction method is defined as formula (8); S6.2: Preprocessing module: After self-correction, the mean and standard deviation of the first 1 / 3 continuous features in the sober state are calculated based on the data collected during the correction period, which are used as the parameters for subsequent data standardization; the subsequent collected data is standardized by using the parameters to ensure the stability and accuracy of the model running, and the preprocessing method is defined as formula (9); S6.3: Integrated model classification module: save the trained optimal model as a persistent file, load the model file when using, and classify the standardized features; through the integrated learning method, the classification results of multiple models are combined to form the final prediction result by weighting, and the weight ratio of random forest RF, multilayer perception MLP and extreme gradient boosting XGBoost is 2:1:3; S6.4: Combine the self-correction feature extraction module, preprocessing module, and integrated model classification module with the target detection model and face key point detection model to build a driver real-time fatigue detection system. 2.The fatigue detection method based on machine learning and hybrid features according to claim 1, characterized in that: The S1 is specifically: each video in the real-life fatigue dataset is divided into 30-second non-overlapping segments, and each segment is used as an independent data sample; in order to avoid overfitting caused by the high correlation of features in adjacent time periods, ensure that all segments are non-overlapping during the division process. 3.The fatigue detection method based on machine learning and hybrid features according to claim 2, characterized in that: The S2 includes the following steps: S2.1: Replace the backbone network of YOLOv8 with Faster-Net to realize the lightweight of the model, and use the Wider-Face dataset for training; during deployment, use TensorRT technology to realize model quantization to realize hardware acceleration and low-latency face detection; S2.2: Use YOLOv8 for face recognition to obtain the region of interest ROI, and ensure that the face is always located in the center of the picture to reduce the influence of the change of the position of the face in the two-dimensional image on the detection of key point coordinates by the face grid model. 4.The fatigue detection method based on machine learning and hybrid features according to claim 1, characterized in that: The S5 includes the following steps: S5.1: Use the leave-one-out method, 10-fold cross-validation, stratified cross-validation, and subject-independent validation LOPO method to divide the dataset, and evaluate the random forest RF, multilayer perception MLP, and extreme gradient boosting XGBoost models trained by the corresponding training set based on different proportions, data balance, and independence of the generated test set to obtain the optimal model.

Images