Fatigue driving monitoring and early warning method based on multi-dimensional feature fusion

Abstract

The invention discloses a fatigue driving monitoring and early warning method based on multi-dimensional feature fusion, and relates to the field of intelligent traffic and safe driving, and the method comprises the steps: carrying out the feature extraction of a collected face image of a vehicle driver through a feature extraction model based on a convolutional neural network, and obtaining a face key point; then calculating the eye closing rate, the mouth opening rate, the head posture data and the facial muscle data to finally obtain multi-dimensional features, and fusing the multi-dimensional features by using a multi-layer perceptron to obtain fused facial features; and then, a fatigue state detection model obtained based on recurrent neural network training is utilized to detect the facial feature time sequence at a plurality of continuous sampling moments so as to detect whether a fatigue driving state occurs and trigger alarm feedback. According to the method, the fatigue state of the driver can be identified and judged more accurately through combination of multi-dimensional data fusion and a time sequence analysis algorithm, and the monitoring and early warning accuracy and reliability are improved. And the system can keep high robustness even in an illumination change or complex driving environment.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation and safe driving, and in particular to a fatigue driving monitoring and early warning method based on multi-dimensional feature fusion. Background Art

[0002] With the popularization of modern transportation and the increase in driving frequency, fatigue driving has become one of the important causes of traffic accidents. Long-term driving, especially at night or on highways, can easily cause drivers to become fatigued, which in turn leads to problems such as slow reaction, decreased judgment, and lack of concentration, which seriously threatens road traffic safety. According to statistics, traffic accidents caused by fatigue driving account for a high proportion of the total number of accidents. How to effectively prevent fatigue driving has become an urgent problem to be solved.

[0003] At present, there are some driver status monitoring systems on the market. These systems usually determine whether the driver is in a state of fatigue by monitoring the driver's eye closure rate. However, the eye closure rate is difficult to accurately represent the driver's actual fatigue status, resulting in low inspection accuracy and prone to false alarms or missed alarms. Summary of the invention

[0004] In response to the above problems and technical requirements, this application proposes a fatigue driving monitoring and early warning method based on multi-dimensional feature fusion. The technical solution of this application is as follows:

[0005] A fatigue driving monitoring and early warning method based on multi-dimensional feature fusion, the fatigue driving monitoring and early warning method comprising:

[0006] The facial image of the vehicle driver is collected by the vehicle-mounted image collection module according to the sampling frequency;

[0007] The facial image is feature extracted using a feature extraction model based on a convolutional neural network to obtain facial key points, which include at least eye key points, mouth key points, eyebrow key points, nose tip key points and chin key points; eyebrow key points include eyebrow center key points and eyebrow tail key points, eye key points include eye head key points, eye tail key points and eyelid key points, and mouth key points include mouth corner key points and lip key points;

[0008] The eye closure rate, mouth opening rate, head posture data and facial muscle data at the current sampling moment are calculated according to the facial key points, the duration of the eye closure rate being lower than the closure rate threshold before the current sampling moment is calculated as the eye closure duration, the duration of the mouth opening rate reaching the opening rate threshold before the current sampling moment is calculated as the mouth opening duration, and the dynamic change characteristics of the facial muscle data at the current sampling moment and multiple consecutive sampling moments before and after the current sampling moment are extracted to obtain the expression change feature vector;

[0009] A multi-layer perceptron is used to fuse the eye closure duration, mouth opening duration, head posture data, and expression change feature vector to obtain the fused facial features at the current sampling moment.

[0010] Extract the facial feature time series composed of the fused facial features at the current sampling moment and multiple consecutive sampling moments before the current sampling moment, and use the fatigue state detection model obtained by recursive neural network training to detect the facial feature time series to obtain the fatigue index. The higher the fatigue index, the higher the fatigue degree of the vehicle driver represented.

[0011] When it is detected that the fatigue index reaches the fatigue threshold for a consecutive number of times reaching the number threshold, it is determined that the vehicle driver is in a fatigue driving state and an alarm feedback is triggered.

[0012] A further technical solution is to calculate facial muscle data based on facial key points, including:

[0013] Calculate the vertical distance between the eyebrow center key point and the eyebrow tail key points on both sides along the length direction of the face; and calculate the relative distance between the nose tip key point and the mouth corner key points on both sides; and calculate the relative distance between the nose tip key point and the eye tail key points on both sides.

[0014] A further technical solution is to calculate head posture data based on facial key points, including:

[0015] Use the PnP algorithm to project the nose tip key point, the eyebrow center key point, the two eye tail key points, the two mouth corner key points and the chin key point into the two-dimensional image to obtain the projection coordinates;

[0016] The solvePnP algorithm is used to calculate the head posture data including the left and right rotation angles, up and down tilt angles and horizontal rotation angles of the head based on the projection coordinates of the nose tip key point, the eyebrow center key point, the two eye tail key points, the two mouth corner key points and the chin key point.

[0017] A further technical solution is that the fatigue driving monitoring and early warning method further includes:

[0018] The number of false alarms and missed alarms is determined based on the change curve of the fatigue index during the historical monitoring process and the alarm feedback triggering record, wherein the fatigue index decline rate before and after the alarm feedback is triggered that does not reach the decline rate threshold is identified as a false alarm, and the fatigue index decline rate reaches the decline rate threshold but does not trigger the alarm feedback is identified as a missed alarm;

[0019] The fatigue threshold is dynamically adjusted according to the number of false alarms and missed alarms. The more false alarms there are and the lower the number of missed alarms, the higher the fatigue threshold is adjusted to.

[0020] A further technical solution is that the fatigue driving monitoring and early warning method further includes:

[0021] Obtain behavioral pattern data during historical monitoring, including eye closure rate, mouth opening rate, head posture data, and facial muscle data obtained at each sampling moment, as well as fused facial features and detected fatigue index at each sampling moment, and the number of false alarms and missed alarms; dynamically adjust the closure rate threshold and the opening rate threshold according to the behavioral pattern data, and dynamically adjust the model parameters of the multi-layer perceptron and the fatigue state detection model.

[0022] A further technical solution is that the fatigue driving monitoring and early warning method further includes:

[0023] obtaining the heart rate data of the vehicle driver through a heart rate monitoring device;

[0024] Detect the fluctuation characteristics of heart rate data by sliding window;

[0025] When an abnormal fluctuation characteristic of the heart rate data is detected, and when the fatigue index is detected to have reached the fatigue threshold for a consecutive number of times reaching the number threshold, it is determined that the vehicle driver is in a fatigue driving state and an alarm feedback is triggered.

[0026] A further technical solution is that the image acquisition module includes an RGB camera and an infrared camera, and the vehicle-mounted image acquisition module acquires the facial image of the vehicle driver according to a predetermined sampling frequency and performs feature extraction, including:

[0027] When the ambient light sensor is used to detect that the real-time lighting condition meets the lighting requirement, the camera parameters of the RGB camera are adjusted according to the real-time lighting condition, and a facial image in RGB format is acquired by using the RGB camera, and features of the facial image are extracted by using a first feature extraction model, where the first feature extraction model is pre-trained using the RGB image based on a convolutional neural network;

[0028] When the ambient light sensor detects that the real-time lighting conditions do not meet the lighting requirements, an infrared camera is used to obtain a facial image in infrared format, and a second feature extraction model is used to extract features from the facial image. The second feature extraction model is pre-trained using infrared images based on a convolutional neural network.

[0029] A further technical solution is to determine that the vehicle driver is in a fatigue driving state and trigger an alarm feedback:

[0030] When it is determined that the vehicle driver is in a fatigued driving state, the vehicle is triggered to enter the automatic driving mode, and an alarm feedback is triggered according to the fatigue index. The higher the fatigue index, the higher the intensity of the triggered alarm feedback.

[0031] A further technical solution is that the fatigue driving monitoring and early warning method further includes:

[0032] The sampling frequency is dynamically adjusted according to the driving scenario. The higher the vehicle speed and / or the longer the vehicle continuously drives, the higher the sampling frequency.

[0033] A further technical solution is that the fatigue driving monitoring and early warning method further includes:

[0034] According to the frequency of the vehicle driver's fatigue driving state and the duration of each fatigue driving state, and when the frequency of the fatigue driving state reaches a frequency threshold and / or the duration of the fatigue driving state reaches a duration threshold, a driving safety reminder message is output.

[0035] The beneficial technical effects of this application are:

[0036] This application discloses a fatigue driving monitoring and early warning method based on multi-dimensional feature fusion. This method uses a deep learning algorithm to extract and fuse multi-dimensional features such as eye closure rate, mouth opening rate, head posture, and facial muscle data, and then uses a recursive neural network to detect fatigue feature changes in a continuous time period through a time series analysis algorithm to achieve judgment of the driver's fatigue state. It can more accurately identify and judge the driver's fatigue state and improve the accuracy and reliability of monitoring and early warning. Even in light changes or complex driving environments, the system can maintain high robustness, and this method is based on a lightweight neural network model, which ensures real-time and high efficiency and is suitable for vehicle-mounted environments.

[0037] This method can also dynamically update the model and various thresholds based on the behavioral pattern data during actual operation, so that it has good environmental adaptability and personalized adjustment capabilities, and has personalized reminder functions, which is conducive to reducing false alarm rates and missed alarm rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a method flow chart of a fatigue driving monitoring and early warning method according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] The specific implementation of the present application is further described below in conjunction with the accompanying drawings.

[0040] This application discloses a fatigue driving monitoring and early warning method based on multi-dimensional feature fusion, please refer to Figure 1 As shown in the flowchart, the fatigue driving monitoring and early warning method includes:

[0041] Step 1: Collect the facial image of the vehicle driver through the vehicle-mounted image acquisition module according to the sampling frequency. When the image acquisition module is installed, it is necessary to ensure that the field of view covers the complete facial area of ​​the vehicle driver. The sampling frequency must be high enough to ensure that the driver's facial changes are captured in real time. It is usually set to 30 frames per second. Considering that the vehicle-mounted scene is greatly affected by the lighting conditions, in one embodiment, the image acquisition module includes an RGB camera and an infrared camera, and uses an ambient light sensor to detect the real-time lighting conditions:

[0042] When the ambient light sensor detects that the real-time light condition meets the light requirement, the camera parameters of the RGB camera are adjusted according to the real-time light condition and the facial image in RGB format is acquired by the RGB camera. The adjusted camera parameters include brightness and contrast, so as to ensure the clarity of the acquired facial features as much as possible.

[0043] However, when the ambient light sensor detects that the real-time lighting conditions do not meet the lighting requirements, such as in complex environments such as night scenes or strong light, the performance of the traditional visual detection system decreases, so it automatically switches to using an infrared camera to obtain facial images in infrared format, thereby reducing the detection error caused by lighting effects.

[0044] Step 2: Use a feature extraction model based on a convolutional neural network to extract features from facial images to obtain facial key points.

[0045] In one embodiment, in order to improve the accuracy of feature extraction, the facial image is first preprocessed before feature extraction. The image preprocessing operation includes: (1) mean filtering, first eliminating the noise in the facial image through mean filtering to enhance the smoothness of the facial image. (2) histogram equalization, further enhancing the contrast of the image, especially in an environment with poor lighting conditions, this step helps to improve the clarity of facial features. (3) binarization, binarizing the facial image that has completed the mean filtering to extract the facial contour of the vehicle driver, thereby reducing irrelevant background interference.

[0046] Then, the feature extraction model is used to extract facial key points, including at least the classic 68 facial key points. The extracted facial key points include at least eye key points, mouth key points, eyebrow key points, nose tip key points and chin key points. The eyebrow key points include the eyebrow center key points and eyebrow tail key points, and actually also include the key points at the eyebrow arch, etc. The eye key points include the eye head key points, eye tail key points and multiple eyelid key points. The mouth key points include the mouth corner key points and multiple lip key points.

[0047] The feature extraction model is pre-trained based on a convolutional neural network using a large-scale driver facial image dataset. The convolutional neural network uses lightweight architectures such as MobileNet to reduce computational complexity while maintaining high detection accuracy. The convolutional neural network generates multiple feature layers, representing facial features at different levels, and extracts facial key points including eyes, mouth, eyebrows, nose, and chin. The dataset used for model training contains driver facial images under different lighting conditions and different posture angles. In order to enhance the generalization ability of the model, data enhancement techniques such as random affine transformation and brightness adjustment are added during the training process. By learning from massive data, the system has strong adaptability and can maintain high-precision detection effects in complex driving environments.

[0048] In addition, since the image acquisition module of the present application integrates an RGB camera and an infrared camera, the pre-trained feature extraction model also includes a first feature extraction model pre-trained using an RGB image based on a convolutional neural network, and a second feature extraction model pre-trained using an infrared image based on a convolutional neural network. When step 1 acquires a facial image in RGB format, the first feature extraction model is used to extract features from the facial image. When step 1 acquires a facial image in infrared format, the second feature extraction model is used to extract features from the facial image.

[0049] Step 3: Calculate the eye closure rate, mouth opening rate, head posture data and facial muscle data at the current sampling moment based on the facial key points.

[0050] The eye closure rate is calculated using eye key points, including eye head key points, eye tail key points and multiple eyelid key points at the upper and lower eyelids. First, the eye vertical distance is calculated using the eyelid key points at the upper and lower eyelids, and then the eye horizontal distance is calculated using the eye head key points and eye tail key points. The ratio of the eye vertical distance to the eye horizontal distance is used as the eye closure rate. The specific calculation method can refer to the existing calculation method and will not be repeated here. Then, the time when the eye closure rate is detected to be lower than the closure rate threshold before the current sampling moment is calculated as the eye closure duration, including determining the number of times the eye closure rate is lower than the closure rate threshold detected continuously at the current sampling moment and multiple previous sampling moments. The eye closure duration can be calculated in combination with the sampling interval. The closure rate threshold here is generally set to 0.2 according to experience.

[0051] The mouth opening rate is calculated using the mouth key points, including the mouth corner key points and multiple lip key points at the upper and lower lips. First, the vertical distance of the mouth is calculated using the lip key points at the upper and lower lips, and then the horizontal distance of the mouth is calculated using the mouth corner key points on both sides. The ratio of the vertical distance of the mouth to the horizontal distance of the mouth is used as the mouth opening rate. The specific calculation method can refer to the existing calculation method, which will not be repeated here. Then, the time when the mouth opening rate is detected to reach the opening rate threshold before the current sampling time is calculated as the mouth opening time, including determining the number of times the mouth opening rate is continuously detected at the current sampling time and the previous multiple sampling times to reach the opening rate threshold. The mouth opening time can be calculated in combination with the sampling interval. The opening rate threshold here is generally set to 0.6 according to experience.

[0052] Calculating the head posture data based on the facial key points includes: using the PnP algorithm to project the nose tip key point, the eyebrow key point, the two eye tail key points, the two mouth corner key points and the chin key point to the two-dimensional image to obtain the projection coordinates. Then, the solvePnP algorithm is used to calculate the rotation matrix based on the projection coordinates of the nose tip key point, the eyebrow key point, the two eye tail key points, the two mouth corner key points and the chin key point, so that the head posture data including the left and right rotation angle, the up and down tilt angle and the horizontal rotation angle of the head can be extracted from the rotation matrix. The specific algorithm process of the solvePnP algorithm will not be repeated here, or other similar algorithms can be used for calculation.

[0053] Calculating facial muscle data includes calculating the relative distances between multiple groups of facial key points, including: calculating the vertical distances between the eyebrow center key point and the eyebrow tail key points on both sides along the length of the face; and calculating the relative distances between the nose tip key point and the mouth corner key points on both sides; and calculating the relative distances between the nose tip key point and the eye tail key points on both sides. Then, the dynamic change characteristics of the facial muscle data at the current sampling moment and the previous multiple continuous sampling moments are extracted to obtain the expression change feature vector, including taking the dynamic changes of the time series of each group of relative distances, and using the dimensionality reduction method to generate the expression change feature vector. The dynamic change here can be achieved by differential or rate change, and the dimensionality reduction method used can be principal component analysis or other deep learning methods. The expression change feature vector obtained is a high-dimensional vector that represents the relationship between different facial points.

[0054] Step 4: Use a multi-layer perceptron to fuse the eye closure duration, mouth opening duration, head posture data, and expression change feature vectors to obtain the fused facial features at the current sampling moment. This step uses a multi-layer perceptron (MLP) instead of a simple weighted sum to fuse the above-mentioned multiple features. MLP can capture the complex nonlinear relationship between different features through activation functions (such as ReLU, Sigmoid, etc.), and uses a graph neural network (GNN). In GNN, each feature can be regarded as a node in the graph, and the coupling relationship between different features can be represented by edges. Through the propagation mechanism of the graph neural network, information can flow between feature nodes, further capturing the dependency and interaction effects between features.

[0055] Step 5, extract the facial feature time series composed of the fused facial features of the current sampling moment and the multiple consecutive sampling moments before it, and use the fatigue state detection model obtained by recursive neural network training to detect the facial feature time series to obtain the fatigue index. Then the fatigue index will be compared with the fatigue threshold. The higher the fatigue index obtained, the higher the fatigue degree of the vehicle driver represented. When the fatigue index reaches the fatigue threshold, it means that the vehicle driver is detected to be in a fatigue state.

[0056] That is, this application does not only detect a single feature at a single moment, but uses a recursive neural network to perform time series analysis on the fused facial features, thereby effectively capturing the dynamic information of the driver's facial features changing over time, and improving the model's ability to capture long-term fatigue states. Through time series analysis of multiple features such as eyes, mouth, and head posture, the system can accurately predict the driver's fatigue trend at an early stage and issue a warning in a timely manner.

[0057] In the initial use stage, the system uses multi-dimensional physiological and behavioral data to train the fatigue state detection model based on the recursive neural network and initialize the fatigue threshold through long-term interactive learning. In order to adapt to the individual differences of different vehicle drivers, in one embodiment, the behavior pattern data in the historical monitoring process will also be obtained, and the behavior pattern data here includes the eye closure rate, mouth opening rate, head posture data and facial muscle data obtained at each sampling time, as well as the fatigue index obtained by fusion facial features and detection at each sampling time, and the number of false alarms and missed alarms. Then, the closure rate threshold and the opening rate threshold are dynamically adjusted according to the behavior pattern data, and the model parameters of the multi-layer perceptron and the fatigue state detection model are dynamically adjusted. And the fatigue threshold is dynamically adjusted according to the number of false alarms and missed alarms. The more false alarms and the lower the number of missed alarms, the higher the fatigue threshold is adjusted. Among them, the number of false alarms and the number of missed alarms can be statistically determined based on the change curve of the fatigue index in the historical monitoring process and the alarm feedback trigger record, including identifying the fatigue index decline rate before and after the alarm feedback is triggered as a false alarm if it does not reach the decline rate threshold, and identifying the fatigue index decline rate as a missed alarm if it reaches the decline rate threshold but does not trigger the alarm feedback.

[0058] The above-mentioned adaptive calibration process based on behavioral pattern data can meet the high adaptability and flexibility for different vehicle drivers by real-time monitoring and feedback of the driver's physiological and behavioral changes, intelligently adjusting the model parameters and fatigue thresholds. In actual operation, by collecting the driver's characteristic data, dynamically updating the model parameters, and making personalized adjustments according to different scenarios and driver's behavioral habits, it can always maintain high reliability and accuracy in complex and changing driving environments, and finally realize a real-time self-adjustment and feedback mechanism based on the driver's status. Through the self-supervised learning mechanism, the accuracy of the detection model can be continuously improved without the need for a large amount of labeled data. For example, different vehicle drivers have different physiological differences (some people have a low eye closure rate but are not tired), driving habits (frequently lowering their heads or keeping their eyes open for a long time), environmental adaptability (poor light at night causes increased feature noise) and behavioral response patterns (speed and effectiveness of response to alarms). Based on these behavioral pattern data, the weights in the multi-layer perceptron can be dynamically adjusted to dynamically adjust the sensitivity of different feature fusions, such as adaptively increasing the weight of eye closure time for drivers with longer eye closure time, or increasing the importance of head posture when the tendency to lower the head is obvious. Through reinforcement learning or online learning mechanisms, the system continuously updates model parameters and gradually forms a fatigue detection solution that is personalized and optimized for each driver, improving detection accuracy and reducing false alarms and missed alarms. The system improves the model by dynamically adjusting fatigue thresholds, optimizing feature weights, and updating time windows (extending the judgment period of fatigue trends). Each optimized parameter directly acts on the detection process, and through continuous iteration, the system's adaptability to driver behavior and scenarios is improved, reducing false and missed judgments, and improving detection accuracy.

[0059] Step 6: When it is detected that the fatigue index reaches the fatigue threshold for a consecutive number of times reaching the number threshold, it is determined that the vehicle driver is in a fatigue driving state and an alarm feedback is triggered.

[0060] In another embodiment, the method also obtains the heart rate data of the vehicle driver through a heart rate monitoring device. The heart rate monitoring device here can be a system-installed device, or a wearable device used by the vehicle driver himself, and the heart rate data can be obtained by interacting with the heart rate monitoring device. Then, the fluctuation characteristics of the heart rate data are detected by means of a sliding window, including the heart rate change rate and the fluctuation amplitude. When the fluctuation characteristics of the heart rate data are detected to be abnormal, and when it is detected that the fatigue index reaches the fatigue threshold for a consecutive number of times reaching the number threshold, it is finally determined that the vehicle driver is in a fatigue driving state and an alarm feedback is triggered. In this embodiment, the heart rate analysis and facial feature analysis can be combined to finally determine whether the vehicle driver is in a fatigue driving state, which is conducive to improving the accuracy and reliability of the judgment.

[0061] When it is determined that the vehicle driver is in a fatigue driving state and triggers an alarm feedback, the alarm feedback is triggered according to the fatigue index. The higher the fatigue index, the higher the intensity of the triggered alarm feedback. The specific trigger feedback has multiple forms, including visual, auditory and tactile. For example, when the fatigue index is low, a low-intensity alarm feedback is issued by flashing the instrument panel lights, while when the fatigue index is high, a high-intensity alarm feedback is triggered by voice alarm combined with seat vibration.

[0062] In addition, when the vehicle has an automatic driving function, it can also trigger the vehicle to enter the automatic driving mode when it is determined that the vehicle driver is in a fatigue driving state, thereby taking over the control of the vehicle and avoiding the potential safety risks caused by fatigue driving.

[0063] In another embodiment, the sampling frequency is dynamically adjusted according to the driving scenario. The higher the vehicle speed and / or the longer the vehicle continuously drives, the higher the sampling frequency. In this way, in high-speed driving or long-distance driving scenarios, the frequency of fatigue monitoring is increased to reduce the false alarm rate and improve safety.

[0064] In another embodiment, based on the frequency of the vehicle driver's fatigue driving state and the duration of each fatigue driving state, and when the frequency of the fatigue driving state reaches a frequency threshold and / or the duration of the fatigue driving state reaches a duration threshold, driving safety reminder information is output to provide rest suggestions, which is conducive to helping the driver form safe driving habits.

[0065] The above is only a preferred embodiment of the present application, and the present application is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present application should be considered to be included in the protection scope of the present application.

Claims

1. A fatigue driving monitoring and early warning method based on multi-dimensional feature fusion, characterized in that: The fatigue driving monitoring and early warning method comprises: The facial image of the vehicle driver is collected by the vehicle-mounted image collection module according to the sampling frequency; The facial image is subjected to feature extraction using a feature extraction model based on a convolutional neural network to obtain facial key points, wherein the facial key points include at least eye key points, mouth key points, eyebrow key points, nose tip key points and chin key points; eyebrow key points include eyebrow center key points and eyebrow tail key points, eye key points include eye head key points, eye tail key points and eyelid key points, and mouth key points include mouth corner key points and lip key points; The eye closure rate, mouth opening rate, head posture data and facial muscle data at the current sampling moment are calculated according to the facial key points, the duration of the eye closure rate being lower than the closure rate threshold before the current sampling moment is calculated as the eye closure duration, the duration of the mouth opening rate reaching the opening rate threshold before the current sampling moment is calculated as the mouth opening duration, and the dynamic change characteristics of the facial muscle data at the current sampling moment and multiple consecutive sampling moments before and after the current sampling moment are extracted to obtain the expression change feature vector; A multi-layer perceptron is used to fuse the eye closure duration, mouth opening duration, head posture data, and expression change feature vector to obtain the fused facial features at the current sampling moment. Extract the facial feature time series composed of the fused facial features at the current sampling moment and multiple consecutive sampling moments before the current sampling moment, and use the fatigue state detection model obtained by recursive neural network training to detect the facial feature time series to obtain the fatigue index. The higher the fatigue index, the higher the fatigue degree of the vehicle driver represented. When it is detected that the fatigue index reaches the fatigue threshold for a consecutive number of times reaching the number threshold, it is determined that the vehicle driver is in a fatigue driving state and an alarm feedback is triggered.

2. The fatigue driving monitoring and early warning method according to claim 1 is characterized in that: Calculating facial muscle data based on facial key points includes: Calculate the vertical distance between the eyebrow center key point and the eyebrow tail key points on both sides along the length direction of the face; and calculate the relative distance between the nose tip key point and the mouth corner key points on both sides; and calculate the relative distance between the nose tip key point and the eye tail key points on both sides.

3. The fatigue driving monitoring and early warning method according to claim 1 is characterized in that: Calculating head posture data based on facial key points includes: Use the PnP algorithm to project the nose tip key point, the eyebrow center key point, the two eye tail key points, the two mouth corner key points and the chin key point into the two-dimensional image to obtain the projection coordinates; The solvePnP algorithm is used to calculate the head posture data including the left and right rotation angles, up and down tilt angles and horizontal rotation angles of the head based on the projection coordinates of the nose tip key point, the eyebrow center key point, the two eye tail key points, the two mouth corner key points and the chin key point.

4. The fatigue driving monitoring and early warning method according to claim 1 is characterized in that: The fatigue driving monitoring and early warning method also includes: The number of false alarms and missed alarms is determined based on the change curve of the fatigue index during the historical monitoring process and the alarm feedback triggering record, wherein the fatigue index decline rate before and after the alarm feedback is triggered that does not reach the decline rate threshold is identified as a false alarm, and the fatigue index decline rate reaches the decline rate threshold but does not trigger the alarm feedback is identified as a missed alarm; The fatigue threshold is dynamically adjusted according to the number of false alarms and the number of missed alarms. The more false alarms there are and the lower the number of missed alarms there are, the higher the fatigue threshold is adjusted to.

5. The fatigue driving monitoring and early warning method according to claim 4 is characterized in that: The fatigue driving monitoring and early warning method also includes: Obtain behavioral pattern data during historical monitoring, including eye closure rate, mouth opening rate, head posture data, and facial muscle data obtained at each sampling moment, as well as fused facial features and detected fatigue index at each sampling moment, and the number of false alarms and missed alarms; dynamically adjust the closure rate threshold and the opening rate threshold according to the behavioral pattern data, and dynamically adjust the model parameters of the multi-layer perceptron and the fatigue state detection model.

6. The fatigue driving monitoring and early warning method according to claim 1 is characterized in that: The fatigue driving monitoring and early warning method also includes: obtaining the heart rate data of the vehicle driver through a heart rate monitoring device; Detect the fluctuation characteristics of heart rate data by sliding window; When it is detected that the fluctuation characteristics of the heart rate data are abnormal, and when it is detected that the fatigue index reaches the fatigue threshold for a consecutive number of times that reaches the number threshold, it is determined that the vehicle driver is in a fatigue driving state and an alarm feedback is triggered.

7. The fatigue driving monitoring and early warning method according to claim 1 is characterized in that: The image acquisition module includes an RGB camera and an infrared camera. The vehicle-mounted image acquisition module acquires the facial image of the vehicle driver at a predetermined sampling frequency and performs feature extraction, including: When the ambient light sensor is used to detect that the real-time lighting condition meets the lighting requirement, the camera parameters of the RGB camera are adjusted according to the real-time lighting condition, and a facial image in RGB format is acquired by using the RGB camera, and features are extracted from the facial image by using a first feature extraction model, where the first feature extraction model is pre-trained using the RGB image based on a convolutional neural network; When the ambient light sensor detects that the real-time lighting conditions do not meet the lighting requirements, an infrared camera is used to obtain a facial image in infrared format, and a second feature extraction model is used to extract features from the facial image. The second feature extraction model is pre-trained using infrared images based on a convolutional neural network.

8. The fatigue driving monitoring and early warning method according to claim 1 is characterized in that: Determining that the vehicle driver is in a fatigue driving state and triggering an alarm feedback: When it is determined that the vehicle driver is in a fatigue driving state, the vehicle is triggered to enter the automatic driving mode, and an alarm feedback is triggered according to the fatigue index. The higher the fatigue index, the higher the intensity of the triggered alarm feedback.

9. The fatigue driving monitoring and early warning method according to claim 1 is characterized in that: The fatigue driving monitoring and early warning method also includes: The sampling frequency is dynamically adjusted according to the driving scenario. The higher the vehicle speed and / or the longer the vehicle continuously drives, the higher the sampling frequency.

10. The fatigue driving monitoring and early warning method according to claim 1 is characterized in that: The fatigue driving monitoring and early warning method also includes: According to the frequency of the vehicle driver's fatigue driving state and the duration of each fatigue driving state, and when the frequency of the fatigue driving state reaches a frequency threshold and / or the duration of the fatigue driving state reaches a duration threshold, a driving safety reminder message is output.

Images