A method and system for monitoring driver fatigue status

Abstract

The invention relates to the technical field of driver fatigue monitoring, and specifically discloses a method and system for monitoring driver fatigue status. It includes: obtaining the driver's real-time driving information, and obtaining the driver's multimodal data and vehicle driving operation data based on the driver's real-time driving information, wherein the multimodal data includes the driver's eye data, the driver's mouth data and the driver's head position data; obtaining the blinking frequency, pupil diameter change value and gaze duration based on the driver's eye data, and obtaining the driver's stress assessment value based on the vehicle driving operation data, blinking frequency, pupil diameter change value and gaze duration; obtaining the driver's fatigue assessment value based on the driver's eye data, the driver's mouth data and the driver's head position data. By comprehensively considering the three key factors of stress, fatigue and emotion, the individual's comprehensive fatigue state can be evaluated more comprehensively and accurately.

Description

Technical Field

[0001] The present invention relates to the technical field of driver fatigue monitoring, and in particular to a method and system for monitoring a driver's fatigue state. Background Art

[0002] Driver fatigue is a long-standing and highly concerned issue in road traffic safety. With the development of modern transportation, drivers are faced with longer driving time, more complex road environments and higher work pressure, which leads to an increasing risk of fatigue driving. Fatigue can cause a decrease in the driver's attention, reaction ability and judgment, increasing the possibility of traffic accidents.

[0003] Existing driver fatigue assessment methods often only focus on indicators such as driver working hours, such as facial expression indicators, driver emotions and stress. Although they provide us with reference information about the driver's fatigue status to a certain extent, they often have certain limitations and ignore the complex connections and mutual influences between these indicators. Existing monitoring methods cannot accurately link facial expression indicators, driver working hours, driver emotions and stress together, resulting in low accuracy of driver fatigue assessment. Summary of the invention

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A method for monitoring a driver's fatigue state, comprising:

[0006] Acquire real-time driving information of the driver, and acquire multimodal data of the driver and vehicle driving operation data according to the real-time driving information of the driver, wherein the multimodal data includes driver eye data, driver mouth data and driver head position data;

[0007] Obtaining blinking frequency, pupil diameter change value and gaze duration according to the driver's eye data, and obtaining a driver stress assessment value according to the vehicle driving operation data, blinking frequency, pupil diameter change value and gaze duration;

[0008] Obtaining a driver's fatigue assessment value based on the driver's eye data, the driver's mouth data, and the driver's head position data;

[0009] Acquire the driver's facial expression information according to the driver's eye data and the driver's mouth data, and acquire the duration of the facial expression according to the driver's facial expression information;

[0010] Obtaining the driver's vehicle operation frequency from the vehicle driving operation data according to the duration of the expression, and obtaining an emotion evaluation value according to the operation frequency and the driver's expression information;

[0011] The comprehensive fatigue value is calculated based on the stress assessment value, fatigue assessment value and emotion assessment value.

[0012] Preferably, the step of acquiring the driver's multimodal data and the vehicle driving operation data according to the driver's real-time driving information includes:

[0013] Acquire multiple driver face image information and multiple vehicle operation image information according to the real-time driving information of the driver at a preset time;

[0014] Acquire a set of eye image information, a set of mouth image information, and a set of head position image information according to the facial image information of each driver;

[0015] Acquiring driver eye data according to multiple sets of eye image information;

[0016] Acquire driver mouth data according to multiple sets of mouth image information;

[0017] Acquiring driver's head position data according to multiple sets of head position image information;

[0018] The optical flow image information of eye changes, mouth changes and head position changes are fused to obtain multimodal data;

[0019] Acquire steering wheel position image information, brake pedal angle information and fuel pedal angle information according to each vehicle operation image information;

[0020] Acquiring steering wheel position change information according to a plurality of steering wheel position image information;

[0021] Acquire brake pedal angle change information according to multiple brake pedal angle information;

[0022] Obtaining the refueling plate angle change information according to the multiple refueling plate angle information;

[0023] The steering wheel position change information, brake pedal angle change information and gas pedal angle change information are integrated to obtain the vehicle driving operation data.

[0024] Preferably, the step of obtaining the blinking frequency, pupil diameter change value and gaze duration according to the driver's eye data, and obtaining the driver's stress assessment value according to the vehicle driving operation data, blinking frequency, pupil diameter change value and gaze duration comprises:

[0025] Acquire multiple eye intercept image information according to the driver's eye data;

[0026] Acquire multiple eye feature point information according to each eye intercepted image information;

[0027] Obtaining eye center coordinates according to multiple eye feature point information;

[0028] Obtaining the blinking frequency according to the eye center coordinates corresponding to the multiple eye intercepted image information;

[0029] Obtain pupil diameter of each eye intercept image information;

[0030] Obtaining a pupil diameter change value according to pupil diameters corresponding to a plurality of eye intercepted image information;

[0031] Obtain pupil fixation points according to intercepted image information of each eye;

[0032] Obtain the gaze direction of each eye intercept image information based on the pupil gaze point,

[0033] Obtaining the number of eye-intercepted image information with gaze direction changes, and using the number as the gaze duration;

[0034] Obtaining driver operation time based on vehicle driving operation data;

[0035] Acquire corresponding multiple eye intercepted image information based on the operation time, and acquire the blinking frequency, pupil diameter change value and gaze duration of the driver when operating according to the corresponding eye intercepted image information;

[0036] Obtain the weight values ​​of blink frequency, pupil diameter change and gaze duration;

[0037] The driver's stress assessment value is calculated based on the driver's blinking frequency, pupil diameter change and gaze duration when the driver is operating. The calculation formula is:

[0038] ;

[0039] in, is the driver stress assessment value, is the logistic sigmoid function, is the blink frequency, is the weight value of blink frequency, is the change in pupil diameter, is the weight value of the pupil diameter change value, For the gaze duration, is the weight value of gaze duration, is the bias vector.

[0040] Preferably, the step of obtaining the driver's fatigue assessment value according to the driver's eye data, the driver's mouth data and the driver's head position data comprises:

[0041] Obtaining an eye closing speed value according to the driver's eye data;

[0042] Get the weight value of the eye closing speed value;

[0043] Obtaining a yawning frequency value based on the driver's mouth data;

[0044] Get the frequency and weight of yawning

[0045] Obtaining a nodding frequency value according to the driver's head position data;

[0046] Get the weight value of the nodding frequency value;

[0047] The driver's fatigue assessment value is calculated based on the eye closure speed value, yawning frequency value, and nodding frequency value. The calculation formula is:

[0048] ;

[0049] in, is the driver’s fatigue assessment value, is the eye closure speed value, is the weight value of the eye closing speed value, is the yawning frequency value, is the weight value of the yawning frequency value, For the nod rating value, It is the weight value of the nod rating value.

[0050] Preferably, the step of obtaining the driver's facial expression information according to the driver's eye data and the driver's mouth data, and obtaining the duration of the facial expression according to the driver's facial expression information comprises:

[0051] Acquire an eye optical flow image according to the driver's eye data;

[0052] Obtain a mouth optical flow image according to the driver's mouth data;

[0053] Based on CNN, feature extraction is performed on the eye optical flow image and the mouth optical flow image to obtain the eye change feature and the mouth change feature;

[0054] The driver's expression information is obtained by fusing the eye change features and mouth change features;

[0055] Get the start time of the change of the driver's facial expression information;

[0056] Get the end change time of the driver's expression information;

[0057] The difference between the start change time and the end change time is calculated to get the duration of the expression.

[0058] Preferably, the step of obtaining the driver's vehicle operation frequency from the vehicle driving operation data according to the duration of the expression, and obtaining the emotion evaluation value according to the operation frequency and the driver's expression information includes:

[0059] The vehicle driving operation data is intercepted according to the duration of the expression to obtain the intercepted vehicle operation data;

[0060] Acquire the steering wheel operation frequency, the brake pedal operation frequency and the gas pedal operation frequency according to the intercepted vehicle operation data;

[0061] The vehicle operation frequency is obtained by cross-synchronizing and fusing the steering wheel operation frequency, the brake pedal operation frequency and the gas pedal operation frequency;

[0062] Obtain facial muscle movement features, eye opening and closing features, and mouth corner change features based on the driver's facial expression information;

[0063] The vehicle operation frequency, facial muscle movement features, eye opening and closing features, and mouth corner change features are fused to obtain the emotion expression fusion features;

[0064] Obtain an emotion feature vector based on the emotion representation fusion features;

[0065] Get the weight matrix of the emotion feature vector;

[0066] Obtain the bias vector of the emotion feature vector;

[0067] The emotion evaluation value is calculated based on the emotion feature vector, weight matrix and bias vector, where the calculation formula is:

[0068] ;

[0069] in, is the sentiment evaluation value, is the emotion feature vector, is the weight matrix, is the bias vector.

[0070] Preferably, the step of calculating the comprehensive fatigue value according to the stress assessment value, the fatigue assessment value and the emotion assessment value comprises:

[0071] Obtain stress assessment values, fatigue assessment values, and mood assessment values;

[0072] Get the weight value of the pressure assessment value;

[0073] Obtain the weight value of the fatigue assessment value;

[0074] Get the weight value of the sentiment evaluation value;

[0075] The comprehensive fatigue value is calculated according to the stress evaluation value, fatigue evaluation value and emotion evaluation value, as well as the weight value of the stress evaluation value, the weight value of the fatigue evaluation value and the weight value of the emotion evaluation value. The calculation formula is:

[0076] ;

[0077] in, is the comprehensive fatigue value, is the pressure assessment value, is the weight value of the pressure assessment value, is the fatigue assessment value, is the weight value of fatigue assessment value, is the sentiment evaluation value, is the weight of the sentiment evaluation value, is the bias vector of the comprehensive fatigue value.

[0078] The present application also provides a system for monitoring driver fatigue status, which is characterized by comprising:

[0079] A first acquisition module is used to acquire the real-time driving information of the driver, and acquire the multimodal data of the driver and the vehicle driving operation data according to the real-time driving information of the driver, wherein the multimodal data includes the driver's eye data, the driver's mouth data and the driver's head position data;

[0080] A second acquisition module is used to obtain a blinking frequency, a pupil diameter change value, and a gaze duration according to the driver's eye data, and to obtain a driver stress assessment value according to the vehicle driving operation data, the blinking frequency, the pupil diameter change value, and the gaze duration;

[0081] A third acquisition module is used to acquire a fatigue assessment value of the driver according to the driver's eye data, the driver's mouth data and the driver's head position data;

[0082] A fourth acquisition module is used to acquire the driver's expression information according to the driver's eye data and the driver's mouth data, and acquire the duration of the expression according to the driver's expression information;

[0083] A fifth acquisition module, used to acquire the driver's vehicle operation frequency from the vehicle driving operation data according to the duration of the expression, and to acquire the emotion evaluation value according to the operation frequency and the driver's expression information;

[0084] A calculation module, used for calculating a comprehensive fatigue value according to the stress assessment value, the fatigue assessment value and the emotion assessment value;

[0085] Determine whether the comprehensive fatigue value is greater than the preset value;

[0086] If it is less than that, it is determined that the driver is not in a fatigue state;

[0087] If it is greater, the driver is judged to be in a fatigued state and an audible and visual warning is given to the driver.

[0088] Preferably, the second acquisition module includes:

[0089] A first acquisition unit, used for acquiring a plurality of eye intercepted image information according to the driver's eye data;

[0090] A second acquisition unit is used to acquire a plurality of eye feature point information according to each eye intercepted image information;

[0091] A third acquisition unit, used for acquiring eye center coordinates according to a plurality of eye feature point information;

[0092] A fourth acquisition unit, configured to acquire a blinking frequency according to eye center coordinates corresponding to a plurality of eye intercepted image information;

[0093] A fifth acquisition unit, used for acquiring the pupil diameter of each eye intercepted image information;

[0094] A sixth acquisition unit, configured to obtain a pupil diameter change value according to pupil diameters corresponding to a plurality of eye intercepted image information;

[0095] A seventh acquisition unit, used for acquiring pupil fixation points according to each eye intercepted image information;

[0096] An eighth acquisition unit is used to acquire the gaze direction of each eye intercepted image information according to the pupil gaze point,

[0097] a ninth obtaining unit, configured to obtain the number of intercepted eye image information with gaze direction changes, and use the number as the gaze duration;

[0098] a tenth obtaining unit, configured to obtain the driver's operation time according to the vehicle driving operation data;

[0099] an eleventh acquisition unit, configured to acquire corresponding multiple eye intercepted image information based on the operation time, and acquire the blinking frequency, pupil diameter change value and gaze duration of the driver when the driver is operating according to the corresponding eye intercepted image information;

[0100] A twelfth acquisition unit is used to acquire weight values ​​of blink frequency, pupil diameter change value and gaze duration;

[0101] The thirteenth calculation unit is used to calculate the driver's stress assessment value according to the blinking frequency, pupil diameter change value and gaze duration of the driver during operation, and the calculation formula is:

[0102] ;

[0103] in, is the driver stress assessment value, is the logistic sigmoid function, is the blink frequency, is the weight value of blink frequency, is the change in pupil diameter, is the weight value of the pupil diameter change value, For the gaze duration, is the weight value of gaze duration, is the bias vector.

[0104] Preferably, the third acquisition module includes:

[0105] a fourteenth obtaining unit, configured to obtain an eye closing speed value according to the driver's eye data;

[0106] A fifteenth obtaining unit, used for obtaining a weight value of the eye closing speed value;

[0107] A sixteenth obtaining unit, configured to obtain a yawning frequency value according to the driver's mouth data;

[0108] A seventeenth obtaining unit is used to obtain a frequency value weight value of yawning;

[0109] An eighteenth obtaining unit, configured to obtain a nodding frequency value according to the driver's head position data;

[0110] A nineteenth obtaining unit, used for obtaining a weight value of the nodding frequency value;

[0111] The second calculation unit is used to calculate the driver's fatigue assessment value according to the eye closure speed value, the yawning frequency value, and the nodding frequency value, and the calculation formula is:

[0112] ;

[0113] in, is the driver’s fatigue assessment value, is the eye closure speed value, is the weight value of the eye closing speed value, is the yawning frequency value, is the weight value of the yawning frequency value, For the nod rating value, It is the weight value of the nod rating value.

[0114] The beneficial effects of the present application are as follows: by comprehensively considering the three key factors of stress, fatigue and emotion, the comprehensive fatigue state of an individual can be evaluated more comprehensively and accurately. These three factors are usually interrelated and have an important impact on the overall fatigue of an individual. The quantitative evaluation method is adopted to avoid subjective assumptions and prejudices, making the evaluation results more objective and reliable. Different individuals have different tolerances to stress, fatigue and emotion. By real-time monitoring of data such as the driver's eyes, mouth and head position, the driver's fatigue state can be discovered in time, and the driver can be reminded to rest or take other measures when necessary. At the same time, through the captured expression information, the driver's emotional state can be reflected in real time, which helps to discover the driver's negative emotions in time. By analyzing the driver's emotional changes and driving behavior data, the specific impact of emotions on driving behavior can be evaluated, and potential driving risks can be predicted. In addition, by judging the driver's fatigue state through the comprehensive fatigue value, the driver's fatigue can be comprehensively evaluated, driving safety can be improved, and a judgment basis can be provided for early warning and intervention. These benefits help to improve the driver's overall fatigue level and work efficiency, while also ensuring the safety and smoothness of road traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] Figure 1 The figure is a schematic diagram of a method flow of an embodiment of the present application.

[0116] Figure 2 A schematic diagram of the system structure of an embodiment of the present application.

[0117] Figure 3 This is a distribution diagram of eye and mouth feature points according to an embodiment of the present application.

[0118] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0119] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0120] like Figure 1-Figure 3 As shown, the present application provides a method for monitoring a driver's fatigue state, comprising:

[0121] S1. Acquire real-time driving information of the driver, and acquire multimodal data of the driver and vehicle driving operation data according to the real-time driving information of the driver, wherein the multimodal data includes driver eye data, driver mouth data and driver head position data;

[0122] S2. Obtaining blinking frequency, pupil diameter change value and gaze duration according to the driver's eye data, and obtaining a driver stress assessment value according to the vehicle driving operation data, blinking frequency, pupil diameter change value and gaze duration;

[0123] S3, obtaining a fatigue assessment value of the driver according to the driver's eye data, the driver's mouth data and the driver's head position data;

[0124] S4, obtaining the driver's facial expression information according to the driver's eye data and the driver's mouth data, and obtaining the duration of the facial expression according to the driver's facial expression information;

[0125] S5, obtaining the driver's vehicle operation frequency from the vehicle driving operation data according to the duration of the expression, and obtaining an emotion evaluation value according to the operation frequency and the driver's expression information;

[0126] S6. Calculate a comprehensive fatigue value according to the stress assessment value, the fatigue assessment value, and the emotion assessment value;

[0127] S7, judging whether the comprehensive fatigue value is greater than a preset value;

[0128] If it is less than that, it is determined that the driver is not in a fatigue state;

[0129] If it is greater, the driver is judged to be in a fatigued state and an audible and visual warning is given to the driver.

[0130] As described in the above steps S1-S7, the driver fatigue status monitoring method plays a vital role in the modern transportation field. With the increasing busyness of road traffic and the increasing complexity of driving tasks, the driver's fatigue status has an increasingly significant impact on traffic safety and driving efficiency. Therefore, the development and application of an effective driver fatigue status monitoring method has become an urgent need. First, the driver fatigue status monitoring method needs to have a high degree of accuracy and reliability. This is because any misjudgment of the driver's fatigue status may lead to serious consequences, including the occurrence of traffic accidents. Therefore, the monitoring method must be able to accurately identify various fatigue states of the driver, such as fatigue, distraction, illness, etc., and give reliable evaluation results. Secondly, the driver fatigue status monitoring method needs to be real-time and dynamic. The driver's fatigue status may change at any time, especially when driving for a long time or encountering an emergency. Therefore, the monitoring method must be able to monitor the driver's fatigue status in real time.

[0131] The present invention can timely detect that the driver may be in a state of fatigue by real-time monitoring of the driver's multimodal data (such as driver stress, behavior patterns, emotional state, etc.), and timely warn the driver through sound and light, so as to avoid the potential fatigue risk from turning into an actual safety accident. The driver's fatigue state directly affects his driving performance and safety. By obtaining the driver's stress assessment value, it can be used as a part of evaluating the driver's fatigue. The stress assessment value represents the driver's current stress level. When the driver is in a high-stress state, their reaction time, judgment and decision-making ability may be affected, thereby increasing the risk of accidents. In addition, the driver's fatigue assessment value is obtained by obtaining the driver's eye data, the driver's mouth data and the driver's head position data, so as to monitor the driver's fatigue in real time, and timely detect that the driver is in a state of fatigue, thereby reducing the risk of traffic accidents. For example, after driving for a long time, the driver often enters a fatigue state, which may lead to problems such as decreased reaction speed and distraction, thereby increasing the risk of traffic accidents. By real-time monitoring the driver's eyes (such as blinking frequency, pupil By using data such as the size of the eyes, mouth (such as yawning frequency) and head position (such as nodding frequency), it is possible to detect in time that the driver is in a state of fatigue, and to remind the driver through sound and light equipment when necessary; at the same time, the facial expression information captured by the eye and mouth data can reflect the driver's emotional state in real time. This real-time monitoring helps to detect the driver's negative emotions in a timely manner, such as fatigue, tension or anger. When the driver is in different emotional states, his driving behavior will also change accordingly. Through the correlation analysis of the duration of expression and the frequency of vehicle operation, the specific impact of emotions on driving behavior can be evaluated. For example, when the driver is tired, he may change lanes more frequently or brake too hard. Combining the emotion evaluation value and driving behavior data, potential driving risks can be predicted. For example, when the driver is emotionally excited, his driving behavior may be more aggressive, thereby increasing the risk of accidents. By timely detecting and warning of these risks, it is helpful to reduce the incidence of traffic accidents. In addition, through emotional evaluation, personalized safe driving guidance can be provided to the driver. For example, when the driver is detected to be fatigued, he can be reminded to rest or take other measures to restore his energy. This personalized guidance helps improve driving safety and reduce traffic accidents caused by driver emotional problems, while long-term emotional assessment data can be used for driver fatigue management. By analyzing the driver's emotional changes in different time periods and road conditions, the relationship between emotional changes and fatigue status can be found, so as to timely remind the driver to pay attention to his own fatigue, judge the driver's fatigue status through comprehensive fatigue values, comprehensively assess the driver's fatigue status, and warn the driver through sound and light, so that the driver can make self-adjustments based on the sound and light warnings.

[0132] In one embodiment, the step of acquiring the driver's multimodal data and the vehicle driving operation data according to the driver's real-time driving information includes:

[0133] S101, acquiring a set of eye image information, a set of mouth image information, and a set of head position image information according to facial image information of each driver;

[0134] S102, acquiring driver's eye data according to multiple sets of eye image information;

[0135] S103, acquiring driver's mouth data according to multiple groups of mouth image information;

[0136] S104, obtaining driver's head position data according to multiple sets of head position image information;

[0137] S105, fusing the optical flow image information of eye changes, the optical flow image information of mouth changes, and the optical flow image information of head position changes to obtain multimodal data;

[0138] S106, acquiring steering wheel position image information, brake pedal angle information, and gas pedal angle information according to each vehicle operation image information;

[0139] S107, acquiring steering wheel position change information according to a plurality of steering wheel position image information;

[0140] S108, obtaining brake pedal angle change information according to multiple brake pedal angle information;

[0141] S109, obtaining the refueling plate angle change information according to the multiple refueling plate angle information;

[0142] S110, integrating the steering wheel position change information, the brake pedal angle change information and the gas pedal angle change information to obtain vehicle driving operation data.

[0143] As described in the above steps S101-S110, the present invention forms multimodal data by fusing the image information of the driver's eyes, mouth and head positions. The multimodal data provides rich information from different modes (eyes, mouth and head positions). This information can more comprehensively and accurately reflect the driver's physiological and psychological state, which helps to more accurately evaluate the driver's fatigue condition. In addition, the multimodal data fuses information from multiple modes, which can provide a more comprehensive background and help reduce the deviation or uncertainty that may be caused by a single data source. In addition, through the vehicle driving operation data, the driver's driving habits and behavior patterns can be deeply analyzed. For example, the driver's steering wheel operation frequency, braking and acceleration response speed under different road conditions can be analyzed, so as to understand the driver's driving habits and safety awareness. By real-time monitoring of the changes in the steering wheel, brake pedal and refueling plate (the change angles of the steering wheel, brake pedal and refueling plate can be obtained by the LVDT displacement sensor), the driver's abnormal operation or negligence, such as sharp turns, emergency braking or sudden acceleration, can be discovered in time, which is used to collect the driver's driving operation behavior, judge the driver's stress, emotion and fatigue, and improve the accuracy of the judgment.

[0144] In one embodiment, the method of obtaining the blinking frequency, pupil diameter change value and gaze duration according to the driver's eye data, and obtaining the driver's stress assessment value according to the vehicle driving operation data, blinking frequency, pupil diameter change value and gaze duration, includes:

[0145] S201, acquiring a plurality of eye captured image information according to the driver's eye data;

[0146] S202, obtaining a plurality of eye feature point information according to each eye intercepted image information;

[0147] S203, obtaining eye center coordinates according to multiple eye feature point information;

[0148] S204, obtaining a blinking frequency according to the eye center coordinates corresponding to the multiple eye intercepted image information;

[0149] S205, obtaining the pupil diameter of each eye intercepted image information;

[0150] S206, obtaining a pupil diameter change value according to pupil diameters corresponding to the plurality of eye intercepted image information;

[0151] S207, obtaining pupil fixation points according to the intercepted image information of each eye;

[0152] S208, obtaining the gaze direction of each eye intercepted image information according to the pupil gaze point,

[0153] S209, obtaining the number of intercepted image information of the eyes with gaze direction changes, and using the number as the gaze duration;

[0154] S210, obtaining the driver's operation time according to the vehicle driving operation data;

[0155] S211, acquiring corresponding multiple eye intercepted image information based on the operation time, and acquiring the blinking frequency, pupil diameter change value and gaze duration of the driver when operating according to the corresponding eye intercepted image information;

[0156] S212, obtaining weight values ​​of blink frequency, pupil diameter change value and gaze duration;

[0157] S213, calculating the driver's stress assessment value according to the driver's blinking frequency, pupil diameter change value and gaze duration when the driver is operating, and the calculation formula is:

[0158] ;

[0159] in, is the driver stress assessment value, is the logistic sigmoid function, is the blink frequency, is the weight value of blink frequency, is the change in pupil diameter, is the weight value of the pupil diameter change value, For the gaze duration, is the weight value of gaze duration, is the bias vector.

[0160] As described in the above steps S201-S213, multiple eye intercept image information is obtained through the driver's eye data, and the blink frequency is obtained by analyzing the eye feature point information and the eye center coordinates. Figure 3 As shown, by obtaining the feature points p0-p5 of the eyes or mouth,

[0161] Indicates the coordinates of 6 eye markers or 6 mouth markers;

[0162] Calculate the coordinates of the center point of the eye or mouth based on the 6 coordinates. The calculation formula is: ; ;

[0163] By cutting a square, ( , ) is the center, and the distance between the two corner points P0 and P3 is the side length of the square;

[0164] Afterwards, by real-time monitoring of the driver's eye data and intercepting image information, the driver's eye activities, such as blinking frequency, can be quickly analyzed. Real-time monitoring and data analysis help to promptly detect possible increases in driver stress. By analyzing the driver's blinking frequency, their current stress level can be indirectly assessed. An abnormal increase or decrease in blinking frequency may be a sign of increased stress. When an abnormal change in the driver's blinking frequency is detected, an early warning can be issued to remind the driver to pay attention to his or her own status or take necessary measures. Early warnings help drivers adjust their status in a timely manner to avoid driving errors or traffic accidents caused by excessive stress. The driver's gaze duration refers to the time the driver continues to stare at a certain direction or object within a specific period of time. Long-term staring may indicate that the driver is highly concentrating, or it may be a manifestation of increased stress. Changes in pupil diameter can also be used as an important indicator for assessing the driver's stress level. Under stress, the pupil diameter may change, such as shrinking or expanding. Changes in pupil diameter can also be used as an important indicator for assessing the driver's stress level. Under stress, pupil diameter may change, such as shrinking or dilating. By combining the three indicators of gaze duration, blinking frequency and pupil diameter change, a more comprehensive assessment of the driver's stress level can be made to address the problem of inaccurate stress judgment in existing technologies.

[0165] In one embodiment, the step of obtaining the driver's fatigue assessment value based on the driver's eye data, the driver's mouth data, and the driver's head position data includes:

[0166] S301, obtaining an eye closing speed value according to the driver's eye data;

[0167] S302, obtaining a weight value of the eye closure speed value;

[0168] S303, obtaining a yawning frequency value according to the driver's mouth data;

[0169] S304: Obtaining the frequency value weight of yawning

[0170] S305, obtaining a nodding frequency value according to the driver's head position data;

[0171] S306, obtaining a weight value of the nodding frequency value;

[0172] S307. Calculate the driver's fatigue assessment value according to the eye closure speed value, the yawning frequency value, and the nodding frequency value. The calculation formula is:

[0173] ;

[0174] in, is the driver’s fatigue assessment value, is the eye closure speed value, is the weight value of the eye closing speed value, is the yawning frequency value, is the weight value of the yawning frequency value, For the nod rating value, It is the weight value of the nod rating value.

[0175] As described in the above steps S301-S307, fatigue driving is one of the important causes of traffic accidents. By continuously monitoring the driver's eye closure speed, yawning frequency and nodding frequency, and different drivers may behave differently in a fatigued state, by comprehensively considering multiple indicators such as eye closure speed, yawning frequency and nodding frequency, the driver's fatigue level can be evaluated more comprehensively and accurately, and misjudgment can be prevented during the detection of different drivers. It is more suitable for different drivers, improves the accuracy and reliability of the evaluation, and can judge the driver's fatigue level in real time. By calculating the fatigue evaluation value, the driver can understand his fatigue state more objectively, thereby avoiding continuing to drive in a fatigued state.

[0176] In one embodiment, the step of obtaining the driver's facial expression information based on the driver's eye data and the driver's mouth data, and obtaining the duration of the facial expression based on the driver's facial expression information, includes:

[0177] S401, acquiring an eye optical flow image according to the driver's eye data;

[0178] S402, obtaining a mouth optical flow image according to the driver's mouth data;

[0179] S403, extracting features of the eye optical flow image and the mouth optical flow image based on CNN to obtain four eye change signs and mouth change features;

[0180] S405, performing feature fusion on the eye change features and the mouth change features to obtain the driver's expression information;

[0181] S406, obtaining the start time of change of the driver's facial expression information;

[0182] S407, obtaining the end change time of the driver's expression information;

[0183] S408: Calculate the difference between the start change time and the end change time to obtain the duration of the expression.

[0184] As described in the above steps S401-S408, by using the eye and mouth data to obtain the eye optical flow image and the mouth optical flow image respectively, this method can more accurately capture the subtle changes in the driver's expression. The optical flow method is a method for estimating the movement speed of pixels in an image sequence, which can capture the dynamic change process of the expression. Based on CNN (convolutional neural network), feature extraction of the eye and mouth optical flow images can obtain more accurate eye change features and mouth change features. Feature fusion of the eye and mouth change features can obtain more comprehensive driver expression information. Obtaining the start change time and the end change time of the driver's expression information is an important parameter for evaluating the driver's emotional state. For example, a long smile may indicate that the driver is in a happy state, while a slow blink may indicate that the driver is tired. In the driver fatigue state monitoring, by real-time monitoring of the driver's expression and the duration of the expression, the driver's fatigue, distraction and other bad driving conditions can be discovered in time, and corresponding measures can be taken to improve driving safety.

[0185] In one embodiment, the step of obtaining the driver's vehicle operation frequency from the vehicle driving operation data according to the duration of the expression, and obtaining the emotion evaluation value according to the operation frequency and the driver's expression information includes:

[0186] S501, intercepting vehicle driving operation data according to the duration of the expression to obtain intercepted vehicle operation data;

[0187] S502, acquiring a steering wheel operation frequency, a brake pedal operation frequency, and a gas pedal operation frequency according to intercepted vehicle operation data;

[0188] S503, obtaining a vehicle operation frequency by cross-synchronizing and fusing the steering wheel operation frequency, the brake pedal operation frequency, and the gas pedal operation frequency;

[0189] S504, obtaining facial muscle movement features, eye opening and closing features, and mouth corner change features according to the driver's facial expression information;

[0190] S505, performing feature fusion on the vehicle operation frequency, facial muscle movement features, eye opening and closing features, and mouth corner change features to obtain emotion representation fusion features;

[0191] S506, obtaining an emotion feature vector according to the emotion representation fusion feature;

[0192] S507, obtaining a weight matrix of the emotion feature vector;

[0193] S508, obtaining a bias vector of the emotion feature vector;

[0194] S509, calculating the emotion evaluation value according to the emotion feature vector, the weight matrix and the bias vector, wherein the calculation formula is:

[0195] ;

[0196] in, is the sentiment evaluation value, is the emotion feature vector, is the weight matrix, is the bias vector.

[0197] As described in the above steps S501-S509, the defects caused by ignoring or insufficiently considering the driver's emotional state in driver status monitoring and vehicle safety control are solved. The driver's emotional state directly affects his driving behavior and reaction ability. Negative emotions such as anger, frustration or overexcitement may cause the driver to take risky or improper driving behaviors, thereby increasing the risk of accidents. Through the emotional evaluation value, the driver's emotional state can be monitored in real time, and warnings or interventions can be provided when necessary to reduce potential dangers. By considering the duration of expression occurrence, it can be ensured that the analyzed data is closely related to the driver's specific emotional state, thereby improving the accuracy of emotion recognition. The duration of expression can provide clues to the intensity of emotion, which helps to more accurately judge the driver's emotional state, and intercept vehicle operation data, including the operating frequency of the steering wheel, brake pedal and refueling plate, which can fully reflect the driver's driving behavior in a specific time period. These operating frequencies are the direct embodiment of the driver's emotional state in driving behavior. By cross-synchronizing the operating frequencies of the steering wheel, brake pedal and refueling plate, information can be integrated more efficiently, redundant data can be reduced, and analysis efficiency can be improved.

[0198] In one embodiment, the step of calculating the comprehensive fatigue value according to the stress assessment value, the fatigue assessment value and the emotion assessment value comprises:

[0199] S601, obtaining stress assessment value, fatigue assessment value and emotion assessment value;

[0200] S602, obtaining a weight value of a pressure assessment value;

[0201] S603, obtaining a weight value of the fatigue assessment value;

[0202] S604, obtaining a weight value of the emotion evaluation value;

[0203] S605: Calculate the comprehensive fatigue value according to the stress evaluation value, the fatigue evaluation value and the emotion evaluation value, and the weight value of the stress evaluation value, the weight value of the fatigue evaluation value and the weight value of the emotion evaluation value. The calculation formula is:

[0204] ;

[0205] in, is the comprehensive fatigue value, is the pressure assessment value, is the weight value of the pressure assessment value, is the fatigue assessment value, is the weight value of fatigue assessment value, is the sentiment evaluation value, is the weight of the sentiment evaluation value, is the bias vector of the comprehensive fatigue value.

[0206] As described in the above steps S601-S605, by comprehensively considering the three key factors of stress, fatigue and emotion, the comprehensive fatigue state of an individual can be evaluated more comprehensively and accurately. These three factors are usually interrelated and have an important impact on the overall fatigue of the individual. The use of quantitative evaluation methods avoids subjective assumptions and biases, making the evaluation results more objective and reliable. Different individuals have different tolerance to stress, fatigue and emotions. By introducing weight values, personalized evaluations can be performed based on the actual situation of the individual, making the evaluation results more in line with the individual's actual situation. This solves the problem that existing monitoring methods often only focus on indicators in one aspect, such as physical indicators, psychological indicators, etc., and ignore the mutual influence and correlation between different indicators. By comprehensively considering multiple indicators such as stress, fatigue and emotion, the fatigue state of an individual can be evaluated more comprehensively.

[0207] like Figure 2 As shown, the present application also provides a monitoring system for driver fatigue status, comprising:

[0208] The first acquisition module 1 is used to acquire the real-time driving information of the driver, and acquire the multimodal data of the driver and the vehicle driving operation data according to the real-time driving information of the driver, wherein the multimodal data includes the driver's eye data, the driver's mouth data and the driver's head position data;

[0209] The second acquisition module 2 is used to obtain the blinking frequency, pupil diameter change value and gaze duration according to the driver's eye data, and to obtain the driver's stress assessment value according to the vehicle driving operation data, blinking frequency, pupil diameter change value and gaze duration;

[0210] A third acquisition module 3, used to acquire a driver's fatigue assessment value according to the driver's eye data, the driver's mouth data and the driver's head position data;

[0211] The fourth acquisition module 4 is used to acquire the driver's expression information according to the driver's eye data and the driver's mouth data, and acquire the duration of the expression according to the driver's expression information;

[0212] A fifth acquisition module 5, used to acquire the driver's vehicle operation frequency from the vehicle driving operation data according to the duration of the expression, and to acquire the emotion evaluation value according to the operation frequency and the driver's expression information;

[0213] A calculation module 6, used for calculating a comprehensive fatigue value according to the stress evaluation value, the fatigue evaluation value and the emotion evaluation value;

[0214] A judgment module 7 is used to judge whether the comprehensive fatigue value is greater than a preset value;

[0215] If it is less than that, it is determined that the driver is not in a fatigue state;

[0216] If it is greater, the driver is judged to be in a fatigued state and an audible and visual warning is given to the driver.

[0217] In one embodiment, the second acquisition module 2 includes:

[0218] A first acquisition unit, used for acquiring a plurality of eye intercepted image information according to the driver's eye data;

[0219] A second acquisition unit is used to acquire a plurality of eye feature point information according to each eye intercepted image information;

[0220] A third acquisition unit, used for acquiring eye center coordinates according to a plurality of eye feature point information;

[0221] A fourth acquisition unit, configured to acquire a blinking frequency according to eye center coordinates corresponding to a plurality of eye intercepted image information;

[0222] A fifth acquisition unit, used for acquiring the pupil diameter of each eye intercepted image information;

[0223] A sixth acquisition unit, configured to obtain a pupil diameter change value according to pupil diameters corresponding to a plurality of eye intercepted image information;

[0224] A seventh acquisition unit, used for acquiring pupil fixation points according to each eye intercepted image information;

[0225] An eighth acquisition unit is used to acquire the gaze direction of each eye intercepted image information according to the pupil gaze point,

[0226] a ninth obtaining unit, configured to obtain the number of intercepted eye image information with gaze direction changes, and use the number as the gaze duration;

[0227] a tenth obtaining unit, configured to obtain the driver's operation time according to the vehicle driving operation data;

[0228] an eleventh acquisition unit, configured to acquire corresponding multiple eye intercepted image information based on the operation time, and acquire the blinking frequency, pupil diameter change value and gaze duration of the driver when the driver is operating according to the corresponding eye intercepted image information;

[0229] A twelfth acquisition unit is used to acquire weight values ​​of blink frequency, pupil diameter change value and gaze duration;

[0230] The thirteenth calculation unit is used to calculate the driver's stress assessment value according to the blinking frequency, pupil diameter change value and gaze duration of the driver during operation, and the calculation formula is:

[0231] ;

[0232] in, is the driver stress assessment value, is the logistic sigmoid function, is the blink frequency, is the weight value of blink frequency, is the change in pupil diameter, is the weight value of the pupil diameter change value, For the gaze duration, is the weight value of gaze duration, is the bias vector.

[0233] In one embodiment, the third acquisition module 3 includes:

[0234] a fourteenth obtaining unit, configured to obtain an eye closing speed value according to the driver's eye data;

[0235] A fifteenth obtaining unit, used for obtaining a weight value of the eye closing speed value;

[0236] A sixteenth obtaining unit, configured to obtain a yawning frequency value according to the driver's mouth data;

[0237] A seventeenth obtaining unit is used to obtain a frequency value weight value of yawning;

[0238] An eighteenth obtaining unit, configured to obtain a nodding frequency value according to the driver's head position data;

[0239] A nineteenth obtaining unit, used for obtaining a weight value of the nodding frequency value;

[0240] The second calculation unit is used to calculate the driver's fatigue assessment value according to the eye closure speed value, the yawning frequency value, and the nodding frequency value, and the calculation formula is:

[0241] ;

[0242] in, is the driver’s fatigue assessment value, is the eye closure speed value, is the weight value of the eye closing speed value, is the yawning frequency value, is the weight value of the yawning frequency value, For the nod rating value, It is the weight value of the nod rating value.

[0243] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0244] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0245] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for monitoring driver fatigue status, characterized in that: include: Acquire real-time driving information of the driver, and acquire multimodal data of the driver and vehicle driving operation data according to the real-time driving information of the driver, wherein the multimodal data includes driver eye data, driver mouth data and driver head position data; Obtaining blinking frequency, pupil diameter change value and gaze duration according to the driver's eye data, and obtaining a driver stress assessment value according to the vehicle driving operation data, blinking frequency, pupil diameter change value and gaze duration; Obtaining a driver's fatigue assessment value based on the driver's eye data, the driver's mouth data, and the driver's head position data; Acquire the driver's facial expression information according to the driver's eye data and the driver's mouth data, and acquire the duration of the facial expression according to the driver's facial expression information; Obtaining the driver's vehicle operation frequency from the vehicle driving operation data according to the duration of the expression, and obtaining an emotion evaluation value according to the operation frequency and the driver's expression information; A comprehensive fatigue value is calculated based on the stress assessment value, fatigue assessment value and emotion assessment value; Determine whether the comprehensive fatigue value is greater than the preset value; If it is less than that, it is determined that the driver is not in a fatigue state; If it is greater, the driver is judged to be in a fatigue state and an audible and visual warning signal is issued; The step of obtaining the driver's facial expression information according to the driver's eye data and the driver's mouth data, and obtaining the duration of the facial expression according to the driver's facial expression information, comprises: Acquire an eye optical flow image according to the driver's eye data; Obtain a mouth optical flow image according to the driver's mouth data; Based on CNN, feature extraction is performed on the eye optical flow image and the mouth optical flow image to obtain the eye change feature and the mouth change feature; The driver's expression information is obtained by fusing the eye change features and mouth change features; Get the start time of the change of the driver's facial expression information; Get the end change time of the driver's expression information; Calculate the difference between the start change time and the end change time to get the duration of the expression; The step of obtaining the driver's vehicle operation frequency from the vehicle driving operation data according to the duration of the expression, and obtaining the emotion evaluation value according to the operation frequency and the driver's expression information includes: The vehicle driving operation data is intercepted according to the duration of the expression to obtain the intercepted vehicle operation data; Acquire the steering wheel operation frequency, the brake pedal operation frequency and the gas pedal operation frequency according to the intercepted vehicle operation data; The vehicle operation frequency is obtained by cross-synchronizing and fusing the steering wheel operation frequency, the brake pedal operation frequency and the gas pedal operation frequency; Obtain facial muscle movement features, eye opening and closing features, and mouth corner change features based on the driver's facial expression information; The vehicle operation frequency, facial muscle movement features, eye opening and closing features, and mouth corner change features are fused to obtain the emotion expression fusion features; Obtain an emotion feature vector based on the emotion representation fusion features; Get the weight matrix of the emotion feature vector; Obtain the bias vector of the emotion feature vector; The emotion evaluation value is calculated based on the emotion feature vector, weight matrix and bias vector, where the calculation formula is: ； in, is the sentiment evaluation value, is the emotion feature vector, is the weight matrix, is the bias vector.

2. The method for monitoring driver fatigue status according to claim 1, characterized in that: The step of acquiring the driver's multimodal data and the vehicle driving operation data according to the driver's real-time driving information includes: Acquire multiple driver face image information and multiple vehicle operation image information according to the real-time driving information of the driver at a preset time; Acquire a set of eye image information, a set of mouth image information, and a set of head position image information according to the facial image information of each driver; Acquiring driver eye data according to multiple sets of eye image information; Acquire driver mouth data according to multiple sets of mouth image information; Acquiring driver's head position data according to multiple sets of head position image information; The optical flow image information of eye changes, mouth changes and head position changes are fused to obtain multimodal data; Acquire steering wheel position image information, brake pedal angle information and fuel pedal angle information according to each vehicle operation image information; Acquiring steering wheel position change information according to a plurality of steering wheel position image information; Acquire brake pedal angle change information according to multiple brake pedal angle information; Obtaining the refueling plate angle change information according to the multiple refueling plate angle information; The steering wheel position change information, brake pedal angle change information and gas pedal angle change information are integrated to obtain the vehicle driving operation data.

3. The method for monitoring driver fatigue status according to claim 1, characterized in that: The steps of obtaining the blinking frequency, pupil diameter change value and gaze duration according to the driver's eye data, and obtaining the driver's stress assessment value according to the vehicle driving operation data, blinking frequency, pupil diameter change value and gaze duration, include: Acquire multiple eye intercept image information according to the driver's eye data; Acquire multiple eye feature point information according to each eye intercepted image information; Obtaining eye center coordinates according to multiple eye feature point information; Obtaining the blinking frequency according to the eye center coordinates corresponding to the multiple eye intercepted image information; Obtain pupil diameter of each eye intercept image information; Obtaining a pupil diameter change value according to pupil diameters corresponding to a plurality of eye intercepted image information; Obtain pupil fixation points according to intercepted image information of each eye; Obtain the gaze direction of each eye intercept image information based on the pupil gaze point, Obtaining the number of eye-intercepted image information with gaze direction changes, and using the number as the gaze duration; Obtaining driver operation time based on vehicle driving operation data; Acquire corresponding multiple eye intercepted image information based on the operation time, and acquire the blinking frequency, pupil diameter change value and gaze duration of the driver when operating according to the corresponding eye intercepted image information; Obtain the weight values ​​of blink frequency, pupil diameter change and gaze duration; The driver's stress assessment value is calculated based on the driver's blinking frequency, pupil diameter change and gaze duration when the driver is operating. The calculation formula is: ; in, is the driver stress assessment value, is the logistic sigmoid function, is the blink frequency, is the weight value of blink frequency, is the change in pupil diameter, is the weight value of the pupil diameter change value, For the gaze duration, is the weight value of gaze duration, is the bias vector.

4. The method for monitoring driver fatigue status according to claim 1, characterized in that: The step of obtaining the driver's fatigue assessment value according to the driver's eye data, the driver's mouth data and the driver's head position data comprises: Obtaining an eye closing speed value according to the driver's eye data; Get the weight value of the eye closing speed value; Obtaining a yawning frequency value based on the driver's mouth data; Get the frequency value and weight value of yawning; Obtaining a nodding frequency value according to the driver's head position data; Get the weight value of the nodding frequency value; The driver's fatigue assessment value is calculated based on the eye closure speed value, yawning frequency value, and nodding frequency value. The calculation formula is: ； in, is the driver’s fatigue assessment value, is the eye closure speed value, is the weight value of the eye closing speed value, is the yawning frequency value, is the weight value of the yawning frequency value, For the nod rating value, It is the weight value of the nod rating value.

5. The method for monitoring driver fatigue status according to claim 1, characterized in that: The step of calculating the comprehensive fatigue value according to the stress assessment value, the fatigue assessment value and the emotion assessment value comprises: Obtain stress assessment values, fatigue assessment values, and mood assessment values; Get the weight value of the pressure assessment value; Obtain the weight value of the fatigue assessment value; Get the weight value of the sentiment evaluation value; The comprehensive fatigue value is calculated according to the stress evaluation value, fatigue evaluation value and emotion evaluation value, as well as the weight value of the stress evaluation value, the weight value of the fatigue evaluation value and the weight value of the emotion evaluation value. The calculation formula is: ； in, is the comprehensive fatigue value, is the pressure assessment value, is the weight value of the pressure assessment value, is the fatigue assessment value, is the weight value of fatigue assessment value, is the sentiment evaluation value, is the weight of the sentiment evaluation value, is the bias vector of the comprehensive fatigue value.

6. A system for monitoring driver fatigue, characterized in that: include: A first acquisition module is used to acquire the real-time driving information of the driver, and acquire the multimodal data of the driver and the vehicle driving operation data according to the real-time driving information of the driver, wherein the multimodal data includes the driver's eye data, the driver's mouth data and the driver's head position data; A second acquisition module is used to obtain a blinking frequency, a pupil diameter change value, and a gaze duration according to the driver's eye data, and to obtain a driver stress assessment value according to the vehicle driving operation data, the blinking frequency, the pupil diameter change value, and the gaze duration; A third acquisition module is used to acquire a fatigue assessment value of the driver according to the driver's eye data, the driver's mouth data and the driver's head position data; A fourth acquisition module is used to acquire the driver's expression information according to the driver's eye data and the driver's mouth data, and acquire the duration of the expression according to the driver's expression information; A fifth acquisition module, used to acquire the driver's vehicle operation frequency from the vehicle driving operation data according to the duration of the expression, and to acquire the emotion evaluation value according to the operation frequency and the driver's expression information; A calculation module, used for calculating a comprehensive fatigue value according to the stress assessment value, the fatigue assessment value and the emotion assessment value; A judgment module, used to judge whether the comprehensive fatigue value is greater than a preset value; If it is less than that, it is determined that the driver is not in a fatigue state; If it is greater than that, the driver is judged to be in a fatigue state and an audible and visual warning is given to the driver; Wherein, the second acquisition module includes: A first acquisition unit, used for acquiring a plurality of eye intercepted image information according to the driver's eye data; A second acquisition unit is used to acquire a plurality of eye feature point information according to each eye intercepted image information; A third acquisition unit, used for acquiring eye center coordinates according to a plurality of eye feature point information; A fourth acquisition unit, configured to acquire a blinking frequency according to eye center coordinates corresponding to a plurality of eye intercepted image information; A fifth acquisition unit, used for acquiring the pupil diameter of each eye intercepted image information; A sixth acquisition unit, configured to obtain a pupil diameter change value according to pupil diameters corresponding to a plurality of eye intercepted image information; A seventh acquisition unit, used for acquiring pupil fixation points according to each eye intercepted image information; An eighth acquisition unit is used to acquire the gaze direction of each eye intercepted image information according to the pupil gaze point, a ninth obtaining unit, configured to obtain the number of intercepted eye image information with gaze direction changes, and use the number as the gaze duration; a tenth obtaining unit, configured to obtain the driver's operation time according to the vehicle driving operation data; an eleventh acquisition unit, configured to acquire corresponding multiple eye intercepted image information based on the operation time, and acquire the blinking frequency, pupil diameter change value and gaze duration of the driver when the driver is operating according to the corresponding eye intercepted image information; A twelfth acquisition unit is used to acquire weight values ​​of blink frequency, pupil diameter change value and gaze duration; The thirteenth calculation unit is used to calculate the driver's stress assessment value according to the blinking frequency, pupil diameter change value and gaze duration of the driver during operation, and the calculation formula is: ； in, is the driver stress assessment value, is the logistic sigmoid function, is the blink frequency, is the weight value of blink frequency, is the change in pupil diameter, is the weight value of the pupil diameter change value, For the gaze duration, is the weight value of gaze duration, is the bias vector; The third acquisition module includes: a fourteenth obtaining unit, configured to obtain an eye closing speed value according to the driver's eye data; A fifteenth obtaining unit, used for obtaining a weight value of the eye closing speed value; A sixteenth obtaining unit, configured to obtain a yawning frequency value according to the driver's mouth data; A seventeenth obtaining unit is used to obtain a frequency value weight value of yawning; An eighteenth obtaining unit, configured to obtain a nodding frequency value according to the driver's head position data; A nineteenth obtaining unit, used for obtaining a weight value of the nodding frequency value; The second calculation unit is used to calculate the driver's fatigue assessment value according to the eye closure speed value, the yawning frequency value, and the nodding frequency value, and the calculation formula is: ； in, is the driver’s fatigue assessment value, is the eye closure speed value, is the weight value of the eye closing speed value, is the yawning frequency value, is the weight value of the yawning frequency value, For the nod rating value, It is the weight value of the nod rating value.

Images