Method and device for monitoring fatigue state of driver

Abstract

The invention provides a method and a device for monitoring the fatigue state of a driver. The monitoring method comprises the following steps: acquiring various original monitoring data related to fatigue state monitoring; feature extraction processing is carried out on each kind of original monitoring data, and monitoring features of each kind of original monitoring data are determined; and analyzing and processing all the monitoring characteristics, and determining a fatigue monitoring result of the target driver. Thus, according to the technical scheme, the fatigue state of the driver can be automatically monitored in real time, the fatigue monitoring result is determined through multiple monitoring characteristics, and the accuracy of the fatigue monitoring result can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of automobile technology, and in particular to a method and device for monitoring a driver's fatigue state. Background Art

[0002] In modern society, with the increase in the number of vehicles and the increasing traffic pressure, driver fatigue has become one of the important causes of traffic accidents. Studies have shown that fatigue driving can significantly reduce the driver's attention, reaction speed and judgment ability, which in turn leads to traffic accidents. Especially when driving for a long time or at night, fatigue is more likely to occur, and the driver's danger also increases.

[0003] Therefore, it is of great significance to monitor the driver's fatigue state. When the system detects that the driver may be in a fatigued state, it can issue an alarm in time to remind the driver to rest or take other measures, thereby effectively preventing potential traffic accidents. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a method and device for monitoring the driver's fatigue status, which can automatically monitor the driver's fatigue status in real time, and determine the fatigue monitoring results through multiple monitoring characteristics, thereby effectively improving the accuracy of the fatigue monitoring results.

[0005] The present application provides a method for monitoring a driver's fatigue state, the monitoring method comprising:

[0006] Acquire various raw monitoring data related to fatigue status monitoring;

[0007] Perform feature extraction processing on each type of original monitoring data respectively to determine the monitoring features of each type of original monitoring data;

[0008] All monitoring features are analyzed and processed to determine the fatigue monitoring results of the target driver.

[0009] Optionally, the acquiring of a variety of original monitoring data related to fatigue status monitoring includes:

[0010] Acquiring physiological and behavioral monitoring data of the target driver using a sensing device;

[0011] Using the vehicle dynamic monitoring system to obtain the driving monitoring data of the vehicle where the target driver is located;

[0012] Using a pressure monitoring device to obtain force change monitoring data of a target driver;

[0013] The multiple types of original monitoring data are determined based on the physiological and behavioral monitoring data, the driving monitoring data, and the force change monitoring data.

[0014] Optionally, performing feature extraction processing on each type of original monitoring data to determine the monitoring features of each type of original monitoring data includes:

[0015] Performing feature extraction processing on the physiological and behavioral monitoring data to determine multimodal behavioral features;

[0016] Performing feature extraction processing on the driving monitoring data to determine the dynamic characteristics of the vehicle;

[0017] Performing feature extraction processing on the force change monitoring data to determine pressure features;

[0018] The monitoring characteristics are determined according to the multimodal behavior characteristics, the vehicle dynamic characteristics and the pressure characteristics.

[0019] Optionally, analyzing and processing all monitoring features to determine the fatigue monitoring result of the target driver includes:

[0020] Determining a first monitoring result according to the multimodal behavior characteristics;

[0021] Determining a second monitoring result according to the vehicle dynamic characteristics;

[0022] Determining a third monitoring result according to the pressure characteristic;

[0023] The first monitoring result, the second monitoring result and the third monitoring result are weighted to determine the fatigue monitoring result of the target driver.

[0024] Optionally, analyzing and processing all monitoring features to determine the fatigue monitoring result of the target driver includes:

[0025] Determining a first monitoring result according to the multimodal behavior characteristics;

[0026] Determining a second monitoring result according to the vehicle dynamic characteristics;

[0027] Determining a third monitoring result according to the pressure characteristic;

[0028] Determining a fourth monitoring result according to the multimodal behavior characteristics and the vehicle dynamic characteristics;

[0029] determining a fifth monitoring result according to the multimodal behavior characteristics and the pressure characteristics;

[0030] Determining a sixth monitoring result according to the vehicle dynamic characteristics and the pressure characteristics;

[0031] The first monitoring result, the second monitoring result, the third monitoring result, the fourth monitoring result, the fifth monitoring result and the sixth monitoring result are weightedly processed to determine the fatigue monitoring result of the target driver.

[0032] Optionally, analyzing and processing all monitoring features to determine the fatigue monitoring result of the target driver includes:

[0033] Determining fusion features according to the multimodal behavior features, vehicle dynamic features, and pressure features;

[0034] The fusion feature is analyzed to determine the fatigue monitoring result of the target driver. Optionally, after determining the fatigue monitoring result of the target driver, the monitoring method further includes:

[0035] Determining whether the fatigue monitoring result meets the warning condition;

[0036] When the conditions are met, a corresponding early warning strategy is determined according to the fatigue monitoring result, and early warning processing is performed on the target driver according to the early warning strategy.

[0037] The embodiment of the present application also provides a device for monitoring a driver's fatigue state, the monitoring device comprising:

[0038] An acquisition module, used to acquire various raw monitoring data related to fatigue status monitoring;

[0039] The extraction module is used to perform feature extraction processing on each type of original monitoring data to determine the monitoring features of each type of original monitoring data;

[0040] The determination module is used to analyze and process all monitoring features to determine the fatigue monitoring results of the target driver.

[0041] Optionally, when the acquisition module is used to acquire a variety of original monitoring data related to fatigue state monitoring, the acquisition module is used to:

[0042] Acquiring physiological and behavioral monitoring data of the target driver using a sensing device;

[0043] Using the vehicle dynamic monitoring system to obtain the driving monitoring data of the vehicle where the target driver is located;

[0044] Using a pressure monitoring device to obtain force change monitoring data of a target driver;

[0045] The multiple types of original monitoring data are determined based on the physiological and behavioral monitoring data, the driving monitoring data, and the force change monitoring data.

[0046] Optionally, when the extraction module is used to perform feature extraction processing on each type of original monitoring data respectively to determine the monitoring features of each type of original monitoring data, the extraction module is used to:

[0047] Performing feature extraction processing on the physiological and behavioral monitoring data to determine multimodal behavioral features;

[0048] Performing feature extraction processing on the driving monitoring data to determine the dynamic characteristics of the vehicle;

[0049] Performing feature extraction processing on the force change monitoring data to determine pressure features;

[0050] The monitoring characteristics are determined according to the multimodal behavior characteristics, the vehicle dynamic characteristics and the pressure characteristics.

[0051] Optionally, when the determination module is used to analyze and process all monitoring features to determine the fatigue monitoring result of the target driver, the determination module is used to:

[0052] Determining a first monitoring result according to the multimodal behavior characteristics;

[0053] Determining a second monitoring result according to the vehicle dynamic characteristics;

[0054] Determining a third monitoring result according to the pressure characteristic;

[0055] The first monitoring result, the second monitoring result and the third monitoring result are weighted to determine the fatigue monitoring result of the target driver.

[0056] Optionally, when the determination module is used to analyze and process all monitoring features to determine the fatigue monitoring result of the target driver, the determination module is used to:

[0057] Determining a first monitoring result according to the multimodal behavior characteristics;

[0058] Determining a second monitoring result according to the vehicle dynamic characteristics;

[0059] Determining a third monitoring result according to the pressure characteristic;

[0060] Determining a fourth monitoring result according to the multimodal behavior characteristics and the vehicle dynamic characteristics;

[0061] determining a fifth monitoring result according to the multimodal behavior characteristics and the pressure characteristics;

[0062] Determining a sixth monitoring result according to the vehicle dynamic characteristics and the pressure characteristics;

[0063] The first monitoring result, the second monitoring result, the third monitoring result, the fourth monitoring result, the fifth monitoring result and the sixth monitoring result are weightedly processed to determine the fatigue monitoring result of the target driver.

[0064] Optionally, when the determination module is used to analyze and process all monitoring features to determine the fatigue monitoring result of the target driver, the determination module is used to:

[0065] Determining fusion features according to the multimodal behavior features, vehicle dynamic features, and pressure features;

[0066] The fusion features are analyzed to determine the fatigue monitoring result of the target driver.

[0067] Optionally, the monitoring device further includes an early warning module, and the early warning module is used to:

[0068] After determining the fatigue monitoring result of the target driver, determining whether the fatigue monitoring result meets the warning condition;

[0069] When the conditions are met, a corresponding early warning strategy is determined according to the fatigue monitoring result, and early warning processing is performed on the target driver according to the early warning strategy.

[0070] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the monitoring method described above are performed.

[0071] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the monitoring method described above are executed.

[0072] An embodiment of the present application provides a method and device for monitoring a driver's fatigue status, and the monitoring method includes: acquiring a variety of original monitoring data related to fatigue status monitoring; performing feature extraction processing on each type of original monitoring data respectively to determine the monitoring characteristics of each type of original monitoring data; analyzing and processing all monitoring characteristics to determine the fatigue monitoring results of the target driver.

[0073] In this way, through the technical solution of the present application, the driver's fatigue state can be automatically monitored in real time, and the fatigue monitoring results can be determined through a variety of monitoring characteristics, which can effectively improve the accuracy of the fatigue monitoring results.

[0074] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0076] Figure 1 A flowchart of a method for monitoring driver fatigue status provided in an embodiment of the present application;

[0077] Figure 2 One of the structural schematic diagrams of a device for monitoring driver fatigue status provided in an embodiment of the present application;

[0078] Figure 3 A second structural schematic diagram of a driver fatigue status monitoring device provided in an embodiment of the present application;

[0079] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0081] In modern society, with the increase in the number of vehicles and the increasing traffic pressure, driver fatigue has become one of the important causes of traffic accidents. Studies have shown that fatigue driving can significantly reduce the driver's attention, reaction speed and judgment ability, which in turn leads to traffic accidents. Especially when driving for a long time or at night, fatigue is more likely to occur, and the driver's danger also increases.

[0082] Therefore, it is of great significance to monitor the driver's fatigue state. When the system detects that the driver may be in a fatigued state, it can issue an alarm in time to remind the driver to rest or take other measures, thereby effectively preventing potential traffic accidents.

[0083] Based on this, an embodiment of the present application provides a method and device for monitoring the driver's fatigue status, which can automatically monitor the driver's fatigue status in real time, and determine the fatigue monitoring results through multiple monitoring characteristics, which can effectively improve the accuracy of the fatigue monitoring results.

[0084] See also Figure 1 , Figure 1 The following is a flow chart of a method for monitoring driver fatigue status provided in an embodiment of the present application. Figure 1 As shown in , the monitoring method provided in the embodiment of the present application includes:

[0085] S101. Acquire various original monitoring data related to fatigue status monitoring.

[0086] S102: Perform feature extraction processing on each type of original monitoring data to determine the monitoring features of each type of original monitoring data.

[0087] S103: Analyze and process all monitoring features to determine the fatigue monitoring result of the target driver.

[0088] With respect to step S101 , different types of basic data (original monitoring data) on the driver's fatigue status can be provided by acquiring the data through different devices or the same device, thereby providing comprehensive monitoring information.

[0089] For example, the original monitoring data includes physiological and behavioral monitoring data, vehicle driving monitoring data, and force change monitoring data.

[0090] Continuing with step S101, in one embodiment provided in the present application, the obtaining of a variety of original monitoring data related to fatigue status monitoring includes:

[0091] S1011. Acquire the physiological and behavioral monitoring data of the target driver using a sensing device.

[0092] S1012: Acquire driving monitoring data of the vehicle in which the target driver is located using a vehicle dynamic monitoring system.

[0093] S1013. Obtain force change monitoring data of the target driver using a pressure monitoring device.

[0094] S1014. Determine the multiple original monitoring data according to the physiological and behavioral monitoring data, the driving monitoring data, and the force change monitoring data.

[0095] For step S1011, illustratively, the sensing device includes but is not limited to an image acquisition device and a sensor; the physiological and behavioral monitoring data includes physiological monitoring data and behavioral monitoring data.

[0096] For example, the image acquisition device may include an RGB camera, an infrared camera, a depth camera, etc., and the sensor may include a heart rate sensor, a breathing sensor, etc.

[0097] The use of sensing equipment to obtain the target driver's physiological and behavioral monitoring data can be: using an image acquisition device to capture the driver's head movements, facial movements, facial expressions, and the distance between the two eyes, etc., and using a heart rate sensor, a breathing sensor, etc. to monitor the driver's heart rate, breathing frequency and other physiological indicators.

[0098] With respect to step S1012, illustratively, the vehicle dynamic monitoring system may include: an accelerometer, a gyroscope, a vehicle speed sensor, and a GPS system.

[0099] The driving monitoring data may include data such as speed changes and motion straightness during the movement of the vehicle.

[0100] The method of using the vehicle dynamic monitoring system to obtain the driving monitoring data of the vehicle in which the target driver is located includes: using an accelerometer to obtain the acceleration change data of the vehicle in various directions; using a gyroscope to obtain the angular velocity of the vehicle; combining the data obtained by the gyroscope with the data obtained by the accelerometer to determine the vehicle's motion straightness, turning angle and other information; using a vehicle speed sensor to measure the vehicle's driving speed; and using a GPS system to obtain the vehicle's real-time location information.

[0101] For step S1013, illustratively, the pressure monitoring device may be a pressure sensor, a strain gauge, a force sensor, or the like.

[0102] The force change monitoring data may specifically include force changes in the back, waist and buttocks of the vehicle seat.

[0103] Therefore, the pressure monitoring device can be installed in the back, waist and hip areas of the seat.

[0104] With respect to step S102 , in this step, for each type of original monitoring data, a corresponding feature extraction algorithm is used to perform feature extraction processing on the original monitoring data to determine the monitoring features of the original monitoring data.

[0105] In one embodiment provided in the present application, the feature extraction process is performed on each type of original monitoring data to determine the monitoring features of each type of original monitoring data, including:

[0106] S1021. Perform feature extraction processing on the physiological and behavioral monitoring data to determine multimodal behavioral features.

[0107] S1022: Perform feature extraction processing on the driving monitoring data to determine vehicle dynamic features.

[0108] S1023: Perform feature extraction processing on the force change monitoring data to determine pressure features.

[0109] S1024. Determine monitoring characteristics according to the multimodal behavior characteristics, vehicle dynamic characteristics, and pressure characteristics.

[0110] For step S1021, for example, when determining multimodal behavioral features in this step, it may include: identifying and tracking the movement direction and amplitude features of the head and face from physiological and behavioral monitoring data through facial detection and tracking algorithms (such as convolutional neural networks, key point detection); using pre-trained deep learning models (such as convolutional neural networks) to analyze changes in facial muscles and extract facial expression features such as smiles and frowns from images or video streams. Use eye detection algorithms (such as Haar features, Dlib library) to identify and locate the position of both eyes, and calculate the distance between the two eyes (pupillary distance) to obtain eye-related features. Detect the breathing frequency by analyzing small movements of the chest or abdomen (or extract the breathing frequency through Fourier transform or peak detection algorithms).

[0111] For step S1021, for example, when determining the dynamic characteristics of the vehicle in this step, it may specifically include: Feature extraction method: By integrating the acceleration data, the speed data can be obtained, and after filtering and denoising the acceleration signal, the speed change characteristics are calculated using the numerical integration method; by integrating the angular velocity data, the attitude angle change of the vehicle is obtained, so as to determine whether the vehicle deviates from straight driving. Identify the speed at different time points and calculate the amplitude and trend of the speed change. By analyzing the position data, the curvature characteristics of the vehicle's motion trajectory are calculated, so as to evaluate the motion straightness.

[0112] For step S1023, for example, when determining the pressure characteristics in this step, it may include: using average value calculation, peak value detection or pressure distribution analysis method to process the force change monitoring data, and calculating the force distribution characteristics and change characteristics of different areas.

[0113] Among them, when performing feature extraction in steps S1021-S1023, feature extraction processing can be performed through one or more pre-trained models to determine the characteristics of each data.

[0114] With respect to step S103, in this step, all monitoring features are used to perform comprehensive analysis and processing to determine the fatigue monitoring result of the target driver.

[0115] Here, after analyzing and processing all monitoring features to determine the fatigue monitoring result of the target driver, three different implementation methods are provided as examples.

[0116] In an implementation mode 1 provided in the present application, all monitoring features are analyzed and processed to determine the fatigue monitoring result of the target driver, including: determining a first monitoring result based on the multimodal behavior feature; determining a second monitoring result based on the vehicle dynamic feature; determining a third monitoring result based on the pressure feature; and using the first monitoring result, the second monitoring result, and the third monitoring result for weighted processing to determine the fatigue monitoring result of the target driver.

[0117] In this step, the first monitoring result is determined based on the first judgment condition combined with the multimodal behavior characteristics; the second monitoring result is determined based on the second judgment condition combined with the vehicle dynamic characteristics; the third monitoring result is determined based on the third judgment condition combined with the pressure characteristics; based on the weights corresponding to the first monitoring result, the second monitoring result and the third monitoring result, the first monitoring result, the second monitoring result and the third monitoring result are weighted and summed to determine the fatigue monitoring result of the target driver.

[0118] Here, the first monitoring result, the second monitoring result and the third monitoring result may be fatigue probability, or may be a binary classification result of fatigue type, fatigue level or whether fatigued or not.

[0119] The first monitoring result, the second monitoring result and the third monitoring result may be specifically: the first monitoring result is visual-fatigue level; the second monitoring result is movement-fatigue level; the third monitoring result is force-fatigue level.

[0120] For example, when determining the first monitoring result, it may be: if it is monitored that the driver's eyes are closed for a long time, the head nods frequently (dozing off), and the breathing rate slows down significantly, it can be determined that the driver may be in a fatigued state, forming a first monitoring result, such as "high fatigue risk".

[0121] When determining the second monitoring result, it may be: if the monitored vehicle frequently deviates from the straight driving trajectory during driving, or the acceleration changes abnormally (such as frequent sudden acceleration and sudden braking), the system can determine that the driver's vehicle control ability has decreased, forming a second monitoring result, such as "distraction or fatigue".

[0122] When determining the third monitoring result, it may be: if it is detected that the driver maintains the same sitting posture for a long time or has an unnatural pressure distribution (such as suddenly leaning forward), this may indicate physical fatigue of the driver, forming a third monitoring result, such as "physical fatigue or discomfort".

[0123] Finally, the first monitoring result (multimodal behavior characteristics), the second monitoring result (vehicle dynamic characteristics) and the third monitoring result (pressure characteristics) are weighted, and the weights of each result are combined to determine the final driver fatigue monitoring result. The monitoring result can be a result in the form of fatigue level, fatigue score, etc.

[0124] The first monitoring result, the second monitoring result and the third monitoring result may be determined by their respective independent models or by the same model. The weight values ​​corresponding to different monitoring results may be the same or different, which is not limited here.

[0125] In another embodiment provided in the present application, the analysis and processing of all monitoring characteristics to determine the fatigue monitoring result of the target driver includes: determining a first monitoring result based on the multimodal behavior characteristics; determining a second monitoring result based on the vehicle dynamic characteristics; determining a third monitoring result based on the pressure characteristics; determining a fourth monitoring result based on the multimodal behavior characteristics and the vehicle dynamic characteristics; determining a fifth monitoring result based on the multimodal behavior characteristics and the pressure characteristics; determining a sixth monitoring result based on the vehicle dynamic characteristics and the pressure characteristics; and using the first monitoring result, the second monitoring result, the third monitoring result, the fourth monitoring result, the fifth monitoring result, and the sixth monitoring result for weighted processing to determine the fatigue monitoring result of the target driver.

[0126] By way of example, the embodiment may be: performing visual decision analysis on the multimodal behavior characteristics through a visual detection model to determine the visual-fatigue level (first monitoring result); performing motion decision analysis on the vehicle dynamic characteristics through a motion detection model to determine the motion-fatigue level (second monitoring result); performing force decision analysis on the pressure characteristics through a force detection model to determine the force-fatigue level (third monitoring result); performing visual and motion decisions on the multimodal behavior characteristics and the vehicle dynamic characteristics through a visual and motion detection model to determine the visual and motion-fatigue level (fourth monitoring result); performing visual and force decisions on the multimodal behavior characteristics and the pressure characteristics through a visual and force detection model to determine the visual and force-fatigue level (fifth monitoring result); performing motion and force decisions on the vehicle dynamic characteristics and the pressure characteristics through a motion and force detection model to determine the motion and force-fatigue level (sixth monitoring result); and then weighting the first monitoring result, the second monitoring result, the third monitoring result, the fourth monitoring result, the fifth monitoring result, and the sixth monitoring result based on the weights of each detection model to determine the fatigue monitoring result of the target driver.

[0127] In another embodiment provided in the present application, when the determination module is used to analyze and process all monitoring features to determine the fatigue monitoring results of the target driver, the determination module is used to: determine the fusion features based on the multimodal behavior features, vehicle dynamic features and pressure features; analyze the fusion features to determine the fatigue monitoring results of the target driver.

[0128] By way of example, the embodiment may be: inputting the multimodal behavior characteristics, vehicle dynamic characteristics, and pressure characteristics into a pre-trained fatigue state prediction model to determine fusion characteristics; and analyzing the fusion characteristics through the fatigue state prediction model to determine the fatigue monitoring results of the target driver.

[0129] Here, the fatigue state prediction model, which can be trained by reinforcement learning, can accurately predict the driver's fatigue state from the input monitoring features.

[0130] The model will perform a fusion analysis on the input features, that is, comprehensively consider the relationship and mutual influence among all features to evaluate the fatigue status, which can effectively improve the accuracy of the prediction.

[0131] The model outputs a monitoring result about the driver's fatigue state based on the results of the fusion analysis. This result can be a specific fatigue level (such as "mild fatigue", "moderate fatigue" or "severe fatigue"), or a probability value indicating the possibility of the driver being in a fatigued state.

[0132] For example, suppose a driver is driving and the system collects the following monitoring features: Multimodal behavioral features: eyes closed for a long time, head nodding slightly, and breathing rate is slightly slower. Vehicle dynamic features: vehicle speed changes smoothly, but occasionally deviates from straight driving. Pressure features: seat pressure distribution shows increased lumbar pressure, which may be due to the driver relaxing on the seat back. These features are input into the pre-trained fatigue state prediction model. After fusion analysis, the model combines the interrelationships of these features to obtain the following results: The model determines that the driver is in a "moderate fatigue" state.

[0133] For example, the fatigue monitoring result can also be a fatigue score, and the corresponding fatigue level can be determined based on the fatigue score, for example: Fatigue Level 1: Awake [0-25 points); Fatigue Level 2: Mildly tired, but no effort is needed to stay awake [25-50 points); Fatigue Level 3: Fatigue, and effort is needed to stay awake [50-75 points); Fatigue Level 4: Fatigue, and great effort is needed to stay awake [75-100 points].

[0134] In addition, in another embodiment provided in the present application, after determining the fatigue monitoring result of the target driver, the monitoring method also includes: determining whether the fatigue monitoring result meets the warning condition; when it meets the condition, determining the corresponding warning strategy according to the fatigue monitoring result, and performing warning processing on the target driver according to the warning strategy.

[0135] Here, the warning condition can be a specific fatigue level (such as "moderate fatigue" or "severe fatigue"), or a fatigue probability exceeding a set threshold. For example, if the driver's fatigue state is judged to be "severe fatigue" and this state exceeds the preset safety threshold, the system will consider that the fatigue monitoring result at this time meets the warning condition and further measures need to be taken.

[0136] When it is determined that the fatigue monitoring results meet the warning conditions, an appropriate warning strategy is selected based on the fatigue monitoring results. Among them, these strategies may vary depending on the severity of fatigue, the current driving environment, and the individual characteristics of the driver (such as past reaction records).

[0137] For example, at fatigue level 1, no reminder is given if the warning conditions are not met; at fatigue level 2, the warning conditions are met, and the corresponding warning strategy may be primary voice reminder, such as a reminder to rest as soon as possible; at fatigue level 3, the warning conditions are met, and the corresponding warning strategy may be seat vibration (different intensities, random vibration), high-pitched voice reminder, etc., for forced rest; at fatigue level 4, the warning conditions are met, and the corresponding warning strategy may be system takeover, parking nearby, and immediate forced rest.

[0138] In this way, through the technical solution of the present application, the driver's fatigue state can be automatically monitored in real time, and the fatigue monitoring results can be determined through a variety of monitoring characteristics, which can effectively improve the accuracy of the fatigue monitoring results.

[0139] Based on the same inventive concept, a monitoring device corresponding to the monitoring method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned monitoring method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0140] See also Figure 2 , Figure 3 , Figure 2 This is one of the structural schematic diagrams of a driver fatigue status monitoring device provided in an embodiment of the present application. Figure 3 This is a second structural diagram of a device for monitoring driver fatigue status provided in an embodiment of the present application. Figure 2 As shown in , the monitoring device 200 includes:

[0141] An acquisition module 210 is used to acquire various raw monitoring data related to fatigue status monitoring;

[0142] The extraction module 220 is used to perform feature extraction processing on each type of original monitoring data to determine the monitoring features of each type of original monitoring data;

[0143] The determination module 230 is used to analyze and process all monitoring features to determine the fatigue monitoring result of the target driver.

[0144] Optionally, when the acquisition module 210 is used to acquire a variety of original monitoring data related to fatigue state monitoring, the acquisition module 210 is used to:

[0145] Acquiring physiological and behavioral monitoring data of the target driver using a sensing device;

[0146] Using the vehicle dynamic monitoring system to obtain the driving monitoring data of the vehicle where the target driver is located;

[0147] Using a pressure monitoring device to obtain force change monitoring data of a target driver;

[0148] The multiple types of original monitoring data are determined based on the physiological and behavioral monitoring data, the driving monitoring data, and the force change monitoring data.

[0149] Optionally, when the extraction module 220 is used to perform feature extraction processing on each type of original monitoring data to determine the monitoring features of each type of original monitoring data, the extraction module 220 is used to:

[0150] Performing feature extraction processing on the physiological and behavioral monitoring data to determine multimodal behavioral features;

[0151] Performing feature extraction processing on the driving monitoring data to determine the dynamic characteristics of the vehicle;

[0152] Performing feature extraction processing on the force change monitoring data to determine pressure features;

[0153] The monitoring characteristics are determined according to the multimodal behavior characteristics, the vehicle dynamic characteristics and the pressure characteristics.

[0154] Optionally, when the determination module 230 is used to analyze and process all monitoring features to determine the fatigue monitoring result of the target driver, the determination module 230 is used to:

[0155] Determining a first monitoring result according to the multimodal behavior characteristics;

[0156] Determining a second monitoring result according to the vehicle dynamic characteristics;

[0157] Determining a third monitoring result according to the pressure characteristic;

[0158] The first monitoring result, the second monitoring result and the third monitoring result are weighted to determine the fatigue monitoring result of the target driver.

[0159] Optionally, when the determination module 230 is used to analyze and process all monitoring features to determine the fatigue monitoring result of the target driver, the determination module 230 is used to:

[0160] Determining a first monitoring result according to the multimodal behavior characteristics;

[0161] Determining a second monitoring result according to the vehicle dynamic characteristics;

[0162] Determining a third monitoring result according to the pressure characteristic;

[0163] Determining a fourth monitoring result according to the multimodal behavior characteristics and the vehicle dynamic characteristics;

[0164] determining a fifth monitoring result according to the multimodal behavior characteristics and the pressure characteristics;

[0165] Determining a sixth monitoring result according to the vehicle dynamic characteristics and the pressure characteristics;

[0166] The first monitoring result, the second monitoring result, the third monitoring result, the fourth monitoring result, the fifth monitoring result and the sixth monitoring result are weightedly processed to determine the fatigue monitoring result of the target driver.

[0167] Optionally, when the determination module 230 is used to analyze and process all monitoring features to determine the fatigue monitoring result of the target driver, the determination module 230 is used to:

[0168] Determining fusion features according to the multimodal behavior features, vehicle dynamic features, and pressure features;

[0169] The fusion features are analyzed to determine the fatigue monitoring result of the target driver.

[0170] Optional, such as Figure 3 As shown, the monitoring device 200 further includes an early warning module 240, and the early warning module 240 is used to:

[0171] After determining the fatigue monitoring result of the target driver, determining whether the fatigue monitoring result meets the warning condition;

[0172] When the conditions are met, a corresponding early warning strategy is determined according to the fatigue monitoring result, and early warning processing is performed on the target driver according to the early warning strategy.

[0173] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in , the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0174] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 The specific implementation of the steps in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0175] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The specific implementation of the steps in the method embodiment shown can be found in the method embodiment, and will not be repeated here.

[0176] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0177] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0178] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0180] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0181] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for monitoring driver fatigue status, characterized in that: The monitoring method comprises: Acquire various raw monitoring data related to fatigue status monitoring; Perform feature extraction processing on each type of original monitoring data respectively to determine the monitoring features of each type of original monitoring data; All monitoring features are analyzed and processed to determine the fatigue monitoring results of the target driver.

2. The monitoring method according to claim 1, characterized in that: The acquisition of various raw monitoring data related to fatigue status monitoring includes: Acquiring physiological and behavioral monitoring data of the target driver using a sensing device; Using the vehicle dynamic monitoring system to obtain the driving monitoring data of the vehicle where the target driver is located; Using a pressure monitoring device to obtain force change monitoring data of a target driver; The multiple types of original monitoring data are determined based on the physiological and behavioral monitoring data, the driving monitoring data, and the force change monitoring data.

3. The monitoring method according to claim 2, characterized in that: The feature extraction process is performed on each type of original monitoring data to determine the monitoring features of each type of original monitoring data, including: Performing feature extraction processing on the physiological and behavioral monitoring data to determine multimodal behavioral features; Performing feature extraction processing on the driving monitoring data to determine the dynamic characteristics of the vehicle; Performing feature extraction processing on the force change monitoring data to determine pressure features; The monitoring characteristics are determined according to the multimodal behavior characteristics, the vehicle dynamic characteristics and the pressure characteristics.

4. The monitoring method according to claim 3, characterized in that: The analysis and processing of all monitoring features to determine the fatigue monitoring result of the target driver includes: Determining a first monitoring result according to the multimodal behavior characteristics; Determining a second monitoring result according to the vehicle dynamic characteristics; Determining a third monitoring result according to the pressure characteristic; The first monitoring result, the second monitoring result and the third monitoring result are weighted to determine the fatigue monitoring result of the target driver.

5. The monitoring method according to claim 3, characterized in that: The analysis and processing of all monitoring features to determine the fatigue monitoring result of the target driver includes: Determining a first monitoring result according to the multimodal behavior characteristics; Determining a second monitoring result according to the vehicle dynamic characteristics; Determining a third monitoring result according to the pressure characteristic; Determining a fourth monitoring result according to the multimodal behavior characteristics and the vehicle dynamic characteristics; determining a fifth monitoring result according to the multimodal behavior characteristics and the pressure characteristics; Determining a sixth monitoring result according to the vehicle dynamic characteristics and the pressure characteristics; The first monitoring result, the second monitoring result, the third monitoring result, the fourth monitoring result, the fifth monitoring result and the sixth monitoring result are weightedly processed to determine the fatigue monitoring result of the target driver.

6. The monitoring method according to claim 3, characterized in that: The analysis and processing of all monitoring features to determine the fatigue monitoring result of the target driver includes: Determining fusion features according to the multimodal behavior features, vehicle dynamic features, and pressure features; The fusion features are analyzed to determine the fatigue monitoring result of the target driver.

7. The monitoring method according to claim 1, characterized in that: After determining the fatigue monitoring result of the target driver, the monitoring method further includes: Determining whether the fatigue monitoring result meets the warning condition; When the conditions are met, a corresponding early warning strategy is determined according to the fatigue monitoring result, and early warning processing is performed on the target driver according to the early warning strategy.

8. A device for monitoring driver fatigue status, characterized in that: The monitoring device comprises: An acquisition module, used to acquire various raw monitoring data related to fatigue status monitoring; The extraction module is used to perform feature extraction processing on each type of original monitoring data to determine the monitoring features of each type of original monitoring data; The determination module is used to analyze and process all monitoring features to determine the fatigue monitoring results of the target driver.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the monitoring method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the monitoring method according to any one of claims 1 to 7 are executed.

Images