Fatigue driving monitoring method and system based on feature analysis
Through the feature analysis method, the driver's reaction time and trend are monitored by using vehicle video data, and the problems of poor versatility and stability in the existing technology are solved, and accurate judgment and reminder of the driver's fatigue status is achieved, thereby avoiding driving due to fatigue traffic accident.
Patent Information
- Application Number
- CN202411148377.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-29
AI Technical Summary
The existing fatigue driving monitoring system relies on physiological signal monitoring and eye image monitoring, which has low versatility and poor stability, making it difficult to effectively understand the changing trend of driver responses, resulting in inaccurate identification of fatigue status and is difficult to avoid traffic caused by fatigue driving. ACCIDENT.
The fatigue driving monitoring method based on feature analysis is adopted. By obtaining the vehicle's driving exterior and interior videos, the target image frame of the emergencies is filtered, the driver's actual reaction time is obtained, the reaction time difference curve chart is generated, and the reaction trend analysis model is input. Judge the driver's reaction force change trend. If there is fatigue, a fatigue warning command will be generated and the driver will be reminded.
Fatigue monitoring of different drivers is achieved, the system is universal and anti-interference ability is improved, and the accuracy of fatigue monitoring is continuously ensured, effectively avoiding traffic accidents caused by fatigue driving.
Smart Images

Figure CN119049220A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of driving monitoring, and more specifically, to a method and system for fatigue driving monitoring based on feature analysis. Background Art
[0002] With the rapid development of modern society, transportation plays an increasingly important role in daily life and economic activities; however, with the increase in the number of vehicles, the incidence of traffic accidents is also rising, among which driver fatigue driving is an important cause of frequent traffic accidents; fatigue driving can cause drivers to be slow to react, inattentive, and make operational errors, greatly increasing the risk of traffic accidents; therefore, how to effectively monitor driver fatigue driving has become a technical problem that needs to be solved urgently.
[0003] At present, the existing fatigue driving monitoring system mainly relies on two methods: physiological signal monitoring and eye image monitoring. For example, the patent with the authorization publication number CN104881955B discloses a driver fatigue driving detection method and system. Such a method realizes fatigue driving judgment by monitoring eye images. However, through research and practical application of the above method and the prior art, it is found that the above method and the prior art have at least the following defects:
[0004] (1) Physiological signal monitoring requires wearing additional hardware equipment, which is easily affected by individual differences, resulting in low versatility and cannot be applied to scenarios where drivers are frequently replaced. Eye image monitoring has high requirements for eye image quality and is easily affected by lighting conditions, resulting in low monitoring stability and difficulty in continuously ensuring the accuracy of fatigue monitoring.
[0005] (2) It is impossible to understand the driver's reaction trend and provide strong data support for fatigue status identification. It is also impossible to analyze and judge the driver's fatigue status based on the reaction trend. Furthermore, it is difficult to effectively avoid traffic accidents caused by fatigue driving. Summary of the invention
[0006] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a fatigue driving monitoring method and system based on feature analysis.
[0007] To achieve the above object, the present invention provides the following technical solutions:
[0008] A method for monitoring fatigue driving based on feature analysis, the method comprising:
[0009] S101: within a second time interval [T, T+N], obtaining an exterior video and an interior video of a vehicle while it is traveling, and screening out first target image frames in which an emergency occurs in the exterior video, to obtain an image frame set A including M first target image frames, where T, N and M are all integers greater than zero;
[0010] S102: Acquire the actual reaction time of the driver when all emergencies occur based on the interior video, and obtain a reaction time data set including R actual reaction times, where R is an integer greater than zero;
[0011] S103: Obtaining a driver's reaction time threshold, and parsing a reaction time data set based on the reaction time threshold to obtain the driver's reaction performance in a second time interval [T, T+N];
[0012] S104: Generate a reaction time difference curve graph according to the reaction performance, input the reaction time difference curve graph into a preconfigured reaction trend analysis model to determine the driver's reaction force change trend type in the second time interval [T, T+N], and judge whether the driver is fatigued based on the reaction force change trend type. If not, set N=N+K and return to step S101; if so, generate a fatigue warning instruction, K is an integer greater than zero;
[0013] S105: Remind the driver of fatigue driving according to the fatigue warning instruction.
[0014] Furthermore, the step of screening out the first target image frame where the emergency occurs in the outdoor video includes:
[0015] a1: performing frame processing on the outdoor scene video to obtain an outdoor scene frame set including Q outdoor scene frames, where Q is an integer greater than zero;
[0016] a2: extract the vth exterior scene frame in the exterior scene frame set, where v is an integer greater than zero;
[0017] a3: Use the pre-configured event detection model to perform emergency event recognition on the v-th frame of the exterior scene to obtain an emergency event recognition result; if the emergency event recognition result shows that "an emergency event has occurred", mark the corresponding v-th exterior scene frame as the first target image frame; if the emergency event recognition result shows that "no emergency event has occurred", set v=v+1 and return to step a2;
[0018] The training logic of the preconfigured event detection model is as follows:
[0019] Acquire historical emergency training data, and divide the historical emergency training data into an emergency training set and an emergency test set; the historical emergency training data includes a plurality of exterior scene frames and their corresponding annotation labels;
[0020] The label includes "0" and "1", wherein "0" represents "no emergency event occurs", and "1" represents "an emergency event occurs";
[0021] Constructing a first classifier, taking the exterior scene frames in the emergency training set as input data of the first classifier, taking the annotated labels in the emergency training set as output data of the first classifier, training the first classifier, and obtaining an initial event detection network;
[0022] Using the emergency event test set to perform model verification on the initial event detection network, and outputting an initial event detection network that is greater than or equal to a preset first test accuracy threshold as a preconfigured event detection model;
[0023] a4: Repeat the above steps a2 to a3 until v=Q, then end the loop and obtain an image frame set A including M first target image frames.
[0024] Furthermore, the actual reaction time of the driver when all emergencies occur is obtained, including:
[0025] b1: acquiring an image frame set B including W second target image frames based on the interior video;
[0026] b2: extracting the g-th first target image frame from the image frame set A, extracting the timestamp of the g-th first target image frame, and marking the timestamp of the g-th first target image frame as the first timestamp;
[0027] b3: extracting the hth second target image frame from the image frame set B, extracting the timestamp of the hth second target image frame, and marking the timestamp of the hth second target image frame as the second timestamp;
[0028] b4: Determine whether the second timestamp satisfies the preset selection condition. If not, set h=h+1 and return to step b3. If so, calculate the difference between the second timestamp and the first timestamp, and mark the difference between the second timestamp and the first timestamp as the actual reaction time of the driver, set g=g+1 and return to step b2.
[0029] The preset selection condition is that the second timestamp is adjacent to the first timestamp, and the second timestamp is less than or equal to the first timestamp;
[0030] b5: Repeat steps b3 to b4 above until a second timestamp that meets the preset selection condition is obtained, and then end the loop;
[0031] b6: Repeat steps b2 to b4 until g=M, then end the loop and obtain a reaction time data set including R actual reaction times.
[0032] Furthermore, the acquisition logic of the image frame set B is as follows:
[0033] c1: performing frame processing on the interior scene video to obtain an interior scene frame set including U interior scene frames, where U is an integer greater than zero;
[0034] c2: extract the rth interior scene frame in the interior scene frame set, where r is an integer greater than zero;
[0035] c3: using the pre-configured reaction detection model to perform reaction action recognition on the r-th interior scene frame to obtain a reaction action recognition result; if the reaction action recognition result shows that "reaction action occurs", the r-th interior scene frame is marked as the second target image frame; if the reaction action recognition result shows that "no reaction action occurs", r=r+1 is set, and the process returns to step c2;
[0036] The training logic of the preconfigured reaction detection model is as follows:
[0037] Acquire historical reaction detection training data, and divide the historical reaction detection training data into a reaction detection training set and a reaction detection test set; the historical reaction detection training data includes a plurality of interior scene frames and their corresponding annotation labels;
[0038] The label includes "0" and "1", wherein "0" represents "no reaction action occurs", and "1" represents "reaction action occurs";
[0039] Constructing a second classifier, taking the interior scene frames in the reaction detection training set as input data of the second classifier, and taking the annotated labels in the reaction detection training set as output data of the second classifier, training the second classifier, and obtaining an initial reaction detection network;
[0040] Using the reaction detection test set to perform model verification on the initial reaction detection network, and outputting an initial reaction detection network that is greater than or equal to a preset second test accuracy threshold as a preconfigured reaction detection model;
[0041] c4: Repeat the above steps c2 to c3 until r=U, then end the loop and obtain an image frame set B including W second target image frames.
[0042] Furthermore, the step of obtaining the driver's reaction time threshold includes:
[0043] The time when the driver starts the vehicle is marked as K, and the face image of the driver in the main driving seat at time K is collected;
[0044] Retrieving the facial image of the vehicle owner, calculating the similarity between the facial image of the driver and the facial image of the vehicle owner, and if the similarity is greater than a preset similarity threshold, retrieving the average reaction time of the vehicle owner, and taking the average reaction time as the driver's reaction time threshold; if the similarity is less than or equal to the preset similarity threshold, collecting the average actual reaction time of the driver in the first time interval;
[0045] Wherein, the end time of the first time interval is earlier than the start time of the second time interval;
[0046] The average actual reaction time in the first time interval is used as the driver's reaction time threshold.
[0047] Furthermore, parsing the reaction time data set based on the reaction time threshold includes:
[0048] d1: extract the jth actual reaction time in the reaction time dataset, where j is an integer greater than zero;
[0049] d2: Calculate the difference between the jth actual reaction time and the reaction time threshold, take the difference between the jth actual reaction time and the reaction time threshold as the reaction difference, set j=j+1, and return to step d1;
[0050] d3: Repeat the above steps d1 to d3 until j=R, then end the loop and obtain a reaction difference set containing P reaction differences. The reaction difference set is used as the driver's reaction performance in the second time interval [T, T+N], where P is an integer greater than zero.
[0051] Furthermore, the training logic of the preconfigured reaction trend analysis model is as follows:
[0052] Acquire historical reaction trend training data, and divide the historical reaction trend training data into a reaction trend training set and a reaction trend test set; the historical reaction trend training data includes a plurality of reaction time difference curve graphs and their corresponding reaction force change trend type annotation labels;
[0053] The reaction force change trend types include an upward trend, a downward trend and a fluctuating trend;
[0054] Constructing a third classifier, taking the reaction time difference curve graph in the reaction trend training set as input data of the third classifier, and taking the annotation label of the reaction force change trend type in the reaction trend training set as output data of the third classifier, training the third classifier, and obtaining an initial reaction trend analysis network;
[0055] The initial reaction trend analysis network is model verified using the reaction trend test set, and an initial reaction trend analysis network with a value greater than or equal to a third accuracy threshold is output as a preconfigured reaction trend analysis model.
[0056] Furthermore, the labeling logic of the labeling label of the reaction force change trend type is as follows:
[0057] Get the coordinates of each data point in the reaction time difference curve graph;
[0058] Substitute the coordinates of each data point into the pre-built mathematical calculation model to obtain the trend analysis coefficient; wherein the expression of the mathematical calculation model is: ; Where: is the trend analysis coefficient, is the coordinate of the ith data point, is the coordinate of the i+1th data point, is the total number of data points;
[0059] Retrieving a trend analysis coefficient threshold, wherein the trend analysis coefficient threshold includes a first trend analysis coefficient threshold and a second trend analysis coefficient threshold, wherein the first trend analysis coefficient threshold is greater than the second trend analysis coefficient threshold;
[0060] The trend analysis coefficient is compared with the first trend analysis coefficient threshold. If the trend analysis coefficient is greater than or equal to the first trend analysis coefficient threshold, the corresponding reaction time difference curve graph will be marked as a reaction force change trend type belonging to an upward trend; if the trend analysis coefficient is less than the first trend analysis coefficient threshold and greater than the second trend analysis coefficient threshold, the corresponding reaction time difference curve graph will be marked as a reaction force change trend type belonging to a fluctuating trend; if the trend analysis coefficient is less than or equal to the second trend analysis coefficient threshold, the corresponding reaction time difference curve graph will be marked as a reaction force change trend type belonging to a downward trend.
[0061] Furthermore, the determining whether the driver is driving fatigued includes:
[0062] Extract the driver's reaction force change trend type in the second time interval [T, T+N];
[0063] If the driver's reaction force change trend type in the second time interval [T, T+N] is an upward trend or a fluctuating trend, it is determined that the driver is not driving fatigued;
[0064] If the driver's reaction force change trend type in the second time interval [T, T+N] is a downward trend, it is determined that the driver is driving fatigued.
[0065] A fatigue driving monitoring system based on feature analysis, comprising:
[0066] The data acquisition module is used to acquire the exterior video and interior video of the vehicle in motion within the second time interval [T, T+N], and filter out the first target image frame in which the emergency occurs in the exterior video, to obtain an image frame set A including M first target image frames, where T, N and M are all integers greater than zero;
[0067] A data analysis module is used to obtain the actual reaction time of the driver when all emergencies occur based on the interior video, and obtain a reaction time data set including R actual reaction times, where R is an integer greater than zero;
[0068] A reaction determination module, used to obtain a driver's reaction time threshold, and parse a reaction time data set based on the reaction time threshold to obtain the driver's reaction performance in a second time interval [T, T+N];
[0069] A fatigue identification module is used to generate a reaction time difference curve diagram according to the reaction performance, input the reaction time difference curve diagram into a preconfigured reaction trend analysis model to determine the driver's reaction force change trend type in the second time interval [T, T+N], and judge whether the driver is fatigued based on the reaction force change trend type. If not, set N=N+K and return to the trigger data acquisition module; if so, generate a fatigue warning instruction, K is an integer greater than zero;
[0070] The fatigue reminder module is used to remind the driver of fatigue driving according to the fatigue warning instruction.
[0071] Compared with the prior art, the present invention has the following beneficial effects:
[0072] The present application discloses a fatigue driving monitoring method and system based on feature analysis, including: obtaining exterior video and interior video, and filtering out a first target image frame to obtain an image frame set A; obtaining the actual reaction time of the driver when all emergencies occur based on the interior video to obtain a reaction time data set; parsing the reaction time data set based on a reaction time threshold to obtain the driver's reaction performance in a second time interval; generating a reaction time difference curve graph based on the reaction performance, inputting the reaction time difference curve graph into a reaction trend analysis model to determine the reaction force change trend type, and judging whether the driver is fatigue driving based on the reaction force change trend type; and performing fatigue warning on the driver according to the fatigue warning instruction. Driving reminder; Based on the above characteristics, the present invention can have insight into the driver's reaction change trend, and then can analyze and judge the driver's fatigue state according to the reaction change trend, and further, can effectively avoid traffic accidents caused by fatigue driving; In addition, compared with the existing physiological signal monitoring method, the present invention does not need to wear additional hardware equipment, and the reaction time threshold is changed in real time by setting an iteration mechanism, so that the present invention has strong versatility and can realize fatigue monitoring of different driving objects; Compared with the existing eye image monitoring method, the present invention has strong anti-interference ability, does not pursue the ultimate image quality, and by monitoring the reaction change trend, the present invention can continuously ensure the accuracy of fatigue monitoring. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] Figure 1 A flowchart of a method for monitoring fatigue driving based on feature analysis provided by the present invention;
[0074] Figure 2 This is a module structure diagram of the feature analysis-based fatigue driving monitoring system provided by the present invention. DETAILED DESCRIPTION
[0075] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0076] Example 1
[0077] See also Figure 1 As shown, this embodiment discloses a fatigue driving monitoring method based on feature analysis, and the method includes:
[0078] S101: within a second time interval [T, T+N], obtaining an exterior video and an interior video of a vehicle while it is traveling, and screening out first target image frames in which an emergency occurs in the exterior video, to obtain an image frame set A including M first target image frames, where T, N and M are all integers greater than zero;
[0079] It should be noted that: at least two high-definition camera devices are installed on the vehicle, one of which is externally mounted on the vehicle and is used to capture the exterior video directly in front of the vehicle during driving; the other high-definition camera device is internally mounted on the vehicle and is used to capture the interior video of the instrument panel during driving; wherein the instrument panel is provided with a brake light display mark;
[0080] In implementation, the step of screening out the first target image frame in which the emergency occurs in the outdoor video includes:
[0081] a1: performing frame processing on the outdoor scene video to obtain an outdoor scene frame set including Q outdoor scene frames, where Q is an integer greater than zero;
[0082] a2: extract the vth exterior scene frame in the exterior scene frame set, where v is an integer greater than zero;
[0083] a3: Use the pre-configured event detection model to perform emergency event recognition on the v-th frame of the exterior scene to obtain an emergency event recognition result; if the emergency event recognition result shows that "an emergency event has occurred", mark the corresponding v-th exterior scene frame as the first target image frame; if the emergency event recognition result shows that "no emergency event has occurred", set v=v+1 and return to step a2;
[0084] The training logic of the preconfigured event detection model is as follows:
[0085] Acquire historical emergency training data, and divide the historical emergency training data into an emergency training set and an emergency test set; the historical emergency training data includes a plurality of exterior scene frames and their corresponding annotation labels;
[0086] The label includes "0" and "1", wherein "0" represents "no emergency event occurs", and "1" represents "an emergency event occurs";
[0087] It should be noted that: the exterior scene frames in the historical emergency training data are obtained by performing video frame processing on the exterior scene video collected by the external high-definition camera device on the vehicle. For details, please refer to the above description and will not be repeated here; and the annotation labels in the historical emergency training data are annotated in advance by technicians or AI data annotation specialists;
[0088] It should also be noted that: the emergency event specifically refers to one or more events such as the appearance of obstacles (such as pedestrians) or parking signs (such as red lights); further exemplary, for example: when a pedestrian appears in the center of an exterior scene frame, it is determined that an emergency event occurs in the exterior scene frame, and the technician or AI data annotation specialist annotates the exterior scene as "1"; on the contrary, when no pedestrian appears in the center of an exterior scene frame, it is determined that no emergency event occurs in the exterior scene frame, and the technician or AI data annotation specialist annotates the exterior scene as "0";
[0089] Constructing a first classifier, taking the exterior scene frames in the emergency training set as input data of the first classifier, taking the annotated labels in the emergency training set as output data of the first classifier, training the first classifier, and obtaining an initial event detection network;
[0090] Using the emergency event test set to perform model verification on the initial event detection network, and outputting an initial event detection network that is greater than or equal to a preset first test accuracy threshold as a preconfigured event detection model;
[0091] It should be noted that: the first classifier is one of the algorithm models for performing classification, such as random forest, decision tree, support vector machine, Bayesian or neural network;
[0092] a4: Repeat the above steps a2 to a3 until v=Q, then end the loop and obtain an image frame set A including M first target image frames.
[0093] S102: Acquire the actual reaction time of the driver when all emergencies occur based on the interior video, and obtain a reaction time data set including R actual reaction times, where R is an integer greater than zero;
[0094] In implementation, the actual reaction time of the driver when all emergencies occur is obtained, including:
[0095] b1: acquiring an image frame set B including W second target image frames based on the interior video;
[0096] Specifically, the acquisition logic of the image frame set B is as follows:
[0097] c1: performing frame processing on the interior scene video to obtain an interior scene frame set including U interior scene frames, where U is an integer greater than zero;
[0098] It should be noted that: the frame processing of the exterior video or interior video is implemented in a frame-by-frame manner based on time interval or scene change detection;
[0099] c2: extract the rth interior scene frame in the interior scene frame set, where r is an integer greater than zero;
[0100] c3: using the pre-configured reaction detection model to perform reaction action recognition on the r-th interior scene frame to obtain a reaction action recognition result; if the reaction action recognition result shows that "reaction action occurs", the r-th interior scene frame is marked as the second target image frame; if the reaction action recognition result shows that "no reaction action occurs", r=r+1 is set, and the process returns to step c2;
[0101] The training logic of the preconfigured reaction detection model is as follows:
[0102] Acquire historical reaction detection training data, and divide the historical reaction detection training data into a reaction detection training set and a reaction detection test set; the historical reaction detection training data includes a plurality of interior scene frames and their corresponding annotation labels;
[0103] The label includes "0" and "1", wherein "0" represents "no reaction action occurs", and "1" represents "reaction action occurs";
[0104] It should be noted that the interior scene frames in the historical reaction detection training data are obtained by performing video frame processing on the interior scene video captured by the built-in high-definition camera device on the vehicle. For details, please refer to the above description and will not be repeated here; and the annotation labels in the historical reaction detection training data are annotated in advance by technicians or AI data annotation specialists;
[0105] It should also be noted that the reaction action refers to the reaction performance to an emergency, for example, when an obstacle (such as a pedestrian) appears, turning the brake light on the dashboard green is used as a reaction action;
[0106] It is understandable that, compared with monitoring the braking action of the driver's legs, the on and off of the brake lights is easier to monitor and is less affected by factors, and the braking action is closely related to the on and off of the brake lights; however, it should be noted that this reaction action mode is only one of the implementation scenarios of the present invention, and the actual content is not limited to this; for example, it can also be achieved by monitoring the sudden change of vehicle speed (such as monitoring the swing amplitude of the speed pointer on the dashboard or monitoring the high and low changes of the vehicle speed value);
[0107] In a specific implementation, the technician or AI data annotation specialist annotates the interior scene frame containing the green light of the brake light as "1"; conversely, the exterior scene frame containing the green light of the brake light is off is annotated as "0";
[0108] Constructing a second classifier, taking the interior scene frames in the reaction detection training set as input data of the second classifier, and taking the annotated labels in the reaction detection training set as output data of the second classifier, training the second classifier, and obtaining an initial reaction detection network;
[0109] Using the reaction detection test set to perform model verification on the initial reaction detection network, and outputting an initial reaction detection network that is greater than or equal to a preset second test accuracy threshold as a preconfigured reaction detection model;
[0110] Similar to the first classifier, the second classifier is one of the algorithm models for performing classification, such as random forest, decision tree, support vector machine, Bayesian or neural network;
[0111] c4: repeat the above steps c2 to c3 until r=U, then end the loop and obtain an image frame set B including W second target image frames;
[0112] b2: extracting the g-th first target image frame from the image frame set A, extracting the timestamp of the g-th first target image frame, and marking the timestamp of the g-th first target image frame as the first timestamp;
[0113] b3: extracting the hth second target image frame from the image frame set B, extracting the timestamp of the hth second target image frame, and marking the timestamp of the hth second target image frame as the second timestamp;
[0114] b4: Determine whether the second timestamp satisfies the preset selection condition. If not, set h=h+1 and return to step b3. If so, calculate the difference between the second timestamp and the first timestamp, and mark the difference between the second timestamp and the first timestamp as the actual reaction time of the driver, set g=g+1 and return to step b2.
[0115] The preset selection condition is that the second timestamp is adjacent to the first timestamp, and the second timestamp is less than or equal to the first timestamp;
[0116] It should be understood that: in the actual driving process, several emergencies will be encountered, and each emergency will be accompanied by a corresponding reaction action, and the reaction action is generally later than the occurrence time of the emergency. Therefore, when filtering the second timestamp, if it does not meet the preset selection conditions, it is impossible to accurately find and distinguish the corresponding reaction actions of different emergencies;
[0117] For example, it is assumed that the appearance of an obstacle (pedestrian) is regarded as an emergency, and the green light of the brake light on the dashboard is regarded as a reaction action to the emergency; and it is further assumed that there are three first target image frames containing emergencies (i.e., the appearance of pedestrians), namely f1, f2 and f3, wherein the first timestamp of f1 is 8:00, the first timestamp of f2 is 8:10, and the first timestamp of f3 is 8:15; it is also assumed that there are three first target image frames containing reaction actions (i.e., the green light of the brake light on the dashboard), namely L1, L2 and L3, wherein the second timestamp of L1 is 8:03, the second timestamp of f2 is 8:15. The second timestamp of f3 is 8:18, and the first timestamp of f2 is 8:10, and the second timestamps adjacent to the first timestamp of f2 are 8:03 and 8:12. At this time, it is obviously unreasonable to use 8:03 as the second timestamp adjacent to the first timestamp of f2, because the reaction action cannot be earlier than the occurrence of the emergency. In other words, the driver cannot react before the emergency occurs. Therefore, according to the preset selection condition, 8:12 can be accurately obtained as the second timestamp adjacent to the first timestamp of f2.
[0118] b5: Repeat steps b3 to b4 above until a second timestamp that meets the preset selection condition is obtained, and then end the loop;
[0119] b6: Repeat the above steps b2 to b4 until g=M, then end the loop and obtain a reaction time data set including R actual reaction times, that is, obtain the actual reaction time of the driver when all emergencies occur.
[0120] S103: Obtaining a driver's reaction time threshold, and parsing a reaction time data set based on the reaction time threshold to obtain the driver's reaction performance in a second time interval [T, T+N];
[0121] In implementation, the step of obtaining the driver's reaction time threshold includes:
[0122] The time when the driver starts the vehicle is marked as K, and the face image of the driver in the main driving seat at time K is collected;
[0123] Retrieving the facial image of the vehicle owner, calculating the similarity between the facial image of the driver and the facial image of the vehicle owner, and if the similarity is greater than a preset similarity threshold, retrieving the average reaction time of the vehicle owner, and taking the average reaction time as the driver's reaction time threshold; if the similarity is less than or equal to the preset similarity threshold, collecting the average actual reaction time of the driver in the first time interval;
[0124] It should be noted that: the facial image of the vehicle owner is pre-stored in the system database; the similarity between the facial image of the driver and the facial image of the vehicle owner is calculated by a similarity algorithm (such as a cosine similarity algorithm); the average reaction time of the vehicle owner is pre-stored in the system database, and the average reaction time of the vehicle owner is obtained by prior experimental records;
[0125] Wherein, the end time of the first time interval is earlier than the start time of the second time interval;
[0126] It should also be noted that the average actual reaction time in the first time interval is obtained by averaging multiple actual reaction times in the first time interval; and the logic for obtaining the multiple actual reaction times in the first time interval is the same as the principle for obtaining the R actual reaction times in the above reaction time data set, which is described in detail above and will not be repeated here;
[0127] It should also be noted that the end time of the first time interval is earlier than the start time of the second time interval, which is designed based on the cause of fatigue driving and is also designed to adapt to different driving subjects; further explanation is that fatigue driving is mainly caused by the driver driving for a long time, and in the early stage of driving, since the driver has just driven the vehicle, most of the drivers are in a sober state; therefore, by recording multiple actual reaction times of the driver encountering emergencies during the early driving period, and averaging the multiple actual reaction times, and taking the average actual reaction time of the driver as the reaction time threshold, the reaction force of the non-target driver (i.e., the car owner) can be accurately obtained, which is conducive to facilitating the system to adapt to different driving subjects, so as to realize fatigue monitoring of different driving subjects, and improve the versatility of the system;
[0128] The average actual reaction time in the first time interval is used as the driver's reaction time threshold;
[0129] In implementation, parsing the reaction time dataset based on the reaction time threshold includes:
[0130] d1: extract the jth actual reaction time in the reaction time dataset, where j is an integer greater than zero;
[0131] d2: Calculate the difference between the jth actual reaction time and the reaction time threshold, take the difference between the jth actual reaction time and the reaction time threshold as the reaction difference, set j=j+1, and return to step d1;
[0132] d3: Repeat the above steps d1 to d3 until j=R, then end the loop and obtain a reaction difference set containing P reaction differences. The reaction difference set is used as the driver's reaction performance in the second time interval [T, T+N], where P is an integer greater than zero.
[0133] S104: Generate a reaction time difference curve graph according to the reaction performance, input the reaction time difference curve graph into a preconfigured reaction trend analysis model to determine the driver's reaction force change trend type in the second time interval [T, T+N], and judge whether the driver is fatigued based on the reaction force change trend type. If not, set N=N+K and return to step S101; if so, generate a fatigue warning instruction, K is an integer greater than zero;
[0134] It should be noted that: the reaction time difference curve graph is generated by importing the reaction difference set into a chart generation tool (such as using Python, Matplotlib library or Origin, etc.); further explained is that P reaction differences are input into the chart generation tool according to the generation time or generation order, with the generation time or generation order as the horizontal axis and the reaction difference (value) as the vertical axis, to obtain the reaction time difference curve graph;
[0135] In implementation, the training logic of the preconfigured reaction trend analysis model is as follows:
[0136] Acquire historical reaction trend training data, and divide the historical reaction trend training data into a reaction trend training set and a reaction trend test set; the historical reaction trend training data includes a plurality of reaction time difference curve graphs and their corresponding reaction force change trend type annotation labels;
[0137] The reaction force change trend types include an upward trend, a downward trend and a fluctuating trend;
[0138] It should be noted that the reaction time difference curve in the historical reaction trend training data is generated according to the driver's reaction performance. The logic for obtaining the reaction performance is described above and will not be repeated here.
[0139] The labeling logic of the labeling label of the reaction force change trend type is as follows:
[0140] Get the coordinates of each data point in the reaction time difference curve graph;
[0141] Substitute the coordinates of each data point into the pre-built mathematical calculation model to obtain the trend analysis coefficient; wherein the expression of the mathematical calculation model is: ; Where: is the trend analysis coefficient, is the coordinate of the ith data point, is the coordinate of the i+1th data point, is the total number of data points;
[0142] Retrieving a trend analysis coefficient threshold, wherein the trend analysis coefficient threshold includes a first trend analysis coefficient threshold and a second trend analysis coefficient threshold, wherein the first trend analysis coefficient threshold is greater than the second trend analysis coefficient threshold;
[0143] The trend analysis coefficient is compared with the first trend analysis coefficient threshold value. If the trend analysis coefficient is greater than or equal to the first trend analysis coefficient threshold value, the corresponding reaction time difference curve graph is marked as a reaction force change trend type belonging to an upward trend; if the trend analysis coefficient is less than the first trend analysis coefficient threshold value and greater than the second trend analysis coefficient threshold value, the corresponding reaction time difference curve graph is marked as a reaction force change trend type belonging to a fluctuating trend; if the trend analysis coefficient is less than or equal to the second trend analysis coefficient threshold value, the corresponding reaction time difference curve graph is marked as a reaction force change trend type belonging to a downward trend;
[0144] Constructing a third classifier, taking the reaction time difference curve graph in the reaction trend training set as input data of the third classifier, and taking the annotation label of the reaction force change trend type in the reaction trend training set as output data of the third classifier, training the third classifier, and obtaining an initial reaction trend analysis network;
[0145] Using the reaction trend test set to perform model verification on the initial reaction trend analysis network, and outputting the initial reaction trend analysis network that is greater than or equal to a third accuracy threshold as a preconfigured reaction trend analysis model;
[0146] Like the first classifier or the second classifier, the third classifier is one of the algorithm models for performing classification, such as random forest, decision tree, support vector machine, Bayesian or neural network.
[0147] In implementation, the determination of whether the driver is driving fatigued includes:
[0148] Extract the driver's reaction force change trend type in the second time interval [T, T+N];
[0149] If the driver's reaction force change trend type in the second time interval [T, T+N] is an upward trend or a fluctuating trend, it is determined that the driver is not driving fatigued;
[0150] If the driver's reaction force change trend type in the second time interval [T, T+N] is a downward trend, it is determined that the driver is driving fatigued.
[0151] S105: reminding the driver of fatigue driving according to the fatigue warning instruction;
[0152] It should be noted that the fatigue warning instruction can be an instruction to wake up the device, or it can be the execution of playing a preset reminder; for example, when the driver is found to be driving fatigued, the fatigue warning instruction can be used to wake up the steering wheel heating, wake up the window control device to open the window ventilation, or wake up the seat vibration to remind the driver of fatigue driving; or the fatigue warning instruction can be used to play a pre-recorded voice such as "Do not drive while tired" to remind the driver of fatigue driving;
[0153] By understanding the driver's reaction change trend, the present invention can analyze and determine the driver's fatigue state according to the reaction change trend, and further, can effectively avoid traffic accidents caused by fatigue driving; in addition, compared with the existing physiological signal monitoring method, the present invention does not need to wear additional hardware equipment, and the reaction time threshold is changed in real time by setting an alternation mechanism, so that the present invention has strong versatility and can realize fatigue monitoring of different driving objects; compared with the existing eye image monitoring method, the present invention has strong anti-interference ability, does not pursue the ultimate image quality, and by monitoring the reaction change trend, the present invention can continuously ensure the accuracy of fatigue monitoring.
[0154] Example 2
[0155] See also Figure 2 As shown, based on the same inventive concept, this embodiment discloses a fatigue driving monitoring system based on feature analysis. For details not provided in this embodiment, please refer to the relevant parts of the above embodiment 1. The system includes:
[0156] The data acquisition module 210 is used to acquire the exterior video and interior video of the vehicle during driving in the second time interval [T, T+N], and filter out the first target image frame in which the emergency occurs in the exterior video, to obtain an image frame set A including M first target image frames, where T, N and M are all integers greater than zero;
[0157] The data analysis module 220 is used to obtain the actual reaction time of the driver when all emergencies occur based on the interior video, and obtain a reaction time data set including R actual reaction times, where R is an integer greater than zero;
[0158] A reaction determination module 230, configured to obtain a driver's reaction time threshold, and parse a reaction time data set based on the reaction time threshold to obtain the driver's reaction performance in a second time interval [T, T+N];
[0159] The fatigue identification module 240 is used to generate a reaction time difference curve diagram according to the reaction performance, input the reaction time difference curve diagram into a preconfigured reaction trend analysis model to determine the driver's reaction force change trend type in the second time interval [T, T+N], and judge whether the driver is fatigued based on the reaction force change trend type. If not, set N=N+K, and return to the trigger data acquisition module 210; if so, generate a fatigue warning instruction, K is an integer greater than zero;
[0160] The fatigue reminder module 250 is used to remind the driver of fatigue driving according to the fatigue warning instruction.
[0161] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters, weights and thresholds in the formula are set by technicians in this field according to actual conditions.
[0162] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired network or a wireless network. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media sets. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD) or a semiconductor medium. The semiconductor medium may be a solid-state hard disk.
[0163] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0164] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0165] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only one, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0166] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0167] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0168] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
[0169] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A fatigue driving monitoring method based on feature analysis, characterized in that: The method comprises: S101: within a second time interval [T, T+N], obtaining an exterior video and an interior video of a vehicle while it is traveling, and screening out first target image frames in which an emergency occurs in the exterior video, to obtain an image frame set A including M first target image frames, where T, N and M are all integers greater than zero; S102: Acquire the actual reaction time of the driver when all emergencies occur based on the interior video, and obtain a reaction time data set including R actual reaction times, where R is an integer greater than zero; S103: Obtaining a driver's reaction time threshold, and parsing a reaction time data set based on the reaction time threshold to obtain the driver's reaction performance in a second time interval [T, T+N]; S104: Generate a reaction time difference curve graph according to the reaction performance, input the reaction time difference curve graph into a preconfigured reaction trend analysis model to determine the driver's reaction force change trend type in the second time interval [T, T+N], and judge whether the driver is fatigued based on the reaction force change trend type. If not, set N=N+K and return to step S101; if so, generate a fatigue warning instruction, K is an integer greater than zero; S105: Remind the driver of fatigue driving according to the fatigue warning instruction.
2. The method for monitoring fatigue driving based on feature analysis according to claim 1, characterized in that: The step of screening out the first target image frame in which the emergency occurs in the outdoor video includes: a1: performing frame processing on the outdoor scene video to obtain an outdoor scene frame set including Q outdoor scene frames, where Q is an integer greater than zero; a2: extract the vth exterior scene frame in the exterior scene frame set, where v is an integer greater than zero; a3: Use the pre-configured event detection model to perform emergency event recognition on the v-th frame of the exterior scene to obtain the emergency event recognition result; if the emergency event recognition result shows "an emergency event has occurred", the corresponding v-th exterior scene frame is marked as the first target image frame; if the emergency event recognition result shows "no emergency event has occurred", set v=v+1 and return to step a2; The training logic of the preconfigured event detection model is as follows: Acquire historical emergency training data, and divide the historical emergency training data into an emergency training set and an emergency test set; the historical emergency training data includes a plurality of exterior scene frames and their corresponding annotation labels; The label includes "0" and "1", wherein "0" represents "no emergency occurred", and "1" represents "an emergency occurred"; Constructing a first classifier, taking the exterior scene frames in the emergency training set as input data of the first classifier, taking the annotated labels in the emergency training set as output data of the first classifier, training the first classifier, and obtaining an initial event detection network; Using the emergency event test set to perform model verification on the initial event detection network, and outputting an initial event detection network that is greater than or equal to a preset first test accuracy threshold as a preconfigured event detection model; a4: Repeat the above steps a2 to a3 until v=Q, then end the loop and obtain an image frame set A including M first target image frames.
3. The method for monitoring driving fatigue based on feature analysis according to claim 2, characterized in that: The actual reaction time of the driver when all emergencies occur is obtained, including: b1: acquiring an image frame set B including W second target image frames based on the interior video; b2: extracting the g-th first target image frame from the image frame set A, extracting the timestamp of the g-th first target image frame, and marking the timestamp of the g-th first target image frame as the first timestamp; b3: extracting the hth second target image frame from the image frame set B, extracting the timestamp of the hth second target image frame, and marking the timestamp of the hth second target image frame as the second timestamp; b4: Determine whether the second timestamp satisfies the preset selection condition. If not, set h=h+1 and return to step b3. If so, calculate the difference between the second timestamp and the first timestamp, and mark the difference between the second timestamp and the first timestamp as the actual reaction time of the driver, set g=g+1 and return to step b2. The preset selection condition is that the second timestamp is adjacent to the first timestamp, and the second timestamp is less than or equal to the first timestamp; b5: Repeat steps b3 to b4 above until a second timestamp that meets the preset selection condition is obtained, and then end the loop; b6: Repeat steps b2 to b4 until g=M, then end the loop and obtain a reaction time data set including R actual reaction times.
4. The method for monitoring fatigue driving based on feature analysis according to claim 3 is characterized in that: The acquisition logic of the image frame set B is as follows: c1: performing frame processing on the interior scene video to obtain an interior scene frame set including U interior scene frames, where U is an integer greater than zero; c2: extract the rth interior scene frame in the interior scene frame set, where r is an integer greater than zero; c3: using the pre-configured reaction detection model to perform reaction action recognition on the rth interior scene frame to obtain a reaction action recognition result; if the reaction action recognition result shows "reaction action occurs", the rth interior scene frame is marked as the second target image frame; if the reaction action recognition result shows "no reaction action occurs", r=r+1 is set, and the process returns to step c2; The training logic of the preconfigured reaction detection model is as follows: Acquire historical reaction detection training data, and divide the historical reaction detection training data into a reaction detection training set and a reaction detection test set; the historical reaction detection training data includes a plurality of interior scene frames and their corresponding annotation labels; The label includes "0" and "1", wherein "0" represents "no reaction action occurs", and "1" represents "reaction action occurs"; Constructing a second classifier, taking the interior scene frames in the reaction detection training set as input data of the second classifier, and taking the annotated labels in the reaction detection training set as output data of the second classifier, training the second classifier, and obtaining an initial reaction detection network; Using the reaction detection test set to perform model verification on the initial reaction detection network, and outputting an initial reaction detection network that is greater than or equal to a preset second test accuracy threshold as a preconfigured reaction detection model; c4: Repeat the above steps c2 to c3 until r=U, then end the loop and obtain an image frame set B including W second target image frames.
5. The method for monitoring driving fatigue based on feature analysis according to claim 4 is characterized in that: The step of obtaining the driver's reaction time threshold comprises: The time when the driver starts the vehicle is marked as K, and the face image of the driver in the main driving seat at time K is collected; Retrieving the facial image of the vehicle owner, calculating the similarity between the facial image of the driver and the facial image of the vehicle owner, and if the similarity is greater than a preset similarity threshold, retrieving the average reaction time of the vehicle owner, and taking the average reaction time as the driver's reaction time threshold; if the similarity is less than or equal to the preset similarity threshold, collecting the average actual reaction time of the driver in the first time interval; Wherein, the end time of the first time interval is earlier than the start time of the second time interval; The average actual reaction time in the first time interval is used as the driver's reaction time threshold.
6. The method for monitoring driving fatigue based on feature analysis according to claim 5, characterized in that: The parsing of the reaction time data set based on the reaction time threshold includes: d1: extract the jth actual reaction time in the reaction time dataset, where j is an integer greater than zero; d2: Calculate the difference between the jth actual reaction time and the reaction time threshold, take the difference between the jth actual reaction time and the reaction time threshold as the reaction difference, set j=j+1, and return to step d1; d3: Repeat the above steps d1 to d3 until j=R, then end the loop and obtain a reaction difference set containing P reaction differences. The reaction difference set is used as the driver's reaction performance in the second time interval [T, T+N], where P is an integer greater than zero.
7. The method for monitoring driving fatigue based on feature analysis according to claim 6, characterized in that: The training logic of the preconfigured reaction trend analysis model is as follows: Acquire historical reaction trend training data, and divide the historical reaction trend training data into a reaction trend training set and a reaction trend test set; the historical reaction trend training data includes a plurality of reaction time difference curve graphs and their corresponding reaction force change trend type annotation labels; The reaction force change trend types include an upward trend, a downward trend and a fluctuating trend; Constructing a third classifier, taking the reaction time difference curve graph in the reaction trend training set as input data of the third classifier, and taking the annotation label of the reaction force change trend type in the reaction trend training set as output data of the third classifier, training the third classifier, and obtaining an initial reaction trend analysis network; The initial reaction trend analysis network is model verified using the reaction trend test set, and an initial reaction trend analysis network with a value greater than or equal to a third accuracy threshold is output as a preconfigured reaction trend analysis model.
8. The method for monitoring driving fatigue based on feature analysis according to claim 7, characterized in that: The labeling logic of the reaction force change trend type labeling label is as follows: Get the coordinates of each data point in the reaction time difference curve graph; Substitute the coordinates of each data point into the pre-built mathematical calculation model to obtain the trend analysis coefficient; wherein the expression of the mathematical calculation model is: ; Where: is the trend analysis coefficient, is the coordinate of the ith data point, is the coordinate of the i+1th data point, is the total number of data points; Retrieving a trend analysis coefficient threshold, wherein the trend analysis coefficient threshold includes a first trend analysis coefficient threshold and a second trend analysis coefficient threshold, wherein the first trend analysis coefficient threshold is greater than the second trend analysis coefficient threshold; The trend analysis coefficient is compared with the first trend analysis coefficient threshold. If the trend analysis coefficient is greater than or equal to the first trend analysis coefficient threshold, the corresponding reaction time difference curve graph will be marked as a reaction force change trend type belonging to an upward trend; if the trend analysis coefficient is less than the first trend analysis coefficient threshold and greater than the second trend analysis coefficient threshold, the corresponding reaction time difference curve graph will be marked as a reaction force change trend type belonging to a fluctuating trend; if the trend analysis coefficient is less than or equal to the second trend analysis coefficient threshold, the corresponding reaction time difference curve graph will be marked as a reaction force change trend type belonging to a downward trend.
9. The method for monitoring driving fatigue based on feature analysis according to claim 8, characterized in that: The determining whether the driver is driving fatigued includes: Extract the driver's reaction force change trend type in the second time interval [T, T+N]; If the driver's reaction force change trend type in the second time interval [T, T+N] is an upward trend or a fluctuating trend, it is determined that the driver is not driving fatigued; If the driver's reaction force change trend type in the second time interval [T, T+N] is a downward trend, it is determined that the driver is driving fatigued.
10. A fatigue driving monitoring system based on feature analysis, characterized in that: include: The data acquisition module is used to acquire the exterior video and interior video of the vehicle in motion within the second time interval [T, T+N], and filter out the first target image frame in which the emergency occurs in the exterior video, to obtain an image frame set A including M first target image frames, where T, N and M are all integers greater than zero; A data analysis module is used to obtain the actual reaction time of the driver when all emergencies occur based on the interior video, and obtain a reaction time data set including R actual reaction times, where R is an integer greater than zero; A reaction determination module, used to obtain a driver's reaction time threshold, and parse a reaction time data set based on the reaction time threshold to obtain the driver's reaction performance in a second time interval [T, T+N]; a fatigue identification module, for generating a reaction time difference curve diagram according to the reaction performance, inputting the reaction time difference curve diagram into a preconfigured reaction trend analysis model to determine the driver's reaction force change trend type in the second time interval [T, T+N], and judging whether the driver is fatigued based on the reaction force change trend type, if not, setting N=N+K, and returning to trigger the data acquisition module; If it exists, a fatigue warning instruction is generated, and K is an integer greater than zero; The fatigue reminder module is used to remind the driver of fatigue driving according to the fatigue warning instruction.
