Early warning method for abnormal driving state of driver and related equipment

Abstract

The invention provides an early warning method for an abnormal driving state of a driver and related equipment, and is applied to the technical field of vehicle safety auxiliary driving, and the method comprises the steps: obtaining the eyeball movement information of the driver collected by a calibrated eyeball tracking sensor, processing the eyeball movement information through a plurality of detection models, and obtaining an abnormal driving state of the driver; a plurality of state detection results are obtained, an abnormal driving detection strategy is applied to process eyeball operation information, and a fatigue detection result and an attention distraction detection result are obtained; and according to the fatigue detection result, the attention distraction detection result and the plurality of state detection results, determining whether to carry out abnormal driving state early warning reminding. The eyeball movement information, collected by the eyeball tracking sensor, of the driver is processed through multiple modes such as the detection model and the abnormal driving detection strategy, multiple modes of detection results are obtained, driving state detection is comprehensively carried out, the abnormal driving state of the driver is detected in time, and early warning is carried out in time.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle safety assisted driving, and in particular to a method for early warning of a driver's abnormal driving state and related equipment. Background Art

[0002] With the continuous increase in car ownership and people's growing demand for travel, traffic safety issues are receiving increasing attention. A large amount of data shows that driver fatigue and distraction are one of the main causes of traffic accidents.

[0003] Detecting the driver's driving status and providing timely warnings when abnormalities occur can effectively reduce the probability of traffic accidents caused by fatigue and distracted driving. Detecting abnormalities such as driver distraction and fatigue, and providing timely warnings, has become a pressing issue in improving driving safety. Summary of the Invention

[0004] In view of this, the embodiments of the present application provide a warning method and related equipment for abnormal driving status of a driver. The scheme provided in the embodiments of the present application is applied to obtain the driver's eye movement information collected by the eye tracking sensor, and use a variety of methods such as detection models and abnormal driving detection strategies to detect the driver's status, obtain multiple detection results, and comprehensively detect the driver's driving status, avoiding errors caused by a single detection method, and timely detect whether the driver's driving status is abnormal, and timely issue a warning to improve driving safety.

[0005] To achieve the above objectives, the present invention provides the following technical solutions:

[0006] The first aspect of the present application discloses a method for early warning of an abnormal driving state of a driver, comprising:

[0007] obtaining the driver's eye movement information collected by a calibrated eye tracking sensor;

[0008] Processing the eye movement information using at least one preset detection model to obtain a state detection result output by each detection model;

[0009] Based on the abnormal driving detection strategy, the eye movement information is processed to obtain the driver's fatigue detection result and attention distraction detection result;

[0010] When the fatigue detection result, the attention distraction detection result, and each of the state detection results meet the preset abnormal driving state warning conditions, an abnormal driving state warning reminder is executed.

[0011] The second aspect of the present application discloses a warning system for abnormal driving state of a driver, comprising:

[0012] a first acquiring unit, configured to acquire in real time the driver's eye movement information collected by a calibrated eye tracking sensor;

[0013] A second acquisition unit is configured to process the eye movement information using at least one preset detection model to obtain a state detection result output by each detection model;

[0014] a third acquiring unit, configured to process the eye movement information based on the abnormal driving detection strategy to acquire a fatigue detection result and a distraction detection result of the driver;

[0015] The early warning unit is used to execute an abnormal driving state early warning reminder when the fatigue detection result, the attention distraction detection result and each of the state detection results meet the preset abnormal driving state early warning conditions.

[0016] A third aspect of the present application discloses a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned method for warning of abnormal driving state of the driver.

[0017] The fourth aspect of the present application discloses an electronic device comprising a memory and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by one or more processors to implement the above-mentioned warning method for abnormal driving state of the driver.

[0018] Compared with the prior art, this application has the following advantages:

[0019] The present application provides a method and related equipment for warning of abnormal driving status of a driver, including: obtaining eye movement information of the driver collected by a calibrated eye tracking sensor, using multiple detection models to process the eye movement information to obtain multiple state detection results, applying an abnormal driving detection strategy to process the eye movement information to obtain fatigue detection results and attention distraction detection results; then determining whether to issue an abnormal driving status warning reminder based on the fatigue detection results, attention distraction detection results and multiple state detection results. By processing the eye movement information of the driver collected by the eye tracking sensor using multiple methods such as detection models and abnormal driving detection strategies, multiple detection results can be obtained, which can comprehensively detect the driver's driving state abnormality. This can avoid the situation where it is difficult to detect the driver's driving state abnormality in a timely manner due to a single detection method, and can promptly detect whether the driver's driving state is abnormal so that a warning can be issued in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0021] Figure 1 A flowchart of a method for warning of abnormal driving status of a driver provided in an embodiment of the present application;

[0022] Figure 2 A flowchart of an abnormal driving detection strategy based on an embodiment of the present application, which processes eye movement information and obtains driver fatigue detection results and attention distraction detection results;

[0023] Figure 3 A flow chart for calibrating an eye tracking sensor according to an embodiment of the present application;

[0024] Figure 4 A schematic diagram of the structure of a warning system for abnormal driving status of a driver provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0027] In this application, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0028] With the continuous increase in the number of cars and the growing demand for travel, traffic safety issues are receiving increasing attention. Driver fatigue and distraction are among the major causes of traffic accidents. During long driving periods, drivers are prone to fatigue symptoms such as frequent blinking, drooping eyelids, and yawning. At the same time, behaviors such as using mobile phones and talking to passengers while driving can also cause distraction and prolonged deviation from the road. Some existing automotive safety systems focus primarily on passive safety protection for vehicles, such as airbags and seatbelts, but are insufficient in actively detecting and warning the driver's own condition.

[0029] Currently, some cars are equipped with simple driver status detection technologies. For example, some vehicles indirectly determine driver fatigue by monitoring the frequency and force of steering wheel operation. However, this method is inaccurate and cannot directly reflect the driver's true condition. Other vehicles use facial recognition technology, which uses cameras to detect the driver's facial expressions. However, this technology is less accurate in detecting the driver's eye state, making it difficult to distinguish between normal blinking and frequent blinking caused by fatigue, resulting in inaccurate driver status detection results.

[0030] In order to solve the above problems and the problems mentioned in the background technology, the present application provides an early warning scheme for the driver's abnormal driving state. The scheme obtains the driver's eye movement information collected by a calibrated eye tracking sensor, uses multiple detection models to process the eye movement information, obtains multiple state detection results, applies an abnormal driving detection strategy to process the eye movement information, obtains fatigue detection results and attention distraction detection results; then determines whether to issue an abnormal driving state early warning reminder based on the fatigue detection results, attention distraction detection results and multiple state detection results. The present application collects the driver's eye movement information, uses detection models and abnormal driving detection strategies and other methods to process the eye movement information, and can obtain detection results of multiple detection methods. The entire detection scheme is more comprehensive, avoiding the inaccuracy of a single detection method, and can timely detect whether the driver's driving state is abnormal and give the driver a timely warning.

[0031] The present application can be used in many general or special vehicle computing device environments or configurations, such as vehicle processors and automobile central processing units.

[0032] Reference Figure 1 , is a flow chart of a method for warning of abnormal driving status of a driver provided in an embodiment of the present application, and is specifically described as follows:

[0033] S101: Acquire driver's eye movement information collected by a calibrated eye tracking sensor.

[0034] This application uses a high-precision and high-frame-rate eye-tracking sensor. The eye-tracking sensor is installed above the dashboard or near the rearview mirror in the cockpit. The location of the sensor needs to be convenient for collecting data from the driver's eyes. It can be fixed with a bracket or embedded installation method to ensure the stability of the eye-tracking sensor, thereby more accurately obtaining the driver's eye movement information and providing a data basis for subsequent accurate judgment of the driver's status.

[0035] In this application, the eye tracking sensor collects data on the movement of the driver's eyes while driving a vehicle, and the collected content includes data on multiple parameters such as eye position coordinates, pupil size, and line of sight direction. The collected initial data includes data on multiple parameters such as eye position coordinates, pupil size, and line of sight direction; the eye tracking sensor collects the optical image of the driver's eyes, and the optical components inside the eye tracking sensor convert the optical image of the driver's eyes into an electrical signal, and the initial data is obtained after preliminary processing by the internal circuit; the initial data can be stored in a matrix format, with each row representing data at a certain moment, and different columns corresponding to different parameters.

[0036] The initial data collected by the eye tracking sensor can be directly used as eye movement information, and the initial data collected by the eye tracking sensor can also be preprocessed to obtain eye movement information.

[0037] Furthermore, the purpose of preprocessing the initial data collected by the eye tracking sensor is to remove noise and outliers in the data, thereby improving the accuracy of the data.

[0038] The process of preprocessing the initial data collected by the eye tracking sensor includes: using a filtering algorithm to remove noise from the initial data, the filtering algorithm is such as a Gaussian filtering algorithm; when using the filtering algorithm to process the noise of the initial data, for each parameter in the initial data, set appropriate filtering parameters respectively to remove the noise generated by external interference in the data of the parameter, so that the data is smoother and more stable; wherein, the filtering parameters include but are not limited to the Gaussian kernel size and standard deviation, and the filtering parameters of different parameters in the initial data are different; after the initial data is denoised, the denoised initial data is processed for outliers, specifically: for each parameter in the initial data, set a reasonable threshold range for the parameter, the threshold range is determined based on a large amount of experimental data and actual driving experience, and then the data points that exceed the threshold position are regarded as outliers. For outliers, according to their number and the degree of influence on the data, a method of elimination or correction is adopted to ensure the accuracy of the data.

[0039] S102: Use at least one preset detection model to process the eye movement information and obtain a state detection result output by each detection model.

[0040] For each detection model, based on the feature requirements of the detection model, a feature vector is extracted from the eye movement information, and the feature vector is input into the detection model so that the detection model processes the feature vector and outputs a state detection result indicating whether the driver is in a normal state or an abnormal state. Preferably, the feature vectors extracted from the eye movement information include, but are not limited to, vectors such as the driver's blinking frequency, rate of change of eye size, line of sight direction, speed of eye movement, acceleration of eye movement, and distribution of eye movement in different areas. Different detection models may have different requirements for features, such as the type of data and specific features that the detection model can accept.

[0041] It should be noted that different detection models require different feature data, and the structure and data processing methods of each detection model are different. Different detection models have different advantages. Therefore, eye movement information can be processed from different angles and directions, and all-round coverage analysis can be carried out to avoid a single detection model outputting incorrect detection results due to inaccurate or incomplete data during analysis, resulting in the inability to issue timely warnings. Using multiple detection models to process eye movement information can output status detection results analyzed from different angles and directions.

[0042] Exemplarily, the present application uses two detection models, one of which is a support vector machine (SVM) model and the other is a neural network (NN) model.

[0043] The SVM model normalizes the extracted feature vectors so that they have the same scale, selects a suitable kernel function (such as a linear kernel, a radial basis kernel, etc.) and adjusts the parameters. Then, based on these processed feature vectors, it uses its classification decision boundary to classify the driver's status and determine whether the driver's driving status is abnormal.

[0044] The NN model comprises a multi-layered neural network structure, consisting of an input layer, hidden layers, and an output layer. The input layer receives the extracted feature vectors, the hidden layer processes and transmits the data using neuron activation functions, and the output layer outputs the classification results. The number of hidden layers and the number of neurons in each layer are determined based on the complexity of the data and the classification requirements. By adjusting these structural parameters and selecting appropriate activation functions (such as the Sigmoid function and ReLU function), the model can learn the characteristics of different data categories and thus accurately judge the driver's status. The activation function is used to activate neurons and transmit data. Different activation functions have different effects on model performance, and the selection should be based on experimental results.

[0045] Preferably, the detection model may classify the driver's state into three categories: normal, fatigued, and distracted. When the driver's state is classified as fatigued or distracted, the detection model outputs a state detection result indicating that the driver's driving state is abnormal. When the driver's state is classified as normal, the model outputs a state detection result indicating that the driver's driving state is normal. Alternatively, when the detection model classifies the driver's state, there may be two classification results: one indicating a normal driving state and the other indicating an abnormal driving state.

[0046] Furthermore, the detection model needs to be trained before it can be put into use, and can only be put into use after training.

[0047] When training the detection model, a training set and a validation set are first constructed. A large number of driver eye movement data are collected as sample data, which is labeled with categories such as fatigue, distraction, and normal. The sample data is divided into a training set and a validation set according to a specific ratio, such as 7:3. The training set is used to train the detection model's parameters, enabling it to learn the characteristics of different types of data. During training, the detection model's parameters are continuously adjusted through an optimization algorithm to reduce classification error rates. For example, for SVM models, toolkits such as libsvm can be used for training; for NN models, deep learning frameworks such as TensorFlow and PyTorch can be used for training. The trained detection model is then validated using the validation set. Once the validation is successful, the model can be put into operation.

[0048] It's important to note that real-world driving scenarios are complex and ever-changing, and drivers may exhibit different performance when fatigued or distracted. Detection models, through training and learning from extensive data, can better adapt to these individual differences and complex situations. For example, the normal blinking frequency of different drivers may vary slightly. The detection model can automatically capture these differences during the learning process, improving its adaptability. Different detection models utilize different data analysis methods, allowing them to analyze a driver's driving state from multiple perspectives to effectively avoid misjudgments.

[0049] S103: Based on the abnormal driving detection strategy, the eye movement information is processed to obtain the driver's fatigue detection results and attention distraction detection results.

[0050] Reference Figure 2 , which is a flowchart of an abnormal driving detection strategy based on an embodiment of the present application, processing eye movement information, and obtaining driver fatigue detection results and attention distraction detection results, as described in detail below:

[0051] S201. Determine various fatigue detection dimensions and various attention detection dimensions based on an abnormal driving detection strategy.

[0052] Abnormal driving detection strategies include strategies for detecting whether the driver is fatigued and strategies for detecting whether the driver is distracted.

[0053] Furthermore, the strategy for detecting whether the driver is fatigued records various fatigue detection dimensions, and the strategy for detecting whether the driver is distracted records various attention detection dimensions.

[0054] The various fatigue detection dimensions include blinking frequency detection dimension, eyeball vertical position change detection dimension, and driver pupil size change rate detection dimension; the various attention detection dimensions include the driver's line of sight deviation angle detection dimension from the road center direction, the driver's line of sight stay time in non-road related areas detection dimension, and driver pupil size change rate detection dimension.

[0055] There are corresponding detection strategies for different detection dimensions.

[0056] S202. For each fatigue detection dimension, extract fatigue state detection data of the fatigue detection dimension from the eye movement information, and generate a first detection result of the fatigue detection dimension based on the fatigue state detection data and the detection strategy of the fatigue detection dimension.

[0057] It should be noted that there are two situations for the first detection result. One situation indicates that the driver is in a fatigued state, and the other situation indicates that the driver is not in a fatigued state.

[0058] For the blinking frequency detection dimension in each fatigue detection dimension, the driver's blinking data within the first preset time window is extracted from the eye movement information. The blinking data is the fatigue state detection data of the blinking frequency detection dimension. The driver's blinking frequency within the first preset time window is determined through the blinking data. When the blinking frequency is higher than the preset blinking frequency in the detection strategy of the fatigue detection dimension, a first detection result is generated, indicating that the driver is in a fatigue state. When the blinking frequency is not higher than the preset blinking frequency in the detection strategy of the fatigue detection dimension, a first detection result is generated, indicating that the driver is not in a fatigue state. The preset blinking frequency can be set according to actual needs, for example, it can be 15 times / minute, and the preset time window can also be set according to actual needs. Preferably, the first preset time window can be within the last one minute.

[0059] For the eyeball position change detection dimension in each fatigue detection dimension, the driver's eyeball movement data in the vertical direction within the second preset time window is extracted from the eyeball movement information. The movement data is the fatigue state detection data of the eyeball position change detection dimension in the vertical direction. The distance over which the driver's eyeball position continues to drop in the treatment direction within the second preset time window is determined through the movement data. When the distance exceeds the preset distance in the detection strategy of the fatigue detection dimension, a first detection result is generated, indicating that the driver is in a fatigue state; when the distance does not exceed the preset distance in the detection strategy of the fatigue detection dimension, a first detection result is generated, indicating that the driver is not in a fatigue state. The preset distance can be set according to actual needs, for example, it can be set to 5mm, and the second preset time window can be set according to actual needs, for example, it can be set to 5 minutes.

[0060] For the driver's pupil size change rate detection dimension in each fatigue detection dimension, the driver's pupil size data is extracted from the eye movement information. The pupil size data is the fatigue state detection data of the driver's pupil size change rate detection dimension. The driver's pupil change rate is determined through the pupil size data. When the pupil change rate is lower than the first preset pupil change rate in the detection strategy of the fatigue detection dimension, a first detection result is generated, indicating that the driver is in a fatigue state. When the pupil change rate is not lower than the first preset pupil change rate in the detection strategy of the fatigue detection dimension, a first detection result is generated, indicating that the driver is not in a fatigue state.

[0061] The driver's pupil change rate = the driver's current pupil size / the driver's pupil baseline, where the driver's pupil size is obtained from the pupil size data; the driver's pupil baseline is data collected when the driver is in a normal state.

[0062] The first preset pupil change rate is obtained by processing a large amount of data. Specifically, data on pupil change rates is collected from a large number of drivers in normal driving conditions and a large number of drivers in fatigued driving conditions. For example, 1,000 drivers are selected and data is collected under normal driving conditions and different driving durations (simulating fatigue conditions). At least 100,000 sets of valid data are collected for each condition. This data is then organized and statistical quantities such as the mean and standard deviation of the pupil size change rate under different conditions are calculated.

[0063] By analyzing a large amount of data, a judgment threshold is set. For example, statistics show that the mean pupil size change rate during normal driving is 0.05, with a standard deviation of 0.02. During fatigue driving, the mean pupil change rate is 0.03, with a standard deviation of 0.015. The judgment threshold = the mean pupil size change rate during normal driving - N * the standard deviation of the pupil change rate during fatigue driving, where N is a positive integer. For example, if N is 1, the judgment difference is 0.035. The judgment threshold here is the first preset pupil change rate.

[0064] S203 : Based on each first detection result, generate a fatigue detection result indicating whether the driver is in a fatigue state.

[0065] When there is a first detection result among the first detection results indicating that the driver is in a fatigue state, a fatigue detection result indicating that the driver is in a fatigue state is generated; when all the first detection results indicate that the driver is not in a fatigue state, a fatigue detection result indicating that the driver is not in a fatigue state is generated.

[0066] It should be noted that, when the number of first detection results indicating that the driver is in a fatigued state is greater, the possibility that the driver is in a fatigued state is greater, and the accuracy of the generated fatigue detection result indicating that the driver is in a fatigued state is higher.

[0067] S204. For each attention detection dimension, extract the attention state detection data of the attention detection dimension from the eye movement information, and generate a second detection result of the attention detection dimension based on the attention state detection data and the detection strategy of the attention detection dimension.

[0068] There are two situations for the second detection result, one is a result indicating that the driver is in a distracted state, and the other is a result indicating that the driver is not in a distracted state.

[0069] For the detection dimension of the deviation angle of the driver's line of sight from the center direction of the road in each attention detection dimension, the driver's line of sight direction data is extracted from the eye movement information, and the line of sight direction data is the fatigue state detection data of the detection dimension of the deviation angle of the driver's line of sight from the center direction of the road. The deviation angle of the driver's line of sight from the center direction of the road is determined through the line of sight direction data, and the duration of the deviation angle is determined. When the deviation angle exceeds the preset angle in the detection strategy of the attention detection dimension, and the duration of the deviation angle exceeds the first preset duration in the detection strategy of the attention detection dimension, a second detection result is generated indicating that the driver is in a distracted state; otherwise, a second detection result is generated indicating that the driver is not in a distracted state. The first preset duration and the preset angle can be set according to actual needs. For example, the preset angle can be 30 degrees, and the first preset duration can be 5 seconds.

[0070] For each attention detection dimension, the time the driver's gaze remains in non-road-related areas is detected. Statistical data on the time the driver's gaze remains in non-road-related areas is extracted from eye movement information. This statistical data serves as fatigue state detection data for the time the driver's gaze remains in non-road-related areas. The statistical data is used to determine the proportion of the driver's time spent in non-road-related areas and the duration of the driver's stay in non-road-related areas. When the proportion of the driver's time spent in non-road-related areas exceeds a preset proportion in the detection strategy for that attention detection dimension, and the duration of the driver's stay in non-road-related areas exceeds a second preset duration in the detection strategy for that attention detection dimension, a second detection result is generated indicating that the driver is in a distracted state. Otherwise, a second detection result is generated indicating that the driver is not in a distracted state. The preset proportion and second preset duration can be set according to actual needs, such as a preset proportion of 30% and a second preset duration of 5 seconds. It should be noted that non-road-related areas can include mobile phone screens, the positions of other passengers in the vehicle, and the like.

[0071] For the driver's pupil size change rate detection dimension in each attention detection dimension, the driver's pupil size data is extracted from the eye movement information. The pupil size data is the attention detection data of the driver's pupil size change rate detection dimension. The driver's pupil change rate is determined through the pupil size data. When the pupil change rate is lower than the second preset pupil change rate in the detection strategy of the attention detection dimension, a second detection result is generated, indicating that the driver is in a distracted state. When the pupil change rate is not lower than the second preset pupil change rate in the detection strategy of the attention detection dimension, a second detection result is generated, indicating that the driver is not in a distracted state.

[0072] It should be noted that the process of determining the driver's pupil change rate is referred to the above content and will not be repeated here.

[0073] The second preset pupil change rate is obtained by processing a large amount of data. Specifically, data on pupil change rates is collected from a large number of drivers in normal driving conditions and a large number of drivers in distracted driving conditions. For example, 1,000 drivers are selected and data is collected under normal driving conditions and different distraction scenarios (simulating distracted driving conditions). At least 100,000 sets of valid data are collected for each condition. This data is then organized and statistical quantities such as the mean and standard deviation of the pupil size change rate under different conditions are calculated.

[0074] By analyzing a large amount of data, a judgment threshold is set. For example, statistics show that the mean pupil size change rate during normal driving is 0.05, with a standard deviation of 0.02. During distracted driving, the mean pupil change rate is 0.08, with a standard deviation of 0.03. The judgment threshold = the mean pupil size change rate during normal driving + M * the standard deviation of the pupil change rate during distracted driving, where M is a positive integer. For example, if M is 1, the judgment difference is 0.07. The judgment threshold here is the second preset pupil change rate.

[0075] S205 : Based on each second detection result, generate a distraction detection result indicating whether the driver is in a distracted state.

[0076] When there is a second detection result among the second detection results indicating that the driver is in a distracted state, a distraction detection result indicating that the driver is in a distracted state is generated; when all the second detection results indicate that the driver is not in a distracted state, a distraction detection result indicating that the driver is not in a distracted state is generated.

[0077] It should be noted that, when the number of second detection results indicating that the driver is in a distracted state is greater, the possibility that the driver is in a distracted state is greater, and the accuracy of the generated distraction detection result indicating that the driver is in a distracted state is higher.

[0078] In the present application, the detection of whether the driver is in a fatigue state and the detection of whether the driver is distracted can be performed simultaneously.

[0079] In the method provided in the embodiment of the present application, whether the driver is in a fatigue driving state and whether the driver is in a distracted state are detected from multiple dimensions; whether the driver is fatigued or distracted is effectively and quickly determined, and the multi-dimensional detection results are more accurate and comprehensive.

[0080] S104: When the fatigue detection result, the attention distraction detection result, and the various state detection results meet the preset abnormal driving state warning conditions, an abnormal driving state warning reminder is executed.

[0081] When there is a state detection result indicating that the driver's state is abnormal among the state detection results, a first conclusion indicating that the driver's state is abnormal is generated; when all the state detection results indicate that the driver's state is normal, a first conclusion indicating that the driver's state is normal is generated;

[0082] When the fatigue detection result indicates that the driver is in a fatigued state and / or the attention distraction detection result indicates that the driver is in a distracted state, generating a second conclusion indicating that the driver's state is abnormal; when the fatigue detection result indicates that the driver is not in a fatigued state and the attention distraction detection result indicates that the driver is not in a distracted state, generating a second conclusion indicating that the driver's state is normal;

[0083] When both the first conclusion and the second conclusion indicate that the driver is in a normal state, no warning reminder is performed;

[0084] When one of the first conclusions or the second conclusion indicates that the driver's state is abnormal, the fatigue detection result, the attention distraction detection result, and the various state detection results are determined, and a first-level warning reminder is executed; the first-level warning reminder is a milder warning method, which indicates that the driver's driving state is slightly abnormal or may be abnormal; at this time, a warning message can be displayed on the instrument panel to remind the driver, which can effectively remind the driver that the driving state is abnormal and the driving state needs to be adjusted in time;

[0085] When both the first conclusion and the second conclusion indicate that the driver's state is abnormal, the fatigue detection results, the attention distraction detection results and the various state detection results are determined, and a second-level early warning reminder is performed; the second-level early warning reminder is a strong early warning method. When this method is used, it indicates that the driver's driving state has become seriously abnormal, and the driver can be reminded by sounding an alarm and vibrating the seat. The frequency and volume of the alarm are set according to the severity, and the intensity and frequency of the seat vibration can be adjusted according to the severity. At the same time, a warning message is displayed on the instrument panel to clearly prompt the driver's state and precautions; further, the severity of the abnormal driving state of the driver can be determined based on the sum of the number of detection results indicating that the driver's state is abnormal, the number of the first detection results indicating that the driver is in a fatigued state, and the number of the second results indicating that the driver is in a distracted state. The larger the total, the more serious the degree of abnormal driving state of the driver.

[0086] In the method provided in the embodiment of the present application, the driver's eye movement information collected by a calibrated eye tracking sensor is obtained; the eye movement information is processed using at least one detection model to obtain the state detection result output by each detection model; the fatigue detection result and attention distraction detection result obtained by processing the eye movement information based on the abnormal driving detection strategy are obtained; when the fatigue detection result, attention distraction detection result and each state detection result meet the preset abnormal driving state warning conditions, an abnormal driving state warning reminder is executed. By using a variety of methods such as detection models and abnormal driving detection strategies to process the eye movement information, the detection results of multiple detection methods are obtained. The entire detection scheme is more comprehensive, avoiding the inaccuracy of a single detection method, and can timely detect whether the driver's driving state is abnormal, and timely warn the driver, reducing the possibility of traffic accidents caused by the driver's abnormal driving state, and providing a safe driving environment for the driver.

[0087] Reference Figure 3 , which is a flow chart for calibrating an eye tracking sensor according to an embodiment of the present application, is described in detail as follows:

[0088] S301 : Collect eye data of the driver, and set sensor parameters of the eye tracking sensor based on the eye data.

[0089] In the embodiment provided in the present application, the eye tracking sensor is calibrated when preset calibration conditions are met; the preset calibration conditions are met such as: 1) when the eye tracking sensor is activated and used for the first time; 2) when the driver changes.

[0090] The collected driver's eye data includes but is not limited to the driver's eye distance, eye socket shape, relative height and horizontal position of the eyes and the dashboard, etc.

[0091] The sensor parameters are preliminarily set based on the collected eye data. The sensor parameters include but are not limited to focal length, field of view, etc. A reasonable setting of the focal length can ensure that the sensor clearly captures the driver's eye image and ensures that the collected eye movement data is accurate; the setting of the field of view must take into account the ability to fully cover the driver's eye activity area while avoiding excessive collection of irrelevant information to improve data processing efficiency.

[0092] S302: Acquire current environmental information, adjust the light intensity in the cockpit where the driver is located based on the environmental information, and collect eye data of the driver looking at each preset calibration area under various light intensities.

[0093] Environmental information includes the driver's driving habit data, current time, weather, location, etc.; driving habit data includes the driver's sitting posture, driving habits, etc., such as the driver's seat information.

[0094] When adjusting the light intensity in the driver's cockpit based on environmental information, collecting the driver's eye data when looking at each calibration area under various light intensities can enable the eye tracking sensor to adapt to the complex lighting environment in the cockpit and improve the accuracy of the collected data.

[0095] In the embodiments provided herein, a PID dimming algorithm can be used to adjust the cockpit backlight intensity to adapt the light intensity to different scenarios (e.g., daytime / nighttime, sunny / rainy days). For example, the backlight intensity can be adjusted to a higher setting when the external light is strong on sunny days, and to a lower setting at night.

[0096] Different ambient lighting conditions can affect the accuracy of eye-tracking sensor data (e.g., pupil size and eye reflection). Adjusting the lighting allows the sensor to accurately capture eye data in a variety of lighting conditions, ensuring calibration effectiveness. By adjusting light intensity, the eye-tracking sensor adapts to the complex lighting conditions within the cockpit, improving the accuracy of collected data and, in turn, enhancing the sensor's tracking of the driver's eye movements. This provides reliable data for subsequent driving status detection (e.g., fatigue and distraction).

[0097] In the method provided in the embodiment of the present application, the eye data includes but is not limited to the relative height and horizontal position of the driver's line of sight and the calibration area, the driver's eye rotation angle, pupil size, eye position coordinates, line of sight direction, blinking frequency, pupil size change rate, etc.

[0098] The preset calibration areas are key areas where the driver's vision may be involved during driving; the calibration areas include but are not limited to the rearview mirror area, left mirror area, right mirror area, front area, instrument panel area, etc.

[0099] S303: Use the data of each eyeball to adjust the sensing parameters of the eye tracking sensor.

[0100] In the embodiment provided in this application, the core steps of adjusting the sensing parameters include:

[0101] Step 1. Data processing: Filter and denoise the collected eye data (such as gaze coordinates, pupil size, and eye rotation angle) (such as Kalman filtering) to remove outliers caused by environmental interference.

[0102] Step 2. Deviation calculation: Compare the gaze point in the eye data with the actual coordinates of the preset calibration area to calculate the position deviation (such as horizontal / vertical angle deviation).

[0103] Step 3. Iterative parameter optimization: Use the least squares method or gradient descent method to iteratively adjust the sensor's optical parameters (such as focal length, white balance) or mapping matrix to gradually reduce the deviation.

[0104] Step 4. Error verification: Collect data again after adjustment. If the tracking error is ≤ 0.5°, the calibration is complete; otherwise, continue optimization.

[0105] Let's use an example scenario to illustrate: During calibration, when the driver looks at the calibration point on the left rearview mirror, the sensor initially outputs line-of-sight coordinates of (-25°, +8°), while the actual coordinates are (-30°, +5°), with a deviation of (5°, -3°). The adjustment process is as follows: Step 1. Data Processing: Gaussian filtering is performed on the eye data to remove jitter noise. Step 2. Parameter Adjustment: Gradient descent is used to increase the focal length from 80mm to 83mm, and the rotation matrix parameters between the image coordinate system and the world coordinate system are corrected (e.g., -2° rotation around the X axis and +1° rotation around the Y axis). Step 3. Verification Results: After adjustment, data is collected again, and the line-of-sight coordinate deviation is reduced to (1°, 0.5°), meeting the error requirements, completing parameter optimization.

[0106] Adjusting the sensing parameters of the eye tracking sensor is conducive to the application of the eye sensor in various scenarios and improving the accuracy of the collected data.

[0107] In the embodiment provided in this application, the process of calibrating the eye tracking sensor is described by taking the scenario of triggering calibration when a new driver uses the system for the first time. The specific steps are as follows:

[0108] Step 1: Collect eye data (interocular distance 62mm, vertical distance between eyeball and instrument panel 115mm), initially setting the sensor focal length to 75mm and the field of view to 110°×80°.

[0109] Step 2: The system adjusts the cockpit backlight to 400 lux at 2:00 PM on a sunny day (external lighting 500 lux). The driver then looks at five calibration areas, including the road ahead and the left rearview mirror. The sensor records a horizontal eye rotation angle of -28° and a pupil size of 4.2 mm while looking at the left rearview mirror.

[0110] Step 3: Adjust the focal length to 78mm based on the data. After optimization, the tracking error is reduced from 0.8° to 0.4°, completing the calibration.

[0111] This application calibrates the eye tracking sensor so that it can quickly adapt to the complex lighting in the cockpit. The eye tracking sensor can be optimized differently according to the driver, thereby improving the tracking accuracy of the eye tracking sensor, accurately tracking the driver's eye movement information, and improving the accuracy of the collected data.

[0112] Some existing vehicles indirectly determine driver fatigue by monitoring the frequency and force of steering wheel operation, but this is less accurate. Other vehicles use facial recognition technology, which can inaccurately monitor the driver's eye state. This system, however, directly utilizes eye-tracking sensors to capture the driver's eye movement information. Compared to existing technologies, it can more accurately monitor driver fatigue and distraction, significantly improving monitoring accuracy. Compared to existing technologies that lack active driver status monitoring and early warning, this system architecture forms a complete closed loop, from hardware deployment to data processing to early warning response. Through the detailed design of the data transmission and processing mechanisms between modules, active driver status monitoring and multi-level early warning are achieved, addressing the shortcomings of existing technologies and enhancing active vehicle driving safety.

[0113] This application effectively avoids traffic accidents caused by driver fatigue or distraction by timely and accurate monitoring of driver fatigue and distraction, and issuing warnings. Compared to existing passive safety systems, this invention monitors the driver's condition and issues warnings at the source, representing an active safety measure that further improves driving safety. Prompt reminders when the driver shows signs of fatigue or distraction help the driver adjust their state, improve driving comfort and convenience, and thus enhance the user experience.

[0114] This application improves the safety of car driving by actively monitoring and warning the driver's status, avoiding safety problems caused by driver fatigue or distraction, and can be effectively applied in actual driving scenarios to promptly detect driver fatigue and distraction, and is highly practical.

[0115] Although the present invention depicts operations in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order. Multitasking and parallel processing may be advantageous under certain circumstances.

[0116] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0117] and Figure 1 Corresponding to the method shown, the present application provides a warning system for abnormal driving state of the driver, which is used to support Figure 1 For the implementation of the method shown, refer to Figure 4 , is a schematic diagram of the structure of a warning system for abnormal driving state of a driver provided in an embodiment of the present application, specifically comprising:

[0118] A first acquiring unit 401 is configured to acquire, in real time, the driver's eye movement information collected by a calibrated eye tracking sensor;

[0119] A second acquiring unit 402 is configured to process the eye movement information using at least one preset detection model to acquire a state detection result output by each detection model;

[0120] A third acquisition unit 403 is configured to process the eye movement information based on the abnormal driving detection strategy to obtain a fatigue detection result and a distraction detection result of the driver;

[0121] The warning unit 404 is configured to execute an abnormal driving state warning reminder when the fatigue detection result, the distraction detection result, and each of the state detection results meet preset abnormal driving state warning conditions.

[0122] In the system provided by the embodiment of the present application, the first acquisition unit 401, the second acquisition unit 402, and the third acquisition unit 403 can be integrated with a control unit. The present application constructs a complete mechanism from sensor data collection, to data preprocessing by the control unit (such as using a filtering algorithm to remove noise, setting a threshold to process abnormal values), to driving the early warning module to work based on the processed data. The system architecture forms a complete closed loop from hardware deployment to data processing to early warning response. By detailed design of the data transmission and processing mechanism between each unit, active monitoring of the driver's status and multi-level early warning are achieved, which makes up for the shortcomings of the existing technology and enhances the active safety of automobile driving. The close cooperation of various links realizes real-time monitoring and early warning of the driver's status. This systematic data processing and interaction method is innovative.

[0123] In another embodiment provided by the present application, the system further includes: a calibration unit;

[0124] The calibration unit is used to calibrate the eye tracking sensor, and the specific process includes:

[0125] collecting eye data of the driver and setting sensor parameters of the eye tracking sensor based on the eye data;

[0126] obtaining current environmental information, adjusting the light intensity in the cockpit where the driver is located based on the environmental information, and collecting eye data of the driver looking at each preset calibration area under various light intensities;

[0127] The sensing parameters of the eye tracking sensor are adjusted using the eye data.

[0128] Furthermore, the calibration unit can also be integrated into the above control unit.

[0129] In another embodiment provided by the present application, the second acquisition unit 402 of the system executes a process of processing the eye movement information using at least one preset detection model to obtain the state detection result output by each detection model, including:

[0130] For each of the detection models, based on the feature requirements of the detection model, a feature vector is extracted from the eye movement information, and the feature vector is input into the detection model, so that the detection model processes the feature vector and outputs a status detection result indicating whether the driver's status is normal or abnormal.

[0131] In another embodiment provided by the present application, the third acquisition unit 403 of the system processes the eye movement information based on the abnormal driving detection strategy to obtain the driver's fatigue detection result and attention distraction detection result, including:

[0132] Determining various fatigue detection dimensions and various attention detection dimensions based on the abnormal driving detection strategy;

[0133] For each fatigue detection dimension, extracting fatigue state detection data of the fatigue detection dimension from the eyeball movement information, and generating a first detection result of the fatigue detection dimension based on the fatigue state detection data and the detection strategy of the fatigue detection dimension;

[0134] generating a fatigue detection result indicating whether the driver is in a fatigue state based on each of the first detection results;

[0135] For each of the attention detection dimensions, extracting attention state detection data of the attention detection dimension from the eye movement information, and generating a second detection result of the attention detection dimension based on the attention state detection data and the detection strategy of the attention detection dimension;

[0136] Based on each of the second detection results, a distraction detection result indicating whether the driver is in a distracted state is generated.

[0137] In another embodiment provided in the present application, the fatigue detection dimensions of the system include a blinking frequency detection dimension, an eyeball position change detection dimension in the vertical direction, and a driver's pupil size change rate detection dimension; the attention detection dimensions include a deviation angle detection dimension of the driver's line of sight from the center direction of the road, a stay time of the driver's line of sight in non-road related areas, and a driver's pupil size change rate detection dimension.

[0138] In another embodiment provided by the present application, the third acquisition unit 403 of the system performs a process of generating a fatigue detection result indicating whether the driver is in a fatigue state based on each of the first detection results, including:

[0139] When there is a first detection result indicating that the driver is in a fatigue state among the first detection results, generating a fatigue detection result indicating that the driver is in a fatigue state;

[0140] When all of the first detection results indicate that the driver is not in a fatigue state, a fatigue detection result indicating that the driver is not in a fatigue state is generated.

[0141] In another embodiment provided by the present application, the warning unit 404 of the system executes a process of performing an abnormal driving state warning reminder when the fatigue detection result, the distraction detection result, and each of the state detection results meet a preset abnormal driving state warning condition, including:

[0142] When there is a state detection result indicating that the driver's state is abnormal among the state detection results, a first conclusion indicating that the driver's state is abnormal is generated; when all the state detection results indicate that the driver's state is normal, a first conclusion indicating that the driver's state is normal is generated;

[0143] When the fatigue detection result indicates that the driver is in a fatigued state and / or the distraction detection result indicates that the driver is in a distracted state, generating a second conclusion indicating that the driver's state is abnormal; when the fatigue detection result indicates that the driver is not in a fatigued state and the distraction detection result indicates that the driver is not in a distracted state, generating a second conclusion indicating that the driver's state is normal;

[0144] When one of the first conclusion or the second conclusion indicates that the driver's state is abnormal, determining the fatigue detection result, the distraction detection result, and each of the state detection results, and performing a first-level warning reminder;

[0145] When both the first conclusion and the second conclusion indicate that the driver's state is abnormal, the fatigue detection result, the attention distraction detection result, and each of the state detection results are determined, and a second-level warning reminder is performed.

[0146] An embodiment of the present invention further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned method for warning the driver of abnormal driving status.

[0147] The embodiment of the present invention further provides an electronic device, the structural diagram of which is shown in FIG. Figure 5As shown, the system specifically includes a memory 601 and one or more instructions 602, wherein the one or more instructions 602 are stored in the memory 601 and are configured to be executed by one or more processors 603 to perform the following operations:

[0148] obtaining the driver's eye movement information collected by a calibrated eye tracking sensor;

[0149] Processing the eye movement information using at least one preset detection model to obtain a state detection result output by each detection model;

[0150] Based on the abnormal driving detection strategy, the eye movement information is processed to obtain the driver's fatigue detection result and attention distraction detection result;

[0151] When the fatigue detection result, the attention distraction detection result, and each of the state detection results meet the preset abnormal driving state warning conditions, an abnormal driving state warning reminder is executed.

[0152] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of the relevant regions.

[0153] The specific implementation processes and derivative methods of the above embodiments are all within the protection scope of the present invention.

[0154] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0155] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0156] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for warning a driver of abnormal driving status, characterized in that: include: obtaining the driver's eye movement information collected by a calibrated eye tracking sensor; Processing the eye movement information using at least one preset detection model to obtain a state detection result output by each detection model; Based on the abnormal driving detection strategy, the eye movement information is processed to obtain the driver's fatigue detection result and attention distraction detection result; When the fatigue detection result, the distraction detection result, and each of the state detection results meet preset abnormal driving state warning conditions, an abnormal driving state warning reminder is executed.

2. The method according to claim 1, characterized in that The process of calibrating the eye tracking sensor includes: collecting eye data of the driver and setting sensor parameters of the eye tracking sensor based on the eye data; obtaining current environmental information, adjusting the light intensity in the cockpit where the driver is located based on the environmental information, and collecting eye data of the driver looking at each preset calibration area under various light intensities; The sensing parameters of the eye tracking sensor are adjusted using the eye data.

3. The method according to claim 1, characterized in that The processing of the eye movement information using at least one preset detection model to obtain a state detection result output by each detection model includes: For each of the detection models, based on the feature requirements of the detection model, a feature vector is extracted from the eye movement information, and the feature vector is input into the detection model, so that the detection model processes the feature vector and outputs a status detection result indicating whether the driver's status is normal or abnormal.

4. The method according to claim 1, wherein The abnormal driving detection strategy is based on processing the eye movement information to obtain the driver's fatigue detection result and attention distraction detection result, including: Determining various fatigue detection dimensions and various attention detection dimensions based on the abnormal driving detection strategy; For each fatigue detection dimension, extracting fatigue state detection data of the fatigue detection dimension from the eyeball movement information, and generating a first detection result of the fatigue detection dimension based on the fatigue state detection data and the detection strategy of the fatigue detection dimension; generating a fatigue detection result indicating whether the driver is in a fatigue state based on each of the first detection results; For each of the attention detection dimensions, extracting attention state detection data of the attention detection dimension from the eye movement information, and generating a second detection result of the attention detection dimension based on the attention state detection data and the detection strategy of the attention detection dimension; Based on each of the second detection results, a distraction detection result indicating whether the driver is in a distracted state is generated.

5. The method according to claim 4, characterized in that Each of the fatigue detection dimensions includes a blink frequency detection dimension, an eyeball position change detection dimension in the vertical direction, and a driver's pupil size change rate detection dimension; each of the attention detection dimensions includes a driver's line of sight deviation angle detection dimension from the road center direction, a driver's line of sight stay time in non-road related areas detection dimension, and a driver's pupil size change rate detection dimension.

6. The method according to claim 4, characterized in that The step of generating a fatigue detection result indicating whether the driver is in a fatigue state based on each of the first detection results includes: When there is a first detection result indicating that the driver is in a fatigue state among the first detection results, generating a fatigue detection result indicating that the driver is in a fatigue state; When all of the first detection results indicate that the driver is not in a fatigue state, a fatigue detection result indicating that the driver is not in a fatigue state is generated.

7. The method according to claim 1, characterized in that When the fatigue detection result, the distraction detection result, and each of the state detection results meet the preset abnormal driving state warning conditions, executing the abnormal driving state warning reminder includes: When there is a state detection result indicating that the driver's state is abnormal among the state detection results, a first conclusion indicating that the driver's state is abnormal is generated; when all the state detection results indicate that the driver's state is normal, a first conclusion indicating that the driver's state is normal is generated; When the fatigue detection result indicates that the driver is in a fatigued state and / or the distraction detection result indicates that the driver is in a distracted state, generating a second conclusion indicating that the driver's state is abnormal; when the fatigue detection result indicates that the driver is not in a fatigued state and the distraction detection result indicates that the driver is not in a distracted state, generating a second conclusion indicating that the driver's state is normal; When one of the first conclusion or the second conclusion indicates that the driver's state is abnormal, determining the fatigue detection result, the distraction detection result, and each of the state detection results, and performing a first-level warning reminder; When both the first conclusion and the second conclusion indicate that the driver's state is abnormal, the fatigue detection result, the attention distraction detection result and each of the state detection results are determined, and a second-level warning reminder is performed.

8. A warning system for abnormal driving state of a driver, characterized in that: include: a first acquiring unit, configured to acquire in real time the driver's eye movement information collected by a calibrated eye tracking sensor; A second acquisition unit is configured to process the eye movement information using at least one preset detection model to obtain a state detection result output by each detection model; a third acquiring unit, configured to process the eye movement information based on the abnormal driving detection strategy to acquire a fatigue detection result and a distraction detection result of the driver; The early warning unit is used to execute an abnormal driving state early warning reminder when the fatigue detection result, the attention distraction detection result and each of the state detection results meet the preset abnormal driving state early warning conditions.

9. A storage medium, characterized in that: The storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the method for warning of abnormal driving state of the driver as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes a memory and one or more instructions, wherein the one or more instructions are stored in the memory and are configured to be executed by one or more processors to implement the early warning method for the driver's abnormal driving state as described in any one of claims 1-7.

Images