Drink state detection method, system, equipment and medium

Abstract

The invention discloses a drunkenness state detection method, system and device and a medium, and the method comprises the steps: obtaining the eye data of a driver in a preset time period, and enabling the eye data to comprise eyeball state data; performing time domain feature extraction on the eyeball state data to obtain time domain feature data; performing frequency domain feature extraction on the eyeball state data to obtain frequency domain feature data; fusing the time domain feature data and the frequency domain feature data to obtain eyeball tremor feature information; and based on the eyeball tremor characteristic data, utilizing a neural network model to detect the drunkenness state of the driver. Through fusion of frequency domain and time domain feature information, the eyeball tremor state of the driver is evaluated from multiple features, the robustness of data and the generalization ability of a model can be improved, and the drunk driving state detection precision is improved. And the judgment result can be fused with a drunkenness state detection result based on the physiological feature information of the driver and the vehicle state information, so that the accuracy of deducing the drunkenness state of the driver is further improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle control technology, and in particular to a drunkenness detection method, device, equipment and medium. Background Art

[0002] Drunk driving has become a high-risk factor in traffic accidents, but it is difficult for drivers to implement self-control, external supervision, active safety intervention and other measures while driving. As a result, drivers are prone to collisions, loss of control and other problems when driving after drinking, which poses some danger to the safety of the vehicle and the driver himself, as well as the environment around the vehicle.

[0003] In the existing technology, one way is to determine whether the driver is drunk by adding alcohol detection equipment in the car. Although this method is feasible in theory, in practice it greatly increases the cost of the whole vehicle, which makes many car companies reluctant to adopt it and difficult to popularize it widely. This not only involves the purchase and installation costs of the alcohol detection hardware equipment itself, but also covers a series of expenses such as subsequent maintenance and calibration, which undoubtedly brings huge challenges to vehicle cost control.

[0004] Another way is to use the Driver Monitoring System (DMS) to collect driver information to determine the drunken state. Due to the huge differences in characteristics, habits and postures between different drivers, DMS simply relies on limited dimensional information such as nystagmus status for analysis, which is prone to false alarms and missed reports. At present, the change in the time domain of eye data is usually used to judge the degree of driver's nystagmus in the video stream, thereby confirming whether the driver is in a normal driver state. This will cause misjudgment of detection due to instantaneous state, noise or abnormal fluctuations when processing the driver's eye data. On the one hand, this inaccurate detection result will interfere with the driver's normal driving, frequently trigger unnecessary warnings, and reduce the driving experience; on the other hand, if the drunken state is not accurately identified at a critical moment, effective safety protection measures will not be able to be initiated in time, burying a major hidden danger to vehicle driving safety.

[0005] In addition, the existing drunk detection method on vehicles directly prohibits starting the vehicle when the driver is driving drunk. Simply prohibiting starting the vehicle will not only affect the user experience when there is a misjudgment or the driver has to drive the vehicle, but may also cause harm to the driver's life or property. Summary of the invention

[0006] In view of the shortcomings of the prior art described above, the purpose of the present application is to provide a method, system, device and medium for detecting a drunken state, so as to improve the problem of increased costs caused by adding alcohol detection equipment to vehicles in the prior art, and the problem of insufficient accuracy in detecting a drunken state using a DMS system.

[0007] To achieve the above objectives and other related objectives, the present application provides a drunkenness detection method comprising:

[0008] Acquiring eye data of the driver during a preset period of time, wherein the eye data includes eyeball state data;

[0009] Performing time domain feature extraction on the eyeball state data to obtain time domain feature data;

[0010] Performing frequency domain feature extraction on the eyeball state data to obtain frequency domain feature data;

[0011] fusing the time domain feature data and the frequency domain feature data to obtain nystagmus feature information;

[0012] Based on the nystagmus feature data, a first neural network model is used to detect the drunken state of the driver.

[0013] In an optional embodiment of the present application, the eye data also includes eyelid opening and closing degree data and sight point data;

[0014] Based on the nystagmus feature data, a neural network model is used to detect the drunk state of the driver, including:

[0015] Based on the eyelid opening and closing degree data, the line of sight landing point data and the nystagmus characteristic data, the first neural network model is used to detect the drunkenness state of the driver.

[0016] In an optional embodiment of the present application, the eye state data includes eye position data, eye movement speed data and eye movement acceleration data;

[0017] Extracting time domain features from the eyeball state data to obtain time domain feature data includes:

[0018] Preprocessing the eyeball state data, wherein the preprocessing includes low-pass filtering and smoothing;

[0019] The amplitude, frequency, phase, velocity and acceleration information of the preprocessed eye state data are extracted to obtain the time domain feature data.

[0020] In an optional embodiment of the present application, the eye state data includes eye position data, eye movement speed data and eye movement acceleration data;

[0021] Performing frequency domain feature extraction on the eyeball state data to obtain frequency domain feature data includes:

[0022] Performing frequency domain conversion on the eyeball state data to obtain eyeball state frequency domain data;

[0023] The energy distribution of different frequency components in the eyeball state frequency domain data, the power spectrum density of different frequency components and the frequency band energy ratio within a preset frequency band are extracted to obtain the frequency domain feature data.

[0024] In an optional embodiment of the present application, based on the nystagmus feature data, a first neural network model is used to detect the drunk state of the driver, including:

[0025] Based on the nystagmus feature data and the driver's historical nystagmus feature data, the first neural network model is used to detect the drunken state of the driver.

[0026] To achieve the above objectives and other related objectives, the present application provides a method for detecting drunkenness, comprising:

[0027] Acquiring the driver's eye data, heart rate data and vehicle status data during a preset period; wherein the eye data includes eyeball status data;

[0028] Based on the eyeball state data, using a first neural network model to obtain a first drunkenness confidence level of the driver;

[0029] Based on the heart rate data, obtaining a second drunkenness confidence level of the driver using a second neural network model;

[0030] Based on the vehicle state data, using a third neural network model to further obtain a third drunkenness confidence level of the driver;

[0031] Detecting the drunkenness state of the driver according to the first drunkenness confidence level, the second drunkenness confidence level, and the third drunkenness confidence level;

[0032] Wherein, based on the eyeball state data, using a first neural network model to obtain a first drunk confidence level of the driver includes:

[0033] Extracting time domain features and frequency domain features from the eyeball state data respectively to obtain time domain feature data and frequency domain feature data;

[0034] fusing the time domain feature data and the frequency domain feature data to obtain nystagmus feature information;

[0035] Based on the nystagmus feature data, the first neural network model is used to detect the drunkenness state of the driver to obtain a first drunkenness confidence level of the driver.

[0036] In an optional embodiment of the present application, the drunk state detection of the driver is performed according to the first drunk confidence level, the second drunk confidence level and the third drunk confidence level, including:

[0037] fusing the first drunkenness confidence level, the second drunkenness confidence level, and the third drunkenness confidence level to obtain a drunkenness confidence fusion value;

[0038] Determine whether the drunk confidence fusion value is greater than the drunk threshold:

[0039] If yes, the driver is intoxicated;

[0040] If no, the driver is not intoxicated.

[0041] In an optional embodiment of the present application, the method further includes:

[0042] When the driver is intoxicated and still driving the vehicle, the lane keeping function of the advanced driving assistance system is activated and the sensitivity of the automatic emergency braking function is increased.

[0043] To achieve the above objectives and other related objectives, the present application provides a drunkenness detection system, including:

[0044] The in-vehicle sensing system includes a driver monitoring camera, a capacitive steering wheel, a steering system and an acceleration and braking system. The driver monitoring camera is used to collect the driver's eye data; the capacitive steering wheel is used to collect the driver's heart rate data, and the steering system and the acceleration and braking system are used to collect vehicle status data;

[0045] A controller is respectively connected to the driver monitoring camera, the capacitive steering wheel, the steering system and the acceleration and braking system, and is used to execute any one of the above-mentioned drunkenness detection methods.

[0046] In an optional embodiment of the present application, the drunkenness detection system further includes a feedback system, including an advanced driver assistance system, a communication unit, and an entertainment system;

[0047] The controller is also used to control the feedback system to perform safety precaution operations when it is detected that the driver is in a drunken state.

[0048] To achieve the above-mentioned purpose and other related purposes, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of any of the above-mentioned methods when executing the computer program.

[0049] To achieve the above-mentioned purpose and other related purposes, the present application provides a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0050] The drunk state detection method of the present application obtains the eye data of the driver in a preset time period, wherein the eye data includes eye state data; extracts time domain features from the eye state data to obtain time domain feature data; extracts frequency domain features from the eye state data to obtain frequency domain feature data; fuses the time domain feature data and the frequency domain feature data to obtain nystagmus feature information; and uses a first neural network model to detect the drunk state of the driver based on the nystagmus feature data. By fusing the frequency domain and time domain feature information to evaluate the driver's nystagmus state, the robustness of the data and the generalization ability of the model can be increased to better adapt to different types of nystagmus state detection and improve the accuracy of drunk driving state detection based on eye state data.

[0051] The drunkenness detection method of the present application can utilize the existing hardware in the vehicle to integrate the drunkenness detection results based on different data sources such as eye state data, driver's physiological characteristics information, vehicle state information, etc., to comprehensively analyze and determine whether the driver is in a drunken state by combining multiple drunkenness evaluation mechanisms. This can avoid the misjudgment or omission of drunken behavior by a single system as much as possible, increase the credibility of the system, and further improve the accuracy of the driver's drunkenness state inference.

[0052] The drunk state detection method of the present application can also prevent the driver from driving a vehicle while drunk or reduce the risks in the driving process by presetting a safety prevention strategy when it is detected that the driver is in a drunk state. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A structural block diagram of a drunkenness detection system in an embodiment of the present application is shown;

[0054] Figure 2 A flow chart of a drunkenness detection method based on single system data in an embodiment of the present application is shown;

[0055] Figure 3 A flow chart of a drunkenness detection method based on multi-system data fusion in an embodiment of the present application is shown;

[0056] Figure 4 A flowchart of drunkenness detection based on multi-system data fusion in an exemplary embodiment of the present application is shown;

[0057] Figure 5A structural block diagram of an electronic device according to an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0058] The following describes the implementation of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementations, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application.

[0059] It should be noted that the illustrations provided in this embodiment only illustrate the basic concept of the present application in a schematic manner, and therefore the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0060] like Figure 1 As shown, the present application provides a drunkenness detection system 100 , which includes an in-vehicle sensing system 120 , a controller 110 and a feedback system 130 .

[0061] Among them, the in-vehicle sensing system 120 includes a driver monitoring camera 121 (Driver Monitor Camera, DMC), a capacitive steering wheel 122, a steering system 123 and an acceleration and braking system 124; the driver monitoring camera 121 is used to identify the driver's ID information, and collect eye data such as the driver's eyelid opening and closing degree, line of sight, eyeball status, etc.; the capacitive steering wheel 122 is used to collect the driver's heart rate data, and the steering system 123 and the acceleration and braking system 124 are used to collect vehicle status data such as steering wheel rotation data, brake and accelerator pedal stability data, and turn signal data.

[0062] The controller 110 is respectively connected to the driver monitoring camera 121, the capacitive steering wheel 122, the steering system 123 and the acceleration and braking system 124, and is used to execute the drunken state detection method based on single-system data and the drunken state detection method based on multi-system data fusion introduced below in this application. The controller 110 can be, for example, a driver state monitoring controller (DMS controller), a vehicle controller, a domain controller, etc.

[0063] The feedback system 130 includes an advanced driver assistance system 131 (ADAS), a communication unit 132 and an entertainment system 133. The advanced driver assistance system 131 (ADAS) is a collection of advanced technologies designed to improve driving safety, comfort and convenience. For example, it may have functions such as autonomous emergency braking (AEB), lane departure warning, lane keeping assist, and adaptive cruise control. The controller 110 is also used to control the operation of each unit of the feedback system 130 to perform safety prevention operations when it is detected that the driver is in a drunken state.

[0064] The drunk state detection system 100 of the present application can analyze and fuse the data information collected by multiple hardwares of the vehicle, and can complete the detection of whether the driver is in a drunk state without increasing the hardware cost, so as to increase the application of the function by vehicle manufacturing companies and promote the implementation and landing of the function on a larger scale. By collecting the driver's visual information, tactile information and the collected vehicle status information, the existing collected data and historical data are used to judge and obtain the driver's drunk confidence under each parameter, and the obtained drunk confidence is fused to determine whether the driver is in a drunk state at this time. If the driver is in a drunk state, the driver is reminded in various ways that the current state is not suitable for driving the vehicle, including but not limited to the sound and text reminder of the instrument, the air conditioner blowing cold air and the seat belt vibration, notifying relatives and friends to persuade them to give up driving the vehicle, activating the ADAS function and increasing the sensitivity. The driver is reminded to give up driving the vehicle actively or passively in a gradient manner from a variety of ways, and the danger to the surrounding environment caused by the driver's insistence on driving the vehicle is avoided through active and safe intervention.

[0065] Figure 2 The flowchart of the drunkenness detection method based on single system data in an exemplary embodiment of the present application is shown, including steps S110 to S150, which can be executed by the controller 110 on the vehicle side or the cloud side. The controller 110 on the vehicle side can be, for example, a driver status monitoring controller (DMS controller), a vehicle controller, a domain controller, etc. Figure 2 To elaborate on the implementation process of the drunken state detection method based on single system data.

[0066] First, step S110 is executed to obtain the driver's eye data in a preset period of time, wherein the eye data includes eyeball state data.

[0067] When the driver enters the vehicle, or when the DMS processor receives a specific trigger event, such as the driver requests the DMS processor to activate the drunkenness detection function, the system will begin to obtain the driver's behavior information, and the relevant information includes: the driver's ID information and eye data, among which the driver's ID information is used to subsequently obtain the driver's historical eye data from the database.

[0068] Because the degree of eyelid opening and closing and the variability of vision when a person is drunk are not consistent with those in the normal state, and there is a state of nystagmus. According to long-term statistical results and relevant literature, when a person consumes alcohol, alcohol will paralyze the human nervous system, thereby causing a series of changes in the eyes and vision. Compared with the normal state, the degree of eyelid opening and closing after drinking is significantly reduced. This is because alcohol interferes with the nerve conduction pathway, causing the muscles that control the movement of the eyelids to be unable to operate flexibly, making it difficult for the eyelids to fully open, and the degree of opening and closing is greatly reduced. Not only that, alcohol will also stimulate the brain nerves, causing many problems for the driver's vision, such as blurred vision, double vision, and a smaller field of vision. In this case, when the driver is in a drunken state, the change in the point of sight is significantly reduced compared to the normal state. In order to be able to see the surrounding things clearly, the driver often stares at a fixed position for a long time, trying to make up for the visual defects caused by drunkenness in this way. In addition, alcohol also acts on the vestibular system of the human body. In order to maintain body balance, the brain will cause the eyeballs to be in a state of tremor, that is, the position of the eyeballs will continue to tremble rapidly, in order to offset the interference of alcohol on the body's sense of balance and ensure that the head remains as stable as possible. Therefore, this application uses eye data such as eyelid opening and closing, line of sight, eyeball state, etc. as the basis for determining the drunken state, among which the eyeball state data may include eyeball position, eyeball movement speed, eyeball movement acceleration, etc.

[0069] The drunk state detection system 100 of the present application adopts the DMS system in the actual vehicle as the basic module of the system. The driver's eye data such as eyelid opening and closing, line of sight, eye state, etc. can be obtained through the driver monitoring camera (Driver Monitor Camera, DMC) in the DMS system, and the collected eye data is transmitted to the DMS controller 110 for subsequent processing to obtain the driver's eye nystagmus characteristic information.

[0070] Next, step S120 is executed to extract time domain features from the eye state data to obtain time domain feature data.

[0071] In the step of extracting time domain features from the eye state data to obtain time domain feature data, the eye state data may first be preprocessed by filtering and smoothing to eliminate invalid data and maintain data continuity; then the amplitude, frequency, phase, velocity and acceleration information of the preprocessed eye state data may be extracted to obtain the time domain feature data.

[0072] In a specific embodiment, the eye state data can be filtered through a low-pass filter to remove high-frequency noise and retain useful signals; the eye state data can be smoothed through a smoothing algorithm (such as moving average filtering, weighted moving average filtering) to reduce data fluctuations and make the signal more stable.

[0073] In a specific embodiment, the extraction of time domain features mainly involves extracting the amplitude A of nystagmus, the frequency F of nystagmus, the phase t of nystagmus, the speed v of nystagmus, the acceleration a of nystagmus and other time domain features from the preprocessed eye state data to form time domain feature data, wherein,

[0074] The amplitude A of nystagmus is the maximum displacement of the eye movement: A = |y max -y center |, where A is the amplitude, y max is the maximum value of the signal, y center is the center position of the signal;

[0075] The frequency F of nystagmus is the number of eye movements per unit time: Where F is the frequency of nystagmus, T is the time interval between two adjacent peaks (or peak valleys);

[0076] The phase of nystagmus is the time distribution of the fast phase and the slow phase: t=t_s+t_f, where t is the phase of nystagmus, t_s is the peak time of the signal, and t_f is the valley time of the signal.

[0077] Next, step S130 is executed to extract frequency domain features from the eye state data to obtain frequency domain feature data.

[0078] When using time domain changes to determine the degree of the driver's nystagmus in the video stream and use this to infer the driver's drunken state, misdetection may occur due to instantaneous states, noise or abnormal fluctuations when processing the driver's eye data. Therefore, this application introduces frequency domain feature information and uses periodic signals to improve the credibility of nystagmus detection.

[0079] In the step of extracting frequency domain features from the eye state data to obtain frequency domain feature data, the eye state data can first be converted into frequency domain by discrete Fourier transform, discrete wavelet transform or the like to obtain eye state frequency domain data; then, the energy distribution of different frequency components in the eye state frequency domain data, the power spectrum density of different frequency components and the frequency band energy ratio within a preset frequency band are extracted to obtain the frequency domain feature data.

[0080] Specifically, extracting the energy distribution of different frequency components in the eye state frequency domain data, the power spectrum density of different frequency components, and the frequency band energy ratio within a preset frequency band to obtain the frequency domain feature data may further include:

[0081] The energy distribution of different frequency components is extracted from the eye state frequency domain data. The energy distribution of different frequency components can provide quantitative information about the intensity and characteristics of nystagmus, wherein the energy distribution X[k] of different frequency components k is expressed as follows:

[0082]

[0083] Where k represents the kth frequency component in the frequency domain, N represents the number of points in the frequency domain, n is 0, 1, ..., N-1, and x[n] represents the nth signal state quantity in the frequency discrete domain;

[0084] The power spectrum density of different frequency components is extracted from the eye state frequency domain data. The power spectrum density represents the energy density of different frequency components in the eye state frequency domain data. The energy density of nystagmus in a lower frequency range can be used to determine the cause of the driver's nystagmus at the corresponding frequency. The power spectrum density expression is as follows:

[0085]

[0086] Wherein, k represents the kth frequency component in the frequency domain, PSD[k] represents the power spectral density of the kth frequency component in the frequency domain, |X[k]| represents the modulus of the energy distribution value of the kth frequency component in the frequency domain, and N represents the number of points in the frequency domain.

[0087] The frequency band energy ratio (also called relative energy) within a preset frequency band is extracted from the eye state frequency domain data. The frequency band energy ratio can reflect the energy ratio value within the preset frequency band, which can not only distinguish different types of nystagmus, but also evaluate the severity of nystagmus and detect abnormal frequency components. The frequency band energy ratio R within the preset frequency band is expressed as follows:

[0088]

[0089] In the formula, represents the energy value between f1 and f2, Represents 0 and Wherein, f1 and f2 are the start and end indexes of the preset frequency band, and N represents the number of points in the frequency domain.

[0090] It should be noted that steps S120 and S130 are not limited to the execution order specified in the flowchart, and the order can be interchanged at will, and of course can also be executed simultaneously.

[0091] Next, step S140 is executed to fuse the time domain feature data and the frequency domain feature data to obtain nystagmus feature information.

[0092] In the past, the analysis of eye data was mainly based on the extracted frequency domain feature information to determine the nystagmus state of the eye, but the use of time domain feature information mainly captured the instantaneous changes in eye movement and could not fully feedback the nystagmus state of the driver in the cycle state. This application introduces the fusion of frequency domain feature information and time domain feature information to evaluate the driver's nystagmus state from multiple features, which can increase the robustness of the data and the generalization ability of the model to better adapt to different types of nystagmus state detection and improve the accuracy of drunk driving state detection based on eye state data.

[0093] Specifically, the corresponding time domain feature data A, F, t, v, a and the frequency domain feature data

[0094] X[k], PSD[k], and R are combined into a comprehensive feature vector M:

[0095] M=[A,F,t,v,a,X[k],PSD[k],R]

[0096] Where A represents the amplitude of the eye state data, F represents the frequency of nystagmus, t represents the phase of nystagmus, v represents the speed of eye movement, a represents the acceleration of eye movement, X[k] represents the energy distribution of frequency classification k, PSD[k] represents the power spectral density of frequency component k, and R represents the frequency band energy ratio within the preset frequency band.

[0097] Finally, step S150 is executed to detect the drunken state of the driver using a first neural network model based on the nystagmus feature data, wherein the first neural network may be, for example, a YOLO model or other neural network models.

[0098] It should be noted that in order to eliminate the influence of some special groups on the perception state, when using the first neural network model to detect the driver's drunken state based on the nystagmus feature data, it is necessary to introduce the driver's historical nystagmus feature data as an auxiliary judgment condition. Based on the nystagmus feature data and the driver's historical nystagmus feature data, the first neural network model can be used to compare the driver's current nystagmus feature data and the driver's historical nystagmus feature data to output a first drunkenness confidence level, and the driver's drunkenness state is detected based on the first drunkenness confidence level and the confidence threshold. When the first drunkenness confidence level is greater than the confidence threshold, the driver is in a drunken state; otherwise, the driver is in a non-drunk state.

[0099] It should be noted that in order to improve the accuracy of drunk driving status detection, when using the first neural network model to detect the driver's drunken status based on the nystagmus feature data, the eyelid opening and closing data and the line of sight point data can be added. In other words, the driver's drunken status can be detected based on the eyelid opening and closing data, the line of sight point data and the nystagmus feature data using the first neural network model.

[0100] Similarly, in order to eliminate the influence of some special groups on the perception state, based on the eyelid opening and closing data, the sight point data and the nystagmus characteristic data, when the first neural network model is used to detect the driver's drunken state, the driver's current eyelid opening and closing data, sight point data and nystagmus characteristic data can be compared with the corresponding historical data to obtain the driver's drunkenness probability (first drunkenness confidence), and the driver's drunkenness state is detected based on the first drunkenness confidence and the confidence threshold. When the first drunkenness confidence is greater than the confidence threshold, the driver is in a drunken state; otherwise, the driver is in a non-drunk state.

[0101] When the driver is in a drunk state, the DMS controller 110 will control the feedback system 130 of the vehicle to perform corresponding safety precautions. Since driving a vehicle in a drunk state is an active behavior of the driver, it is difficult for the driver to give up driving the vehicle without external intervention. It is necessary to use system warnings and persuasion from relatives and friends to make the driver aware of the dangers of drunkenness to avoid the driver driving the vehicle in a drunk state. If the driver still insists on driving the vehicle in the above two situations, the relevant system in the car will be activated to improve the driver's sobriety and keep the vehicle safe. Among them, the system warning mainly uses the information obtained by the DMS system to integrate multiple systems to determine that the driver is in a drunk state at this time, and then warns the danger of drunk driving through the instrument pop-up window and voice of the vehicle entertainment system 133. At the same time, the communication unit 132 of the feedback system 130 can also send text messages and phone calls to inform relatives and friends that the driver is driving the vehicle in a drunk state this time, hoping that they will call to persuade them to give up the behavior of driving the vehicle this time. If the above two methods still cannot make the driver give up driving the vehicle this time, the system activates the wake-up mode and activates the air conditioning cold air system to keep the driver awake and drive the vehicle. The ADAS system is triggered synchronously to activate the lane keeping function and increase the sensitivity of the AEB function. When the vehicle is at risk of colliding with static or dynamic surrounding objects, the vehicle will be braked urgently to avoid the risk of the vehicle running out of the lane or colliding with surrounding obstacles during driving, so as to minimize the harm to the vehicle itself and the degree of harm to the surrounding environment. In addition, in this state, the driver is not allowed to actively exit after the AEB function is triggered. The ADAS system needs to judge the state of the surrounding environment before allowing the driver to actively control the driving of the vehicle. This application uses the inherent system in the car to obtain whether the driver is in a drunk driving state to adopt certain strategies, which can minimize the harm to the vehicle itself and the degree of harm to the surrounding environment.

[0102] In this application, in order to further improve the accuracy of driver drunk state detection and avoid the problem that feature judgment extracted by only using single system data (such as the above-mentioned eye data) may lead to false positives or false negatives in some extreme scenarios or for a certain group of people, this application also provides another method such as Figure 3 The drunkenness status detection method based on multi-system data fusion shown includes steps S210 to S250, which can be executed by a controller 110 on the vehicle side or in the cloud. The controller 110 on the vehicle side can be, for example, a driver status monitoring controller (DMS controller), a vehicle computer controller, a domain controller, etc. Figure 4 A specific example of a drunkenness detection method based on multi-system data fusion is given. Figure 3 and Figure 4 To elaborate on the implementation process of the drunken state detection method based on multi-system data fusion.

[0103] First, step S210 is executed to obtain the driver's eye data, heart rate data and vehicle status data during a preset period of time.

[0104] Wherein, the eye data includes eyeball status data.

[0105] When the driver enters the vehicle, or when the DMS processor receives a specific trigger event, such as the driver requests the DMS processor to activate the drunk status detection function, the system will begin to obtain the driver's behavior information, and the relevant information includes: the driver's ID information, eye data, heart rate data, and vehicle status data. Among them, the driver's ID information is used to subsequently obtain the driver's historical eye data, historical heart rate data, and historical vehicle status data from the database; the eye data may include eyelid opening and closing data, line of sight data, and eyeball status data; the heart rate data may include inductive steering wheel data information; the vehicle status data includes steering wheel rotation data, brake and accelerator pedal stability data, and turn signal data, etc.

[0106] Next, step S220 is executed to obtain a first drunkenness confidence level of the driver using a first neural network model based on the eye state data, wherein the first neural network may be, for example, a YOLO model or other neural network models.

[0107] In one embodiment of the present application, based on the eye state data, obtaining the first drunk confidence of the driver using the first neural network model may further include: extracting the time domain features and frequency domain features of the eye state data respectively to obtain time domain feature data and frequency domain feature data; fusing the time domain feature data and the frequency domain feature data to obtain nystagmus feature information; based on the nystagmus feature data, using the first neural network model to detect the drunk state of the driver to obtain the first drunk confidence of the driver. For detailed descriptions, please refer to the descriptions in the above steps S120-150, which will not be repeated here.

[0108] Next, step S230 is executed to obtain a second drunkenness confidence level of the driver using a second neural network model based on the heart rate data, wherein the second neural network may be, for example, a YOLO model or other neural network models.

[0109] When in a drunken state, the effect of alcohol on the driver's sympathetic nerves causes the autonomic nerves of the heart to be suppressed, resulting in a certain variability in the heart rate. In addition, the driver's heartbeat speeds up under the stimulation of alcohol after drinking. Therefore, the capacitive steering wheel 122 can detect the electrical signal between the left and right hands to obtain the driver's heartbeat data within a preset time period (for example, 5 minutes after getting on the car), and the heartbeat data is input into the DMS controller 110. The driver's heartbeat information is analyzed using the second neural network model, and the driver's heartbeat variability, heartbeat change rate and other heartbeat information are compared with the driver's historical heartbeat data to determine the driver's current probability of drunkenness, that is, the second drunkenness confidence level.

[0110] Next, step S240 is executed to obtain a third drunkenness confidence level of the driver using a third neural network model based on the vehicle state data, wherein the third neural network may be, for example, a YOLO model or other neural network models.

[0111] Because the driver may be slow to react or have insufficient risk judgment under the stimulation of alcohol, the driver may slam on the accelerator and brake, and because the driver is confused after drinking, he may frequently turn the steering wheel to ensure the normal driving of the vehicle, or there may be inconsistencies between turning on the turn signal and turning. If the driver has multiple similar behaviors within a period of time, it is considered abnormal driving behavior. The driver collects vehicle status data such as brakes, accelerators, steering wheels, and turn signals during vehicle startup and driving, and transmits the data information obtained within a preset time period (for example, 5 minutes after getting on the car) to the DMS controller 110. The third neural network model is used to analyze the vehicle startup, brake wake-up, and steering and turn signal consistency, and the driver's current probability of drunkenness is analyzed and judged in combination with the driver's historical driving habit information, that is, the third drunkenness confidence.

[0112] It should be noted that steps S220, S230, and S240 are not limited to the execution order specified in the flowchart, and the order can be interchanged at will, and of course, they can also be partially or completely executed synchronously.

[0113] Finally, step S250 is executed to detect the drunkenness state of the driver according to the first drunkenness confidence level, the second drunkenness confidence level and the third drunkenness confidence level.

[0114] In one embodiment, detecting the drunken state of the driver according to the first drunkenness confidence level, the second drunkenness confidence level, and the third drunkenness confidence level may further include:

[0115] fusing the first drunkenness confidence level, the second drunkenness confidence level, and the third drunkenness confidence level to obtain a drunkenness confidence fusion value;

[0116] Determine whether the drunk confidence fusion value is greater than the drunk threshold:

[0117] If yes, the driver is intoxicated;

[0118] If no, the driver is not intoxicated.

[0119] By integrating the drunkenness detection results based on different data sources such as eye state data, driver's physiological characteristics information, vehicle state information: the first drunkenness confidence level, the second drunkenness confidence level and the third drunkenness confidence level, a variety of drunkenness evaluation mechanisms are combined to analyze and determine whether the driver is in a drunken state. This can avoid misjudgment or omission of drunken behavior by a single system, increase the credibility of the system, and further improve the accuracy of inferring the driver's drunkenness state.

[0120] like Figure 4 As shown, when the driver is in a drunk state, the DMS controller will control the vehicle's feedback system 130 to perform corresponding safety precautions. Since driving a vehicle under a drunk state is an active behavior of the driver, it is difficult for the driver to give up driving the vehicle without external intervention. It is necessary to use system warnings and persuasion from relatives and friends to make the driver aware of the dangers of drunkenness to avoid the driver driving the vehicle under a drunken state. If the driver still insists on driving the vehicle in the above two situations, the relevant system in the car will be activated to improve the driver's sobriety and keep the vehicle safe. Among them, the system warning mainly uses the information obtained by the DMS system to integrate multiple systems to determine that the driver is in a drunken state at this time, and then warns the danger of drunk driving through the instrument pop-up window and voice of the vehicle entertainment system 133. At the same time, the communication unit 132 can also send text messages and phone calls to inform relatives and friends that the driver is driving the vehicle under a drunken state this time, hoping that they will call to persuade them to give up the behavior of driving the vehicle this time. If the above two methods still cannot make the driver give up driving the vehicle this time, the system activates the wake-up mode and activates the air conditioning cold air system to keep the driver sober and drive the vehicle. The ADAS system is triggered synchronously to activate the lane keeping function and increase the sensitivity of the AEB function. When the vehicle is at risk of colliding with static or dynamic surrounding objects, the vehicle will be braked urgently to avoid the risk of the vehicle running out of the lane or colliding with surrounding obstacles during driving, so as to minimize the harm to the vehicle itself and the degree of harm to the surrounding environment. In addition, in this state, the driver is not allowed to actively exit after the AEB function is triggered. The ADAS system needs to judge the state of the surrounding environment before allowing the driver to actively control the driving of the vehicle. This application uses the inherent system in the car to obtain whether the driver is in a drunk driving state to adopt certain strategies, which can minimize the harm to the vehicle itself and the degree of harm to the surrounding environment.

[0121] In summary, the drunk state detection method of the present application obtains the eye data of the driver in a preset time period, wherein the eye data includes eye state data; performs time domain feature extraction on the eye state data to obtain time domain feature data; performs frequency domain feature extraction on the eye state data to obtain frequency domain feature data; fuses the time domain feature data and the frequency domain feature data to obtain nystagmus feature information; and uses a first neural network model to detect the drunk state of the driver based on the nystagmus feature data. By fusing the frequency domain and time domain feature information to evaluate the driver's nystagmus state, the robustness of the data and the generalization ability of the model can be increased to better adapt to different types of nystagmus state detection and improve the accuracy of drunk driving state detection based on eye state data. The drunk state detection method of the present application can utilize the existing hardware in the vehicle to obtain data from different data sources such as the driver's eye state data, the driver's physiological characteristics information, and the vehicle state information within a preset time period, and utilize the corresponding drunk evaluation mechanism to summarize, analyze and process the data from different data sources to obtain the probability that the driver's current state corresponding to the different data sources is in a drunk state (i.e., drunk confidence), and merge these results to comprehensively analyze and determine whether the driver is in a drunk state using multiple drunk evaluation mechanisms, which can avoid the misjudgment or missed judgment of drunk behavior by a single system as much as possible, increase the credibility of the system, and further improve the accuracy of the driver's drunk state inference. The drunk state detection method of the present application can also, when detecting that the driver is in a drunk state, prevent the driver from driving a vehicle while drunk or reduce the risk of driving a vehicle by presetting a safety prevention strategy.

[0122] like Figure 5 The figure is a schematic diagram of the structure of an electronic device for implementing the drunken state detection method of the present application.

[0123] The electronic device 1 may include a memory 11, a processor 12 and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 12, such as a drunkenness detection program.

[0124] Among them, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 11 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code for drunken state detection, etc., but also can be used to temporarily store data that has been output or is to be output.

[0125] In some embodiments, the processor 12 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 12 is the control core (Control Unit) of the electronic device 1, and uses various interfaces and lines to connect various components of the entire electronic device 1, and executes or executes programs or modules (such as drunkenness detection programs, etc.) stored in the memory 11, and calls data stored in the memory 11 to execute various functions of the electronic device 1 and process data.

[0126] The processor 12 executes the operating system and various installed applications of the electronic device 1. The processor 12 executes the applications to implement the steps in the above-mentioned drunkenness detection method.

[0127] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 11 and executed by the processor 12 to complete the drunken state detection program of the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 1.

[0128] In the description herein, many specific details are provided, such as examples of parts and / or methods, to provide a complete understanding of the embodiments of the present application. However, those skilled in the art will recognize that the embodiments of the present application can be practiced without one or more specific details or by other devices, systems, components, methods, parts, materials, parts, etc. In other cases, known structures, materials or operations are not specifically shown or described in detail to avoid blurring the content of the embodiments of the present application.

[0129] The above description of the embodiment shown in the application (including the content described in the abstract of the specification) is not intended to be exhaustive or to limit the application to the precise form disclosed herein. Although the specific embodiment of the application and the example of the application are described herein for the purpose of illustration only, as those skilled in the art will recognize and understand, various equivalent modifications can be within the spirit and scope of the present invention. As pointed out, these modifications can be made to the application according to the above description of the embodiment described in the application, and these modifications will be within the spirit and scope of the application.

Claims

1. A method for detecting drunkenness, characterized in that: include: Acquiring eye data of the driver during a preset period of time, wherein the eye data includes eyeball state data; Performing time domain feature extraction on the eyeball state data to obtain time domain feature data; Performing frequency domain feature extraction on the eyeball state data to obtain frequency domain feature data; fusing the time domain feature data and the frequency domain feature data to obtain nystagmus feature information; Based on the nystagmus feature data, the drunken state of the driver is detected using a first neural network model.

2. The drunkenness detection method according to claim 1, characterized in that: The eye data also includes eyelid opening and closing degree data and sight point data; Based on the nystagmus feature data, a neural network model is used to detect the drunk state of the driver, including: Based on the eyelid opening and closing degree data, the line of sight landing point data and the nystagmus characteristic data, the first neural network model is used to detect the drunkenness state of the driver.

3. The drunkenness detection method according to claim 1, characterized in that: The eye state data includes eye position data, eye movement speed data and eye movement acceleration data; Extracting time domain features from the eyeball state data to obtain time domain feature data includes: Preprocessing the eyeball state data, wherein the preprocessing includes low-pass filtering and smoothing; The amplitude, frequency, phase, velocity and acceleration information of the preprocessed eye state data are extracted to obtain the time domain feature data.

4. The drunkenness detection method according to claim 1, characterized in that: The eye state data includes eye position data, eye movement speed data and eye movement acceleration data; Performing frequency domain feature extraction on the eyeball state data to obtain frequency domain feature data includes: Performing frequency domain conversion on the eyeball state data to obtain eyeball state frequency domain data; The energy distribution of different frequency components in the eyeball state frequency domain data, the power spectrum density of different frequency components and the frequency band energy ratio within a preset frequency band are extracted to obtain the frequency domain feature data.

5. The drunkenness detection method according to claim 1, characterized in that: Based on the nystagmus feature data, a first neural network model is used to detect the drunk state of the driver, including: Based on the nystagmus feature data and the driver's historical nystagmus feature data, the first neural network model is used to detect the drunken state of the driver.

6. A method for detecting drunkenness, characterized in that: include: Acquiring the driver's eye data, heart rate data and vehicle status data during a preset period; wherein the eye data includes eyeball status data; Based on the eyeball state data, using a first neural network model to obtain a first drunkenness confidence level of the driver; Based on the heart rate data, obtaining a second drunkenness confidence level of the driver using a second neural network model; Based on the vehicle state data, using a third neural network model to further obtain a third drunkenness confidence level of the driver; Detecting the drunkenness state of the driver according to the first drunkenness confidence level, the second drunkenness confidence level, and the third drunkenness confidence level; Wherein, based on the eyeball state data, using a first neural network model to obtain a first drunk confidence level of the driver includes: Extracting time domain features and frequency domain features from the eyeball state data respectively to obtain time domain feature data and frequency domain feature data; fusing the time domain feature data and the frequency domain feature data to obtain nystagmus feature information; Based on the nystagmus feature data, a first neural network model is used to detect the drunkenness state of the driver to obtain a first drunkenness confidence level of the driver.

7. The drunkenness detection method according to claim 6, characterized in that: Detecting the drunken state of the driver according to the first drunkenness confidence level, the second drunkenness confidence level, and the third drunkenness confidence level includes: fusing the first drunkenness confidence level, the second drunkenness confidence level, and the third drunkenness confidence level to obtain a drunkenness confidence fusion value; Determine whether the drunk confidence fusion value is greater than the drunk threshold: If yes, the driver is intoxicated; If no, the driver is not intoxicated.

8. The drunkenness detection method according to any one of claims 1 to 7, characterized in that: The method further comprises: When the driver is intoxicated and still driving the vehicle, the lane keeping function of the advanced driving assistance system is activated and the sensitivity of the automatic emergency braking function is increased.

9. A drunkenness detection system, characterized in that: include: An in-vehicle sensing system, including a driver monitoring camera, a capacitive steering wheel, a steering system, and an acceleration and braking system. The driver monitoring camera is used to collect the driver's eye data; The capacitive steering wheel is used to collect the driver's heart rate data, and the steering system and the acceleration and braking system are used to collect vehicle status data; A controller is respectively connected to the driver monitoring camera, the capacitive steering wheel, the steering system and the acceleration and braking system, and is used to execute the drunken state detection method described in any one of claims 1-8.

10. The drunkenness detection system according to claim 9, characterized in that: The drunkenness detection system also includes a feedback system, including an advanced driver assistance system, a communication unit, and an entertainment system; The controller is also used to control the feedback system to perform safety precaution operations when it is detected that the driver is in a drunken state.

11. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 8 when executing the computer program.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images