Driver state monitoring method, system and device based on machine vision and medium

Abstract

The invention discloses a driver state monitoring method, system and device based on machine vision and a medium. The method comprises the steps that displacement time sequence data of a brake pedal, rotation angle time sequence data of a steering wheel and image time sequence data of a road in front of a vehicle are obtained; extracting emergency braking time sequence characteristics of a brake pedal according to the displacement time sequence data, extracting emergency steering time sequence characteristics of a steering wheel according to the turning angle time sequence data, and extracting direction time sequence characteristics of an obstacle in front of the vehicle according to the road image time sequence data; inputting the emergency braking time sequence characteristics, the emergency steering time sequence characteristics and the obstacle orientation time sequence characteristics into a pre-trained driver state recognition model to obtain a current driver state; and early warning the driver according to the current driver state. The method improves the accuracy and timeliness of driver state monitoring, also improves the driving safety, and can be applied to the technical field of vehicle monitoring.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle monitoring, and in particular to a driver state monitoring method, system and device based on machine vision and a medium. BACKGROUND

[0002] During driving, the driver may be tired or inattentive. The mainstream technology currently used is to collect driver facial data through a camera and analyze the driver state. However, this method is easily affected by the light in the vehicle and the driver's actions (such as turning his head or lowering his head), resulting in poor accuracy and timeliness of driver state monitoring. SUMMARY

[0003] The present application aims to at least partially solve one of the problems in the prior art.

[0004] To this end, one object of the present application is to provide a driver state monitoring method based on machine vision, which improves the accuracy and timeliness of driver state monitoring and also improves driving safety.

[0005] Another object of the present application is to provide a driver state monitoring system based on machine vision.

[0006] To achieve the above technical objects, the technical solutions adopted by the embodiments of the present application include:

[0007] In a first aspect, the present application provides a driver state monitoring method based on machine vision, comprising the following steps:

[0008] obtaining displacement time series data of a brake pedal, rotation angle time series data of a steering wheel and road image time series data in front of the vehicle;

[0009] extracting emergency braking time series features of the brake pedal according to the displacement time series data, extracting emergency steering time series features of the steering wheel according to the rotation angle time series data, and extracting obstacle orientation time series features in front of the vehicle according to the road image time series data;

[0010] inputting the emergency braking time series features, the emergency steering time series features and the obstacle orientation time series features into a pre-trained driver state recognition model to obtain a current driver state;

[0011] warning the driver according to the current driver state.

[0012] Further, in one embodiment of the present application, the obtaining of the displacement time series data of the brake pedal, the rotation angle time series data of the steering wheel and the road image time series data in front of the vehicle specifically includes:

[0013] the displacement time-series data is acquired by a displacement sensor arranged on the brake pedal;

[0014] the rotation angle time-series data is acquired by a rotation angle sensor arranged on the rotation shaft of the steering wheel;

[0015] the road image time-series data is acquired by a camera arranged in front of the vehicle.

[0016] Further, in an embodiment of the present application, the extracting according to the displacement time-series data extracts an emergency braking time-series feature of the brake pedal, and the extracting according to the rotation angle time-series data extracts an emergency steering time-series feature of the steering wheel, which specifically comprises:

[0017] performing short-time Fourier transform on the displacement time-series data to obtain first frequency domain information, and extracting first key frequency domain information from the first frequency domain information according to a preset first frequency band region;

[0018] performing inverse Fourier transform on the first key frequency domain information to obtain a first time-series signal, and extracting an instantaneous displacement change rate of the first time-series signal to obtain the emergency braking time-series feature;

[0019] performing short-time Fourier transform on the rotation angle time-series data to obtain second frequency domain information, and extracting second key frequency domain information from the second frequency domain information according to a preset second frequency band region;

[0020] performing inverse Fourier transform on the second key frequency domain information to obtain a second time-series signal, and extracting an instantaneous rotation angle change rate of the second time-series signal to obtain the emergency steering time-series feature.

[0021] Further, in an embodiment of the present application, the extracting according to the road image time-series data extracts an obstacle azimuth time-series feature in front of the vehicle, which specifically comprises:

[0022] determining a plurality of continuous road image information according to the road image time-series data;

[0023] performing obstacle detection on the road image information to obtain an obstacle direction and an obstacle distance, and determining an obstacle azimuth feature according to the obstacle direction and the obstacle distance;

[0024] generating the obstacle azimuth time-series feature according to the obstacle azimuth features corresponding to the plurality of continuous road image information.

[0025] Further, in an embodiment of the present application, the driver state recognition model is obtained by training through the following steps:

[0026] obtain brake pedal displacement time sequence samples, steering wheel rotation angle time sequence samples and front road image time sequence samples in a historical driving scene;

[0027] extract emergency braking time sequence feature samples according to the brake pedal displacement time sequence samples, extract emergency steering time sequence feature samples according to the steering wheel rotation angle time sequence samples, and extract obstacle orientation time sequence feature samples according to the front road image time sequence samples;

[0028] construct training samples according to the emergency braking time sequence feature samples, the emergency steering time sequence feature samples and the obstacle orientation time sequence feature samples, determine driver state labels corresponding to the training samples through artificial labeling, construct a training data set according to the training samples and the driver state labels;

[0029] input the training data set into a pre-constructed bidirectional long short-term memory network for training to obtain a trained driver state recognition model;

[0030] The driver state labels include fatigue degree labels and attention concentration degree labels.

[0031] Further, in an embodiment of the present application, the bidirectional long short-term memory network includes an input layer, a forward LSTM layer, a backward LSTM layer and an output layer, and the inputting of the training data set into the pre-constructed bidirectional long short-term memory network for training to obtain the trained driver state recognition model specifically includes:

[0032] input the training samples into the input layer and calculate a forward hidden state vector through the forward LSTM layer;

[0033] input the training samples into the input layer in reverse order and calculate a backward hidden state vector through the backward LSTM layer;

[0034] perform reverse order processing on the backward hidden state vector and splice the processed backward hidden state vector with the forward hidden state vector to obtain a target hidden state vector;

[0035] input the target hidden state vector into the output layer to output a driver state recognition result;

[0036] determine a loss value according to the driver state recognition result and the driver state labels;

[0037] update parameters of the bidirectional long short-term memory network according to the loss value to obtain the trained driver state recognition model.

[0038] Further, in one embodiment of the present application, the warning the driver according to the current driver state specifically comprises:

[0039] determining whether the driver is in fatigue driving or inattentive driving according to the current driver state;

[0040] warning the driver according to preset first prompt information when the driver is in fatigue driving or inattentive driving.

[0041] In a second aspect, the embodiments of the present application provide a driver state monitoring system based on machine vision, comprising:

[0042] a data acquisition module configured to acquire displacement time sequence data of a brake pedal, rotation angle time sequence data of a steering wheel, and road image time sequence data in front of a vehicle;

[0043] a feature extraction module configured to extract emergency braking time sequence features of the brake pedal according to the displacement time sequence data, extract emergency steering time sequence features of the steering wheel according to the rotation angle time sequence data, and extract obstacle direction time sequence features in front of the vehicle according to the road image time sequence data;

[0044] a state recognition module configured to input the emergency braking time sequence features, the emergency steering time sequence features, and the obstacle direction time sequence features into a pre-trained driver state recognition model to obtain a current driver state;

[0045] a warning module configured to warn the driver according to the current driver state.

[0046] In a third aspect, the embodiments of the present application provide a driver state monitoring device based on machine vision, comprising:

[0047] at least one processor;

[0048] at least one memory configured to store at least one program;

[0049] when the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned driver state monitoring method based on machine vision.

[0050] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, wherein a program executable by a processor is stored, and the program executable by the processor is used to execute the above-mentioned driver state monitoring method based on machine vision when executed by the processor.

[0051] The advantages and beneficial effects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood through the practice of the present application:

[0052] The embodiment of the present application obtains displacement time sequence data of a brake pedal, rotation angle time sequence data of a steering wheel and road image time sequence data in front of a vehicle, extracts emergency braking time sequence features of the brake pedal according to the displacement time sequence data, extracts emergency steering time sequence features of the steering wheel according to the rotation angle time sequence data, and extracts obstacle direction time sequence features in front of the vehicle according to the road image time sequence data, inputs the emergency braking time sequence features, the emergency steering time sequence features and the obstacle direction time sequence features into a pre-trained driver state recognition model to obtain a current driver state, and pre-warns the driver according to the current driver state. The embodiment of the present application extracts the emergency braking time sequence features according to the displacement time sequence data of the brake pedal, extracts the emergency steering time sequence features according to the rotation angle time sequence data of the steering wheel, extracts the obstacle direction time sequence features according to the road image time sequence data in front of the vehicle, infers whether the driver appears emergency avoidance behavior due to fatigue or inattention based on the pre-trained driver state recognition model, thereby recognizing the current driver state, overcomes the defect that the existing technology needs to collect driver facial images which are easily affected by the light in the vehicle and the driver's actions, improves the accuracy and timeliness of the driver state monitoring, and improves the driving safety. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following introduces the drawings needed to be used in the embodiments of the present application. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing some embodiments in the technical solutions of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0054] Figure 1 A step flow chart of a driver state monitoring method based on machine vision provided by the embodiment of the present application is shown in the figure.

[0055] Figure 2 A structural block diagram of a driver state monitoring system based on machine vision provided by the embodiment of the present application is shown in the figure.

[0056] Figure 3 A structural block diagram of a driver state monitoring device based on machine vision provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0057] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0058] In the description of this invention, "multiple" means two or more. The use of "first" and "second" is for distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or the order of the indicated technical features. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art.

[0059] Reference Figure 1 This invention provides a driver state monitoring method based on machine vision, specifically including the following steps:

[0060] S101. Acquire the timing data of brake pedal displacement, steering wheel angle, and road image in front of the vehicle.

[0061] S102. Extract the emergency braking timing features of the brake pedal based on the displacement timing data, extract the emergency steering timing features of the steering wheel based on the turning angle timing data, and extract the location timing features of obstacles in front of the vehicle based on the road image timing data.

[0062] S103. Input the emergency braking timing features, emergency steering timing features, and obstacle orientation timing features into the pre-trained driver state recognition model to obtain the current driver state.

[0063] S104. Issue a warning to the driver based on the current driver status.

[0064] Specifically, when the driver is fatigued or inattentive, it is particularly prone to the situation that the front obstacle is not timely discovered and needs to be avoided in an emergency, such as avoiding vehicles, pedestrians, roadblocks, etc., which is manifested in the data in the vehicle as emergency steering and emergency braking. The embodiments of the present application infer the driver state by analyzing the displacement time sequence data of the brake pedal, the steering angle time sequence data of the steering wheel, and the road image time sequence data in front of the vehicle. Specifically, the embodiments of the present application extract the emergency braking time sequence feature according to the displacement time sequence data of the brake pedal, extract the emergency steering time sequence feature according to the steering angle time sequence data of the steering wheel, and extract the obstacle direction time sequence feature according to the road image time sequence data in front of the vehicle. Based on the pre-trained driver state recognition model, it is inferred whether the driver appears emergency avoidance behavior due to fatigue or inattention, so as to recognize the current driver state, overcome the defect that the existing technology needs to collect driver facial images which are easily affected by the light in the vehicle and the driver's actions, improve the accuracy and timeliness of driver state monitoring, and also improve the driving safety.

[0065] Further as an optional implementation, the displacement time sequence data of the brake pedal, the steering angle time sequence data of the steering wheel, and the road image time sequence data in front of the vehicle are obtained, which specifically include:

[0066] S1011, acquiring displacement time sequence data through a displacement sensor arranged on the brake pedal;

[0067] S1012, acquiring steering angle time sequence data through a steering angle sensor arranged on the steering shaft of the steering wheel;

[0068] S1013, acquiring road image time sequence data through a camera device arranged in front of the vehicle.

[0069] Further as an optional implementation, the emergency braking time sequence feature of the brake pedal is extracted according to the displacement time sequence data, and the emergency steering time sequence feature of the steering wheel is extracted according to the steering angle time sequence data, which specifically include:

[0070] S1021, performing short-time Fourier transform on the displacement time sequence data to obtain first frequency domain information, and extracting first key frequency domain information from the first frequency domain information according to a preset first frequency band region;

[0071] S1022, performing inverse Fourier transform on the first key frequency domain information to obtain a first time sequence signal, and extracting an instantaneous displacement change rate of the first time sequence signal to obtain an emergency braking time sequence feature;

[0072] S1023, performing short-time Fourier transform on the steering angle time sequence data to obtain second frequency domain information, and extracting second key frequency domain information from the second frequency domain information according to a preset second frequency band region;

[0073] S1024, inverse Fourier transform of the second key frequency domain information to obtain the second time sequence signal, and extracting the instantaneous corner change rate of the second time sequence signal to obtain the emergency turning time sequence feature.

[0074] Specifically, the extraction process of the emergency braking time sequence feature is as follows:

[0075] 1) Short-time Fourier transform (STFT) to obtain the first frequency domain information:

[0076] Input: displacement time series data (i.e., time series data, representing the displacement change of the object, obtained by a displacement sensor arranged on the brake pedal).

[0077] Processing: Apply STFT to convert the time domain signal to a time-frequency domain representation. STFT divides the signal into short-time windows (such as Hamming window or Hanning window) and performs Fourier transform on each window to obtain a matrix of frequency components changing with time (called time-frequency spectrum).

[0078] Output: The first frequency domain information is a complex matrix with dimensions (frequency points x time frames). Each element represents the amplitude and phase at a specific time and frequency.

[0079] Purpose: The displacement signal is usually non-stationary (frequency content changes over time) during emergency braking, and STFT can capture such transient characteristics, such as specific frequency vibrations caused by braking (such as high-frequency impact components).

[0080] 2) Extract the first key frequency domain information:

[0081] Processing: Extract a subset from the first frequency domain information according to a pre-set first frequency band region (such as 50-200Hz). This is equivalent to applying a band-pass filter on the time-frequency spectrum, only retaining the frequency band related to emergency braking. For example, if the first frequency band region is set as [f_min, f_max], only the rows (frequency points) within this frequency range are retained, and other frequency components are set to zero or ignored. In the vehicle braking system, the emergency braking frequency band may be 50-150Hz (corresponding to brake disc vibration), which can be determined by historical data spectrum analysis (such as FFT or STFT average spectrum).

[0082] Purpose: Emergency braking events often exhibit significant characteristics in specific frequency bands (such as brake disc vibration or vehicle body resonance frequency band), and extracting key frequency bands can remove noise and irrelevant components, improving the signal-to-noise ratio of the feature.

[0083] 3) Inverse Fourier transform (IFFT) to obtain the first time sequence signal:

[0084] Processing: Inverse Fourier Transform (IFFT) is applied to the first frequency domain information to reconstruct the time domain signal. Since the input is a subset of STFT (only key frequency bands), the IFFT will generate a filtered time domain signal, called the first time series signal. STFT is usually processed in frames, so window overlap and window consistency should be ensured during IFFT to avoid boundary effects (such as using overlap-add method).

[0085] Objective: Convert the key frequency band information back to the time domain to obtain a "pure" displacement signal that only contains dynamic components related to emergency braking (such as vibration waveforms during braking).

[0086] 4) Extract the instantaneous displacement rate of change as the emergency braking time series feature:

[0087] Processing: Calculate the instantaneous displacement rate of change from the first time series signal, which can be obtained by numerical differentiation.

[0088] Output: Emergency braking time series feature, a time series representing the instantaneous rate of change of the filtered displacement signal.

[0089] Objective: During emergency braking, the displacement rate of change (speed) often experiences a sharp decline or abnormal fluctuation (such as sudden speed drop accompanied by high-frequency jitter). This feature can directly reflect the braking intensity and time point, and is used for event detection or classification.

[0090] It can be recognized that emergency braking events often trigger resonance in specific frequency bands (such as the natural frequency of the vehicle body or mechanical structure), and the pre-set first frequency band region can extract these features specifically, avoiding low-frequency noise (such as road bumps) or high-frequency interference. By reconstructing the signal through IFFT, the time domain dynamics of the key frequency band are preserved, making the rate of change calculation more sensitive to braking events. The instantaneous displacement rate of change is directly related to acceleration or deceleration, and in emergency braking analysis, the sudden change in deceleration is a core indicator (for example, in vehicle braking systems, deceleration exceeding a threshold is considered emergency braking).

[0091] The extraction process of the emergency steering time series feature is as follows:

[0092] 1) Short-time Fourier Transform (STFT) to obtain second frequency domain information:

[0093] Input: Steering angle time series data (collected by steering wheel angle sensor).

[0094] Processing: Segment the signal through sliding window (commonly use Hamming or Hanning window), and perform Fourier transform on each segment to obtain the second frequency domain information (time-frequency matrix).

[0095] Key parameters: (1) Window length: affects time-frequency resolution. Longer window, higher frequency resolution, but lower time resolution; shorter window, vice versa; (2) Overlap rate: reduces boundary effects caused by segmentation, improves continuity.

[0096] Objective: Capture the transient frequency characteristics of non-stationary signals in emergency steering (such as steering mechanism resonance frequency).

[0097] 2) Extract the second key frequency domain information:

[0098] Pre-set second frequency band region: set according to the physical characteristics of the steering system (for example, 5-20Hz, corresponding to the mechanical response frequency band). The second frequency band needs to be determined in combination with the vehicle dynamics model or experimental data. For example: low frequency band (1-5Hz): steering wheel hand force input frequency; medium-high frequency band (10-30Hz): steering column vibration or motor response frequency.

[0099] Operation: In the time-frequency matrix, only the data of the target frequency band is retained, and the other frequency bands are set to zero, which is equivalent to time-frequency domain band-pass filtering.

[0100] Objective: Focus on the frequency band strongly related to emergency steering (such as steering assist motor vibration, tire lateral force mutation frequency).

[0101] 3) Inverse Fourier transform to reconstruct the time domain signal:

[0102] Inverse Fourier transform (ISTFT) of the key frequency domain information to obtain the second time sequence signal.

[0103] Output characteristics: retain the components of the original signal in the target frequency band, and filter out irrelevant noise (such as road bump interference). The reconstructed signal highlights the transient characteristics of emergency steering (such as steering angle acceleration mutation).

[0104] 3) Extract the instantaneous steering angle change rate

[0105] Calculation method: first-order derivative of the second time sequence signal.

[0106] Emergency steering time sequence characteristics: instantaneous change rate peak value identifies the steering urgency (such as rapid steering when avoiding obstacles); high sustained value of change rate may indicate continuous emergency operation (such as S-type avoidance).

[0107] Further as an optional implementation, the orientation time sequence characteristics of the obstacle in front of the vehicle are obtained according to the road image time sequence data, which specifically includes:

[0108] S1025, determining a plurality of continuous road image information according to the road image time sequence data;

[0109] S1026, obstacle direction and obstacle distance are obtained by performing obstacle detection on the road image information, and the obstacle position feature is determined according to the obstacle direction and the obstacle distance;

[0110] S1027, the obstacle position time sequence feature is generated according to the obstacle position features corresponding to the plurality of continuous road image information.

[0111] Specifically, according to the road image time sequence data, a plurality of continuous road image information is determined, the road image information is input into the obstacle detection model to obtain the direction of the obstacle relative to the vehicle and the distance from the vehicle, thereby determining the obstacle position feature, and the obstacle position features corresponding to the plurality of continuous road image information are sequentially processed to generate the obstacle position time sequence feature.

[0112] Further, as an optional implementation, the driver state recognition model is obtained by the following steps:

[0113] S201, brake pedal displacement time sequence samples, steering wheel angle time sequence samples and front road image time sequence samples in a historical driving scene are obtained;

[0114] S202, emergency braking time sequence feature samples are extracted according to the brake pedal displacement time sequence samples, emergency steering time sequence feature samples are extracted according to the steering wheel angle time sequence samples, and obstacle position time sequence feature samples are extracted according to the front road image time sequence samples;

[0115] S203, training samples are constructed according to the emergency braking time sequence feature samples, the emergency steering time sequence feature samples and the obstacle position time sequence feature samples, the driver state labels corresponding to the training samples are determined by artificial labeling, and a training data set is constructed according to the training samples and the driver state labels;

[0116] S204, the training data set is input into the pre-constructed bidirectional long short-term memory network for training to obtain the trained driver state recognition model;

[0117] The driver state labels include fatigue degree labels and attention concentration degree labels.

[0118] Specifically, the brake pedal displacement time sequence samples, the steering wheel angle time sequence samples and the front road image time sequence samples in the historical driving scene are obtained (pre-obtained user authorization), the emergency braking time sequence feature samples are extracted according to the brake pedal displacement time sequence samples, the emergency steering time sequence feature samples are extracted according to the steering wheel angle time sequence samples, and the obstacle position time sequence feature samples are extracted according to the front road image time sequence samples, and whether the user has fatigue driving or inattention is determined by artificial analysis when the vehicle performs emergency avoidance behavior in the sample, thereby determining the fatigue degree label and the attention concentration degree label.

[0119] Further as an optional implementation, the bidirectional long short-term memory network comprises an input layer, a forward LSTM layer, a backward LSTM layer and an output layer, the training data set is input to the pre-constructed bidirectional long short-term memory network for training to obtain a trained driver state recognition model, which specifically comprises:

[0120] S2041, input the training sample to the input layer, and calculate the forward hidden state vector through the forward LSTM layer;

[0121] S2042, input the training sample in reverse order to the input layer, and calculate the backward hidden state vector through the backward LSTM layer;

[0122] S2043, after the reverse processing of the backward hidden state vector, the forward hidden state vector is spliced to obtain a target hidden state vector;

[0123] S2044, input the target hidden state vector to the output layer, and output to obtain a driver state recognition result;

[0124] S2045, determine a loss value according to the driver state recognition result and the driver state label;

[0125] S2046, update the parameters of the bidirectional long short-term memory network according to the loss value to obtain a trained driver state recognition model.

[0126] Specifically, the calculation process of the forward LSTM layer and the backward LSTM layer is as follows:

[0127]

[0128] wherein, and respectively represent the forward hidden state vector and the backward hidden state vector, and respectively represent the forward calculation function and the backward calculation function, represents the input variable data, represents the target hidden state vector.

[0129] The target hidden state vector is input to the output layer, and a driver state recognition result is output. A loss value is calculated based on a preset loss function according to the driver state recognition result and the driver state label. The parameters of the bidirectional long short-term memory network are updated according to the loss value, and then the next round of iteration training is entered until a preset convergence condition (the loss value is lower than a preset threshold, the number of training times reaches a preset number) is reached, that is, a trained driver state recognition model is obtained.

[0130] Further as an optional implementation, the driver is warned according to the current driver state, which specifically includes:

[0131] S1041, judging whether the driver has fatigue driving or inattentive driving according to the current driver state;

[0132] S1042, warning the driver according to the preset first prompt information when the driver has fatigue driving or inattentive driving.

[0133] Specifically, the emergency braking timing feature, the emergency steering timing feature and the obstacle orientation timing feature extracted in the foregoing steps are input to the pre-trained driver state recognition model, so that the current driver state is obtained; whether the driver has fatigue driving or inattentive driving is judged according to the current driver state, and the driver is warned according to the preset first prompt information when the driver has fatigue driving or inattentive driving. In addition, when the driver has serious fatigue driving, the vehicle can be directly controlled to slow down and remind the driver to stop and rest.

[0134] The method steps of the embodiments of the application are described above. It can be recognized that the embodiments of the application extract the emergency braking timing feature according to the displacement timing data of the brake pedal, extract the emergency steering timing feature according to the rotation angle timing data of the steering wheel, extract the obstacle orientation timing feature according to the road image timing data in front of the vehicle, and infer whether the driver has the emergency avoidance behavior due to fatigue or inattentive driving based on the pre-trained driver state recognition model, so as to recognize the current driver state, overcome the defects that the existing technology needs to collect the driver's face image which is easily affected by the light in the vehicle and the driver's action, improve the accuracy and timeliness of the driver state monitoring, and also improve the driving safety.

[0135] Referring to Figure 2 , the embodiments of the application provide a driver state monitoring system based on machine vision, which comprises:

[0136] A data acquisition module is configured to acquire displacement timing data of a brake pedal, rotation angle timing data of a steering wheel and road image timing data in front of a vehicle;

[0137] A feature extraction module is configured to extract an emergency braking timing feature of the brake pedal according to the displacement timing data, extract an emergency steering timing feature of the steering wheel according to the rotation angle timing data, and extract an obstacle orientation timing feature in front of the vehicle according to the road image timing data;

[0138] A state recognition module is configured to input the emergency braking timing feature, the emergency steering timing feature and the obstacle orientation timing feature to a pre-trained driver state recognition model to obtain a current driver state;

[0139] The early warning module is configured to give a warning to the driver according to the current driver state.

[0140] The contents in the method embodiments are applicable to the system embodiments, the system embodiments achieve the same functions as the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0141] Referring to Figure 3 The embodiment of the present application provides a driver state monitoring device based on machine vision, which comprises:

[0142] at least one processor;

[0143] at least one memory for storing at least one program;

[0144] When the at least one program is executed by the at least one processor, the at least one processor implements the driver state monitoring method based on machine vision.

[0145] The contents in the method embodiments are applicable to the device embodiments, the device embodiments achieve the same functions as the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0146] The embodiment of the present application further provides a computer readable storage medium, wherein a program executable by a processor is stored, and the program executable by the processor is used for executing the driver state monitoring method based on machine vision when the processor executes the program.

[0147] The computer readable storage medium of the embodiment of the present application can execute the driver state monitoring method based on machine vision provided by the method embodiment of the present application, can execute the step of any combination of the method embodiment, has the corresponding functions and beneficial effects of the method.

[0148] The embodiment of the present application further discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method shown in the figure. Figure 1

[0149] ​In alternative embodiments, the functions / operations in the flow diagrams can occur in sequences other than those depicted. For example, two operations shown in succession can in fact be executed substantially concurrently or the operations can sometimes be executed in the reverse order depending upon the functionality / operations involved. Such variations are contemplated to be within the scope of the present application. Embodiments presented and described in the flow diagrams are examples only and are used to provide an enabling teaching for the present application. The processes disclosed are not limited to the order or specific blocks described. Alternative embodiments are contemplated, in which the order of the blocks is changed and where some blocks are performed in parallel rather than sequentially.

[0150] Moreover, while the present application has been described in the context of functional modules, it is to be understood that one or more of the functions and / or features described above can be integrated within a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It will also be appreciated that detailed discussion of the actual implementation of each module is not necessary for an enabling understanding of the application. Rather, the actual implementation is most readily derived from the description of the functionality of the various functional modules, in conjunction with the understanding of the properties, functions and interrelationships of the various functional modules presented in the context of the device disclosed herein. Therefore, the scope of the application is best understood from the appended claims, in conjunction with the full description and examples provided. It is to be understood that the specific concepts presented are merely illustrative of the application and are not intended to limit the scope of the application as defined by the claims. The scope of the application is defined by the claims and the full extent of equivalents to which such claims are entitled.

[0151] If the above functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0152] The logic and / or steps represented in the flow diagrams or otherwise described herein, for example, can be embodied in non-transitory computer- readable media, executed by an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or in conjunction with which the instructions can be executed. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium.

[0153] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and stored in a computer memory.

[0154] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or combinations thereof, can be used: a discrete logic circuit having logic gates for implementing logic functions upon data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), and / or the like.

[0155] In the above description of the present specification, reference is made to the description of terms such as "one embodiment / one example", "another embodiment / another example", or "certain embodiments / certain examples" and the like, which means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. The illustrative appearances of the above-mentioned terms in the description are not necessarily referred to the same embodiment or example throughout the specification. Moreover, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0156] While the embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary and are not to be taken as limiting the scope of the application. The scope of the application is defined by the claims and their equivalents.

[0157] The above is the specific description of the preferred embodiment of the application, but the application is not limited to the above-mentioned embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the application.

Claims

1. A method of driver state monitoring based on machine vision, characterized in that, The method comprises the following steps: obtaining time sequence data of displacement of a brake pedal, time sequence data of a steering wheel angle, and time sequence data of a road image in front of a vehicle; extracting an emergency braking time sequence feature of the brake pedal according to the time sequence data of displacement, extracting an emergency steering time sequence feature of the steering wheel according to the time sequence data of the steering wheel angle, and extracting an obstacle orientation time sequence feature in front of the vehicle according to the time sequence data of the road image; inputting the emergency braking time sequence feature, the emergency steering time sequence feature, and the obstacle orientation time sequence feature into a pre-trained driver state recognition model to obtain a current driver state; warning the driver according to the current driver state.

2. The method of claim 1, wherein the method further comprises: The obtaining of the time sequence data of displacement of the brake pedal, the time sequence data of the steering wheel angle, and the time sequence data of the road image in front of the vehicle specifically comprises: obtaining the time sequence data of displacement through a displacement sensor arranged on the brake pedal; obtaining the time sequence data of the steering wheel angle through a steering angle sensor arranged on a rotating shaft of the steering wheel; obtaining the time sequence data of the road image through a camera arranged in front of the vehicle.

3. The method of claim 1, wherein the method further comprises: The extracting of the emergency braking time sequence feature of the brake pedal according to the time sequence data of displacement, and the extracting of the emergency steering time sequence feature of the steering wheel according to the time sequence data of the steering wheel angle specifically comprises: performing short-time Fourier transform on the time sequence data of displacement to obtain first frequency domain information, and extracting first key frequency domain information from the first frequency domain information according to a preset first frequency band region; performing inverse Fourier transform on the first key frequency domain information to obtain a first time sequence signal, and extracting an instantaneous displacement change rate of the first time sequence signal to obtain the emergency braking time sequence feature; performing short-time Fourier transform on the time sequence data of the steering wheel angle to obtain second frequency domain information, and extracting second key frequency domain information from the second frequency domain information according to a preset second frequency band region; performing inverse Fourier transform on the second key frequency domain information to obtain a second time sequence signal, and extracting an instantaneous steering angle change rate of the second time sequence signal to obtain the emergency steering time sequence feature.

4. The method of claim 1, wherein the method further comprises: The extracting of the obstacle orientation time sequence feature in front of the vehicle according to the time sequence data of the road image specifically comprises: determining a plurality of continuous road image information according to the time sequence data of the road image; performing obstacle detection on the road image information to obtain an obstacle direction and an obstacle distance, and determining an obstacle orientation feature according to the obstacle direction and the obstacle distance; generating the obstacle orientation time sequence feature according to the obstacle orientation features corresponding to the plurality of continuous road image information.

5. The method of claim 1, wherein the method further comprises: The driver state recognition model is obtained through the following steps of training: obtaining brake pedal displacement time sequence samples, steering wheel angle time sequence samples, and front road image time sequence samples in a historical driving scene; extracting emergency braking time sequence feature samples according to the brake pedal displacement time sequence samples, extracting emergency steering time sequence feature samples according to the steering wheel angle time sequence samples, and extracting obstacle orientation time sequence feature samples according to the front road image time sequence samples; construct a training sample according to the emergency braking time sequence feature sample, the emergency steering time sequence feature sample, and the obstacle orientation time sequence feature sample, determine a driver state label corresponding to the training sample through manual labeling, construct a training data set according to the training sample and the driver state label; input the training data set to a pre-constructed bidirectional long short-term memory network for training, and obtain a trained driver state recognition model; wherein the driver state label includes a fatigue degree label and a concentration degree label.

6. The method of claim 5, wherein the method further comprises: The bidirectional long short-term memory network includes an input layer, a forward LSTM layer, a backward LSTM layer, and an output layer. The training data set is input to the pre-constructed bidirectional long short-term memory network for training, and the trained driver state recognition model is obtained. Specifically, it includes: input the training sample to the input layer, and calculate a forward hidden state vector through the forward LSTM layer; input the training sample to the input layer in reverse order, and calculate a backward hidden state vector through the backward LSTM layer; after the backward hidden state vector is processed in reverse order, the forward hidden state vector is spliced to obtain a target hidden state vector; input the target hidden state vector to the output layer, and output a driver state recognition result; determine a loss value according to the driver state recognition result and the driver state label; update the parameters of the bidirectional long short-term memory network according to the loss value, and obtain the trained driver state recognition model.

7. A machine vision-based driver state monitoring method according to any one of claims 1 to 6, characterized in that, According to the current driver state, the driver is prewarned. Specifically, it includes: determine whether the driver is fatigued or not concentrated according to the current driver state; when the driver is fatigued or not concentrated, prewarn the driver according to a preset first prompt information.

8. A machine vision based driver state monitoring system, characterized in that It includes: a data acquisition module for acquiring displacement time sequence data of a brake pedal, rotation angle time sequence data of a steering wheel, and road image time sequence data in front of a vehicle; a feature extraction module for extracting an emergency braking time sequence feature of the brake pedal according to the displacement time sequence data, an emergency steering time sequence feature of the steering wheel according to the rotation angle time sequence data, and an obstacle orientation time sequence feature in front of the vehicle according to the road image time sequence data; a state recognition module for inputting the emergency braking time sequence feature, the emergency steering time sequence feature, and the obstacle orientation time sequence feature to a pre-trained driver state recognition model to obtain a current driver state; a warning module for prewarning the driver according to the current driver state.

9. A machine vision-based driver state monitoring apparatus characterized by comprising: It includes: at least one processor; at least one memory for storing at least one program; when the at least one program is executed by the at least one processor, the at least one processor implements a machine vision-based driver state monitoring method according to any one of claims 1 to 7.

10. A computer readable storage medium having stored therein a program which is executable by a processor, characterized in that, The program executable by the processor, when executed by the processor, is used to perform a machine vision-based driver state monitoring method as claimed in any one of claims 1 to 7. The program executable by the processor, when executed by the processor, is used to perform a machine vision-based driver state monitoring method as claimed in any one of claims 1 to 7.

Images