Brain-vision fusion driver abnormal state detection early warning method based on L3 automatic driving technology

Abstract

The invention discloses a brain-vision fusion driver abnormal state detection early warning method based on an L3 automatic driving technology. The method specifically comprises the steps that a face image and an electroencephalogram signal of a driver are acquired in real time, and face feature points of the driver are extracted from the face image; calculating various face judgment indexes of the driver in real time; performing false removal and noise reduction on the extracted electroencephalogram signals in real time; according to the real-time face judgment index and the noise-reduced electroencephalogram signal, whether the driver enters a corresponding abnormal state under the two driving modes of taking over driving and automatic driving is judged, so that early warning is performed on the driver; the minimum attention concentration degree required in each driving mode can be met when the driver uses the automatic driving vehicle, and the potential safety hazard of L3 automatic driving is reduced to the maximum extent.

Description

Technical Field

[0001] The present invention relates to driver anomaly detection technology, specifically to a brain-vision fusion driver anomaly detection and early warning method based on L3 autonomous driving technology. Background Art

[0002] In recent years, my country's autonomous driving technology has grown by leaps and bounds. With its intelligence and efficiency, it has brought new production and creative vitality to modern vehicle traffic, and greatly promoted the steady development of domestic intelligent networking. At present, the country greatly encourages various research companies to invest in the development of autonomous driving technology, and provides policy support for the legal use of L3 and L4 autonomous driving on the road. At the same time, there is also relatively mature research in the visual aspect of abnormal detection and warning of driving ordinary vehicles. However, since L3 has a different driving mode compared to the previous ordinary driving and assisted driving, it is impossible to judge the degree of driver slackness in the autonomous driving mode by relying solely on visual detection. However, if the driver is too lazy in the autonomous driving mode and fails to respond to the request for the vehicle to take over the mode in time, it is likely to bring potential great hidden dangers to road traffic safety. This is also a driving safety challenge that must be considered and solved if autonomous driving technology wants to be fully promoted on the road.

[0003] Since most of the current driver abnormality detection and warning are still at the stage of visual detection, they can only reflect the surface physiological indicators and cannot deeply understand the specific reasons for the driver's abnormality. In the autonomous driving mode, since the driver may be in a resting state with his eyes and mouth closed and his head leaning on the seat for a long time, it is difficult to determine whether the driver has entered a deep relaxation state where he cannot respond to the takeover request by visual means. Therefore, L3 autonomous driving technology requires a new detection method that can detect driver abnormalities in both driving modes to meet the safety requirements of autonomous driving. Summary of the invention

[0004] In order to solve the shortcomings of autonomous driving technology in abnormal state detection and early warning, the present invention proposes a brain-vision fusion driver abnormal state detection and early warning method based on L3 autonomous driving technology.

[0005] The technical solution to achieve the purpose of the present invention is: a method for detecting and warning abnormal driver behavior based on brain-vision fusion of L3 autonomous driving technology, comprising:

[0006] Step 1: Acquire the driver's facial image and EEG signal in real time and extract the driver's facial feature points on the facial image;

[0007] Step 2: Calculate various facial judgment indicators of the driver in real time;

[0008] Step 3: De-spurious and denoise the extracted EEG signals in real time;

[0009] Step 4: Compare the calculated facial judgment index and the denoised EEG signal with the normal facial features and EEG signal respectively;

[0010] Step 5: Based on the comparison results, determine whether the driver enters an abnormal state during takeover and automatic driving, and issue a warning.

[0011] Preferably, the specific steps of extracting the driver's facial feature points are:

[0012] Initialize the Dlib model face detector and create facial landmark predictions;

[0013] Get the face location detector using the Dlib model;

[0014] After determining the face position, use the Dlib model to obtain the facial feature position detector;

[0015] The pixel coordinates of the driver's facial feature points are obtained through a facial feature position detector, wherein the facial feature points include left and right eye contour feature points and mouth contour feature points.

[0016] Preferably, the various facial judgment indicators include: the absolute value EAR of the difference in eye aspect ratio between the previous and next frames, the absolute value MAR of the difference in mouth aspect ratio between the previous and next frames, the head pitch rotation data HPE between the previous and next frames, and the tracking gaze direction GAZE of the pupil and eye corner positioning inference between the previous and next frames.

[0017] Preferably, the calculation methods of various facial judgment indicators are:

[0018] The eye aspect ratio EAR is calculated based on the left and right eye contour feature points. The specific calculation formula is as follows:

[0019]

[0020] Where P1~P6 are the coordinates of the feature points of the left and right eyes;

[0021] According to the mouth contour feature points, the mouth length-to-width ratio is calculated. The specific calculation formula is as follows:

[0022]

[0023] Where M1~M6 are the coordinates of the feature points of the mouth;

[0024] Match facial feature points with 3D face model;

[0025] Solve the conversion relationship between the 3D world space coordinates and the corresponding 2D pixel point coordinates, and solve the head-down Euler angle and rotation Euler angle according to the rotation matrix in the conversion relationship. The calculation formula is as follows:

[0026]

[0027] Roll=atan(-r 12 ,r 22 )

[0028] Among them, R 3×3 represents the rotation matrix, r 00 ~r 22 are the constant values ​​in the rotation matrix, Pich refers to the head-down Euler angle, and Roll refers to the rotation Euler angle;

[0029] The head pitch rotation data HPE of the front and rear frames consists of the head down Euler angle and the rotation Euler angle;

[0030] Star ray iteration is used to lock the pupil center position, with the pupil center as the moving point and the eye corner as the fixed point, and the relative position of the pupil center and the eye corner is determined to determine the line of sight direction GAZE.

[0031] Preferably, the extracted EEG signal is subjected to denoising and de-noising, and the specific steps are as follows:

[0032] 3.1. Let the extracted EEG signal be the initial estimation vector w(0), set its modulus to 1, and set k = 1;

[0033] 3.2. Let w(k) = E[x(w(k-1) T x) 3 ]-3w(k-1), x represents the transition vector of the EEG signal;

[0034] 3.3. If ||w(k) T w(k-1)-1||>δ, δ is the set threshold, then set k=k+1 and return to step 3.2 to execute, otherwise the output vector w(k) is the EEG signal after denoising.

[0035] Preferably, the calculated facial judgment index and the denoised EEG signal are used to compare with normal facial features and EEG signals respectively, and the specific method for obtaining the comparison result is:

[0036] When the EAR value drops below the set threshold and then rises above it, it is considered a blink. If the blinking frequency within 1 minute is higher than the set frequency, it is considered an abnormal driving state.

[0037] The process in which the MAR value changes from less than the threshold to greater than the threshold, and then from greater than the threshold to less than the threshold, and lasts for more than 3 seconds is recorded as a yawning process. At the same time, when the number of yawns exceeds the set number within 1 minute, it is determined that the current driving state is abnormal;

[0038] In a time period, if the time proportion of the head-down Euler angle |Pich|≥20° or the head-tilt Euler angle |Roll|≥20° exceeds the set duration, the current driving state is considered abnormal;

[0039] If the sight direction GAZE stays in one direction for more than the set time within 1 minute, the current driving state is considered abnormal;

[0040] If the ratio of theta waves to alpha waves in the EEG or MEG exceeds 1.5, or the blood oxygen saturation in the cerebral blood oxygen level NIRS is lower than 95%, the current driving state is considered abnormal.

[0041] Preferably, the specific method of judging whether the driver enters an abnormal state in takeover driving and automatic driving according to the comparison result and issuing an early warning is:

[0042] When the vehicle is in takeover driving mode, if any facial judgment indicator determines that the current driving state is abnormal, it is determined as driving abnormality and a voice warning is given; when the vehicle is in automatic driving mode, if any EEG signal determines that the current driving state is abnormal, it is determined as driving abnormality and seat vibration is used to stimulate awareness.

[0043] Compared with the prior art, the present invention has the following significant advantages:

[0044] (1) The present invention fills the gap in L3 autonomous driving technology in driver anomaly detection and warning, and fully considers and analyzes the differences in anomaly detection evaluation indicators under dual driving modes in the future autonomous driving era;

[0045] (2) The present invention expands various indicators for detecting driver abnormalities in terms of visual detection, thereby ensuring the accuracy of early warning of driving abnormalities in the takeover driving mode;

[0046] (3) The present invention incorporates brain-computer interface technology and links some physiological indicators in EEG signals with driving abnormalities, thus making up for the difficulty of detecting driver abnormalities in the autonomous driving mode;

[0047] (4) The present invention reasonably integrates visual perception and brain-computer interface, which can reduce the missed detection rate of driving abnormalities to a certain extent.

[0048] Other features of the present invention will be described in the following description, and part of them will become obvious from the description. The purpose of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0050] Figure 1 It is a flow chart of the present invention.

[0051] Figure 2 It is a schematic diagram of the device structure of the present invention. DETAILED DESCRIPTION

[0052] It is easy to understand that according to the technical solution of the present invention, without changing the essential spirit of the present invention, a person skilled in the art can imagine various embodiments of the present invention. Therefore, the following specific embodiments and drawings are only exemplary descriptions of the technical solution of the present invention, and should not be regarded as the whole of the present invention or as a limitation or limitation of the technical solution of the present invention. On the contrary, the purpose of providing these embodiments is to enable those skilled in the art to understand the present invention more thoroughly. The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the innovative concept of the present invention.

[0053] A driver abnormality detection and warning method based on L3 autonomous driving technology using brain-vision fusion, which combines visual means with brain-computer interface to achieve driver abnormality detection in both takeover driving and autonomous driving modes, specifically including:

[0054] Step 1: Acquire the driver's facial image and EEG signal in real time and extract the driver's facial feature points;

[0055] A high-resolution monocular RGB camera is installed on the upper edge of the vehicle's front windshield to obtain the driver's facial image, and a non-invasive brain-computer interface is installed on the main driver's seat to collect various types of EEG signals.

[0056] Capture the driver's facial feature points on the acquired facial image. The specific steps are:

[0057] Initialize the Dlib model face detector and create facial landmark predictions.

[0058] Use the dlib.get_frontal_face_detector() function that comes with the Dlib model to get the face position detector;

[0059] After determining the face position, use the dlib.shape_predictor() function that comes with the Dlib model to obtain the facial feature position detector;

[0060] The pixel coordinates of 68 feature points on the driver's face are obtained through a facial feature position detector; the facial feature points include left and right eye contour feature points and mouth contour feature points.

[0061] The various types of EEG signals collected by the brain-computer interface include: the electroencephalogram (EEG) that records changes in scalp electrical potential, the magnetoencephalogram (MEG) that captures the weak magnetic fields generated by brain activity, and the cerebral blood oxygen level (NIRS) that provides information on the concentration of oxygenated hemoglobin and deoxygenated hemoglobin in the brain.

[0062] Step 2: Calculate various driver facial judgment indicators (EAR, MAR, HPE, GAZE) in real time based on the pixel coordinates of the 68 extracted feature points.

[0063] The various judgment indicators of the driver's face include: the absolute value of the difference between the eye aspect ratio of the previous and next frames EAR, the absolute value of the difference between the mouth aspect ratio of the previous and next frames MAR, the head pitch rotation data HPE of the previous and next frames, and the tracking gaze direction GAZE of the pupil and eye corner positioning inference of the previous and next frames. The specific method for calculating the above judgment indicators is as follows:

[0064] The absolute value EAR of the difference between the previous and next frames is determined as follows:

[0065] Among the 68 extracted feature points, 6 feature points of the left and right eye contours are selected to calculate the eye aspect ratio EAR. The specific calculation formula is as follows:

[0066]

[0067] Where P1~P6 are the coordinates of the feature points of the left and right eye contours.

[0068] The absolute value MAR of the difference between the length-to-width ratio of the mouth in the previous and next frames is determined as follows:

[0069] Six feature points of the mouth contour are selected from the 68 extracted feature points to calculate the mouth aspect ratio MAR. The specific calculation formula is:

[0070]

[0071] Where M1~M6 are the coordinates of the feature points of the mouth contour.

[0072] The method for determining the head pitch rotation data HPE of the front and rear frames is:

[0073] The pixel coordinates of the 68 feature points on the driver's face are regarded as the 2D facial key point coordinates;

[0074] Match the 2D facial key point coordinates with the 3D face model;

[0075] The conversion relationship between the 3D world space coordinates and the corresponding 2D pixel coordinates is solved by the known camera internal and external parameters and camera coordinates of the monocular RGB camera. The conversion relationship formula is:

[0076]

[0077] In the formula, Z c is the camera coordinate, (u,v) is the coordinate of the 2D pixel point, is the camera internal parameter, where f x ,f y Respectively represent the focal length of the camera in the horizontal and vertical directions, u0, v0 represent the coordinates of the principal point of the image in the horizontal and vertical directions, [R 3×3 T 3×1 ] is the camera external parameter, where R 3×3 represents the rotation matrix, T 3×1 represents the translation vector, (X W ,Y W ,Z W ) are 3D world coordinates.

[0078] Solve the Euler angle according to the obtained rotation matrix. The solution formula is:

[0079]

[0080] Roll=atan(-r 12 ,r 22 )

[0081] Among them, R 3×3 represents the rotation matrix, r 00 ~r 22 are the constant values ​​in the rotation matrix, Pich refers to the head-down Euler angle, Roll refers to the rotation Euler angle, and the judgment index HPE is jointly expressed by Pich and Roll.

[0082] The method for determining the pupil and eye corner positioning inference tracking gaze direction GAZE in the previous and next frames is:

[0083] Use star ray iteration to lock the pupil center position. The specific steps are:

[0084] 2.4.1. Determine the center of the eye from the acquired left and right eye contour feature points and set it as the origin. Then emit a number of rays outward from the origin.

[0085] 2.4.2. Use Canny edge detection for each ray to find its intersection with the pupil edge, so that the pupil edge is composed of several intersection points;

[0086] 2.4.3. Set the initial value of the pupil center position within the eye range, and continuously update the estimated value of the pupil center according to the intersection of the ray and the edge to gradually approach the actual pupil center position;

[0087] 2.4.4. Repeat the steps in 2.4.3. After each iteration, check whether the change of the current pupil center position is small enough. If it has converged, stop the iteration.

[0088] 2.4.5. After obtaining the converged pupil center position, it is necessary to optimize the positioning of the pupil center, that is, use the Otsu dynamic threshold to output a binary image of the driver's facial image, and then use the longitudinal erosion filter to obtain the final iris area of ​​the eye, and use the center point of the iris area as the center of the pupil.

[0089] Step 3: Perform real-time denoising and denoising on the extracted EEG signals. The specific method is as follows:

[0090] Use the FastICA algorithm to denoise and purify various EEG signals (EEG, MEG, NIRS) collected in step 1.

[0091] Since the "source" of electrooculographic interference in the observed EEG signal is independent of the "effective source" of a large number of neural activity organizations, the ICA method is considered to remove the electrooculographic interference, and the various components of ICA are more physiologically meaningful. If the Infomax algorithm is used for decomposition, the inappropriate selection of the learning factor μ will lead to non-convergence of the iterative results. And due to the complexity of the EEG signal itself, when the number of independent signals is less than the number of recorded leads, it is also possible that the ICA result will not converge. At the same time, due to the convergence speed, the speed of EEG signal decomposition will be slow. Therefore, the FastICA algorithm is used to remove artifacts from the EEG signal. The specific steps of the FastICA algorithm for removing artifacts and denoising the above EEG signals are as follows:

[0092] 3.1. Set the initial estimated vector w(0) of the EEG signal to 1, and set k = 1;

[0093] 3.2. Let w(k) = E[x(w(k-1) T x) 3 ]-3w(k-1), its expected value can be calculated using a certain number of sampling points x, where x represents the transition vector of the EEG signal;

[0094] 3.3. If ||w(k) T w(k-1)-1||>δ, δ is the set threshold, then set k=k+1 and return to step 3.2 to execute, otherwise the output vector w(k) is the EEG signal after denoising.

[0095] The final vector w(k) is a non-Gaussian signal in the mixed signal x(k). T x(k), k=1,2,... is the EEG signal obtained by noise reduction and purification.

[0096] Step 4: Compare the calculated facial judgment index and the denoised EEG signal with the normal facial features and EEG signal respectively;

[0097] Under normal circumstances, people blink very quickly, and it usually takes 1 to 3 frames to complete a blink. However, in a fatigued state, the blinking speed will decrease and the blinking frequency will increase. When the EAR value drops below a certain threshold and then quickly rises above this threshold, it is judged as a blink. If the blinking frequency within 1 minute is significantly higher than in other cases, it is judged as fatigue. Under normal circumstances, people blink 15 to 20 times within 1 minute. Therefore, this detection system is set to determine that the current driving state is abnormal when the blinking frequency exceeds 20 times within 1 minute.

[0098] Normally, when a person is speaking, the continuous opening time of the mouth will not exceed 3 seconds. Therefore, this detection system records the process of the MAR value changing from less than the threshold to greater than the threshold, and then from greater than the threshold to less than the threshold, and the duration exceeds 3 seconds as a yawning process. At the same time, it is set that when the number of yawns exceeds 3 times within 1 minute, it is judged as abnormal driving status.

[0099] The parameter threshold is set to 0.3. Within a time period, such as 10 seconds, when the time proportion of the head-down Euler angle |Pich|≥20° or the head-tilt Euler angle |Roll|≥20° exceeds 0.3, the current driving state is considered abnormal.

[0100] By expanding the coordinates of the eye feature points to a certain extent and adjusting the parameters appropriately, a better corresponding proportional relationship between the eye area ratio and the eye coordinates can be obtained, which can be used as the ROI area for pupil recognition, and the left and right boundaries of the ROI area are regarded as the positions of the eye corners. Then, the pupil center is used as the moving point and the eye corners are used as the fixed points, and the gaze direction GAZE is determined by the relative positions of the two. If the gaze stays in one direction for more than 4 seconds more than 3 times within 1 minute, the current driving state is considered abnormal.

[0101] After noise reduction of the EEG signal, if during driving, the ratio of theta waves to alpha waves in EEG and MEG continues to exceed 1.5 or the blood oxygen saturation (SpO2) in NIRS is lower than 95% for more than 4 seconds, it can be determined that the current driving state is abnormal.

[0102] Step 5: Based on the comparison results, determine whether the driver enters an abnormal state during takeover and automatic driving, and issue a warning.

[0103] L3 autonomous driving vehicles are divided into two driving modes (takeover driving and autonomous driving) according to the current driving scenario. The scenarios that require takeover driving are divided into two categories. The first type of takeover scenario means that the vehicle actively exits the autonomous driving mode due to insufficient road coverage or specific scenarios are not preset; the second type of takeover scenario means that the vehicle actively exits the autonomous driving mode due to instantaneous problems (such as sudden situations such as the need to quickly avoid vehicles or pedestrians) that cannot be handled.

[0104] When the vehicle is in takeover driving mode, the calculated facial judgment indicators are compared with normal facial features. If they exceed the threshold, the driving state is judged to be abnormal and a voice warning is played. When the vehicle is in automatic driving mode, the noise-reduced EEG signals are compared with normal EEG signals. If they exceed the threshold, the driving state is judged to be abnormal and the seat vibration stimulates consciousness.

[0105] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

[0106] It should be understood that in order to simplify the present invention and help those skilled in the art understand the various aspects of the present invention, in the above description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes described in a single embodiment or described with reference to a single figure. However, the present invention should not be interpreted as the features included in the exemplary embodiments are all necessary technical features of the patent claims.

[0107] It should be understood that the modules, units, components, etc. included in the device of an embodiment of the present invention can be adaptively changed to be set in a device different from the embodiment. The different modules, units, or components included in the device of the embodiment can be combined into one module, unit, or component, or they can be divided into multiple sub-modules, sub-units, or sub-components.

Claims

1. A method for detecting and warning abnormal driver behavior based on brain-vision fusion of L3 autonomous driving technology, characterized in that: include: Step 1: Acquire the driver's facial image and EEG signal in real time and extract the driver's facial feature points on the facial image, wherein the EEG signal includes the EEG of scalp potential changes, the MEG of magnetoencephalogram, and the NIRS of brain blood oxygen level; Step 2: Calculate various facial judgment indicators of the driver in real time; Step 3: De-spurious and denoise the extracted EEG signals in real time; Step 4: Compare the calculated facial judgment index and the denoised EEG signal with the normal facial features and EEG signal respectively; Step 5: Based on the comparison results, determine whether the driver enters an abnormal state during takeover and automatic driving, and issue a warning.

2. The method for detecting and warning driver abnormalities based on brain-vision fusion and L3 autonomous driving technology according to claim 1 is characterized in that: The specific steps for extracting the driver's facial feature points are: Initialize the Dlib model face detector and create facial landmark predictions; Get the face location detector using the Dlib model; After determining the face position, use the Dlib model to obtain the facial feature position detector; The pixel coordinates of the driver's facial feature points are obtained through a facial feature position detector, wherein the facial feature points include left and right eye contour feature points and mouth contour feature points.

3. The method for detecting and warning abnormal driver behavior based on L3 autonomous driving technology using brain-vision fusion according to claim 1 is characterized in that: Various facial judgment indicators include: the absolute value of the difference in the eye aspect ratio EAR between the previous and next frames, the absolute value of the difference in the mouth aspect ratio MAR between the previous and next frames, the head pitch rotation data HPE between the previous and next frames, and the pupil and eye corner positioning inference tracking gaze direction GAZE between the previous and next frames.

4. The method for detecting and warning abnormal driver behavior based on L3 autonomous driving technology using brain-vision fusion according to claim 1, characterized in that: The calculation methods of various facial judgment indicators are: The eye aspect ratio EAR is calculated based on the left and right eye contour feature points. The specific calculation formula is as follows: Where P1~P6 are the coordinates of the feature points of the left and right eyes; According to the mouth contour feature points, the mouth length-to-width ratio is calculated. The specific calculation formula is as follows: Where M1~M6 are the coordinates of the feature points of the mouth; Match facial feature points with 3D face model; Solve the conversion relationship between the 3D world space coordinates and the corresponding 2D pixel point coordinates, and solve the head-down Euler angle and rotation Euler angle according to the rotation matrix in the conversion relationship. The calculation formula is as follows: Roll=atan(-r 12 ,r 22 ) Among them, R 3×3 represents the rotation matrix, r 00 ~r 22 are the constant values ​​in the rotation matrix, Pich refers to the head-down Euler angle, and Roll refers to the rotation Euler angle; The head pitch rotation data HPE of the front and rear frames consists of the head down Euler angle and the rotation Euler angle; Star ray iteration is used to lock the pupil center position, with the pupil center as the moving point and the eye corner as the fixed point, and the relative position of the pupil center and the eye corner is determined to determine the line of sight direction GAZE.

5. The method for detecting and warning abnormal driver behavior based on L3 autonomous driving technology using brain-vision fusion according to claim 1, characterized in that: The extracted EEG signal is subjected to denoising and de-noising. The specific steps are as follows: 3.

1. Let the extracted EEG signal be the initial estimation vector w(0), set its modulus to 1, and set k = 1; 3.

2. Let w(k) = E[x(w(k-1) T x) 3 ]-3w(k-1), x represents the transition vector of the EEG signal; 3.

3. If ||w(k) T w(k-1)-1||>δ, δ is the set threshold, then set k=k+1 and return to step 3.2 to execute, otherwise the output vector w(k) is the EEG signal after denoising.

6. The method for detecting and warning abnormal driver behavior based on L3 autonomous driving technology using brain-vision fusion according to claim 1, characterized in that: The calculated facial judgment index and the denoised EEG signal are used to compare with normal facial features and EEG signals respectively. The specific method for obtaining the comparison result is as follows: When the EAR value drops below the set threshold and then rises above it, it is considered a blink. If the blinking frequency within 1 minute is higher than the set frequency, it is considered an abnormal driving state. The process in which the MAR value changes from less than the threshold to greater than the threshold, and then from greater than the threshold to less than the threshold, and lasts for more than 3 seconds is recorded as a yawning process. At the same time, when the number of yawns exceeds the set number within 1 minute, it is determined that the current driving state is abnormal; In a time period, if the time proportion of the head-down Euler angle |Pich|≥20° or the head-tilt Euler angle |Roll|≥20° exceeds the set duration, the current driving state is considered abnormal; If the sight direction GAZE stays in one direction for more than the set time within 1 minute, the current driving state is considered abnormal; If the ratio of theta waves to alpha waves in the EEG or MEG exceeds 1.5, or the blood oxygen saturation in the cerebral blood oxygen level NIRS is lower than 95%, the current driving state is considered abnormal.

7. The method for detecting and warning abnormal driver behavior based on L3 autonomous driving technology using brain-vision fusion according to claim 6 is characterized in that: The specific method for judging whether the driver has entered an abnormal state under takeover driving and automatic driving according to the comparison results and issuing an early warning is as follows: When the vehicle is in takeover driving mode, if any facial judgment indicator determines that the current driving state is abnormal, it is determined as driving abnormality and a voice warning is given; when the vehicle is in automatic driving mode, if any EEG signal determines that the current driving state is abnormal, it is determined as driving abnormality and seat vibration is used to stimulate awareness.

Images