Driving State Prediction Method and Device for Drivers Based on Multimodal Data

Abstract

The present invention discloses a method and device for predicting the driving state of a driver based on multimodal data. The method for predicting the driving state of a driver based on multimodal data includes: obtaining an initial prediction result for the driving state of the driver based on the physiological state data of the driver; and obtaining a target prediction result for the driving state of the vehicle based on the initial prediction result and the vehicle driving data. The method for predicting the driving state of a driver based on multimodal data according to the present invention combines the physiological state data of the driver and the vehicle driving data to predict the driving state of the vehicle, improves the prediction accuracy of the driving state, and thus improves driving safety.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence, human factor intelligence, etc., and particularly relates to a method and device for predicting a driving state of a driver based on multimodal data. Background Art

[0002] Research shows that driver distraction will lead to a significant decline in the driver's ability to perceive danger, that is, the driver's ability to respond to danger decreases, resulting in the driver's untimely and inaccurate judgment of the road environment, leading to improper operation behaviors and causing traffic accidents. If the driver can be prompted when the driver is distracted while driving, some accidents can be effectively avoided. Statistics show that if the driver's reaction time can be advanced by 0.5 seconds, the possibility of an accident will be reduced by about 60%, and even if an accident occurs, the intensity of the accident can be appropriately reduced. However, in related technologies, the prediction accuracy of the driving state prediction method is low, the prediction efficiency is low, and the calculation amount is large. Summary of the Invention

[0003] The embodiments of the present application aim to at least solve one of the technical problems in the related technologies to some extent. For this reason, the purpose of the embodiments of the present application is to propose a method, device, electronic device, and storage medium for predicting a driving state of a driver based on multimodal data.

[0004] The embodiments of the present application provide a method for predicting a driving state of a driver based on multimodal data, the method including: obtaining an initial prediction result for the driving state of the driver based on the physiological state data of the driver; and obtaining a target prediction result for the driving state of the vehicle based on the initial prediction result and the vehicle driving data.

[0005] Exemplarily, the vehicle driving data includes lateral speed offset information; the obtaining a target prediction result for the driving state of the vehicle based on the initial prediction result and the vehicle driving data includes: determining that the driving state of the driver is a risky driving state when both the initial prediction result and the lateral speed offset information indicate a risky driving state.

[0006] Exemplarily, the vehicle driving data includes a lateral speed value and lateral speed offset information; the obtaining a target prediction result for the driving state of the vehicle based on the initial prediction result and the vehicle driving data includes: determining the driving state of the driver based on the initial prediction result and the lateral speed offset information when the lateral speed value is less than or equal to a preset speed threshold; and determining that the driving state of the driver is a risky driving state when both the initial prediction result and the lateral speed offset information indicate a risky driving state.

[0007] Exemplarily, the vehicle driving data includes the true value of kinematic information, and the lateral speed offset information includes a first lateral speed offset value; the method further includes: predicting a predicted value of kinematic information based on the true value of the kinematic information; obtaining a prediction error of the kinematic information based on the true value of the kinematic information and the predicted value of the kinematic information; and obtaining the first lateral speed offset value based on the prediction error.

[0008] Exemplarily, the lateral speed offset information indicates that the driving state of the driver is a risky driving, including: in a case where the first lateral speed offset value is greater than a preset speed offset threshold, determining that the lateral speed offset information indicates that the driving state of the driver is a risky driving.

[0009] Exemplarily, the predicting a predicted value of kinematic information based on the true value of the kinematic information includes: predicting using a plurality of state-space models based on the true value of the kinematic information to obtain a plurality of predicted values corresponding one-to-one to the plurality of state-space models; and performing a weighted calculation on the plurality of predicted values based on the weights corresponding to the plurality of state-space models to obtain the predicted value of the kinematic information.

[0010] Exemplarily, the vehicle driving data further includes a lateral speed value, and the lateral speed offset information includes a second lateral speed offset value; the method further includes: determining an initial lateral speed offset value based on the lateral speed value; and determining an average value of the initial lateral speed offset value as the second lateral speed offset value based on the initial lateral speed offset value and a smoothing coefficient.

[0011] Exemplarily, the lateral speed offset information indicates that the driving state of the driver is a risky driving, including: in a case where the second lateral speed offset value is within a risk confidence interval, determining that the lateral speed offset information indicates that the driving state of the driver is a risky driving.

[0012] Exemplarily, in a case where both the initial prediction result and the lateral speed offset information indicate that the driving state of the driver is a risky driving, determining that the driving state of the driver is a risky driving includes: in a case where the initial prediction result indicates that the driving state of the driver is a risky driving within a target number of time periods, and the lateral speed offset information indicates that the driving state of the driver is a risky driving, determining a risky driving level based on the target number, where the target number is positively correlated with the number of levels of the risky driving level, and the risk degree of a larger level is greater than that of a smaller level.

[0013] Exemplarily, the method further includes: outputting risk prompt information in a corresponding risk prompt manner according to the risk driving level.

[0014] Exemplarily, the physiological state data includes at least one of eye movement data and electroencephalogram data; the obtaining of an initial prediction result of the driving state of the driver based on the physiological state data of the driver includes: inputting at least one of the eye movement data and the electroencephalogram data into a trained deep learning model for prediction to obtain the initial prediction result.

[0015] Exemplarily, the preprocessing of the eye movement data includes at least one of the following: removing data with abnormal pupil size change, occluded pupil, and artifacts at the pupil edge in the eye movement data; removing data representing gaze line deviation in the eye movement data; removing data with the line of sight outside the region of interest in the eye movement data; removing data representing a saccade angular velocity greater than a preset angular velocity in the eye movement data.

[0016] Exemplarily, the preprocessing of the electroencephalogram data includes at least one of the following: averaging multi-channel electroencephalogram data to obtain an average value and retaining the difference between the electroencephalogram data of each channel and the average value; filtering the electroencephalogram data and retaining data in a preset frequency band; removing interference data caused by blinking or body movement in the electroencephalogram data; extracting features from the electroencephalogram data to obtain power spectral density feature data for a specific frequency band.

[0017] Exemplarily, the initial prediction result characterizes whether the driver is distracted during driving, and the target prediction result characterizes whether a vehicle driving deviation is caused by the driver's distraction during driving.

[0018] Exemplarily, the method further includes: switching the driving mode of the vehicle based on the target prediction result.

[0019] Exemplarily, the switching of the driving mode of the vehicle based on the target prediction result includes: when the target prediction result characterizes that the vehicle is in a risk driving state, switching the driving mode of the vehicle to an autonomous driving mode.

[0020] Exemplarily, the method further includes: detecting the comfort information of the driver when the vehicle is driving based on the autonomous driving mode; when the comfort information characterizes that the comfort level of the driver is less than a preset level, switching the driving mode of the vehicle from the autonomous driving mode to an assisted driving mode or a conventional driving mode.

[0021] Exemplarily, when the vehicle is traveling based on the autonomous driving mode, detecting the comfort information of the driver includes: when the vehicle is traveling based on the autonomous driving mode, collecting at least one of the physiological information of the driver and the vehicle driving information; and detecting the comfort information of the driver based on at least one of the physiological information and the vehicle driving information.

[0022] Another embodiment of the present application provides a driving state prediction device for a driver based on multi-modal data. The device includes: a first obtaining module for obtaining an initial prediction result for the driving state of the driver based on the physiological state data of the driver; and a second obtaining module for obtaining a target prediction result for the driving state of the vehicle based on the initial prediction result and the vehicle driving data.

[0023] Another embodiment of the present application provides an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the method according to any one of the above embodiments are implemented.

[0024] Another embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method according to any one of the above embodiments are implemented.

[0025] Another embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device can execute the steps of the method according to any one of the above embodiments.

[0026] In the above embodiments, an initial prediction result for the driving state of the driver is obtained based on the physiological state data of the driver; and a target prediction result for the driving state of the vehicle is obtained based on the initial prediction result and the vehicle driving data. The driving state prediction method for a driver based on multi-modal data of the present invention can combine the physiological state data of the driver and the vehicle driving data to predict the driving state of the vehicle, improve the prediction accuracy of the driving state, and thus improve driving safety. Description of the Drawings

[0027] Figure 1 It is a flowchart of the driving state prediction method for a driver based on multi-modal data provided by an embodiment of the present application;

[0028] Figure 2 It is a schematic diagram of the driving state prediction system provided by an embodiment of the present application;

[0029] Figure 3 It is a schematic flowchart of preprocessing electroencephalogram data provided by an embodiment of the present application;

[0030] Figure 4 Flow chart for obtaining a target prediction result provided by an embodiment of the present application;

[0031] Figure 5 Flow chart for obtaining a first lateral velocity offset value provided by an embodiment of the present application;

[0032] Figure 6 Flow chart for obtaining a predicted value of kinematic information provided by an embodiment of the present application;

[0033] Figure 7 Flow chart for obtaining a second lateral velocity offset value provided by an embodiment of the present application;

[0034] Figure 8 Schematic diagram of a driving state prediction device for a driver based on multimodal data provided by an embodiment of the present application;

[0035] Figure 9 Block diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0036] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

[0037] Research shows that driver distraction can lead to a significant decline in the driver's ability to perceive danger, that is, the driver's ability to respond to danger decreases, resulting in untimely and inaccurate judgment of the road environment by the driver, leading to improper operation behaviors and causing traffic accidents. If the driver can be prompted when the driver is distracted while driving, some accidents can be effectively avoided. Statistics show that if the driver's reaction time can be advanced by 0.5 seconds, the likelihood of an accident will be reduced by approximately 60%, and even if an accident occurs, the intensity of the accident can be appropriately reduced.

[0038] In one embodiment, traditional machine learning and deep learning methods can be used to predict the state of a driver. Traditional machine learning methods have a relatively small computational load and low requirements for the hardware platform, and are suitable for embedded platforms, but they require a complex feature extraction process. Deep learning methods can automatically extract effective features, simplify the classification process, improve generalization, and have excellent classification performance when there is sufficient data volume, but they have a large computational load and high requirements for the hardware platform.

[0039] Deep learning is an "end-to-end" algorithm, which belongs to a type of representation learning. It only requires data input and has a corresponding target output, eliminating complex feature engineering in the middle and being able to self-learn and extract useful features. Due to the advantages of self-learning features and excellent performance of deep learning, deep learning methods can be applied to the problem of driver distraction.

[0040] For example, a spatio-temporal convolutional neural network based on electroencephalogram (EEG) signals can be used to detect driver distraction. By combining the time-domain features and frequency-domain features of EEG signals, a high accuracy is obtained. Compared with traditional algorithms, the computational efficiency and parameter update speed of this framework are more advantageous and can be applied to an online distraction monitoring system. For example, a Long Short-Term Memory (LSTM) model for detecting the fatigue state of drivers can be built to predict the fatigue state of drivers. The LSTM model can make full use of the temporal characteristics of EEG signals and optimize the model by combining the visualization of the training process. Compared with traditional machine learning algorithms, the accuracy has been greatly improved. For example, a convolutional neural network based on channels can be used to predict the fatigue state of drivers. This method makes full use of the channel correlation characteristics of EEG signals and achieves good results. For example, by constructing a simulated driving experiment to design different driving scenarios, the attention state is evaluated by the driver's braking reaction time, and deep convolutional networks are used to process EEG signals, thus realizing the detection of the braking intention of drivers in different attention states.

[0041] In one example, driving behavior experiments are mainly divided into simulated driving experiments and real vehicle experiments. For example, in a real vehicle environment, image data of the driver during the driving process is collected by a camera, including images in the normal driving state and images in the distracted state (talking on the phone with the left or right hand, sending text messages with the left or right hand, taking items in the back row of the driver's seat of the car, operating the in-vehicle radio), and recognition is carried out through image technology. For example, a driving simulator with visual feedback, auditory feedback, and vehicle sensors can be used for distraction research. For example, a vehicle turning experiment can be implemented using the driving simulator, and driving parameters such as the steering wheel angle, vehicle speed, brake pedal signal, lateral acceleration, environmental road information, and driving time during the driving process are collected to explore the steering pattern under low attention, so as to monitor driver distraction based on the steering performance of the vehicle. On this basis, a real vehicle experiment is carried out to eliminate the influence of extreme working conditions on the steering sensor and enhance the robustness.

[0042] In one example, with the continuous development of computer hardware and the easier availability of a large amount of driving distraction data, it has become possible to apply machine learning algorithms to the study of driving distraction. For example, one-dimensional discrete wavelet analysis can be used to analyze electroencephalogram (EEG) signals to obtain various wavelet frequency bands, and the statistics of wavelet coefficients in the δ-wave, θ-wave, α-wave, and β-wave frequency bands can be extracted as features and input into a neural network. Combining with a fuzzy model, the attention status of the driver can be classified. For example, the power spectrum of the EEG and the driving scenario can be combined as input features, combined with a Support Vector Machine (SVM), and the particle swarm algorithm can be used to optimize the system parameters to obtain a driving attention dispersion detection model for different driving scenarios. For example, the frequency domain features of EEG data can be extracted, and based on Principal Component Analysis (PCA), the feature dimension can be reduced. The reduced features are respectively input into the Maximum Likelihood Estimation (MLE) and the K-Nearest Neighbor (KNN) model for classification. For example, the EEG data of the driver in a distracted state can be collected, and its power spectrum can be extracted as a feature, and then input into an SVM based on the Radial Basis Function (RBF) to construct a driver attention focus tracking system.

[0043] It can be understood that by monitoring and predicting the driving state of the driver through the driver distraction monitoring system, corrections can be made to the driver before danger occurs when the driver is distracted, reducing the degree of the driver's incorrect perception and judgment of the environment and reducing the reaction time to dangerous situations. These have important theoretical and application values for reducing the accident rate and improving traffic safety.

[0044] In view of this, the present invention proposes a method for predicting the driving state of a driver based on multimodal data, which combines multimodal data to predict the state of the driver. The multimodal data includes the physiological state data of the driver and the vehicle driving data, and the monitoring information is fed back to the Advance Driver Assistance System (ADAS) to warn or remind the distracted driver. This function combined with the existing ADAS can reduce the distracted behavior of the driver, thereby improving driving safety.

[0045] Figure 1 It is a flowchart of the method for predicting the driving state of a driver based on multimodal data according to an embodiment of the present invention.

[0046] As Figure 1As shown in the figure, the method for predicting the driving state of a driver based on multimodal data includes S1 - S2.

[0047] S1. Based on the physiological state data of the driver, obtain an initial prediction result for the driving state of the driver.

[0048] S2. Based on the initial prediction result and the vehicle driving data, obtain a target prediction result for the driving state of the vehicle.

[0049] Specifically, the causes of accidents are mostly due to the driver being distracted during driving. To improve driving safety, it is necessary to predict the driving state of the driver during driving. For example, the physiological state information of the driving state can be collected through sensors, and the collected physiological state information of the driver can reflect the attention information of the driver to a certain extent. Based on the physiological state information of the driver, an initial prediction result for the driving state is obtained. The initial prediction result of the driving state can be a distracted driving state or a non - distracted driving state. The initial prediction result for the driving state of the vehicle based on the physiological state data of the driver can be predicted through a deep learning network or a machine learning method. Based on the initial prediction result of the driver and the vehicle driving data, a target prediction result for the driving state is obtained. The target prediction result can include the prediction result of the vehicle's dangerous driving. Combining the initial prediction result of the driving state and the vehicle driving data to predict whether the vehicle is in dangerous driving improves the accuracy and efficiency of the prediction. At the same time, when it is predicted that the vehicle is about to be in dangerous driving, it warns or reminds the distracted driver to improve driving safety.

[0050] Figure 2 This is a schematic diagram of the driving state prediction system provided by the embodiment of the present application.

[0051] As Figure 2 shown, the driving state prediction system of the present invention can be divided into two major modules. The first module includes a model training module. The model training module is used to train a deep learning model, and the deep learning model can be a cognitive distraction classification model. The first module can be used to obtain the physiological state data under different distraction states as training set data for model training to obtain a cognitive distraction classification model. In one example, the physiological state data includes at least one of eye movement data and electroencephalogram data. Of course, the physiological state data can also include other types of data, such as the movement data of specific body parts (hands, feet, etc.). The movement data of specific body parts can reflect whether the driver is distracted to a certain extent. For example, when the driver operates a mobile phone with the hand during driving, the movement data of the hand at this time reflects that the driver is distracted.

[0052] The second module includes an online cognitive distraction detection module, and the second module consists of at least three systems: a data acquisition system, a data processing system, and a model prediction system. The data acquisition system is used to collect electroencephalogram (EEG) data, eye movement data, and vehicle data (i.e., vehicle driving data) in real time. The data processing system is used to process the collected data in real time, which is divided into processing EEG data, eye movement data, and vehicle driving data. Subsequently, the processing results of EEG data and eye movement data are sent to the trained cognitive distraction classification model in the first module for prediction to obtain the cognitive distraction classification result (i.e., the initial prediction result). The collected vehicle driving data is processed to obtain the vehicle position prediction error rate. Combining the vehicle position prediction error rate, in the model prediction system, the predicted driving state is obtained according to certain rules.

[0053] When making a prediction based on EEG data and eye movement data, the EEG data and eye movement data can be pre-processed first.

[0054] As an example, based on the physiological state data of the driver, the initial prediction result for the driving state is obtained, including: inputting at least one of the eye movement data and EEG data into the trained deep learning model for prediction to obtain the initial prediction result.

[0055] Specifically, the physiological state information of the driver is collected. The physiological state information is used to determine whether the driver is distracted. Therefore, the collected physiological state information includes at least one of the eye movement data and EEG data. At least one of the eye movement data and EEG data is input into the trained deep learning model. The deep learning model extracts features from the input eye movement data and EEG data and predicts the driving state to obtain the initial prediction result. The initial prediction result is used to indicate whether the driver is distracted.

[0056] As an example, the trained deep learning model includes a trained cognitive distraction classification model. When training this model, first, a training data set is obtained. The eye movement data and EEG data of one or more drivers can be collected respectively when they are performing a driving dual-task experiment and a driving single-task. At the initial stage of establishing each classification model, a large amount of data under different scenarios needs to be accumulated. In this application, the eye movement data and EEG data information of one driver when performing dual-task and single-task, as well as the eye movement data and EEG data information of multiple different drivers in dual-task and single-task states, can be collected respectively. Among them, when collecting data of one driver, it can be collected multiple times and recorded. Further, the data collected multiple times can be compared, and the data information with large errors can be removed, and the others can be averaged and saved. Dual-task means that the driver performs a distraction task while performing the driving task, and single-task means that the driver only performs the driving task.

[0057] The acquisition data information corresponding to multiple different drivers also needs to be collected multiple times at different times. For example, collect data information at set time intervals, or collect data information at different time periods. For example, it can be at 6 am, 10 am, 1 pm, 3 pm, 7 pm, 12 am, 2 am, etc. in the morning. Record the eye movement data and electroencephalogram data in different states according to different times. Because everyone's distraction state is different at different times, when collecting the training data of the model, it is necessary to consider comprehensively to increase the completeness and accuracy of the data, so as to better improve the accuracy of the data set and the prediction accuracy of the driving state, thereby improving driving safety.

[0058] The prediction results of training the deep learning model can include label results. The data label is 1 in the dual-task state, representing cognitive distraction, and the data label is 0 in the single-task driving state, representing no cognitive distraction. After training, a trained cognitive distraction classification model is obtained. When making an online prediction, at least one of the eye movement data and the electroencephalogram data is input into the trained cognitive distraction classification model for prediction to obtain an initial prediction result.

[0059] As an example, the initial prediction result can also include label results. The data label is 1 in the dual-task state, representing cognitive distraction, and the data label is 0 in the single-task driving state, representing no cognitive distraction.

[0060] As an example, preprocessing the eye movement data includes at least one of the following: removing data with abnormal pupil size changes, blocked pupils, and artifacts at the pupil edge in the eye movement data; removing data representing gaze line deviation in the eye movement data; removing data with the line of sight outside the region of interest in the eye movement data; removing data representing a saccade angular velocity greater than a preset angular velocity in the eye movement data.

[0061] Specifically, the collected eye movement data needs to be preprocessed before being input into the trained deep learning model, which can improve the prediction accuracy of the model. The eye movement data can include the original pupil size time series, the gaze position information of the user collected by the eye tracker and the on-site camera. Abnormal pupil size changes in the eye movement data include non-positive pupil sizes, due to missing eye targets, eyelid occlusion, or blinking.

[0062] As an example, to remove the eye movement data where the line of sight is outside the region of interest, a Velocity Threshold Identification fixation filter (I-VT) can be used to extract the features of fixations and saccades. The threshold can be set to 30 degrees. That is, when the angular velocity of a saccade is greater than the threshold of 30 degrees, it indicates interference caused by head turning, and this part of the interference needs to be removed to identify the eye movement behaviors of fixations and saccades.

[0063] As an example, to remove the eye movement data where the saccade angular velocity is greater than the preset angular velocity, an angular region can be determined as the region of interest with the distance between the center point of the road in front of the vehicle and the eyes as the diameter or radius and the eyes as the center. That is, the line connecting the driver's eyes to the center point of the road ahead is used as the angle bisector, and the region within a 16° angle is the region of interest. The data outside the region of interest is deleted, and only the fixation points within the region of interest are recorded.

[0064] As an example, preprocessing of electroencephalogram (EEG) data includes at least one of the following: averaging multi-channel EEG data to obtain an average value and retaining the difference between the EEG data of each channel and the average value; filtering the EEG data and retaining the data within a preset frequency band; removing the interference data caused by blinking or body movement in the EEG data; extracting features from the EEG data to obtain power spectral density feature data for a specific frequency band.

[0065] Specifically, the collected EEG data also needs to be preprocessed before being input into the trained deep learning model. The EEG data can be collected through a head-mounted electroencephalograph.

[0066] As an example, as Figure 3 shown in the process of preprocessing EEG data, whole-brain average reference is performed on the original EEG data, that is, the data collected from multiple channels is averaged to obtain an average value, and the difference between the data of each channel and the average value can be used as the whole-brain average reference data. Then, band-stop filtering is performed, and the filtering frequency can be set to 0.5 - 45 Hz, that is, the signals outside this frequency band are removed. Then, independent component analysis (ICA) is used to identify and remove the interference data caused by blinking or body movement. Since blinking will interfere with the EEG signal and body movement will also interfere with the EEG signal, both need to be removed. Finally, feature extraction is performed through power spectral density analysis to obtain power spectral density feature data for a specific frequency band. The power spectral density features that can be extracted include theta waves of 3 - 7 Hz, alpha waves of 8 - 12 Hz, and beta waves of 13 - 30 Hz.

[0067] As an example, the initial prediction result characterizes whether the driver is distracted during driving, and the target prediction result characterizes whether there is a vehicle driving deviation caused by the driver's distraction during driving.

[0068] Specifically, at least one of the preprocessed eye movement data and electroencephalogram data is input into the trained deep learning model to obtain the initial prediction result, which can characterize whether the driver is distracted during driving. For example, if the driver is distracted, the output label is 1, and if the driver is not distracted, the output label is 0. To further improve the prediction accuracy, the vehicle driving data can also be combined for judgment. By processing the vehicle driving data, it can be determined whether the vehicle has deviated. For example, combining the initial prediction result and the vehicle driving data, the target prediction result for the driving state can be obtained, which can characterize whether there is a vehicle driving deviation caused by the driver's distraction during driving.

[0069] As an example, the vehicle driving data includes lateral speed offset information; based on the initial prediction result and the vehicle driving data, the target prediction result for the driving state is obtained, including: when both the initial prediction result and the lateral speed offset information indicate that the driving state is a risky driving, it is determined that the driver's driving state is a risky driving.

[0070] Specifically, when the vehicle is driving normally, its lateral speed offset is relatively stable or small. For example, when the vehicle is driving straight, the theoretical value of the lateral speed offset of the vehicle is zero. When an emergency occurs, the driver instinctively turns the steering wheel, and at this time, the lateral speed offset of the vehicle is large. Therefore, by analyzing the lateral speed offset information of the vehicle, it can be judged whether the driving state is a risky driving. The initial prediction result characterizes whether the driver is distracted during driving. If the driver is distracted, it is a risky driving situation, otherwise it is a non-risky driving situation. Combining the initial prediction result and the vehicle driving data, the target prediction result for the driving state is obtained. When the initial prediction result is a risky driving and the result of judging the lateral speed offset information is also a risky driving, the target prediction result for the driving state is a risky driving, and this risky driving is caused by the driver's distraction resulting in a vehicle driving deviation.

[0071] In another example, the vehicle driving data includes the lateral speed value and the lateral speed offset information. Compared with determining the risky driving situation based on the initial prediction result and the lateral speed offset information, the lateral speed value can be further considered to determine the risky driving situation to improve the accuracy. As Figure 4 shown, based on the initial prediction result and the vehicle driving data, the target prediction result for the driving state of the vehicle is obtained, including S401 - S402.

[0072] S401. When the lateral speed value is less than or equal to the preset speed threshold, determine the driving state of the driver based on the initial prediction result and the lateral speed offset information.

[0073] S402. When both the initial prediction result and the lateral speed offset information indicate that the driving state is a risky driving state, determine that the driving state of the driver is a risky driving state.

[0074] Specifically, driving behaviors such as vehicle turning around and lane changing will also affect the lateral speed offset information. Therefore, when the lateral speed offset is large due to normal vehicle turning around or lane changing, it can be determined that there is no driving risk. Specifically, when the lateral speed value is less than or equal to the preset speed threshold, that is, the lateral speed value is less than or equal to the lateral speed value during vehicle turning around or lane changing, if both the initial prediction result and the lateral speed offset information indicate that the driving state is a risky driving state, determine that the driving state of the driver is a risky driving state.

[0075] As an example, the lateral speed value can be directly measured by a speed detector on the vehicle, the preset speed threshold can be set to 2 m / s, and the lateral speed can be denoted as The lateral speed value is the absolute value of the lateral speed and is denoted as When the lateral speed value is less than or equal to 2 m / s, it indicates that the vehicle is not in a driving situation of turning around or lane changing. At this time, if both the initial prediction result and the lateral speed offset information indicate that the driving state is a risky driving state, determine that the driving state is a risky driving state.

[0076] As an example, when the lateral speed value is greater than the preset speed threshold and the initial prediction result is a risky driving state, the driving state is not a risky driving state. For example, when the lateral speed value is greater than 2 m / s, it is not monitored as cognitive distraction.

[0077] As an example, the vehicle driving data includes the true value of kinematic information, and the lateral speed offset information includes the first lateral speed offset value; the driving state prediction method for the driver based on multi-modal data further includes S501 - S503.

[0078] S501. Based on the true value of kinematic information, predict the predicted value of kinematic information.

[0079] S502. Based on the true value of kinematic information and the predicted value of kinematic information, obtain the prediction error of kinematic information.

[0080] S503. Based on the prediction error, obtain the first lateral speed offset value.

[0081] Specifically, the vehicle driving data includes the true values of kinematic information, which can be directly obtained through acquisition. The kinematic information may include information such as the vehicle's position, speed, acceleration, etc. Based on the true values of the kinematic information, predicted values of the kinematic information are obtained. The predicted values of the kinematic information can correspond one by one to the true values of the kinematic information. For example, the information such as the position, speed, acceleration, etc. of the vehicle at the next moment is predicted one by one to obtain the corresponding predicted values. Based on the true values of the kinematic information and the predicted values of the kinematic information, the prediction error of the kinematic information (the prediction error includes the vehicle position prediction error rate mentioned above) is obtained. The predicted value at the next moment obtained by prediction is compared with the true value obtained by acquisition to obtain the prediction error of the kinematic information. Based on the prediction error, the first lateral velocity offset value is obtained.

[0082] As an example, all the data of the vehicle driving data need to be synchronously acquired, that is, the time starting points of all the data are the same. The true values of the kinematic information can be sampled at a certain frequency band of Hertz frequency, for example, it can be 10HZ. t denotes the true value of the vehicle kinematic information collected at time t, and each Z t denotes the vector of q-dimensional (such as including multiple dimensions such as position, speed, acceleration, etc.) vehicle driving data measured at time t. Z t-1 denotes all q-dimensional vehicle kinematic information up to time t-1. Through the state space model, according to Z t-1 (the true values of all q-dimensional vehicle kinematic information up to time t-1), the predicted value of the vehicle kinematic information at time t is predicted, and the predicted value of the kinematic information at time t obtained by prediction is denoted as ( denotes the prediction results of q dimensions such as position, speed, acceleration, etc.). Based on the true values of the kinematic information and the predicted values of the kinematic information, the prediction error of the kinematic information is obtained. The lane position prediction error can be used as the prediction error evaluation scale between the true value and the predicted value, or the average lane position prediction error can be used as the prediction error evaluation scale between the true value and the predicted value. The average lane position prediction error represents the average value of the lane position prediction errors at multiple moments within the time window. If the lane position prediction error is small, the driving behavior has not deviated greatly; if the lane position prediction error is large, the driving behavior has deviated greatly and dangerous driving may occur. Based on the lane position prediction error, the first lateral velocity offset value is calculated.

[0083] As an example, as Figure 6 shown, based on the true values of the kinematic information, the predicted values of the kinematic information are obtained, including S601-S602.

[0084] S601. Use multiple spatial state models to make predictions based on the true value of kinematic information, and obtain multiple predicted values corresponding one-to-one to the multiple spatial state models.

[0085] S602. Based on the weights corresponding to the multiple spatial state models, perform weighted calculation on the multiple predicted values to obtain the predicted value of kinematic information.

[0086] Specifically, the spatial state model can be used to predict the true value of kinematic information. The spatial state model has multiple driving motion patterns \(c\in\{1,\ldots,C\}\), where \(C\) is the number of spatial state models. For example, \(C = 6\). The 6 spatial state models include Constant Velocity (CV), Constant Acceleration (CA), Constant Turn Rate and Velocity (CTRV), Constant Turn Rate and Acceleration (CTRA), Constant Steering Angle and Velocity (CSAV), and Constant Curvature and Acceleration model (CCA). Fuse the multiple spatial state models to obtain multiple predicted values corresponding one-to-one to the multiple spatial state models. For example, perform weighted calculation on the multiple predicted values to obtain the predicted value of kinematic information.

[0087] As an example, the Autonomous Multiple Model (AMM) algorithm can be used to fuse the multiple spatial state models. The fused predicted value is obtained through formula (1):

[0088]

[0089] Formula (1) means that each of the 6 spatial state models outputs a predicted value, obtaining 6 predicted values, and perform weighted average on the 6 predicted values to obtain the fused predicted value.

[0090] Among them, is the normalized weight of each state space model at time \(t\). The weight of each state space model can be set according to requirements. For example, the weights of each state space model may be different under different road conditions. Perform weighted calculation on the predicted values obtained by each state space model to obtain the predicted value of kinematic information. It should be noted that the predicted value of kinematic information is the predicted value of multiple dimensions (position, velocity, acceleration, etc., \(q\) dimensions).

[0091] The lane position prediction error can be used as the prediction error evaluation scale between the true value and the predicted value, or the average lane position prediction error can be used as the prediction error evaluation scale between the true value and the predicted value.

[0092] As an example, denote the lane position prediction error as ε t , which is the predicted value of the kinematic information and the difference between the true value Z of the kinematic information t . ε t is the lane position prediction error in multiple dimensions (position, speed, acceleration, etc., q dimensions) at time t.

[0093] As an example, denote the average lane position prediction error as In the i-th time window, a time window includes multiple moments. If a time window includes the n-th moment, the average lane position prediction error

[0094]

[0095] As an example, based on the lane position prediction error, the first lateral speed offset value is calculated, and the first lateral speed offset value is calculated by formula (2):

[0096]

[0097]

[0098]

[0099]

[0100] where σ f is the standard deviation of the lane position prediction error ε t in the fusion state. In the fusion state, ε t is the weighted average of the lane position deviations in different states, and the weights of each state are the weights corresponding to the C state space models in the above text. The weighted average of the lane position deviations in C different states gives in formula (2). is the first lateral speed offset value of the i-th time window in the positive direction, is the first lateral speed offset value of the 0-th time window in the positive direction, and this value is zero. is the first lateral speed offset value of the i-th time window in the negative direction,

[0101] Using multiple spatial state models to predict the true value of the collected kinematic information, obtaining the predicted value of the kinematic information, and obtaining the prediction error of the kinematics based on the difference between the true value of the kinematic information and the predicted value of the kinematic information. Based on the prediction error, the first lateral velocity offset is calculated, and whether the driving state is a risky driving can be determined through the first lateral velocity offset.

[0102] As an example, the lateral velocity offset information indicates that the driving state of the driver is risky driving, including: when the first lateral velocity offset value is greater than the preset velocity offset threshold, it is determined that the lateral velocity offset information indicates that the driving state of the driver is risky driving.

[0103] Specifically, the first horizontal velocity offset value includes the lateral velocity offset values in two directions, which are the first lateral velocity offset value in the positive direction and the first lateral velocity offset value in the negative direction When the first lateral velocity offset value is greater than the preset velocity offset threshold, it includes the first lateral velocity offset value in the positive direction and the first lateral velocity offset value in the negative direction in the case that at least one of them is greater than the preset velocity offset threshold. The preset velocity offset threshold can be set to 8. When the first lateral velocity offset value in the positive direction and the first lateral velocity offset value in the negative direction at least one of them is greater than 8, it is determined that the lateral velocity offset information indicates that the driving state is risky driving.

[0104] In addition to obtaining the lateral velocity offset information through the prediction error, the lateral velocity offset information can also be obtained through the lateral velocity.

[0105] As an example, as Figure 7 shown, the vehicle driving data includes the lateral velocity value, and the lateral velocity offset information includes the second lateral velocity offset value; the driving state prediction method of the driver based on multi-modal data further includes S701 - S702.

[0106] S701, based on the lateral velocity value, determine the initial lateral velocity offset value.

[0107] S702, based on the initial lateral velocity offset value and the smoothing coefficient, determine the mean value of the initial lateral velocity offset value as the second lateral velocity offset value.

[0108] Specifically, the lateral velocity can be directly measured, and the lateral velocity value is the absolute value of the lateral velocity. An initial lateral velocity offset value is obtained based on the lateral velocity value, and the initial lateral velocity offset value is obtained from the lateral velocity values within multiple windows. For example, the initial time window can be 1 s, and within the time window \(i\in\{1,\ldots,N\}\), the initial lateral velocity offset value is as shown in Equation (3):

[0109]

[0110] Next, according to the initial lateral velocity offset value and the smoothing coefficient, the mean value of the initial lateral velocity offset value is determined, as shown in Equation (4):

[0111]

[0112] where, is the mean value of the initial lateral velocity offset value, is the initial lateral velocity offset value within the \(i\)-th time window, is the initial lateral velocity offset value within the \((i - 1)\)-th time window, and \(\lambda\) is the smoothing parameter. The mean value of the initial lateral velocity offset value can be used as the second lateral velocity offset value. Whether the driving state is a risky driving is judged according to the second lateral velocity offset value.

[0113] As an example, the lateral velocity offset information indicates that the driving state of the driver is risky driving, including: when the second lateral velocity offset value is within the risk confidence interval, it is determined that the lateral velocity offset information indicates that the driving state of the driver is risky driving.

[0114] Specifically, a risk confidence interval can be set. If the second lateral velocity offset value is 0 or relatively small, it indicates a small degree of lateral offset, that is, there is no risky driving. If the second lateral velocity offset value falls within the risk confidence interval, it indicates a large degree of lateral offset, that is, there is risky driving.

[0115] As an example, setting the risk confidence interval \(1-\alpha\) can be expressed by Equation (4):

[0116]

[0117]

[0118] where \(L\) is the \((1-\alpha / 2)\)-th percentile of the standard normal distribution, and \(\sigma\) 0 is the standard deviation. \(\alpha\) can be set to 0.0027, and \(L = 3\). UCL i represents the upper limit of the risk confidence interval, and LCL i represents the lower limit of the confidence interval. By judging the second lateral velocity offset value Whether it is within the risk confidence interval to determine whether it is a risky driving during the driving process. If the second lateral speed offset value is outside the risk confidence interval, that is, there is no risky driving. If the second lateral speed offset value falls within the risk confidence interval, it indicates a large degree of lateral offset, that is, there is risky driving.

[0119] By inputting the physiological state data of the driver into the trained cognitive distraction classification model, an initial prediction result is obtained. The initial prediction result characterizes whether the driver is distracted. If the driver is distracted, it is a dangerous driving. If the driver is not distracted, it is not a dangerous driving. This result is a preliminary prediction result and needs to be further combined with the analysis of vehicle driving data. The vehicle driving data includes lateral speed offset information, and the driving state of the vehicle is determined to be risky driving by calculating the lateral speed offset amount. The present invention proposes two example methods for calculating the lateral speed offset amount, which will not be elaborated here. Combining the initial prediction result and the vehicle driving data, a target prediction result is obtained. The target prediction result characterizes whether the vehicle driving deviation is caused by the driver's distraction during the driving process.

[0120] As an example, in the case where both the initial prediction result and the lateral speed offset information indicate that the driving state is risky driving of the driver, determining that the driving state of the driver is risky driving includes: in the case where the initial prediction result indicates that the driving state of the driver in the target number of time periods is risky driving, and the lateral speed offset information indicates that the driving state of the driver is risky driving, determining the risky driving level based on the target number, where the target number is positively correlated with the number of levels of the risky driving level, and the risk degree of the level with a larger number is greater than that of the level with a smaller number.

[0121] Specifically, after the preprocessed eye movement data and electroencephalogram data are input in real time, through the cognitive distraction classification model, a classification result label is obtained. For example, the classification result label of 1 indicates that the driving state is risky driving, and the classification label of 0 indicates that the driving state is not risky driving. The lateral speed offset information is in units of time windows. For example, the eye movement data and electroencephalogram data of each time window are input into the distraction classification model to obtain the driving state within the corresponding time window time period. According to the number of time periods (i.e., time windows) in which the driving state is risky driving, the risky driving is classified. Obviously, the more the number of time periods in which the driving state is risky driving, the higher the level of risky driving. That is, the target number is positively correlated with the number of levels of the risky driving level, and the risk degree of the level with a larger number is greater than that of the level with a smaller number.

[0122] As an example, the risk level can be divided into three levels according to the target quantity. Denote the target quantity as Di. If Di = 1: it means that the predicted label result corresponding to the ith time window is 1 (the physiological state data in the ith time window indicates that the driver is in a distracted state), and when the lateral speed offset information indicates that the driving state is a risky driving state, the ADAS system warns of the first level of risky driving; Di = 2: it means that the predicted label results corresponding to the ith time window and the i - 1th time window are 1 (the physiological state data in the ith time window and the i - 1th time window indicate that the driver is in a distracted state), and when the lateral speed offset information indicates that the driving state is a risky driving state, the ADAS system warns of the second level of risky driving; Di = 3: it means that the predicted label results corresponding to the ith time window and the i - 1th, i - 2th time windows are 1 (the physiological state data in the ith time window and the i - 1th, i - 2th time windows indicate that the driver is in a distracted state), and when the lateral speed offset information indicates that the driving state is a risky driving state, the ADAS system warns of the third level of risky driving. The third level is greater than the second level, and the second level is greater than the first level. The higher the level, the greater the risk. It can be understood that more levels of risk levels can be set according to the actual situation.

[0123] As an example, the method for predicting the driving state of a driver based on multi-modal data further includes: outputting risk warning information in a corresponding risk warning manner according to the risky driving level.

[0124] Specifically, different risk warning methods can be output according to the risky driving level. For example, the greater the risky driving level, the stronger the warning information output, such as the louder the warning sound.

[0125] It should be noted that in order to avoid continuous warnings for the same distraction behavior, when reaching the highest level 3, the first lateral speed offset value and the second lateral speed offset value are reset to 0.

[0126] It should also be noted that when the data is unstable or lost, when the sensor data such as electroencephalogram and eye movement transmission is a null value, Di can be set to 0.

[0127] The present invention obtains an initial prediction result for the driving state based on the physiological state data of the driver; and obtains a target prediction result for the driving state based on the initial prediction result and the vehicle driving data. The method for predicting the driving state of a driver based on multi-modal data of the present invention can combine the physiological state data of the driver and the vehicle driving data to predict the state of the driver, so as to warn or remind distracted drivers, thereby improving driving safety.

[0128] In another example, the driving mode of the vehicle can also be switched based on the target prediction result.

[0129] Exemplarily, after obtaining the target prediction result, the driving mode of the vehicle can be switched based on the target prediction result. For example, the initial driving mode of the vehicle is the normal driving mode or the assisted driving mode. In the normal driving mode, the driver operates the vehicle, and in the assisted driving mode, the driver can assist in driving on the basis of the autonomous driving mode. It can be seen that switching the driving mode of the vehicle based on the target prediction result can improve driving safety.

[0130] In one example, when the target prediction result indicates that the vehicle is in a risky driving state, the driving mode of the vehicle is switched to the autonomous driving mode. For example, if the target prediction result indicates that the vehicle is in a risky driving state, it further shows that the driver's driving state is not good when driving based on the normal driving mode or the assisted driving mode, such as being distracted or fatigued. At this time, in order to ensure driving safety, it is necessary to switch to the autonomous driving mode.

[0131] In one example, after switching the driving mode of the vehicle to the autonomous driving mode, during the process of the vehicle driving based on the autonomous driving mode, the comfort information of the driver in the autonomous driving mode can be detected in real time. The comfort information indicates whether the driver trusts the autonomous driving mode, whether the driver can relax physically and mentally in the autonomous driving mode, and whether the mood is stable.

[0132] In the case where the comfort information indicates that the comfort level of the driver is less than the preset level, it means that the driver does not trust the autonomous driving mode enough, and the driver is in a state of tension, fatigue, and unstable mood in the autonomous driving mode. At this time, the driving mode of the vehicle can be switched from the autonomous driving mode to the assisted driving mode or the normal driving mode to improve the comfort of the driver.

[0133] In one example, when the vehicle is driving based on the autonomous driving mode, at least one of the physiological information of the driver and the vehicle driving information can be collected, and based on at least one of the physiological information and the vehicle driving information, the comfort information of the driver can be detected. The physiological information includes, for example, the driver's electroencephalogram data, emotional data, fatigue data, skin conductance data, etc. The physiological information can be processed to obtain the comfort of the driver. For example, it can be known whether the driver is tense or fatigued. The vehicle driving information includes, for example, the vehicle steering angle, speed, acceleration, whether to change lanes during driving, etc. The vehicle driving information will affect the comfort of the driver to a certain extent. For example, when the vehicle makes a sharp turn, drives fast, or changes lanes frequently, the driver may be in a tense state. Therefore, when detecting the comfort information of the driver, the physiological information and the vehicle driving information can be combined.

[0134] The embodiment of the present application provides a driving state prediction device for a driver based on multimodal data.

[0135] As shown Figure 8 in FIG. 340, a driving state prediction device 800 for a driver based on multimodal data includes: a first obtaining module 810, configured to obtain an initial prediction result for the driving state of the driver based on the physiological state data of the driver; and a second obtaining module 820, configured to obtain a target prediction result for the driving state of the vehicle based on the initial prediction result and the vehicle driving data.

[0136] Exemplarily, the vehicle driving data includes lateral speed offset information; the second obtaining module 820 is configured to: when both the initial prediction result and the lateral speed offset information indicate that the driving state is a risky driving state, determine that the driving state of the driver is a risky driving state.

[0137] Exemplarily, the vehicle driving data includes a lateral speed value and lateral speed offset information; the second obtaining module 820 is configured to: when the lateral speed value is less than or equal to a preset speed threshold, determine the driving state of the driver based on the initial prediction result and the lateral speed offset information; and when both the initial prediction result and the lateral speed offset information indicate that the driving state is a risky driving state, determine that the driving state of the driver is a risky driving state.

[0138] Exemplarily, the vehicle driving data includes the true value of kinematic information, and the lateral speed offset information includes a first lateral speed offset value; the driving state prediction device 800 for a driver based on multimodal data further includes: a prediction module, configured to predict a predicted value of kinematic information based on the true value of kinematic information; a third obtaining module, configured to obtain a prediction error of kinematic information based on the true value of kinematic information and the predicted value of kinematic information; and a fourth obtaining module, configured to obtain the first lateral speed offset value based on the prediction error.

[0139] Exemplarily, that the lateral speed offset information indicates that the driving state of the driver is a risky driving state includes: when the first lateral speed offset value is greater than a preset speed offset threshold, determining that the lateral speed offset information indicates that the driving state of the driver is a risky driving state.

[0140] Exemplarily, predicting a predicted value of kinematic information based on the true value of kinematic information includes: using a plurality of spatial state models to perform prediction based on the true value of kinematic information to obtain a plurality of predicted values corresponding one-to-one to the plurality of spatial state models; and performing weighted calculation on the plurality of predicted values based on the weights corresponding to the plurality of spatial state models to obtain the predicted value of kinematic information.

[0141] Exemplarily, the vehicle driving data further includes a lateral speed value, and the lateral speed offset information includes a second lateral speed offset value; the driver's driving state prediction device 800 based on multimodal data further includes: a first determination module, configured to determine an initial lateral speed offset value based on the lateral speed value; and a second determination module, configured to determine the mean value of the initial lateral speed offset value as the second lateral speed offset value based on the initial lateral speed offset value and a smoothing coefficient.

[0142] Exemplarily, the lateral speed offset information indicating that the driver's driving state is risky driving includes: in the case where the second lateral speed offset value is within a risk confidence interval, determining that the lateral speed offset information indicates that the driver's driving state is risky driving.

[0143] Exemplarily, in the case where both the initial prediction result and the lateral speed offset information indicate that the driving state is risky driving, determining that the driver's driving state is risky driving includes: in the case where the initial prediction result indicates that the driver's driving state is risky driving within a target number of time periods, and the lateral speed offset information indicates that the driver's driving state is risky driving, determining a risky driving level based on the target number, where the target number is positively correlated with the number of levels of the risky driving level, and the risk degree of a larger level is greater than that of a smaller level.

[0144] Exemplarily, the driver's driving state prediction device 800 based on multimodal data further includes an output module, configured to output risk prompt information in a corresponding risk prompt manner according to the risky driving level.

[0145] Exemplarily, the physiological state data includes at least one of eye movement data and electroencephalogram data; the first acquisition module 820 is further configured to input at least one of the eye movement data and the electroencephalogram data into a trained deep learning model for prediction to obtain an initial prediction result.

[0146] Exemplarily, preprocessing the eye movement data includes at least one of the following: removing data with abnormal pupil size changes, occluded pupils, and artifacts at the pupil edge in the eye movement data; removing data indicating gaze line deviation in the eye movement data; removing data where the line of sight is outside the region of interest in the eye movement data; removing data indicating that the saccade angular velocity is greater than a preset angular velocity in the eye movement data.

[0147] Exemplarily, preprocessing the electroencephalogram data includes at least one of the following: averaging multi-channel electroencephalogram data to obtain a mean value, and retaining the difference between the electroencephalogram data of each channel and the mean value; filtering the electroencephalogram data to retain data in a preset frequency band; removing interference data caused by blinking or body movement in the electroencephalogram data; extracting features from the electroencephalogram data to obtain power spectral density feature data for a specific frequency band.

[0148] Exemplarily, the initial prediction result characterizes whether the driver is distracted during driving, and the target prediction result characterizes whether there is a vehicle driving deviation caused by the driver's distraction during driving.

[0149] Exemplarily, the driving state prediction device 800 for the driver based on multimodal data further includes: a first switching module, configured to switch the driving mode of the vehicle based on the target prediction result.

[0150] Exemplarily, the first switching module is further configured to switch the driving mode of the vehicle to the autonomous driving mode when the target prediction result characterizes that the vehicle is in a risky driving state.

[0151] Exemplarily, the driving state prediction device 800 for the driver based on multimodal data further includes: a detection module and a second switching module. The detection module is configured to detect the comfort information of the driver when the vehicle is driving based on the autonomous driving mode; the second switching module is configured to switch the driving mode of the vehicle from the autonomous driving mode to the assisted driving mode or the conventional driving mode when the comfort information characterizes that the comfort level of the driver is less than the preset level.

[0152] Exemplarily, the detection module is further configured to: when the vehicle is driving based on the autonomous driving mode, collect at least one of the physiological information of the driver and the vehicle driving information; and detect the comfort information of the driver based on at least one of the physiological information and the vehicle driving information.

[0153] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.

[0154] Figure 9 It is a block diagram of an electronic device provided by an embodiment of the present application.

[0155] An embodiment of the present application provides an electronic device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the driving state prediction method for the driver based on multimodal data as described above is implemented.

[0156] As Figure 9 shown, for the sake of easy understanding, an embodiment of the present application shows a specific electronic device 900.

[0157] The electronic device 900 is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as, personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0158] As Figure 9 shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0159] A plurality of components in the electronic device 900 are connected to the I / O interface 905. The plurality of components include: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0160] The computing unit 901 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 executes the various methods described above, such as the method for predicting the driving state of a driver based on multimodal data. For example, in some embodiments, the method for predicting the driving state of a driver based on multimodal data can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, any one or more steps of the method for predicting the driving state of a driver based on multimodal data described above can be executed. Alternatively, in other embodiments, the computing unit 901 can be configured to execute the method for predicting the driving state of a driver based on multimodal data in any other suitable manner (e.g., by means of firmware).

[0161] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatus, or devices. For the purposes of this application, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which a program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0162] It should be understood that each part of this application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0163] In the description of this application, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this application, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0164] In the description of this application, it should be understood that the orientation or positional relationships indicated by terms such as "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. are based on the orientation or positional relationships shown in the drawings, and are only for the convenience of describing this application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation of this application.

[0165] In addition, the terms "first", "second", etc. used in the embodiments of this application are only for descriptive purposes, and cannot be understood as indicating or implying relative importance, or implicitly indicating the quantity of the technical features indicated in this embodiment. Thus, the features defined with terms such as "first", "second", etc. in the embodiments of this application can explicitly or implicitly indicate that at least one such feature is included in this embodiment. In the description of this application, the meaning of the word "plurality" is at least two or more, such as two, three, four, etc., unless otherwise specifically defined in the embodiment.

[0166] In this application, unless otherwise clearly specified or limited in the embodiments, the terms "installed", "connected", "coupled" and "fixed" etc. appearing in the embodiments shall be understood in a broad sense. For example, the connection can be a fixed connection, a detachable connection, or integrated. Understandably, it can also be a mechanical connection, an electrical connection, etc.; of course, it can also be directly connected, or indirectly connected through an intermediate medium, or it can be the communication inside two elements, or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to the specific implementation circumstances.

[0167] In this application, unless otherwise clearly specified and limited, the first feature being "on" or "under" the second feature may be that the first and second features are in direct contact, or the first and second features are indirectly in contact through an intermediate medium. Moreover, the first feature being "above", "over" and "on top of" the second feature may be that the first feature is directly above or obliquely above the second feature, or merely indicates that the first feature has a higher horizontal height than the second feature. The first feature being "under", "below" and "beneath" the second feature may be that the first feature is directly below or obliquely below the second feature, or merely indicates that the first feature has a lower horizontal height than the second feature.

[0168] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present application.

Claims

1. A method for predicting the driving state of a driver based on multimodal data, characterized in that, the method includes: obtaining an initial prediction result of the driving state of the driver based on the physiological state data of the driver, wherein the physiological state data includes at least one of eye movement data and electroencephalogram data; and obtaining a target prediction result of the driving state of the vehicle based on the initial prediction result and the vehicle driving data, wherein the vehicle driving data includes a lateral speed value and lateral speed offset information; The obtaining the target prediction result of the driving state of the vehicle based on the initial prediction result and the vehicle driving data includes: determining whether the current lateral driving speed of the vehicle is less than or equal to a preset speed threshold, and the preset speed threshold is used to represent the lateral speed threshold for the vehicle to make a U-turn or change lanes; in the case where the lateral speed value is less than or equal to the preset speed threshold, determining the driving state of the driver based on the initial prediction result and the lateral speed offset information; and in the case where both the initial prediction result and the lateral speed offset information indicate that the driving state is a risky driving, determining that the driving state of the driver is a risky driving; the initial prediction result characterizes whether the driver is distracted during driving, and the target prediction result characterizes whether there is a driving deviation of the vehicle due to the driver's distraction during driving; the lateral speed offset information includes a second lateral speed offset value; the method further includes: determining an initial lateral speed offset value based on the lateral speed value; and determining the mean value of the initial lateral speed offset value as the second lateral speed offset value based on the initial lateral speed offset value and the smoothing coefficient; The lateral speed offset information indicating that the driving state is the risky driving of the driver includes: in the case where the second lateral speed offset value is within the risk confidence interval, determining that the lateral speed offset information indicates that the driving state of the driver is a risky driving.

2. The method according to claim 1, characterized in that, the vehicle driving data includes the true value of the kinematic information, and the lateral speed offset information includes a first lateral speed offset value; the method further includes: predicting a predicted value of the kinematic information based on the true value of the kinematic information; obtaining a prediction error of the kinematic information based on the true value of the kinematic information and the predicted value of the kinematic information; and obtaining the first lateral speed offset value based on the prediction error.

3. The method according to claim 2, characterized in that, The lateral speed offset information indicating that the driving state of the driver is a risky driving includes: in the case where the first lateral speed offset value is greater than a preset speed offset threshold, determining that the lateral speed offset information indicates that the driving state of the driver is a risky driving.

4. The method according to claim 2, characterized in that, The predicting a predicted value of the kinematic information based on the true value of the kinematic information includes: Predict using multiple spatial state models based on the true value of the kinematic information to obtain multiple predicted values corresponding one by one to the multiple spatial state models; Based on the weights corresponding to the multiple spatial state models, perform weighted calculation on the multiple predicted values to obtain the predicted value of the kinematic information.

5. The method according to claim 1, wherein, when both the initial prediction result and the lateral velocity offset information indicate that the driving state is a risky driving state of the driver, determining that the driving state of the driver is a risky driving state includes: when the initial prediction result indicates that the driving state of the driver is a risky driving state within a target number of time periods, and the lateral velocity offset information indicates that the driving state of the driver is a risky driving state, determining a risky driving level based on the target number, wherein, the target number has a positive correlation with the number of levels of the risky driving level, and the risk degree of a larger level is greater than that of a smaller level.

6. The method according to claim 5, wherein, the method further includes: Output risk warning information in a corresponding risk warning manner according to the risky driving level.

7. The method according to claim 1, wherein, obtaining an initial prediction result for the driving state of the driver based on the physiological state data of the driver includes: Input at least one of the eye movement data and the electroencephalogram data into a trained deep learning model for prediction to obtain the initial prediction result.

8. The method according to claim 7, wherein, Preprocessing the eye movement data includes at least one of the following: Removing data with abnormal pupil size changes, pupil occlusion, and artifacts at the pupil edge in the eye movement data; Removing data representing gaze line deviation in the eye movement data; Removing data with the line of sight outside the region of interest in the eye movement data; Removing data representing a saccade angular velocity greater than a preset angular velocity in the eye movement data.

9. The method according to claim 7, wherein, Preprocessing the electroencephalogram data includes at least one of the following: Averaging multi-channel electroencephalogram data to obtain an average value, and retaining the difference between the electroencephalogram data of each channel and the average value; Filtering the electroencephalogram data to retain data in a preset frequency band; Removing interference data caused by blinking or body movement in the electroencephalogram data; Performing feature extraction on the electroencephalogram data to obtain power spectral density feature data for a specific frequency band.

10. The method according to claim 1, wherein, the method further includes: Switching the driving mode of the vehicle based on the target prediction result.

11. The method according to claim 10, wherein, switching the driving mode of the vehicle based on the target prediction result includes: When the target prediction result indicates that the vehicle is in a risky driving state, switching the driving mode of the vehicle to an autonomous driving mode.

12. The method according to claim 11, wherein, the method further includes: When the vehicle is traveling based on the autonomous driving mode, detect the comfort information of the driver; and When the comfort information indicates that the comfort level of the driver is less than a preset level, switch the driving mode of the vehicle from the autonomous driving mode to the assisted driving mode or the conventional driving mode.

13. The method according to claim 12,[[]]END]] wherein,[[]]END]] The step of detecting the comfort information of the driver when the vehicle is traveling based on the autonomous driving mode includes:[[]]END]] When the vehicle is traveling based on the autonomous driving mode, collect at least one of the physiological information of the driver and the vehicle driving information;[[]]END]] Based on at least one of the physiological information and the vehicle driving information, detect the comfort information of the driver.[[]]END]] 14. A driving state prediction device for a driver based on multi-modal data, the device is applied to the method of any one of claims 1-13,[[]]END]] wherein,[[]]END]] The device includes:[[]]END]] A first obtaining module, configured to obtain an initial prediction result for the driving state of the driver based on the physiological state data of the driver; and[[]]END]] A second obtaining module, configured to obtain a target prediction result for the driving state of the vehicle based on the initial prediction result and the vehicle driving data.[[]]END]] 15. An electronic device, including a memory and a processor, the memory stores a computer program,[[]]END]] wherein,[[]]END]] When the processor executes the computer program, the steps of the method described in any one of claims 1-13 are implemented.[[]]END]] 16. A computer-readable storage medium, on which a computer program is stored,[[]]END]] wherein,[[]]END]] When the computer program is executed by a processor, the steps of the method described in any one of claims 1-13 are implemented.[[]]END]]

Images