A driver cognitive state identification method and system based on multi-source information fusion

Abstract

The present invention proposes a driver cognitive state identification method and system based on multi-source information fusion, which includes seven parts: driver distraction experiment, data preprocessing, feature extraction, standardization processing, model training, feature importance analysis and cognitive state determination. The model identification result is the driver's three-level cognitive state, including concentration, mild distraction and severe distraction. The present invention integrates vehicle information, driver eye movement information and driver EEG information, and takes into account road characteristics and individual driver characteristics. The trained model realizes a relatively accurate identification of the driver's cognitive state. The extracted features that have a greater impact on the identification results can be used in real vehicle driver state detection.

Description

Technical Field

[0001] The present invention relates to the field of driver status monitoring, and in particular to a driver cognitive status identification method and system based on multi-source information fusion. Background Art

[0002] In recent years, with the popularization of smart terminals and the improvement of in-vehicle information systems, distracted driving behavior has become more and more common. Distracted driving behavior reduces the driver's ability to respond to the traffic environment and will lead to wrong decisions and operations. For example, when a driver is driving and making a phone call, his attention will be diverted from the driving task to the call, resulting in an increase in the driver's reaction time and a decrease in the ability to operate the vehicle, which makes it very easy for traffic accidents to occur in the event of emergencies. Therefore, if the driver's cognitive state can be accurately identified, and when dangerous behavior is detected, the driver can be given necessary reminders to prompt the driver to return to normal driving from a distracted state, traffic accidents can be effectively avoided.

[0003] A certain existing technology discloses a driving system and judgment method for detecting and reminding drivers of distraction, which identifies the driver's cognitive state by collecting signals such as steering wheel angle, vehicle driving path, visual gaze position, and steering wheel grip. The method has a relatively simple structure and is easy to implement, and can achieve the purpose of detection. However, the driver's distraction state is judged by whether the defined distracted driving parameter index reaches the threshold, which is less persuasive and has certain limitations in its application. Another existing technology also discloses a driver cognitive distraction monitoring method based on multi-source information fusion. The data used in this method include steering wheel angle, vehicle speed, driver's head orientation and gaze point position, and a support vector machine algorithm is used to monitor the driver's cognitive distraction state in real time. However, this method only collects vehicle signals and driver's eye movement signals, and has not yet considered the manifestation of changes in the driver's cognitive state in the EEG signal, and has not considered individual characteristics and road characteristics, so it has poor universality for diverse roads and different drivers.

[0004] In addition, both of the above methods only propose methods for identifying the driver's cognitive state, and do not further study which factors have a stronger ability to represent the driver's cognitive state and which features are more suitable for identifying the driver's distracted state. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a driver cognitive state identification method and system based on multi-source information fusion with high universality for diverse roads and different drivers.

[0006] To achieve the above objectives, the present invention adopts the following technical solutions: In a first aspect, a method for identifying a driver's cognitive state based on multi-source information fusion is provided, comprising:

[0007] Establish a cognitive state recognition model;

[0008] Acquire and process vehicle data and driver's physiological data simultaneously in the real vehicle;

[0009] The processed data is input into the established cognitive state identification model to determine the driver's cognitive state.

[0010] Furthermore, the establishing of the cognitive state recognition model includes:

[0011] Conduct driver distraction experiments on a driving simulator and simultaneously obtain experimental vehicle data and driver physiological data;

[0012] Preprocessing the acquired vehicle data and driver's physiological data;

[0013] Extract features from the preprocessed vehicle data and the driver's physiological data, and extract features of the experimental road and individual features of the driver;

[0014] Standardize the extracted features;

[0015] According to the standardized features, a cognitive state recognition model is established, and the features with the top several average SHAP values ​​among the standardized features are screened out.

[0016] Furthermore, the driver distraction experiment is conducted on the driving simulator, and the vehicle data of the experiment and the physiological data of the driver are simultaneously obtained, including:

[0017] Calibrate the eye tracker and position the electrode cap of the semi-dry electrode EEG system;

[0018] The drivers conducted a preliminary experiment on a driving simulator based on a calibrated eye tracker and a positioned semi-dry electrode EEG system;

[0019] Based on the calibrated eye tracker and the positioned semi-dry electrode EEG system, the driver conducted a formal experiment on a driving simulator, and simultaneously obtained the corresponding experimental vehicle data as well as the driver's eye movement data and EEG data.

[0020] Furthermore, the formal experiment includes a first experiment, a second experiment and a third experiment, and each formal experiment enables a driver to operate a driving simulator and drive from a starting point to an end point on a given road;

[0021] In the first experiment, no distraction was applied to the driver; in the second experiment, the driver was subjected to distraction of the first set of tasks; in the third experiment, the driver was subjected to distraction of the second set of tasks.

[0022] Further, the characteristics of the vehicle data include a steering wheel angle standard deviation, a steering wheel angle mean, a steering wheel angle maximum, a steering wheel angle minimum, a lateral deviation standard deviation, a lateral deviation mean, a lateral deviation maximum, a lateral deviation minimum, a longitudinal vehicle speed standard deviation, a longitudinal vehicle speed mean, a longitudinal vehicle speed maximum, and a longitudinal vehicle speed minimum;

[0023] The features of the eye movement data include the horizontal moving speed of the left eye gaze point, the vertical moving speed of the left eye gaze point, the forward and backward moving speed of the left eye gaze point, the horizontal moving speed of the right eye gaze point, the vertical moving speed of the right eye gaze point, the forward and backward moving speed of the right eye gaze point, the average pupil diameter of the left eye, the average pupil diameter of the right eye, the mean of the angular velocity of eyeball rotation, the standard deviation of the angular velocity of eyeball rotation, the mean of the horizontal position of the left eye gaze point, the mean of the vertical position of the left eye gaze point, the mean of the horizontal position of the right eye gaze point, the mean of the vertical position of the right eye gaze point, the standard deviation of the horizontal position of the left eye gaze point, the standard deviation of the vertical position of the left eye gaze point, the standard deviation of the horizontal position of the right eye gaze point, the standard deviation of the vertical position of the right eye gaze point, the eye saccade frequency and the gaze frequency;

[0024] The characteristics of the EEG data include the ratio of the average power spectral density of alpha waves to beta waves, and the sum of the average power spectral density of alpha waves and beta waves;

[0025] The experimental road is characterized by road curvature;

[0026] The individual characteristic of the driver is driving experience.

[0027] Furthermore, the preprocessing of the acquired vehicle data and the driver's physiological data includes:

[0028] Eliminate the acquired vehicle data and the driver's eye movement data and EEG data;

[0029] Synchronize the time axis of the vehicle data after data removal and the driver's eye movement data and EEG data;

[0030] The vehicle data after time axis synchronization as well as the driver's eye movement data and EEG data are downsampled.

[0031] Furthermore, the cognitive state recognition model is established based on the standardized features, and the features with the top several average SHAP values ​​in the standardized features are screened out, including:

[0032] According to the standardized features, the LightGBM algorithm is used for model training to establish a cognitive state recognition model, where the input of the cognitive state recognition model is the standardized features and the output is the driver's cognitive state;

[0033] According to the standardized features, feature importance analysis based on SHAP value is performed to screen out the features with the top several average SHAP values ​​among the standardized features.

[0034] In a second aspect, a driver cognitive state identification system based on multi-source information fusion is provided, comprising:

[0035] A model building module, used to build a cognitive state recognition model;

[0036] The real vehicle data acquisition module simultaneously acquires vehicle data and the driver's physiological data in the real vehicle and processes them;

[0037] The cognitive state determination module is used to input the processed data into the established cognitive state identification model to determine the cognitive state of the driver.

[0038] In a third aspect, a processing device is provided, comprising computer program instructions, wherein the computer program instructions, when executed by the processing device, are used to implement the steps corresponding to the above-mentioned driver cognitive state identification method based on multi-source information fusion.

[0039] In a fourth aspect, a computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, are used to implement the steps corresponding to the above-mentioned driver cognitive state identification method based on multi-source information fusion.

[0040] The present invention has the following advantages due to the adoption of the above technical solution:

[0041] 1. The present invention simultaneously collects vehicle data, driver eye movement data, and driver EEG data in the driver distraction experiment, and additionally considers the impact of road characteristics and driver individual characteristics on the driver's cognitive state. The above five types of characteristics are integrated to identify the driver's cognitive state, and an identification model with better training results and stronger universality can be obtained.

[0042] 2. The present invention uses SHAP value to analyze the importance of features and extracts 25 features that have an important impact on the identification results of the driver's cognitive state, which can improve the interpretability of the model and has certain inspiring significance for studying the subjective and objective characterization of the driver's cognitive state. The extracted more important features can be used for real-time detection of the driver's state in a real vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The following will analyze the benefits of the invention in detail by describing the preferred embodiments. The accompanying drawings are only used to illustrate the preferred embodiments and are not intended to limit the invention. In the accompanying drawings:

[0044] Figure 1 It is a schematic diagram of a method flow provided by an embodiment of the present invention;

[0045] Figure 2 It is a flowchart of the formal experimental stage of the method provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0046] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0047] It should be understood that the terms used in the text are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used in the text may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that alternative steps may be used.

[0048] The driver cognitive state identification method and system based on multi-source information fusion provided by the embodiment of the present invention integrates vehicle data, driver eye movement data, and driver EEG data, while taking into account road characteristics and driver individual characteristics. The trained model realizes a relatively accurate identification of the driver's cognitive state, and the extracted features that have a greater impact on the identification results can be used in real-time detection of the driver's state.

[0049] Example 1

[0050] like Figure 1 As shown, this embodiment provides a method for identifying a driver's cognitive state based on multi-source information fusion, comprising the following steps:

[0051] 1) Conduct a driver distraction experiment on a driving simulator, and simultaneously obtain the experimental vehicle data and the driver's physiological data. The vehicle data includes steering wheel angle, lateral deviation, and longitudinal vehicle speed, and the physiological data includes the driver's eye movement data and 32-channel EEG data, specifically:

[0052] 1.1) Calibrate the eye tracker and position the electrode cap of the semi-dry electrode EEG system.

[0053] Specifically, the driver's eye movement data is collected using an eye tracker, and the driver's EEG data is collected using a 32-channel semi-dry electrode EEG system (including an electrode cap and an EEG signal amplifier).

[0054] Specifically, the calibration process of the eye tracker includes: the driver looks at the continuously moving red dot on the screen with both eyes; the experimenter evaluates the calibration result by the distance between the average fixation point and the target fixation point; if the calibration result does not meet the preset requirements or there is no calibration feedback, recalibration is performed.

[0055] Specifically, the positioning process of the electrode cap includes: measuring the driver's head size, finding the midpoint Cz between the Nz point (the depression between the driver's eyes and nose) and the Iz point (the bulge directly behind the driver's head); the driver wears the electrode cap based on the found midpoint Cz to facilitate the positioning of the electrode. After the Cz point is aligned, the other positions of the electrode cap will also be aligned accordingly.

[0056] Furthermore, the naming and position distribution of the electrodes of the electrode cap are consistent with the 10-20 standard system proposed by the International Federation of Brain Mapping Societies.

[0057] 1.2) The driver conducts a preliminary experiment on a driving simulator based on the calibrated eye tracker and the positioned semi-dry electrode EEG system:

[0058] The driver manipulates the steering wheel of the driving simulator according to natural driving habits, drives along the given road on the screen, keeps the vehicle in the center of the path, and drives to the end of the road.

[0059] Specifically, the driver is asked to practice driving for about 10 minutes to familiarize himself with the operation of the driving simulator and the experimental road. During the driving process, the driver is required to always maintain a comfortable and natural driving posture, while ensuring that the electrode cap fixed on the driver's head does not interfere with the driving simulator seat and other components in space.

[0060] 1.3) If Figure 2As shown, the driver was asked to conduct a formal experiment on the driving simulator, and the vehicle data, the driver's eye movement data and 32-channel EEG data were obtained at the same time. The formal experiment included the first experiment, the second experiment and the third experiment. In each formal experiment, the driver operated the driving simulator and drove from the starting point to the end point on the given road on the screen. The timing was measured from the time the vehicle started to drive, and each experiment took about 7 minutes and 30 seconds.

[0061] Specifically, the first experiment is a baseline experiment for comparison, in which no distraction interference is imposed on the driver, and the driver's cognitive state is focused driving. In the second experiment, the driver is interfered with by the first set of tasks. In the third experiment, the driver is interfered with by the second set of tasks.

[0062] Specifically, both the first and second group of tasks include: calling the driver within 0-1min, and the driver uses a Bluetooth headset to answer the call; when the time is 2min, 3min, 4min, 5min and 6min, 5 questions are asked to the driver in turn, and the driver needs to answer the questions within 1min (i.e. before the next question is asked). Further, the content of the first and second group of tasks can be shown in Table 1 below. The content of the first group of tasks belongs to the category of daily conversations and is relatively low in difficulty; the questions of the second group of tasks are four arithmetic operations, which are more difficult than the first group of tasks and require the driver to shift more attention to the task of answering questions. Therefore, from the first experiment to the second experiment, and then to the third experiment, the driver's distraction level deepens successively, and correspondingly, the difficulty of maintaining a better driving performance also increases successively. In the second experiment, considering that the entire process of hearing questions and answering questions after the driver answers the phone belongs to the execution of the driving sub-task, it is believed that the driver's cognitive state is slightly distracted from 1 to 7min. In the third experiment, considering that the entire process of the driver hearing the question and answering the question after answering the phone is a more complex driving sub-task, it is believed that the driver's cognitive state is severely distracted during 1 to 7 minutes.

[0063] Table 1: Questions for the first and second set of tasks

[0064]

[0065] 2) Preprocess the vehicle data and driver's physiological data obtained from the experiment, specifically:

[0066] 2.1) The vehicle data, driver’s eye movement data and EEG data obtained in the experiment are eliminated.

[0067] Specifically, at the beginning of each experiment, due to the delay effect of information transmission between software and hardware, when the driver's physiological data begins to be recorded, the vehicle signal cannot be recorded synchronously. Therefore, it is necessary to exclude the physiological data of the first period of time (about 2 seconds) of each experiment.

[0068] 2.2) Synchronize the time axis of the vehicle data after data removal, the driver's eye movement data and EEG data.

[0069] Specifically, in each experiment, the time when vehicle data starts to be recorded is set as time zero.

[0070] 2.3) Downsample the vehicle data, driver's eye movement data and EEG data after time axis synchronization. The original sampling frequency of each data is shown in Table 2 below.

[0071] Table 2: Original sampling frequency of the data

[0072]

[0073] For vehicle data, the sampling frequency of the steering wheel angle is reduced from 1000Hz to 100Hz, consistent with the frequency of lateral deviation and vehicle speed. For EEG data, according to the sampling theorem, a sampling rate of more than twice the highest frequency of the studied frequency band can restore the waveform within the frequency (EEG data usually requires at least 3 to 4 times the sampling rate). The EEG frequency range studied in the embodiment of the present invention is 1Hz to 30Hz, so in order to improve computing efficiency, the frequency of EEG data is reduced to 100Hz. The frequency of eye movement data remains unchanged.

[0074] 3) Extract features from the preprocessed vehicle data and driver's physiological data, and extract features of the experimental road and individual features of the driver. The calculation cycle of vehicle data features and driver physiological data features is 1 second.

[0075] Specifically, the characteristics of vehicle data include steering wheel angle standard deviation, steering wheel angle mean, steering wheel angle maximum, steering wheel angle minimum, lateral deviation standard deviation, lateral deviation mean, lateral deviation maximum, lateral deviation minimum, longitudinal vehicle speed standard deviation, longitudinal vehicle speed mean, longitudinal vehicle speed maximum and longitudinal vehicle speed minimum.

[0076] Specifically, the characteristics of eye movement data include the horizontal movement speed of the left eye gaze point, the vertical movement speed of the left eye gaze point, the forward and backward movement speed of the left eye gaze point, the horizontal movement speed of the right eye gaze point, the vertical movement speed of the right eye gaze point, the forward and backward movement speed of the right eye gaze point, the average pupil diameter of the left eye, the average pupil diameter of the right eye, the mean angular velocity of the eye rotation, the standard deviation of the angular velocity of the eye rotation, the mean horizontal position of the left eye gaze point, the mean vertical position of the left eye gaze point, the mean horizontal position of the right eye gaze point, the mean vertical position of the right eye gaze point, the standard deviation of the horizontal position of the left eye gaze point, the standard deviation of the vertical position of the left eye gaze point, the standard deviation of the horizontal position of the right eye gaze point, the standard deviation of the vertical position of the right eye gaze point, the eye saccade frequency and the gaze frequency.

[0077] Specifically, the characteristics of the EEG data include the ratio of the average power spectral density of α waves to that of β waves, and the sum of the average power spectral density of α waves and that of β waves.

[0078] Specifically, the characteristic of the experimental road is the road curvature, and the individual characteristic of the driver is the driving experience.

[0079] In summary, 36 features are obtained as shown in Table 3 below.

[0080] Table 3: 36 features obtained after feature extraction

[0081]

[0082]

[0083] 4) The extracted features are standardized to eliminate the impact caused by the inconsistent dimensions of the feature data, while retaining the relative size relationship of each data within the same feature.

[0084] Specifically, the Min-max normalization method is used to linearly transform the extracted features so that they are mapped to the [0,1] interval. The conversion formula is:

[0085]

[0086] Where x is a feature; norm is the result obtained after feature x is standardized; max is the maximum value in feature x; min is the minimum value in feature x.

[0087] 5) Based on the standardized features, the LightGBM (Light Gradient Boosting Machine) algorithm is used for model training to establish a cognitive state recognition model, specifically:

[0088] 5.1) Using the four-fold cross-validation method, the data obtained in step 4) is divided into four parts, of which three parts are used as training sets and one part is used as a validation set.

[0089] 5.2) Based on the LightGBM algorithm, the training set and validation set were used for model training and model validation respectively, and a cognitive state recognition model was obtained. The input of the cognitive state recognition model was the standardized features in Table 3, and the output was the driver's three-level cognitive state, namely, concentration, mild distraction, and severe distraction.

[0090] 6) Based on the standardized features, perform feature importance analysis based on SHAP (SHapley Additive exPlanations) values ​​to screen out the top features with the highest average SHAP values ​​from large to small among the standardized features.

[0091] Specifically, a feature importance analysis based on SHAP value is performed to screen out the top 25 features with the top average SHAP values ​​among the standardized features, and the 25 features are divided into three groups according to the size of the average SHAP value. The SHAP value interval of the first group of features is SHAP>0.8, the SHAP value interval of the second group of features is SHAP=0.4~0.8, and the SHAP value interval of the third group of features is SHAP=0.1~0.4. The importance of the three groups of features to the driver state identification model decreases in turn.

[0092] More specifically, the first group of features includes eye saccade frequency, steering wheel angle standard deviation, lateral deviation standard deviation, driver's driving experience, left eye gaze point horizontal position standard deviation, right eye gaze point horizontal position standard deviation, left eye average pupil diameter and right eye average pupil diameter, the second group of features includes the minimum steering wheel angle, the maximum lateral deviation, the mean vertical position of the left eye gaze point, the mean vertical position of the right eye gaze point and the mean horizontal position of the left eye gaze point, and the third group of features includes road curvature, the maximum steering wheel angle, the mean steering wheel angle, the mean lateral deviation, the minimum lateral deviation, the mean horizontal position of the right eye gaze point, the standard deviation of the left eye gaze point vertical position, the standard deviation of the right eye gaze point vertical position, the horizontal movement speed of the left eye gaze point, the horizontal movement speed of the right eye gaze point, the mean angular velocity of the eye rotation and the standard deviation of the angular velocity of the eye rotation, as shown in Table 4 below.

[0093] Table 4: Top 25 features

[0094]

[0095] 7) In the actual vehicle, vehicle data and the driver's physiological data are collected simultaneously, the first set of features in step 6) is calculated, or the first set of features and the second set of features in step 6) are calculated, and the feature data calculated in real time is input into the established cognitive state recognition model to determine the cognitive state of the actual vehicle driver.

[0096] The effectiveness of the LightGBM algorithm used in the present invention is described in detail below through specific examples:

[0097] The support vector machine (SVM) algorithm, K-nearest neighbors (KNN) algorithm, decision tree algorithm, quadratic discriminant analysis (QDA) algorithm, random forest algorithm, extra trees algorithm, CatBoost (Categorical Boosting) algorithm and XGBoost (eXtreme Gradient Boosting) algorithm are used in turn to train the model and obtain the corresponding model.

[0098] The eight models obtained after training with the eight algorithms are compared with the model obtained after training with the LightGBM algorithm.

[0099] Specifically, five performance indicators are used to evaluate the quality of the model, including accuracy, precision, recall, F1 value and AUC value. The training results of the above nine algorithms are shown in Table 5 below.

[0100] Table 5: Comparison of training results of LightGBM and other eight algorithms

[0101] algorithm Accuracy Accuracy Recall F1 value AUC value SVM 0.771 0.767 0.767 0.767 0.89 KNN 0.740 0.737 0.737 0.737 0.86 Decision Tree 0.762 0.758 0.743 0.750 0.91 QDA (Quadratic Discriminant Analysis) 0.700 0.701 0.695 0.698 0.87 Random Forest 0.772 0.767 0.767 0.767 0.91 Extra Trees 0.775 0.758 0.759 0.759 0.89 CatBoost (Categorical Boosting) 0.778 0.775 0.774 0.774 0.92 XGBoost(eXtreme Gradient Boosting) 0.780 0.776 0.774 0.775 0.92 LightGBM 0.782 0.779 0.777 0.778 0.92

[0102] It can be seen that the LightGBM algorithm used in the present invention is higher than the other eight algorithms in accuracy, precision, recall rate, F1 value and AUC value. The driver cognitive state identification based on the LightGBM algorithm is more accurate.

[0103] Example 2

[0104] This embodiment provides a driver cognitive state identification system based on multi-source information fusion, including:

[0105] The experimental module is used to conduct driver distraction experiments and obtain experimental vehicle data and driver physiological data, including equipment calibration, pre-experiments and formal experiments.

[0106] The data preprocessing module is used to preprocess the experimental data, including data elimination, time axis synchronization and data downsampling.

[0107] The feature extraction module is used to extract features from the preprocessed vehicle data and the driver's physiological data, and to extract road features and individual features of the driver.

[0108] The feature standardization processing module is used to standardize the extracted features.

[0109] The model training module is used to train the model based on the LightGBM algorithm according to the standardized features and establish a cognitive state recognition model.

[0110] The feature importance analysis module is used to perform feature importance analysis based on SHAP values ​​according to the standardized features, and screen out the top several features with the largest average SHAP values.

[0111] The cognitive state determination module is used to input the processed data into the established cognitive state identification model to determine the actual cognitive state of the driver.

[0112] Example 3

[0113] This embodiment provides a processing device corresponding to the driver cognitive state identification method based on multi-source information fusion provided in this embodiment 1. The processing device can be a processing device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of embodiment 1.

[0114] The processing device includes a processor, a memory, a communication interface and a bus, and the processor, the memory and the communication interface are connected through the bus to complete mutual communication. The memory stores a computer program that can be run on the processing device, and when the processing device runs the computer program, the driver cognitive state identification method based on multi-source information fusion provided in this embodiment 1 is executed.

[0115] In some implementations, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0116] In some other implementations, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.

[0117] Example 4

[0118] This embodiment provides a computer program product corresponding to the driver cognitive state identification method based on multi-source information fusion provided in this embodiment 1. The computer program product may include a computer-readable storage medium on which computer-readable program instructions for executing the driver cognitive state identification method based on multi-source information fusion described in this embodiment 1 are loaded.

[0119] Computer readable storage media can be tangible devices that hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof.

[0120] The above embodiments are only used to illustrate the present invention, wherein the structure, connection mode and manufacturing process of each component may be changed. Any equivalent transformations and improvements based on the technical solution of the present invention should not be excluded from the protection scope of the present invention.

Claims

1. A method for identifying driver cognitive state based on multi-source information fusion, characterized in that: include: Establish a cognitive state recognition model; Acquire and process vehicle data and driver's physiological data simultaneously in the real vehicle; Inputting the processed data into the established cognitive state identification model to determine the driver's cognitive state; The establishing of the cognitive state recognition model comprises: Conduct driver distraction experiments on a driving simulator and obtain experimental vehicle data and driver physiological data, including: Calibrate the eye tracker and position the electrode cap of the semi-dry electrode EEG system; The drivers conducted a preliminary experiment on a driving simulator based on a calibrated eye tracker and a positioned semi-dry electrode EEG system; The driver conducts a formal experiment on a driving simulator based on a calibrated eye tracker and a positioned semi-dry electrode EEG system, and simultaneously obtains vehicle data of the corresponding experiment as well as the driver's eye movement data and EEG data. The formal experiment includes a first experiment, a second experiment, and a third experiment. Each formal experiment enables the driver to operate the driving simulator and drive from a starting point to an end point on a given road. In the first experiment, no distraction is imposed on the driver. In the second experiment, the driver is interfered with by a first set of tasks. In the third experiment, the driver is interfered with by a second set of tasks. Preprocessing the acquired vehicle data and driver's physiological data; Extract features from the preprocessed vehicle data and the driver's physiological data, and extract features of the experimental road and individual features of the driver; Standardize the extracted features; Based on the standardized features, a cognitive state recognition model was established, and the features with the top average SHAP values ​​among the standardized features were selected, including: According to the standardized features, the LightGBM algorithm is used for model training to establish a cognitive state recognition model, where the input of the cognitive state recognition model is the standardized features and the output is the driver's cognitive state; According to the standardized features, feature importance analysis based on SHAP value is performed to screen out the features with the top several average SHAP values ​​among the standardized features.

2. The method for identifying driver cognitive state based on multi-source information fusion according to claim 1, characterized in that: The characteristics of the vehicle data include steering wheel angle standard deviation, steering wheel angle mean, steering wheel angle maximum, steering wheel angle minimum, lateral deviation standard deviation, lateral deviation mean, lateral deviation maximum, lateral deviation minimum, longitudinal vehicle speed standard deviation, longitudinal vehicle speed mean, longitudinal vehicle speed maximum and longitudinal vehicle speed minimum; The features of the eye movement data include the horizontal moving speed of the left eye gaze point, the vertical moving speed of the left eye gaze point, the forward and backward moving speed of the left eye gaze point, the horizontal moving speed of the right eye gaze point, the vertical moving speed of the right eye gaze point, the forward and backward moving speed of the right eye gaze point, the average pupil diameter of the left eye, the average pupil diameter of the right eye, the mean of the angular velocity of eyeball rotation, the standard deviation of the angular velocity of eyeball rotation, the mean of the horizontal position of the left eye gaze point, the mean of the vertical position of the left eye gaze point, the mean of the horizontal position of the right eye gaze point, the mean of the vertical position of the right eye gaze point, the standard deviation of the horizontal position of the left eye gaze point, the standard deviation of the vertical position of the left eye gaze point, the standard deviation of the horizontal position of the right eye gaze point, the standard deviation of the vertical position of the right eye gaze point, the eye saccade frequency and the gaze frequency; The characteristics of the EEG data include Wave and The ratio of the average power spectral density of the wave, Wave and The sum of the average power spectral densities of the waves; The experimental road is characterized by road curvature; The individual characteristic of the driver is driving experience.

3. The driver cognitive state identification method based on multi-source information fusion according to claim 1, characterized in that: The preprocessing of the acquired vehicle data and the driver's physiological data includes: Eliminate the acquired vehicle data and the driver's eye movement data and EEG data; Synchronize the time axis of the vehicle data after data removal and the driver's eye movement data and EEG data; The vehicle data after time axis synchronization as well as the driver's eye movement data and EEG data are downsampled.

4. A driver cognitive state identification system based on multi-source information fusion, characterized in that: include: A model building module, used to build a cognitive state recognition model; The real vehicle data acquisition module simultaneously acquires vehicle data and the driver's physiological data in the real vehicle and processes them; A cognitive state determination module, used to input the processed data into an established cognitive state identification model to determine the cognitive state of the driver; The establishing of the cognitive state recognition model comprises: Conduct driver distraction experiments on a driving simulator and obtain experimental vehicle data and driver physiological data, including: Calibrate the eye tracker and position the electrode cap of the semi-dry electrode EEG system; The drivers conducted a preliminary experiment on a driving simulator based on a calibrated eye tracker and a positioned semi-dry electrode EEG system; The driver conducts a formal experiment on a driving simulator based on a calibrated eye tracker and a positioned semi-dry electrode EEG system, and simultaneously obtains vehicle data of the corresponding experiment as well as the driver's eye movement data and EEG data. The formal experiment includes a first experiment, a second experiment, and a third experiment. Each formal experiment enables the driver to operate the driving simulator and drive from a starting point to an end point on a given road. In the first experiment, no distraction is imposed on the driver. In the second experiment, the driver is interfered with by a first set of tasks. In the third experiment, the driver is interfered with by a second set of tasks. Preprocessing the acquired vehicle data and driver's physiological data; Extract features from the preprocessed vehicle data and the driver's physiological data, and extract features of the experimental road and individual features of the driver; Standardize the extracted features; Based on the standardized features, a cognitive state recognition model was established, and the features with the top average SHAP values ​​among the standardized features were selected, including: According to the standardized features, the LightGBM algorithm is used for model training to establish a cognitive state recognition model, where the input of the cognitive state recognition model is the standardized features and the output is the driver's cognitive state; According to the standardized features, feature importance analysis based on SHAP value is performed to screen out the features with the top several average SHAP values ​​among the standardized features.

5. A processing device, characterized in that: It includes computer program instructions, wherein the computer program instructions, when executed by a processing device, are used to implement the steps corresponding to the driver cognitive state identification method based on multi-source information fusion described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, wherein the computer program instructions, when executed by a processor, are used to implement the steps corresponding to the driver cognitive state identification method based on multi-source information fusion according to any one of claims 1 to 3.

Images